CHANGELOG.md 10.3 KB
Newer Older
1
# Changelog of pygeodyn
Loic Huder's avatar
Loic Huder committed
2

Mathieu Istas's avatar
Mathieu Istas committed
3
4
5
6
1.4.0 - 2021-03-23
------------------
*Cdf data is now handled, P_bb covariance matrix is updated at each analysis, Huber norm is implemented for better results in presence of outliers*

GitLab Runner's avatar
GitLab Runner committed
7
8
9
10
1.3.2 - 2020-04-01
------------------
Skipped analyses now set misfits to NaN. This fixes the crash of `run_algo` when no observation was available in analysis steps.

GitLab Runner's avatar
GitLab Runner committed
11
12
13
14
15
16
17
1.3.1 - 2020-02-03
------------------
Refactoring towards cleaner handling of PCA in augkf and zonkf:
- PCA algorithms are now in dedicated objects
- The Algo object is created through a factory function and will create the appropriate objects (Forecaster, Analysers)
- Updated documentation in consequence

GitLab Runner's avatar
GitLab Runner committed
18
19
20
21
22
23
24
25
26
27
28
29
30
1.3.0 - 2020-01-28
------------------
Code structure improvements:
- Added the Algo object that has an interface defined in GenericAlgo. All Algo deriving from GenericAlgo can be used in run.algorithm.
- forecast and analysis step were broken down in substeps (analyse_B, analyse_SV...)
- ZonkfAlgo derives from AugkfAlgo and overrides methods as needed (similarly for Forecaster and Analyser objects)
- Several config members were replaced by queries to avoid inconsistent changes of parameters
- Code clean-up
- Updated docstings and tests accordingly

Bug fixes:
- ZonKF algorithm is now working when using diagonal AR process (no PCA)

GitLab Runner's avatar
GitLab Runner committed
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
1.2.0 - 2019-11-28
------------------
**New features:**

* Implemented ZonKF algorithm: Augmented Kalman filter with zonal and non-zonal coefficients of U separately treated. This algorithm has two 'flavours' which be can changed by setting `coupled_zonals` in the configuration file:
    * `coupled`: The zonal and non-zonal coefficients are coupled through the forecast and by cross-covariances in the analyse.
    * `isolated`: The zonal and non-zonal coefficients are isolated. There is no forecast of the zonal coefficients nor cross-covariances with the non-zonal coefficients in the analyse.
    
While the algorithms are working, their scientific development is still a work in progress.

* Added the possibility to use COVOBS.x2 as observations
* Added the possiblity to use modelisation errors through `obs_mod_errors_dir` member in configuration file.

**Fixes:**

* Variance of COVOBS is now computed on *all* realisations

GitLab Runner's avatar
GitLab Runner committed
48
49
50
51
1.1.3 - 2019-10-14
------------------
Fixed Kalman gain matrix expression in AugKF documentation

GitLab Runner's avatar
GitLab Runner committed
52
53
54
55
1.1.2 - 2019-08-27
------------------
Updated pygeodyn_data with GOVO_2019 (August 2019) dataset. Minor improvements to doc and CR.

GitLab Runner's avatar
GitLab Runner committed
56
57
58
59
1.1.1 - 2019-08-06
------------------
Fixed the reading order for errors of GOVO (wrong format of _COV.obs files was assumed).

GitLab Runner's avatar
GitLab Runner committed
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
1.1.0 - 2019-07-05
------------------
**Major changes:**
- Numerous improvements to the documentation
    - Reorganised into RST files
    - Expanded with tutorials, complete description configuration parameters, ...
    - RST files are now picked up by Sphinx (`make_doc.sh` script) to generate a navigable documentation in HTML also deployed online
- Data in `pygeodyn/data` is now stored as a submodule (`pygeodyn_data`)

**In addition:**
- `init_algorithm` can now accept a configuration as a file or a dict
- `tmax` was renamed `Nth_legendre` in configuration files
- `TauU` and `TauE` are now stored as `timedelta64`
- Default value for `-m` was set to 20 (when calling `run_algo.py`)
- Added Continuous Release

Loic Huder's avatar
Loic Huder committed
76
77
78
79
80
81
82
83
84
85
86
87
88
1.0.0 - 2019-04-16
------------------

#### :confetti\_ball: **Release of pygeodyn** :confetti\_ball: 

**Numerous clean-ups for release:**
- Now packaging-ready: added a manifest, PyPI classfiers and a logo
- Reorganized tests and release tools
- Removed debug prints and unneccessary files

**New features:**
- Global seed is now saved as an attribute of the output hdf5 file

Loic Huder's avatar
Loic Huder committed
89
90
91
92
93
94
95
96
97
98
99
100
0.8.0
-----
Numerous improvements in view of future release:
* Renamed the project in **pygeodyn**
* Switch to semantic versioning

In addition :
* Added continous release tools (not used for now), CHANGELOG, version handling
* Clean-up of fortran source files
* Removed deprecated files
* Added benchmarks files for functional tests of the algorithm (only implemented for regular AugKF)

Loic Huder's avatar
Loic Huder committed
101
102
103
104
105
106
107
108
109
110
0.7
---
New features:
* PCA_U can now performed with energy normalisation (PCA is performed on core flow energy coefficents rather than directly on core flow coefficients)
* CoreState can now be initialised in several fashions:
    * *constant*: equal to the average prior (for MF, U and ER)
    * *normal*: normal draw around the average prior (variance = deviation of the prior) (for MF, U and ER)
    * *from_file*: equal to the corestate of a given file (*init_file*) at a given date (*init_date*) (for all measures)
* Added higher resolution COVOBS files (COVOBS_hd). The COVOBS loading method now computes the error matrix R directly from the files rather than using a `var_*` file.
* Configuration file can now be generated from a hdf5 file of a previous calculation
111

Loic Huder's avatar
Loic Huder committed
112
113
114
115
116
Other improvements:
* Matrix inversions required by misfits computation were refactored
* Misfits of the PCA_U are now saved
* Added tests
* Updated README.md
117

Loic Huder's avatar
Loic Huder committed
118
119
120
121
122
0.6
---
**Important bug fixes:**
- Fixed the time sampling for dense AR matrices that led to no variation in forecasts (#54)
- Observation error matrices are now properly set for COVOBS (#57)
123
124

In addition:
Loic Huder's avatar
Loic Huder committed
125
126
127
128
129
- Seed arg is now working (#56): it allows to have reproducible stochastic processes **when using the same number of MPI processes**. Noising of GO_VO observations does not use this global seed but prints the seeds used in debug logs.
- Misfits between the analysed states and the observations are now saved for analysis times (CoreState-like behaviour) (#58)
- Updated GO_VO data with 2018 data
- Updated guide for advanced users
- Added some tests (coverage: 79%)
130

Loic Huder's avatar
Loic Huder committed
131
0.5
132
---
Loic Huder's avatar
Loic Huder committed
133
134
:warning: **Important bug fix**:
* Fixed the covariance matrices of analysed states Z that was not scaled (led to inconsistent results when using GO/VO)
135

Loic Huder's avatar
Loic Huder committed
136
137
138
139
140
141
Improvements towards a usable package:
* Explained the arguments of `run_algo.py` in README.md
* Added an in-depth guide in `doc` to explain how to change the input data (priors/observations) and how the low-level features (forecast/analysis/_CoreState_) work (#50)
* Refactored the reading of observations/priors in separate functions that are called dynamically according to the types asked in config (#49)
* Redesigned the observations to have a single object _Observation_ handling data, operator and errors.
* Renamed _Forecasts_/_Analysis_ objects in _Forecaster_/_Analyser_
142

Loic Huder's avatar
Loic Huder committed
143
144
145
Testing:
* Implemented the use of hypothesis.strategies and composite strategies as input for several tests
* Added basic functional tests of `run_algo.py` and other tests (coverage: 78% !)
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168

0.4
---
:warning: **This version introduced a bug in analysis_step (P_zz not scaled) ! Use version 0.5 instead (same features but with bug fixed).**

Scientific improvements of the code:
* Implementation of a AR-1 process using dense drift matrices (#42 and #48)
* Implementation of several scaling methods for dense matrices: v1 is the same for diagonal and dense AR, v2 is the recommended for dense AR.


This was possible thanks to the following improvements in the code:
* Added high-resolution midpath data as hdf5 file
* Redesigned CoreState to be able to dynamically set the measures in it (#46)
* Implemented the possibility to do the computation on a principal componenent analysis of U (PCA_U)
* The covariance of analysed states Z is computed at each analyse and now enters in the expression of the covariance matrice P_zz used for the augmented state Kalman filter.

Other minor improvements:
* Changed magnetic field key from 'B' to 'MF'
* Spectral observations are now truncated by the number of asked coefficients for computation (Nb, Nsv)
* Asking for max degrees higher than the degree of priors will raise a ValueError (#41)
* Fixed docstrings of functions decorated by with_core_state_of_dimensions
* Added instructions to run the code on IST-OAR

Loic Huder's avatar
Loic Huder committed
169
0.3
170
---
Loic Huder's avatar
Loic Huder committed
171
172
173
174
175
176
Oops ! Computation of diffusion using cross-covariances only works for samples of the same statistical ensemble... As a consequence :
* Deprecation of the AugKF algorithm with diffusion computation (now 'legacy.augkf' in deprecated module)
* Deprecation of the CorDE algorithm (now in deprecated module as well)
* New implementation of the AugKF algorithm (formely AugKF_dpe) using master equation: $`\dot{b} = A(b)u + e`$ (DIFF is taken as a contribution of ER in:
  - AR-1 process on ER
  - Analysis on augmented state $`z= [u^Te^T] `$
177

Loic Huder's avatar
Loic Huder committed
178
179
180
Changes in observations:
* VO and GO data were added to the repository and can be used as observations for analyses.
* Dates in the algorithm are now handled with [NumPy datetime64](https://docs.scipy.org/doc/numpy/reference/arrays.datetime.html#arrays-dtypes-dateunits). This allows notably to have $`\Delta t`$ in months in the config files rather than floating dates that can have infinite decimals.
181
182


Loic Huder's avatar
Loic Huder committed
183
184
185
186
187
188
189
190
191
192
193
194
In addition:
* $`\Delta t`$ for Euler schemes and $`\Delta t_{forecasts}`$  are no longer different.
* Covariance matrices can be supplied as files in the prior folders. They will be computed from priors if not found.
* Deprecated files were put in a dedicated module that triggers a warning if imported.
* Various code improvements: clean-ups, docs written and tests added.

v0.2
----
Numerous improvements aiming to optimise the code, the most important being:
* The possibility to run forecasts in parallel using `mpi4py`
* The possibility to save results during computation in `hdf5` or `ASCII` format.
* Number of forecasts and analysis are now determined **ONLY** from the config file
195
196

In addition:
Loic Huder's avatar
Loic Huder committed
197
198
199
200
201
* Possibility to run analysis in parallel (not very interesting as the algorithm used in this case is way slower and scientifically equivalent...)
* Some clean-up: 
    - All raw data are now in the data folder
    - *forecast* folder containing the Fortran sources was renamed to *fortran*
* Launching an algorithm can now be done in one line in `run_algo.py` thanks to the encapsulation of the `run` module.
202
203


Loic Huder's avatar
Loic Huder committed
204
205
206
207
208
209
210
211
212
213
214
215
v0.1
----

Important milestones comprise:
- Par_forecast returns scientifically correct results.
- Fortran and Python implementation are interchangeable for the setup, the diffusion computation and the forecasts.

In addition:
- Implemented visualisation tools for trajectories and Lowes spectra.
- Possibility to save files in a format readable by [pygeodyn](https://gricad-gitlab.univ-grenoble-alpes.fr/Geodynamo/pygeodyn)
- Unitary and functional tests on around 35% of the code.
- Generated doc using Sphinx