Prerequisites/Numpy.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
+
+# <!-- TITLE --> [NP1] - A short introduction to Numpy
+<!-- DESC --> Numpy is an essential tool for the Scientific Python.
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Comprendre les grands principes de Numpy et son potentiel
+
+Note : This notebook is strongly inspired by the UGA Python Introduction Course
+See : **https://gricad-gitlab.univ-grenoble-alpes.fr/python-uga/py-training-2017**
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Numpy the beginning
+
+Code using `numpy` usually starts with the import statement
+
+%% Cell type:code id: tags:
+
+``` python
+import numpy as np
+```
+
+%% Cell type:markdown id: tags:
+
+NumPy provides the type `np.ndarray`. Such array are multidimensionnal sequences of homogeneous elements. They can be created for example with the commands:
+
+%% Cell type:code id: tags:
+
+``` python
+# from a list
+l = [10.0, 12.5, 15.0, 17.5, 20.0]
+np.array(l)
+```
+
+%% Output
+
+    array([10. , 12.5, 15. , 17.5, 20. ])
+
+%% Cell type:code id: tags:
+
+``` python
+# fast but the values can be anything
+np.empty(4)
+```
+
+%% Output
+
+    array([1.27880790e-316, 0.00000000e+000, 6.91986808e-310, 1.57378525e-316])
+
+%% Cell type:code id: tags:
+
+``` python
+# slower than np.empty but the values are all 0.
+np.zeros([2, 6])
+```
+
+%% Output
+
+    array([[0., 0., 0., 0., 0., 0.],
+           [0., 0., 0., 0., 0., 0.]])
+
+%% Cell type:code id: tags:
+
+``` python
+# multidimensional array
+a = np.ones([2, 3, 4])
+print(a.shape, a.size, a.dtype)
+a
+```
+
+%% Output
+
+    (2, 3, 4) 24 float64
+
+    array([[[1., 1., 1., 1.],
+            [1., 1., 1., 1.],
+            [1., 1., 1., 1.]],
+    
+           [[1., 1., 1., 1.],
+            [1., 1., 1., 1.],
+            [1., 1., 1., 1.]]])
+
+%% Cell type:code id: tags:
+
+``` python
+# like range but produce 1D numpy array
+np.arange(4)
+```
+
+%% Output
+
+    array([0, 1, 2, 3])
+
+%% Cell type:code id: tags:
+
+``` python
+# np.arange can produce arrays of floats
+np.arange(4.)
+```
+
+%% Output
+
+    array([0., 1., 2., 3.])
+
+%% Cell type:code id: tags:
+
+``` python
+# another convenient function to generate 1D arrays
+np.linspace(10, 20, 5)
+```
+
+%% Output
+
+    array([10. , 12.5, 15. , 17.5, 20. ])
+
+%% Cell type:markdown id: tags:
+
+A NumPy array can be easily converted to a Python list.
+
+%% Cell type:code id: tags:
+
+``` python
+a = np.linspace(10, 20 ,5)
+list(a)
+```
+
+%% Output
+
+    [10.0, 12.5, 15.0, 17.5, 20.0]
+
+%% Cell type:code id: tags:
+
+``` python
+# Or even better
+a.tolist()
+```
+
+%% Output
+
+    [10.0, 12.5, 15.0, 17.5, 20.0]
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Access elements
+
+Elements in a `numpy` array can be accessed using indexing and slicing in any dimension. It also offers the same functionalities available in Fortan or Matlab.
+
+### 2.1 - Indexes and slices
+For example, we can create an array `A` and perform any kind of selection operations on it.
+
+%% Cell type:code id: tags:
+
+``` python
+A = np.random.random([4, 5])
+A
+```
+
+%% Output
+
+    array([[0.37962202, 0.76975047, 0.73922844, 0.88922672, 0.99782046],
+           [0.1551213 , 0.6275266 , 0.38079222, 0.35535393, 0.86290208],
+           [0.03586265, 0.37427542, 0.67578338, 0.80472164, 0.21367888],
+           [0.27197917, 0.08610806, 0.16481586, 0.72530108, 0.49244613]])
+
+%% Cell type:code id: tags:
+
+``` python
+# Get the element from second line, first column
+A[1, 0]
+```
+
+%% Output
+
+    0.4336510750584107
+
+%% Cell type:code id: tags:
+
+``` python
+# Get the first two lines
+A[:2]
+```
+
+%% Output
+
+    array([[0.89925962, 0.31519992, 0.17170063, 0.06102236, 0.6055506 ],
+           [0.43365108, 0.67461267, 0.34962124, 0.75648088, 0.53096922]])
+
+%% Cell type:code id: tags:
+
+``` python
+# Get the last column
+A[:, -1]
+```
+
+%% Output
+
+    array([0.6055506 , 0.53096922, 0.14067726, 0.54727527])
+
+%% Cell type:code id: tags:
+
+``` python
+# Get the first two lines and the columns with an even index
+A[:2, ::2]
+```
+
+%% Output
+
+    array([[0.89925962, 0.17170063, 0.6055506 ],
+           [0.43365108, 0.34962124, 0.53096922]])
+
+%% Cell type:markdown id: tags:
+
+### 2.2 -  Using a mask to select elements validating a condition:
+
+%% Cell type:code id: tags:
+
+``` python
+cond = A > 0.5
+print(cond)
+print(A[cond])
+```
+
+%% Output
+
+    [[ True False False False  True]
+     [False  True False  True  True]
+     [ True False  True  True False]
+     [ True False  True False  True]]
+    [0.89925962 0.6055506  0.67461267 0.75648088 0.53096922 0.65643503
+     0.77202087 0.50192904 0.80709755 0.65465368 0.54727527]
+
+%% Cell type:markdown id: tags:
+
+The mask is in fact a particular case of the advanced indexing capabilities provided by NumPy. For example, it is even possible to use lists for indexing:
+
+%% Cell type:code id: tags:
+
+``` python
+# Selecting only particular columns
+print(A)
+A[:, [0, 1, 4]]
+```
+
+%% Output
+
+    [[0.89925962 0.31519992 0.17170063 0.06102236 0.6055506 ]
+     [0.43365108 0.67461267 0.34962124 0.75648088 0.53096922]
+     [0.65643503 0.4723704  0.77202087 0.50192904 0.14067726]
+     [0.80709755 0.2314217  0.65465368 0.28459125 0.54727527]]
+
+    array([[0.89925962, 0.31519992, 0.6055506 ],
+           [0.43365108, 0.67461267, 0.53096922],
+           [0.65643503, 0.4723704 , 0.14067726],
+           [0.80709755, 0.2314217 , 0.54727527]])
+
+%% Cell type:markdown id: tags:
+
+## Step 3 -  Perform array manipulations
+### 3.1 - Apply arithmetic operations to whole arrays (element-wise):
+
+%% Cell type:code id: tags:
+
+``` python
+(A+5)**2
+```
+
+%% Output
+
+    array([[34.80126403, 28.25135024, 26.7464874 , 25.61394735, 31.42219749],
+           [29.52456401, 32.20122896, 28.61844741, 33.13707212, 30.59162046],
+           [31.99525724, 29.94683782, 33.31622493, 30.27122313, 26.42656267],
+           [33.72238198, 27.36777304, 31.97510827, 27.92690466, 30.77226288]])
+
+%% Cell type:markdown id: tags:
+
+### 3.2 - Apply functions element-wise:
+
+%% Cell type:code id: tags:
+
+``` python
+np.exp(A) # With numpy arrays, use the functions from numpy !
+```
+
+%% Output
+
+    array([[2.45778274, 1.37053329, 1.18732233, 1.06292268, 1.83226077],
+           [1.54288042, 1.9632724 , 1.41853016, 2.13076459, 1.70057974],
+           [1.92790714, 1.60379132, 2.16413527, 1.65190478, 1.15105309],
+           [2.24139301, 1.26039064, 1.92447592, 1.3292186 , 1.72853679]])
+
+%% Cell type:markdown id: tags:
+
+### 3.3 - Setting parts of arrays
+
+%% Cell type:code id: tags:
+
+``` python
+A[:, 0] = 0.
+print(A)
+```
+
+%% Output
+
+    [[0.         0.31519992 0.17170063 0.06102236 0.6055506 ]
+     [0.         0.67461267 0.34962124 0.75648088 0.53096922]
+     [0.         0.4723704  0.77202087 0.50192904 0.14067726]
+     [0.         0.2314217  0.65465368 0.28459125 0.54727527]]
+
+%% Cell type:code id: tags:
+
+``` python
+# BONUS: Safe element-wise inverse with masks
+cond = (A != 0)
+A[cond] = 1./A[cond]
+print(A)
+```
+
+%% Output
+
+    [[ 0.          3.17258959  5.82409047 16.387435    1.65138967]
+     [ 0.          1.48233207  2.86023812  1.32191048  1.88334836]
+     [ 0.          2.11698277  1.29530177  1.99231351  7.10846954]
+     [ 0.          4.32111589  1.5275252   3.51381149  1.82723405]]
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Attributes and methods of `np.ndarray` (see the [doc](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.html#numpy.ndarray))
+
+%% Cell type:code id: tags:
+
+``` python
+for i,v in enumerate([s for s in dir(A) if not s.startswith('__')]):
+    print(f'{v:16}', end='')
+    if (i+1) % 6 == 0 :print('')
+```
+
+%% Output
+
+    T               all             any             argmax          argmin          argpartition
+    argsort         astype          base            byteswap        choose          clip
+    compress        conj            conjugate       copy            ctypes          cumprod
+    cumsum          data            diagonal        dot             dtype           dump
+    dumps           fill            flags           flat            flatten         getfield
+    imag            item            itemset         itemsize        max             mean
+    min             nbytes          ndim            newbyteorder    nonzero         partition
+    prod            ptp             put             ravel           real            repeat
+    reshape         resize          round           searchsorted    setfield        setflags
+    shape           size            sort            squeeze         std             strides
+    sum             swapaxes        take            tobytes         tofile          tolist
+    tostring        trace           transpose       var             view
+
+%% Cell type:code id: tags:
+
+``` python
+
+# Ex1: Get the mean through different dimensions
+
+print(A)
+print('Mean value',  A.mean())
+print('Mean line',   A.mean(axis=0))
+print('Mean column', A.mean(axis=1))
+```
+
+%% Output
+
+    [[0.37962202 0.76975047 0.73922844 0.88922672 0.99782046]
+     [0.1551213  0.6275266  0.38079222 0.35535393 0.86290208]
+     [0.03586265 0.37427542 0.67578338 0.80472164 0.21367888]
+     [0.27197917 0.08610806 0.16481586 0.72530108 0.49244613]]
+    Mean value 0.5001158248338762
+    Mean line [0.21064629 0.46441514 0.49015498 0.69365084 0.64171189]
+    Mean column [0.75512962 0.47633923 0.42086439 0.34813006]
+
+%% Cell type:code id: tags:
+
+``` python
+
+# Ex2: Convert a 2D array in 1D keeping all elements
+
+print(A)
+print(A.shape)
+A_flat = A.flatten()
+print(A_flat, A_flat.shape)
+```
+
+%% Output
+
+    [[0.37962202 0.76975047 0.73922844 0.88922672 0.99782046]
+     [0.1551213  0.6275266  0.38079222 0.35535393 0.86290208]
+     [0.03586265 0.37427542 0.67578338 0.80472164 0.21367888]
+     [0.27197917 0.08610806 0.16481586 0.72530108 0.49244613]]
+    (4, 5)
+    [0.37962202 0.76975047 0.73922844 0.88922672 0.99782046 0.1551213
+     0.6275266  0.38079222 0.35535393 0.86290208 0.03586265 0.37427542
+     0.67578338 0.80472164 0.21367888 0.27197917 0.08610806 0.16481586
+     0.72530108 0.49244613] (20,)
+
+%% Cell type:markdown id: tags:
+
+### 4.1 - Remark: dot product
+
+%% Cell type:code id: tags:
+
+``` python
+b = np.linspace(0, 10, 11)
+c = b @ b
+# before 3.5:
+# c = b.dot(b)
+print(b)
+print(c)
+```
+
+%% Output
+
+    [ 0.  1.  2.  3.  4.  5.  6.  7.  8.  9. 10.]
+    385.0
+
+%% Cell type:markdown id: tags:
+
+### 4.2 -  For Matlab users
+
+|     ` `       | Matlab | Numpy |
+| ------------- | ------ | ----- |
+| element wise  |  `.*`  |  `*`  |
+|  dot product  |  `*`   |  `@`  |
+
+%% Cell type:markdown id: tags:
+
+`numpy` arrays can also be sorted, even when they are composed of complex data if the type of the columns are explicitly stated with `dtypes`.
+
+%% Cell type:markdown id: tags:
+
+### 4.3 -  NumPy and SciPy sub-packages:
+
+We already saw `numpy.random` to generate `numpy` arrays filled with random values. This submodule also provides functions related to distributions (Poisson, gaussian, etc.) and permutations.
+
+%% Cell type:markdown id: tags:
+
+To perform linear algebra with dense matrices, we can use the submodule `numpy.linalg`. For instance, in order to compute the determinant of a random matrix, we use the method `det`
+
+%% Cell type:code id: tags:
+
+``` python
+A = np.random.random([5,5])
+print(A)
+np.linalg.det(A)
+```
+
+%% Output
+
+    [[0.47138506 0.41353868 0.09441948 0.225147   0.82335198]
+     [0.04490952 0.14682972 0.31792846 0.22918746 0.73823443]
+     [0.50485749 0.99705961 0.51896582 0.93318595 0.11375617]
+     [0.37148317 0.0477689  0.29061475 0.41826056 0.47950005]
+     [0.70324502 0.82838271 0.92172528 0.79532669 0.56698101]]
+
+    0.06968780805887545
+
+%% Cell type:code id: tags:
+
+``` python
+squared_subA = A[1:3, 1:3]
+print(squared_subA)
+np.linalg.inv(squared_subA)
+```
+
+%% Output
+
+    [[0.14682972 0.31792846]
+     [0.99705961 0.51896582]]
+
+    array([[-2.15522717,  1.32033369],
+           [ 4.14071576, -0.6097731 ]])
+
+%% Cell type:markdown id: tags:
+
+### 4.4 -  Introduction to Pandas: Python Data Analysis Library
+
+Pandas is an open source library providing high-performance, easy-to-use data structures and data analysis tools for Python.
+
+[Pandas tutorial](https://pandas.pydata.org/pandas-docs/stable/10min.html)
+[Grenoble Python Working Session](https://github.com/iutzeler/Pres_Pandas/)
+[Pandas for SQL Users](https://hackernoon.com/pandas-cheatsheet-for-sql-people-part-1-2976894acd0)
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>

README.md

+[<img width="600px" src="fidle/img/00-Fidle-titre-01.svg"></img>](#)
+
+## A propos
+
+This repository contains all the documents and links of the **Fidle Training**.  
+
+The objectives of this training, co-organized by the Formation Permanente CNRS and the SARI and DEVLOG networks, are :
+ - Understanding the **bases of deep learning** neural networks (Deep Learning)
+ - Develop a **first experience** through simple and representative examples
+ - Understand the different types of networks, their **architectures** and their **use cases**.
+ - Understanding **Tensorflow/Keras and Jupyter lab** technologies on the GPU
+ - Apprehend the **academic computing environments** Tier-2 (meso) and/or Tier-1 (national)
+
+## Course materials
+**[<img width="50px" src="fidle/img/00-Fidle-pdf.svg"></img>
+Get the course slides](https://cloud.univ-grenoble-alpes.fr/index.php/s/z7XZA36xKkMcaTS)**  
+
+
+
+<!-- ![pdf](fidle/img/00-Fidle-pdf.png) -->
+Useful information is also available in the [wiki](https://gricad-gitlab.univ-grenoble-alpes.fr/talks/fidle/-/wikis/home)
+
+
+## Jupyter notebooks
+
+[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/git/https%3A%2F%2Fgricad-gitlab.univ-grenoble-alpes.fr%2Ftalks%2Fdeeplearning.git/master?urlpath=lab/tree/index.ipynb)
+
+
+<!-- DO NOT REMOVE THIS TAG !!! -->
+<!-- INDEX -->
+<!-- INDEX_BEGIN -->
+[[NP1] - A short introduction to Numpy](Prerequisites/Numpy.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Numpy is an essential tool for the Scientific Python.  
+[[LINR1] - Linear regression with direct resolution](LinearReg/01-Linear-Regression.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Direct determination of linear regression   
+[[GRAD1] - Linear regression with gradient descent](LinearReg/02-Gradient-descent.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;An example of gradient descent in the simple case of a linear regression.  
+[[FIT1] - Complexity Syndrome](LinearReg/03-Polynomial-Regression.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Illustration of the problem of complexity with the polynomial regression  
+[[LOGR1] - Logistic regression, in pure Tensorflow](LinearReg/04-Logistic-Regression.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Logistic Regression with Mini-Batch Gradient Descent using pure TensorFlow.   
+[[MNIST1] - Simple classification with DNN](MNIST/01-DNN-MNIST.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Example of classification with a fully connected neural network  
+[[BHP1] - Regression with a Dense Network (DNN)](BHPD/01-DNN-Regression.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;A Simple regression with a Dense Neural Network (DNN) - BHPD dataset  
+[[BHP2] - Regression with a Dense Network (DNN) - Advanced code](BHPD/02-DNN-Regression-Premium.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;More advanced example of DNN network code - BHPD dataset  
+[[GTS1] - CNN with GTSRB dataset - Data analysis and preparation](GTSRB/01-Preparation-of-data.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Episode 1: Data analysis and creation of a usable dataset  
+[[GTS2] - CNN with GTSRB dataset - First convolutions](GTSRB/02-First-convolutions.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Episode 2 : First convolutions and first results  
+[[GTS3] - CNN with GTSRB dataset - Monitoring ](GTSRB/03-Tracking-and-visualizing.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Episode 3: Monitoring and analysing training, managing checkpoints  
+[[GTS4] - CNN with GTSRB dataset - Data augmentation ](GTSRB/04-Data-augmentation.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Episode 4: Improving the results with data augmentation  
+[[GTS5] - CNN with GTSRB dataset - Full convolutions ](GTSRB/05-Full-convolutions.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Episode 5: A lot of models, a lot of datasets and a lot of results.  
+[[GTS6] - CNN with GTSRB dataset - Full convolutions as a batch](GTSRB/06-Full-convolutions-batch.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Episode 6 : Run Full convolution notebook as a batch  
+[[GTS7] - Full convolutions Report](GTSRB/07-Full-convolutions-reports.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Displaying the reports of the different jobs  
+[[TSB1] - Tensorboard with/from Jupyter ](GTSRB/99-Scripts-Tensorboard.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;4 ways to use Tensorboard from the Jupyter environment  
+[[IMDB1] - Text embedding with IMDB](IMDB/01-Embedding-Keras.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;A very classical example of word embedding for text classification (sentiment analysis)  
+[[IMDB2] - Text embedding with IMDB - Reloaded](IMDB/02-Prediction.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Example of reusing a previously saved model  
+[[IMDB3] - Text embedding/LSTM model with IMDB](IMDB/03-LSTM-Keras.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Still the same problem, but with a network combining embedding and LSTM  
+[[VAE1] - Variational AutoEncoder (VAE) with MNIST](VAE/01-VAE-with-MNIST.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;First generative network experience with the MNIST dataset  
+[[VAE2] - Variational AutoEncoder (VAE) with MNIST - Analysis](VAE/02-VAE-with-MNIST-post.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Use of the previously trained model, analysis of the results  
+[[VAE3] - About the CelebA dataset](VAE/03-Prepare-CelebA.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;New VAE experience, but with a larger and more fun dataset  
+[[VAE4] - Preparation of the CelebA dataset](VAE/04-Prepare-CelebA-batch.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Preparation of a clustered dataset, batchable  
+[[VAE5] - Checking the clustered CelebA dataset](VAE/05-Check-CelebA.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Verification of prepared data from CelebA dataset  
+[[VAE6] - Variational AutoEncoder (VAE) with CelebA (small)](VAE/06-VAE-with-CelebA-s.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;VAE with a more fun and realistic dataset - small resolution and batchable  
+[[VAE7] - Variational AutoEncoder (VAE) with CelebA (medium)](VAE/07-VAE-with-CelebA-m.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;VAE with a more fun and realistic dataset - medium resolution and batchable  
+[[VAE12] - Variational AutoEncoder (VAE) with CelebA - Analysis](VAE/12-VAE-withCelebA-post.ipynb)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Use of the previously trained model with CelebA, analysis of the results  
+[[BASH1] - OAR batch script](VAE/batch-oar.sh)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Bash script for OAR batch submission of a notebook  
+[[BASH2] - SLURM batch script](VAE/batch-slurm.sh)  
+&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Bash script for SLURM batch submission of a notebook  
+<!-- INDEX_END -->
+
+
+## Installation
+
+A procedure for **configuring** and **starting Jupyter** is available in the **[Wiki](https://gricad-gitlab.univ-grenoble-alpes.fr/talks/fidle/-/wikis/howto-jupyter)**.
+
+## Licence
+
+[<img width="100px" src="fidle/img/00-fidle-CC BY-NC-SA.svg"></img>](https://creativecommons.org/licenses/by-nc-sa/4.0/)  
+\[en\] Attribution - NonCommercial - ShareAlike 4.0 International (CC BY-NC-SA 4.0)  
+\[Fr\] Attribution - Pas d’Utilisation Commerciale - Partage dans les Mêmes Conditions 4.0 International  
+See [License](https://creativecommons.org/licenses/by-nc-sa/4.0/legalcode).  
+See [Disclaimer](https://creativecommons.org/licenses/by-nc-sa/4.0/#).  
+
+
+----
+[<img width="80px" src="fidle/img/00-Fidle-logo-01.svg"></img>](#)
\ No newline at end of file

VAE/01-VAE-with-MNIST.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
+
+# <!-- TITLE --> [VAE1] - Variational AutoEncoder (VAE) with MNIST
+<!-- DESC --> First generative network experience with the MNIST dataset
+
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Understanding and implementing a variational autoencoder neurals network (VAE)
+ - Understanding a more advanced programming model
+
+The calculation needs being important, it is preferable to use a very simple dataset such as MNIST to start with.
+
+## What we're going to do :
+
+ - Defining a VAE model
+ - Build the model
+ - Train it
+ - Follow the learning process with Tensorboard
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init python stuff
+
+%% Cell type:code id: tags:
+
+``` python
+import numpy as np
+import sys, importlib
+
+import modules.vae
+import modules.loader_MNIST
+
+from modules.vae          import VariationalAutoencoder
+from modules.loader_MNIST import Loader_MNIST
+
+VariationalAutoencoder.about()
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Get data
+
+%% Cell type:code id: tags:
+
+``` python
+(x_train, y_train), (x_test, y_test) = Loader_MNIST.load()
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Get VAE model
+
+%% Cell type:code id: tags:
+
+``` python
+tag = 'MNIST.001'
+
+input_shape = (28,28,1)
+z_dim       = 2
+verbose     = 0
+
+encoder= [ {'type':'Conv2D',          'filters':32, 'kernel_size':(3,3), 'strides':1, 'padding':'same', 'activation':'relu'},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':1, 'padding':'same', 'activation':'relu'}
+         ]
+
+decoder= [ {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':1, 'padding':'same', 'activation':'relu'},
+           {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Conv2DTranspose', 'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Conv2DTranspose', 'filters':1,  'kernel_size':(3,3), 'strides':1, 'padding':'same', 'activation':'sigmoid'}
+         ]
+
+vae = modules.vae.VariationalAutoencoder(input_shape    = input_shape,
+                                         encoder_layers = encoder,
+                                         decoder_layers = decoder,
+                                         z_dim          = z_dim,
+                                         verbose        = verbose,
+                                         run_tag        = tag)
+vae.save(model=None)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Compile it
+
+%% Cell type:code id: tags:
+
+``` python
+r_loss_factor = 1000
+
+vae.compile( optimizer='adam', r_loss_factor=r_loss_factor)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Train
+
+%% Cell type:code id: tags:
+
+``` python
+batch_size        = 100
+epochs            = 100
+initial_epoch     = 0
+k_size            = 1      # 1 mean using 100% of the dataset
+```
+
+%% Cell type:code id: tags:
+
+``` python
+vae.train(x_train,
+          x_test,
+          batch_size        = batch_size,
+          epochs            = epochs,
+          initial_epoch     = initial_epoch,
+          k_size            = k_size
+         )
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>

VAE/04-Prepare-CelebA-batch.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
+
+# <!-- TITLE --> [VAE4] - Preparation of the CelebA dataset
+<!-- DESC --> Preparation of a clustered dataset, batchable
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Formatting our dataset in cluster files, using batch mode
+ - Adapting a notebook for batch use
+
+
+The [CelebFaces Attributes Dataset (CelebA)](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) contains about 200,000 images (202599,218,178,3).
+
+
+## What we're going to do :
+ - Lire les images
+ - redimensionner et normaliser celles-ci,
+ - Constituer des clusters d'images en format npy
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Import and init
+### 1.2 - Import
+
+%% Cell type:code id: tags:
+
+``` python
+import numpy as np
+import matplotlib.pyplot as plt
+import pandas as pd
+from skimage import io, transform
+
+import os,time,sys,json,glob
+import csv
+import math, random
+
+from importlib import reload
+
+sys.path.append('..')
+import fidle.pwk as ooo
+
+ooo.init()
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Directories and files :
+
+%% Cell type:code id: tags:
+
+``` python
+place, dataset_dir = ooo.good_place( { 'GRICAD' : f'{os.getenv("SCRATCH_DIR","")}/PROJECTS/pr-fidle/datasets/celeba',
+                                       'IDRIS'  : f'{os.getenv("WORK","")}/datasets/celeba'    } )
+
+dataset_csv  = f'{dataset_dir}/list_attr_celeba.csv'
+dataset_img  = f'{dataset_dir}/img_align_celeba'
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Read and shuffle filenames catalog
+
+%% Cell type:code id: tags:
+
+``` python
+dataset_desc = pd.read_csv(dataset_csv, header=0)
+dataset_desc = dataset_desc.reindex(np.random.permutation(dataset_desc.index))
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Save as clusters of n images
+
+%% Cell type:markdown id: tags:
+
+### 4.2 - Cooking function
+
+%% Cell type:code id: tags:
+
+``` python
+def read_and_save( dataset_img, dataset_desc,
+                   cluster_size=1000, cluster_dir='./dataset_cluster', cluster_name='images',
+                   image_size=(128,128)):
+
+    def save_cluster(imgs,desc,cols,id):
+        file_img  = f'{cluster_dir}/{cluster_name}-{id:03d}.npy'
+        file_desc = f'{cluster_dir}/{cluster_name}-{id:03d}.csv'
+        np.save(file_img,  np.array(imgs))
+        df=pd.DataFrame(data=desc,columns=cols)
+        df.to_csv(file_desc, index=False)
+        return [],[],id+1
+
+    start_time = time.time()
+    cols = list(dataset_desc.columns)
+
+    # ---- Check if cluster files exist
+    #
+    if os.path.isfile(f'{cluster_dir}/images-000.npy'):
+        print('\n*** Oops. There are already clusters in the target folder!\n')
+        return 0,0
+
+    # ---- Create cluster_dir
+    #
+    os.makedirs(cluster_dir, mode=0o750, exist_ok=True)
+
+    # ---- Read and save clusters
+    #
+    imgs, desc, cluster_id = [],[],0
+    #
+    for i,row in dataset_desc.iterrows():
+        #
+        filename = f'{dataset_img}/{row.image_id}'
+        #
+        # ---- Read image, resize (and normalize)
+        #
+        img = io.imread(filename)
+        img = transform.resize(img, image_size)
+        #
+        # ---- Add image and description
+        #
+        imgs.append( img )
+        desc.append( row.values )
+        #
+        # ---- Progress bar
+        #
+        ooo.update_progress(f'Cluster {cluster_id:03d} :',len(imgs),cluster_size)
+        #
+        # ---- Save cluster if full
+        #
+        if len(imgs)==cluster_size:
+            imgs,desc,cluster_id=save_cluster(imgs,desc,cols, cluster_id)
+
+    # ---- Save uncomplete cluster
+    if len(imgs)>0 : imgs,desc,cluster_id=save_cluster(imgs,desc,cols,cluster_id)
+
+    duration=time.time()-start_time
+    return cluster_id,duration
+```
+
+%% Cell type:markdown id: tags:
+
+### 4.3 - Cluster building
+Reading the 200,000 images can take a long time (>20 minutes)
+200,000 images will be used for training (x_train), the rest for validation (x_test)
+If the target folder is not empty, the construction is blocked.
+
+Image Sizes: 128x128 : 74 GB
+Image Sizes: 192x160 : 138 GB
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Cluster size
+
+cluster_size_train = 10000
+cluster_size_test  = 10000
+image_size         = (128,128)
+
+# ---- Clusters location
+
+train_dir  = f'{dataset_dir}/clusters.train'
+test_dir   = f'{dataset_dir}/clusters.test'
+
+# ---- x_train, x_test
+#
+n1,d1 = read_and_save(dataset_img, dataset_desc[:200000],
+                      cluster_size = cluster_size_train,
+                      cluster_dir  = train_dir,
+                      image_size   = image_size )
+
+n2,d2 = read_and_save(dataset_img, dataset_desc[200000:],
+                      cluster_size = cluster_size_test,
+                      cluster_dir  = test_dir,
+                      image_size   = image_size )
+
+print(f'\n\nDuration : {d1+d2:.2f} s or {ooo.hdelay(d1+d2)}')
+print(f'Train clusters : {train_dir}')
+print(f'Test  clusters : {test_dir}')
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>

VAE/05-Check-CelebA.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
+
+# <!-- TITLE --> [VAE5] - Checking the clustered CelebA dataset
+<!-- DESC --> Verification of prepared data from CelebA dataset
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Making sure our clustered dataset is correct
+ - Do a little bit of python while waiting to build and train our VAE model.
+
+The [CelebFaces Attributes Dataset (CelebA)](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) contains about 200,000 images (202599,218,178,3).
+
+
+## What we're going to do :
+
+ - Reload our dataset
+ - Check and verify our clustered dataset
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Import and init
+### 1.2 - Import
+
+%% Cell type:code id: tags:
+
+``` python
+import numpy as np
+import pandas as pd
+
+import os,time,sys,json,glob,importlib
+import math, random
+
+import modules.data_generator
+from modules.data_generator import DataGenerator
+
+sys.path.append('..')
+import fidle.pwk as ooo
+
+ooo.init()
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Directories and files :
+
+%% Cell type:code id: tags:
+
+``` python
+place, dataset_dir = ooo.good_place( { 'GRICAD' : f'{os.getenv("SCRATCH_DIR","")}/PROJECTS/pr-fidle/datasets/celeba',
+                                       'IDRIS'  : f'{os.getenv("WORK","")}/datasets/celeba'    } )
+
+train_dir    = f'{dataset_dir}/clusters.train'
+test_dir     = f'{dataset_dir}/clusters.test'
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Data verification
+What we're going to do:
+ - Recover all clusters by normalizing images
+ - Make some statistics to be sure we have all the data
+ - picking one image per cluster to check that everything is good.
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Return a legend from a description
+def get_legend(x_desc,i):
+    cols  = x_desc.columns
+    desc  = x_desc.iloc[i]
+    legend =[]
+    for i,v in enumerate(desc):
+        if v==1 : legend.append(cols[i])
+    return str('\n'.join(legend))
+
+start_time = time.time()
+
+# ---- get cluster list
+
+clusters_name = [ os.path.splitext(f)[0] for f in glob.glob( f'{train_dir}/*.npy') ]
+
+# ---- Counters set to 0
+
+imax  = len(clusters_name)
+i,n1,n2,s = 0,0,0,0
+imgs,desc = [],[]
+
+# ---- Reload all clusters
+
+ooo.update_progress('Load clusters :',i,imax, redraw=True)
+for cluster_name in clusters_name:
+
+    # ---- Reload images and normalize
+
+    x_data = np.load(cluster_name+'.npy')
+
+    # ---- Reload descriptions
+
+    x_desc = pd.read_csv(cluster_name+'.csv', header=0)
+
+    # ---- Counters
+
+    n1 += len(x_data)
+    n2 += len(x_desc.index)
+    s  += x_data.nbytes
+    i  += 1
+
+    # ---- Get somes images/legends
+
+    j=random.randint(0,len(x_data)-1)
+    imgs.append( x_data[j].copy() )
+    desc.append( get_legend(x_desc,j) )
+    x_data=None
+
+    # ---- To appear professional
+
+    ooo.update_progress('Load clusters :',i,imax, redraw=True)
+
+d=time.time()-start_time
+
+print(f'Loading time      : {d:.2f} s or {ooo.hdelay(d)}')
+print(f'Number of cluster : {i}')
+print(f'Number of images  : {n1}')
+print(f'Number of desc.   : {n2}')
+print(f'Total size of img : {ooo.hsize(s)}')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+ooo.plot_images(imgs,desc,x_size=2,y_size=2,fontsize=8,columns=7,y_padding=2.5)
+```
+
+%% Cell type:markdown id: tags:
+
+<div class='nota'>
+    <b>Note :</b> With this approach, the use of data is much much more effective !
+    <ul>
+        <li>Data loading speed : <b>x 10</b> (81 s vs 16 min.)</li>
+    </ul>
+</div>
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - How we will read our data during the train session
+We are going to use a "dataset reader", which is a [tensorflow.keras.utils.Sequence](https://www.tensorflow.org/api_docs/python/tf/keras/utils/Sequence)
+The batches will be requested to our DataGenerator, which will read the clusters as they come in.
+
+### 3.1 - An example to understand
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- A very small dataset
+
+clusters_dir = f'{dataset_dir}/clusters-xs.train'
+
+# ---- Our DataGenerator
+#      with small batch size, debug mode and 50% of the dataset
+
+data_gen = DataGenerator(clusters_dir, 32, debug=True, k_size=1)
+
+# ---- We ask him to retrieve all batchs
+
+batch_sizes=[]
+for i in range( len(data_gen)):
+    x,y = data_gen[i]
+    batch_sizes.append(len(x))
+
+print(f'\n\ntotal number of items : {sum(batch_sizes)}')
+print(f'batch sizes      : {batch_sizes}')
+print(f'Last batch shape : {x.shape}')
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>

VAE/06-VAE-with-CelebA-s.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
+
+# <!-- TITLE --> [VAE6] - Variational AutoEncoder (VAE) with CelebA (small)
+<!-- DESC --> VAE with a more fun and realistic dataset - small resolution and batchable
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Build and train a VAE model with a large dataset in small resolution(>70 GB)
+ - Understanding a more advanced programming model with data generator
+
+The [CelebFaces Attributes Dataset (CelebA)](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) contains about 200,000 images (202599,218,178,3).
+
+## What we're going to do :
+
+ - Defining a VAE model
+ - Build the model
+ - Train it
+ - Follow the learning process with Tensorboard
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Setup environment
+### 1.1 - Python stuff
+
+%% Cell type:code id: tags:
+
+``` python
+import tensorflow as tf
+import numpy as np
+import os,sys
+from importlib import reload
+
+import modules.vae
+import modules.data_generator
+
+reload(modules.data_generator)
+reload(modules.vae)
+
+from modules.vae  import VariationalAutoencoder
+from modules.data_generator import DataGenerator
+
+sys.path.append('..')
+import fidle.pwk as ooo
+reload(ooo)
+
+ooo.init()
+
+VariationalAutoencoder.about()
+DataGenerator.about()
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - The good place
+
+%% Cell type:code id: tags:
+
+``` python
+place, dataset_dir = ooo.good_place( { 'GRICAD' : f'{os.getenv("SCRATCH_DIR","")}/PROJECTS/pr-fidle/datasets/celeba',
+                                       'IDRIS'  : f'{os.getenv("WORK","")}/datasets/celeba'    } )
+
+# ---- train/test datasets
+
+train_dir    = f'{dataset_dir}/clusters.train'
+test_dir     = f'{dataset_dir}/clusters.test'
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - DataGenerator and validation data
+Ok, everything's perfect, now let's instantiate our generator for the entire dataset.
+
+%% Cell type:code id: tags:
+
+``` python
+data_gen = DataGenerator(train_dir, 32, k_size=1)
+x_test   = np.load(f'{test_dir}/images-000.npy')
+
+print(f'Data generator : {len(data_gen)} batchs of {data_gen.batch_size} images, or {data_gen.dataset_size} images')
+print(f'x_test : {len(x_test)} images')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Get VAE model
+
+%% Cell type:code id: tags:
+
+``` python
+tag = f'CelebA.006-S.{os.getenv("SLURM_JOB_ID","unknown")}'
+
+input_shape = (128, 128, 3)
+z_dim       = 200
+verbose     = 1
+
+encoder= [ {'type':'Conv2D',          'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+         ]
+
+decoder= [ {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2DTranspose', 'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2DTranspose', 'filters':3,  'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'sigmoid'}
+         ]
+
+vae = modules.vae.VariationalAutoencoder(input_shape    = input_shape,
+                                         encoder_layers = encoder,
+                                         decoder_layers = decoder,
+                                         z_dim          = z_dim,
+                                         verbose        = verbose,
+                                         run_tag        = tag)
+vae.save(model=None)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Compile it
+
+%% Cell type:code id: tags:
+
+``` python
+optimizer = tf.keras.optimizers.Adam(1e-4)
+# optimizer     = 'adam'
+r_loss_factor = 10000
+
+vae.compile(optimizer, r_loss_factor)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Train
+For 10 epochs, adam optimizer :
+- Run time at IDRIS : 1299.77 sec. - 0:21:39
+- Run time at GRICAD : 2092.77 sec. - 0:34:52
+
+%% Cell type:code id: tags:
+
+``` python
+epochs            = 10
+initial_epoch     = 0
+```
+
+%% Cell type:code id: tags:
+
+``` python
+vae.train(data_generator    = data_gen,
+          x_test            = x_test,
+          epochs            = epochs,
+          initial_epoch     = initial_epoch
+         )
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>

VAE/07-VAE-with-CelebA-m.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
+
+# <!-- TITLE --> [VAE7] - Variational AutoEncoder (VAE) with CelebA (medium)
+<!-- DESC --> VAE with a more fun and realistic dataset - medium resolution and batchable
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Build and train a VAE model with a large dataset in small resolution(>140 GB)
+ - Understanding a more advanced programming model with data generator
+
+The [CelebFaces Attributes Dataset (CelebA)](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html) contains about 200,000 images (202599,218,178,3).
+
+## What we're going to do :
+
+ - Defining a VAE model
+ - Build the model
+ - Train it
+ - Follow the learning process with Tensorboard
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Setup environment
+### 1.1 - Python stuff
+
+%% Cell type:code id: tags:
+
+``` python
+import tensorflow as tf
+import numpy as np
+import os,sys
+from importlib import reload
+
+import modules.vae
+import modules.data_generator
+
+reload(modules.data_generator)
+reload(modules.vae)
+
+from modules.vae  import VariationalAutoencoder
+from modules.data_generator import DataGenerator
+
+sys.path.append('..')
+import fidle.pwk as ooo
+reload(ooo)
+
+ooo.init()
+
+VariationalAutoencoder.about()
+DataGenerator.about()
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - The good place
+
+%% Cell type:code id: tags:
+
+``` python
+place, dataset_dir = ooo.good_place( { 'GRICAD' : f'{os.getenv("SCRATCH_DIR","")}/PROJECTS/pr-fidle/datasets/celeba',
+                                       'IDRIS'  : f'{os.getenv("WORK","")}/datasets/celeba'    } )
+
+# ---- train/test datasets
+
+train_dir    = f'{dataset_dir}/clusters-M.train'
+test_dir     = f'{dataset_dir}/clusters-M.test'
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - DataGenerator and validation data
+Ok, everything's perfect, now let's instantiate our generator for the entire dataset.
+
+%% Cell type:code id: tags:
+
+``` python
+data_gen = DataGenerator(train_dir, 32, k_size=1)
+x_test   = np.load(f'{test_dir}/images-000.npy')
+
+print(f'Data generator : {len(data_gen)} batchs of {data_gen.batch_size} images, or {data_gen.dataset_size} images')
+print(f'x_test : {len(x_test)} images')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Get VAE model
+
+%% Cell type:code id: tags:
+
+``` python
+tag = f'CelebA.007-M.{os.getenv("SLURM_JOB_ID","unknown")}'
+
+input_shape = (192, 160, 3)
+z_dim       = 200
+verbose     = 0
+
+encoder= [ {'type':'Conv2D',          'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+         ]
+
+decoder= [ {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2DTranspose', 'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2DTranspose', 'filters':3,  'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'sigmoid'}
+         ]
+
+vae = modules.vae.VariationalAutoencoder(input_shape    = input_shape,
+                                         encoder_layers = encoder,
+                                         decoder_layers = decoder,
+                                         z_dim          = z_dim,
+                                         verbose        = verbose,
+                                         run_tag        = tag)
+vae.save(model=None)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Compile it
+
+%% Cell type:code id: tags:
+
+``` python
+optimizer     = tf.keras.optimizers.Adam(1e-4)
+r_loss_factor = 10000
+
+vae.compile(optimizer, r_loss_factor)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Train
+For 10 epochs, adam optimizer :
+- Run time at IDRIS : 1299.77 sec. - 0:21:39
+- Run time at GRICAD : 2092.77 sec. - 0:34:52
+
+%% Cell type:code id: tags:
+
+``` python
+epochs            = 20
+initial_epoch     = 0
+```
+
+%% Cell type:code id: tags:
+
+``` python
+vae.train(data_generator    = data_gen,
+          x_test            = x_test,
+          epochs            = epochs,
+          initial_epoch     = initial_epoch
+         )
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>

VAE/08-VAE-with-CelebA-m.nbconvert.ipynb

+%% Cell type:markdown id: tags:
+
+Variational AutoEncoder (VAE) with CelebA
+=========================================
+---
+Formation Introduction au Deep Learning  (FIDLE) - S. Arias, E. Maldonado, JL. Parouty - CNRS/SARI/DEVLOG - 2020
+
+## Episode 1 - Train a model
+
+ - Defining a VAE model
+ - Build the model
+ - Train it
+ - Follow the learning process with Tensorboard
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Setup environment
+### 1.1 - Python stuff
+
+%% Cell type:code id: tags:
+
+``` python
+import tensorflow as tf
+import numpy as np
+import os,sys
+from importlib import reload
+
+import modules.vae
+import modules.data_generator
+
+reload(modules.data_generator)
+reload(modules.vae)
+
+from modules.vae  import VariationalAutoencoder
+from modules.data_generator import DataGenerator
+
+sys.path.append('..')
+import fidle.pwk as ooo
+reload(ooo)
+
+ooo.init()
+
+VariationalAutoencoder.about()
+DataGenerator.about()
+```
+
+%% Output
+
+
+    
+    FIDLE 2020 - Practical Work Module
+    Version              : 0.2.8
+    Run time             : Friday 14 February 2020, 00:07:28
+    TensorFlow version   : 2.0.0
+    Keras version        : 2.2.4-tf
+    
+    FIDLE 2020 - Variational AutoEncoder (VAE)
+    TensorFlow version   : 2.0.0
+    VAE version          : 1.28
+    
+    FIDLE 2020 - DataGenerator
+    Version              : 0.4.1
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - The good place
+
+%% Cell type:code id: tags:
+
+``` python
+place, dataset_dir = ooo.good_place( { 'GRICAD' : f'{os.getenv("SCRATCH_DIR","")}/PROJECTS/pr-fidle/datasets/celeba',
+                                       'IDRIS'  : f'{os.getenv("WORK","")}/datasets/celeba'    } )
+
+# ---- train/test datasets
+
+train_dir    = f'{dataset_dir}/clusters-M.train'
+test_dir     = f'{dataset_dir}/clusters-M.test'
+```
+
+%% Output
+
+    Well, we should be at IDRIS !
+    We are going to use: /gpfswork/rech/mlh/uja62cb/datasets/celeba
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - DataGenerator and validation data
+Ok, everything's perfect, now let's instantiate our generator for the entire dataset.
+
+%% Cell type:code id: tags:
+
+``` python
+data_gen = DataGenerator(train_dir, 32, k_size=1)
+x_test   = np.load(f'{test_dir}/images-000.npy')
+
+print(f'Data generator : {len(data_gen)} batchs of {data_gen.batch_size} images, or {data_gen.dataset_size} images')
+print(f'x_test : {len(x_test)} images')
+```
+
+%% Output
+
+    Data generator : 6250 batchs of 32 images, or 200000 images
+    x_test : 2599 images
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Get VAE model
+
+%% Cell type:code id: tags:
+
+``` python
+tag = f'CelebA.052-M.{os.getenv("SLURM_JOB_ID","unknown")}'
+
+input_shape = (192, 160, 3)
+z_dim       = 200
+verbose     = 0
+
+encoder= [ {'type':'Conv2D',          'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+         ]
+
+decoder= [ {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2DTranspose', 'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
+           {'type':'Dropout',         'rate':0.25},
+           {'type':'Conv2DTranspose', 'filters':3,  'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'sigmoid'}
+         ]
+
+vae = modules.vae.VariationalAutoencoder(input_shape    = input_shape,
+                                         encoder_layers = encoder,
+                                         decoder_layers = decoder,
+                                         z_dim          = z_dim,
+                                         verbose        = verbose,
+                                         run_tag        = tag)
+vae.save(model=None)
+```
+
+%% Output
+
+    Model initialized.
+    Outputs will be in  : ./run/CelebA.052-M.810156
+    Config saved in     : ./run/CelebA.052-M.810156/models/vae_config.json
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Compile it
+
+%% Cell type:code id: tags:
+
+``` python
+optimizer     = tf.keras.optimizers.Adam(1e-4)
+r_loss_factor = 10000
+
+vae.compile(optimizer, r_loss_factor)
+```
+
+%% Output
+
+    Compiled.
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Train
+For 10 epochs, adam optimizer :
+- Run time at IDRIS : 1299.77 sec. - 0:21:39
+- Run time at GRICAD : 2092.77 sec. - 0:34:52
+
+%% Cell type:code id: tags:
+
+``` python
+epochs            = 20
+initial_epoch     = 0
+```
+
+%% Cell type:code id: tags:
+
+``` python
+vae.train(data_generator    = data_gen,
+          x_test            = x_test,
+          epochs            = epochs,
+          initial_epoch     = initial_epoch
+         )
+```
+
+%% Output
+
+    Epoch 1/20
+    6250/6250 [==============================] - 342s 55ms/step - loss: 332.4792 - vae_r_loss: 282.9655 - vae_kl_loss: 49.5134 - val_loss: 235.4288 - val_vae_r_loss: 187.4334 - val_vae_kl_loss: 48.1192
+    Epoch 2/20
+    6250/6250 [==============================] - 337s 54ms/step - loss: 224.0962 - vae_r_loss: 166.4602 - vae_kl_loss: 57.6360 - val_loss: 210.9632 - val_vae_r_loss: 155.0925 - val_vae_kl_loss: 56.0114
+    Epoch 4/20
+    6250/6250 [==============================] - 333s 53ms/step - loss: 214.5463 - vae_r_loss: 155.7666 - vae_kl_loss: 58.7794 - val_loss: 203.8241 - val_vae_r_loss: 147.3248 - val_vae_kl_loss: 56.5778
+    Epoch 7/20
+    6250/6250 [==============================] - 327s 52ms/step - loss: 211.1459 - vae_r_loss: 152.4026 - vae_kl_loss: 58.7437 - val_loss: 201.1862 - val_vae_r_loss: 145.6906 - val_vae_kl_loss: 55.6326
+    Epoch 10/20
+    6250/6250 [==============================] - 335s 54ms/step - loss: 209.7628 - vae_r_loss: 151.0874 - vae_kl_loss: 58.6756 - val_loss: 202.3954 - val_vae_r_loss: 147.0956 - val_vae_kl_loss: 55.4047
+    Epoch 12/20
+    6250/6250 [==============================] - 333s 53ms/step - loss: 207.9830 - vae_r_loss: 149.3870 - vae_kl_loss: 58.5959 - val_loss: 198.5626 - val_vae_r_loss: 142.5848 - val_vae_kl_loss: 56.0871
+    Epoch 16/20
+    6250/6250 [==============================] - 330s 53ms/step - loss: 206.6382 - vae_r_loss: 148.0522 - vae_kl_loss: 58.5863 - val_loss: 197.5800 - val_vae_r_loss: 142.6799 - val_vae_kl_loss: 54.9832
+    
+    Train duration : 6638.61 sec. - 1:50:38
+
+%% Cell type:markdown id: tags:
+
+----
+That's all folks !

VAE/batch-oar.sh

+#!/bin/bash
+#OAR -n VAE with CelebA
+#OAR -t gpu
+#OAR -l /nodes=1/gpudevice=1,walltime=01:00:00
+#OAR --stdout _batch/VAE_CelebA_%jobid%.out
+#OAR --stderr _batch/VAE_CelebA_%jobid%.err
+#OAR --project fidle
+
+#---- For cpu
+# use :
+# OAR -l /nodes=1/core=32,walltime=01:00:00
+# and add a 2>/dev/null to ipython xxx
+
+# -----------------------------------------------
+#         _           _       _
+#        | |__   __ _| |_ ___| |__
+#        | '_ \ / _` | __/ __| '_ \
+#        | |_) | (_| | || (__| | | |
+#        |_.__/ \__,_|\__\___|_| |_|
+#                             Fidle at GRICAD
+# -----------------------------------------------
+#
+# <!-- TITLE --> [BASH1] - OAR batch script
+# <!-- DESC --> Bash script for OAR batch submission of a notebook 
+# <!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+CONDA_ENV=fidle
+RUN_DIR=~/fidle/VAE
+RUN_IPYNB='06-VAE-with-CelebA-s.ipynb'
+
+# ---- Cuda Conda initialization
+
+echo '------------------------------------------------------------'
+echo "Start : $0"
+echo '------------------------------------------------------------'
+
+source /applis/environments/cuda_env.sh dahu 10.0
+source /applis/environments/conda.sh
+
+conda activate "$CONDA_ENV"
+
+# ---- Run it...
+#
+cd $RUN_DIR
+
+jupyter nbconvert --to notebook --execute "$RUN_IPYNB"

VAE/batch-slurm.sh

+#!/bin/bash
+
+#SBATCH --job-name="VAE_bizness"                       # nom du job
+#SBATCH --ntasks=1                                     # nombre de tâche (un unique processus ici)
+#SBATCH --gres=gpu:1                                   # nombre de GPU à réserver (un unique GPU ici)
+#SBATCH --cpus-per-task=10                             # nombre de coeurs à réserver (un quart du noeud)
+#SBATCH --hint=nomultithread                           # on réserve des coeurs physiques et non logiques
+#SBATCH --time=02:00:00                                # temps exécution maximum demande (HH:MM:SS)
+#SBATCH --output="_batch/VAE_%j.out"                   # nom du fichier de sortie
+#SBATCH --error="_batch/VAE_%j.err"                    # nom du fichier d'erreur (ici commun avec la sortie)
+#SBATCH --mail-user=Jean-Luc.Parouty@grenoble-inp.fr
+#SBATCH --mail-type=ALL
+
+# -----------------------------------------------
+#         _           _       _
+#        | |__   __ _| |_ ___| |__
+#        | '_ \ / _` | __/ __| '_ \
+#        | |_) | (_| | || (__| | | |
+#        |_.__/ \__,_|\__\___|_| |_|
+#                              Fidle at IDRIS
+# -----------------------------------------------
+#
+# <!-- TITLE --> [BASH2] - SLURM batch script
+# <!-- DESC --> Bash script for SLURM batch submission of a notebook 
+# <!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+MODULE_ENV="tensorflow-gpu/py3/2.0.0"
+RUN_DIR="$WORK/fidle/VAE"
+RUN_IPYNB="06-VAE-with-CelebA-s.ipynb"
+
+# ---- Welcome...
+
+echo '------------------------------------------------------------'
+echo "Start : $0"
+echo '------------------------------------------------------------'
+echo "Job id        : $SLURM_JOB_ID"
+echo "Job name      : $SLURM_JOB_NAME"
+echo "Job node list : $SLURM_JOB_NODELIST"
+echo '------------------------------------------------------------'
+echo "Notebook      : $RUN_IPYNB"
+echo "Run in        : $RUN_DIR"
+echo "With env.     : $MODULE_ENV"
+echo '------------------------------------------------------------'
+
+
+# ---- Module
+
+module load $MODULE_ENV
+
+# ---- Run it...
+
+cd $RUN_DIR
+
+jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --execute "$RUN_IPYNB"

VAE/modules/callbacks.py

+from tensorflow.keras.callbacks import Callback
+import numpy as np
+import matplotlib.pyplot as plt
+
+class ImagesCallback(Callback):
+    
+   
+    def __init__(self, filename= 'image-{epoch:03d}-{i:02d}.jpg', z_dim=0, decoder=None, nb_images=5):
+        self.filename  = filename
+        self.z_dim     = z_dim
+        self.decoder   = decoder
+        self.nb_images = nb_images
+        
+        
+    def on_epoch_end(self, epoch, logs={}):  
+        
+        # ---- Get random latent points
+        
+        z_new   = np.random.normal(size = (self.nb_images,self.z_dim))
+        
+        # ---- Predict an image
+        
+        images = self.decoder.predict(np.array(z_new))
+        
+        # ---- Save images
+        
+        for i,image in enumerate(images):
+            
+            # ---- Squeeze it if monochrome : (lx,ly,1) -> (lx,ly) 
+            image = image.squeeze()
+        
+            # ---- Save it
+
+            filename = self.filename.format(epoch=epoch,i=i)
+            
+            if len(image.shape) == 2:
+                plt.imsave(filename, image, cmap='gray_r')
+            else:
+                plt.imsave(filename, image)

VAE/modules/data_generator.py

+
+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/       DataGenerator
+#    |_|   |_|\__,_|_|\___|       for clustered CelebA sataset
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/SARI/DEVLOG 2020 - S. Arias, E. Maldonado, JL. Parouty
+# ------------------------------------------------------------------
+# Initial version by JL Parouty, feb 2020
+
+
+import numpy as np
+import pandas as pd
+import math
+import os,glob
+
+from tensorflow.keras.utils import Sequence
+
+class DataGenerator(Sequence):
+
+    version = '0.4.1'
+    
+    def __init__(self, clusters_dir='./data', batch_size=32, debug=False, k_size=1):
+        '''
+        Instanciation of the data generator
+        args:
+            cluster_dir : Directory of the clusters files
+            batch_size  : Batch size (32)
+            debug       : debug mode (False)
+        '''
+        if debug : self.about()
+        #
+        # ---- Get the list of clusters
+        #      
+        clusters_name = [ os.path.splitext(f)[0] for f in glob.glob( f'{clusters_dir}/*.npy') ]
+        clusters_size = len(clusters_name)
+        #
+        # ---- Read each cluster description
+        #      because we need the full dataset size
+        #
+        dataset_size  = 0
+        for c in clusters_name:
+            df = pd.read_csv(c+'.csv', header=0)
+            dataset_size+=len(df.index)
+        #
+        # ---- If we only want to use a part of the dataset...
+        #
+        dataset_size = int(dataset_size * k_size)
+        #
+        if debug: 
+            print(f'\nClusters nb  : {len(clusters_name)} files')
+            print(f'Dataset size : {dataset_size}')
+            print(f'Batch size   : {batch_size}')
+
+        #
+        # ---- Remember all of that
+        #
+        self.clusters_dir  = clusters_dir
+        self.batch_size    = batch_size
+        self.clusters_name = clusters_name
+        self.clusters_size = clusters_size
+        self.dataset_size  = dataset_size
+        self.debug         = debug
+        #
+        # ---- Read a first cluster
+        #
+        self.cluster_i = clusters_size
+        self.read_next_cluster()
+        
+        
+    def __len__(self):
+        return math.floor(self.dataset_size / self.batch_size)
+
+    
+    def __getitem__(self, idx):
+        #
+        # ---- Get the next item index
+        #
+        i=self.data_i
+        #
+        # ---- Get a batch
+        #
+        batch = self.data[i:i+self.batch_size]
+        #
+        # ---- Cluster is large enough
+        #
+        if len(batch) == self.batch_size:
+            self.data_i += self.batch_size
+            if self.debug: print(f'({len(batch)}) ',end='')
+            return batch,batch
+        #
+        # ---- Not enough...
+        #
+        if self.debug: print(f'({len(batch)}..) ',end='')
+        #
+        self.read_next_cluster()
+        batch2 = self.data[ 0:self.batch_size-len(batch) ]
+        self.data_i = self.batch_size-len(batch)
+        batch  = np.concatenate( (batch,batch2) )
+        #
+        if self.debug: print(f'(..{len(batch2)}) ',end='')
+        return batch, batch
+    
+    
+    def on_epoch_end(self):
+        self.cluster_i = self.clusters_size
+        self.read_next_cluster()
+    
+    
+    def read_next_cluster(self):
+        #
+        # ---- Get the next cluster name
+        #      If we have reached the end of the list, we mix and
+        #      start again from the beginning. 
+        #
+        i = self.cluster_i + 1
+        if i >= self.clusters_size:
+            np.random.shuffle(self.clusters_name)
+            i = 0
+            if self.debug : print(f'\n[shuffle!]')
+        #
+        # ---- Read it (images still normalized)
+        #
+        data = np.load( self.clusters_name[i]+'.npy', mmap_mode='r' )
+        #
+        # ---- Remember all of that
+        #
+        self.data      = data
+        self.data_i    = 0
+        self.cluster_i = i
+        #
+        if self.debug: print(f'\n[Load {self.cluster_i:02d},s={len(self.data):3d}] ',end='')
+          
+        
+    @classmethod
+    def about(cls):
+        print('\nFIDLE 2020 - DataGenerator')
+        print('Version              :', cls.version)
\ No newline at end of file

VAE/modules/loader_MNIST.py

+
+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/SARI/DEVLOG 2020 - S. Arias, E. Maldonado, JL. Parouty
+# ------------------------------------------------------------------
+# Initial version by JL Parouty, feb 2020
+
+import numpy as np
+import tensorflow as tf
+import tensorflow.keras.datasets.mnist as mnist
+
+class Loader_MNIST():
+    
+    version = '0.1'
+    
+    def __init__(self):
+        pass
+
+    @classmethod
+    def about(cls):
+        print('\nFIDLE 2020 - Very basic MNIST dataset loader)')
+        print('TensorFlow version   :',tf.__version__)
+        print('Loader version       :', cls.version)
+    
+    @classmethod
+    def load(normalize=True, expand=True, verbose=1):
+
+        # ---- Get data
+
+        (x_train, y_train), (x_test, y_test) = mnist.load_data()
+        if verbose>0: print('Dataset loaded.')
+
+
+        # ---- Normalization
+
+        if normalize:
+            x_train = x_train.astype('float32') / 255.
+            x_test  = x_test.astype( 'float32') / 255.
+            if verbose>0: print('Normalized.')
+            
+        # ---- Reshape : (28,28) -> (28,28,1)
+
+        if expand:
+            x_train = np.expand_dims(x_train, axis=3)
+            x_test  = np.expand_dims(x_test,  axis=3)
+            if verbose>0: print(f'Reshaped to {x_train.shape}')
+
+        return (x_train,y_train),(x_test,y_test)
\ No newline at end of file

VAE/modules/vae.py

+
+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/SARI/DEVLOG 2020 - S. Arias, E. Maldonado, JL. Parouty
+# ------------------------------------------------------------------
+# by JL Parouty (feb 2020), based on David Foster examples.
+
+import numpy as np
+import math
+
+import tensorflow as tf
+import tensorflow.keras as keras
+from tensorflow.keras import backend as K
+from tensorflow.keras.layers import Input, Conv2D, Flatten, Dense, Conv2DTranspose, Reshape, Lambda
+from tensorflow.keras.layers import Activation, BatchNormalization, LeakyReLU, Dropout
+from tensorflow.keras.models import Model
+from tensorflow.keras.callbacks import ModelCheckpoint, TensorBoard
+from tensorflow.keras.optimizers import Adam
+from tensorflow.keras.utils import plot_model
+
+from modules.callbacks      import ImagesCallback
+from modules.data_generator import DataGenerator
+
+
+import os, json, time, datetime
+
+
+
+class VariationalAutoencoder():
+
+    version = '1.28'
+    
+    def __init__(self, input_shape=None, encoder_layers=None, decoder_layers=None, z_dim=None, run_tag='000', verbose=0):
+               
+        self.name           = 'Variational AutoEncoder'
+        self.input_shape    = list(input_shape)
+        self.encoder_layers = encoder_layers
+        self.decoder_layers = decoder_layers
+        self.z_dim          = z_dim
+        self.run_tag        = str(run_tag)
+        self.verbose        = verbose
+        self.run_directory  = f'./run/{run_tag}'
+        
+        # ---- Create run directories
+        for d in ('','/models','/figs','/logs','/images'):
+            os.makedirs(self.run_directory+d, mode=0o750, exist_ok=True)
+        
+        # ==== Encoder ================================================================
+        
+        # ---- Input layer
+        encoder_input = Input(shape=self.input_shape, name='encoder_input')
+        x = encoder_input
+        
+        # ---- Add next layers
+        i=1
+        for l_config in encoder_layers:
+            l_type   = l_config['type']
+            l_params = l_config.copy()
+            l_params.pop('type')
+            if l_type=='Conv2D':
+                layer = Conv2D(**l_params)
+            if l_type=='Dropout':
+                layer = Dropout(**l_params)
+            x = layer(x)
+            i+=1
+            
+        # ---- Flatten
+        shape_before_flattening = K.int_shape(x)[1:]
+        x = Flatten()(x)
+        
+        # ---- mu <-> log_var
+        self.mu      = Dense(self.z_dim, name='mu')(x)
+        self.log_var = Dense(self.z_dim, name='log_var')(x)
+
+        self.encoder_mu_log_var = Model(encoder_input, (self.mu, self.log_var))
+
+        # ---- output layer
+        def sampling(args):
+            mu, log_var = args
+            epsilon = K.random_normal(shape=K.shape(mu), mean=0., stddev=1.)
+            return mu + K.exp(log_var / 2) * epsilon
+
+        encoder_output = Lambda(sampling, name='encoder_output')([self.mu, self.log_var])
+
+        self.encoder = Model(encoder_input, encoder_output)
+
+        # ==== Decoder ================================================================
+
+        # ---- Input layer
+        decoder_input = Input(shape=(self.z_dim,), name='decoder_input')
+        
+        # ---- First dense layer
+        x = Dense(np.prod(shape_before_flattening))(decoder_input)
+        x = Reshape(shape_before_flattening)(x)
+
+        # ---- Add next layers
+        i=1
+        for l_config in decoder_layers:
+            l_type   = l_config['type']
+            l_params = l_config.copy()
+            l_params.pop('type')
+            if l_type=='Conv2DTranspose':
+                layer = Conv2DTranspose(**l_params)
+            if l_type=='Dropout':
+                layer = Dropout(**l_params)
+            x = layer(x)
+            i+=1
+
+        decoder_output = x
+
+        self.decoder = Model(decoder_input, decoder_output)
+        
+        # ==== Encoder-Decoder ========================================================
+        
+        model_input = encoder_input
+        model_output = self.decoder(encoder_output)
+
+        self.model = Model(model_input, model_output)
+
+        # ==== Verbosity ==============================================================
+
+        print('Model initialized.')
+        print(f'Outputs will be in  : {self.run_directory}')
+        
+        if verbose>0 :
+            print('\n','-'*10,'Encoder','-'*50,'\n')
+            self.encoder.summary()
+            print('\n','-'*10,'Encoder','-'*50,'\n')
+            self.decoder.summary()
+            self.plot_model()
+        
+        
+        
+    def compile(self, optimizer='adam', r_loss_factor='1000'):
+        self.r_loss_factor = r_loss_factor
+
+        def vae_r_loss(y_true, y_pred):
+            r_loss = K.mean(K.square(y_true - y_pred), axis = [1,2,3])
+            return r_loss_factor * r_loss
+
+        def vae_kl_loss(y_true, y_pred):
+            kl_loss =  -0.5 * K.sum(1 + self.log_var - K.square(self.mu) - K.exp(self.log_var), axis = 1)
+            return kl_loss
+
+        def vae_loss(y_true, y_pred):
+            r_loss = vae_r_loss(y_true, y_pred)
+            kl_loss = vae_kl_loss(y_true, y_pred)
+            return  r_loss + kl_loss
+
+        self.model.compile(optimizer=optimizer, 
+                           loss = vae_loss,
+                           metrics = [vae_r_loss, vae_kl_loss], 
+                           experimental_run_tf_function=False)
+        print('Compiled.')
+    
+    
+    
+    def train(self, 
+              x_train=None,
+              x_test=None,
+              data_generator=None,
+              batch_size=32, 
+              epochs=20,
+              initial_epoch=0,
+              k_size=1
+             ):
+
+        # ---- Data given or via generator
+        mode_data = (data_generator is None)
+        
+        # ---- Size of the dataset we are going to use
+        #      k_size ==1 : mean 100%
+        #      Unused with data generator
+        #
+        if mode_data:
+            n_train = int(x_train.shape[0] * k_size)
+            n_test  = int(x_test.shape[0]  * k_size)
+                   
+        # ---- Callback : Images
+        filename = self.run_directory+"/images/image-{epoch:03d}-{i:02d}.jpg"
+        callbacks_images = ImagesCallback(filename, z_dim=self.z_dim, decoder=self.decoder)
+        
+        # ---- Callback : Checkpoint
+        filename = self.run_directory+"/models/model-{epoch:03d}.h5"
+        callback_chkpts = ModelCheckpoint(filename, save_freq='epoch' ,verbose=0)
+
+        # ---- Callback : Best model
+        filename = self.run_directory+"/models/best_model.h5"
+        callback_bestmodel = ModelCheckpoint(filename, save_best_only=True, mode='min',monitor='val_loss',verbose=0)
+
+        # ---- Callback tensorboard
+        dirname = self.run_directory+"/logs"
+        callback_tensorboard = TensorBoard(log_dir=dirname, histogram_freq=1)
+
+        callbacks_list = [callbacks_images, callback_chkpts, callback_bestmodel, callback_tensorboard]
+#         callbacks_list = [callback_chkpts, callback_bestmodel, callback_tensorboard]
+
+        # ---- Let's go...
+        start_time   = time.time()
+        
+        if mode_data:
+            #
+            # ---- With pure data (x_train) -----------------------------------------
+            #                             
+            self.history = self.model.fit(x_train[:n_train], x_train[:n_train],
+                                          batch_size = batch_size,
+                                          shuffle = True,
+                                          epochs = epochs,
+                                          initial_epoch = initial_epoch,
+                                          callbacks = callbacks_list,
+                                          validation_data = (x_test[:n_test], x_test[:n_test])
+                                          )
+            #
+        else:
+            # ---- With Data Generator ----------------------------------------------
+            #
+            self.history = self.model.fit(data_generator,
+                                          shuffle = True,
+                                          epochs = epochs,
+                                          initial_epoch = initial_epoch,
+                                          callbacks = callbacks_list,
+                                          validation_data = (x_test, x_test)
+                                         )
+        
+        end_time  = time.time()
+        dt  = end_time-start_time
+        dth = str(datetime.timedelta(seconds=int(dt)))
+        self.duration = dt
+        print(f'\nTrain duration : {dt:.2f} sec. - {dth:}')
+
+
+        
+        
+    def plot_model(self):
+        d=self.run_directory+'/figs'
+        plot_model(self.model,   to_file=f'{d}/model.png',   show_shapes = True, show_layer_names = True, expand_nested=True)
+        plot_model(self.encoder, to_file=f'{d}/encoder.png', show_shapes = True, show_layer_names = True)
+        plot_model(self.decoder, to_file=f'{d}/decoder.png', show_shapes = True, show_layer_names = True)
+
+        
+        
+        
+    def save(self,config='vae_config.json', model='model.h5', force=False):
+        
+        # ---- Check if the place is still used
+        
+        if os.path.isfile(self.run_directory+'/models/best_model.h5') and not force:
+            print('\n*** Oops. There are already stuff in the target folder !\n')
+            assert False, f'Tag directory {self.run_directory} is not empty...'
+           
+        # ---- Save config in json
+        
+        if config!=None:
+            to_save  = ['input_shape', 'encoder_layers', 'decoder_layers', 'z_dim', 'run_tag', 'verbose']
+            data     = { i:self.__dict__[i] for i in to_save }
+            filename = self.run_directory+'/models/'+config
+            with open(filename, 'w') as outfile:
+                json.dump(data, outfile)
+            print(f'Config saved in     : {filename}')
+        
+        # ---- Save model
+        
+        if model!=None:
+            filename = self.run_directory+'/models/'+model
+            self.model.save(filename)
+            print(f'Model saved in      : {filename}')
+
+        
+        
+    def load_weights(self,model='model.h5'):
+        filename = self.run_directory+'/models/'+model
+        self.model.load_weights(filename)
+        print(f'Weights loaded from : {filename}')
+    
+            
+    @classmethod
+    def load(cls, run_tag='000', config='vae_config.json', weights='model.h5'):
+        # ---- Instantiate a new vae
+        filename = f'./run/{run_tag}/models/{config}'
+        with open(filename, 'r') as infile:
+            params=json.load(infile)
+            vae=cls( **params)
+        # ---- weights==None, just return it
+        if weights==None: return vae
+        # ---- weights!=None, get weights
+        vae.load_weights(weights)
+        return vae
+    
+    @classmethod
+    def about(cls):
+        print('\nFIDLE 2020 - Variational AutoEncoder (VAE)')
+        print('TensorFlow version   :',tf.__version__)
+        print('VAE version          :', cls.version)
\ No newline at end of file

environment.yml

+name: fidle
+channels:
+  - defaults
+dependencies:
+  - _libgcc_mutex=0.1=main
+  - _tflow_select=2.1.0=gpu
+  - absl-py=0.9.0=py37_0
+  - astor=0.8.0=py37_0
+  - attrs=19.3.0=py_0
+  - backcall=0.1.0=py37_0
+  - blas=1.0=mkl
+  - bleach=3.1.0=py37_0
+  - blosc=1.16.3=hd408876_0
+  - bzip2=1.0.8=h7b6447c_0
+  - c-ares=1.15.0=h7b6447c_1001
+  - ca-certificates=2020.1.1=0
+  - cairo=1.14.12=h8948797_3
+  - certifi=2019.11.28=py37_0
+  - cloudpickle=1.3.0=py_0
+  - cudatoolkit=10.0.130=0
+  - cudnn=7.6.5=cuda10.0_0
+  - cupti=10.0.130=0
+  - cycler=0.10.0=py37_0
+  - cytoolz=0.10.1=py37h7b6447c_0
+  - dask-core=2.10.1=py_0
+  - dbus=1.13.12=h746ee38_0
+  - decorator=4.4.1=py_0
+  - defusedxml=0.6.0=py_0
+  - entrypoints=0.3=py37_0
+  - expat=2.2.6=he6710b0_0
+  - fontconfig=2.13.0=h9420a91_0
+  - freetype=2.9.1=h8a8886c_1
+  - fribidi=1.0.5=h7b6447c_0
+  - gast=0.2.2=py37_0
+  - glib=2.63.1=h5a9c865_0
+  - gmp=6.1.2=h6c8ec71_1
+  - google-pasta=0.1.8=py_0
+  - graphite2=1.3.13=h23475e2_0
+  - graphviz=2.40.1=h21bd128_2
+  - grpcio=1.16.1=py37hf8bcb03_1
+  - gst-plugins-base=1.14.0=hbbd80ab_1
+  - gstreamer=1.14.0=hb453b48_1
+  - h5py=2.10.0=py37h7918eee_0
+  - harfbuzz=1.8.8=hffaf4a1_0
+  - hdf5=1.10.4=hb1b8bf9_0
+  - icu=58.2=h9c2bf20_1
+  - imageio=2.6.1=py37_0
+  - importlib_metadata=1.5.0=py37_0
+  - intel-openmp=2020.0=166
+  - ipykernel=5.1.4=py37h39e3cac_0
+  - ipython=7.12.0=py37h5ca1d4c_0
+  - ipython_genutils=0.2.0=py37_0
+  - jedi=0.16.0=py37_0
+  - jinja2=2.11.1=py_0
+  - joblib=0.14.1=py_0
+  - jpeg=9b=h024ee3a_2
+  - json5=0.9.1=py_0
+  - jsonschema=3.2.0=py37_0
+  - jupyter_client=5.3.4=py37_0
+  - jupyter_core=4.6.1=py37_0
+  - jupyterlab=1.2.6=pyhf63ae98_0
+  - jupyterlab_server=1.0.6=py_0
+  - keras-applications=1.0.8=py_0
+  - keras-preprocessing=1.1.0=py_1
+  - kiwisolver=1.1.0=py37he6710b0_0
+  - ld_impl_linux-64=2.33.1=h53a641e_7
+  - libedit=3.1.20181209=hc058e9b_0
+  - libffi=3.2.1=hd88cf55_4
+  - libgcc-ng=9.1.0=hdf63c60_0
+  - libgfortran-ng=7.3.0=hdf63c60_0
+  - libpng=1.6.37=hbc83047_0
+  - libprotobuf=3.11.3=hd408876_0
+  - libsodium=1.0.16=h1bed415_0
+  - libstdcxx-ng=9.1.0=hdf63c60_0
+  - libtiff=4.1.0=h2733197_0
+  - libuuid=1.0.3=h1bed415_2
+  - libxcb=1.13=h1bed415_1
+  - libxml2=2.9.9=hea5a465_1
+  - lz4-c=1.8.1.2=h14c3975_0
+  - lzo=2.10=h49e0be7_2
+  - markdown=3.1.1=py37_0
+  - markupsafe=1.1.1=py37h7b6447c_0
+  - matplotlib=3.1.3=py37_0
+  - matplotlib-base=3.1.3=py37hef1b27d_0
+  - mistune=0.8.4=py37h7b6447c_0
+  - mkl=2020.0=166
+  - mkl-service=2.3.0=py37he904b0f_0
+  - mkl_fft=1.0.15=py37ha843d7b_0
+  - mkl_random=1.1.0=py37hd6b4f25_0
+  - mock=4.0.1=py_0
+  - more-itertools=8.2.0=py_0
+  - nbconvert=5.6.1=py37_0
+  - nbformat=5.0.4=py_0
+  - ncurses=6.1=he6710b0_1
+  - networkx=2.4=py_0
+  - notebook=6.0.3=py37_0
+  - numexpr=2.7.1=py37h423224d_0
+  - numpy=1.18.1=py37h4f9e942_0
+  - numpy-base=1.18.1=py37hde5b4d6_1
+  - olefile=0.46=py37_0
+  - openssl=1.1.1d=h7b6447c_4
+  - opt_einsum=3.1.0=py_0
+  - pandas=1.0.1=py37h0573a6f_0
+  - pandoc=2.2.3.2=0
+  - pandocfilters=1.4.2=py37_1
+  - pango=1.42.4=h049681c_0
+  - parso=0.6.1=py_0
+  - patsy=0.5.1=py37_0
+  - pcre=8.43=he6710b0_0
+  - pexpect=4.8.0=py37_0
+  - pickleshare=0.7.5=py37_0
+  - pillow=7.0.0=py37hb39fc2d_0
+  - pip=20.0.2=py37_1
+  - pixman=0.38.0=h7b6447c_0
+  - prometheus_client=0.7.1=py_0
+  - prompt_toolkit=3.0.3=py_0
+  - protobuf=3.11.3=py37he6710b0_0
+  - ptyprocess=0.6.0=py37_0
+  - pydot=1.4.1=py37_0
+  - pygments=2.5.2=py_0
+  - pyparsing=2.4.6=py_0
+  - pyqt=5.9.2=py37h05f1152_2
+  - pyrsistent=0.15.7=py37h7b6447c_0
+  - pytables=3.6.1=py37h71ec239_0
+  - python=3.7.6=h0371630_2
+  - python-dateutil=2.8.1=py_0
+  - pytz=2019.3=py_0
+  - pywavelets=1.1.1=py37h7b6447c_0
+  - pyzmq=18.1.1=py37he6710b0_0
+  - qt=5.9.7=h5867ecd_1
+  - readline=7.0=h7b6447c_5
+  - scikit-image=0.16.2=py37h0573a6f_0
+  - scikit-learn=0.22.1=py37hd81dba3_0
+  - scipy=1.4.1=py37h0b6359f_0
+  - seaborn=0.10.0=py_0
+  - send2trash=1.5.0=py37_0
+  - setuptools=45.2.0=py37_0
+  - sip=4.19.8=py37hf484d3e_0
+  - six=1.14.0=py37_0
+  - snappy=1.1.7=hbae5bb6_3
+  - sqlite=3.31.1=h7b6447c_0
+  - statsmodels=0.11.0=py37h7b6447c_0
+  - tensorboard=2.0.0=pyhb38c66f_1
+  - tensorflow=2.0.0=gpu_py37h768510d_0
+  - tensorflow-base=2.0.0=gpu_py37h0ec5d1f_0
+  - tensorflow-estimator=2.0.0=pyh2649769_0
+  - tensorflow-gpu=2.0.0=h0d30ee6_0
+  - termcolor=1.1.0=py37_1
+  - terminado=0.8.3=py37_0
+  - testpath=0.4.4=py_0
+  - tk=8.6.8=hbc83047_0
+  - toolz=0.10.0=py_0
+  - tornado=6.0.3=py37h7b6447c_3
+  - traitlets=4.3.3=py37_0
+  - wcwidth=0.1.8=py_0
+  - webencodings=0.5.1=py37_1
+  - werkzeug=0.16.0=py_0
+  - wheel=0.34.2=py37_0
+  - wrapt=1.11.2=py37h7b6447c_0
+  - xz=5.2.4=h14c3975_4
+  - zeromq=4.3.1=he6710b0_3
+  - zipp=2.2.0=py_0
+  - zlib=1.2.11=h7b6447c_3
+  - zstd=1.3.7=h0b5b093_0

fidle/Charte.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
+
+# <!-- TITLE --> Titre_du_notebook
+<!-- DESC --> Description_du_notebook_et_de_sa_thématique
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Objectif
+ - Objectif_pédagogique
+
+
+A_propos_du_dataset
+
+## What we're going to do :
+
+ - Ceci
+ - Cela
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>
+
+%% Cell type:code id: tags:
+
+``` python
+```