VAE.Keras3/02-VAE-with-MNIST.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3VAE2] - VAE, using a custom model class  (MNIST dataset)
+<!-- DESC --> Construction and training of a VAE, using model subclass, with a latent space of small dimension.
+
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Understanding and implementing a **variational autoencoder** neurals network (VAE)
+ - Understanding a still more **advanced programming model**, using a **custom model**
+
+The calculation needs being important, it is preferable to use a very simple dataset such as MNIST to start with.
+...MNIST with a small scale if you haven't a GPU ;-)
+
+## What we're going to do :
+
+ - Defining a VAE model
+ - Build the model
+ - Train it
+ - Have a look on the train process
+
+## Acknowledgements :
+Thanks to **François Chollet** who is at the base of this example (and the creator of Keras !!).
+See : https://keras.io/examples/generative/vae
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init python stuff
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+from keras import layers
+
+import numpy as np
+
+from modules.models    import VAE
+from modules.layers    import SamplingLayer
+from modules.callbacks import ImagesCallback
+from modules.datagen   import MNIST
+
+
+import matplotlib.pyplot as plt
+import scipy.stats
+import sys
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3VAE2')
+
+VAE.about()
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Parameters
+`scale` : with scale=1, we need 1'30s on a GPU V100 ...and >20' on a CPU !
+`latent_dim` : 2 dimensions is small, but usefull to draw !
+`fit_verbosity`: Verbosity of training progress bar: 0=silent, 1=progress bar, 2=One line
+
+`loss_weights` : Our **loss function** is the weighted sum of two loss:
+ - `r_loss` which measures the loss during reconstruction.
+ - `kl_loss` which measures the dispersion.
+
+The weights are defined by: `loss_weights=[k1,k2]` where : `total_loss = k1*r_loss + k2*kl_loss`
+In practice, a value of \[1,.06\] gives good results here.
+
+%% Cell type:code id: tags:
+
+``` python
+latent_dim    = 6
+loss_weights  = [1,.06]
+
+scale         = .2
+seed          = 123
+
+batch_size    = 64
+epochs        = 4
+fit_verbosity = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('latent_dim', 'loss_weights', 'scale', 'seed', 'batch_size', 'epochs', 'fit_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Prepare data
+`MNIST.get_data()` return : `x_train,y_train, x_test,y_test`,  \
+but we only need x_train for our training.
+
+%% Cell type:code id: tags:
+
+``` python
+x_data, y_data, _,_ = MNIST.get_data(seed=seed, scale=scale, train_prop=1 )
+
+fidle.scrawler.images(x_data[:20], None, indices='all', columns=10, x_size=1,y_size=1,y_padding=0, save_as='01-original')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Build model
+In this example, we will use a **custom model**.
+For this, we will use :
+ - `SamplingLayer`, which generates a vector z from the parameters z_mean and z_log_var - See : [SamplingLayer.py](./modules/layers/SamplingLayer.py)
+ - `VAE`, a custom model with a specific train_step - See : [VAE.py](./modules/models/VAE.py)
+
+%% Cell type:markdown id: tags:
+
+#### Encoder
+
+%% Cell type:code id: tags:
+
+``` python
+inputs    = keras.Input(shape=(28, 28, 1))
+x         = layers.Conv2D(32, 3, strides=1, padding="same", activation="relu")(inputs)
+x         = layers.Conv2D(64, 3, strides=2, padding="same", activation="relu")(x)
+x         = layers.Conv2D(64, 3, strides=2, padding="same", activation="relu")(x)
+x         = layers.Conv2D(64, 3, strides=1, padding="same", activation="relu")(x)
+x         = layers.Flatten()(x)
+x         = layers.Dense(16, activation="relu")(x)
+
+z_mean    = layers.Dense(latent_dim, name="z_mean")(x)
+z_log_var = layers.Dense(latent_dim, name="z_log_var")(x)
+z         = SamplingLayer()([z_mean, z_log_var])
+
+encoder = keras.Model(inputs, [z_mean, z_log_var, z], name="encoder")
+encoder.compile()
+```
+
+%% Cell type:markdown id: tags:
+
+#### Decoder
+
+%% Cell type:code id: tags:
+
+``` python
+inputs  = keras.Input(shape=(latent_dim,))
+x       = layers.Dense(7 * 7 * 64, activation="relu")(inputs)
+x       = layers.Reshape((7, 7, 64))(x)
+x       = layers.Conv2DTranspose(64, 3, strides=1, padding="same", activation="relu")(x)
+x       = layers.Conv2DTranspose(64, 3, strides=2, padding="same", activation="relu")(x)
+x       = layers.Conv2DTranspose(32, 3, strides=2, padding="same", activation="relu")(x)
+outputs = layers.Conv2DTranspose(1,  3, padding="same", activation="sigmoid")(x)
+
+decoder = keras.Model(inputs, outputs, name="decoder")
+decoder.compile()
+```
+
+%% Cell type:markdown id: tags:
+
+#### VAE
+`VAE` is a custom model with a specific train_step - See : [VAE.py](./modules/models/VAE.py)
+
+%% Cell type:code id: tags:
+
+``` python
+vae = VAE(encoder, decoder, loss_weights)
+
+vae.compile(optimizer='adam')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Train
+### 5.1 - Using two nice custom callbacks :-)
+Two custom callbacks are used:
+ - `ImagesCallback` : qui va sauvegarder des images durant l'apprentissage - See [ImagesCallback.py](./modules/callbacks/ImagesCallback.py)
+ - `BestModelCallback` : qui sauvegardera le meilleur model - See [BestModelCallback.py](./modules/callbacks/BestModelCallback.py)
+
+%% Cell type:code id: tags:
+
+``` python
+callback_images      = ImagesCallback(x=x_data, z_dim=latent_dim, nb_images=5, from_z=True, from_random=True, run_dir=run_dir)
+
+callbacks_list = [callback_images]
+```
+
+%% Cell type:markdown id: tags:
+
+### 5.2 - Let's train !
+With `scale=1`, need 1'15 on a GPU (V100 at IDRIS) ...or 20' on a CPU
+
+%% Cell type:code id: tags:
+
+``` python
+chrono=fidle.Chrono()
+chrono.start()
+
+history = vae.fit(x_data, epochs=epochs, batch_size=batch_size, callbacks=callbacks_list, verbose=fit_verbosity)
+
+chrono.show()
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Training review
+### 6.1 - History
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history(history,  plot={"Loss":['loss']}, save_as='history')
+```
+
+%% Cell type:markdown id: tags:
+
+### 6.2 - Reconstruction during training
+At the end of each epoch, our callback saved some reconstructed images.
+Where :
+Original image -> encoder -> z -> decoder -> Reconstructed image
+
+%% Cell type:code id: tags:
+
+``` python
+images_z, images_r = callback_images.get_images( range(0,epochs,2) )
+
+fidle.utils.subtitle('Original images :')
+fidle.scrawler.images(x_data[:5], None, indices='all', columns=5, x_size=2,y_size=2, save_as='02-original')
+
+fidle.utils.subtitle('Encoded/decoded images')
+fidle.scrawler.images(images_z, None, indices='all', columns=5, x_size=2,y_size=2, save_as='03-reconstruct')
+
+fidle.utils.subtitle('Original images :')
+fidle.scrawler.images(x_data[:5], None, indices='all', columns=5, x_size=2,y_size=2, save_as=None)
+```
+
+%% Cell type:markdown id: tags:
+
+### 6.3 - Generation (latent -> decoder) during training
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.utils.subtitle('Generated images from latent space')
+fidle.scrawler.images(images_r, None, indices='all', columns=5, x_size=2,y_size=2, save_as='04-encoded')
+```
+
+%% Cell type:markdown id: tags:
+
+### 6.4 - Save model
+
+%% Cell type:code id: tags:
+
+``` python
+os.makedirs(f'{run_dir}/models', exist_ok=True)
+
+vae.save(f'{run_dir}/models/vae_model.keras')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Model evaluation
+
+%% Cell type:markdown id: tags:
+
+### 7.1 - Reload model
+
+%% Cell type:code id: tags:
+
+``` python
+vae=VAE()
+vae.reload(f'{run_dir}/models/vae_model.keras')
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.2 - Image reconstruction
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Select few images
+
+x_show = fidle.utils.pick_dataset(x_data, n=10)
+
+# ---- Get latent points and reconstructed images
+
+z_mean, z_var, z  = vae.encoder.predict(x_show)
+x_reconst         = vae.decoder.predict(z)
+
+# ---- Show it
+
+labels=[ str(np.round(z[i],1)) for i in range(10) ]
+fidle.scrawler.images(x_show,    None, indices='all', columns=10, x_size=2,y_size=2, save_as='05-original')
+fidle.scrawler.images(x_reconst, None, indices='all', columns=10, x_size=2,y_size=2, save_as='06-reconstruct')
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.3 - Visualization of the latent space
+
+%% Cell type:code id: tags:
+
+``` python
+n_show = int(20000*scale)
+
+# ---- Select images
+
+x_show, y_show = fidle.utils.pick_dataset(x_data,y_data, n=n_show)
+
+# ---- Get latent points
+
+z_mean, z_var, z = vae.encoder.predict(x_show)
+
+# ---- Show them
+
+fig = plt.figure(figsize=(14, 10))
+plt.scatter(z[:, 0] , z[:, 1], c=y_show, cmap= 'tab10', alpha=0.5, s=30)
+plt.colorbar()
+fidle.scrawler.save_fig('07-Latent-space')
+plt.show()
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.4 - Generative latent space
+
+%% Cell type:code id: tags:
+
+``` python
+if latent_dim>2:
+
+    print('Sorry, This part can only work if the latent space is of dimension 2')
+
+else:
+
+    grid_size   = 18
+    grid_scale  = 1
+
+    # ---- Draw a ppf grid
+
+    grid=[]
+    for y in scipy.stats.norm.ppf(np.linspace(0.99, 0.01, grid_size),scale=grid_scale):
+        for x in scipy.stats.norm.ppf(np.linspace(0.01, 0.99, grid_size),scale=grid_scale):
+            grid.append( (x,y) )
+    grid=np.array(grid)
+
+    # ---- Draw latentspoints and grid
+
+    fig = plt.figure(figsize=(10, 8))
+    plt.scatter(z[:, 0] , z[:, 1], c=y_show, cmap= 'tab10', alpha=0.5, s=20)
+    plt.scatter(grid[:, 0] , grid[:, 1], c = 'black', s=60, linewidth=2, marker='+', alpha=1)
+    fidle.scrawler.save_fig('08-Latent-grid')
+    plt.show()
+
+    # ---- Plot grid corresponding images
+
+    x_reconst = vae.decoder.predict([grid])
+    fidle.scrawler.images(x_reconst, indices='all', columns=grid_size, x_size=0.5,y_size=0.5, y_padding=0,spines_alpha=0.1, save_as='09-Latent-morphing')
+
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

VAE.Keras3/03-VAE-with-MNIST-post.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3VAE3] - Analysis of the VAE's latent space of MNIST dataset
+<!-- DESC --> Visualization and analysis of the VAE's latent space of the dataset MNIST
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - First data generation from **latent space**
+ - Understanding of underlying principles
+ - Model management
+
+Here, we don't consume data anymore, but we generate them ! ;-)
+
+## What we're going to do :
+
+ - Load a saved model
+ - Reconstruct some images
+ - Latent space visualization
+ - Matrix of generated images
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init python stuff
+
+%% Cell type:markdown id: tags:
+
+### 1.1 - Init python
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+from keras import layers
+
+import numpy as np
+
+from modules.models    import VAE
+from modules.datagen   import MNIST
+
+import matplotlib
+import matplotlib.pyplot as plt
+from barviz import Simplex
+from barviz import Collection
+
+import sys
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3VAE3')
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Parameters
+
+%% Cell type:code id: tags:
+
+``` python
+scale      = 1
+seed       = 123
+models_dir = './run/K3VAE2'
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('scale', 'seed', 'models_dir')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Get data
+
+%% Cell type:code id: tags:
+
+``` python
+x_data, y_data, _,_ = MNIST.get_data(seed=seed, scale=scale, train_prop=1 )
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Reload best model
+
+%% Cell type:code id: tags:
+
+``` python
+vae=VAE()
+vae.reload(f'{models_dir}/models/vae_model')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Image reconstruction
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Select few images
+
+x_show = fidle.utils.pick_dataset(x_data, n=10)
+
+# ---- Get latent points and reconstructed images
+
+z_mean, z_var, z  = vae.encoder.predict(x_show, verbose=0)
+x_reconst         = vae.decoder.predict(z,      verbose=0)
+
+latent_dim        = z.shape[1]
+
+# ---- Show it
+
+labels=[ str(np.round(z[i],1)) for i in range(10) ]
+fidle.utils.subtitle('Originals :')
+fidle.scrawler.images(x_show,    None, indices='all', columns=10, x_size=2,y_size=2, save_as='01-original')
+fidle.utils.subtitle('Reconstructed :')
+fidle.scrawler.images(x_reconst, None, indices='all', columns=10, x_size=2,y_size=2, save_as='02-reconstruct')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Visualizing the latent space
+
+%% Cell type:code id: tags:
+
+``` python
+n_show = min( 20000, len(x_data) )
+
+# ---- Select images
+
+x_show, y_show = fidle.utils.pick_dataset(x_data,y_data, n=n_show)
+
+# ---- Get latent points
+
+z_mean, z_var, z = vae.encoder.predict(x_show, verbose=0)
+```
+
+%% Cell type:markdown id: tags:
+
+### 5.1 - Classic 2d visualisaton
+
+%% Cell type:code id: tags:
+
+``` python
+fig = plt.figure(figsize=(14, 10))
+plt.scatter(z[:, 2] , z[:, 4], c=y_show, cmap= 'tab10', alpha=0.5, s=30)
+plt.colorbar()
+fidle.scrawler.save_fig('03-Latent-space')
+plt.show()
+```
+
+%% Cell type:markdown id: tags:
+
+### 5.2 - Simplex visualisaton
+
+%% Cell type:code id: tags:
+
+``` python
+if latent_dim<4:
+
+    print('Sorry, This part can only work if the latent space is greater than 3')
+
+else:
+
+    # ---- Softmax rescale
+    #
+    zs = np.exp(z)/np.sum(np.exp(z),axis=1,keepdims=True)
+    # zc  = zs * 1/np.max(zs)
+
+    # ---- Create collection
+    #
+    c = Collection(zs, colors=y_show, labels=y_show)
+    c.attrs.markers_colormap     = {'colorscale':'Rainbow','cmin':0,'cmax':latent_dim}
+    c.attrs.markers_size         = 5
+    c.attrs.markers_border_width = 0
+    c.attrs.markers_opacity      = 0.8
+
+    s = Simplex.build(latent_dim)
+    s.attrs.width  = 1000
+    s.attrs.height = 1000
+    s.plot(c)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Generate from latent space (latent_dim==2)
+
+%% Cell type:code id: tags:
+
+``` python
+if latent_dim>2:
+
+    print('Sorry, This part can only work if the latent space is of dimension 2')
+
+else:
+
+    grid_size   = 14
+    grid_scale  = 1.
+
+    # ---- Draw a ppf grid
+
+    grid=[]
+    for y in scipy.stats.norm.ppf(np.linspace(0.99, 0.01, grid_size),scale=grid_scale):
+        for x in scipy.stats.norm.ppf(np.linspace(0.01, 0.99, grid_size),scale=grid_scale):
+            grid.append( (x,y) )
+    grid=np.array(grid)
+
+    # ---- Draw latentspoints and grid
+
+    fig = plt.figure(figsize=(12, 10))
+    plt.scatter(z[:, 0] , z[:, 1], c=y_show, cmap= 'tab10', alpha=0.5, s=20)
+    plt.scatter(grid[:, 0] , grid[:, 1], c = 'black', s=60, linewidth=2, marker='+', alpha=1)
+    fidle.scrawler.save_fig('04-Latent-grid')
+    plt.show()
+
+    # ---- Plot grid corresponding images
+
+    x_reconst = vae.decoder.predict([grid])
+    fidle.scrawler.images(x_reconst, indices='all', columns=grid_size, x_size=0.5,y_size=0.5, y_padding=0,spines_alpha=0.1, save_as='05-Latent-morphing')
+
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

VAE.Keras3/modules/callbacks/ImagesCallback.py

+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|                            ImageCallback
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/SARI/DEVLOG 2020 - S. Arias, E. Maldonado, JL. Parouty
+# ------------------------------------------------------------------
+# 2.0 version by JL Parouty, feb 2021
+
+from keras.callbacks import Callback
+import numpy as np
+import matplotlib.pyplot as plt
+from skimage import io
+import os
+
+class ImagesCallback(Callback):
+    '''
+    Save generated (random mode) or encoded/decoded (z mode) images on epoch end.
+    params:
+        x           : input images, for z mode (None)
+        z_dim       : size of the latent space, for random mode (None)
+        nb_images   : number of images to save
+        from_z      : save images from z (False)
+        from_random : save images from random (False)
+        filename    : images filename
+        run_dir     : output directory to save images        
+    '''
+    
+   
+    def __init__(self, x           = None,
+                       z_dim       = None,
+                       nb_images   = 5,
+                       from_z      = False, 
+                       from_random = False,
+                       filename    = 'image-{epoch:03d}-{i:02d}.jpg',
+                       run_dir     = './run'):
+        
+        # ---- Parameters
+        #
+        
+        self.x = None if x is None else x[:nb_images]
+        self.z_dim       = z_dim
+        
+        self.nb_images   = nb_images
+        self.from_z      = from_z
+        self.from_random = from_random
+
+        self.filename_z       = run_dir + '/images-z/'      + filename
+        self.filename_random  = run_dir + '/images-random/' + filename
+        
+        if from_z:      os.makedirs( run_dir + '/images-z/',     mode=0o750, exist_ok=True)
+        if from_random: os.makedirs( run_dir + '/images-random/', mode=0o750, exist_ok=True)
+        
+    
+    
+    def save_images(self, images, filename, epoch):
+        '''Save images as <filename>'''
+        
+        for i,image in enumerate(images):
+            
+            image = image.squeeze()  # Squeeze it if monochrome : (lx,ly,1) -> (lx,ly) 
+        
+            filenamei = filename.format(epoch=epoch,i=i)
+            
+            if len(image.shape) == 2:
+                plt.imsave(filenamei, image, cmap='gray_r')
+            else:
+                plt.imsave(filenamei, image)
+
+    
+    
+    def on_epoch_end(self, epoch, logs={}):
+        '''Called at the end of each epoch'''
+        
+        encoder     = self.model.get_layer('encoder')
+        decoder     = self.model.get_layer('decoder')
+
+        if self.from_random:
+            z      = np.random.normal( size=(self.nb_images,self.z_dim) )
+            images = decoder.predict(z)
+            self.save_images(images, self.filename_random, epoch)
+            
+        if self.from_z:
+            z_mean, z_var, z  = encoder.predict(self.x)
+            images            = decoder.predict(z)
+            self.save_images(images, self.filename_z, epoch)
+
+
+    def get_images(self, epochs=None, from_z=True,from_random=True):
+        '''Read and return saved images. epochs is a range'''
+        if epochs is None : return
+        images_z = []
+        images_r = []
+        for epoch in list(epochs):
+            for i in range(self.nb_images):
+                if from_z:
+                    f = self.filename_z.format(epoch=epoch,i=i)
+                    images_z.append( io.imread(f) )
+                if from_random:
+                    f = self.filename_random.format(epoch=epoch,i=i)
+                    images_r.append( io.imread(f) )
+        return images_z, images_r
+            

VAE.Keras3/modules/callbacks/__init__.py

+from modules.callbacks.ImagesCallback    import ImagesCallback
\ No newline at end of file

VAE.Keras3/modules/datagen/MNIST.py

+
+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|                        MNIST Data loader
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI - https://fidle.cnrs.fr
+# ------------------------------------------------------------------
+# JL Parouty (Mars 2024)
+
+
+import h5py
+import os
+import numpy as np
+from hashlib import blake2b
+import keras
+import keras.datasets.mnist as mnist
+
+
+# ------------------------------------------------------------------
+#   A usefull class to manage our MNIST dataset
+#   This class allows to manage datasets derived from the original MNIST
+# ------------------------------------------------------------------
+
+
+class MNIST():
+    
+    version = '0.1'
+    
+    def __init__(self):
+        pass
+   
+    @classmethod
+    def get_data(cls, normalize=True, expand=True, scale=1., train_prop=0.8, shuffle=True, seed=None):
+        """
+        Return original MNIST dataset
+        args:
+            normalize   : Normalize dataset or not (True)
+            expand      : Reshape images as (28,28,1) instead (28,28) (True)
+            scale      : Scale of dataset to use. 1. mean 100% (1.)
+            train_prop : Ratio of train/test (0.8)
+            shuffle    : Shuffle data if True (True)
+            seed       : Random seed value. False mean no seed, None mean using /dev/urandom (None)
+        returns:
+            x_train,y_train,x_test,y_test
+        """
+
+        # ---- Seed
+        #
+        if seed is not False:
+            np.random.seed(seed)
+            print(f'Seeded ({seed})')
+
+        # ---- Get data
+        #
+        (x_train, y_train), (x_test, y_test) = mnist.load_data()
+        print('Dataset loaded.')
+        
+        # ---- Concatenate
+        #
+        x_data = np.concatenate([x_train, x_test], axis=0)
+        y_data = np.concatenate([y_train, y_test])
+        print('Concatenated.')
+
+        # ---- Shuffle
+        #
+        if shuffle:
+            p = np.random.permutation(len(x_data))
+            x_data, y_data = x_data[p], y_data[p]
+            print('Shuffled.')     
+        
+        # ---- Rescale
+        #
+        n = int(scale*len(x_data))
+        x_data, y_data = x_data[:n], y_data[:n]
+        print(f'rescaled ({scale}).') 
+
+        # ---- Normalization
+        #
+        if normalize:
+            x_data = x_data.astype('float32') / 255.
+            print('Normalized.')
+            
+        # ---- Reshape : (28,28) -> (28,28,1)
+        #
+        if expand:
+            x_data = np.expand_dims(x_data, axis=-1)
+            print('Reshaped.')
+
+        # ---- Split
+        #
+        n=int(len(x_data)*train_prop)
+        x_train, x_test = x_data[:n], x_data[n:]
+        y_train, y_test = y_data[:n], y_data[n:]
+        print(f'splited ({train_prop}).') 
+
+        # ---- Hash
+        #
+        h = blake2b(digest_size=10)
+        for a in [x_train,x_test, y_train,y_test]:
+            h.update(a)
+            
+        # ---- About and return
+        #
+        print('x_train shape is  : ', x_train.shape)
+        print('x_test  shape is  : ', x_test.shape)
+        print('y_train shape is  : ', y_train.shape)
+        print('y_test  shape is  : ', y_test.shape)
+        print('Blake2b digest is : ', h.hexdigest())
+        return  x_train,y_train, x_test,y_test
+                
+            

VAE.Keras3/modules/datagen/__init__.py

+from modules.datagen.MNIST         import MNIST
+
+

VAE.Keras3/modules/layers/SamplingLayer.py

+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|                            SamplingLayer
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI - https://fidle.cnrs.fr
+# ------------------------------------------------------------------
+# JL Parouty (Mars 2024)
+
+import keras
+import torch
+from torch.distributions.normal import Normal
+
+# Note : https://keras.io/guides/making_new_layers_and_models_via_subclassing/
+
+class SamplingLayer(keras.layers.Layer):
+    '''A custom layer that receive (z_mean, z_var) and sample a z vector'''
+
+    def call(self, inputs):
+        
+        z_mean, z_log_var = inputs
+        
+        batch_size, latent_dim = z_mean.shape
+        
+        epsilon = Normal(0, 1).sample((batch_size, latent_dim)).to(z_mean.device)
+
+        z = z_mean + torch.exp(0.5 * z_log_var) * epsilon 
+        
+        return z
\ No newline at end of file

VAE.Keras3/modules/layers/VariationalLossLayer.py

+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|                            SamplingLayer
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI - https://fidle.cnrs.fr
+# ------------------------------------------------------------------
+# JL Parouty (mars 2024)
+
+import keras
+import torch
+
+# See : https://keras.io/guides/making_new_layers_and_models_via_subclassing/
+
+class VariationalLossLayer(keras.layers.Layer):
+   
+    def __init__(self, loss_weights=[3,7]):
+        super().__init__()
+        self.k1 = loss_weights[0]
+        self.k2 = loss_weights[1]
+
+
+    def call(self, inputs):
+        
+        k1 = self.k1
+        k2 = self.k2
+        
+        # ---- Retrieve inputs
+        #
+        x, z_mean, z_log_var, y = inputs
+        
+        # ---- Compute : reconstruction loss
+        #                
+        r_loss = torch.nn.functional.binary_cross_entropy(y, x, reduction='sum')
+        
+        #
+        # ---- Compute : kl_loss
+        # 
+        kl_loss = - torch.sum(1+ z_log_var - z_mean.pow(2) - z_log_var.exp())
+        
+        # ---- Compute total loss, and add it
+        #
+        loss = r_loss*k1 + kl_loss*k2
+        self.add_loss(loss)
+        
+        return y
+
+    
+    def get_config(self):
+        return {'loss_weights':[self.k1,self.k2]}
+    
\ No newline at end of file

VAE.Keras3/modules/layers/__init__.py

+from modules.layers.SamplingLayer        import SamplingLayer
+from modules.layers.VariationalLossLayer import VariationalLossLayer

VAE.Keras3/modules/models/VAE.py

+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|                              VAE Example
+# ------------------------------------------------------------------
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI - https://fidle.cnrs.fr
+# ------------------------------------------------------------------
+# JL Parouty (mars 2024
+
+import numpy as np
+import keras
+import torch
+
+from IPython.display import display,Markdown
+from modules.layers    import SamplingLayer
+import os
+
+
+# Note : https://keras.io/guides/making_new_layers_and_models_via_subclassing/
+
+
+
+class VAE(keras.Model):
+    '''
+    A VAE model, built from given encoder and decoder
+    '''
+
+    version = '2.0'
+
+    def __init__(self, encoder=None, decoder=None, loss_weights=[1,1], **kwargs):
+        '''
+        VAE instantiation with encoder, decoder and r_loss_factor
+        args :
+            encoder : Encoder model
+            decoder : Decoder model
+            loss_weights : Weight of the loss functions: reconstruction_loss and kl_loss
+            r_loss_factor : Proportion of reconstruction loss for global loss (0.3)
+        return:
+            None
+        '''
+        super(VAE, self).__init__(**kwargs)
+        self.encoder      = encoder
+        self.decoder      = decoder
+        self.loss_weights = loss_weights
+        print(f'Fidle VAE is ready :-)  loss_weights={list(self.loss_weights)}')
+       
+        
+    def call(self, inputs):
+        '''
+        Model forward pass, when we use our model
+        args:
+            inputs : Model inputs
+        return:
+            output : Output of the model 
+        '''
+        z_mean, z_log_var, z = self.encoder(inputs)
+        output               = self.decoder(z)
+        return output
+                
+        
+    def train_step(self, input):
+        '''
+        Implementation of the training update.
+        Receive an input, compute loss, get gradient, update weights and return metrics.
+        Here, our metrics are loss.
+        args:
+            inputs : Model inputs
+        return:
+            loss    : Total loss
+            r_loss  : Reconstruction loss
+            kl_loss : KL loss
+        '''
+        
+        # ---- Get the input we need, specified in the .fit()
+        #
+        if isinstance(input, tuple):
+            input = input[0]
+        
+        k1,k2 = self.loss_weights
+        
+        # ---- Reset grad
+        #
+        self.zero_grad()
+        
+        # ---- Forward pass
+        #
+        # Get encoder outputs
+        #
+        z_mean, z_log_var, z = self.encoder(input)
+            
+        # ---- Get reconstruction from decoder
+        #
+        reconstruction       = self.decoder(z)
+        
+        # ---- Compute loss
+        #      Total loss = Reconstruction loss + KL loss
+        #
+        r_loss  = torch.nn.functional.binary_cross_entropy(reconstruction, input, reduction='sum')
+        kl_loss = - torch.sum(1+ z_log_var - z_mean.pow(2) - z_log_var.exp())
+        loss    = r_loss*k1 + kl_loss*k2
+        
+        # ---- Compute gradients for the weights
+        #
+        loss.backward()
+        
+        # ---- Adjust learning weights
+        #
+        trainable_weights = [v for v in self.trainable_weights]
+        gradients = [v.value.grad for v in trainable_weights]
+
+        with torch.no_grad():
+            self.optimizer.apply(gradients, trainable_weights)
+        
+        # ---- Update metrics (includes the metric that tracks the loss)
+        #
+        for metric in self.metrics:
+            if metric.name == "loss":
+                metric.update_state(loss)
+            else:
+                metric.update_state(input, reconstruction)
+        
+        # ---- Return a dict mapping metric names to current value
+        # Note that it will include the loss (tracked in self.metrics).
+        #
+        return {m.name: m.result() for m in self.metrics}
+        
+        
+        
+        
+        # # ---- Forward pass
+        # #      Run the forward pass and record 
+        # #      operations on the GradientTape.
+        # #
+        # with tf.GradientTape() as tape:
+            
+        #     # ---- Get encoder outputs
+        #     #
+        #     z_mean, z_log_var, z = self.encoder(input)
+            
+        #     # ---- Get reconstruction from decoder
+        #     #
+        #     reconstruction       = self.decoder(z)
+         
+        #     # ---- Compute loss
+        #     #      Reconstruction loss, KL loss and Total loss
+        #     #
+        #     reconstruction_loss  = k1 * tf.reduce_mean( keras.losses.binary_crossentropy(input, reconstruction) )
+
+        #     kl_loss = 1 + z_log_var - tf.square(z_mean) - tf.exp(z_log_var)
+        #     kl_loss = -tf.reduce_mean(kl_loss) * k2
+
+        #     total_loss = reconstruction_loss + kl_loss
+
+        # # ---- Retrieve gradients from gradient_tape
+        # #      and run one step of gradient descent
+        # #      to optimize trainable weights
+        # #
+        # grads = tape.gradient(total_loss, self.trainable_weights)
+        # self.optimizer.apply_gradients(zip(grads, self.trainable_weights))
+        
+        # return {
+        #     "loss":     total_loss,
+        #     "r_loss":   reconstruction_loss,
+        #     "kl_loss":  kl_loss,
+        # }
+    
+    
+    def predict(self,inputs):
+        '''Our predict function...'''
+        z_mean, z_var, z  = self.encoder.predict(inputs)
+        outputs           = self.decoder.predict(z)
+        return outputs
+
+        
+    def save(self,filename):
+        '''Save model in 2 part'''
+        filename, extension = os.path.splitext(filename)
+        self.encoder.save(f'{filename}-encoder.keras')
+        self.decoder.save(f'{filename}-decoder.keras')
+
+    
+    def reload(self,filename):
+        '''Reload a 2 part saved model.'''
+        filename, extension = os.path.splitext(filename)
+        self.encoder = keras.models.load_model(f'{filename}-encoder.keras', custom_objects={'SamplingLayer': SamplingLayer})
+        self.decoder = keras.models.load_model(f'{filename}-decoder.keras')
+        print('Reloaded.')
+                
+        
+    @classmethod
+    def about(cls):
+        '''Basic whoami method'''
+        display(Markdown('<br>**FIDLE 2024 - VAE**'))
+        print('Version              :', cls.version)
+        print('Keras version        :', keras.__version__)

VAE.Keras3/modules/models/__init__.py

+from modules.models.VAE import VAE
\ No newline at end of file

VAE/03.1-Batch.ipynb

-%% Cell type:markdown id: tags:
-
-Celeb Faces Dataset (CelebA)
-=================================================
----
-Introduction au Deep Learning  (IDLE) - S. Arias, E. Maldonado, JL. Parouty - CNRS/SARI/DEVLOG - 2020
-
-We'll do the same thing again but with a more interesting dataset:  CelebFaces
-About this dataset : http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html
-
-## Episode 1 : Preparation of data - Batch mode
-
- - Save enhanced datasets in h5 file format
-
-%% Cell type:markdown id: tags:
-
-## Step 1 - Import and init
-### 1.2 - Import
-
-%% Cell type:code id: tags:
-
-``` python
-import numpy as np
-import matplotlib.pyplot as plt
-import pandas as pd
-from skimage import io, transform
-
-import os,time,sys,json,glob
-import csv
-import math, random
-
-from importlib import reload
-
-sys.path.append('..')
-import fidle.pwk as ooo
-
-ooo.init()
-```
-
-%% Cell type:markdown id: tags:
-
-### 1.2 - Directories and files :
-
-%% Cell type:code id: tags:
-
-``` python
-place, dataset_dir = ooo.good_place( { 'GRICAD' : f'{os.getenv("SCRATCH_DIR","")}/PROJECTS/pr-fidle/datasets/celeba',
-                                       'IDRIS'  : f'{os.getenv("WORK","")}/datasets/celeba'    } )
-
-dataset_csv  = f'{dataset_dir}/list_attr_celeba.csv'
-dataset_img  = f'{dataset_dir}/img_align_celeba'
-```
-
-%% Cell type:markdown id: tags:
-
-## Step 2 - Read filenames catalog
-
-%% Cell type:code id: tags:
-
-``` python
-dataset_desc = pd.read_csv(dataset_csv, header=0)
-dataset_desc = dataset_desc.reindex(np.random.permutation(dataset_desc.index))
-```
-
-%% Cell type:markdown id: tags:
-
-## Step 3 - Save as clusters of n images
-
-%% Cell type:markdown id: tags:
-
-### 4.2 - Cooking function
-
-%% Cell type:code id: tags:
-
-``` python
-def read_and_save( dataset_img, dataset_desc,
-                   cluster_size=1000, cluster_dir='./dataset_cluster', cluster_name='images',
-                   image_size=(128,128)):
-
-    def save_cluster(imgs,desc,cols,id):
-        file_img  = f'{cluster_dir}/{cluster_name}-{id:03d}.npy'
-        file_desc = f'{cluster_dir}/{cluster_name}-{id:03d}.csv'
-        np.save(file_img,  np.array(imgs))
-        df=pd.DataFrame(data=desc,columns=cols)
-        df.to_csv(file_desc, index=False)
-        return [],[],id+1
-
-    start_time = time.time()
-    cols = list(dataset_desc.columns)
-
-    # ---- Check if cluster files exist
-    #
-    if os.path.isfile(f'{cluster_dir}/images-000.npy'):
-        print('\n*** Oops. There are already clusters in the target folder!\n')
-        return 0,0
-
-    # ---- Create cluster_dir
-    #
-    os.makedirs(cluster_dir, mode=0o750, exist_ok=True)
-
-    # ---- Read and save clusters
-    #
-    imgs, desc, cluster_id = [],[],0
-    #
-    for i,row in dataset_desc.iterrows():
-        #
-        filename = f'{dataset_img}/{row.image_id}'
-        #
-        # ---- Read image, resize (and normalize)
-        #
-        img = io.imread(filename)
-        img = transform.resize(img, image_size)
-        #
-        # ---- Add image and description
-        #
-        imgs.append( img )
-        desc.append( row.values )
-        #
-        # ---- Progress bar
-        #
-        ooo.update_progress(f'Cluster {cluster_id:03d} :',len(imgs),cluster_size)
-        #
-        # ---- Save cluster if full
-        #
-        if len(imgs)==cluster_size:
-            imgs,desc,cluster_id=save_cluster(imgs,desc,cols, cluster_id)
-
-    # ---- Save uncomplete cluster
-    if len(imgs)>0 : imgs,desc,cluster_id=save_cluster(imgs,desc,cols,cluster_id)
-
-    duration=time.time()-start_time
-    return cluster_id,duration
-```
-
-%% Cell type:markdown id: tags:
-
-### 4.3 - Cluster building
-
-%% Cell type:code id: tags:
-
-``` python
-# ---- Cluster size
-
-cluster_size_train = 10000
-cluster_size_test  = 10000
-image_size         = (192,160)
-
-# ---- Clusters location
-
-train_dir  = f'{dataset_dir}/clusters-M.train'
-test_dir   = f'{dataset_dir}/clusters-M.test'
-
-# ---- x_train, x_test
-#
-n1,d1 = read_and_save(dataset_img, dataset_desc[:200000],
-                      cluster_size = cluster_size_train,
-                      cluster_dir  = train_dir,
-                      image_size   = image_size )
-
-n2,d2 = read_and_save(dataset_img, dataset_desc[200000:],
-                      cluster_size = cluster_size_test,
-                      cluster_dir  = test_dir,
-                      image_size   = image_size )
-
-print(f'\n\nDuration : {d1+d2:.2f} s or {ooo.hdelay(d1+d2)}')
-print(f'Train clusters : {train_dir}')
-print(f'Test  clusters : {test_dir}')
-```
-
-%% Cell type:markdown id: tags:
-
-----
-That's all folks !
-
-%% Cell type:code id: tags:
-
-``` python
-```

VAE/03.1-Batch.nbconvert.ipynb

-%% Cell type:markdown id: tags:
-
-Celeb Faces Dataset (CelebA)
-=================================================
----
-Introduction au Deep Learning  (IDLE) - S. Arias, E. Maldonado, JL. Parouty - CNRS/SARI/DEVLOG - 2020
-
-We'll do the same thing again but with a more interesting dataset:  CelebFaces
-About this dataset : http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html
-
-## Episode 1 : Preparation of data - Batch mode
-
- - Save enhanced datasets in h5 file format
-
-%% Cell type:markdown id: tags:
-
-## Step 1 - Import and init
-### 1.2 - Import
-
-%% Cell type:code id: tags:
-
-``` python
-import numpy as np
-import matplotlib.pyplot as plt
-import pandas as pd
-from skimage import io, transform
-
-import os,time,sys,json,glob
-import csv
-import math, random
-
-from importlib import reload
-
-sys.path.append('..')
-import fidle.pwk as ooo
-
-ooo.init()
-```
-
-%% Output
-
-
-    
-    FIDLE 2020 - Practical Work Module
-    Version              : 0.2.8
-    Run time             : Thursday 13 February 2020, 23:50:25
-    TensorFlow version   : 2.0.0
-    Keras version        : 2.2.4-tf
-
-%% Cell type:markdown id: tags:
-
-### 1.2 - Directories and files :
-
-%% Cell type:code id: tags:
-
-``` python
-place, dataset_dir = ooo.good_place( { 'GRICAD' : f'{os.getenv("SCRATCH_DIR","")}/PROJECTS/pr-fidle/datasets/celeba',
-                                       'IDRIS'  : f'{os.getenv("WORK","")}/datasets/celeba'    } )
-
-dataset_csv  = f'{dataset_dir}/list_attr_celeba.csv'
-dataset_img  = f'{dataset_dir}/img_align_celeba'
-```
-
-%% Output
-
-    Well, we should be at IDRIS !
-    We are going to use: /gpfswork/rech/mlh/uja62cb/datasets/celeba
-
-%% Cell type:markdown id: tags:
-
-## Step 2 - Read filenames catalog
-
-%% Cell type:code id: tags:
-
-``` python
-dataset_desc = pd.read_csv(dataset_csv, header=0)
-dataset_desc = dataset_desc.reindex(np.random.permutation(dataset_desc.index))
-```
-
-%% Cell type:markdown id: tags:
-
-## Step 3 - Save as clusters of n images
-
-%% Cell type:markdown id: tags:
-
-### 4.2 - Cooking function
-
-%% Cell type:code id: tags:
-
-``` python
-def read_and_save( dataset_img, dataset_desc,
-                   cluster_size=1000, cluster_dir='./dataset_cluster', cluster_name='images',
-                   image_size=(128,128)):
-
-    def save_cluster(imgs,desc,cols,id):
-        file_img  = f'{cluster_dir}/{cluster_name}-{id:03d}.npy'
-        file_desc = f'{cluster_dir}/{cluster_name}-{id:03d}.csv'
-        np.save(file_img,  np.array(imgs))
-        df=pd.DataFrame(data=desc,columns=cols)
-        df.to_csv(file_desc, index=False)
-        return [],[],id+1
-
-    start_time = time.time()
-    cols = list(dataset_desc.columns)
-
-    # ---- Check if cluster files exist
-    #
-    if os.path.isfile(f'{cluster_dir}/images-000.npy'):
-        print('\n*** Oops. There are already clusters in the target folder!\n')
-        return 0,0
-
-    # ---- Create cluster_dir
-    #
-    os.makedirs(cluster_dir, mode=0o750, exist_ok=True)
-
-    # ---- Read and save clusters
-    #
-    imgs, desc, cluster_id = [],[],0
-    #
-    for i,row in dataset_desc.iterrows():
-        #
-        filename = f'{dataset_img}/{row.image_id}'
-        #
-        # ---- Read image, resize (and normalize)
-        #
-        img = io.imread(filename)
-        img = transform.resize(img, image_size)
-        #
-        # ---- Add image and description
-        #
-        imgs.append( img )
-        desc.append( row.values )
-        #
-        # ---- Progress bar
-        #
-        ooo.update_progress(f'Cluster {cluster_id:03d} :',len(imgs),cluster_size)
-        #
-        # ---- Save cluster if full
-        #
-        if len(imgs)==cluster_size:
-            imgs,desc,cluster_id=save_cluster(imgs,desc,cols, cluster_id)
-
-    # ---- Save uncomplete cluster
-    if len(imgs)>0 : imgs,desc,cluster_id=save_cluster(imgs,desc,cols,cluster_id)
-
-    duration=time.time()-start_time
-    return cluster_id,duration
-```
-
-%% Cell type:markdown id: tags:
-
-### 4.3 - Cluster building
-
-%% Cell type:code id: tags:
-
-``` python
-# ---- Cluster size
-
-cluster_size_train = 10000
-cluster_size_test  = 10000
-image_size         = (192,160)
-
-# ---- Clusters location
-
-train_dir  = f'{dataset_dir}/clusters-M.train'
-test_dir   = f'{dataset_dir}/clusters-M.test'
-
-# ---- x_train, x_test
-#
-n1,d1 = read_and_save(dataset_img, dataset_desc[:200000],
-                      cluster_size = cluster_size_train,
-                      cluster_dir  = train_dir,
-                      image_size   = image_size )
-
-n2,d2 = read_and_save(dataset_img, dataset_desc[200000:],
-                      cluster_size = cluster_size_test,
-                      cluster_dir  = test_dir,
-                      image_size   = image_size )
-
-print(f'\n\nDuration : {d1+d2:.2f} s or {ooo.hdelay(d1+d2)}')
-print(f'Train clusters : {train_dir}')
-print(f'Test  clusters : {test_dir}')
-```
-
-%% Output
-
-    
-    *** Oops. There are already clusters in the target folder!
-    
-    
-    *** Oops. There are already clusters in the target folder!
-    
-    
-    
-    Duration : 0.00 s or 0:00:00
-    Train clusters : /gpfswork/rech/mlh/uja62cb/datasets/celeba/clusters-M.train
-    Test  clusters : /gpfswork/rech/mlh/uja62cb/datasets/celeba/clusters-M.test
-
-%% Cell type:markdown id: tags:
-
-----
-That's all folks !
-
-%% Cell type:code id: tags:
-
-``` python
-```

VAE/05-VAE-with-CelebA.ipynb

-%% Cell type:markdown id: tags:
-
-Variational AutoEncoder (VAE) with CelebA
-=========================================
----
-Formation Introduction au Deep Learning  (FIDLE) - S. Arias, E. Maldonado, JL. Parouty - CNRS/SARI/DEVLOG - 2020
-
-## Episode 1 - Train a model
-
- - Defining a VAE model
- - Build the model
- - Train it
- - Follow the learning process with Tensorboard
-
-%% Cell type:markdown id: tags:
-
-## Step 1 - Setup environment
-### 1.1 - Python stuff
-
-%% Cell type:code id: tags:
-
-``` python
-import tensorflow as tf
-import numpy as np
-import os,sys
-from importlib import reload
-
-import modules.vae
-import modules.data_generator
-
-reload(modules.data_generator)
-reload(modules.vae)
-
-from modules.vae  import VariationalAutoencoder
-from modules.data_generator import DataGenerator
-
-sys.path.append('..')
-import fidle.pwk as ooo
-reload(ooo)
-
-ooo.init()
-
-VariationalAutoencoder.about()
-DataGenerator.about()
-```
-
-%% Output
-
-
-    
-    FIDLE 2020 - Practical Work Module
-    Version              : 0.2.8
-    Run time             : Thursday 13 February 2020, 21:38:51
-    TensorFlow version   : 2.0.0
-    Keras version        : 2.2.4-tf
-    
-    FIDLE 2020 - Variational AutoEncoder (VAE)
-    TensorFlow version   : 2.0.0
-    VAE version          : 1.27
-    
-    FIDLE 2020 - DataGenerator
-    Version              : 0.4.1
-
-%% Cell type:markdown id: tags:
-
-### 1.2 - The good place
-
-%% Cell type:code id: tags:
-
-``` python
-place, dataset_dir = ooo.good_place( { 'GRICAD' : f'{os.getenv("SCRATCH_DIR","")}/PROJECTS/pr-fidle/datasets/celeba',
-                                       'IDRIS'  : f'{os.getenv("WORK","")}/datasets/celeba'    } )
-
-# ---- train/test datasets
-
-train_dir    = f'{dataset_dir}/clusters.train'
-test_dir     = f'{dataset_dir}/clusters.test'
-```
-
-%% Output
-
-    Well, we should be at IDRIS !
-    We are going to use: /gpfswork/rech/mlh/uja62cb/datasets/celeba
-
-%% Cell type:markdown id: tags:
-
-## Step 2 - DataGenerator and validation data
-Ok, everything's perfect, now let's instantiate our generator for the entire dataset.
-
-%% Cell type:code id: tags:
-
-``` python
-data_gen = DataGenerator(train_dir, 32, k_size=1)
-x_test   = np.load(f'{test_dir}/images-000.npy')
-
-print(f'Data generator : {len(data_gen)} batchs of {data_gen.batch_size} images, or {data_gen.dataset_size} images')
-print(f'x_test : {len(x_test)} images')
-```
-
-%% Output
-
-    Data generator : 6250 batchs of 32 images, or 200000 images
-    x_test : 2599 images
-
-%% Cell type:markdown id: tags:
-
-## Step 3 - Get VAE model
-
-%% Cell type:code id: tags:
-
-``` python
-tag = 'CelebA.test'
-
-input_shape = (128, 128, 3)
-z_dim       = 200
-verbose     = 1
-
-encoder= [ {'type':'Conv2D',          'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-         ]
-
-decoder= [ {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2DTranspose', 'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2DTranspose', 'filters':3,  'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'sigmoid'}
-         ]
-
-vae = modules.vae.VariationalAutoencoder(input_shape    = input_shape,
-                                         encoder_layers = encoder,
-                                         decoder_layers = decoder,
-                                         z_dim          = z_dim,
-                                         verbose        = verbose,
-                                         run_tag        = tag)
-vae.save(model=None)
-```
-
-%% Output
-
-    Model initialized.
-    Outputs will be in  : ./run/CelebA.test
-    
-     ---------- Encoder --------------------------------------------------
-    
-    Model: "model_1"
-    __________________________________________________________________________________________________
-    Layer (type)                    Output Shape         Param #     Connected to
-    ==================================================================================================
-    encoder_input (InputLayer)      [(None, 128, 128, 3) 0
-    __________________________________________________________________________________________________
-    conv2d (Conv2D)                 (None, 64, 64, 32)   896         encoder_input[0][0]
-    __________________________________________________________________________________________________
-    dropout (Dropout)               (None, 64, 64, 32)   0           conv2d[0][0]
-    __________________________________________________________________________________________________
-    conv2d_1 (Conv2D)               (None, 32, 32, 64)   18496       dropout[0][0]
-    __________________________________________________________________________________________________
-    dropout_1 (Dropout)             (None, 32, 32, 64)   0           conv2d_1[0][0]
-    __________________________________________________________________________________________________
-    conv2d_2 (Conv2D)               (None, 16, 16, 64)   36928       dropout_1[0][0]
-    __________________________________________________________________________________________________
-    dropout_2 (Dropout)             (None, 16, 16, 64)   0           conv2d_2[0][0]
-    __________________________________________________________________________________________________
-    conv2d_3 (Conv2D)               (None, 8, 8, 64)     36928       dropout_2[0][0]
-    __________________________________________________________________________________________________
-    dropout_3 (Dropout)             (None, 8, 8, 64)     0           conv2d_3[0][0]
-    __________________________________________________________________________________________________
-    flatten (Flatten)               (None, 4096)         0           dropout_3[0][0]
-    __________________________________________________________________________________________________
-    mu (Dense)                      (None, 200)          819400      flatten[0][0]
-    __________________________________________________________________________________________________
-    log_var (Dense)                 (None, 200)          819400      flatten[0][0]
-    __________________________________________________________________________________________________
-    encoder_output (Lambda)         (None, 200)          0           mu[0][0]
-                                                                     log_var[0][0]
-    ==================================================================================================
-    Total params: 1,732,048
-    Trainable params: 1,732,048
-    Non-trainable params: 0
-    __________________________________________________________________________________________________
-    
-     ---------- Encoder --------------------------------------------------
-    
-    Model: "model_2"
-    _________________________________________________________________
-    Layer (type)                 Output Shape              Param #
-    =================================================================
-    decoder_input (InputLayer)   [(None, 200)]             0
-    _________________________________________________________________
-    dense (Dense)                (None, 4096)              823296
-    _________________________________________________________________
-    reshape (Reshape)            (None, 8, 8, 64)          0
-    _________________________________________________________________
-    conv2d_transpose (Conv2DTran (None, 16, 16, 64)        36928
-    _________________________________________________________________
-    dropout_4 (Dropout)          (None, 16, 16, 64)        0
-    _________________________________________________________________
-    conv2d_transpose_1 (Conv2DTr (None, 32, 32, 64)        36928
-    _________________________________________________________________
-    dropout_5 (Dropout)          (None, 32, 32, 64)        0
-    _________________________________________________________________
-    conv2d_transpose_2 (Conv2DTr (None, 64, 64, 32)        18464
-    _________________________________________________________________
-    dropout_6 (Dropout)          (None, 64, 64, 32)        0
-    _________________________________________________________________
-    conv2d_transpose_3 (Conv2DTr (None, 128, 128, 3)       867
-    =================================================================
-    Total params: 916,483
-    Trainable params: 916,483
-    Non-trainable params: 0
-    _________________________________________________________________
-    Failed to import pydot. You must install pydot and graphviz for `pydotprint` to work.
-    Failed to import pydot. You must install pydot and graphviz for `pydotprint` to work.
-    Failed to import pydot. You must install pydot and graphviz for `pydotprint` to work.
-    Config saved in     : ./run/CelebA.test/models/vae_config.json
-
-%% Cell type:markdown id: tags:
-
-## Step 4 - Compile it
-
-%% Cell type:code id: tags:
-
-``` python
-optimizer = tf.keras.optimizers.Adam(1e-4)
-# optimizer     = 'adam'
-r_loss_factor = 10000
-
-vae.compile(optimizer, r_loss_factor)
-```
-
-%% Output
-
-    Compiled.
-
-%% Cell type:markdown id: tags:
-
-## Step 5 - Train
-For 10 epochs, adam optimizer :
-- Run time at IDRIS : 1299.77 sec. - 0:21:39
-- Run time at GRICAD : 2092.77 sec. - 0:34:52
-
-%% Cell type:code id: tags:
-
-``` python
-epochs            = 10
-initial_epoch     = 0
-```
-
-%% Cell type:code id: tags:
-
-``` python
-vae.train(data_generator    = data_gen,
-          x_test            = x_test,
-          epochs            = epochs,
-          initial_epoch     = initial_epoch
-         )
-```
-
-%% Output
-
-    Epoch 1/10
-    6250/6250 [==============================] - 175s 28ms/step - loss: 349.6490 - vae_r_loss: 301.2042 - vae_kl_loss: 48.4450 - val_loss: 236.8924 - val_vae_r_loss: 189.8669 - val_vae_kl_loss: 47.1441
-    Epoch 2/10
-    6250/6250 [==============================] - 128s 20ms/step - loss: 241.0380 - vae_r_loss: 187.2187 - vae_kl_loss: 53.8191 - val_loss: 215.4507 - val_vae_r_loss: 162.7656 - val_vae_kl_loss: 52.8317
-    Epoch 3/10
-    3141/6250 [==============>...............] - ETA: 1:02 - loss: 230.1868 - vae_r_loss: 175.1927 - vae_kl_loss: 54.9941
-
-%% Cell type:code id: tags:
-
-``` python
-```
-
-%% Cell type:code id: tags:
-
-``` python
-```

VAE/05.2-Variant.ipynb

-%% Cell type:markdown id: tags:
-
-Variational AutoEncoder (VAE) with CelebA
-=========================================
----
-Formation Introduction au Deep Learning  (FIDLE) - S. Arias, E. Maldonado, JL. Parouty - CNRS/SARI/DEVLOG - 2020
-
-## Episode 1 - Train a model
-
- - Defining a VAE model
- - Build the model
- - Train it
- - Follow the learning process with Tensorboard
-
-%% Cell type:markdown id: tags:
-
-## Step 1 - Setup environment
-### 1.1 - Python stuff
-
-%% Cell type:code id: tags:
-
-``` python
-import tensorflow as tf
-import numpy as np
-import os,sys
-from importlib import reload
-
-import modules.vae
-import modules.data_generator
-
-reload(modules.data_generator)
-reload(modules.vae)
-
-from modules.vae  import VariationalAutoencoder
-from modules.data_generator import DataGenerator
-
-sys.path.append('..')
-import fidle.pwk as ooo
-reload(ooo)
-
-ooo.init()
-
-VariationalAutoencoder.about()
-DataGenerator.about()
-```
-
-%% Cell type:markdown id: tags:
-
-### 1.2 - The good place
-
-%% Cell type:code id: tags:
-
-``` python
-place, dataset_dir = ooo.good_place( { 'GRICAD' : f'{os.getenv("SCRATCH_DIR","")}/PROJECTS/pr-fidle/datasets/celeba',
-                                       'IDRIS'  : f'{os.getenv("WORK","")}/datasets/celeba'    } )
-
-# ---- train/test datasets
-
-train_dir    = f'{dataset_dir}/clusters-M.train'
-test_dir     = f'{dataset_dir}/clusters-M.test'
-```
-
-%% Cell type:markdown id: tags:
-
-## Step 2 - DataGenerator and validation data
-Ok, everything's perfect, now let's instantiate our generator for the entire dataset.
-
-%% Cell type:code id: tags:
-
-``` python
-data_gen = DataGenerator(train_dir, 32, k_size=1)
-x_test   = np.load(f'{test_dir}/images-000.npy')
-
-print(f'Data generator : {len(data_gen)} batchs of {data_gen.batch_size} images, or {data_gen.dataset_size} images')
-print(f'x_test : {len(x_test)} images')
-```
-
-%% Cell type:markdown id: tags:
-
-## Step 3 - Get VAE model
-
-%% Cell type:code id: tags:
-
-``` python
-tag = f'CelebA.052-M.{os.getenv("SLURM_JOB_ID","unknown")}'
-
-input_shape = (192, 160, 3)
-z_dim       = 200
-verbose     = 0
-
-encoder= [ {'type':'Conv2D',          'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-         ]
-
-decoder= [ {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2DTranspose', 'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2DTranspose', 'filters':3,  'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'sigmoid'}
-         ]
-
-vae = modules.vae.VariationalAutoencoder(input_shape    = input_shape,
-                                         encoder_layers = encoder,
-                                         decoder_layers = decoder,
-                                         z_dim          = z_dim,
-                                         verbose        = verbose,
-                                         run_tag        = tag)
-vae.save(model=None)
-```
-
-%% Cell type:markdown id: tags:
-
-## Step 4 - Compile it
-
-%% Cell type:code id: tags:
-
-``` python
-optimizer     = tf.keras.optimizers.Adam(1e-4)
-r_loss_factor = 10000
-
-vae.compile(optimizer, r_loss_factor)
-```
-
-%% Cell type:markdown id: tags:
-
-## Step 5 - Train
-For 10 epochs, adam optimizer :
-- Run time at IDRIS : 1299.77 sec. - 0:21:39
-- Run time at GRICAD : 2092.77 sec. - 0:34:52
-
-%% Cell type:code id: tags:
-
-``` python
-epochs            = 20
-initial_epoch     = 0
-```
-
-%% Cell type:code id: tags:
-
-``` python
-vae.train(data_generator    = data_gen,
-          x_test            = x_test,
-          epochs            = epochs,
-          initial_epoch     = initial_epoch
-         )
-```
-
-%% Cell type:markdown id: tags:
-
-----
-That's all folks !

VAE/05.2-Variant.nbconvert.ipynb

-%% Cell type:markdown id: tags:
-
-Variational AutoEncoder (VAE) with CelebA
-=========================================
----
-Formation Introduction au Deep Learning  (FIDLE) - S. Arias, E. Maldonado, JL. Parouty - CNRS/SARI/DEVLOG - 2020
-
-## Episode 1 - Train a model
-
- - Defining a VAE model
- - Build the model
- - Train it
- - Follow the learning process with Tensorboard
-
-%% Cell type:markdown id: tags:
-
-## Step 1 - Setup environment
-### 1.1 - Python stuff
-
-%% Cell type:code id: tags:
-
-``` python
-import tensorflow as tf
-import numpy as np
-import os,sys
-from importlib import reload
-
-import modules.vae
-import modules.data_generator
-
-reload(modules.data_generator)
-reload(modules.vae)
-
-from modules.vae  import VariationalAutoencoder
-from modules.data_generator import DataGenerator
-
-sys.path.append('..')
-import fidle.pwk as ooo
-reload(ooo)
-
-ooo.init()
-
-VariationalAutoencoder.about()
-DataGenerator.about()
-```
-
-%% Output
-
-
-    
-    FIDLE 2020 - Practical Work Module
-    Version              : 0.2.8
-    Run time             : Friday 14 February 2020, 00:07:28
-    TensorFlow version   : 2.0.0
-    Keras version        : 2.2.4-tf
-    
-    FIDLE 2020 - Variational AutoEncoder (VAE)
-    TensorFlow version   : 2.0.0
-    VAE version          : 1.28
-    
-    FIDLE 2020 - DataGenerator
-    Version              : 0.4.1
-
-%% Cell type:markdown id: tags:
-
-### 1.2 - The good place
-
-%% Cell type:code id: tags:
-
-``` python
-place, dataset_dir = ooo.good_place( { 'GRICAD' : f'{os.getenv("SCRATCH_DIR","")}/PROJECTS/pr-fidle/datasets/celeba',
-                                       'IDRIS'  : f'{os.getenv("WORK","")}/datasets/celeba'    } )
-
-# ---- train/test datasets
-
-train_dir    = f'{dataset_dir}/clusters-M.train'
-test_dir     = f'{dataset_dir}/clusters-M.test'
-```
-
-%% Output
-
-    Well, we should be at IDRIS !
-    We are going to use: /gpfswork/rech/mlh/uja62cb/datasets/celeba
-
-%% Cell type:markdown id: tags:
-
-## Step 2 - DataGenerator and validation data
-Ok, everything's perfect, now let's instantiate our generator for the entire dataset.
-
-%% Cell type:code id: tags:
-
-``` python
-data_gen = DataGenerator(train_dir, 32, k_size=1)
-x_test   = np.load(f'{test_dir}/images-000.npy')
-
-print(f'Data generator : {len(data_gen)} batchs of {data_gen.batch_size} images, or {data_gen.dataset_size} images')
-print(f'x_test : {len(x_test)} images')
-```
-
-%% Output
-
-    Data generator : 6250 batchs of 32 images, or 200000 images
-    x_test : 2599 images
-
-%% Cell type:markdown id: tags:
-
-## Step 3 - Get VAE model
-
-%% Cell type:code id: tags:
-
-``` python
-tag = f'CelebA.052-M.{os.getenv("SLURM_JOB_ID","unknown")}'
-
-input_shape = (192, 160, 3)
-z_dim       = 200
-verbose     = 0
-
-encoder= [ {'type':'Conv2D',          'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2D',          'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-         ]
-
-decoder= [ {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2DTranspose', 'filters':64, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2DTranspose', 'filters':32, 'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'relu'},
-           {'type':'Dropout',         'rate':0.25},
-           {'type':'Conv2DTranspose', 'filters':3,  'kernel_size':(3,3), 'strides':2, 'padding':'same', 'activation':'sigmoid'}
-         ]
-
-vae = modules.vae.VariationalAutoencoder(input_shape    = input_shape,
-                                         encoder_layers = encoder,
-                                         decoder_layers = decoder,
-                                         z_dim          = z_dim,
-                                         verbose        = verbose,
-                                         run_tag        = tag)
-vae.save(model=None)
-```
-
-%% Output
-
-    Model initialized.
-    Outputs will be in  : ./run/CelebA.052-M.810156
-    Config saved in     : ./run/CelebA.052-M.810156/models/vae_config.json
-
-%% Cell type:markdown id: tags:
-
-## Step 4 - Compile it
-
-%% Cell type:code id: tags:
-
-``` python
-optimizer     = tf.keras.optimizers.Adam(1e-4)
-r_loss_factor = 10000
-
-vae.compile(optimizer, r_loss_factor)
-```
-
-%% Output
-
-    Compiled.
-
-%% Cell type:markdown id: tags:
-
-## Step 5 - Train
-For 10 epochs, adam optimizer :
-- Run time at IDRIS : 1299.77 sec. - 0:21:39
-- Run time at GRICAD : 2092.77 sec. - 0:34:52
-
-%% Cell type:code id: tags:
-
-``` python
-epochs            = 20
-initial_epoch     = 0
-```
-
-%% Cell type:code id: tags:
-
-``` python
-vae.train(data_generator    = data_gen,
-          x_test            = x_test,
-          epochs            = epochs,
-          initial_epoch     = initial_epoch
-         )
-```
-
-%% Output
-
-    Epoch 1/20
-    6250/6250 [==============================] - 342s 55ms/step - loss: 332.4792 - vae_r_loss: 282.9655 - vae_kl_loss: 49.5134 - val_loss: 235.4288 - val_vae_r_loss: 187.4334 - val_vae_kl_loss: 48.1192
-    Epoch 2/20
-    6250/6250 [==============================] - 337s 54ms/step - loss: 224.0962 - vae_r_loss: 166.4602 - vae_kl_loss: 57.6360 - val_loss: 210.9632 - val_vae_r_loss: 155.0925 - val_vae_kl_loss: 56.0114
-    Epoch 4/20
-    6250/6250 [==============================] - 333s 53ms/step - loss: 214.5463 - vae_r_loss: 155.7666 - vae_kl_loss: 58.7794 - val_loss: 203.8241 - val_vae_r_loss: 147.3248 - val_vae_kl_loss: 56.5778
-    Epoch 7/20
-    6250/6250 [==============================] - 327s 52ms/step - loss: 211.1459 - vae_r_loss: 152.4026 - vae_kl_loss: 58.7437 - val_loss: 201.1862 - val_vae_r_loss: 145.6906 - val_vae_kl_loss: 55.6326
-    Epoch 10/20
-    6250/6250 [==============================] - 335s 54ms/step - loss: 209.7628 - vae_r_loss: 151.0874 - vae_kl_loss: 58.6756 - val_loss: 202.3954 - val_vae_r_loss: 147.0956 - val_vae_kl_loss: 55.4047
-    Epoch 12/20
-    6250/6250 [==============================] - 333s 53ms/step - loss: 207.9830 - vae_r_loss: 149.3870 - vae_kl_loss: 58.5959 - val_loss: 198.5626 - val_vae_r_loss: 142.5848 - val_vae_kl_loss: 56.0871
-    Epoch 16/20
-    6250/6250 [==============================] - 330s 53ms/step - loss: 206.6382 - vae_r_loss: 148.0522 - vae_kl_loss: 58.5863 - val_loss: 197.5800 - val_vae_r_loss: 142.6799 - val_vae_kl_loss: 54.9832
-    
-    Train duration : 6638.61 sec. - 1:50:38
-
-%% Cell type:markdown id: tags:
-
-----
-That's all folks !