.gitignore

 .ipynb_checkpoints
+.vscode
 */.ipynb_checkpoints/*
 __pycache__
 */__pycache__/*
+campains/
+*==*==.ipynb
+!*==done==.ipynb
+stderr.txt
+stdout.txt
+debug.log
 run/
+figs/
+data/
 GTSRB/data
+GTSRB/05-full_convolutions.py
 IMDB/data
 MNIST/data
+SYNOP/data
 VAE/data
-BHPD/data/*
-!BHPD/data/BostonHousing.csv
+BHPD/data
+AE/data

AE.Keras3/01-Prepare-MNIST-dataset.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3AE1] - Prepare a noisy MNIST dataset
+<!-- DESC --> Episode 1: Preparation of a noisy MNIST dataset
+
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Prepare a MNIST noisy dataset, usable with our denoiser autoencoder (duration : <50s)
+
+## What we're going to do :
+
+ - Load original MNIST dataset
+ - Adding noise, a lot !
+ - Save it :-)
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init and set parameters
+### 1.1 - Init python
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+import sys
+
+from skimage import io
+from skimage.util import random_noise
+
+import modules.MNIST
+from modules.MNIST     import MNIST
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3AE1')
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Parameters
+`prepared_dataset` : Filename of the future prepared dataset (example : ./data/mnist-noisy.h5)\
+`scale` : Dataset scale. 1 mean 100% of the dataset - set 0.1 for tests\
+`progress_verbosity`: Verbosity of progress bar: 0=silent, 1=progress bar, 2=One line
+
+%% Cell type:code id: tags:
+
+``` python
+prepared_dataset   = './data/mnist-noisy.h5'
+scale              = 1
+progress_verbosity = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('prepared_dataset', 'scale', 'progress_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Get original dataset
+We load :
+`clean_data` : Original and clean images - This is what we will want to ontain at the **output** of the AE
+`class_data` : Image classes - Useless, because the training will be unsupervised
+We'll build :
+`noisy_data` : Noisy images - These are the images that we will give as **input** to our AE
+
+%% Cell type:code id: tags:
+
+``` python
+clean_data, class_data = MNIST.get_origine(scale=scale)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Add noise
+We add noise to the original images (clean_data) to obtain noisy images (noisy_data)
+Need 30-40 seconds
+
+%% Cell type:code id: tags:
+
+``` python
+def noise_it(data):
+    new_data = np.copy(data)
+    for i,image in enumerate(new_data):
+        fidle.utils.update_progress('Add noise : ',i+1,len(data),verbosity=progress_verbosity)
+        image=random_noise(image, mode='gaussian', mean=0, var=0.3)
+        image=random_noise(image, mode='s&p',      amount=0.2, salt_vs_pepper=0.5)
+        image=random_noise(image, mode='poisson')
+        image=random_noise(image, mode='speckle',  mean=0, var=0.1)
+        new_data[i]=image
+    print('Done.')
+    return new_data
+
+# ---- Add noise to input data : x_data
+#
+noisy_data = noise_it(clean_data)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Have a look
+
+%% Cell type:code id: tags:
+
+``` python
+print('Clean dataset (clean_data) : ',clean_data.shape)
+print('Noisy dataset (noisy_data) : ',noisy_data.shape)
+
+fidle.utils.subtitle("Noisy images we'll have in input (or x)")
+fidle.scrawler.images(noisy_data[:5], None, indices='all', columns=5, x_size=3,y_size=3, interpolation=None, save_as='01-noisy')
+fidle.utils.subtitle('Clean images we want to obtain (or y)')
+fidle.scrawler.images(clean_data[:5], None, indices='all', columns=5, x_size=3,y_size=3, interpolation=None, save_as='02-original')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Shuffle dataset
+
+%% Cell type:code id: tags:
+
+``` python
+p = np.random.permutation(len(clean_data))
+clean_data, noisy_data, class_data = clean_data[p], noisy_data[p], class_data[p]
+print('Shuffled.')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Save our prepared dataset
+
+%% Cell type:code id: tags:
+
+``` python
+MNIST.save_prepared_dataset( clean_data, noisy_data, class_data, filename=prepared_dataset )
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

AE.Keras3/02-AE-with-MNIST.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3AE2] - Building and training an AE denoiser model
+<!-- DESC --> Episode 1 : Construction of a denoising autoencoder and training of it with a noisy MNIST dataset.
+
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Understanding and implementing a denoizing **autoencoder** neurals network (AE)
+ - First overview or example of Keras procedural syntax
+
+The calculation needs being important, it is preferable to use a very simple dataset such as MNIST.
+The use of a GPU is often indispensable.
+
+## What we're going to do :
+
+ - Defining an AE model
+ - Build the model
+ - Train it
+ - Follow the learning process with Tensorboard
+
+## Data Terminology :
+- `clean_train`, `clean_test` for noiseless images
+- `noisy_train`, `noisy_test` for noisy images
+- `denoised_test` for denoised images at the output of the model
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init python stuff
+### 1.1 - Init
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+from skimage import io
+import random
+
+from keras import layers
+from keras.callbacks import ModelCheckpoint, TensorBoard
+
+import os
+from importlib import reload
+
+from modules.MNIST          import MNIST
+from modules.ImagesCallback import ImagesCallback
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3AE2')
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Parameters
+`prepared_dataset` : Filename of the prepared dataset (Need 400 Mo, but can be in ./data)
+`dataset_seed` : Random seed for shuffling dataset
+`scale` : % of the dataset to use (1. for 100%)
+`latent_dim` : Dimension of the latent space
+`train_prop` : Percentage for train (the rest being for the test)
+`batch_size` : Batch size
+`epochs` : Nb of epochs for training\
+`fit_verbosity` is the verbosity during training : 0 = silent, 1 = progress bar, 2 = one line per epoch
+
+Note : scale=.2, epoch=20 => 3'30s on a laptop
+
+%% Cell type:code id: tags:
+
+``` python
+prepared_dataset = './data/mnist-noisy.h5'
+dataset_seed     = 123
+
+scale            = .1
+
+latent_dim       = 10
+
+train_prop       = .8
+batch_size       = 128
+epochs           = 20
+fit_verbosity    = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('prepared_dataset', 'dataset_seed', 'scale', 'latent_dim')
+fidle.override('train_prop', 'batch_size', 'epochs', 'fit_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve dataset
+With our MNIST class, in one call, we can reload, rescale, shuffle and split our previously saved dataset :-)
+
+%% Cell type:code id: tags:
+
+``` python
+clean_train,clean_test, noisy_train,noisy_test, _,_ = MNIST.reload_prepared_dataset(scale      = scale,
+                                                                                    train_prop = train_prop,
+                                                                                    seed       = dataset_seed,
+                                                                                    shuffle    = True,
+                                                                                    filename=prepared_dataset )
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Build models
+
+%% Cell type:markdown id: tags:
+
+#### Encoder
+
+%% Cell type:code id: tags:
+
+``` python
+inputs    = keras.Input(shape=(28, 28, 1))
+x         = layers.Conv2D(32, 3, activation="relu", strides=2, padding="same")(inputs)
+x         = layers.Conv2D(64, 3, activation="relu", strides=2, padding="same")(x)
+x         = layers.Flatten()(x)
+x         = layers.Dense(16, activation="relu")(x)
+z         = layers.Dense(latent_dim)(x)
+
+encoder = keras.Model(inputs, z, name="encoder")
+# encoder.summary()
+```
+
+%% Cell type:markdown id: tags:
+
+#### Decoder
+
+%% Cell type:code id: tags:
+
+``` python
+inputs  = keras.Input(shape=(latent_dim,))
+x       = layers.Dense(7 * 7 * 64, activation="relu")(inputs)
+x       = layers.Reshape((7, 7, 64))(x)
+x       = layers.Conv2DTranspose(64, 3, activation="relu", strides=2, padding="same")(x)
+x       = layers.Conv2DTranspose(32, 3, activation="relu", strides=2, padding="same")(x)
+outputs = layers.Conv2DTranspose(1, 3, activation="sigmoid", padding="same")(x)
+
+decoder = keras.Model(inputs, outputs, name="decoder")
+# decoder.summary()
+```
+
+%% Cell type:markdown id: tags:
+
+#### AE
+
+%% Cell type:code id: tags:
+
+``` python
+inputs    = keras.Input(shape=(28, 28, 1))
+
+latents   = encoder(inputs)
+outputs   = decoder(latents)
+
+ae = keras.Model(inputs,outputs, name="ae")
+
+ae.compile(optimizer=keras.optimizers.Adam(), loss='binary_crossentropy')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Train
+20' on a CPU
+1'12 on a GPU (V100, IDRIS)
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Callback : Images
+#
+fidle.utils.mkdir( run_dir + '/images')
+filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'
+callback_images = ImagesCallback(filename, x=clean_test[:5], encoder=encoder,decoder=decoder)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+chrono = fidle.Chrono()
+chrono.start()
+
+history = ae.fit(noisy_train, clean_train,
+                 batch_size      = batch_size,
+                 epochs          = epochs,
+                 verbose         = fit_verbosity,
+                 validation_data = (noisy_test, clean_test),
+                 callbacks       = [ callback_images ] )
+
+chrono.show()
+```
+
+%% Cell type:markdown id: tags:
+
+Save model
+
+%% Cell type:code id: tags:
+
+``` python
+os.makedirs(f'{run_dir}/models', exist_ok=True)
+
+encoder.save(f'{run_dir}/models/encoder.keras')
+decoder.save(f'{run_dir}/models/decoder.keras')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - History
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history(history,  plot={'loss':['loss','val_loss']}, save_as='01-history')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Denoising progress
+
+%% Cell type:code id: tags:
+
+``` python
+imgs=[]
+for epoch in range(0,epochs,2):
+    for i in range(5):
+        filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'.format(epoch=epoch, i=i)
+        img      = io.imread(filename)
+        imgs.append(img)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='02-original-real')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='03-original-noisy')
+
+fidle.utils.subtitle('Evolution during the training period (denoised_test) :')
+fidle.scrawler.images(imgs, None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, y_padding=0.1, save_as='04-learning')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Evaluation
+**Note :** We will use the following data:\
+`clean_train`, `clean_test` for noiseless images \
+`noisy_train`, `noisy_test` for noisy images\
+`denoised_test` for denoised images at the output of the model
+
+### 7.1 - Reload model
+
+%% Cell type:code id: tags:
+
+``` python
+encoder = keras.models.load_model(f'{run_dir}/models/encoder.keras')
+decoder = keras.models.load_model(f'{run_dir}/models/decoder.keras')
+
+inputs    = keras.Input(shape=(28, 28, 1))
+
+latents   = encoder(inputs)
+outputs   = decoder(latents)
+
+ae_reloaded = keras.Model(inputs,outputs, name="ae")
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.2 - Let's make a prediction
+
+%% Cell type:code id: tags:
+
+``` python
+denoised_test = ae_reloaded.predict(noisy_test, verbose=0)
+
+print('Denoised images   (denoised_test) shape : ',denoised_test.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.3 - Denoised images
+
+%% Cell type:code id: tags:
+
+``` python
+i=random.randint(0,len(denoised_test)-8)
+j=i+8
+
+fidle.utils.subtitle('Noisy test images (input):')
+fidle.scrawler.images(noisy_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='05-test-noisy')
+
+fidle.utils.subtitle('Denoised images (output):')
+fidle.scrawler.images(denoised_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='06-test-predict')
+
+fidle.utils.subtitle('Real test images :')
+fidle.scrawler.images(clean_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='07-test-real')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

AE.Keras3/03-AE-with-MNIST-post.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3AE3] - Playing with our denoiser model
+<!-- DESC --> Episode 2 : Using the previously trained autoencoder to denoise data
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Retrieve and use our denoiser model
+
+
+## What we're going to do :
+
+ - Reload our dataset and saved best model
+ - Encode/decode some test images (neved used, never seen by the model)
+
+## Data Terminology :
+- `clean_train`, `clean_test` for noiseless images
+- `noisy_train`, `noisy_test` for noisy images
+- `denoised_test` for denoised images at the output of the model
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init python stuff
+### 1.1 - Init
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+import matplotlib.pyplot as plt
+import random
+
+from modules.MNIST import MNIST
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3AE3')
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Parameters
+These **parameters must be identical** to those used during the training in order to have the **same dataset**.\
+`prepared_dataset` : Filename of the prepared dataset (Need 400 Mo, but can be in ./data)
+`dataset_seed` : Random seed for shuffling dataset
+`scale` : % of the dataset to use (1. for 100%)
+`train_prop` : Percentage for train (the rest being for the test)
+
+%% Cell type:code id: tags:
+
+``` python
+prepared_dataset = './data/mnist-noisy.h5'
+saved_models     = './run/K3AE2/models'
+dataset_seed     = 123
+scale            = 1
+train_prop       = .8
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('prepared_dataset', 'dataset_seed', 'scale', 'train_prop')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve dataset
+With our MNIST class, in one call, we can reload, rescale, shuffle and split our previously saved dataset :-)
+**Important :** Make sure that the **digest is identical** to the one used during the training !\
+See : [AE2 / Step 2 - Retrieve dataset](./02-AE-with-MNIST.ipynb#Step-2---Retrieve-dataset)
+
+%% Cell type:code id: tags:
+
+``` python
+clean_train,clean_test, noisy_train,noisy_test, _,_ = MNIST.reload_prepared_dataset(scale      = scale,
+                                                                                    train_prop = train_prop,
+                                                                                    seed       = dataset_seed,
+                                                                                    shuffle    = True,
+                                                                                    filename=prepared_dataset )
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Evaluation
+**Note :** We will use the following data:\
+`clean_train`, `clean_test` for noiseless images \
+`noisy_train`, `noisy_test` for noisy images\
+`denoised_test` for denoised images at the output of the model
+
+### 3.1 - Reload our best model
+
+%% Cell type:code id: tags:
+
+``` python
+# model = keras.models.load_model(f'{saved_models}/model.keras')
+
+encoder = keras.models.load_model(f'{saved_models}/encoder.keras')
+decoder = keras.models.load_model(f'{saved_models}/decoder.keras')
+
+inputs    = keras.Input(shape=(28, 28, 1))
+
+latents   = encoder(inputs)
+outputs   = decoder(latents)
+
+model = keras.Model(inputs,outputs, name="ae")
+```
+
+%% Cell type:markdown id: tags:
+
+### 3.2 - Let's make a prediction
+
+%% Cell type:code id: tags:
+
+``` python
+from tabnanny import verbose
+
+
+denoised_test = model.predict(noisy_test,verbose=0)
+
+print('Denoised images   (denoised_test) shape : ',denoised_test.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 3.3 - Denoised images
+
+%% Cell type:code id: tags:
+
+``` python
+i=random.randint(0,len(denoised_test)-8)
+j=i+8
+
+fidle.utils.subtitle('Noisy test images (input):')
+fidle.scrawler.images(noisy_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='05-test-noisy')
+
+fidle.utils.subtitle('Denoised images (output):')
+fidle.scrawler.images(denoised_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='06-test-predict')
+
+fidle.utils.subtitle('Real test images :')
+fidle.scrawler.images(clean_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='07-test-real')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Looking at the latent space
+### 4.1 - Getting clean data and class
+
+%% Cell type:code id: tags:
+
+``` python
+clean_data,_, _,_, class_data,_ = MNIST.reload_prepared_dataset(scale      = 1,
+                                                                train_prop = 1,
+                                                                seed       = dataset_seed,
+                                                                shuffle    = False,
+                                                                filename   = prepared_dataset )
+```
+
+%% Cell type:markdown id: tags:
+
+### 4.2 - Retrieve encoder
+
+%% Cell type:code id: tags:
+
+``` python
+encoder=model.get_layer('encoder')
+```
+
+%% Cell type:markdown id: tags:
+
+### 4.3 Showing latent space
+Here is the digit distribution in the latent space
+
+%% Cell type:code id: tags:
+
+``` python
+n_show = min( 20000, len(clean_data) )
+
+# ---- Select images
+
+x_show, y_show = fidle.utils.pick_dataset(clean_data, class_data, n=n_show)
+
+# ---- Get latent points
+
+z = encoder.predict(x_show)
+
+# ---- Show them
+
+fig = plt.figure(figsize=(14, 10))
+plt.scatter(z[:, 0] , z[:, 1], c=y_show, cmap= 'tab10', alpha=0.5, s=30)
+plt.colorbar()
+fidle.scrawler.save_fig('08-Latent-space')
+plt.show()
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

AE.Keras3/04-ExtAE-with-MNIST.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3AE4] - Denoiser and classifier model
+<!-- DESC --> Episode 4 : Construction of a denoiser and classifier model
+
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Building a multiple output model, able to **denoise** and **classify**
+ - Understanding a more **advanced programming model**
+
+The calculation needs being important, it is preferable to use a very simple dataset such as MNIST.
+The use of a GPU is often indispensable.
+
+## What we're going to do :
+
+ - Defining a multiple output model using Keras procedural programing model
+ - Build the model
+ - Train it
+ - Follow the learning process
+
+## Data Terminology :
+- `clean_train`, `clean_test` for noiseless images
+- `noisy_train`, `noisy_test` for noisy images
+- `class_train`, `class_test` for the classes to which the images belong
+- `denoised_test` for denoised images at the output of the model
+- `classcat_test` for class prediction in model output (is a softmax)
+- `classid_test` class prediction (ie: argmax of classcat_test)
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init python stuff
+### 1.1 - Init
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+from skimage import io
+import random
+
+from modules.AE4_builder    import AE4_builder
+from modules.MNIST          import MNIST
+from modules.ImagesCallback import ImagesCallback
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3AE4')
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Parameters
+`prepared_dataset` : Filename of the prepared dataset (Need 400 Mo, but can be in ./data)
+`dataset_seed` : Random seed for shuffling dataset. 'None' mean using /dev/urandom
+`scale` : % of the dataset to use (1. for 100%)
+`latent_dim` : Dimension of the latent space
+`train_prop` : Percentage for train (the rest being for the test)
+`batch_size` : Batch size
+`epochs` : Nb of epochs for training\
+`fit_verbosity` is the verbosity during training : 0 = silent, 1 = progress bar, 2 = one line per epoch
+
+scale=0.1, epochs=20  => 2' on a laptop
+
+%% Cell type:code id: tags:
+
+``` python
+prepared_dataset = './data/mnist-noisy.h5'
+dataset_seed     = None
+
+scale            = .1
+
+train_prop       = .8
+batch_size       = 128
+epochs           = 10
+fit_verbosity    = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('prepared_dataset', 'dataset_seed', 'scale')
+fidle.override('train_prop', 'batch_size', 'epochs', 'fit_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve dataset
+With our MNIST class, in one call, we can reload, rescale, shuffle and split our previously saved dataset :-)
+
+%% Cell type:code id: tags:
+
+``` python
+clean_train,clean_test, noisy_train,noisy_test, class_train,class_test = MNIST.reload_prepared_dataset(
+                                                                                    scale      = scale,
+                                                                                    train_prop = train_prop,
+                                                                                    seed       = dataset_seed,
+                                                                                    shuffle    = True,
+                                                                                    filename   = prepared_dataset )
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Build models
+
+%% Cell type:code id: tags:
+
+``` python
+builder = AE4_builder( ae={ 'latent_dim':10 }, cnn = { 'lc1':8, 'lc2':16, 'ld':100 } )
+
+model = builder.create_model()
+```
+
+%% Cell type:code id: tags:
+
+``` python
+model.compile(optimizer='rmsprop',
+              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
+              loss_weights={'ae':1., 'classifier':1.},
+              metrics={'classifier':'accuracy'} )
+```
+
+%% Cell type:code id: tags:
+
+``` python
+# keras.utils.plot_model(model, "multi_input_and_output_model.png", show_shapes=True)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Train
+20' on a CPU
+1'12 on a GPU (V100, IDRIS)
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Callback : Images
+#
+fidle.utils.mkdir( run_dir + '/images')
+filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'
+
+encoder = model.get_layer('ae').get_layer('encoder')
+decoder = model.get_layer('ae').get_layer('decoder')
+
+callback_images = ImagesCallback(filename, x=clean_test[:5], encoder=encoder,decoder=decoder)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+chrono = fidle.Chrono()
+chrono.start()
+
+history = model.fit(noisy_train, [clean_train, class_train],
+                 batch_size      = batch_size,
+                 epochs          = epochs,
+                 verbose         = fit_verbosity,
+                 validation_data = (noisy_test, [clean_test, class_test]),
+                 callbacks       = [ callback_images ]  )
+
+chrono.show()
+```
+
+%% Cell type:markdown id: tags:
+
+Save model weights
+
+%% Cell type:code id: tags:
+
+``` python
+os.makedirs(f'{run_dir}/models', exist_ok=True)
+
+model.save_weights(f'{run_dir}/models/model.weights.h5')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - History
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history(history,  plot={'Loss':['loss', 'val_loss'],
+                                 'Accuracy':['classifier_accuracy','val_classifier_accuracy']}, save_as='01-history')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Denoising progress
+
+%% Cell type:code id: tags:
+
+``` python
+imgs=[]
+for epoch in range(0,epochs,4):
+    for i in range(5):
+        filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'.format(epoch=epoch, i=i)
+        img      = io.imread(filename)
+        imgs.append(img)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='02-original-real')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='03-original-noisy')
+
+fidle.utils.subtitle('Evolution during the training period (denoised_test) :')
+fidle.scrawler.images(imgs, None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, y_padding=0.1, save_as='04-learning')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Evaluation
+**Note :** We will use the following data:\
+`clean_train`, `clean_test` for noiseless images \
+`noisy_train`, `noisy_test` for noisy images\
+`class_train`, `class_test` for the classes to which the images belong \
+`denoised_test` for denoised images at the output of the model\
+`classcat_test` for class prediction in model output (is a softmax)\
+`classid_test` class prediction (ie: argmax of classcat_test)
+
+### 7.1 - Reload our model (weights)
+
+%% Cell type:code id: tags:
+
+``` python
+builder = AE4_builder( ae={ 'latent_dim':10 }, cnn = { 'lc1':8, 'lc2':16, 'ld':100 } )
+
+model = builder.create_model()
+
+model.load_weights(f'{run_dir}/models/model.weights.h5')
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.2 - Let's make a prediction
+Note that our model will returns 2 outputs : **denoised images** from output 1 and **class prediction** from output 2
+
+%% Cell type:code id: tags:
+
+``` python
+outputs = model.predict(noisy_test, verbose=0)
+
+denoised = outputs['ae']
+classcat = outputs['classifier']
+
+print('Denoised images   (denoised_test) shape : ', denoised.shape)
+print('Predicted classes (classcat_test) shape : ', classcat.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.3 - Denoised images
+
+%% Cell type:code id: tags:
+
+``` python
+i=random.randint(0,len(denoised)-8)
+j=i+8
+
+fidle.utils.subtitle('Noisy test images (input):')
+fidle.scrawler.images(noisy_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='05-test-noisy')
+
+fidle.utils.subtitle('Denoised images (output):')
+fidle.scrawler.images(denoised[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='06-test-predict')
+
+fidle.utils.subtitle('Real test images :')
+fidle.scrawler.images(clean_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='07-test-real')
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.4 - Class prediction
+Note: The evaluation requires the noisy images as input (noisy_test) and the 2 expected outputs:
+ - the images without noise (clean_test)
+ - the classes (class_test)
+
+%% Cell type:code id: tags:
+
+``` python
+# We need to (re)compile our resurrected model (to specify loss and metrics)
+#
+model.compile(optimizer='rmsprop',
+              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
+              loss_weights={'ae':1., 'classifier':1.},
+              metrics={'classifier':'accuracy'} )
+
+
+# Get an evaluation
+#
+score = model.evaluate(noisy_test, [clean_test, class_test], verbose=0)
+
+# And show results
+#
+fidle.utils.subtitle("Accuracy :")
+print(f'Classification accuracy : {score[1]:4.4f}')
+
+fidle.utils.subtitle("Few examples :")
+classid_test  = np.argmax(classcat, axis=-1)
+fidle.scrawler.images(noisy_test, class_test, range(0,200), columns=12, x_size=1, y_size=1, y_pred=classid_test, save_as='04-predictions')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

AE.Keras3/05-ExtAE-with-MNIST.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3AE5] - Advanced denoiser and classifier model
+<!-- DESC --> Episode 5 : Construction of an advanced denoiser and classifier model
+
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Building a multiple output model, able to **denoise** and **classify**
+ - Understanding a more complex **advanced programming model**
+
+The calculation needs being important, it is preferable to use a very simple dataset such as MNIST.
+The use of a GPU is often indispensable.
+
+## What we're going to do :
+
+ - Defining a multiple output model using Keras procedural programing model
+ - Build the model
+ - Train it
+ - Follow the learning process
+
+## Data Terminology :
+- `clean_train`, `clean_test` for noiseless images
+- `noisy_train`, `noisy_test` for noisy images
+- `class_train`, `class_test` for the classes to which the images belong
+- `denoised_test` for denoised images at the output of the model
+- `classcat_test` for class prediction in model output (is a softmax)
+- `classid_test` class prediction (ie: argmax of classcat_test)
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init python stuff
+### 1.1 - Init
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+from skimage import io
+import random
+
+from modules.AE5_builder    import AE5_builder
+from modules.MNIST          import MNIST
+from modules.ImagesCallback import ImagesCallback
+
+import fidle
+
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3AE5')
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Parameters
+`prepared_dataset` : Filename of the prepared dataset (Need 400 Mo, but can be in ./data)
+`dataset_seed` : Random seed for shuffling dataset
+`scale` : % of the dataset to use (1. for 100%)
+`latent_dim` : Dimension of the latent space
+`train_prop` : Percentage for train (the rest being for the test)
+`batch_size` : Batch size
+`epochs` : Nb of epochs for training\
+`fit_verbosity` is the verbosity during training : 0 = silent, 1 = progress bar, 2 = one line per epoch
+
+%% Cell type:code id: tags:
+
+``` python
+prepared_dataset = './data/mnist-noisy.h5'
+dataset_seed     = None
+
+scale            = .1
+
+train_prop       = .8
+batch_size       = 128
+epochs           = 10
+fit_verbosity    = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('prepared_dataset', 'dataset_seed', 'scale')
+fidle.override('train_prop', 'batch_size', 'epochs', 'fit_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve dataset
+With our MNIST class, in one call, we can reload, rescale, shuffle and split our previously saved dataset :-)
+
+%% Cell type:code id: tags:
+
+``` python
+clean_train,clean_test, noisy_train,noisy_test, class_train,class_test = MNIST.reload_prepared_dataset(
+                                                                                    scale      = scale,
+                                                                                    train_prop = train_prop,
+                                                                                    seed       = dataset_seed,
+                                                                                    shuffle    = True,
+                                                                                    filename   = prepared_dataset )
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Build model
+
+%% Cell type:code id: tags:
+
+``` python
+builder = AE5_builder( ae   = { 'latent_dim':10 },
+                       cnn1 = { 'lc1':8,  'lc2':16, 'ld':100 },
+                       cnn2 = { 'lc1':32, 'lc2':64, 'ld':50 } )
+
+model = builder.create_model()
+```
+
+%% Cell type:code id: tags:
+
+``` python
+model.compile(optimizer='rmsprop',
+              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
+              loss_weights={'ae':1., 'classifier':1.},
+              metrics={'classifier':'accuracy'} )
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Train
+20' on a CPU
+1'30 on a GPU (V100, IDRIS)
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Callback : Images
+#
+fidle.utils.mkdir( run_dir + '/images')
+filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'
+
+encoder = model.get_layer('ae').get_layer('encoder')
+decoder = model.get_layer('ae').get_layer('decoder')
+
+callback_images = ImagesCallback(filename, x=clean_test[:5], encoder=encoder,decoder=decoder)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+chrono = fidle.Chrono()
+chrono.start()
+
+history = model.fit(noisy_train, [clean_train, class_train],
+                 batch_size      = batch_size,
+                 epochs          = epochs,
+                 verbose         = fit_verbosity,
+                 validation_data = (noisy_test, [clean_test, class_test]),
+                 callbacks       = [ callback_images ] )
+
+
+chrono.show()
+```
+
+%% Cell type:markdown id: tags:
+
+Save model weights
+
+%% Cell type:code id: tags:
+
+``` python
+os.makedirs(f'{run_dir}/models', exist_ok=True)
+
+model.save_weights(f'{run_dir}/models/model.weights.h5')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - History
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history(history,  plot={'Loss':['loss', 'val_loss'],
+                                 'Accuracy':['classifier_accuracy','val_classifier_accuracy']}, save_as='01-history')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Denoising progress
+
+%% Cell type:code id: tags:
+
+``` python
+imgs=[]
+for epoch in range(0,epochs,4):
+    for i in range(5):
+        filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'.format(epoch=epoch, i=i)
+        img      = io.imread(filename)
+        imgs.append(img)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='02-original-real')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='03-original-noisy')
+
+fidle.utils.subtitle('Evolution during the training period (denoised_test) :')
+fidle.scrawler.images(imgs, None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, y_padding=0.1, save_as='04-learning')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Evaluation
+**Note :** We will use the following data:\
+`clean_train`, `clean_test` for noiseless images \
+`noisy_train`, `noisy_test` for noisy images\
+`class_train`, `class_test` for the classes to which the images belong \
+`denoised_test` for denoised images at the output of the model\
+`classcat_test` for class prediction in model output (is a softmax)\
+`classid_test` class prediction (ie: argmax of classcat_test)
+
+### 7.1 - Reload our best model
+
+%% Cell type:code id: tags:
+
+``` python
+builder = AE5_builder( ae   = { 'latent_dim':10 },
+                       cnn1 = { 'lc1':8,  'lc2':16, 'ld':100 },
+                       cnn2 = { 'lc1':32, 'lc2':64, 'ld':50 } )
+
+model = builder.create_model()
+
+model.load_weights(f'{run_dir}/models/model.weights.h5')
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.2 - Let's make a prediction
+Note that our model will returns 2 outputs : **denoised images** from output 1 and **class prediction** from output 2
+
+%% Cell type:code id: tags:
+
+``` python
+outputs = model.predict(noisy_test, verbose=0)
+
+denoised = outputs['ae']
+classcat = outputs['classifier']
+
+print('Denoised images   (denoised_test) shape : ', denoised.shape)
+print('Predicted classes (classcat_test) shape : ', classcat.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.3 - Denoised images
+
+%% Cell type:code id: tags:
+
+``` python
+i=random.randint(0,len(denoised)-8)
+j=i+8
+
+fidle.utils.subtitle('Noisy test images (input):')
+fidle.scrawler.images(noisy_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='05-test-noisy')
+
+fidle.utils.subtitle('Denoised images (output):')
+fidle.scrawler.images(denoised[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='06-test-predict')
+
+fidle.utils.subtitle('Real test images :')
+fidle.scrawler.images(clean_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='07-test-real')
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.4 - Class prediction
+Note: The evaluation requires the noisy images as input (noisy_test) and the 2 expected outputs:
+ - the images without noise (clean_test)
+ - the classes (class_test)
+
+%% Cell type:code id: tags:
+
+``` python
+# We need to (re)compile our resurrected model (to specify loss and metrics)
+#
+model.compile(optimizer='rmsprop',
+              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
+              loss_weights={'ae':1., 'classifier':1.},
+              metrics={'classifier':'accuracy'} )
+
+
+# Get an evaluation
+#
+score = model.evaluate(noisy_test, [clean_test, class_test], verbose=0)
+
+# And show results
+#
+fidle.utils.subtitle("Accuracy :")
+print(f'Classification accuracy : {score[1]:4.4f}')
+
+fidle.utils.subtitle("Few examples :")
+classid_test  = np.argmax(classcat, axis=-1)
+fidle.scrawler.images(noisy_test, class_test, range(0,200), columns=12, x_size=1, y_size=1, y_pred=classid_test, save_as='04-predictions')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

AE.Keras3/modules/AE4_builder.py

+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI 2024 - JL. Parouty
+# ------------------------------------------------------------------
+
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+import keras.layers as layers
+
+
+# ------------------------------------------------------------------
+#   A usefull class to manage our AE model
+# ------------------------------------------------------------------
+
+
+class AE4_builder():
+    
+    version = '0.1'
+    
+    def __init__(self,  ae  = { 'latent_dim':16 },
+                        cnn = { 'lc1':8, 'lc2':16, 'ld':100 }
+                ):
+        
+        self.ae   = ae
+        self.cnn  = cnn
+    
+    
+    
+    def create_encoder(self):
+        
+        latent_dim = self.ae['latent_dim']
+                
+        inputs    = keras.Input(shape=(28,28,1))
+        x         = layers.Conv2D(32, 3, activation="relu", strides=2, padding="same")(inputs)
+        x         = layers.Conv2D(64, 3, activation="relu", strides=2, padding="same")(x)
+        x         = layers.Flatten()(x)
+        x         = layers.Dense(16,     activation="relu")(x)
+        z         = layers.Dense(latent_dim, name='latent')(x)
+
+        encoder = keras.Model(inputs, z, name="encoder")    
+        return encoder
+        
+        
+        
+    def create_decoder(self):
+
+        latent_dim = self.ae['latent_dim']
+
+        inputs  = keras.Input(shape=(latent_dim,))
+        x       = layers.Dense(7 * 7 * 64, activation="relu")(inputs)
+        x       = layers.Reshape((7, 7, 64))(x)
+        x       = layers.Conv2DTranspose(64, 3, activation="relu", strides=2, padding="same")(x)
+        x       = layers.Conv2DTranspose(32, 3, activation="relu", strides=2, padding="same")(x)
+        outputs = layers.Conv2DTranspose(1, 3, activation="sigmoid", padding="same", name='denoiser')(x)
+
+        decoder = keras.Model(inputs, outputs, name="decoder")
+        return decoder
+    
+        
+        
+    def create_cnn(self):
+
+        lc1     = self.cnn['lc1']
+        lc2     = self.cnn['lc2']
+        ld      = self.cnn['ld']
+
+        inputs    = keras.Input(shape=(28, 28, 1))
+
+        x         = keras.layers.Conv2D(lc1, (3,3),  activation='relu')(inputs)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.2)(x)
+
+        x         = keras.layers.Conv2D(lc2, (3,3), activation='relu')(x)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.2)(x)
+
+        x         = keras.layers.Flatten()(x)
+        x         = keras.layers.Dense(ld, activation='relu')(x)
+        x         = keras.layers.Dropout(0.5)(x)
+
+        outputs   = keras.layers.Dense(10, activation='softmax', name='cnn')(x)
+
+        cnn       = keras.Model(inputs, outputs, name='classifier')
+        return cnn
+    
+    
+    
+    def create_model(self):
+        
+        # ---- Recover all elementary bricks
+        #
+        encoder = self.create_encoder()
+        decoder = self.create_decoder()
+        cnn     = self.create_cnn()
+        
+        # ---- Build ae
+        #
+        inputs    = keras.Input(shape=(28, 28, 1))
+        latents   = encoder(inputs)
+        outputs   = decoder(latents)
+
+        ae = keras.Model(inputs,outputs, name='ae')
+
+        # ---- Assembling final model
+        #
+        inputs    = keras.Input(shape=(28, 28, 1))
+
+        denoised  = ae(inputs)
+        classcat  = cnn(inputs)
+
+        model = keras.Model(inputs, outputs={ 'ae':denoised, 'classifier':classcat})
+        
+        return model
\ No newline at end of file

AE.Keras3/modules/AE5_builder.py

+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI 2024 - JL. Parouty
+# ------------------------------------------------------------------
+
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+import keras.layers as layers
+
+
+# ------------------------------------------------------------------
+#   A usefull class to manage our AE model
+# ------------------------------------------------------------------
+
+
+class AE5_builder():
+    
+    version = '0.1'
+    
+    def __init__(self,  ae   = { 'latent_dim':16 },
+                        cnn1 = { 'lc1':8, 'lc2':16, 'ld':100 },
+                        cnn2 = { 'lc1':32, 'lc2':64, 'ld':50 }
+                ):
+        
+        self.ae   = ae
+        self.cnn1  = cnn1
+        self.cnn2  = cnn2
+    
+    
+    
+    def create_encoder(self):
+        
+        latent_dim = self.ae['latent_dim']
+                
+        inputs    = keras.Input(shape=(28,28,1))
+        x         = layers.Conv2D(32, 3, activation="relu", strides=2, padding="same")(inputs)
+        x         = layers.Conv2D(64, 3, activation="relu", strides=2, padding="same")(x)
+        x         = layers.Flatten()(x)
+        x         = layers.Dense(16,     activation="relu")(x)
+        z         = layers.Dense(latent_dim, name='latent')(x)
+
+        encoder = keras.Model(inputs, z, name="encoder")    
+        return encoder
+        
+        
+        
+    def create_decoder(self):
+
+        latent_dim = self.ae['latent_dim']
+
+        inputs  = keras.Input(shape=(latent_dim,))
+        x       = layers.Dense(7 * 7 * 64, activation="relu")(inputs)
+        x       = layers.Reshape((7, 7, 64))(x)
+        x       = layers.Conv2DTranspose(64, 3, activation="relu", strides=2, padding="same")(x)
+        x       = layers.Conv2DTranspose(32, 3, activation="relu", strides=2, padding="same")(x)
+        outputs = layers.Conv2DTranspose(1, 3, activation="sigmoid", padding="same", name='denoiser')(x)
+
+        decoder = keras.Model(inputs, outputs, name="decoder")
+        return decoder
+    
+        
+        
+    def create_cnn1(self):
+
+        lc1     = self.cnn1['lc1']
+        lc2     = self.cnn1['lc2']
+        ld      = self.cnn1['ld']
+
+        inputs    = keras.Input(shape=(28, 28, 1))
+
+        x         = keras.layers.Conv2D(lc1, (3,3),  activation='relu')(inputs)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.2)(x)
+
+        x         = keras.layers.Conv2D(lc2, (3,3), activation='relu')(x)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.2)(x)
+
+        x         = keras.layers.Flatten()(x)
+        x         = keras.layers.Dense(ld, activation='relu')(x)
+        x         = keras.layers.Dropout(0.5)(x)
+
+        outputs   = keras.layers.Dense(10, activation='softmax', name='classifier')(x)
+
+        cnn       = keras.Model(inputs, outputs, name='cnn')
+        return cnn
+    
+    
+    def create_cnn2(self):
+
+        lc1     = self.cnn2['lc1']
+        lc2     = self.cnn2['lc2']
+        ld      = self.cnn2['ld']
+        
+        inputs    = keras.Input(shape=(28, 28, 1))
+
+        x         = keras.layers.Conv2D(lc1, (5,5),  activation='relu')(inputs)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.3)(x)
+
+        x         = keras.layers.Conv2D(lc2, (5,5), activation='relu')(x)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.3)(x)
+
+        x         = keras.layers.Flatten()(x)
+        x         = keras.layers.Dense(ld, activation='relu')(x)
+        outputs   = keras.layers.Dropout(0.3)(x)
+
+        cnn2       = keras.Model(inputs, outputs, name='cnn2')
+        return cnn2
+    
+    
+    
+    
+    def create_model(self):
+        
+        # ---- Recover all elementary bricks
+        #
+        encoder = self.create_encoder()
+        decoder = self.create_decoder()
+        cnn1    = self.create_cnn1()
+        cnn2    = self.create_cnn2()
+        
+        # ---- Build ae
+        #
+        inputs    = keras.Input(shape=(28, 28, 1))
+        latents   = encoder(inputs)
+        outputs   = decoder(latents)
+
+        ae = keras.Model(inputs,outputs, name='ae')
+
+        # ---- Build classifier
+        #
+        branch_1 = cnn1(inputs)
+        branch_2 = cnn2(inputs)
+        
+        x        = keras.layers.concatenate([branch_1,branch_2], axis=1)
+        outputs  = keras.layers.Dense(10, activation='softmax')(x)
+
+        classifier = keras.Model(inputs,outputs, name='classifier')
+
+        # ---- Assembling final model
+        #
+        inputs    = keras.Input(shape=(28, 28, 1))
+
+        denoised  = ae(inputs)
+        classcat  = classifier(inputs)
+        
+        model = keras.Model(inputs, outputs={ 'ae':denoised, 'classifier':classcat})
+        
+        return model
\ No newline at end of file

VAE/modules/callbacks.py → AE.Keras3/modules/ImagesCallback.py

-from tensorflow.keras.callbacks import Callback
+from keras.callbacks import Callback
 import numpy as np
 import matplotlib.pyplot as plt
 
 class ImagesCallback(Callback):
     
    
-    def __init__(self, filename= 'image-{epoch:03d}-{i:02d}.jpg', z_dim=0, decoder=None, nb_images=5):
+    def __init__(self, filename='image-{epoch:03d}-{i:02d}.jpg', 
+                       x=None,
+                       encoder=None, decoder=None):
         self.filename  = filename
-        self.z_dim     = z_dim
+        self.x         = x
+        self.encoder   = encoder
         self.decoder   = decoder
-        self.nb_images = nb_images
-        
+        if len(x)>100:
+            print('***Warning : The number of images is reduced to 100')
+            self.x=x[:100]
         
     def on_epoch_end(self, epoch, logs={}):  
         
-        # ---- Get random latent points
-        
-        z_new   = np.random.normal(size = (self.nb_images,self.z_dim))
+        # ---- Get latent points
+        #
+        z_new  = self.encoder.predict(self.x)
         
         # ---- Predict an image
-        
+        #
         images = self.decoder.predict(np.array(z_new))
         
         # ---- Save images
-        
+        #
         for i,image in enumerate(images):
             
             # ---- Squeeze it if monochrome : (lx,ly,1) -> (lx,ly) 
+            #
             image = image.squeeze()
         
             # ---- Save it
-
+            #
             filename = self.filename.format(epoch=epoch,i=i)
             
             if len(image.shape) == 2:

AE.Keras3/modules/MNIST.py

+
+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/SARI/DEVLOG 2020 - S. Arias, E. Maldonado, JL. Parouty
+# ------------------------------------------------------------------
+# 2.0 version by JL Parouty, feb 2021
+
+import h5py
+import os
+import numpy as np
+from hashlib import blake2b
+import keras
+import keras.datasets.mnist as mnist
+
+
+# ------------------------------------------------------------------
+#   A usefull class to manage our MNIST dataset
+#   This class allows to manage datasets derived from the original MNIST
+# ------------------------------------------------------------------
+
+
+class MNIST():
+    
+    version = '0.1'
+    
+    def __init__(self):
+        pass
+   
+    @classmethod
+    def get_origine(cls, scale=1, normalize=True, expand=True, concatenate=True):
+        """
+        Return original MNIST dataset
+        args:
+            scale       : Proportion of the requested dataset
+            normalize   : Normalize dataset or not (True)
+            expand      : Reshape images as (28,28,1) instead (28,28) (True)
+            concatenate : Concatenate train and test sets (True)
+        returns:
+            x_data,y_data                   if concatenate is False
+            x_train,y_train,x_test,y_test   if concatenate is True
+        """
+
+        # ---- Get data
+        #
+        (x_train, y_train), (x_test, y_test) = mnist.load_data()
+        print('Dataset loaded.')
+        
+        # ---- Normalization
+        #
+        if normalize:
+            x_train = x_train.astype('float32') / 255.
+            x_test  = x_test.astype( 'float32') / 255.
+            print('Normalized.')
+            
+        # ---- Reshape : (28,28) -> (28,28,1)
+        #
+        if expand:
+            x_train = np.expand_dims(x_train, axis=-1)
+            x_test  = np.expand_dims(x_test,  axis=-1)
+            print('Reshaped.')
+
+        # ---- scale
+        #
+        n1 = int(len(x_train)*scale)
+        n2 = int(len(x_test)*scale)
+        x_train = x_train[:n1]
+        y_train = y_train[:n1]
+        x_test  = x_test[:n2]
+        y_test  = y_test[:n2]
+
+        # ---- Concatenate
+        #
+        if concatenate:
+            x_data = np.concatenate([x_train, x_test], axis=0)
+            y_data = np.concatenate([y_train, y_test])
+            print('Concatenate.')
+            print('x shape :', x_data.shape)
+            print('y shape :', y_data.shape)
+            return x_data,y_data
+        else:
+            print('x_train shape :', x_train.shape)
+            print('y_train shape :', y_train.shape)
+            print('x_test  shape :', x_test.shape)
+            print('y_test  shape :', y_test.shape)
+            return x_train,y_train,x_test,y_test
+        
+        
+    @classmethod
+    def save_prepared_dataset(cls, clean_data, noisy_data, class_data, filename='./data/mnist-noisy.h5'):
+        """
+        Save a prepared dataset in a h5 file
+        args:
+            clean_data, noisy_data, class_data : clean, noisy and class dataset
+            filename                      : filename
+        return:
+            None
+        """
+        path=os.path.dirname(filename)
+        os.makedirs(path, mode=0o750, exist_ok=True)
+
+        with h5py.File(filename, "w") as f:
+            f.create_dataset("clean_data", data=clean_data)
+            f.create_dataset("noisy_data", data=noisy_data)
+            f.create_dataset("class_data", data=class_data)
+        print('Saved.')
+        print('clean_data shape is : ',clean_data.shape)
+        print('noisy_data shape is : ',noisy_data.shape)
+        print('class_data shape is : ',class_data.shape)
+            
+            
+    @classmethod    
+    def reload_prepared_dataset(cls, scale=1., train_prop=0.8, shuffle=True, seed=False, filename='./data/mnist-noisy.h5'):
+        """
+        Reload a saved dataset
+        args:
+            scale      : Scale of dataset to use. 1. mean 100% (1.)
+            train_prop : Ratio of train/test
+            shuffle    : Shuffle data if True
+            seed       : Random seed value. False mean no seed, None mean using /dev/urandom (None)
+            filename   : filename of the prepared dataset
+        returns:
+            clean_train,clean_test, noisy_train,noisy_test, class_train,class_test
+        """
+        # ---- Load saved dataset
+        #
+        with  h5py.File(filename,'r') as f:
+            clean_data  = f['clean_data'][:]
+            noisy_data  = f['noisy_data'][:]
+            class_data  = f['class_data'][:]
+        print('Loaded.')
+        
+        # ---- Rescale
+        #
+        n = int(scale*len(clean_data))
+        clean_data, noisy_data, class_data = clean_data[:n], noisy_data[:n], class_data[:n]
+        print(f'rescaled ({scale}).') 
+        
+        # ---- Seed
+        #
+        if seed is not False:
+            np.random.seed(seed)
+            print(f'Seeded ({seed})')
+        
+        # ---- Shuffle
+        #
+        if shuffle:
+            p = np.random.permutation(len(clean_data))
+            clean_data, noisy_data, class_data = clean_data[p], noisy_data[p], class_data[p]
+            print('Shuffled.')
+        
+        # ---- Split
+        #
+        n=int(len(clean_data)*train_prop)
+        clean_train, clean_test = clean_data[:n], clean_data[n:]
+        noisy_train, noisy_test = noisy_data[:n], noisy_data[n:]
+        class_train, class_test = class_data[:n], class_data[n:]
+        print(f'splited ({train_prop}).') 
+
+        # ---- Hash
+        #
+        h = blake2b(digest_size=10)
+        for a in [clean_train,clean_test, noisy_train,noisy_test, class_train,class_test]:
+            h.update(a)
+        
+        print('clean_train shape is : ', clean_train.shape)
+        print('clean_test  shape is : ', clean_test.shape)
+        print('noisy_train shape is : ', noisy_train.shape)
+        print('noisy_test  shape is : ', noisy_test.shape)
+        print('class_train shape is : ', class_train.shape)
+        print('class_test  shape is : ', class_test.shape)
+        print('Blake2b digest is    : ', h.hexdigest())
+        return  clean_train,clean_test, noisy_train,noisy_test, class_train,class_test
\ No newline at end of file

BHPD.Keras3/01-DNN-Regression.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+
+# <!-- TITLE --> [K3BHPD1] - Regression with a Dense Network (DNN)
+<!-- DESC --> Simple example of a regression with the dataset Boston Housing Prices Dataset (BHPD)
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Predicts **housing prices** from a set of house features.
+ - Understanding the **principle** and the **architecture** of a regression with a **dense neural network**
+
+
+The **[Boston Housing Prices Dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html)** consists of price of houses in various places in Boston.
+Alongside with price, the dataset also provide theses informations :
+
+ - CRIM: This is the per capita crime rate by town
+ - ZN: This is the proportion of residential land zoned for lots larger than 25,000 sq.ft
+ - INDUS: This is the proportion of non-retail business acres per town
+ - CHAS: This is the Charles River dummy variable (this is equal to 1 if tract bounds river; 0 otherwise)
+ - NOX: This is the nitric oxides concentration (parts per 10 million)
+ - RM: This is the average number of rooms per dwelling
+ - AGE: This is the proportion of owner-occupied units built prior to 1940
+ - DIS: This is the weighted distances to five Boston employment centers
+ - RAD: This is the index of accessibility to radial highways
+ - TAX: This is the full-value property-tax rate per 10,000 dollars
+ - PTRATIO: This is the pupil-teacher ratio by town
+ - B: This is calculated as 1000(Bk — 0.63)^2, where Bk is the proportion of people of African American descent by town
+ - LSTAT: This is the percentage lower status of the population
+ - MEDV: This is the median value of owner-occupied homes in 1000 dollars
+
+## What we're going to do :
+
+ - Retrieve data
+ - Preparing the data
+ - Build a model
+ - Train the model
+ - Evaluate the result
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Import and init
+
+You can also adjust the verbosity by changing the value of TF_CPP_MIN_LOG_LEVEL :
+- 0 = all messages are logged (default)
+- 1 = INFO messages are not printed.
+- 2 = INFO and WARNING messages are not printed.
+- 3 = INFO , WARNING and ERROR messages are not printed.
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+import matplotlib.pyplot as plt
+import pandas as pd
+import os,sys
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3BHPD1')
+```
+
+%% Cell type:markdown id: tags:
+
+Verbosity during training :
+- 0 = silent
+- 1 = progress bar
+- 2 = one line per epoch
+
+%% Cell type:code id: tags:
+
+``` python
+fit_verbosity = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('fit_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve data
+
+### 2.1 - Option 1  : From Keras
+Boston housing is a famous historic dataset, so we can get it directly from [Keras datasets](https://keras.io/api/datasets)
+
+%% Cell type:code id: tags:
+
+``` python
+# (x_train, y_train), (x_test, y_test) = keras.datasets.boston_housing.load_data(test_split=0.2, seed=113)
+```
+
+%% Cell type:markdown id: tags:
+
+### 2.2 - Option 2 : From a csv file
+More fun !
+
+%% Cell type:code id: tags:
+
+``` python
+data = pd.read_csv(f'{datasets_dir}/BHPD/origine/BostonHousing.csv', header=0)
+
+display(data.head(5).style.format("{0:.2f}").set_caption("Few lines of the dataset :"))
+print('Missing Data : ',data.isna().sum().sum(), '  Shape is : ', data.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Preparing the data
+### 3.1 - Split data
+We will use 70% of the data for training and 30% for validation.
+The dataset is **shuffled** and shared between **learning** and **testing**.
+x will be input data and y the expected output
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Shuffle and Split => train, test
+#
+data       = data.sample(frac=1., axis=0)
+data_train = data.sample(frac=0.7, axis=0)
+data_test  = data.drop(data_train.index)
+
+# ---- Split => x,y (medv is price)
+#
+x_train = data_train.drop('medv',  axis=1)
+y_train = data_train['medv']
+x_test  = data_test.drop('medv',   axis=1)
+y_test  = data_test['medv']
+
+print('Original data shape was : ',data.shape)
+print('x_train : ',x_train.shape, 'y_train : ',y_train.shape)
+print('x_test  : ',x_test.shape,  'y_test  : ',y_test.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 3.2 - Data normalization
+**Note :**
+ - All input data must be normalized, train and test.
+ - To do this we will **subtract the mean** and **divide by the standard deviation**.
+ - But test data should not be used in any way, even for normalization.
+ - The mean and the standard deviation will therefore only be calculated with the train data.
+
+%% Cell type:code id: tags:
+
+``` python
+display(x_train.describe().style.format("{0:.2f}").set_caption("Before normalization :"))
+
+mean = x_train.mean()
+std  = x_train.std()
+x_train = (x_train - mean) / std
+x_test  = (x_test  - mean) / std
+
+display(x_train.describe().style.format("{0:.2f}").set_caption("After normalization :"))
+display(x_train.head(5).style.format("{0:.2f}").set_caption("Few lines of the dataset :"))
+
+x_train, y_train = np.array(x_train), np.array(y_train)
+x_test,  y_test  = np.array(x_test),  np.array(y_test)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Build a model
+About informations about :
+ - [Optimizer](https://keras.io/api/optimizers)
+ - [Activation](https://keras.io/api/layers/activations)
+ - [Loss](https://keras.io/api/losses)
+ - [Metrics](https://keras.io/api/metrics)
+
+%% Cell type:code id: tags:
+
+``` python
+def get_model_v1(shape):
+
+  model = keras.models.Sequential()
+  model.add(keras.layers.Input(shape, name="InputLayer"))
+  model.add(keras.layers.Dense(32, activation='relu', name='Dense_n1'))
+  model.add(keras.layers.Dense(64, activation='relu', name='Dense_n2'))
+  model.add(keras.layers.Dense(32, activation='relu', name='Dense_n3'))
+  model.add(keras.layers.Dense(1, name='Output'))
+
+  model.compile(optimizer = 'adam',
+                loss      = 'mse',
+                metrics   = ['mae', 'mse'] )
+  return model
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Train the model
+### 5.1 - Get it
+
+%% Cell type:code id: tags:
+
+``` python
+model=get_model_v1( (13,) )
+
+model.summary()
+
+# img=keras.utils.plot_model( model, to_file='./run/model.png', show_shapes=True, show_layer_names=True, dpi=96)
+# display(img)
+```
+
+%% Cell type:markdown id: tags:
+
+### 5.2 - Train it
+
+%% Cell type:code id: tags:
+
+``` python
+history = model.fit(x_train,
+                    y_train,
+                    epochs          = 60,
+                    batch_size      = 10,
+                    verbose         = fit_verbosity,
+                    validation_data = (x_test, y_test))
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Evaluate
+### 6.1 - Model evaluation
+MAE =  Mean Absolute Error (between the labels and predictions)
+A mae equal to 3 represents an average error in prediction of $3k.
+
+%% Cell type:code id: tags:
+
+``` python
+score = model.evaluate(x_test, y_test, verbose=0)
+
+print('x_test / loss      : {:5.4f}'.format(score[0]))
+print('x_test / mae       : {:5.4f}'.format(score[1]))
+print('x_test / mse       : {:5.4f}'.format(score[2]))
+```
+
+%% Cell type:markdown id: tags:
+
+### 6.2 - Training history
+What was the best result during our training ?
+
+%% Cell type:code id: tags:
+
+``` python
+df=pd.DataFrame(data=history.history)
+display(df)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+print("min( val_mae ) : {:.4f}".format( min(history.history["val_mae"]) ) )
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history( history, plot={'MSE' :['mse', 'val_mse'],
+                        'MAE' :['mae', 'val_mae'],
+                        'LOSS':['loss','val_loss']}, save_as='01-history')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Make a prediction
+The data must be normalized with the parameters (mean, std) previously used.
+
+%% Cell type:code id: tags:
+
+``` python
+my_data = [ 1.26425925, -0.48522739,  1.0436489 , -0.23112788,  1.37120745,
+       -2.14308942,  1.13489104, -1.06802005,  1.71189006,  1.57042287,
+        0.77859951,  0.14769795,  2.7585581 ]
+real_price = 10.4
+
+my_data=np.array(my_data).reshape(1,13)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+
+predictions = model.predict( my_data, verbose=fit_verbosity )
+print("Prediction : {:.2f} K$".format(predictions[0][0]))
+print("Reality    : {:.2f} K$".format(real_price))
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

BHPD.Keras3/02-DNN-Regression-Premium.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3BHPD2] - Regression with a Dense Network (DNN) - Advanced code
+  <!-- DESC -->  A more advanced implementation of the precedent example, using Keras3
+  <!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Predicts **housing prices** from a set of house features.
+ - Understanding the principle and the architecture of a regression with a dense neural network with backup and restore of the trained model.
+
+The **[Boston Housing Prices Dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html)** consists of price of houses in various places in Boston.
+Alongside with price, the dataset also provide these information :
+
+ - CRIM: This is the per capita crime rate by town
+ - ZN: This is the proportion of residential land zoned for lots larger than 25,000 sq.ft
+ - INDUS: This is the proportion of non-retail business acres per town
+ - CHAS: This is the Charles River dummy variable (this is equal to 1 if tract bounds river; 0 otherwise)
+ - NOX: This is the nitric oxides concentration (parts per 10 million)
+ - RM: This is the average number of rooms per dwelling
+ - AGE: This is the proportion of owner-occupied units built prior to 1940
+ - DIS: This is the weighted distances to five Boston employment centers
+ - RAD: This is the index of accessibility to radial highways
+ - TAX: This is the full-value property-tax rate per 10,000 dollars
+ - PTRATIO: This is the pupil-teacher ratio by town
+ - B: This is calculated as 1000(Bk — 0.63)^2, where Bk is the proportion of people of African American descent by town
+ - LSTAT: This is the percentage lower status of the population
+ - MEDV: This is the median value of owner-occupied homes in 1000 dollars
+
+## What we're going to do :
+
+ - (Retrieve data)
+ - (Preparing the data)
+ - (Build a model)
+ - Train and save the model
+ - Restore saved model
+ - Evaluate the model
+ - Make some predictions
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Import and init
+
+You can also adjust the verbosity by changing the value of TF_CPP_MIN_LOG_LEVEL :
+- 0 = all messages are logged (default)
+- 1 = INFO messages are not printed.
+- 2 = INFO and WARNING messages are not printed.
+- 3 = INFO , WARNING and ERROR messages are not printed.
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+import matplotlib.pyplot as plt
+import pandas as pd
+import os,sys
+
+from IPython.display import Markdown
+from importlib import reload
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3BHPD2')
+```
+
+%% Cell type:markdown id: tags:
+
+Verbosity during training :
+- 0 = silent
+- 1 = progress bar
+- 2 = one line per epoch
+
+%% Cell type:code id: tags:
+
+``` python
+fit_verbosity = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('fit_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve data
+
+### 2.1 - Option 1  : From Keras
+Boston housing is a famous historic dataset, so we can get it directly from [Keras datasets](https://keras.io/api/datasets)
+
+%% Cell type:code id: tags:
+
+``` python
+# (x_train, y_train), (x_test, y_test) = keras.datasets.boston_housing.load_data(test_split=0.2, seed=113)
+```
+
+%% Cell type:markdown id: tags:
+
+### 2.2 - Option 2 : From a csv file
+More fun !
+
+%% Cell type:code id: tags:
+
+``` python
+data = pd.read_csv(f'{datasets_dir}/BHPD/origine/BostonHousing.csv', header=0)
+
+display(data.head(5).style.format("{0:.2f}"))
+print('Missing Data : ',data.isna().sum().sum(), '  Shape is : ', data.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Preparing the data
+### 3.1 - Split data
+We will use 80% of the data for training and 20% for validation.
+x will be input data and y the expected output
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Split => train, test
+#
+data       = data.sample(frac=1., axis=0)
+data_train = data.sample(frac=0.7, axis=0)
+data_test  = data.drop(data_train.index)
+
+# ---- Split => x,y (medv is price)
+#
+x_train = data_train.drop('medv',  axis=1)
+y_train = data_train['medv']
+x_test  = data_test.drop('medv',   axis=1)
+y_test  = data_test['medv']
+
+print('Original data shape was : ',data.shape)
+print('x_train : ',x_train.shape, 'y_train : ',y_train.shape)
+print('x_test  : ',x_test.shape,  'y_test  : ',y_test.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 3.2 - Data normalization
+**Note :**
+ - All input data must be normalized, train and test.
+ - To do this we will subtract the mean and divide by the standard deviation.
+ - But test data should not be used in any way, even for normalization.
+ - The mean and the standard deviation will therefore only be calculated with the train data.
+
+%% Cell type:code id: tags:
+
+``` python
+display(x_train.describe().style.format("{0:.2f}").set_caption("Before normalization :"))
+
+mean = x_train.mean()
+std  = x_train.std()
+x_train = (x_train - mean) / std
+x_test  = (x_test  - mean) / std
+
+display(x_train.describe().style.format("{0:.2f}").set_caption("After normalization :"))
+
+x_train, y_train = np.array(x_train), np.array(y_train)
+x_test,  y_test  = np.array(x_test),  np.array(y_test)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Build a model
+More informations about :
+ - [Optimizer](https://keras.io/api/optimizers)
+ - [Activation](https://keras.io/api/layers/activations)
+ - [Loss](https://keras.io/api/losses)
+ - [Metrics](https://keras.io/api/metrics)
+
+%% Cell type:code id: tags:
+
+``` python
+def get_model_v1(shape):
+
+  model = keras.models.Sequential()
+  model.add(keras.layers.Input(shape, name="InputLayer"))
+  model.add(keras.layers.Dense(64, activation='relu', name='Dense_n1'))
+  model.add(keras.layers.Dense(64, activation='relu', name='Dense_n2'))
+  model.add(keras.layers.Dense(1, name='Output'))
+
+  model.compile(optimizer = 'rmsprop',
+                loss      = 'mse',
+                metrics   = ['mae', 'mse'] )
+  return model
+```
+
+%% Cell type:markdown id: tags:
+
+## 5 - Train the model
+### 5.1 - Get it
+
+%% Cell type:code id: tags:
+
+``` python
+model=get_model_v1( (13,) )
+
+model.summary()
+```
+
+%% Cell type:markdown id: tags:
+
+### 5.2 - Add callback
+
+%% Cell type:code id: tags:
+
+``` python
+os.makedirs(f'{run_dir}/models',   mode=0o750, exist_ok=True)
+save_dir = f'{run_dir}/models/best_model.keras'
+
+savemodel_callback = keras.callbacks.ModelCheckpoint( filepath=save_dir, monitor='val_mae', mode='max', save_best_only=True)
+```
+
+%% Cell type:markdown id: tags:
+
+### 5.3 - Train it
+
+%% Cell type:code id: tags:
+
+``` python
+history = model.fit(x_train,
+                    y_train,
+                    epochs          = 50,
+                    batch_size      = 10,
+                    verbose         = fit_verbosity,
+                    validation_data = (x_test, y_test),
+                    callbacks       = [savemodel_callback])
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Evaluate
+### 6.1 - Model evaluation
+MAE =  Mean Absolute Error (between the labels and predictions)
+A mae equal to 3 represents an average error in prediction of $3k.
+
+%% Cell type:code id: tags:
+
+``` python
+score = model.evaluate(x_test, y_test, verbose=0)
+
+print('x_test / loss      : {:5.4f}'.format(score[0]))
+print('x_test / mae       : {:5.4f}'.format(score[1]))
+print('x_test / mse       : {:5.4f}'.format(score[2]))
+```
+
+%% Cell type:markdown id: tags:
+
+### 6.2 - Training history
+What was the best result during our training ?
+
+%% Cell type:code id: tags:
+
+``` python
+print("min( val_mae ) : {:.4f}".format( min(history.history["val_mae"]) ) )
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history( history, plot={'MSE' :['mse', 'val_mse'],
+                        'MAE' :['mae', 'val_mae'],
+                        'LOSS':['loss','val_loss']}, save_as='01-history')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Restore a model :
+
+%% Cell type:markdown id: tags:
+
+### 7.1 - Reload model
+
+%% Cell type:code id: tags:
+
+``` python
+loaded_model = keras.models.load_model(f'{run_dir}/models/best_model.keras')
+loaded_model.summary()
+print("Loaded.")
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.2 - Evaluate it :
+
+%% Cell type:code id: tags:
+
+``` python
+score = loaded_model.evaluate(x_test, y_test, verbose=0)
+
+print('x_test / loss      : {:5.4f}'.format(score[0]))
+print('x_test / mae       : {:5.4f}'.format(score[1]))
+print('x_test / mse       : {:5.4f}'.format(score[2]))
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.3 - Make a prediction
+
+%% Cell type:code id: tags:
+
+``` python
+my_data = [ 1.26425925, -0.48522739,  1.0436489 , -0.23112788,  1.37120745,
+       -2.14308942,  1.13489104, -1.06802005,  1.71189006,  1.57042287,
+        0.77859951,  0.14769795,  2.7585581 ]
+real_price = 10.4
+
+my_data=np.array(my_data).reshape(1,13)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+predictions = loaded_model.predict( my_data, verbose=fit_verbosity )
+print("Prediction : {:.2f} K$   Reality : {:.2f} K$".format(predictions[0][0], real_price))
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

BHPD.PyTorch/01-DNN-Regression_PyTorch.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+
+# <!-- TITLE --> [PBHPD1] - Regression with a Dense Network (DNN)
+<!-- DESC --> A Simple regression with a Dense Neural Network (DNN) using Pytorch - BHPD dataset
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP), Laurent Risser (CNRS/IMT) -->
+
+## Objectives :
+ - Predicts **housing prices** from a set of house features.
+ - Understanding the **principle** and the **architecture** of a regression with a **dense neural network**
+
+
+The **[Boston Housing Dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html)** consists of price of houses in various places in Boston.
+Alongside with price, the dataset also provide theses informations :
+
+ - CRIM: This is the per capita crime rate by town
+ - ZN: This is the proportion of residential land zoned for lots larger than 25,000 sq.ft
+ - INDUS: This is the proportion of non-retail business acres per town
+ - CHAS: This is the Charles River dummy variable (this is equal to 1 if tract bounds river; 0 otherwise)
+ - NOX: This is the nitric oxides concentration (parts per 10 million)
+ - RM: This is the average number of rooms per dwelling
+ - AGE: This is the proportion of owner-occupied units built prior to 1940
+ - DIS: This is the weighted distances to five Boston employment centers
+ - RAD: This is the index of accessibility to radial highways
+ - TAX: This is the full-value property-tax rate per 10,000 dollars
+ - PTRATIO: This is the pupil-teacher ratio by town
+ - B: This is calculated as 1000(Bk — 0.63)^2, where Bk is the proportion of people of African American descent by town
+ - LSTAT: This is the percentage lower status of the population
+ - MEDV: This is the median value of owner-occupied homes in 1000 dollars
+## What we're going to do :
+
+ - Retrieve data
+ - Preparing the data
+ - Build a model
+ - Train the model
+ - Evaluate the result
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Import and init
+
+%% Cell type:code id: tags:
+
+``` python
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.autograd import Variable
+
+
+import numpy as np
+import matplotlib.pyplot as plt
+import sys,os
+
+import pandas as pd
+
+from modules.fidle_pwk_additional import convergence_history_MSELoss
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('PBHPD1')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve data
+
+
+Boston housing is a famous historic dataset, which can be get here: [Boston housing datasets](https://www.kaggle.com/puxama/bostoncsv)
+
+%% Cell type:code id: tags:
+
+``` python
+data = pd.read_csv('./BostonHousing.csv', header=0)
+
+display(data.head(5).style.format("{0:.2f}").set_caption("Few lines of the dataset :"))
+print('Missing Data : ',data.isna().sum().sum(), '  Shape is : ', data.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Preparing the data
+### 3.1 - Split data
+We will use 70% of the data for training and 30% for validation.
+The dataset is **shuffled** and shared between **learning** and **testing**.
+x will be input data and y the expected output
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Shuffle and Split => train, test
+#
+data_train = data.sample(frac=0.7, axis=0)
+data_test  = data.drop(data_train.index)
+
+# ---- Split => x,y (medv is price)
+#
+x_train = data_train.drop('medv',  axis=1)
+y_train = data_train['medv']
+x_test  = data_test.drop('medv',   axis=1)
+y_test  = data_test['medv']
+
+print('Original data shape was : ',data.shape)
+print('x_train : ',x_train.shape, 'y_train : ',y_train.shape)
+print('x_test  : ',x_test.shape,  'y_test  : ',y_test.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 3.2 - Data normalization
+**Note :**
+ - All input data must be normalized, train and test.
+ - To do this we will **subtract the mean** and **divide by the standard deviation**.
+ - But test data should not be used in any way, even for normalization.
+ - The mean and the standard deviation will therefore only be calculated with the train data.
+
+%% Cell type:code id: tags:
+
+``` python
+display(x_train.describe().style.format("{0:.2f}").set_caption("Before normalization :"))
+
+mean = x_train.mean()
+std  = x_train.std()
+x_train = (x_train - mean) / std
+x_test  = (x_test  - mean) / std
+
+display(x_train.describe().style.format("{0:.2f}").set_caption("After normalization :"))
+display(x_train.head(5).style.format("{0:.2f}").set_caption("Few lines of the dataset :"))
+
+x_train, y_train = np.array(x_train), np.array(y_train)
+x_test,  y_test  = np.array(x_test),  np.array(y_test)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Build a model
+About informations about :
+ - [Optimizer](https://pytorch.org/docs/stable/optim.html)
+ - [Basic neural-network blocks](https://pytorch.org/docs/stable/nn.html)
+ - [Loss](https://pytorch.org/docs/stable/nn.html#loss-functions)
+
+%% Cell type:code id: tags:
+
+``` python
+class model_v1(nn.Module):
+    """
+    Basic fully connected neural-network for tabular data
+    """
+    def __init__(self,num_vars):
+        super(model_v1, self).__init__()
+        self.num_vars=num_vars
+        self.hidden1 = nn.Linear(self.num_vars, 64)
+        self.hidden2 = nn.Linear(64, 64)
+        self.hidden3 = nn.Linear(64, 1)
+
+    def forward(self, x):
+        x = x.view(-1,self.num_vars)   #flatten the observation before using fully-connected layers
+        x = self.hidden1(x)
+        x = F.relu(x)
+        x = self.hidden2(x)
+        x = F.relu(x)
+        x = self.hidden3(x)
+        return x
+
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Train the model
+
+### 5.1 - Stochastic gradient descent strategy to fit the model
+
+%% Cell type:code id: tags:
+
+``` python
+def fit(model,X_train,Y_train,X_test,Y_test, EPOCHS = 5, BATCH_SIZE = 32):
+
+    loss = nn.MSELoss()
+    optimizer = torch.optim.Adam(model.parameters(),lr=1e-3) #lr is the learning rate
+    model.train()
+
+    history=convergence_history_MSELoss()
+
+    history.update(model,X_train,Y_train,X_test,Y_test)
+
+    n=X_train.shape[0] #number of observations in the training data
+
+    #stochastic gradient descent
+    for epoch in range(EPOCHS):
+
+        batch_start=0
+        epoch_shuffler=np.arange(n)
+        np.random.shuffle(epoch_shuffler) #remark that 'utilsData.DataLoader' could be used instead
+
+        while batch_start+BATCH_SIZE < n:
+            #get mini-batch observation
+            mini_batch_observations = epoch_shuffler[batch_start:batch_start+BATCH_SIZE]
+            var_X_batch = Variable(X_train[mini_batch_observations,:]).float()
+            var_Y_batch = Variable(Y_train[mini_batch_observations]).float()
+
+            #gradient descent step
+            optimizer.zero_grad()               #set the parameters gradients to 0
+            Y_pred_batch = model(var_X_batch)   #predict y with the current NN parameters
+
+            curr_loss = loss(Y_pred_batch.view(-1), var_Y_batch.view(-1))  #compute the current loss
+            curr_loss.backward()                         #compute the loss gradient w.r.t. all NN parameters
+            optimizer.step()                             #update the NN parameters
+
+            #prepare the next mini-batch of the epoch
+            batch_start+=BATCH_SIZE
+
+        history.update(model,X_train,Y_train,X_test,Y_test)
+
+    return history
+```
+
+%% Cell type:markdown id: tags:
+
+#### 5.2 - Get the model
+
+%% Cell type:code id: tags:
+
+``` python
+
+
+model=model_v1( x_train[0,:].shape[0] )
+
+print(model)
+
+```
+
+%% Cell type:markdown id: tags:
+
+#### 5.3 - Train the model
+
+%% Cell type:code id: tags:
+
+``` python
+
+
+torch_x_train=torch.from_numpy(x_train)
+torch_y_train=torch.from_numpy(y_train)
+torch_x_test=torch.from_numpy(x_test)
+torch_y_test=torch.from_numpy(y_test)
+
+batch_size  = 10
+epochs      = 100
+
+
+history=fit(model,torch_x_train,torch_y_train,torch_x_test,torch_y_test,EPOCHS=epochs,BATCH_SIZE = batch_size)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Evaluate
+### 6.1 - Model evaluation
+MAE =  Mean Absolute Error (between the labels and predictions)
+A mae equal to 3 represents an average error in prediction of $3k.
+
+%% Cell type:code id: tags:
+
+``` python
+var_x_test = Variable(torch_x_test).float()
+var_y_test = Variable(torch_y_test).float()
+y_pred = model(var_x_test)
+
+nn_loss = nn.MSELoss()
+nn_MAE_loss = nn.L1Loss()
+
+print('x_test / loss      : {:5.4f}'.format(nn_loss(y_pred.view(-1), var_y_test.view(-1)).item()))
+print('x_test / mae       : {:5.4f}'.format(nn_MAE_loss(y_pred.view(-1), var_y_test.view(-1)).item()))
+```
+
+%% Cell type:markdown id: tags:
+
+### 6.2 - Training history
+What was the best result during our training ?
+
+%% Cell type:code id: tags:
+
+``` python
+
+df=pd.DataFrame(data=history.history)
+df.describe()
+```
+
+%% Cell type:code id: tags:
+
+``` python
+print("min( val_mae ) : {:.4f}".format( min(history.history["val_mae"]) ) )
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history(history, plot={'MAE' :['mae', 'val_mae'],
+                                'LOSS':['loss','val_loss']})
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Make a prediction
+The data must be normalized with the parameters (mean, std) previously used.
+
+%% Cell type:code id: tags:
+
+``` python
+my_data = [ 1.26425925, -0.48522739,  1.0436489 , -0.23112788,  1.37120745,
+       -2.14308942,  1.13489104, -1.06802005,  1.71189006,  1.57042287,
+        0.77859951,  0.14769795,  2.7585581 ]
+real_price = 10.4
+
+my_data=np.array(my_data).reshape(1,13)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+torch_my_data=torch.from_numpy(my_data)
+var_my_data = Variable(torch_my_data).float()
+
+predictions = model( var_my_data )
+print("Prediction : {:.2f} K$".format(predictions[0][0]))
+print("Reality    : {:.2f} K$".format(real_price))
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

BHPD.PyTorch/modules/fidle_pwk_additional.py

+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.autograd import Variable
+
+import numpy as np
+
+
+class convergence_history_CrossEntropyLoss:
+  def __init__(self):
+    """
+    Class to save the training converge properties
+    """
+    self.loss=nn.CrossEntropyLoss()
+    self.history={}                #Save convergence measures in the end of each epoch
+    self.history['loss']=[]        #value of the cost function on training data
+    self.history['accuracy']=[]         #percentage of  correctly classified instances on training data (if classification)
+    self.history['val_loss']=[]    #value of the cost function on validation data
+    self.history['val_accuracy']=[]     #percentage of  correctly classified instances on validation data (if classification)
+  
+  def update(self,current_model,xtrain,ytrain,xtest,ytest):
+    
+    #convergence information on the training data 
+    nb_training_obs=xtrain.shape[0]
+    if nb_training_obs>xtest.shape[0]:
+        nb_training_obs=xtest.shape[0]
+        
+    epoch_shuffler=np.arange(xtrain.shape[0]) 
+    np.random.shuffle(epoch_shuffler)
+    mini_batch_observations = epoch_shuffler[:nb_training_obs]
+    var_X_batch = Variable(xtrain[mini_batch_observations,:]).float()
+    var_y_batch = Variable(ytrain[mini_batch_observations])
+    y_pred_batch = current_model(var_X_batch)
+    curr_loss = self.loss(y_pred_batch, var_y_batch)
+
+    self.history['loss'].append(curr_loss.item())
+    self.history['accuracy'].append( float( (torch.argmax(y_pred_batch, dim= 1) == var_y_batch).float().mean()) )
+    
+    #convergence information on the test data 
+    var_X_batch = Variable(xtest[:,:]).float()
+    var_y_batch = Variable(ytest[:])
+    y_pred_batch = current_model(var_X_batch)
+    curr_loss = self.loss(y_pred_batch, var_y_batch)
+
+    self.history['val_loss'].append(curr_loss.item())
+    self.history['val_accuracy'].append( float( (torch.argmax(y_pred_batch, dim= 1) == var_y_batch).float().mean()) )
+
+
+
+class convergence_history_MSELoss:
+  def __init__(self):
+    """
+    Class to save the training converge properties
+    """
+    self.loss = nn.MSELoss()
+    self.MAE_loss = nn.L1Loss()
+    self.history={}                #Save convergence measures in the end of each epoch
+    self.history['loss']=[]        #value of the cost function on training data
+    self.history['mae']=[]         #mean absolute error on training data
+    self.history['val_loss']=[]    #value of the cost function on validation data
+    self.history['val_mae']=[]     #mean absolute error on validation data
+  
+  def update(self,current_model,xtrain,ytrain,xtest,ytest):
+    
+    #convergence information on the training data 
+    nb_training_obs=xtrain.shape[0]
+    if nb_training_obs>xtest.shape[0]:
+        nb_training_obs=xtest.shape[0]
+        
+    epoch_shuffler=np.arange(xtrain.shape[0]) 
+    np.random.shuffle(epoch_shuffler)
+    mini_batch_observations = epoch_shuffler[:nb_training_obs]
+    var_X_batch = Variable(xtrain[mini_batch_observations,:]).float()
+    var_y_batch = Variable(ytrain[mini_batch_observations]).float()
+    y_pred_batch = current_model(var_X_batch)
+    curr_loss = self.loss(y_pred_batch.view(-1), var_y_batch.view(-1))
+
+    self.history['loss'].append(curr_loss.item())
+    self.history['mae'].append(self.MAE_loss(y_pred_batch.view(-1), var_y_batch.view(-1)).item())
+    
+    #convergence information on the test data 
+    var_X_batch = Variable(xtest[:,:]).float()
+    var_y_batch = Variable(ytest[:]).float()
+    y_pred_batch = current_model(var_X_batch)
+    curr_loss = self.loss(y_pred_batch.view(-1), var_y_batch.view(-1))
+
+    self.history['val_loss'].append(curr_loss.item())
+    self.history['val_mae'].append(self.MAE_loss(y_pred_batch.view(-1), var_y_batch.view(-1)).item())

BHPD/01-DNN-Regression.ipynb

-%% Cell type:markdown id: tags:
-
-<!-- <div style="text-align: left"> -->
-
-<!-- </div> -->
-<img src="../fidle/img/00-Fidle-header-01.png" style="width:800px"/>
-
-Deep Neural Network (DNN) - BHPD dataset
-========================================
-
-A very simple and classic example of **regression** :
-
-## Objectives :
-Predicts **housing prices** from a set of house features.
-
-The **[Boston Housing Dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html)** consists of price of houses in various places in Boston.
-Alongside with price, the dataset also provide information such as Crime, areas of non-retail business in the town,
-age of people who own the house and many other attributes...
-
-What we're going to do:
-
- - Retrieve data
- - Preparing the data
- - Build a model
- - Train the model
- - Evaluate the result
-
-%% Cell type:markdown id: tags:
-
-## Step 1 - Import and init
-
-%% Cell type:code id: tags:
-
-``` python
-import tensorflow as tf
-from tensorflow import keras
-
-import numpy as np
-import matplotlib.pyplot as plt
-import pandas as pd
-import os,sys
-
-from IPython.display import display, Markdown
-from importlib import reload
-
-sys.path.append('..')
-import fidle.pwk as ooo
-
-ooo.init()
-os.makedirs('./run/models',   mode=0o750, exist_ok=True)
-```
-
-%% Output
-
-
-    
-    FIDLE 2020 - Practical Work Module
-    Version              : 0.2.9
-    Run time             : Monday 17 February 2020, 22:08:38
-    TensorFlow version   : 2.0.0
-    Keras version        : 2.2.4-tf
-
-%% Cell type:markdown id: tags:
-
-## Step 2 - Retrieve data
-
-### 2.1 - Option 1  : From Keras
-Boston housing is a famous historic dataset, so we can get it directly from [Keras datasets](https://www.tensorflow.org/api_docs/python/tf/keras/datasets)
-
-%% Cell type:raw id: tags:
-
-(x_train, y_train), (x_test, y_test) = keras.datasets.boston_housing.load_data(test_split=0.2, seed=113)
-
-%% Cell type:markdown id: tags:
-
-### 2.2 - Option 2 : From a csv file
-More fun !
-
-%% Cell type:code id: tags:
-
-``` python
-data = pd.read_csv('./data/BostonHousing.csv', header=0)
-
-display(data.head(5).style.format("{0:.2f}"))
-print('Données manquantes : ',data.isna().sum().sum(), '  Shape is : ', data.shape)
-```
-
-%% Output
-
-
-    Données manquantes :  0   Shape is :  (506, 14)
-
-%% Cell type:markdown id: tags:
-
-## Step 3 - Preparing the data
-### 3.1 - Split data
-We will use 70% of the data for training and 30% for validation.
-x will be input data and y the expected output
-
-%% Cell type:code id: tags:
-
-``` python
-# ---- Split => train, test
-#
-data_train = data.sample(frac=0.7, axis=0)
-data_test  = data.drop(data_train.index)
-
-# ---- Split => x,y (medv is price)
-#
-x_train = data_train.drop('medv',  axis=1)
-y_train = data_train['medv']
-x_test  = data_test.drop('medv',   axis=1)
-y_test  = data_test['medv']
-
-print('Original data shape was : ',data.shape)
-print('x_train : ',x_train.shape, 'y_train : ',y_train.shape)
-print('x_test  : ',x_test.shape,  'y_test  : ',y_test.shape)
-```
-
-%% Output
-
-    Original data shape was :  (506, 14)
-    x_train :  (354, 13) y_train :  (354,)
-    x_test  :  (152, 13) y_test  :  (152,)
-
-%% Cell type:markdown id: tags:
-
-### 3.2 - Data normalization
-**Note :**
- - All input data must be normalized, train and test.
- - To do this we will **subtract the mean** and **divide by the standard deviation**.
- - But test data should not be used in any way, even for normalization.
- - The mean and the standard deviation will therefore only be calculated with the train data.
-
-%% Cell type:code id: tags:
-
-``` python
-display(x_train.describe().style.format("{0:.2f}").set_caption("Before normalization :"))
-
-mean = x_train.mean()
-std  = x_train.std()
-x_train = (x_train - mean) / std
-x_test  = (x_test  - mean) / std
-
-display(x_train.describe().style.format("{0:.2f}").set_caption("After normalization :"))
-
-x_train, y_train = np.array(x_train), np.array(y_train)
-x_test,  y_test  = np.array(x_test),  np.array(y_test)
-```
-
-%% Output
-
-
-
-%% Cell type:markdown id: tags:
-
-## Step 4 - Build a model
-About informations about :
- - [Optimizer](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers)
- - [Activation](https://www.tensorflow.org/api_docs/python/tf/keras/activations)
- - [Loss](https://www.tensorflow.org/api_docs/python/tf/keras/losses)
- - [Metrics](https://www.tensorflow.org/api_docs/python/tf/keras/metrics)
-
-%% Cell type:code id: tags:
-
-``` python
-  def get_model_v1(shape):
-
-    model = keras.models.Sequential()
-    model.add(keras.layers.Input(shape, name="InputLayer"))
-    model.add(keras.layers.Dense(64, activation='relu', name='Dense_n1'))
-    model.add(keras.layers.Dense(64, activation='relu', name='Dense_n2'))
-    model.add(keras.layers.Dense(1, name='Output'))
-
-    model.compile(optimizer = 'rmsprop',
-                  loss      = 'mse',
-                  metrics   = ['mae', 'mse'] )
-    return model
-```
-
-%% Cell type:markdown id: tags:
-
-## Step 5 - Train the model
-### 5.1 - Get it
-
-%% Cell type:code id: tags:
-
-``` python
-model=get_model_v1( (13,) )
-
-model.summary()
-keras.utils.plot_model( model, to_file='./run/model.png', show_shapes=True, show_layer_names=True, dpi=96)
-```
-
-%% Output
-
-    Model: "sequential"
-    _________________________________________________________________
-    Layer (type)                 Output Shape              Param #
-    =================================================================
-    Dense_n1 (Dense)             (None, 64)                896
-    _________________________________________________________________
-    Dense_n2 (Dense)             (None, 64)                4160
-    _________________________________________________________________
-    Output (Dense)               (None, 1)                 65
-    =================================================================
-    Total params: 5,121
-    Trainable params: 5,121
-    Non-trainable params: 0
-    _________________________________________________________________
-
-
-    <IPython.core.display.Image object>
-
-%% Cell type:markdown id: tags:
-
-### 5.2 - Train it
-
-%% Cell type:code id: tags:
-
-``` python
-history = model.fit(x_train,
-                    y_train,
-                    epochs          = 100,
-                    batch_size      = 10,
-                    verbose         = 1,
-                    validation_data = (x_test, y_test))
-```
-
-%% Output
-
-    Train on 354 samples, validate on 152 samples
-    Epoch 1/100
-    354/354 [==============================] - 1s 2ms/sample - loss: 414.5603 - mae: 18.2577 - mse: 414.5602 - val_loss: 266.3728 - val_mae: 13.9913 - val_mse: 266.3728
-    Epoch 2/100
-    354/354 [==============================] - 0s 190us/sample - loss: 165.4507 - mae: 10.4618 - mse: 165.4507 - val_loss: 74.4125 - val_mae: 6.0372 - val_mse: 74.4125
-    Epoch 3/100
-    354/354 [==============================] - 0s 187us/sample - loss: 54.2313 - mae: 5.3763 - mse: 54.2313 - val_loss: 47.0203 - val_mae: 4.7399 - val_mse: 47.0203
-    Epoch 4/100
-    354/354 [==============================] - 0s 166us/sample - loss: 32.3303 - mae: 4.2632 - mse: 32.3303 - val_loss: 38.0120 - val_mae: 4.2484 - val_mse: 38.0120
-    Epoch 5/100
-    354/354 [==============================] - 0s 153us/sample - loss: 25.3763 - mae: 3.7745 - mse: 25.3763 - val_loss: 32.4707 - val_mae: 3.8465 - val_mse: 32.4707
-    Epoch 6/100
-    354/354 [==============================] - 0s 153us/sample - loss: 22.2331 - mae: 3.4720 - mse: 22.2331 - val_loss: 29.6142 - val_mae: 3.4844 - val_mse: 29.6142
-    Epoch 7/100
-    354/354 [==============================] - 0s 154us/sample - loss: 19.7834 - mae: 3.2245 - mse: 19.7834 - val_loss: 27.1649 - val_mae: 3.5465 - val_mse: 27.1649
-    Epoch 8/100
-    354/354 [==============================] - 0s 155us/sample - loss: 18.0991 - mae: 3.0669 - mse: 18.0991 - val_loss: 26.0093 - val_mae: 3.5617 - val_mse: 26.0093
-    Epoch 9/100
-    354/354 [==============================] - 0s 161us/sample - loss: 16.9247 - mae: 2.9184 - mse: 16.9247 - val_loss: 23.2549 - val_mae: 3.3243 - val_mse: 23.2549
-    Epoch 10/100
-    354/354 [==============================] - 0s 150us/sample - loss: 16.0827 - mae: 2.8116 - mse: 16.0827 - val_loss: 21.1365 - val_mae: 3.0248 - val_mse: 21.1365
-    Epoch 11/100
-    354/354 [==============================] - 0s 170us/sample - loss: 15.0334 - mae: 2.7214 - mse: 15.0334 - val_loss: 20.0163 - val_mae: 2.9800 - val_mse: 20.0163
-    Epoch 12/100
-    354/354 [==============================] - 0s 180us/sample - loss: 14.4011 - mae: 2.6949 - mse: 14.4011 - val_loss: 19.8958 - val_mae: 2.9262 - val_mse: 19.8958
-    Epoch 13/100
-    354/354 [==============================] - 0s 184us/sample - loss: 13.9168 - mae: 2.5674 - mse: 13.9168 - val_loss: 18.5729 - val_mae: 2.7302 - val_mse: 18.5729
-    Epoch 14/100
-    354/354 [==============================] - 0s 161us/sample - loss: 13.5575 - mae: 2.5442 - mse: 13.5575 - val_loss: 17.8812 - val_mae: 2.6748 - val_mse: 17.8812
-    Epoch 15/100
-    354/354 [==============================] - 0s 166us/sample - loss: 12.8689 - mae: 2.4779 - mse: 12.8689 - val_loss: 18.9649 - val_mae: 2.7560 - val_mse: 18.9649
-    Epoch 16/100
-    354/354 [==============================] - 0s 159us/sample - loss: 12.6470 - mae: 2.4670 - mse: 12.6470 - val_loss: 16.5834 - val_mae: 2.6016 - val_mse: 16.5834
-    Epoch 17/100
-    354/354 [==============================] - 0s 159us/sample - loss: 12.3566 - mae: 2.4280 - mse: 12.3566 - val_loss: 16.7371 - val_mae: 2.6670 - val_mse: 16.7371
-    Epoch 18/100
-    354/354 [==============================] - 0s 158us/sample - loss: 12.3328 - mae: 2.4060 - mse: 12.3328 - val_loss: 16.3754 - val_mae: 2.6027 - val_mse: 16.3754
-    Epoch 19/100
-    354/354 [==============================] - 0s 152us/sample - loss: 11.8357 - mae: 2.3106 - mse: 11.8357 - val_loss: 16.1015 - val_mae: 2.6255 - val_mse: 16.1015
-    Epoch 20/100
-    354/354 [==============================] - 0s 163us/sample - loss: 11.6722 - mae: 2.3482 - mse: 11.6722 - val_loss: 16.1405 - val_mae: 2.6889 - val_mse: 16.1405
-    Epoch 21/100
-    354/354 [==============================] - 0s 175us/sample - loss: 11.2774 - mae: 2.3344 - mse: 11.2774 - val_loss: 15.2110 - val_mae: 2.5038 - val_mse: 15.2110
-    Epoch 22/100
-    354/354 [==============================] - 0s 180us/sample - loss: 11.2491 - mae: 2.3055 - mse: 11.2491 - val_loss: 15.4745 - val_mae: 2.4494 - val_mse: 15.4744
-    Epoch 23/100
-    354/354 [==============================] - 0s 187us/sample - loss: 10.9102 - mae: 2.2171 - mse: 10.9102 - val_loss: 15.1145 - val_mae: 2.4282 - val_mse: 15.1145
-    Epoch 24/100
-    354/354 [==============================] - 0s 168us/sample - loss: 10.7952 - mae: 2.2533 - mse: 10.7952 - val_loss: 14.3789 - val_mae: 2.3683 - val_mse: 14.3789
-    Epoch 25/100
-    354/354 [==============================] - 0s 171us/sample - loss: 10.7250 - mae: 2.2489 - mse: 10.7250 - val_loss: 15.1102 - val_mae: 2.3422 - val_mse: 15.1102
-    Epoch 26/100
-    354/354 [==============================] - 0s 158us/sample - loss: 10.4010 - mae: 2.1702 - mse: 10.4010 - val_loss: 14.3260 - val_mae: 2.3176 - val_mse: 14.3260
-    Epoch 27/100
-    354/354 [==============================] - 0s 149us/sample - loss: 10.1442 - mae: 2.1797 - mse: 10.1442 - val_loss: 13.6694 - val_mae: 2.3864 - val_mse: 13.6694
-    Epoch 28/100
-    354/354 [==============================] - 0s 168us/sample - loss: 10.1391 - mae: 2.1809 - mse: 10.1391 - val_loss: 14.0177 - val_mae: 2.3467 - val_mse: 14.0177
-    Epoch 29/100
-    354/354 [==============================] - 0s 149us/sample - loss: 9.9119 - mae: 2.1267 - mse: 9.9119 - val_loss: 14.0739 - val_mae: 2.4617 - val_mse: 14.0739
-    Epoch 30/100
-    354/354 [==============================] - 0s 164us/sample - loss: 10.0176 - mae: 2.1669 - mse: 10.0176 - val_loss: 13.5116 - val_mae: 2.3158 - val_mse: 13.5116
-    Epoch 31/100
-    354/354 [==============================] - 0s 189us/sample - loss: 9.8259 - mae: 2.1407 - mse: 9.8259 - val_loss: 13.7364 - val_mae: 2.3531 - val_mse: 13.7364
-    Epoch 32/100
-    354/354 [==============================] - 0s 178us/sample - loss: 9.4495 - mae: 2.0922 - mse: 9.4495 - val_loss: 14.1936 - val_mae: 2.3887 - val_mse: 14.1936
-    Epoch 33/100
-    354/354 [==============================] - 0s 164us/sample - loss: 9.6721 - mae: 2.0870 - mse: 9.6721 - val_loss: 13.4267 - val_mae: 2.3508 - val_mse: 13.4267
-    Epoch 34/100
-    354/354 [==============================] - 0s 167us/sample - loss: 9.1042 - mae: 2.0644 - mse: 9.1042 - val_loss: 13.3821 - val_mae: 2.4709 - val_mse: 13.3821
-    Epoch 35/100
-    354/354 [==============================] - 0s 155us/sample - loss: 9.0129 - mae: 2.0482 - mse: 9.0129 - val_loss: 14.2184 - val_mae: 2.2754 - val_mse: 14.2184
-    Epoch 36/100
-    354/354 [==============================] - 0s 160us/sample - loss: 9.2470 - mae: 2.0661 - mse: 9.2470 - val_loss: 14.3466 - val_mae: 2.5561 - val_mse: 14.3466
-    Epoch 37/100
-    354/354 [==============================] - 0s 169us/sample - loss: 9.1695 - mae: 2.0766 - mse: 9.1695 - val_loss: 13.3818 - val_mae: 2.2373 - val_mse: 13.3818
-    Epoch 38/100
-    354/354 [==============================] - 0s 165us/sample - loss: 9.1663 - mae: 2.0617 - mse: 9.1663 - val_loss: 14.7461 - val_mae: 2.5061 - val_mse: 14.7461
-    Epoch 39/100
-    354/354 [==============================] - 0s 159us/sample - loss: 8.7273 - mae: 2.0208 - mse: 8.7273 - val_loss: 12.5890 - val_mae: 2.3037 - val_mse: 12.5890
-    Epoch 40/100
-    354/354 [==============================] - 0s 166us/sample - loss: 8.9038 - mae: 2.0352 - mse: 8.9038 - val_loss: 12.9754 - val_mae: 2.2079 - val_mse: 12.9754
-    Epoch 41/100
-    354/354 [==============================] - 0s 153us/sample - loss: 8.6155 - mae: 2.0267 - mse: 8.6155 - val_loss: 13.9239 - val_mae: 2.3525 - val_mse: 13.9239
-    Epoch 42/100
-    354/354 [==============================] - 0s 163us/sample - loss: 8.5479 - mae: 2.0170 - mse: 8.5479 - val_loss: 13.6362 - val_mae: 2.2694 - val_mse: 13.6362
-    Epoch 43/100
-    354/354 [==============================] - 0s 165us/sample - loss: 8.7087 - mae: 2.0062 - mse: 8.7087 - val_loss: 13.1138 - val_mae: 2.2386 - val_mse: 13.1138
-    Epoch 44/100
-    354/354 [==============================] - 0s 160us/sample - loss: 8.3942 - mae: 1.9622 - mse: 8.3942 - val_loss: 12.3461 - val_mae: 2.2337 - val_mse: 12.3461
-    Epoch 45/100
-    354/354 [==============================] - 0s 168us/sample - loss: 8.4101 - mae: 2.0098 - mse: 8.4101 - val_loss: 13.2116 - val_mae: 2.2682 - val_mse: 13.2116
-    Epoch 46/100
-    354/354 [==============================] - 0s 156us/sample - loss: 8.3264 - mae: 1.9483 - mse: 8.3264 - val_loss: 12.5519 - val_mae: 2.4063 - val_mse: 12.5519
-    Epoch 47/100
-    354/354 [==============================] - 0s 158us/sample - loss: 8.1445 - mae: 1.9549 - mse: 8.1445 - val_loss: 12.1838 - val_mae: 2.2591 - val_mse: 12.1838
-    Epoch 48/100
-    354/354 [==============================] - 0s 156us/sample - loss: 8.0389 - mae: 1.9304 - mse: 8.0389 - val_loss: 12.6978 - val_mae: 2.1907 - val_mse: 12.6978
-    Epoch 49/100
-    354/354 [==============================] - 0s 164us/sample - loss: 8.0705 - mae: 1.9493 - mse: 8.0705 - val_loss: 12.4833 - val_mae: 2.4720 - val_mse: 12.4833
-    Epoch 50/100
-    354/354 [==============================] - 0s 158us/sample - loss: 8.1872 - mae: 1.9630 - mse: 8.1872 - val_loss: 12.0043 - val_mae: 2.2610 - val_mse: 12.0043
-    Epoch 51/100
-    354/354 [==============================] - 0s 158us/sample - loss: 8.0357 - mae: 1.8946 - mse: 8.0357 - val_loss: 11.3982 - val_mae: 2.1770 - val_mse: 11.3982
-    Epoch 52/100
-    354/354 [==============================] - 0s 162us/sample - loss: 7.6882 - mae: 1.8951 - mse: 7.6882 - val_loss: 13.0714 - val_mae: 2.4109 - val_mse: 13.0714
-    Epoch 53/100
-    354/354 [==============================] - 0s 162us/sample - loss: 7.9639 - mae: 1.9103 - mse: 7.9639 - val_loss: 12.4297 - val_mae: 2.2996 - val_mse: 12.4297
-    Epoch 54/100
-    354/354 [==============================] - 0s 183us/sample - loss: 7.7929 - mae: 1.8971 - mse: 7.7929 - val_loss: 11.9751 - val_mae: 2.2491 - val_mse: 11.9751
-    Epoch 55/100
-    354/354 [==============================] - 0s 185us/sample - loss: 7.4411 - mae: 1.8631 - mse: 7.4411 - val_loss: 11.3761 - val_mae: 2.3416 - val_mse: 11.3761
-    Epoch 56/100
-    354/354 [==============================] - 0s 186us/sample - loss: 7.6105 - mae: 1.9111 - mse: 7.6105 - val_loss: 12.4939 - val_mae: 2.4095 - val_mse: 12.4939
-    Epoch 57/100
-    354/354 [==============================] - 0s 190us/sample - loss: 7.5013 - mae: 1.9146 - mse: 7.5013 - val_loss: 11.6668 - val_mae: 2.1468 - val_mse: 11.6668
-    Epoch 58/100
-    354/354 [==============================] - 0s 195us/sample - loss: 7.4096 - mae: 1.8515 - mse: 7.4096 - val_loss: 13.8000 - val_mae: 2.5222 - val_mse: 13.8000
-    Epoch 59/100
-    354/354 [==============================] - 0s 180us/sample - loss: 7.2263 - mae: 1.8241 - mse: 7.2263 - val_loss: 10.8964 - val_mae: 2.2130 - val_mse: 10.8964
-    Epoch 60/100
-    354/354 [==============================] - 0s 161us/sample - loss: 7.1773 - mae: 1.8526 - mse: 7.1773 - val_loss: 10.7862 - val_mae: 2.1088 - val_mse: 10.7862
-    Epoch 61/100
-    354/354 [==============================] - 0s 165us/sample - loss: 7.0812 - mae: 1.8308 - mse: 7.0812 - val_loss: 10.8147 - val_mae: 2.3209 - val_mse: 10.8147
-    Epoch 62/100
-    354/354 [==============================] - 0s 155us/sample - loss: 7.2235 - mae: 1.8367 - mse: 7.2235 - val_loss: 11.0399 - val_mae: 2.2583 - val_mse: 11.0399
-    Epoch 63/100
-    354/354 [==============================] - 0s 155us/sample - loss: 7.0341 - mae: 1.8172 - mse: 7.0341 - val_loss: 10.9894 - val_mae: 2.1429 - val_mse: 10.9894
-    Epoch 64/100
-    354/354 [==============================] - 0s 157us/sample - loss: 6.8729 - mae: 1.7492 - mse: 6.8729 - val_loss: 10.5465 - val_mae: 2.1532 - val_mse: 10.5465
-    Epoch 65/100
-    354/354 [==============================] - 0s 164us/sample - loss: 6.9345 - mae: 1.7837 - mse: 6.9345 - val_loss: 11.5379 - val_mae: 2.1963 - val_mse: 11.5379
-    Epoch 66/100
-    354/354 [==============================] - 0s 166us/sample - loss: 6.8218 - mae: 1.7714 - mse: 6.8218 - val_loss: 10.1486 - val_mae: 2.1617 - val_mse: 10.1486
-    Epoch 67/100
-    354/354 [==============================] - 0s 157us/sample - loss: 6.8711 - mae: 1.8045 - mse: 6.8711 - val_loss: 10.3196 - val_mae: 2.2297 - val_mse: 10.3196
-    Epoch 68/100
-    354/354 [==============================] - 0s 162us/sample - loss: 6.7281 - mae: 1.7762 - mse: 6.7281 - val_loss: 11.2361 - val_mae: 2.2046 - val_mse: 11.2361
-    Epoch 69/100
-    354/354 [==============================] - 0s 158us/sample - loss: 6.5518 - mae: 1.7292 - mse: 6.5518 - val_loss: 10.2378 - val_mae: 2.1494 - val_mse: 10.2378
-    Epoch 70/100
-    354/354 [==============================] - 0s 161us/sample - loss: 6.6489 - mae: 1.7383 - mse: 6.6489 - val_loss: 11.1613 - val_mae: 2.2212 - val_mse: 11.1613
-    Epoch 71/100
-    354/354 [==============================] - 0s 176us/sample - loss: 6.5827 - mae: 1.7564 - mse: 6.5827 - val_loss: 10.0177 - val_mae: 2.2440 - val_mse: 10.0177
-    Epoch 72/100
-    354/354 [==============================] - 0s 168us/sample - loss: 6.3411 - mae: 1.7463 - mse: 6.3411 - val_loss: 10.7929 - val_mae: 2.1946 - val_mse: 10.7929
-    Epoch 73/100
-    354/354 [==============================] - 0s 163us/sample - loss: 6.3621 - mae: 1.7466 - mse: 6.3621 - val_loss: 9.7344 - val_mae: 2.1441 - val_mse: 9.7344
-    Epoch 74/100
-    354/354 [==============================] - 0s 158us/sample - loss: 6.2298 - mae: 1.7411 - mse: 6.2298 - val_loss: 11.2495 - val_mae: 2.1948 - val_mse: 11.2495
-    Epoch 75/100
-    354/354 [==============================] - 0s 159us/sample - loss: 6.3037 - mae: 1.7169 - mse: 6.3037 - val_loss: 10.1339 - val_mae: 2.1716 - val_mse: 10.1339
-    Epoch 76/100
-    354/354 [==============================] - 0s 158us/sample - loss: 6.0780 - mae: 1.6686 - mse: 6.0780 - val_loss: 11.9975 - val_mae: 2.3317 - val_mse: 11.9975
-    Epoch 77/100
-    354/354 [==============================] - 0s 165us/sample - loss: 6.3311 - mae: 1.7082 - mse: 6.3311 - val_loss: 11.6433 - val_mae: 2.2756 - val_mse: 11.6433
-    Epoch 78/100
-    354/354 [==============================] - 0s 155us/sample - loss: 6.0620 - mae: 1.6765 - mse: 6.0620 - val_loss: 13.0159 - val_mae: 2.5073 - val_mse: 13.0159
-    Epoch 79/100
-    354/354 [==============================] - 0s 167us/sample - loss: 6.1819 - mae: 1.7157 - mse: 6.1819 - val_loss: 10.1000 - val_mae: 2.1462 - val_mse: 10.1000
-    Epoch 80/100
-    354/354 [==============================] - 0s 158us/sample - loss: 5.9085 - mae: 1.6720 - mse: 5.9085 - val_loss: 11.7867 - val_mae: 2.5045 - val_mse: 11.7866
-    Epoch 81/100
-    354/354 [==============================] - 0s 168us/sample - loss: 6.0201 - mae: 1.6678 - mse: 6.0201 - val_loss: 10.8789 - val_mae: 2.3031 - val_mse: 10.8789
-    Epoch 82/100
-    354/354 [==============================] - 0s 159us/sample - loss: 6.1278 - mae: 1.6799 - mse: 6.1278 - val_loss: 9.8114 - val_mae: 2.1048 - val_mse: 9.8114
-    Epoch 83/100
-    354/354 [==============================] - 0s 150us/sample - loss: 5.6372 - mae: 1.6280 - mse: 5.6372 - val_loss: 10.0971 - val_mae: 2.1464 - val_mse: 10.0971
-    Epoch 84/100
-    354/354 [==============================] - 0s 153us/sample - loss: 5.9587 - mae: 1.6421 - mse: 5.9587 - val_loss: 9.4731 - val_mae: 2.1915 - val_mse: 9.4731
-    Epoch 85/100
-    354/354 [==============================] - 0s 158us/sample - loss: 5.6189 - mae: 1.6223 - mse: 5.6189 - val_loss: 9.9788 - val_mae: 2.3332 - val_mse: 9.9788
-    Epoch 86/100
-    354/354 [==============================] - 0s 158us/sample - loss: 5.8193 - mae: 1.6930 - mse: 5.8193 - val_loss: 10.4070 - val_mae: 2.1490 - val_mse: 10.4070
-    Epoch 87/100
-    354/354 [==============================] - 0s 155us/sample - loss: 5.5919 - mae: 1.6152 - mse: 5.5919 - val_loss: 9.9985 - val_mae: 2.2546 - val_mse: 9.9985
-    Epoch 88/100
-    354/354 [==============================] - 0s 160us/sample - loss: 5.6652 - mae: 1.6246 - mse: 5.6652 - val_loss: 9.1506 - val_mae: 2.0642 - val_mse: 9.1506
-    Epoch 89/100
-    354/354 [==============================] - 0s 157us/sample - loss: 5.6349 - mae: 1.6108 - mse: 5.6349 - val_loss: 9.8522 - val_mae: 2.0813 - val_mse: 9.8522
-    Epoch 90/100
-    354/354 [==============================] - 0s 159us/sample - loss: 5.6165 - mae: 1.6449 - mse: 5.6165 - val_loss: 9.1553 - val_mae: 2.0421 - val_mse: 9.1553
-    Epoch 91/100
-    354/354 [==============================] - 0s 161us/sample - loss: 5.5416 - mae: 1.6153 - mse: 5.5416 - val_loss: 10.4231 - val_mae: 2.2880 - val_mse: 10.4231
-    Epoch 92/100
-    354/354 [==============================] - 0s 158us/sample - loss: 5.3909 - mae: 1.5863 - mse: 5.3909 - val_loss: 8.8087 - val_mae: 2.1022 - val_mse: 8.8087
-    Epoch 93/100
-    354/354 [==============================] - 0s 155us/sample - loss: 5.3540 - mae: 1.5986 - mse: 5.3540 - val_loss: 9.6963 - val_mae: 2.1931 - val_mse: 9.6963
-    Epoch 94/100
-    354/354 [==============================] - 0s 161us/sample - loss: 5.3198 - mae: 1.6074 - mse: 5.3198 - val_loss: 9.1875 - val_mae: 2.1917 - val_mse: 9.1875
-    Epoch 95/100
-    354/354 [==============================] - 0s 165us/sample - loss: 5.2299 - mae: 1.5638 - mse: 5.2299 - val_loss: 8.8746 - val_mae: 2.1273 - val_mse: 8.8746
-    Epoch 96/100
-    354/354 [==============================] - 0s 163us/sample - loss: 5.2789 - mae: 1.5651 - mse: 5.2789 - val_loss: 9.7351 - val_mae: 2.2359 - val_mse: 9.7351
-    Epoch 97/100
-    354/354 [==============================] - 0s 153us/sample - loss: 5.3399 - mae: 1.6002 - mse: 5.3399 - val_loss: 9.7185 - val_mae: 2.1080 - val_mse: 9.7185
-    Epoch 98/100
-    354/354 [==============================] - 0s 159us/sample - loss: 5.0072 - mae: 1.5055 - mse: 5.0072 - val_loss: 8.3621 - val_mae: 2.0586 - val_mse: 8.3621
-    Epoch 99/100
-    354/354 [==============================] - 0s 156us/sample - loss: 5.2596 - mae: 1.5557 - mse: 5.2596 - val_loss: 8.6406 - val_mae: 2.0527 - val_mse: 8.6406
-    Epoch 100/100
-    354/354 [==============================] - 0s 159us/sample - loss: 5.0983 - mae: 1.5543 - mse: 5.0983 - val_loss: 8.4836 - val_mae: 2.0234 - val_mse: 8.4836
-
-%% Cell type:markdown id: tags:
-
-## Step 6 - Evaluate
-### 6.1 - Model evaluation
-MAE =  Mean Absolute Error (between the labels and predictions)
-A mae equal to 3 represents an average error in prediction of $3k.
-
-%% Cell type:code id: tags:
-
-``` python
-score = model.evaluate(x_test, y_test, verbose=0)
-
-print('x_test / loss      : {:5.4f}'.format(score[0]))
-print('x_test / mae       : {:5.4f}'.format(score[1]))
-print('x_test / mse       : {:5.4f}'.format(score[2]))
-```
-
-%% Output
-
-    x_test / loss      : 8.4836
-    x_test / mae       : 2.0234
-    x_test / mse       : 8.4836
-
-%% Cell type:markdown id: tags:
-
-### 6.2 - Training history
-What was the best result during our training ?
-
-%% Cell type:code id: tags:
-
-``` python
-
-df=pd.DataFrame(data=history.history)
-df.describe()
-```
-
-%% Output
-
-                 loss         mae         mse    val_loss     val_mae     val_mse
-    count  100.000000  100.000000  100.000000  100.000000  100.000000  100.000000
-    mean    15.144930    2.312168   15.144930   17.019036    2.582618   17.019036
-    std     43.707091    1.906713   43.707090   26.587745    1.288267   26.587746
-    min      5.007155    1.505515    5.007155    8.362053    2.023406    8.362053
-    25%      6.285225    1.716563    6.285225   10.419040    2.192718   10.419040
-    50%      8.037316    1.922454    8.037317   12.488579    2.301342   12.488580
-    75%     10.482029    2.189933   10.482029   14.470699    2.503943   14.470701
-    max    414.560260   18.257650  414.560242  266.372801   13.991282  266.372803
-
-%% Cell type:code id: tags:
-
-``` python
-print("min( val_mae ) : {:.4f}".format( min(history.history["val_mae"]) ) )
-```
-
-%% Output
-
-    min( val_mae ) : 2.0234
-
-%% Cell type:code id: tags:
-
-``` python
-ooo.plot_history(history, plot={'MSE' :['mse', 'val_mse'],
-                                'MAE' :['mae', 'val_mae'],
-                                'LOSS':['loss','val_loss']})
-```
-
-%% Output
-
-
-
-
-
-
-
-%% Cell type:markdown id: tags:
-
-## Step 7 - Make a prediction
-
-%% Cell type:code id: tags:
-
-``` python
-my_data = [ 1.26425925, -0.48522739,  1.0436489 , -0.23112788,  1.37120745,
-       -2.14308942,  1.13489104, -1.06802005,  1.71189006,  1.57042287,
-        0.77859951,  0.14769795,  2.7585581 ]
-real_price = 10.4
-
-my_data=np.array(my_data).reshape(1,13)
-```
-
-%% Cell type:code id: tags:
-
-``` python
-
-predictions = model.predict( my_data )
-print("Prédiction : {:.2f} K$".format(predictions[0][0]))
-print("Reality    : {:.2f} K$".format(real_price))
-```
-
-%% Output
-
-    Prédiction : 11.59 K$
-    Reality    : 10.40 K$
-
-%% Cell type:markdown id: tags:
-
----
-<div style="text-align: left">
-    <img src="../fidle/img/00-Fidle-logo-01.svg" style="width:80px"/>
-</div>

BHPD/02-DNN-Regression Premium.ipynb

-%% Cell type:markdown id: tags:
-
-Deep Neural Network (DNN) - BHPD dataset
-========================================
----
-Introduction au Deep Learning  (IDLE) - S. Arias, E. Maldonado, JL. Parouty - CNRS/SARI/DEVLOG - 2020
-
-## A very simple example of **regression** (Premium edition):
-
-Objective is to predicts **housing prices** from a set of house features.
-
-The **[Boston Housing Dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html)** consists of price of houses in various places in Boston.
-Alongside with price, the dataset also provide information such as Crime, areas of non-retail business in the town,
-age of people who own the house and many other attributes...
-
-What we're going to do:
-
- - (Retrieve data)
- - (Preparing the data)
- - (Build a model)
- - Train and save the model
- - Restore saved model
- - Evaluate the model
- - Make some predictions
-
-%% Cell type:markdown id: tags:
-
-## Step 1 - Import and init
-
-%% Cell type:code id: tags:
-
-``` python
-import tensorflow as tf
-from tensorflow import keras
-
-import numpy as np
-import matplotlib.pyplot as plt
-import pandas as pd
-import os,sys
-
-from IPython.display import display, Markdown
-from importlib import reload
-
-sys.path.append('..')
-import fidle.pwk as ooo
-
-ooo.init()
-os.makedirs('./run/models',   mode=0o750, exist_ok=True)
-```
-
-%% Output
-
-
-    
-    FIDLE 2020 - Practical Work Module
-    Version              : 0.2.8
-    Run time             : Saturday 15 February 2020, 12:32:05
-    TensorFlow version   : 2.0.0
-    Keras version        : 2.2.4-tf
-
-%% Cell type:markdown id: tags:
-
-## Step 2 - Retrieve data
-
-### 2.1 - Option 1  : From Keras
-Boston housing is a famous historic dataset, so we can get it directly from [Keras datasets](https://www.tensorflow.org/api_docs/python/tf/keras/datasets)
-
-%% Cell type:raw id: tags:
-
-(x_train, y_train), (x_test, y_test) = keras.datasets.boston_housing.load_data(test_split=0.2, seed=113)
-
-%% Cell type:markdown id: tags:
-
-### 2.2 - Option 2 : From a csv file
-More fun !
-
-%% Cell type:code id: tags:
-
-``` python
-data = pd.read_csv('./data/BostonHousing.csv', header=0)
-
-display(data.head(5).style.format("{0:.2f}"))
-print('Données manquantes : ',data.isna().sum().sum(), '  Shape is : ', data.shape)
-```
-
-%% Output
-
-
-    Données manquantes :  0   Shape is :  (506, 14)
-
-%% Cell type:markdown id: tags:
-
-## Step 3 - Preparing the data
-### 3.1 - Split data
-We will use 80% of the data for training and 20% for validation.
-x will be input data and y the expected output
-
-%% Cell type:code id: tags:
-
-``` python
-# ---- Split => train, test
-#
-data_train = data.sample(frac=0.7, axis=0)
-data_test  = data.drop(data_train.index)
-
-# ---- Split => x,y (medv is price)
-#
-x_train = data_train.drop('medv',  axis=1)
-y_train = data_train['medv']
-x_test  = data_test.drop('medv',   axis=1)
-y_test  = data_test['medv']
-
-print('Original data shape was : ',data.shape)
-print('x_train : ',x_train.shape, 'y_train : ',y_train.shape)
-print('x_test  : ',x_test.shape,  'y_test  : ',y_test.shape)
-```
-
-%% Output
-
-    Original data shape was :  (506, 14)
-    x_train :  (354, 13) y_train :  (354,)
-    x_test  :  (152, 13) y_test  :  (152,)
-
-%% Cell type:markdown id: tags:
-
-### 3.2 - Data normalization
-**Note :**
- - All input data must be normalized, train and test.
- - To do this we will subtract the mean and divide by the standard deviation.
- - But test data should not be used in any way, even for normalization.
- - The mean and the standard deviation will therefore only be calculated with the train data.
-
-%% Cell type:code id: tags:
-
-``` python
-display(x_train.describe().style.format("{0:.2f}").set_caption("Before normalization :"))
-
-mean = x_train.mean()
-std  = x_train.std()
-x_train = (x_train - mean) / std
-x_test  = (x_test  - mean) / std
-
-display(x_train.describe().style.format("{0:.2f}").set_caption("After normalization :"))
-
-x_train, y_train = np.array(x_train), np.array(y_train)
-x_test,  y_test  = np.array(x_test),  np.array(y_test)
-```
-
-%% Output
-
-
-
-%% Cell type:markdown id: tags:
-
-## Step 4 - Build a model
-More informations about :
- - [Optimizer](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers)
- - [Activation](https://www.tensorflow.org/api_docs/python/tf/keras/activations)
- - [Loss](https://www.tensorflow.org/api_docs/python/tf/keras/losses)
- - [Metrics](https://www.tensorflow.org/api_docs/python/tf/keras/metrics)
-
-%% Cell type:code id: tags:
-
-``` python
-  def get_model_v1(shape):
-
-    model = keras.models.Sequential()
-    model.add(keras.layers.Input(shape, name="InputLayer"))
-    model.add(keras.layers.Dense(64, activation='relu', name='Dense_n1'))
-    model.add(keras.layers.Dense(64, activation='relu', name='Dense_n2'))
-    model.add(keras.layers.Dense(1, name='Output'))
-
-    model.compile(optimizer = 'rmsprop',
-                  loss      = 'mse',
-                  metrics   = ['mae', 'mse'] )
-    return model
-```
-
-%% Cell type:markdown id: tags:
-
-## 5 - Train the model
-### 5.1 - Get it
-
-%% Cell type:code id: tags:
-
-``` python
-model=get_model_v1( (13,) )
-
-model.summary()
-keras.utils.plot_model( model, to_file='./run/model.png', show_shapes=True, show_layer_names=True, dpi=96)
-```
-
-%% Output
-
-    Model: "sequential"
-    _________________________________________________________________
-    Layer (type)                 Output Shape              Param #
-    =================================================================
-    Dense_n1 (Dense)             (None, 64)                896
-    _________________________________________________________________
-    Dense_n2 (Dense)             (None, 64)                4160
-    _________________________________________________________________
-    Output (Dense)               (None, 1)                 65
-    =================================================================
-    Total params: 5,121
-    Trainable params: 5,121
-    Non-trainable params: 0
-    _________________________________________________________________
-
-
-    <IPython.core.display.Image object>
-
-%% Cell type:markdown id: tags:
-
-### 5.2 - Add callback
-
-%% Cell type:code id: tags:
-
-``` python
-os.makedirs('./run/models',   mode=0o750, exist_ok=True)
-save_dir = "./run/models/best_model.h5"
-
-savemodel_callback = tf.keras.callbacks.ModelCheckpoint(filepath=save_dir, verbose=0, save_best_only=True)
-```
-
-%% Cell type:markdown id: tags:
-
-### 5.3 - Train it
-
-%% Cell type:code id: tags:
-
-``` python
-history = model.fit(x_train,
-                    y_train,
-                    epochs          = 100,
-                    batch_size      = 10,
-                    verbose         = 1,
-                    validation_data = (x_test, y_test),
-                    callbacks       = [savemodel_callback])
-```
-
-%% Output
-
-    Train on 354 samples, validate on 152 samples
-    Epoch 1/100
-    354/354 [==============================] - 1s 3ms/sample - loss: 446.5069 - mae: 19.1690 - mse: 446.5069 - val_loss: 328.7387 - val_mae: 16.4455 - val_mse: 328.7387
-    Epoch 2/100
-    354/354 [==============================] - 0s 301us/sample - loss: 206.7491 - mae: 12.2281 - mse: 206.7491 - val_loss: 102.8150 - val_mae: 8.6449 - val_mse: 102.8150
-    Epoch 3/100
-    354/354 [==============================] - 0s 302us/sample - loss: 65.8724 - mae: 6.2331 - mse: 65.8724 - val_loss: 33.7508 - val_mae: 4.5848 - val_mse: 33.7508
-    Epoch 4/100
-    354/354 [==============================] - 0s 318us/sample - loss: 33.4179 - mae: 4.2331 - mse: 33.4179 - val_loss: 27.0058 - val_mae: 3.9154 - val_mse: 27.0058
-    Epoch 5/100
-    354/354 [==============================] - 0s 312us/sample - loss: 24.9602 - mae: 3.5624 - mse: 24.9602 - val_loss: 23.2470 - val_mae: 3.5429 - val_mse: 23.2470
-    Epoch 6/100
-    354/354 [==============================] - 0s 316us/sample - loss: 21.4080 - mae: 3.2530 - mse: 21.4080 - val_loss: 22.1707 - val_mae: 3.4498 - val_mse: 22.1707
-    Epoch 7/100
-    354/354 [==============================] - 0s 262us/sample - loss: 18.3586 - mae: 3.0399 - mse: 18.3586 - val_loss: 24.4102 - val_mae: 3.4754 - val_mse: 24.4102
-    Epoch 8/100
-    354/354 [==============================] - 0s 307us/sample - loss: 16.9126 - mae: 2.8925 - mse: 16.9126 - val_loss: 20.1919 - val_mae: 3.2138 - val_mse: 20.1919
-    Epoch 9/100
-    354/354 [==============================] - 0s 312us/sample - loss: 15.5047 - mae: 2.7532 - mse: 15.5047 - val_loss: 19.0378 - val_mae: 3.0763 - val_mse: 19.0378
-    Epoch 10/100
-    354/354 [==============================] - 0s 273us/sample - loss: 14.5763 - mae: 2.6404 - mse: 14.5763 - val_loss: 19.9752 - val_mae: 3.0986 - val_mse: 19.9752
-    Epoch 11/100
-    354/354 [==============================] - 0s 310us/sample - loss: 13.5901 - mae: 2.5801 - mse: 13.5901 - val_loss: 18.9675 - val_mae: 3.0192 - val_mse: 18.9675
-    Epoch 12/100
-    354/354 [==============================] - 0s 270us/sample - loss: 12.9341 - mae: 2.5158 - mse: 12.9341 - val_loss: 20.6757 - val_mae: 3.1029 - val_mse: 20.6757
-    Epoch 13/100
-    354/354 [==============================] - 0s 311us/sample - loss: 12.4520 - mae: 2.5061 - mse: 12.4520 - val_loss: 17.6596 - val_mae: 2.8839 - val_mse: 17.6596
-    Epoch 14/100
-    354/354 [==============================] - 0s 311us/sample - loss: 11.9484 - mae: 2.4710 - mse: 11.9484 - val_loss: 16.7645 - val_mae: 2.8083 - val_mse: 16.7645
-    Epoch 15/100
-    354/354 [==============================] - 0s 269us/sample - loss: 11.6260 - mae: 2.3959 - mse: 11.6260 - val_loss: 17.5048 - val_mae: 2.8007 - val_mse: 17.5048
-    Epoch 16/100
-    354/354 [==============================] - 0s 267us/sample - loss: 11.2504 - mae: 2.3567 - mse: 11.2504 - val_loss: 18.6748 - val_mae: 2.8771 - val_mse: 18.6748
-    Epoch 17/100
-    354/354 [==============================] - 0s 269us/sample - loss: 10.8352 - mae: 2.3051 - mse: 10.8352 - val_loss: 19.4796 - val_mae: 3.0041 - val_mse: 19.4796
-    Epoch 18/100
-    354/354 [==============================] - 0s 267us/sample - loss: 10.6488 - mae: 2.3377 - mse: 10.6488 - val_loss: 17.0329 - val_mae: 2.7640 - val_mse: 17.0329
-    Epoch 19/100
-    354/354 [==============================] - 0s 273us/sample - loss: 10.2134 - mae: 2.2439 - mse: 10.2134 - val_loss: 18.0589 - val_mae: 2.8565 - val_mse: 18.0589
-    Epoch 20/100
-    354/354 [==============================] - 0s 315us/sample - loss: 10.1024 - mae: 2.2432 - mse: 10.1024 - val_loss: 16.5968 - val_mae: 2.7402 - val_mse: 16.5968
-    Epoch 21/100
-    354/354 [==============================] - 0s 277us/sample - loss: 10.0576 - mae: 2.2401 - mse: 10.0576 - val_loss: 18.4496 - val_mae: 2.8156 - val_mse: 18.4496
-    Epoch 22/100
-    354/354 [==============================] - 0s 269us/sample - loss: 9.6590 - mae: 2.1500 - mse: 9.6590 - val_loss: 18.7084 - val_mae: 2.8309 - val_mse: 18.7084
-    Epoch 23/100
-    354/354 [==============================] - 0s 277us/sample - loss: 9.4596 - mae: 2.1967 - mse: 9.4596 - val_loss: 18.0308 - val_mae: 2.7595 - val_mse: 18.0308
-    Epoch 24/100
-    354/354 [==============================] - 0s 272us/sample - loss: 9.2778 - mae: 2.1680 - mse: 9.2778 - val_loss: 18.9343 - val_mae: 2.9152 - val_mse: 18.9343
-    Epoch 25/100
-    354/354 [==============================] - 0s 267us/sample - loss: 9.1075 - mae: 2.1451 - mse: 9.1076 - val_loss: 18.0646 - val_mae: 2.8202 - val_mse: 18.0646
-    Epoch 26/100
-    354/354 [==============================] - 0s 273us/sample - loss: 9.2196 - mae: 2.1282 - mse: 9.2196 - val_loss: 18.7244 - val_mae: 2.8288 - val_mse: 18.7244
-    Epoch 27/100
-    354/354 [==============================] - 0s 267us/sample - loss: 8.5733 - mae: 2.0703 - mse: 8.5733 - val_loss: 16.9568 - val_mae: 2.8123 - val_mse: 16.9568
-    Epoch 28/100
-    354/354 [==============================] - 0s 309us/sample - loss: 8.6252 - mae: 2.0821 - mse: 8.6252 - val_loss: 16.4984 - val_mae: 2.7069 - val_mse: 16.4984
-    Epoch 29/100
-    354/354 [==============================] - 0s 307us/sample - loss: 8.6336 - mae: 2.0822 - mse: 8.6336 - val_loss: 16.0498 - val_mae: 2.6532 - val_mse: 16.0498
-    Epoch 30/100
-    354/354 [==============================] - 0s 321us/sample - loss: 8.5071 - mae: 2.0379 - mse: 8.5071 - val_loss: 15.1042 - val_mae: 2.6004 - val_mse: 15.1042
-    Epoch 31/100
-    354/354 [==============================] - 0s 273us/sample - loss: 8.2888 - mae: 2.0627 - mse: 8.2888 - val_loss: 16.2730 - val_mae: 2.7019 - val_mse: 16.2730
-    Epoch 32/100
-    354/354 [==============================] - 0s 271us/sample - loss: 8.2021 - mae: 2.0000 - mse: 8.2021 - val_loss: 17.2852 - val_mae: 2.7962 - val_mse: 17.2852
-    Epoch 33/100
-    354/354 [==============================] - 0s 272us/sample - loss: 8.2973 - mae: 2.0336 - mse: 8.2973 - val_loss: 16.8973 - val_mae: 2.7318 - val_mse: 16.8973
-    Epoch 34/100
-    354/354 [==============================] - 0s 257us/sample - loss: 8.1033 - mae: 2.0105 - mse: 8.1033 - val_loss: 16.6509 - val_mae: 2.8218 - val_mse: 16.6509
-    Epoch 35/100
-    354/354 [==============================] - 0s 272us/sample - loss: 8.0724 - mae: 2.0170 - mse: 8.0724 - val_loss: 16.0802 - val_mae: 2.6733 - val_mse: 16.0802
-    Epoch 36/100
-    354/354 [==============================] - 0s 257us/sample - loss: 7.7939 - mae: 1.9606 - mse: 7.7939 - val_loss: 17.1008 - val_mae: 2.7384 - val_mse: 17.1008
-    Epoch 37/100
-    354/354 [==============================] - 0s 269us/sample - loss: 7.7812 - mae: 1.9719 - mse: 7.7812 - val_loss: 16.3472 - val_mae: 2.6939 - val_mse: 16.3472
-    Epoch 38/100
-    354/354 [==============================] - 0s 276us/sample - loss: 7.4494 - mae: 1.9224 - mse: 7.4494 - val_loss: 19.3916 - val_mae: 2.9414 - val_mse: 19.3916
-    Epoch 39/100
-    354/354 [==============================] - 0s 271us/sample - loss: 7.8023 - mae: 1.9978 - mse: 7.8023 - val_loss: 16.3499 - val_mae: 2.7018 - val_mse: 16.3499
-    Epoch 40/100
-    354/354 [==============================] - 0s 270us/sample - loss: 7.3681 - mae: 1.9293 - mse: 7.3681 - val_loss: 16.0445 - val_mae: 2.6872 - val_mse: 16.0445
-    Epoch 41/100
-    354/354 [==============================] - 0s 267us/sample - loss: 7.3013 - mae: 1.8820 - mse: 7.3013 - val_loss: 16.5657 - val_mae: 2.7222 - val_mse: 16.5657
-    Epoch 42/100
-    354/354 [==============================] - 0s 274us/sample - loss: 7.3978 - mae: 1.9154 - mse: 7.3978 - val_loss: 15.9821 - val_mae: 2.6576 - val_mse: 15.9821
-    Epoch 43/100
-    354/354 [==============================] - 0s 319us/sample - loss: 6.9832 - mae: 1.9037 - mse: 6.9832 - val_loss: 14.4977 - val_mae: 2.5418 - val_mse: 14.4977
-    Epoch 44/100
-    354/354 [==============================] - 0s 269us/sample - loss: 7.2307 - mae: 1.8968 - mse: 7.2307 - val_loss: 15.0962 - val_mae: 2.6188 - val_mse: 15.0962
-    Epoch 45/100
-    354/354 [==============================] - 0s 256us/sample - loss: 7.0289 - mae: 1.8685 - mse: 7.0289 - val_loss: 17.0531 - val_mae: 2.8123 - val_mse: 17.0531
-    Epoch 46/100
-    354/354 [==============================] - 0s 270us/sample - loss: 6.9010 - mae: 1.8537 - mse: 6.9010 - val_loss: 16.7469 - val_mae: 2.7081 - val_mse: 16.7469
-    Epoch 47/100
-    354/354 [==============================] - 0s 268us/sample - loss: 6.9256 - mae: 1.8664 - mse: 6.9256 - val_loss: 16.1227 - val_mae: 2.7760 - val_mse: 16.1227
-    Epoch 48/100
-    354/354 [==============================] - 0s 273us/sample - loss: 6.8333 - mae: 1.8552 - mse: 6.8333 - val_loss: 14.9262 - val_mae: 2.6213 - val_mse: 14.9262
-    Epoch 49/100
-    354/354 [==============================] - 0s 313us/sample - loss: 6.7351 - mae: 1.8375 - mse: 6.7351 - val_loss: 14.2252 - val_mae: 2.5309 - val_mse: 14.2252
-    Epoch 50/100
-    354/354 [==============================] - 0s 276us/sample - loss: 6.6672 - mae: 1.7913 - mse: 6.6672 - val_loss: 16.5652 - val_mae: 2.7693 - val_mse: 16.5652
-    Epoch 51/100
-    354/354 [==============================] - 0s 271us/sample - loss: 6.6222 - mae: 1.8325 - mse: 6.6222 - val_loss: 14.8928 - val_mae: 2.5921 - val_mse: 14.8928
-    Epoch 52/100
-    354/354 [==============================] - 0s 271us/sample - loss: 6.5606 - mae: 1.8150 - mse: 6.5606 - val_loss: 14.7382 - val_mae: 2.6124 - val_mse: 14.7382
-    Epoch 53/100
-    354/354 [==============================] - 0s 273us/sample - loss: 6.5737 - mae: 1.7757 - mse: 6.5737 - val_loss: 14.8866 - val_mae: 2.6357 - val_mse: 14.8866
-    Epoch 54/100
-    354/354 [==============================] - 0s 264us/sample - loss: 6.3009 - mae: 1.7569 - mse: 6.3009 - val_loss: 14.6100 - val_mae: 2.6115 - val_mse: 14.6100
-    Epoch 55/100
-    354/354 [==============================] - 0s 272us/sample - loss: 6.2524 - mae: 1.7679 - mse: 6.2524 - val_loss: 17.4939 - val_mae: 2.8652 - val_mse: 17.4939
-    Epoch 56/100
-    354/354 [==============================] - 0s 319us/sample - loss: 6.2461 - mae: 1.7830 - mse: 6.2461 - val_loss: 14.0397 - val_mae: 2.5829 - val_mse: 14.0397
-    Epoch 57/100
-    354/354 [==============================] - 0s 267us/sample - loss: 6.3124 - mae: 1.7788 - mse: 6.3124 - val_loss: 15.4946 - val_mae: 2.7133 - val_mse: 15.4946
-    Epoch 58/100
-    354/354 [==============================] - 0s 269us/sample - loss: 6.1133 - mae: 1.7282 - mse: 6.1133 - val_loss: 14.5244 - val_mae: 2.5982 - val_mse: 14.5244
-    Epoch 59/100
-    354/354 [==============================] - 0s 259us/sample - loss: 6.2866 - mae: 1.7860 - mse: 6.2866 - val_loss: 15.8915 - val_mae: 2.7331 - val_mse: 15.8915
-    Epoch 60/100
-    354/354 [==============================] - 0s 311us/sample - loss: 5.9945 - mae: 1.7178 - mse: 5.9945 - val_loss: 13.2656 - val_mae: 2.5189 - val_mse: 13.2656
-    Epoch 61/100
-    354/354 [==============================] - 0s 263us/sample - loss: 6.0649 - mae: 1.7064 - mse: 6.0649 - val_loss: 15.4134 - val_mae: 2.7351 - val_mse: 15.4134
-    Epoch 62/100
-    354/354 [==============================] - 0s 268us/sample - loss: 5.9954 - mae: 1.6767 - mse: 5.9954 - val_loss: 13.8741 - val_mae: 2.5721 - val_mse: 13.8741
-    Epoch 63/100
-    354/354 [==============================] - 0s 254us/sample - loss: 5.9648 - mae: 1.7023 - mse: 5.9648 - val_loss: 15.1974 - val_mae: 2.6602 - val_mse: 15.1974
-    Epoch 64/100
-    354/354 [==============================] - 0s 272us/sample - loss: 5.7276 - mae: 1.7202 - mse: 5.7276 - val_loss: 14.5766 - val_mae: 2.6508 - val_mse: 14.5766
-    Epoch 65/100
-    354/354 [==============================] - 0s 266us/sample - loss: 5.8443 - mae: 1.6907 - mse: 5.8443 - val_loss: 15.5797 - val_mae: 2.6848 - val_mse: 15.5797
-    Epoch 66/100
-    354/354 [==============================] - 0s 273us/sample - loss: 5.8195 - mae: 1.7295 - mse: 5.8195 - val_loss: 14.5484 - val_mae: 2.6527 - val_mse: 14.5484
-    Epoch 67/100
-    354/354 [==============================] - 0s 266us/sample - loss: 5.8216 - mae: 1.6966 - mse: 5.8216 - val_loss: 14.3616 - val_mae: 2.5733 - val_mse: 14.3616
-    Epoch 68/100
-    354/354 [==============================] - 0s 271us/sample - loss: 5.6572 - mae: 1.6543 - mse: 5.6572 - val_loss: 16.1438 - val_mae: 2.8151 - val_mse: 16.1438
-    Epoch 69/100
-    354/354 [==============================] - 0s 259us/sample - loss: 5.5142 - mae: 1.6657 - mse: 5.5142 - val_loss: 14.2295 - val_mae: 2.5796 - val_mse: 14.2295
-    Epoch 70/100
-    354/354 [==============================] - 0s 273us/sample - loss: 5.4965 - mae: 1.6313 - mse: 5.4965 - val_loss: 15.2662 - val_mae: 2.6980 - val_mse: 15.2662
-    Epoch 71/100
-    354/354 [==============================] - 0s 270us/sample - loss: 5.4534 - mae: 1.6717 - mse: 5.4534 - val_loss: 14.5025 - val_mae: 2.6441 - val_mse: 14.5025
-    Epoch 72/100
-    354/354 [==============================] - 0s 253us/sample - loss: 5.5146 - mae: 1.6526 - mse: 5.5146 - val_loss: 13.7906 - val_mae: 2.5753 - val_mse: 13.7906
-    Epoch 73/100
-    354/354 [==============================] - 0s 272us/sample - loss: 5.4499 - mae: 1.6130 - mse: 5.4499 - val_loss: 15.1649 - val_mae: 2.7624 - val_mse: 15.1649
-    Epoch 74/100
-    354/354 [==============================] - 0s 309us/sample - loss: 5.3808 - mae: 1.6297 - mse: 5.3808 - val_loss: 12.9326 - val_mae: 2.5007 - val_mse: 12.9326
-    Epoch 75/100
-    354/354 [==============================] - 0s 258us/sample - loss: 5.3546 - mae: 1.6313 - mse: 5.3546 - val_loss: 13.6397 - val_mae: 2.5810 - val_mse: 13.6397
-    Epoch 76/100
-    354/354 [==============================] - 0s 265us/sample - loss: 5.1666 - mae: 1.5998 - mse: 5.1666 - val_loss: 15.6069 - val_mae: 2.7630 - val_mse: 15.6069
-    Epoch 77/100
-    354/354 [==============================] - 0s 272us/sample - loss: 5.2465 - mae: 1.6192 - mse: 5.2465 - val_loss: 14.8084 - val_mae: 2.6388 - val_mse: 14.8084
-    Epoch 78/100
-    354/354 [==============================] - 0s 265us/sample - loss: 5.1107 - mae: 1.5772 - mse: 5.1107 - val_loss: 13.6319 - val_mae: 2.5756 - val_mse: 13.6319
-    Epoch 79/100
-    354/354 [==============================] - 0s 272us/sample - loss: 5.2677 - mae: 1.5989 - mse: 5.2677 - val_loss: 15.0306 - val_mae: 2.7715 - val_mse: 15.0306
-    Epoch 80/100
-    354/354 [==============================] - 0s 274us/sample - loss: 5.0534 - mae: 1.5504 - mse: 5.0534 - val_loss: 13.3917 - val_mae: 2.5352 - val_mse: 13.3917
-    Epoch 81/100
-    354/354 [==============================] - 0s 272us/sample - loss: 5.1013 - mae: 1.5826 - mse: 5.1013 - val_loss: 14.6761 - val_mae: 2.7158 - val_mse: 14.6761
-    Epoch 82/100
-    354/354 [==============================] - 0s 258us/sample - loss: 5.1137 - mae: 1.5984 - mse: 5.1137 - val_loss: 14.7063 - val_mae: 2.6576 - val_mse: 14.7063
-    Epoch 83/100
-    354/354 [==============================] - 0s 269us/sample - loss: 4.9343 - mae: 1.5545 - mse: 4.9343 - val_loss: 13.6205 - val_mae: 2.5494 - val_mse: 13.6205
-    Epoch 84/100
-    354/354 [==============================] - 0s 277us/sample - loss: 4.9839 - mae: 1.5815 - mse: 4.9839 - val_loss: 13.3857 - val_mae: 2.6047 - val_mse: 13.3857
-    Epoch 85/100
-    354/354 [==============================] - 0s 277us/sample - loss: 4.9946 - mae: 1.5818 - mse: 4.9946 - val_loss: 14.1012 - val_mae: 2.6176 - val_mse: 14.1012
-    Epoch 86/100
-    354/354 [==============================] - 0s 273us/sample - loss: 4.7884 - mae: 1.5321 - mse: 4.7884 - val_loss: 14.5182 - val_mae: 2.6687 - val_mse: 14.5182
-    Epoch 87/100
-    354/354 [==============================] - 0s 311us/sample - loss: 4.8134 - mae: 1.5660 - mse: 4.8134 - val_loss: 12.7966 - val_mae: 2.5734 - val_mse: 12.7966
-    Epoch 88/100
-    354/354 [==============================] - 0s 273us/sample - loss: 4.7923 - mae: 1.5483 - mse: 4.7923 - val_loss: 14.4001 - val_mae: 2.6707 - val_mse: 14.4001
-    Epoch 89/100
-    354/354 [==============================] - 0s 274us/sample - loss: 4.6705 - mae: 1.5086 - mse: 4.6705 - val_loss: 15.3677 - val_mae: 2.7359 - val_mse: 15.3677
-    Epoch 90/100
-    354/354 [==============================] - 0s 280us/sample - loss: 4.8776 - mae: 1.5806 - mse: 4.8776 - val_loss: 14.4442 - val_mae: 2.6343 - val_mse: 14.4442
-    Epoch 91/100
-    354/354 [==============================] - 0s 260us/sample - loss: 4.6349 - mae: 1.5300 - mse: 4.6349 - val_loss: 14.2969 - val_mae: 2.7718 - val_mse: 14.2969
-    Epoch 92/100
-    354/354 [==============================] - 0s 273us/sample - loss: 4.7835 - mae: 1.5637 - mse: 4.7835 - val_loss: 13.1123 - val_mae: 2.5578 - val_mse: 13.1123
-    Epoch 93/100
-    354/354 [==============================] - 0s 277us/sample - loss: 4.6759 - mae: 1.5259 - mse: 4.6759 - val_loss: 14.3508 - val_mae: 2.6888 - val_mse: 14.3507
-    Epoch 94/100
-    354/354 [==============================] - 0s 273us/sample - loss: 4.7856 - mae: 1.5560 - mse: 4.7856 - val_loss: 14.5237 - val_mae: 2.6956 - val_mse: 14.5237
-    Epoch 95/100
-    354/354 [==============================] - 0s 313us/sample - loss: 4.7038 - mae: 1.5331 - mse: 4.7038 - val_loss: 12.7707 - val_mae: 2.5393 - val_mse: 12.7707
-    Epoch 96/100
-    354/354 [==============================] - 0s 277us/sample - loss: 4.6006 - mae: 1.5331 - mse: 4.6006 - val_loss: 13.8540 - val_mae: 2.6720 - val_mse: 13.8540
-    Epoch 97/100
-    354/354 [==============================] - 0s 269us/sample - loss: 4.4720 - mae: 1.4912 - mse: 4.4720 - val_loss: 13.1524 - val_mae: 2.6311 - val_mse: 13.1524
-    Epoch 98/100
-    354/354 [==============================] - 0s 309us/sample - loss: 4.4242 - mae: 1.4854 - mse: 4.4242 - val_loss: 11.7020 - val_mae: 2.4886 - val_mse: 11.7020
-    Epoch 99/100
-    354/354 [==============================] - 0s 280us/sample - loss: 4.5642 - mae: 1.4920 - mse: 4.5642 - val_loss: 12.6523 - val_mae: 2.5232 - val_mse: 12.6523
-    Epoch 100/100
-    354/354 [==============================] - 0s 274us/sample - loss: 4.1971 - mae: 1.4564 - mse: 4.1971 - val_loss: 18.7164 - val_mae: 3.0774 - val_mse: 18.7164
-
-%% Cell type:markdown id: tags:
-
-## Step 6 - Evaluate
-### 6.1 - Model evaluation
-MAE =  Mean Absolute Error (between the labels and predictions)
-A mae equal to 3 represents an average error in prediction of $3k.
-
-%% Cell type:code id: tags:
-
-``` python
-score = model.evaluate(x_test, y_test, verbose=0)
-
-print('x_test / loss      : {:5.4f}'.format(score[0]))
-print('x_test / mae       : {:5.4f}'.format(score[1]))
-print('x_test / mse       : {:5.4f}'.format(score[2]))
-```
-
-%% Output
-
-    x_test / loss      : 18.7164
-    x_test / mae       : 3.0774
-    x_test / mse       : 18.7164
-
-%% Cell type:markdown id: tags:
-
-### 6.2 - Training history
-What was the best result during our training ?
-
-%% Cell type:code id: tags:
-
-``` python
-print("min( val_mae ) : {:.4f}".format( min(history.history["val_mae"]) ) )
-```
-
-%% Output
-
-    min( val_mae ) : 2.4886
-
-%% Cell type:code id: tags:
-
-``` python
-ooo.plot_history(history, plot={'MSE' :['mse', 'val_mse'],
-                                'MAE' :['mae', 'val_mae'],
-                                'LOSS':['loss','val_loss']})
-```
-
-%% Output
-
-
-
-
-
-
-
-%% Cell type:markdown id: tags:
-
-## Step 7 - Restore a model :
-
-%% Cell type:markdown id: tags:
-
-### 7.1 - Reload model
-
-%% Cell type:code id: tags:
-
-``` python
-loaded_model = tf.keras.models.load_model('./run/models/best_model.h5')
-loaded_model.summary()
-print("Loaded.")
-```
-
-%% Output
-
-    Model: "sequential"
-    _________________________________________________________________
-    Layer (type)                 Output Shape              Param #
-    =================================================================
-    Dense_n1 (Dense)             (None, 64)                896
-    _________________________________________________________________
-    Dense_n2 (Dense)             (None, 64)                4160
-    _________________________________________________________________
-    Output (Dense)               (None, 1)                 65
-    =================================================================
-    Total params: 5,121
-    Trainable params: 5,121
-    Non-trainable params: 0
-    _________________________________________________________________
-    Loaded.
-
-%% Cell type:markdown id: tags:
-
-### 7.2 - Evaluate it :
-
-%% Cell type:code id: tags:
-
-``` python
-score = loaded_model.evaluate(x_test, y_test, verbose=0)
-
-print('x_test / loss      : {:5.4f}'.format(score[0]))
-print('x_test / mae       : {:5.4f}'.format(score[1]))
-print('x_test / mse       : {:5.4f}'.format(score[2]))
-```
-
-%% Output
-
-    x_test / loss      : 11.7020
-    x_test / mae       : 2.4886
-    x_test / mse       : 11.7020
-
-%% Cell type:markdown id: tags:
-
-### 7.3 - Make a prediction
-
-%% Cell type:code id: tags:
-
-``` python
-mon_test=[-0.20113196, -0.48631663,  1.23572348, -0.26929877,  2.67879106,
-       -0.89623587,  1.09961251, -1.05826704, -0.55823117, -0.06159088,
-       -1.76085159, -1.97039608,  0.52775666]
-
-mon_test=np.array(mon_test).reshape(1,13)
-```
-
-%% Cell type:code id: tags:
-
-``` python
-predictions = loaded_model.predict( mon_test )
-print("Prédiction : {:.2f} K$   Reality : {:.2f} K$".format(predictions[0][0], y_train[13]))
-```
-
-%% Output
-
-    Prédiction : 16.20 K$   Reality : 21.70 K$
-
-%% Cell type:markdown id: tags:
-
------
-That's all folks !

DCGAN.Lightning/01-DCGAN-PL.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [PLSHEEP3] - A DCGAN to Draw a Sheep, using Pytorch Lightning
+<!-- DESC --> "Draw me a sheep", revisited with a DCGAN, using Pytorch Lightning
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Build and train a DCGAN model with the Quick Draw dataset
+ - Understanding DCGAN
+
+The [Quick draw dataset](https://quickdraw.withgoogle.com/data) contains about 50.000.000 drawings, made by real people...
+We are using a subset of 117.555 of Sheep drawings
+To get the dataset : [https://github.com/googlecreativelab/quickdraw-dataset](https://github.com/googlecreativelab/quickdraw-dataset)
+Datasets in numpy bitmap file : [https://console.cloud.google.com/storage/quickdraw_dataset/full/numpy_bitmap](https://console.cloud.google.com/storage/quickdraw_dataset/full/numpy_bitmap)
+Sheep dataset : [https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/sheep.npy](https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/sheep.npy) (94.3 Mo)
+
+
+## What we're going to do :
+
+ - Have a look to the dataset
+ - Defining a GAN model
+ - Build the model
+ - Train it
+ - Have a look of the results
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init and parameters
+#### Python init
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+import sys
+import shutil
+
+import numpy as np
+import torch
+from lightning import Trainer
+from lightning.pytorch.callbacks                        import ModelCheckpoint
+from lightning.pytorch.loggers.tensorboard              import TensorBoardLogger
+
+import fidle
+
+from modules.QuickDrawDataModule import QuickDrawDataModule
+
+from modules.GAN                 import GAN
+from modules.WGANGP              import WGANGP
+from modules.Generators          import *
+from modules.Discriminators      import *
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('PLSHEEP3')
+```
+
+%% Cell type:markdown id: tags:
+
+#### Few parameters
+scale=1, epochs=20 : Need 22' on a V100
+
+%% Cell type:code id: tags:
+
+``` python
+latent_dim          = 128
+
+gan_name            = 'WGANGP'
+generator_name      = 'Generator_2'
+discriminator_name  = 'Discriminator_3'
+
+scale               = 0.001
+epochs              = 4
+num_workers         = 2
+lr                  = 0.0001
+b1                  = 0.5
+b2                  = 0.999
+lambda_gp           = 10
+batch_size          = 64
+num_img             = 48
+fit_verbosity       = 2
+
+dataset_file        = datasets_dir+'/QuickDraw/origine/sheep.npy'
+data_shape          = (28,28,1)
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('latent_dim', 'gan_name', 'generator_name', 'discriminator_name')
+fidle.override('epochs', 'lr', 'b1', 'b2', 'batch_size', 'num_img', 'fit_verbosity')
+fidle.override('dataset_file', 'data_shape', 'scale', 'num_workers' )
+```
+
+%% Cell type:markdown id: tags:
+
+#### Cleaning
+
+%% Cell type:code id: tags:
+
+``` python
+# You can comment these lines to keep each run...
+shutil.rmtree(f'{run_dir}/figs', ignore_errors=True)
+shutil.rmtree(f'{run_dir}/models', ignore_errors=True)
+shutil.rmtree(f'{run_dir}/tb_logs', ignore_errors=True)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Get some nice data
+
+%% Cell type:markdown id: tags:
+
+#### Get a Nice DataModule
+Our DataModule is defined in [./modules/QuickDrawDataModule.py](./modules/QuickDrawDataModule.py)
+This is a [LightningDataModule](https://pytorch-lightning.readthedocs.io/en/stable/data/datamodule.html)
+
+%% Cell type:code id: tags:
+
+``` python
+dm = QuickDrawDataModule(dataset_file, scale, batch_size, num_workers=num_workers)
+dm.setup()
+```
+
+%% Cell type:markdown id: tags:
+
+#### Have a look
+
+%% Cell type:code id: tags:
+
+``` python
+dl         = dm.train_dataloader()
+batch_data = next(iter(dl))
+
+fidle.scrawler.images( batch_data.reshape(-1,28,28), indices=range(batch_size), columns=12, x_size=1, y_size=1,
+                       y_padding=0,spines_alpha=0, save_as='01-Sheeps')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Get a nice GAN model
+
+Our Generators are defined in [./modules/Generators.py](./modules/Generators.py)
+Our Discriminators are defined in [./modules/Discriminators.py](./modules/Discriminators.py)
+
+
+Our GANs are defined in :
+ - [./modules/GAN.py](./modules/GAN.py)
+ - [./modules/WGANGP.py](./modules/WGANGP.py)
+
+%% Cell type:markdown id: tags:
+
+#### Retrieve class by name
+To be very flexible, we just specify class names as parameters.
+The code below retrieves classes from their names.
+
+%% Cell type:code id: tags:
+
+``` python
+module=sys.modules['__main__']
+Generator_     = getattr(module, generator_name)
+Discriminator_ = getattr(module, discriminator_name)
+GAN_           = getattr(module, gan_name)
+```
+
+%% Cell type:markdown id: tags:
+
+#### Basic test - Just to be sure it (could) works... ;-)
+
+%% Cell type:code id: tags:
+
+``` python
+generator     = Generator_(     latent_dim=latent_dim, data_shape=data_shape )
+discriminator = Discriminator_( latent_dim=latent_dim, data_shape=data_shape )
+
+print('\nFew tests :\n')
+z = torch.randn(batch_size, latent_dim)
+print('z size        : ',z.size())
+
+fake_img = generator.forward(z)
+print('fake_img      : ', fake_img.size())
+
+p = discriminator.forward(fake_img)
+print('pred fake     : ', p.size())
+
+print('batch_data    : ',batch_data.size())
+
+p = discriminator.forward(batch_data)
+print('pred real     : ', p.size())
+
+print('\nShow fake images :')
+nimg = fake_img.detach().numpy()
+fidle.scrawler.images( nimg.reshape(-1,28,28), indices=range(batch_size), columns=12, x_size=1, y_size=1,
+                       y_padding=0,spines_alpha=0, save_as='01-Sheeps')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+print('Fake images : ', fake_img.size())
+print('Batch size  : ', batch_data.size())
+e = torch.distributions.uniform.Uniform(0, 1).sample([batch_size,1])
+e = e[:None,None,None]
+i = fake_img * e + (1-e)*batch_data
+
+print('\ninterpolate images :')
+nimg = i.detach().numpy()
+fidle.scrawler.images( nimg.reshape(-1,28,28), indices=range(batch_size), columns=12, x_size=1, y_size=1,
+                       y_padding=0,spines_alpha=0, save_as='01-Sheeps')
+```
+
+%% Cell type:markdown id: tags:
+
+#### GAN model
+To simplify our code, the GAN class is defined separately in the module [./modules/GAN.py](./modules/GAN.py)
+Passing the classe names for generator/discriminator by parameter allows to stay modular and to use the PL checkpoints.
+
+%% Cell type:code id: tags:
+
+``` python
+gan = GAN_( data_shape          = data_shape,
+            lr                  = lr,
+            b1                  = b1,
+            b2                  = b2,
+            lambda_gp           = lambda_gp,
+            batch_size          = batch_size,
+            latent_dim          = latent_dim,
+            generator_name      = generator_name,
+            discriminator_name  = discriminator_name)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Train it !
+#### Instantiate Callbacks, Logger & co.
+More about :
+- [Checkpoints](https://pytorch-lightning.readthedocs.io/en/stable/common/checkpointing_basic.html)
+- [modelCheckpoint](https://pytorch-lightning.readthedocs.io/en/stable/api/pytorch_lightning.callbacks.ModelCheckpoint.html#pytorch_lightning.callbacks.ModelCheckpoint)
+
+%% Cell type:code id: tags:
+
+``` python
+
+# ---- for tensorboard logs
+#
+logger       = TensorBoardLogger(       save_dir       = f'{run_dir}',
+                                        name           = 'tb_logs'  )
+
+log_dir = os.path.abspath(f'{run_dir}/tb_logs')
+print('To access the logs with tensorboard, use this command line :')
+print(f'tensorboard --logdir {log_dir}')
+
+# ---- To save checkpoints
+#
+callback_checkpoints = ModelCheckpoint( dirpath        = f'{run_dir}/models',
+                                        filename       = 'bestModel',
+                                        save_top_k     = 1,
+                                        save_last      = True,
+                                        every_n_epochs = 1,
+                                        monitor        = "g_loss")
+```
+
+%% Cell type:markdown id: tags:
+
+#### Train it
+
+%% Cell type:code id: tags:
+
+``` python
+
+trainer = Trainer(
+    accelerator        = "auto",
+    max_epochs         = epochs,
+    callbacks          = [callback_checkpoints],
+    log_every_n_steps  = batch_size,
+    logger             = logger
+)
+
+trainer.fit(gan, dm)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Reload our best model
+Note :
+
+%% Cell type:code id: tags:
+
+``` python
+gan = GAN.load_from_checkpoint(f'{run_dir}/models/bestModel.ckpt')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+nb_images = 96
+
+z = torch.randn(nb_images, latent_dim)
+print('z size        : ',z.size())
+
+if torch.cuda.is_available(): z=z.cuda()
+
+fake_img = gan.generator.forward(z)
+print('fake_img      : ', fake_img.size())
+
+nimg = fake_img.cpu().detach().numpy()
+fidle.scrawler.images( nimg.reshape(-1,28,28), indices=range(nb_images), columns=12, x_size=1, y_size=1,
+                       y_padding=0,spines_alpha=0, save_as='01-Sheeps')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

DCGAN.Lightning/modules/Discriminators.py

+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|                         GAN / Generators
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI - https://fidle.cnrs.fr
+# ------------------------------------------------------------------
+# JL Parouty (Mars 2024)
+
+
+import numpy as np
+import torch.nn as nn
+
+
+# -----------------------------------------------------------------------------
+# -- Discriminator n°1
+# -----------------------------------------------------------------------------
+#
+class Discriminator_1(nn.Module):
+    '''
+    A basic DNN discriminator, usable with classic GAN
+    '''
+
+    def __init__(self, latent_dim=None, data_shape=None):
+    
+        super().__init__()
+        self.img_shape = data_shape
+        print('init discriminator 1     : ',data_shape,' to sigmoid')
+
+        self.model = nn.Sequential(
+
+            nn.Flatten(),
+            nn.Linear(int(np.prod(data_shape)), 512),
+            nn.ReLU(),
+            
+            nn.Linear(512, 256),
+            nn.ReLU(),
+
+            nn.Linear(256, 1),
+            nn.Sigmoid(),
+        )
+
+    def forward(self, img):
+        validity = self.model(img)
+
+        return validity
+
+
+
+# -----------------------------------------------------------------------------
+# -- Discriminator n°2
+# -----------------------------------------------------------------------------
+#
+class Discriminator_2(nn.Module):
+    '''
+    A more efficient discriminator,based on CNN, usable with classic GAN
+    '''
+
+    def __init__(self, latent_dim=None, data_shape=None):
+    
+        super().__init__()
+        self.img_shape = data_shape
+        print('init discriminator 2     : ',data_shape,' to sigmoid')
+
+        self.model = nn.Sequential(
+
+            nn.Conv2d(1, 32, kernel_size = 3, stride = 2, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(32),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(32, 64, kernel_size = 3, stride = 1, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(64),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(64, 128, kernel_size = 3, stride = 1, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(128),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(128, 256, kernel_size = 3, stride = 2, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(256),
+            nn.Dropout2d(0.25),
+
+            nn.Flatten(),
+            nn.Linear(12544, 1),
+            nn.Sigmoid(),
+        )
+
+    def forward(self, img):
+        img_nchw = img.permute(0, 3, 1, 2) # reformat from NHWC to NCHW
+        validity = self.model(img_nchw)
+
+        return validity
+
+
+   
+# -----------------------------------------------------------------------------
+# -- Discriminator n°3
+# -----------------------------------------------------------------------------
+#     
+class Discriminator_3(nn.Module):
+    '''
+    A CNN discriminator, usable with a WGANGP.
+    This discriminator has no sigmoid and returns a critical and not a probability
+    '''
+
+    def __init__(self, latent_dim=None, data_shape=None):
+    
+        super().__init__()
+        self.img_shape = data_shape
+        print('init discriminator 3     : ',data_shape,' to sigmoid')
+
+        self.model = nn.Sequential(
+
+            nn.Conv2d(1, 32, kernel_size = 3, stride = 2, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(32),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(32, 64, kernel_size = 3, stride = 1, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(64),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(64, 128, kernel_size = 3, stride = 1, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(128),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(128, 256, kernel_size = 3, stride = 2, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(256),
+            nn.Dropout2d(0.25),
+
+            nn.Flatten(),
+            nn.Linear(12544, 1),
+            nn.Sigmoid(),
+
+        )
+
+    def forward(self, img):
+        img_nchw = img.permute(0, 3, 1, 2) # reformat from NHWC to NCHW
+        validity = self.model(img_nchw)
+
+        return validity
\ No newline at end of file

DCGAN.Lightning/modules/GAN.py

+
+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|                Basic GAN LigthningModule
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI - https://fidle.cnrs.fr
+# ------------------------------------------------------------------
+# JL Parouty (Mars 2024)
+
+
+
+import sys
+import numpy as np
+import torch
+import torch.nn.functional as F
+import torchvision
+from lightning import LightningModule
+
+
+class GAN(LightningModule):
+
+    # -------------------------------------------------------------------------
+    # Init
+    # -------------------------------------------------------------------------
+    #
+    def __init__(
+        self,
+        data_shape          = (None,None,None),
+        latent_dim          = None,
+        lr                  = 0.0002,
+        b1                  = 0.5,
+        b2                  = 0.999,
+        batch_size          = 64,
+        generator_name      = None,
+        discriminator_name  = None,
+        **kwargs,
+    ):
+        super().__init__()
+
+        print('\n---- GAN initialization --------------------------------------------')
+
+        # ---- Hyperparameters
+        #
+        # Enable Lightning to store all the provided arguments under the self.hparams attribute.
+        # These hyperparameters will also be stored within the model checkpoint.
+        #
+        self.save_hyperparameters()
+
+        print('Hyperarameters are :')
+        for name,value in self.hparams.items():
+            print(f'{name:24s} : {value}')
+
+        # ---- Because we have more than one optimizer
+        #
+        self.automatic_optimization = False
+
+        # ---- Generator/Discriminator instantiation
+        #
+        print('Submodels :')
+        module=sys.modules['__main__']
+        class_g = getattr(module, generator_name)
+        class_d = getattr(module, discriminator_name)
+        self.generator     = class_g( latent_dim=latent_dim, data_shape=data_shape)
+        self.discriminator = class_d( latent_dim=latent_dim, data_shape=data_shape)
+
+        # ---- Validation and example data
+        #
+        self.validation_z        = torch.randn(8, self.hparams.latent_dim)
+        self.example_input_array = torch.zeros(2, self.hparams.latent_dim)
+
+
+    def forward(self, z):
+        return self.generator(z)
+
+
+    def adversarial_loss(self, y_hat, y):
+        return F.binary_cross_entropy(y_hat, y)
+
+
+    def training_step(self, batch, batch_idx):
+        imgs       = batch
+        batch_size = batch.size(0)
+
+        optimizer_g, optimizer_d = self.optimizers()
+
+        # ---- Get some latent space vectors
+        #      We use type_as() to make sure we initialize z on the right device (GPU/CPU).
+        #
+        z = torch.randn(batch_size, self.hparams.latent_dim)
+        z = z.type_as(imgs)
+
+        # ---- Train generator ------------------------------------------------
+        #      Generator use optimizer #0
+        #      We try to generate false images that could mislead the discriminator
+        #
+        self.toggle_optimizer(optimizer_g)
+        
+        # Generate fake images
+        self.fake_imgs = self.generator.forward(z)
+
+        # Assemble labels that say all images are real, yes it's a lie ;-)
+        # put on GPU because we created this tensor inside training_loop
+        misleading_labels = torch.ones(batch_size, 1)
+        misleading_labels = misleading_labels.type_as(imgs)
+
+        # Adversarial loss is binary cross-entropy
+        y_hat = self.discriminator.forward(self.fake_imgs)
+        # print(y_hat)
+        g_loss = self.adversarial_loss(self.discriminator.forward(self.fake_imgs), misleading_labels)
+        
+        self.log("g_loss", g_loss, prog_bar=True)
+        
+        # Backward loss
+        self.manual_backward(g_loss)
+        
+        optimizer_g.step()
+        optimizer_g.zero_grad()
+        
+        self.untoggle_optimizer(optimizer_g)
+
+        # ---- Train discriminator --------------------------------------------
+        #      Discriminator use optimizer #1
+        #      We try to make the difference between fake images and real ones 
+        #
+        self.toggle_optimizer(optimizer_d)
+
+        # These images are reals
+        real_labels = torch.ones(batch_size, 1)
+        real_labels = real_labels.type_as(imgs)
+        pred_labels = self.discriminator.forward(imgs)
+
+        real_loss   = self.adversarial_loss(pred_labels, real_labels)
+
+        # These images are fake
+        fake_imgs   = self.generator.forward(z)
+        fake_labels = torch.zeros(batch_size, 1)
+        fake_labels = fake_labels.type_as(imgs)
+
+        fake_loss   = self.adversarial_loss(self.discriminator(fake_imgs.detach()), fake_labels)
+
+        # Discriminator loss is the average
+        d_loss = (real_loss + fake_loss) / 2
+        
+        self.log("d_loss", d_loss, prog_bar=True)
+ 
+        # Backward
+        self.manual_backward(d_loss)
+        
+        optimizer_d.step()
+        optimizer_d.zero_grad()
+        
+        self.untoggle_optimizer(optimizer_d)
+ 
+
+
+    def configure_optimizers(self):
+
+        lr = self.hparams.lr
+        b1 = self.hparams.b1
+        b2 = self.hparams.b2
+
+        # With a GAN, we need 2 separate optimizer.
+        # opt_g = torch.optim.Adam(self.generator.parameters(),     lr=lr, betas=(b1, b2))
+        # opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr, betas=(b1, b2),)
+        opt_g = torch.optim.Adam(self.generator.parameters(),     lr=lr)
+        opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr)
+        return [opt_g, opt_d], []
+
+
+    def on_train_epoch_end(self):
+
+        # ---- Log Graph
+        #
+        if(self.current_epoch==1):
+            sampleImg=torch.rand((1,28,28,1))
+            sampleImg=sampleImg.type_as(self.generator.model[0].weight)
+            self.logger.experiment.add_graph(self.discriminator,sampleImg)
+
+        # ---- Log some of these images
+        #
+        z = torch.randn(self.hparams.batch_size, self.hparams.latent_dim)
+        z = z.type_as(self.generator.model[0].weight)
+        sample_imgs = self.generator(z)
+        sample_imgs = sample_imgs.permute(0, 3, 1, 2) # from NHWC to NCHW
+        grid = torchvision.utils.make_grid(tensor=sample_imgs, nrow=12, )
+        self.logger.experiment.add_image(f"Generated images", grid,self.current_epoch)