.gitignore

 .ipynb_checkpoints
+.vscode
 */.ipynb_checkpoints/*
 __pycache__
 */__pycache__/*
+campains/
+*==*==.ipynb
+!*==done==.ipynb
+stderr.txt
+stdout.txt
+debug.log
 run/
 figs/
+data/
 GTSRB/data
+GTSRB/05-full_convolutions.py
 IMDB/data
 MNIST/data
+SYNOP/data
 VAE/data
-BHPD/data/*
-!BHPD/data/BostonHousing.csv
+BHPD/data
+AE/data

AE.Keras3/01-Prepare-MNIST-dataset.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3AE1] - Prepare a noisy MNIST dataset
+<!-- DESC --> Episode 1: Preparation of a noisy MNIST dataset
+
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Prepare a MNIST noisy dataset, usable with our denoiser autoencoder (duration : <50s)
+
+## What we're going to do :
+
+ - Load original MNIST dataset
+ - Adding noise, a lot !
+ - Save it :-)
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init and set parameters
+### 1.1 - Init python
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+import sys
+
+from skimage import io
+from skimage.util import random_noise
+
+import modules.MNIST
+from modules.MNIST     import MNIST
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3AE1')
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Parameters
+`prepared_dataset` : Filename of the future prepared dataset (example : ./data/mnist-noisy.h5)\
+`scale` : Dataset scale. 1 mean 100% of the dataset - set 0.1 for tests\
+`progress_verbosity`: Verbosity of progress bar: 0=silent, 1=progress bar, 2=One line
+
+%% Cell type:code id: tags:
+
+``` python
+prepared_dataset   = './data/mnist-noisy.h5'
+scale              = 1
+progress_verbosity = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('prepared_dataset', 'scale', 'progress_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Get original dataset
+We load :
+`clean_data` : Original and clean images - This is what we will want to ontain at the **output** of the AE
+`class_data` : Image classes - Useless, because the training will be unsupervised
+We'll build :
+`noisy_data` : Noisy images - These are the images that we will give as **input** to our AE
+
+%% Cell type:code id: tags:
+
+``` python
+clean_data, class_data = MNIST.get_origine(scale=scale)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Add noise
+We add noise to the original images (clean_data) to obtain noisy images (noisy_data)
+Need 30-40 seconds
+
+%% Cell type:code id: tags:
+
+``` python
+def noise_it(data):
+    new_data = np.copy(data)
+    for i,image in enumerate(new_data):
+        fidle.utils.update_progress('Add noise : ',i+1,len(data),verbosity=progress_verbosity)
+        image=random_noise(image, mode='gaussian', mean=0, var=0.3)
+        image=random_noise(image, mode='s&p',      amount=0.2, salt_vs_pepper=0.5)
+        image=random_noise(image, mode='poisson')
+        image=random_noise(image, mode='speckle',  mean=0, var=0.1)
+        new_data[i]=image
+    print('Done.')
+    return new_data
+
+# ---- Add noise to input data : x_data
+#
+noisy_data = noise_it(clean_data)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Have a look
+
+%% Cell type:code id: tags:
+
+``` python
+print('Clean dataset (clean_data) : ',clean_data.shape)
+print('Noisy dataset (noisy_data) : ',noisy_data.shape)
+
+fidle.utils.subtitle("Noisy images we'll have in input (or x)")
+fidle.scrawler.images(noisy_data[:5], None, indices='all', columns=5, x_size=3,y_size=3, interpolation=None, save_as='01-noisy')
+fidle.utils.subtitle('Clean images we want to obtain (or y)')
+fidle.scrawler.images(clean_data[:5], None, indices='all', columns=5, x_size=3,y_size=3, interpolation=None, save_as='02-original')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Shuffle dataset
+
+%% Cell type:code id: tags:
+
+``` python
+p = np.random.permutation(len(clean_data))
+clean_data, noisy_data, class_data = clean_data[p], noisy_data[p], class_data[p]
+print('Shuffled.')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Save our prepared dataset
+
+%% Cell type:code id: tags:
+
+``` python
+MNIST.save_prepared_dataset( clean_data, noisy_data, class_data, filename=prepared_dataset )
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

AE.Keras3/02-AE-with-MNIST.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3AE2] - Building and training an AE denoiser model
+<!-- DESC --> Episode 1 : Construction of a denoising autoencoder and training of it with a noisy MNIST dataset.
+
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Understanding and implementing a denoizing **autoencoder** neurals network (AE)
+ - First overview or example of Keras procedural syntax
+
+The calculation needs being important, it is preferable to use a very simple dataset such as MNIST.
+The use of a GPU is often indispensable.
+
+## What we're going to do :
+
+ - Defining an AE model
+ - Build the model
+ - Train it
+ - Follow the learning process with Tensorboard
+
+## Data Terminology :
+- `clean_train`, `clean_test` for noiseless images
+- `noisy_train`, `noisy_test` for noisy images
+- `denoised_test` for denoised images at the output of the model
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init python stuff
+### 1.1 - Init
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+from skimage import io
+import random
+
+from keras import layers
+from keras.callbacks import ModelCheckpoint, TensorBoard
+
+import os
+from importlib import reload
+
+from modules.MNIST          import MNIST
+from modules.ImagesCallback import ImagesCallback
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3AE2')
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Parameters
+`prepared_dataset` : Filename of the prepared dataset (Need 400 Mo, but can be in ./data)
+`dataset_seed` : Random seed for shuffling dataset
+`scale` : % of the dataset to use (1. for 100%)
+`latent_dim` : Dimension of the latent space
+`train_prop` : Percentage for train (the rest being for the test)
+`batch_size` : Batch size
+`epochs` : Nb of epochs for training\
+`fit_verbosity` is the verbosity during training : 0 = silent, 1 = progress bar, 2 = one line per epoch
+
+Note : scale=.2, epoch=20 => 3'30s on a laptop
+
+%% Cell type:code id: tags:
+
+``` python
+prepared_dataset = './data/mnist-noisy.h5'
+dataset_seed     = 123
+
+scale            = .1
+
+latent_dim       = 10
+
+train_prop       = .8
+batch_size       = 128
+epochs           = 20
+fit_verbosity    = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('prepared_dataset', 'dataset_seed', 'scale', 'latent_dim')
+fidle.override('train_prop', 'batch_size', 'epochs', 'fit_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve dataset
+With our MNIST class, in one call, we can reload, rescale, shuffle and split our previously saved dataset :-)
+
+%% Cell type:code id: tags:
+
+``` python
+clean_train,clean_test, noisy_train,noisy_test, _,_ = MNIST.reload_prepared_dataset(scale      = scale,
+                                                                                    train_prop = train_prop,
+                                                                                    seed       = dataset_seed,
+                                                                                    shuffle    = True,
+                                                                                    filename=prepared_dataset )
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Build models
+
+%% Cell type:markdown id: tags:
+
+#### Encoder
+
+%% Cell type:code id: tags:
+
+``` python
+inputs    = keras.Input(shape=(28, 28, 1))
+x         = layers.Conv2D(32, 3, activation="relu", strides=2, padding="same")(inputs)
+x         = layers.Conv2D(64, 3, activation="relu", strides=2, padding="same")(x)
+x         = layers.Flatten()(x)
+x         = layers.Dense(16, activation="relu")(x)
+z         = layers.Dense(latent_dim)(x)
+
+encoder = keras.Model(inputs, z, name="encoder")
+# encoder.summary()
+```
+
+%% Cell type:markdown id: tags:
+
+#### Decoder
+
+%% Cell type:code id: tags:
+
+``` python
+inputs  = keras.Input(shape=(latent_dim,))
+x       = layers.Dense(7 * 7 * 64, activation="relu")(inputs)
+x       = layers.Reshape((7, 7, 64))(x)
+x       = layers.Conv2DTranspose(64, 3, activation="relu", strides=2, padding="same")(x)
+x       = layers.Conv2DTranspose(32, 3, activation="relu", strides=2, padding="same")(x)
+outputs = layers.Conv2DTranspose(1, 3, activation="sigmoid", padding="same")(x)
+
+decoder = keras.Model(inputs, outputs, name="decoder")
+# decoder.summary()
+```
+
+%% Cell type:markdown id: tags:
+
+#### AE
+
+%% Cell type:code id: tags:
+
+``` python
+inputs    = keras.Input(shape=(28, 28, 1))
+
+latents   = encoder(inputs)
+outputs   = decoder(latents)
+
+ae = keras.Model(inputs,outputs, name="ae")
+
+ae.compile(optimizer=keras.optimizers.Adam(), loss='binary_crossentropy')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Train
+20' on a CPU
+1'12 on a GPU (V100, IDRIS)
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Callback : Images
+#
+fidle.utils.mkdir( run_dir + '/images')
+filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'
+callback_images = ImagesCallback(filename, x=clean_test[:5], encoder=encoder,decoder=decoder)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+chrono = fidle.Chrono()
+chrono.start()
+
+history = ae.fit(noisy_train, clean_train,
+                 batch_size      = batch_size,
+                 epochs          = epochs,
+                 verbose         = fit_verbosity,
+                 validation_data = (noisy_test, clean_test),
+                 callbacks       = [ callback_images ] )
+
+chrono.show()
+```
+
+%% Cell type:markdown id: tags:
+
+Save model
+
+%% Cell type:code id: tags:
+
+``` python
+os.makedirs(f'{run_dir}/models', exist_ok=True)
+
+encoder.save(f'{run_dir}/models/encoder.keras')
+decoder.save(f'{run_dir}/models/decoder.keras')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - History
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history(history,  plot={'loss':['loss','val_loss']}, save_as='01-history')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Denoising progress
+
+%% Cell type:code id: tags:
+
+``` python
+imgs=[]
+for epoch in range(0,epochs,2):
+    for i in range(5):
+        filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'.format(epoch=epoch, i=i)
+        img      = io.imread(filename)
+        imgs.append(img)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='02-original-real')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='03-original-noisy')
+
+fidle.utils.subtitle('Evolution during the training period (denoised_test) :')
+fidle.scrawler.images(imgs, None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, y_padding=0.1, save_as='04-learning')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Evaluation
+**Note :** We will use the following data:\
+`clean_train`, `clean_test` for noiseless images \
+`noisy_train`, `noisy_test` for noisy images\
+`denoised_test` for denoised images at the output of the model
+
+### 7.1 - Reload model
+
+%% Cell type:code id: tags:
+
+``` python
+encoder = keras.models.load_model(f'{run_dir}/models/encoder.keras')
+decoder = keras.models.load_model(f'{run_dir}/models/decoder.keras')
+
+inputs    = keras.Input(shape=(28, 28, 1))
+
+latents   = encoder(inputs)
+outputs   = decoder(latents)
+
+ae_reloaded = keras.Model(inputs,outputs, name="ae")
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.2 - Let's make a prediction
+
+%% Cell type:code id: tags:
+
+``` python
+denoised_test = ae_reloaded.predict(noisy_test, verbose=0)
+
+print('Denoised images   (denoised_test) shape : ',denoised_test.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.3 - Denoised images
+
+%% Cell type:code id: tags:
+
+``` python
+i=random.randint(0,len(denoised_test)-8)
+j=i+8
+
+fidle.utils.subtitle('Noisy test images (input):')
+fidle.scrawler.images(noisy_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='05-test-noisy')
+
+fidle.utils.subtitle('Denoised images (output):')
+fidle.scrawler.images(denoised_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='06-test-predict')
+
+fidle.utils.subtitle('Real test images :')
+fidle.scrawler.images(clean_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='07-test-real')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

AE.Keras3/03-AE-with-MNIST-post.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3AE3] - Playing with our denoiser model
+<!-- DESC --> Episode 2 : Using the previously trained autoencoder to denoise data
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Retrieve and use our denoiser model
+
+
+## What we're going to do :
+
+ - Reload our dataset and saved best model
+ - Encode/decode some test images (neved used, never seen by the model)
+
+## Data Terminology :
+- `clean_train`, `clean_test` for noiseless images
+- `noisy_train`, `noisy_test` for noisy images
+- `denoised_test` for denoised images at the output of the model
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init python stuff
+### 1.1 - Init
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+import matplotlib.pyplot as plt
+import random
+
+from modules.MNIST import MNIST
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3AE3')
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Parameters
+These **parameters must be identical** to those used during the training in order to have the **same dataset**.\
+`prepared_dataset` : Filename of the prepared dataset (Need 400 Mo, but can be in ./data)
+`dataset_seed` : Random seed for shuffling dataset
+`scale` : % of the dataset to use (1. for 100%)
+`train_prop` : Percentage for train (the rest being for the test)
+
+%% Cell type:code id: tags:
+
+``` python
+prepared_dataset = './data/mnist-noisy.h5'
+saved_models     = './run/K3AE2/models'
+dataset_seed     = 123
+scale            = 1
+train_prop       = .8
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('prepared_dataset', 'dataset_seed', 'scale', 'train_prop')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve dataset
+With our MNIST class, in one call, we can reload, rescale, shuffle and split our previously saved dataset :-)
+**Important :** Make sure that the **digest is identical** to the one used during the training !\
+See : [AE2 / Step 2 - Retrieve dataset](./02-AE-with-MNIST.ipynb#Step-2---Retrieve-dataset)
+
+%% Cell type:code id: tags:
+
+``` python
+clean_train,clean_test, noisy_train,noisy_test, _,_ = MNIST.reload_prepared_dataset(scale      = scale,
+                                                                                    train_prop = train_prop,
+                                                                                    seed       = dataset_seed,
+                                                                                    shuffle    = True,
+                                                                                    filename=prepared_dataset )
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Evaluation
+**Note :** We will use the following data:\
+`clean_train`, `clean_test` for noiseless images \
+`noisy_train`, `noisy_test` for noisy images\
+`denoised_test` for denoised images at the output of the model
+
+### 3.1 - Reload our best model
+
+%% Cell type:code id: tags:
+
+``` python
+# model = keras.models.load_model(f'{saved_models}/model.keras')
+
+encoder = keras.models.load_model(f'{saved_models}/encoder.keras')
+decoder = keras.models.load_model(f'{saved_models}/decoder.keras')
+
+inputs    = keras.Input(shape=(28, 28, 1))
+
+latents   = encoder(inputs)
+outputs   = decoder(latents)
+
+model = keras.Model(inputs,outputs, name="ae")
+```
+
+%% Cell type:markdown id: tags:
+
+### 3.2 - Let's make a prediction
+
+%% Cell type:code id: tags:
+
+``` python
+from tabnanny import verbose
+
+
+denoised_test = model.predict(noisy_test,verbose=0)
+
+print('Denoised images   (denoised_test) shape : ',denoised_test.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 3.3 - Denoised images
+
+%% Cell type:code id: tags:
+
+``` python
+i=random.randint(0,len(denoised_test)-8)
+j=i+8
+
+fidle.utils.subtitle('Noisy test images (input):')
+fidle.scrawler.images(noisy_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='05-test-noisy')
+
+fidle.utils.subtitle('Denoised images (output):')
+fidle.scrawler.images(denoised_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='06-test-predict')
+
+fidle.utils.subtitle('Real test images :')
+fidle.scrawler.images(clean_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='07-test-real')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Looking at the latent space
+### 4.1 - Getting clean data and class
+
+%% Cell type:code id: tags:
+
+``` python
+clean_data,_, _,_, class_data,_ = MNIST.reload_prepared_dataset(scale      = 1,
+                                                                train_prop = 1,
+                                                                seed       = dataset_seed,
+                                                                shuffle    = False,
+                                                                filename   = prepared_dataset )
+```
+
+%% Cell type:markdown id: tags:
+
+### 4.2 - Retrieve encoder
+
+%% Cell type:code id: tags:
+
+``` python
+encoder=model.get_layer('encoder')
+```
+
+%% Cell type:markdown id: tags:
+
+### 4.3 Showing latent space
+Here is the digit distribution in the latent space
+
+%% Cell type:code id: tags:
+
+``` python
+n_show = min( 20000, len(clean_data) )
+
+# ---- Select images
+
+x_show, y_show = fidle.utils.pick_dataset(clean_data, class_data, n=n_show)
+
+# ---- Get latent points
+
+z = encoder.predict(x_show)
+
+# ---- Show them
+
+fig = plt.figure(figsize=(14, 10))
+plt.scatter(z[:, 0] , z[:, 1], c=y_show, cmap= 'tab10', alpha=0.5, s=30)
+plt.colorbar()
+fidle.scrawler.save_fig('08-Latent-space')
+plt.show()
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

AE.Keras3/04-ExtAE-with-MNIST.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3AE4] - Denoiser and classifier model
+<!-- DESC --> Episode 4 : Construction of a denoiser and classifier model
+
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Building a multiple output model, able to **denoise** and **classify**
+ - Understanding a more **advanced programming model**
+
+The calculation needs being important, it is preferable to use a very simple dataset such as MNIST.
+The use of a GPU is often indispensable.
+
+## What we're going to do :
+
+ - Defining a multiple output model using Keras procedural programing model
+ - Build the model
+ - Train it
+ - Follow the learning process
+
+## Data Terminology :
+- `clean_train`, `clean_test` for noiseless images
+- `noisy_train`, `noisy_test` for noisy images
+- `class_train`, `class_test` for the classes to which the images belong
+- `denoised_test` for denoised images at the output of the model
+- `classcat_test` for class prediction in model output (is a softmax)
+- `classid_test` class prediction (ie: argmax of classcat_test)
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init python stuff
+### 1.1 - Init
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+from skimage import io
+import random
+
+from modules.AE4_builder    import AE4_builder
+from modules.MNIST          import MNIST
+from modules.ImagesCallback import ImagesCallback
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3AE4')
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Parameters
+`prepared_dataset` : Filename of the prepared dataset (Need 400 Mo, but can be in ./data)
+`dataset_seed` : Random seed for shuffling dataset. 'None' mean using /dev/urandom
+`scale` : % of the dataset to use (1. for 100%)
+`latent_dim` : Dimension of the latent space
+`train_prop` : Percentage for train (the rest being for the test)
+`batch_size` : Batch size
+`epochs` : Nb of epochs for training\
+`fit_verbosity` is the verbosity during training : 0 = silent, 1 = progress bar, 2 = one line per epoch
+
+scale=0.1, epochs=20  => 2' on a laptop
+
+%% Cell type:code id: tags:
+
+``` python
+prepared_dataset = './data/mnist-noisy.h5'
+dataset_seed     = None
+
+scale            = .1
+
+train_prop       = .8
+batch_size       = 128
+epochs           = 10
+fit_verbosity    = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('prepared_dataset', 'dataset_seed', 'scale')
+fidle.override('train_prop', 'batch_size', 'epochs', 'fit_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve dataset
+With our MNIST class, in one call, we can reload, rescale, shuffle and split our previously saved dataset :-)
+
+%% Cell type:code id: tags:
+
+``` python
+clean_train,clean_test, noisy_train,noisy_test, class_train,class_test = MNIST.reload_prepared_dataset(
+                                                                                    scale      = scale,
+                                                                                    train_prop = train_prop,
+                                                                                    seed       = dataset_seed,
+                                                                                    shuffle    = True,
+                                                                                    filename   = prepared_dataset )
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Build models
+
+%% Cell type:code id: tags:
+
+``` python
+builder = AE4_builder( ae={ 'latent_dim':10 }, cnn = { 'lc1':8, 'lc2':16, 'ld':100 } )
+
+model = builder.create_model()
+```
+
+%% Cell type:code id: tags:
+
+``` python
+model.compile(optimizer='rmsprop',
+              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
+              loss_weights={'ae':1., 'classifier':1.},
+              metrics={'classifier':'accuracy'} )
+```
+
+%% Cell type:code id: tags:
+
+``` python
+# keras.utils.plot_model(model, "multi_input_and_output_model.png", show_shapes=True)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Train
+20' on a CPU
+1'12 on a GPU (V100, IDRIS)
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Callback : Images
+#
+fidle.utils.mkdir( run_dir + '/images')
+filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'
+
+encoder = model.get_layer('ae').get_layer('encoder')
+decoder = model.get_layer('ae').get_layer('decoder')
+
+callback_images = ImagesCallback(filename, x=clean_test[:5], encoder=encoder,decoder=decoder)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+chrono = fidle.Chrono()
+chrono.start()
+
+history = model.fit(noisy_train, [clean_train, class_train],
+                 batch_size      = batch_size,
+                 epochs          = epochs,
+                 verbose         = fit_verbosity,
+                 validation_data = (noisy_test, [clean_test, class_test]),
+                 callbacks       = [ callback_images ]  )
+
+chrono.show()
+```
+
+%% Cell type:markdown id: tags:
+
+Save model weights
+
+%% Cell type:code id: tags:
+
+``` python
+os.makedirs(f'{run_dir}/models', exist_ok=True)
+
+model.save_weights(f'{run_dir}/models/model.weights.h5')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - History
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history(history,  plot={'Loss':['loss', 'val_loss'],
+                                 'Accuracy':['classifier_accuracy','val_classifier_accuracy']}, save_as='01-history')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Denoising progress
+
+%% Cell type:code id: tags:
+
+``` python
+imgs=[]
+for epoch in range(0,epochs,4):
+    for i in range(5):
+        filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'.format(epoch=epoch, i=i)
+        img      = io.imread(filename)
+        imgs.append(img)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='02-original-real')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='03-original-noisy')
+
+fidle.utils.subtitle('Evolution during the training period (denoised_test) :')
+fidle.scrawler.images(imgs, None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, y_padding=0.1, save_as='04-learning')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Evaluation
+**Note :** We will use the following data:\
+`clean_train`, `clean_test` for noiseless images \
+`noisy_train`, `noisy_test` for noisy images\
+`class_train`, `class_test` for the classes to which the images belong \
+`denoised_test` for denoised images at the output of the model\
+`classcat_test` for class prediction in model output (is a softmax)\
+`classid_test` class prediction (ie: argmax of classcat_test)
+
+### 7.1 - Reload our model (weights)
+
+%% Cell type:code id: tags:
+
+``` python
+builder = AE4_builder( ae={ 'latent_dim':10 }, cnn = { 'lc1':8, 'lc2':16, 'ld':100 } )
+
+model = builder.create_model()
+
+model.load_weights(f'{run_dir}/models/model.weights.h5')
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.2 - Let's make a prediction
+Note that our model will returns 2 outputs : **denoised images** from output 1 and **class prediction** from output 2
+
+%% Cell type:code id: tags:
+
+``` python
+outputs = model.predict(noisy_test, verbose=0)
+
+denoised = outputs['ae']
+classcat = outputs['classifier']
+
+print('Denoised images   (denoised_test) shape : ', denoised.shape)
+print('Predicted classes (classcat_test) shape : ', classcat.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.3 - Denoised images
+
+%% Cell type:code id: tags:
+
+``` python
+i=random.randint(0,len(denoised)-8)
+j=i+8
+
+fidle.utils.subtitle('Noisy test images (input):')
+fidle.scrawler.images(noisy_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='05-test-noisy')
+
+fidle.utils.subtitle('Denoised images (output):')
+fidle.scrawler.images(denoised[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='06-test-predict')
+
+fidle.utils.subtitle('Real test images :')
+fidle.scrawler.images(clean_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='07-test-real')
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.4 - Class prediction
+Note: The evaluation requires the noisy images as input (noisy_test) and the 2 expected outputs:
+ - the images without noise (clean_test)
+ - the classes (class_test)
+
+%% Cell type:code id: tags:
+
+``` python
+# We need to (re)compile our resurrected model (to specify loss and metrics)
+#
+model.compile(optimizer='rmsprop',
+              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
+              loss_weights={'ae':1., 'classifier':1.},
+              metrics={'classifier':'accuracy'} )
+
+
+# Get an evaluation
+#
+score = model.evaluate(noisy_test, [clean_test, class_test], verbose=0)
+
+# And show results
+#
+fidle.utils.subtitle("Accuracy :")
+print(f'Classification accuracy : {score[1]:4.4f}')
+
+fidle.utils.subtitle("Few examples :")
+classid_test  = np.argmax(classcat, axis=-1)
+fidle.scrawler.images(noisy_test, class_test, range(0,200), columns=12, x_size=1, y_size=1, y_pred=classid_test, save_as='04-predictions')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

AE.Keras3/05-ExtAE-with-MNIST.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3AE5] - Advanced denoiser and classifier model
+<!-- DESC --> Episode 5 : Construction of an advanced denoiser and classifier model
+
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Building a multiple output model, able to **denoise** and **classify**
+ - Understanding a more complex **advanced programming model**
+
+The calculation needs being important, it is preferable to use a very simple dataset such as MNIST.
+The use of a GPU is often indispensable.
+
+## What we're going to do :
+
+ - Defining a multiple output model using Keras procedural programing model
+ - Build the model
+ - Train it
+ - Follow the learning process
+
+## Data Terminology :
+- `clean_train`, `clean_test` for noiseless images
+- `noisy_train`, `noisy_test` for noisy images
+- `class_train`, `class_test` for the classes to which the images belong
+- `denoised_test` for denoised images at the output of the model
+- `classcat_test` for class prediction in model output (is a softmax)
+- `classid_test` class prediction (ie: argmax of classcat_test)
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init python stuff
+### 1.1 - Init
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+from skimage import io
+import random
+
+from modules.AE5_builder    import AE5_builder
+from modules.MNIST          import MNIST
+from modules.ImagesCallback import ImagesCallback
+
+import fidle
+
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3AE5')
+```
+
+%% Cell type:markdown id: tags:
+
+### 1.2 - Parameters
+`prepared_dataset` : Filename of the prepared dataset (Need 400 Mo, but can be in ./data)
+`dataset_seed` : Random seed for shuffling dataset
+`scale` : % of the dataset to use (1. for 100%)
+`latent_dim` : Dimension of the latent space
+`train_prop` : Percentage for train (the rest being for the test)
+`batch_size` : Batch size
+`epochs` : Nb of epochs for training\
+`fit_verbosity` is the verbosity during training : 0 = silent, 1 = progress bar, 2 = one line per epoch
+
+%% Cell type:code id: tags:
+
+``` python
+prepared_dataset = './data/mnist-noisy.h5'
+dataset_seed     = None
+
+scale            = .1
+
+train_prop       = .8
+batch_size       = 128
+epochs           = 10
+fit_verbosity    = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('prepared_dataset', 'dataset_seed', 'scale')
+fidle.override('train_prop', 'batch_size', 'epochs', 'fit_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve dataset
+With our MNIST class, in one call, we can reload, rescale, shuffle and split our previously saved dataset :-)
+
+%% Cell type:code id: tags:
+
+``` python
+clean_train,clean_test, noisy_train,noisy_test, class_train,class_test = MNIST.reload_prepared_dataset(
+                                                                                    scale      = scale,
+                                                                                    train_prop = train_prop,
+                                                                                    seed       = dataset_seed,
+                                                                                    shuffle    = True,
+                                                                                    filename   = prepared_dataset )
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Build model
+
+%% Cell type:code id: tags:
+
+``` python
+builder = AE5_builder( ae   = { 'latent_dim':10 },
+                       cnn1 = { 'lc1':8,  'lc2':16, 'ld':100 },
+                       cnn2 = { 'lc1':32, 'lc2':64, 'ld':50 } )
+
+model = builder.create_model()
+```
+
+%% Cell type:code id: tags:
+
+``` python
+model.compile(optimizer='rmsprop',
+              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
+              loss_weights={'ae':1., 'classifier':1.},
+              metrics={'classifier':'accuracy'} )
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Train
+20' on a CPU
+1'30 on a GPU (V100, IDRIS)
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Callback : Images
+#
+fidle.utils.mkdir( run_dir + '/images')
+filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'
+
+encoder = model.get_layer('ae').get_layer('encoder')
+decoder = model.get_layer('ae').get_layer('decoder')
+
+callback_images = ImagesCallback(filename, x=clean_test[:5], encoder=encoder,decoder=decoder)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+chrono = fidle.Chrono()
+chrono.start()
+
+history = model.fit(noisy_train, [clean_train, class_train],
+                 batch_size      = batch_size,
+                 epochs          = epochs,
+                 verbose         = fit_verbosity,
+                 validation_data = (noisy_test, [clean_test, class_test]),
+                 callbacks       = [ callback_images ] )
+
+
+chrono.show()
+```
+
+%% Cell type:markdown id: tags:
+
+Save model weights
+
+%% Cell type:code id: tags:
+
+``` python
+os.makedirs(f'{run_dir}/models', exist_ok=True)
+
+model.save_weights(f'{run_dir}/models/model.weights.h5')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - History
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history(history,  plot={'Loss':['loss', 'val_loss'],
+                                 'Accuracy':['classifier_accuracy','val_classifier_accuracy']}, save_as='01-history')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Denoising progress
+
+%% Cell type:code id: tags:
+
+``` python
+imgs=[]
+for epoch in range(0,epochs,4):
+    for i in range(5):
+        filename = run_dir + '/images/image-{epoch:03d}-{i:02d}.jpg'.format(epoch=epoch, i=i)
+        img      = io.imread(filename)
+        imgs.append(img)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='02-original-real')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as='03-original-noisy')
+
+fidle.utils.subtitle('Evolution during the training period (denoised_test) :')
+fidle.scrawler.images(imgs, None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, y_padding=0.1, save_as='04-learning')
+
+fidle.utils.subtitle('Noisy images (noisy_test) :')
+fidle.scrawler.images(noisy_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+
+fidle.utils.subtitle('Real images (clean_test) :')
+fidle.scrawler.images(clean_test[:5], None, indices='all', columns=5, x_size=2,y_size=2, interpolation=None, save_as=None)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Evaluation
+**Note :** We will use the following data:\
+`clean_train`, `clean_test` for noiseless images \
+`noisy_train`, `noisy_test` for noisy images\
+`class_train`, `class_test` for the classes to which the images belong \
+`denoised_test` for denoised images at the output of the model\
+`classcat_test` for class prediction in model output (is a softmax)\
+`classid_test` class prediction (ie: argmax of classcat_test)
+
+### 7.1 - Reload our best model
+
+%% Cell type:code id: tags:
+
+``` python
+builder = AE5_builder( ae   = { 'latent_dim':10 },
+                       cnn1 = { 'lc1':8,  'lc2':16, 'ld':100 },
+                       cnn2 = { 'lc1':32, 'lc2':64, 'ld':50 } )
+
+model = builder.create_model()
+
+model.load_weights(f'{run_dir}/models/model.weights.h5')
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.2 - Let's make a prediction
+Note that our model will returns 2 outputs : **denoised images** from output 1 and **class prediction** from output 2
+
+%% Cell type:code id: tags:
+
+``` python
+outputs = model.predict(noisy_test, verbose=0)
+
+denoised = outputs['ae']
+classcat = outputs['classifier']
+
+print('Denoised images   (denoised_test) shape : ', denoised.shape)
+print('Predicted classes (classcat_test) shape : ', classcat.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.3 - Denoised images
+
+%% Cell type:code id: tags:
+
+``` python
+i=random.randint(0,len(denoised)-8)
+j=i+8
+
+fidle.utils.subtitle('Noisy test images (input):')
+fidle.scrawler.images(noisy_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='05-test-noisy')
+
+fidle.utils.subtitle('Denoised images (output):')
+fidle.scrawler.images(denoised[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='06-test-predict')
+
+fidle.utils.subtitle('Real test images :')
+fidle.scrawler.images(clean_test[i:j], None, indices='all', columns=8, x_size=2,y_size=2, interpolation=None, save_as='07-test-real')
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.4 - Class prediction
+Note: The evaluation requires the noisy images as input (noisy_test) and the 2 expected outputs:
+ - the images without noise (clean_test)
+ - the classes (class_test)
+
+%% Cell type:code id: tags:
+
+``` python
+# We need to (re)compile our resurrected model (to specify loss and metrics)
+#
+model.compile(optimizer='rmsprop',
+              loss={'ae':'binary_crossentropy', 'classifier':'sparse_categorical_crossentropy'},
+              loss_weights={'ae':1., 'classifier':1.},
+              metrics={'classifier':'accuracy'} )
+
+
+# Get an evaluation
+#
+score = model.evaluate(noisy_test, [clean_test, class_test], verbose=0)
+
+# And show results
+#
+fidle.utils.subtitle("Accuracy :")
+print(f'Classification accuracy : {score[1]:4.4f}')
+
+fidle.utils.subtitle("Few examples :")
+classid_test  = np.argmax(classcat, axis=-1)
+fidle.scrawler.images(noisy_test, class_test, range(0,200), columns=12, x_size=1, y_size=1, y_pred=classid_test, save_as='04-predictions')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

AE.Keras3/modules/AE4_builder.py

+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI 2024 - JL. Parouty
+# ------------------------------------------------------------------
+
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+import keras.layers as layers
+
+
+# ------------------------------------------------------------------
+#   A usefull class to manage our AE model
+# ------------------------------------------------------------------
+
+
+class AE4_builder():
+    
+    version = '0.1'
+    
+    def __init__(self,  ae  = { 'latent_dim':16 },
+                        cnn = { 'lc1':8, 'lc2':16, 'ld':100 }
+                ):
+        
+        self.ae   = ae
+        self.cnn  = cnn
+    
+    
+    
+    def create_encoder(self):
+        
+        latent_dim = self.ae['latent_dim']
+                
+        inputs    = keras.Input(shape=(28,28,1))
+        x         = layers.Conv2D(32, 3, activation="relu", strides=2, padding="same")(inputs)
+        x         = layers.Conv2D(64, 3, activation="relu", strides=2, padding="same")(x)
+        x         = layers.Flatten()(x)
+        x         = layers.Dense(16,     activation="relu")(x)
+        z         = layers.Dense(latent_dim, name='latent')(x)
+
+        encoder = keras.Model(inputs, z, name="encoder")    
+        return encoder
+        
+        
+        
+    def create_decoder(self):
+
+        latent_dim = self.ae['latent_dim']
+
+        inputs  = keras.Input(shape=(latent_dim,))
+        x       = layers.Dense(7 * 7 * 64, activation="relu")(inputs)
+        x       = layers.Reshape((7, 7, 64))(x)
+        x       = layers.Conv2DTranspose(64, 3, activation="relu", strides=2, padding="same")(x)
+        x       = layers.Conv2DTranspose(32, 3, activation="relu", strides=2, padding="same")(x)
+        outputs = layers.Conv2DTranspose(1, 3, activation="sigmoid", padding="same", name='denoiser')(x)
+
+        decoder = keras.Model(inputs, outputs, name="decoder")
+        return decoder
+    
+        
+        
+    def create_cnn(self):
+
+        lc1     = self.cnn['lc1']
+        lc2     = self.cnn['lc2']
+        ld      = self.cnn['ld']
+
+        inputs    = keras.Input(shape=(28, 28, 1))
+
+        x         = keras.layers.Conv2D(lc1, (3,3),  activation='relu')(inputs)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.2)(x)
+
+        x         = keras.layers.Conv2D(lc2, (3,3), activation='relu')(x)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.2)(x)
+
+        x         = keras.layers.Flatten()(x)
+        x         = keras.layers.Dense(ld, activation='relu')(x)
+        x         = keras.layers.Dropout(0.5)(x)
+
+        outputs   = keras.layers.Dense(10, activation='softmax', name='cnn')(x)
+
+        cnn       = keras.Model(inputs, outputs, name='classifier')
+        return cnn
+    
+    
+    
+    def create_model(self):
+        
+        # ---- Recover all elementary bricks
+        #
+        encoder = self.create_encoder()
+        decoder = self.create_decoder()
+        cnn     = self.create_cnn()
+        
+        # ---- Build ae
+        #
+        inputs    = keras.Input(shape=(28, 28, 1))
+        latents   = encoder(inputs)
+        outputs   = decoder(latents)
+
+        ae = keras.Model(inputs,outputs, name='ae')
+
+        # ---- Assembling final model
+        #
+        inputs    = keras.Input(shape=(28, 28, 1))
+
+        denoised  = ae(inputs)
+        classcat  = cnn(inputs)
+
+        model = keras.Model(inputs, outputs={ 'ae':denoised, 'classifier':classcat})
+        
+        return model
\ No newline at end of file

AE.Keras3/modules/AE5_builder.py

+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI 2024 - JL. Parouty
+# ------------------------------------------------------------------
+
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+import keras.layers as layers
+
+
+# ------------------------------------------------------------------
+#   A usefull class to manage our AE model
+# ------------------------------------------------------------------
+
+
+class AE5_builder():
+    
+    version = '0.1'
+    
+    def __init__(self,  ae   = { 'latent_dim':16 },
+                        cnn1 = { 'lc1':8, 'lc2':16, 'ld':100 },
+                        cnn2 = { 'lc1':32, 'lc2':64, 'ld':50 }
+                ):
+        
+        self.ae   = ae
+        self.cnn1  = cnn1
+        self.cnn2  = cnn2
+    
+    
+    
+    def create_encoder(self):
+        
+        latent_dim = self.ae['latent_dim']
+                
+        inputs    = keras.Input(shape=(28,28,1))
+        x         = layers.Conv2D(32, 3, activation="relu", strides=2, padding="same")(inputs)
+        x         = layers.Conv2D(64, 3, activation="relu", strides=2, padding="same")(x)
+        x         = layers.Flatten()(x)
+        x         = layers.Dense(16,     activation="relu")(x)
+        z         = layers.Dense(latent_dim, name='latent')(x)
+
+        encoder = keras.Model(inputs, z, name="encoder")    
+        return encoder
+        
+        
+        
+    def create_decoder(self):
+
+        latent_dim = self.ae['latent_dim']
+
+        inputs  = keras.Input(shape=(latent_dim,))
+        x       = layers.Dense(7 * 7 * 64, activation="relu")(inputs)
+        x       = layers.Reshape((7, 7, 64))(x)
+        x       = layers.Conv2DTranspose(64, 3, activation="relu", strides=2, padding="same")(x)
+        x       = layers.Conv2DTranspose(32, 3, activation="relu", strides=2, padding="same")(x)
+        outputs = layers.Conv2DTranspose(1, 3, activation="sigmoid", padding="same", name='denoiser')(x)
+
+        decoder = keras.Model(inputs, outputs, name="decoder")
+        return decoder
+    
+        
+        
+    def create_cnn1(self):
+
+        lc1     = self.cnn1['lc1']
+        lc2     = self.cnn1['lc2']
+        ld      = self.cnn1['ld']
+
+        inputs    = keras.Input(shape=(28, 28, 1))
+
+        x         = keras.layers.Conv2D(lc1, (3,3),  activation='relu')(inputs)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.2)(x)
+
+        x         = keras.layers.Conv2D(lc2, (3,3), activation='relu')(x)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.2)(x)
+
+        x         = keras.layers.Flatten()(x)
+        x         = keras.layers.Dense(ld, activation='relu')(x)
+        x         = keras.layers.Dropout(0.5)(x)
+
+        outputs   = keras.layers.Dense(10, activation='softmax', name='classifier')(x)
+
+        cnn       = keras.Model(inputs, outputs, name='cnn')
+        return cnn
+    
+    
+    def create_cnn2(self):
+
+        lc1     = self.cnn2['lc1']
+        lc2     = self.cnn2['lc2']
+        ld      = self.cnn2['ld']
+        
+        inputs    = keras.Input(shape=(28, 28, 1))
+
+        x         = keras.layers.Conv2D(lc1, (5,5),  activation='relu')(inputs)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.3)(x)
+
+        x         = keras.layers.Conv2D(lc2, (5,5), activation='relu')(x)
+        x         = keras.layers.MaxPooling2D((2,2))(x)
+        x         = keras.layers.Dropout(0.3)(x)
+
+        x         = keras.layers.Flatten()(x)
+        x         = keras.layers.Dense(ld, activation='relu')(x)
+        outputs   = keras.layers.Dropout(0.3)(x)
+
+        cnn2       = keras.Model(inputs, outputs, name='cnn2')
+        return cnn2
+    
+    
+    
+    
+    def create_model(self):
+        
+        # ---- Recover all elementary bricks
+        #
+        encoder = self.create_encoder()
+        decoder = self.create_decoder()
+        cnn1    = self.create_cnn1()
+        cnn2    = self.create_cnn2()
+        
+        # ---- Build ae
+        #
+        inputs    = keras.Input(shape=(28, 28, 1))
+        latents   = encoder(inputs)
+        outputs   = decoder(latents)
+
+        ae = keras.Model(inputs,outputs, name='ae')
+
+        # ---- Build classifier
+        #
+        branch_1 = cnn1(inputs)
+        branch_2 = cnn2(inputs)
+        
+        x        = keras.layers.concatenate([branch_1,branch_2], axis=1)
+        outputs  = keras.layers.Dense(10, activation='softmax')(x)
+
+        classifier = keras.Model(inputs,outputs, name='classifier')
+
+        # ---- Assembling final model
+        #
+        inputs    = keras.Input(shape=(28, 28, 1))
+
+        denoised  = ae(inputs)
+        classcat  = classifier(inputs)
+        
+        model = keras.Model(inputs, outputs={ 'ae':denoised, 'classifier':classcat})
+        
+        return model
\ No newline at end of file

VAE/modules/callbacks.py → AE.Keras3/modules/ImagesCallback.py

-from tensorflow.keras.callbacks import Callback
+from keras.callbacks import Callback
 import numpy as np
 import matplotlib.pyplot as plt
 
 class ImagesCallback(Callback):
     
    
-    def __init__(self, filename= 'image-{epoch:03d}-{i:02d}.jpg', z_dim=0, decoder=None, nb_images=5):
+    def __init__(self, filename='image-{epoch:03d}-{i:02d}.jpg', 
+                       x=None,
+                       encoder=None, decoder=None):
         self.filename  = filename
-        self.z_dim     = z_dim
+        self.x         = x
+        self.encoder   = encoder
         self.decoder   = decoder
-        self.nb_images = nb_images
-        
+        if len(x)>100:
+            print('***Warning : The number of images is reduced to 100')
+            self.x=x[:100]
         
     def on_epoch_end(self, epoch, logs={}):  
         
-        # ---- Get random latent points
-        
-        z_new   = np.random.normal(size = (self.nb_images,self.z_dim))
+        # ---- Get latent points
+        #
+        z_new  = self.encoder.predict(self.x)
         
         # ---- Predict an image
-        
+        #
         images = self.decoder.predict(np.array(z_new))
         
         # ---- Save images
-        
+        #
         for i,image in enumerate(images):
             
             # ---- Squeeze it if monochrome : (lx,ly,1) -> (lx,ly) 
+            #
             image = image.squeeze()
         
             # ---- Save it
-
+            #
             filename = self.filename.format(epoch=epoch,i=i)
             
             if len(image.shape) == 2:

AE.Keras3/modules/MNIST.py

+
+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/SARI/DEVLOG 2020 - S. Arias, E. Maldonado, JL. Parouty
+# ------------------------------------------------------------------
+# 2.0 version by JL Parouty, feb 2021
+
+import h5py
+import os
+import numpy as np
+from hashlib import blake2b
+import keras
+import keras.datasets.mnist as mnist
+
+
+# ------------------------------------------------------------------
+#   A usefull class to manage our MNIST dataset
+#   This class allows to manage datasets derived from the original MNIST
+# ------------------------------------------------------------------
+
+
+class MNIST():
+    
+    version = '0.1'
+    
+    def __init__(self):
+        pass
+   
+    @classmethod
+    def get_origine(cls, scale=1, normalize=True, expand=True, concatenate=True):
+        """
+        Return original MNIST dataset
+        args:
+            scale       : Proportion of the requested dataset
+            normalize   : Normalize dataset or not (True)
+            expand      : Reshape images as (28,28,1) instead (28,28) (True)
+            concatenate : Concatenate train and test sets (True)
+        returns:
+            x_data,y_data                   if concatenate is False
+            x_train,y_train,x_test,y_test   if concatenate is True
+        """
+
+        # ---- Get data
+        #
+        (x_train, y_train), (x_test, y_test) = mnist.load_data()
+        print('Dataset loaded.')
+        
+        # ---- Normalization
+        #
+        if normalize:
+            x_train = x_train.astype('float32') / 255.
+            x_test  = x_test.astype( 'float32') / 255.
+            print('Normalized.')
+            
+        # ---- Reshape : (28,28) -> (28,28,1)
+        #
+        if expand:
+            x_train = np.expand_dims(x_train, axis=-1)
+            x_test  = np.expand_dims(x_test,  axis=-1)
+            print('Reshaped.')
+
+        # ---- scale
+        #
+        n1 = int(len(x_train)*scale)
+        n2 = int(len(x_test)*scale)
+        x_train = x_train[:n1]
+        y_train = y_train[:n1]
+        x_test  = x_test[:n2]
+        y_test  = y_test[:n2]
+
+        # ---- Concatenate
+        #
+        if concatenate:
+            x_data = np.concatenate([x_train, x_test], axis=0)
+            y_data = np.concatenate([y_train, y_test])
+            print('Concatenate.')
+            print('x shape :', x_data.shape)
+            print('y shape :', y_data.shape)
+            return x_data,y_data
+        else:
+            print('x_train shape :', x_train.shape)
+            print('y_train shape :', y_train.shape)
+            print('x_test  shape :', x_test.shape)
+            print('y_test  shape :', y_test.shape)
+            return x_train,y_train,x_test,y_test
+        
+        
+    @classmethod
+    def save_prepared_dataset(cls, clean_data, noisy_data, class_data, filename='./data/mnist-noisy.h5'):
+        """
+        Save a prepared dataset in a h5 file
+        args:
+            clean_data, noisy_data, class_data : clean, noisy and class dataset
+            filename                      : filename
+        return:
+            None
+        """
+        path=os.path.dirname(filename)
+        os.makedirs(path, mode=0o750, exist_ok=True)
+
+        with h5py.File(filename, "w") as f:
+            f.create_dataset("clean_data", data=clean_data)
+            f.create_dataset("noisy_data", data=noisy_data)
+            f.create_dataset("class_data", data=class_data)
+        print('Saved.')
+        print('clean_data shape is : ',clean_data.shape)
+        print('noisy_data shape is : ',noisy_data.shape)
+        print('class_data shape is : ',class_data.shape)
+            
+            
+    @classmethod    
+    def reload_prepared_dataset(cls, scale=1., train_prop=0.8, shuffle=True, seed=False, filename='./data/mnist-noisy.h5'):
+        """
+        Reload a saved dataset
+        args:
+            scale      : Scale of dataset to use. 1. mean 100% (1.)
+            train_prop : Ratio of train/test
+            shuffle    : Shuffle data if True
+            seed       : Random seed value. False mean no seed, None mean using /dev/urandom (None)
+            filename   : filename of the prepared dataset
+        returns:
+            clean_train,clean_test, noisy_train,noisy_test, class_train,class_test
+        """
+        # ---- Load saved dataset
+        #
+        with  h5py.File(filename,'r') as f:
+            clean_data  = f['clean_data'][:]
+            noisy_data  = f['noisy_data'][:]
+            class_data  = f['class_data'][:]
+        print('Loaded.')
+        
+        # ---- Rescale
+        #
+        n = int(scale*len(clean_data))
+        clean_data, noisy_data, class_data = clean_data[:n], noisy_data[:n], class_data[:n]
+        print(f'rescaled ({scale}).') 
+        
+        # ---- Seed
+        #
+        if seed is not False:
+            np.random.seed(seed)
+            print(f'Seeded ({seed})')
+        
+        # ---- Shuffle
+        #
+        if shuffle:
+            p = np.random.permutation(len(clean_data))
+            clean_data, noisy_data, class_data = clean_data[p], noisy_data[p], class_data[p]
+            print('Shuffled.')
+        
+        # ---- Split
+        #
+        n=int(len(clean_data)*train_prop)
+        clean_train, clean_test = clean_data[:n], clean_data[n:]
+        noisy_train, noisy_test = noisy_data[:n], noisy_data[n:]
+        class_train, class_test = class_data[:n], class_data[n:]
+        print(f'splited ({train_prop}).') 
+
+        # ---- Hash
+        #
+        h = blake2b(digest_size=10)
+        for a in [clean_train,clean_test, noisy_train,noisy_test, class_train,class_test]:
+            h.update(a)
+        
+        print('clean_train shape is : ', clean_train.shape)
+        print('clean_test  shape is : ', clean_test.shape)
+        print('noisy_train shape is : ', noisy_train.shape)
+        print('noisy_test  shape is : ', noisy_test.shape)
+        print('class_train shape is : ', class_train.shape)
+        print('class_test  shape is : ', class_test.shape)
+        print('Blake2b digest is    : ', h.hexdigest())
+        return  clean_train,clean_test, noisy_train,noisy_test, class_train,class_test
\ No newline at end of file

BHPD.Keras3/01-DNN-Regression.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+
+# <!-- TITLE --> [K3BHPD1] - Regression with a Dense Network (DNN)
+<!-- DESC --> Simple example of a regression with the dataset Boston Housing Prices Dataset (BHPD)
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Predicts **housing prices** from a set of house features.
+ - Understanding the **principle** and the **architecture** of a regression with a **dense neural network**
+
+
+The **[Boston Housing Prices Dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html)** consists of price of houses in various places in Boston.
+Alongside with price, the dataset also provide theses informations :
+
+ - CRIM: This is the per capita crime rate by town
+ - ZN: This is the proportion of residential land zoned for lots larger than 25,000 sq.ft
+ - INDUS: This is the proportion of non-retail business acres per town
+ - CHAS: This is the Charles River dummy variable (this is equal to 1 if tract bounds river; 0 otherwise)
+ - NOX: This is the nitric oxides concentration (parts per 10 million)
+ - RM: This is the average number of rooms per dwelling
+ - AGE: This is the proportion of owner-occupied units built prior to 1940
+ - DIS: This is the weighted distances to five Boston employment centers
+ - RAD: This is the index of accessibility to radial highways
+ - TAX: This is the full-value property-tax rate per 10,000 dollars
+ - PTRATIO: This is the pupil-teacher ratio by town
+ - B: This is calculated as 1000(Bk — 0.63)^2, where Bk is the proportion of people of African American descent by town
+ - LSTAT: This is the percentage lower status of the population
+ - MEDV: This is the median value of owner-occupied homes in 1000 dollars
+
+## What we're going to do :
+
+ - Retrieve data
+ - Preparing the data
+ - Build a model
+ - Train the model
+ - Evaluate the result
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Import and init
+
+You can also adjust the verbosity by changing the value of TF_CPP_MIN_LOG_LEVEL :
+- 0 = all messages are logged (default)
+- 1 = INFO messages are not printed.
+- 2 = INFO and WARNING messages are not printed.
+- 3 = INFO , WARNING and ERROR messages are not printed.
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+import matplotlib.pyplot as plt
+import pandas as pd
+import os,sys
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3BHPD1')
+```
+
+%% Cell type:markdown id: tags:
+
+Verbosity during training :
+- 0 = silent
+- 1 = progress bar
+- 2 = one line per epoch
+
+%% Cell type:code id: tags:
+
+``` python
+fit_verbosity = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('fit_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve data
+
+### 2.1 - Option 1  : From Keras
+Boston housing is a famous historic dataset, so we can get it directly from [Keras datasets](https://keras.io/api/datasets)
+
+%% Cell type:code id: tags:
+
+``` python
+# (x_train, y_train), (x_test, y_test) = keras.datasets.boston_housing.load_data(test_split=0.2, seed=113)
+```
+
+%% Cell type:markdown id: tags:
+
+### 2.2 - Option 2 : From a csv file
+More fun !
+
+%% Cell type:code id: tags:
+
+``` python
+data = pd.read_csv(f'{datasets_dir}/BHPD/origine/BostonHousing.csv', header=0)
+
+display(data.head(5).style.format("{0:.2f}").set_caption("Few lines of the dataset :"))
+print('Missing Data : ',data.isna().sum().sum(), '  Shape is : ', data.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Preparing the data
+### 3.1 - Split data
+We will use 70% of the data for training and 30% for validation.
+The dataset is **shuffled** and shared between **learning** and **testing**.
+x will be input data and y the expected output
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Shuffle and Split => train, test
+#
+data       = data.sample(frac=1., axis=0)
+data_train = data.sample(frac=0.7, axis=0)
+data_test  = data.drop(data_train.index)
+
+# ---- Split => x,y (medv is price)
+#
+x_train = data_train.drop('medv',  axis=1)
+y_train = data_train['medv']
+x_test  = data_test.drop('medv',   axis=1)
+y_test  = data_test['medv']
+
+print('Original data shape was : ',data.shape)
+print('x_train : ',x_train.shape, 'y_train : ',y_train.shape)
+print('x_test  : ',x_test.shape,  'y_test  : ',y_test.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 3.2 - Data normalization
+**Note :**
+ - All input data must be normalized, train and test.
+ - To do this we will **subtract the mean** and **divide by the standard deviation**.
+ - But test data should not be used in any way, even for normalization.
+ - The mean and the standard deviation will therefore only be calculated with the train data.
+
+%% Cell type:code id: tags:
+
+``` python
+display(x_train.describe().style.format("{0:.2f}").set_caption("Before normalization :"))
+
+mean = x_train.mean()
+std  = x_train.std()
+x_train = (x_train - mean) / std
+x_test  = (x_test  - mean) / std
+
+display(x_train.describe().style.format("{0:.2f}").set_caption("After normalization :"))
+display(x_train.head(5).style.format("{0:.2f}").set_caption("Few lines of the dataset :"))
+
+x_train, y_train = np.array(x_train), np.array(y_train)
+x_test,  y_test  = np.array(x_test),  np.array(y_test)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Build a model
+About informations about :
+ - [Optimizer](https://keras.io/api/optimizers)
+ - [Activation](https://keras.io/api/layers/activations)
+ - [Loss](https://keras.io/api/losses)
+ - [Metrics](https://keras.io/api/metrics)
+
+%% Cell type:code id: tags:
+
+``` python
+def get_model_v1(shape):
+
+  model = keras.models.Sequential()
+  model.add(keras.layers.Input(shape, name="InputLayer"))
+  model.add(keras.layers.Dense(32, activation='relu', name='Dense_n1'))
+  model.add(keras.layers.Dense(64, activation='relu', name='Dense_n2'))
+  model.add(keras.layers.Dense(32, activation='relu', name='Dense_n3'))
+  model.add(keras.layers.Dense(1, name='Output'))
+
+  model.compile(optimizer = 'adam',
+                loss      = 'mse',
+                metrics   = ['mae', 'mse'] )
+  return model
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Train the model
+### 5.1 - Get it
+
+%% Cell type:code id: tags:
+
+``` python
+model=get_model_v1( (13,) )
+
+model.summary()
+
+# img=keras.utils.plot_model( model, to_file='./run/model.png', show_shapes=True, show_layer_names=True, dpi=96)
+# display(img)
+```
+
+%% Cell type:markdown id: tags:
+
+### 5.2 - Train it
+
+%% Cell type:code id: tags:
+
+``` python
+history = model.fit(x_train,
+                    y_train,
+                    epochs          = 60,
+                    batch_size      = 10,
+                    verbose         = fit_verbosity,
+                    validation_data = (x_test, y_test))
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Evaluate
+### 6.1 - Model evaluation
+MAE =  Mean Absolute Error (between the labels and predictions)
+A mae equal to 3 represents an average error in prediction of $3k.
+
+%% Cell type:code id: tags:
+
+``` python
+score = model.evaluate(x_test, y_test, verbose=0)
+
+print('x_test / loss      : {:5.4f}'.format(score[0]))
+print('x_test / mae       : {:5.4f}'.format(score[1]))
+print('x_test / mse       : {:5.4f}'.format(score[2]))
+```
+
+%% Cell type:markdown id: tags:
+
+### 6.2 - Training history
+What was the best result during our training ?
+
+%% Cell type:code id: tags:
+
+``` python
+df=pd.DataFrame(data=history.history)
+display(df)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+print("min( val_mae ) : {:.4f}".format( min(history.history["val_mae"]) ) )
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history( history, plot={'MSE' :['mse', 'val_mse'],
+                        'MAE' :['mae', 'val_mae'],
+                        'LOSS':['loss','val_loss']}, save_as='01-history')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Make a prediction
+The data must be normalized with the parameters (mean, std) previously used.
+
+%% Cell type:code id: tags:
+
+``` python
+my_data = [ 1.26425925, -0.48522739,  1.0436489 , -0.23112788,  1.37120745,
+       -2.14308942,  1.13489104, -1.06802005,  1.71189006,  1.57042287,
+        0.77859951,  0.14769795,  2.7585581 ]
+real_price = 10.4
+
+my_data=np.array(my_data).reshape(1,13)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+
+predictions = model.predict( my_data, verbose=fit_verbosity )
+print("Prediction : {:.2f} K$".format(predictions[0][0]))
+print("Reality    : {:.2f} K$".format(real_price))
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

BHPD.Keras3/02-DNN-Regression-Premium.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [K3BHPD2] - Regression with a Dense Network (DNN) - Advanced code
+  <!-- DESC -->  A more advanced implementation of the precedent example, using Keras3
+  <!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Predicts **housing prices** from a set of house features.
+ - Understanding the principle and the architecture of a regression with a dense neural network with backup and restore of the trained model.
+
+The **[Boston Housing Prices Dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html)** consists of price of houses in various places in Boston.
+Alongside with price, the dataset also provide these information :
+
+ - CRIM: This is the per capita crime rate by town
+ - ZN: This is the proportion of residential land zoned for lots larger than 25,000 sq.ft
+ - INDUS: This is the proportion of non-retail business acres per town
+ - CHAS: This is the Charles River dummy variable (this is equal to 1 if tract bounds river; 0 otherwise)
+ - NOX: This is the nitric oxides concentration (parts per 10 million)
+ - RM: This is the average number of rooms per dwelling
+ - AGE: This is the proportion of owner-occupied units built prior to 1940
+ - DIS: This is the weighted distances to five Boston employment centers
+ - RAD: This is the index of accessibility to radial highways
+ - TAX: This is the full-value property-tax rate per 10,000 dollars
+ - PTRATIO: This is the pupil-teacher ratio by town
+ - B: This is calculated as 1000(Bk — 0.63)^2, where Bk is the proportion of people of African American descent by town
+ - LSTAT: This is the percentage lower status of the population
+ - MEDV: This is the median value of owner-occupied homes in 1000 dollars
+
+## What we're going to do :
+
+ - (Retrieve data)
+ - (Preparing the data)
+ - (Build a model)
+ - Train and save the model
+ - Restore saved model
+ - Evaluate the model
+ - Make some predictions
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Import and init
+
+You can also adjust the verbosity by changing the value of TF_CPP_MIN_LOG_LEVEL :
+- 0 = all messages are logged (default)
+- 1 = INFO messages are not printed.
+- 2 = INFO and WARNING messages are not printed.
+- 3 = INFO , WARNING and ERROR messages are not printed.
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+os.environ['KERAS_BACKEND'] = 'torch'
+
+import keras
+
+import numpy as np
+import matplotlib.pyplot as plt
+import pandas as pd
+import os,sys
+
+from IPython.display import Markdown
+from importlib import reload
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('K3BHPD2')
+```
+
+%% Cell type:markdown id: tags:
+
+Verbosity during training :
+- 0 = silent
+- 1 = progress bar
+- 2 = one line per epoch
+
+%% Cell type:code id: tags:
+
+``` python
+fit_verbosity = 1
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('fit_verbosity')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve data
+
+### 2.1 - Option 1  : From Keras
+Boston housing is a famous historic dataset, so we can get it directly from [Keras datasets](https://keras.io/api/datasets)
+
+%% Cell type:code id: tags:
+
+``` python
+# (x_train, y_train), (x_test, y_test) = keras.datasets.boston_housing.load_data(test_split=0.2, seed=113)
+```
+
+%% Cell type:markdown id: tags:
+
+### 2.2 - Option 2 : From a csv file
+More fun !
+
+%% Cell type:code id: tags:
+
+``` python
+data = pd.read_csv(f'{datasets_dir}/BHPD/origine/BostonHousing.csv', header=0)
+
+display(data.head(5).style.format("{0:.2f}"))
+print('Missing Data : ',data.isna().sum().sum(), '  Shape is : ', data.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Preparing the data
+### 3.1 - Split data
+We will use 80% of the data for training and 20% for validation.
+x will be input data and y the expected output
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Split => train, test
+#
+data       = data.sample(frac=1., axis=0)
+data_train = data.sample(frac=0.7, axis=0)
+data_test  = data.drop(data_train.index)
+
+# ---- Split => x,y (medv is price)
+#
+x_train = data_train.drop('medv',  axis=1)
+y_train = data_train['medv']
+x_test  = data_test.drop('medv',   axis=1)
+y_test  = data_test['medv']
+
+print('Original data shape was : ',data.shape)
+print('x_train : ',x_train.shape, 'y_train : ',y_train.shape)
+print('x_test  : ',x_test.shape,  'y_test  : ',y_test.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 3.2 - Data normalization
+**Note :**
+ - All input data must be normalized, train and test.
+ - To do this we will subtract the mean and divide by the standard deviation.
+ - But test data should not be used in any way, even for normalization.
+ - The mean and the standard deviation will therefore only be calculated with the train data.
+
+%% Cell type:code id: tags:
+
+``` python
+display(x_train.describe().style.format("{0:.2f}").set_caption("Before normalization :"))
+
+mean = x_train.mean()
+std  = x_train.std()
+x_train = (x_train - mean) / std
+x_test  = (x_test  - mean) / std
+
+display(x_train.describe().style.format("{0:.2f}").set_caption("After normalization :"))
+
+x_train, y_train = np.array(x_train), np.array(y_train)
+x_test,  y_test  = np.array(x_test),  np.array(y_test)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Build a model
+More informations about :
+ - [Optimizer](https://keras.io/api/optimizers)
+ - [Activation](https://keras.io/api/layers/activations)
+ - [Loss](https://keras.io/api/losses)
+ - [Metrics](https://keras.io/api/metrics)
+
+%% Cell type:code id: tags:
+
+``` python
+def get_model_v1(shape):
+
+  model = keras.models.Sequential()
+  model.add(keras.layers.Input(shape, name="InputLayer"))
+  model.add(keras.layers.Dense(64, activation='relu', name='Dense_n1'))
+  model.add(keras.layers.Dense(64, activation='relu', name='Dense_n2'))
+  model.add(keras.layers.Dense(1, name='Output'))
+
+  model.compile(optimizer = 'rmsprop',
+                loss      = 'mse',
+                metrics   = ['mae', 'mse'] )
+  return model
+```
+
+%% Cell type:markdown id: tags:
+
+## 5 - Train the model
+### 5.1 - Get it
+
+%% Cell type:code id: tags:
+
+``` python
+model=get_model_v1( (13,) )
+
+model.summary()
+```
+
+%% Cell type:markdown id: tags:
+
+### 5.2 - Add callback
+
+%% Cell type:code id: tags:
+
+``` python
+os.makedirs(f'{run_dir}/models',   mode=0o750, exist_ok=True)
+save_dir = f'{run_dir}/models/best_model.keras'
+
+savemodel_callback = keras.callbacks.ModelCheckpoint( filepath=save_dir, monitor='val_mae', mode='max', save_best_only=True)
+```
+
+%% Cell type:markdown id: tags:
+
+### 5.3 - Train it
+
+%% Cell type:code id: tags:
+
+``` python
+history = model.fit(x_train,
+                    y_train,
+                    epochs          = 50,
+                    batch_size      = 10,
+                    verbose         = fit_verbosity,
+                    validation_data = (x_test, y_test),
+                    callbacks       = [savemodel_callback])
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Evaluate
+### 6.1 - Model evaluation
+MAE =  Mean Absolute Error (between the labels and predictions)
+A mae equal to 3 represents an average error in prediction of $3k.
+
+%% Cell type:code id: tags:
+
+``` python
+score = model.evaluate(x_test, y_test, verbose=0)
+
+print('x_test / loss      : {:5.4f}'.format(score[0]))
+print('x_test / mae       : {:5.4f}'.format(score[1]))
+print('x_test / mse       : {:5.4f}'.format(score[2]))
+```
+
+%% Cell type:markdown id: tags:
+
+### 6.2 - Training history
+What was the best result during our training ?
+
+%% Cell type:code id: tags:
+
+``` python
+print("min( val_mae ) : {:.4f}".format( min(history.history["val_mae"]) ) )
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history( history, plot={'MSE' :['mse', 'val_mse'],
+                        'MAE' :['mae', 'val_mae'],
+                        'LOSS':['loss','val_loss']}, save_as='01-history')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Restore a model :
+
+%% Cell type:markdown id: tags:
+
+### 7.1 - Reload model
+
+%% Cell type:code id: tags:
+
+``` python
+loaded_model = keras.models.load_model(f'{run_dir}/models/best_model.keras')
+loaded_model.summary()
+print("Loaded.")
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.2 - Evaluate it :
+
+%% Cell type:code id: tags:
+
+``` python
+score = loaded_model.evaluate(x_test, y_test, verbose=0)
+
+print('x_test / loss      : {:5.4f}'.format(score[0]))
+print('x_test / mae       : {:5.4f}'.format(score[1]))
+print('x_test / mse       : {:5.4f}'.format(score[2]))
+```
+
+%% Cell type:markdown id: tags:
+
+### 7.3 - Make a prediction
+
+%% Cell type:code id: tags:
+
+``` python
+my_data = [ 1.26425925, -0.48522739,  1.0436489 , -0.23112788,  1.37120745,
+       -2.14308942,  1.13489104, -1.06802005,  1.71189006,  1.57042287,
+        0.77859951,  0.14769795,  2.7585581 ]
+real_price = 10.4
+
+my_data=np.array(my_data).reshape(1,13)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+predictions = loaded_model.predict( my_data, verbose=fit_verbosity )
+print("Prediction : {:.2f} K$   Reality : {:.2f} K$".format(predictions[0][0], real_price))
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

BHPD.PyTorch/01-DNN-Regression_PyTorch.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+
+# <!-- TITLE --> [PBHPD1] - Regression with a Dense Network (DNN)
+<!-- DESC --> A Simple regression with a Dense Neural Network (DNN) using Pytorch - BHPD dataset
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP), Laurent Risser (CNRS/IMT) -->
+
+## Objectives :
+ - Predicts **housing prices** from a set of house features.
+ - Understanding the **principle** and the **architecture** of a regression with a **dense neural network**
+
+
+The **[Boston Housing Dataset](https://www.cs.toronto.edu/~delve/data/boston/bostonDetail.html)** consists of price of houses in various places in Boston.
+Alongside with price, the dataset also provide theses informations :
+
+ - CRIM: This is the per capita crime rate by town
+ - ZN: This is the proportion of residential land zoned for lots larger than 25,000 sq.ft
+ - INDUS: This is the proportion of non-retail business acres per town
+ - CHAS: This is the Charles River dummy variable (this is equal to 1 if tract bounds river; 0 otherwise)
+ - NOX: This is the nitric oxides concentration (parts per 10 million)
+ - RM: This is the average number of rooms per dwelling
+ - AGE: This is the proportion of owner-occupied units built prior to 1940
+ - DIS: This is the weighted distances to five Boston employment centers
+ - RAD: This is the index of accessibility to radial highways
+ - TAX: This is the full-value property-tax rate per 10,000 dollars
+ - PTRATIO: This is the pupil-teacher ratio by town
+ - B: This is calculated as 1000(Bk — 0.63)^2, where Bk is the proportion of people of African American descent by town
+ - LSTAT: This is the percentage lower status of the population
+ - MEDV: This is the median value of owner-occupied homes in 1000 dollars
+## What we're going to do :
+
+ - Retrieve data
+ - Preparing the data
+ - Build a model
+ - Train the model
+ - Evaluate the result
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Import and init
+
+%% Cell type:code id: tags:
+
+``` python
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.autograd import Variable
+
+
+import numpy as np
+import matplotlib.pyplot as plt
+import sys,os
+
+import pandas as pd
+
+from modules.fidle_pwk_additional import convergence_history_MSELoss
+
+import fidle
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('PBHPD1')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Retrieve data
+
+
+Boston housing is a famous historic dataset, which can be get here: [Boston housing datasets](https://www.kaggle.com/puxama/bostoncsv)
+
+%% Cell type:code id: tags:
+
+``` python
+data = pd.read_csv('./BostonHousing.csv', header=0)
+
+display(data.head(5).style.format("{0:.2f}").set_caption("Few lines of the dataset :"))
+print('Missing Data : ',data.isna().sum().sum(), '  Shape is : ', data.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Preparing the data
+### 3.1 - Split data
+We will use 70% of the data for training and 30% for validation.
+The dataset is **shuffled** and shared between **learning** and **testing**.
+x will be input data and y the expected output
+
+%% Cell type:code id: tags:
+
+``` python
+# ---- Shuffle and Split => train, test
+#
+data_train = data.sample(frac=0.7, axis=0)
+data_test  = data.drop(data_train.index)
+
+# ---- Split => x,y (medv is price)
+#
+x_train = data_train.drop('medv',  axis=1)
+y_train = data_train['medv']
+x_test  = data_test.drop('medv',   axis=1)
+y_test  = data_test['medv']
+
+print('Original data shape was : ',data.shape)
+print('x_train : ',x_train.shape, 'y_train : ',y_train.shape)
+print('x_test  : ',x_test.shape,  'y_test  : ',y_test.shape)
+```
+
+%% Cell type:markdown id: tags:
+
+### 3.2 - Data normalization
+**Note :**
+ - All input data must be normalized, train and test.
+ - To do this we will **subtract the mean** and **divide by the standard deviation**.
+ - But test data should not be used in any way, even for normalization.
+ - The mean and the standard deviation will therefore only be calculated with the train data.
+
+%% Cell type:code id: tags:
+
+``` python
+display(x_train.describe().style.format("{0:.2f}").set_caption("Before normalization :"))
+
+mean = x_train.mean()
+std  = x_train.std()
+x_train = (x_train - mean) / std
+x_test  = (x_test  - mean) / std
+
+display(x_train.describe().style.format("{0:.2f}").set_caption("After normalization :"))
+display(x_train.head(5).style.format("{0:.2f}").set_caption("Few lines of the dataset :"))
+
+x_train, y_train = np.array(x_train), np.array(y_train)
+x_test,  y_test  = np.array(x_test),  np.array(y_test)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 4 - Build a model
+About informations about :
+ - [Optimizer](https://pytorch.org/docs/stable/optim.html)
+ - [Basic neural-network blocks](https://pytorch.org/docs/stable/nn.html)
+ - [Loss](https://pytorch.org/docs/stable/nn.html#loss-functions)
+
+%% Cell type:code id: tags:
+
+``` python
+class model_v1(nn.Module):
+    """
+    Basic fully connected neural-network for tabular data
+    """
+    def __init__(self,num_vars):
+        super(model_v1, self).__init__()
+        self.num_vars=num_vars
+        self.hidden1 = nn.Linear(self.num_vars, 64)
+        self.hidden2 = nn.Linear(64, 64)
+        self.hidden3 = nn.Linear(64, 1)
+
+    def forward(self, x):
+        x = x.view(-1,self.num_vars)   #flatten the observation before using fully-connected layers
+        x = self.hidden1(x)
+        x = F.relu(x)
+        x = self.hidden2(x)
+        x = F.relu(x)
+        x = self.hidden3(x)
+        return x
+
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Train the model
+
+### 5.1 - Stochastic gradient descent strategy to fit the model
+
+%% Cell type:code id: tags:
+
+``` python
+def fit(model,X_train,Y_train,X_test,Y_test, EPOCHS = 5, BATCH_SIZE = 32):
+
+    loss = nn.MSELoss()
+    optimizer = torch.optim.Adam(model.parameters(),lr=1e-3) #lr is the learning rate
+    model.train()
+
+    history=convergence_history_MSELoss()
+
+    history.update(model,X_train,Y_train,X_test,Y_test)
+
+    n=X_train.shape[0] #number of observations in the training data
+
+    #stochastic gradient descent
+    for epoch in range(EPOCHS):
+
+        batch_start=0
+        epoch_shuffler=np.arange(n)
+        np.random.shuffle(epoch_shuffler) #remark that 'utilsData.DataLoader' could be used instead
+
+        while batch_start+BATCH_SIZE < n:
+            #get mini-batch observation
+            mini_batch_observations = epoch_shuffler[batch_start:batch_start+BATCH_SIZE]
+            var_X_batch = Variable(X_train[mini_batch_observations,:]).float()
+            var_Y_batch = Variable(Y_train[mini_batch_observations]).float()
+
+            #gradient descent step
+            optimizer.zero_grad()               #set the parameters gradients to 0
+            Y_pred_batch = model(var_X_batch)   #predict y with the current NN parameters
+
+            curr_loss = loss(Y_pred_batch.view(-1), var_Y_batch.view(-1))  #compute the current loss
+            curr_loss.backward()                         #compute the loss gradient w.r.t. all NN parameters
+            optimizer.step()                             #update the NN parameters
+
+            #prepare the next mini-batch of the epoch
+            batch_start+=BATCH_SIZE
+
+        history.update(model,X_train,Y_train,X_test,Y_test)
+
+    return history
+```
+
+%% Cell type:markdown id: tags:
+
+#### 5.2 - Get the model
+
+%% Cell type:code id: tags:
+
+``` python
+
+
+model=model_v1( x_train[0,:].shape[0] )
+
+print(model)
+
+```
+
+%% Cell type:markdown id: tags:
+
+#### 5.3 - Train the model
+
+%% Cell type:code id: tags:
+
+``` python
+
+
+torch_x_train=torch.from_numpy(x_train)
+torch_y_train=torch.from_numpy(y_train)
+torch_x_test=torch.from_numpy(x_test)
+torch_y_test=torch.from_numpy(y_test)
+
+batch_size  = 10
+epochs      = 100
+
+
+history=fit(model,torch_x_train,torch_y_train,torch_x_test,torch_y_test,EPOCHS=epochs,BATCH_SIZE = batch_size)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Evaluate
+### 6.1 - Model evaluation
+MAE =  Mean Absolute Error (between the labels and predictions)
+A mae equal to 3 represents an average error in prediction of $3k.
+
+%% Cell type:code id: tags:
+
+``` python
+var_x_test = Variable(torch_x_test).float()
+var_y_test = Variable(torch_y_test).float()
+y_pred = model(var_x_test)
+
+nn_loss = nn.MSELoss()
+nn_MAE_loss = nn.L1Loss()
+
+print('x_test / loss      : {:5.4f}'.format(nn_loss(y_pred.view(-1), var_y_test.view(-1)).item()))
+print('x_test / mae       : {:5.4f}'.format(nn_MAE_loss(y_pred.view(-1), var_y_test.view(-1)).item()))
+```
+
+%% Cell type:markdown id: tags:
+
+### 6.2 - Training history
+What was the best result during our training ?
+
+%% Cell type:code id: tags:
+
+``` python
+
+df=pd.DataFrame(data=history.history)
+df.describe()
+```
+
+%% Cell type:code id: tags:
+
+``` python
+print("min( val_mae ) : {:.4f}".format( min(history.history["val_mae"]) ) )
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.scrawler.history(history, plot={'MAE' :['mae', 'val_mae'],
+                                'LOSS':['loss','val_loss']})
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 7 - Make a prediction
+The data must be normalized with the parameters (mean, std) previously used.
+
+%% Cell type:code id: tags:
+
+``` python
+my_data = [ 1.26425925, -0.48522739,  1.0436489 , -0.23112788,  1.37120745,
+       -2.14308942,  1.13489104, -1.06802005,  1.71189006,  1.57042287,
+        0.77859951,  0.14769795,  2.7585581 ]
+real_price = 10.4
+
+my_data=np.array(my_data).reshape(1,13)
+```
+
+%% Cell type:code id: tags:
+
+``` python
+torch_my_data=torch.from_numpy(my_data)
+var_my_data = Variable(torch_my_data).float()
+
+predictions = model( var_my_data )
+print("Prediction : {:.2f} K$".format(predictions[0][0]))
+print("Reality    : {:.2f} K$".format(real_price))
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

BHPD.PyTorch/modules/fidle_pwk_additional.py

+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from torch.autograd import Variable
+
+import numpy as np
+
+
+class convergence_history_CrossEntropyLoss:
+  def __init__(self):
+    """
+    Class to save the training converge properties
+    """
+    self.loss=nn.CrossEntropyLoss()
+    self.history={}                #Save convergence measures in the end of each epoch
+    self.history['loss']=[]        #value of the cost function on training data
+    self.history['accuracy']=[]         #percentage of  correctly classified instances on training data (if classification)
+    self.history['val_loss']=[]    #value of the cost function on validation data
+    self.history['val_accuracy']=[]     #percentage of  correctly classified instances on validation data (if classification)
+  
+  def update(self,current_model,xtrain,ytrain,xtest,ytest):
+    
+    #convergence information on the training data 
+    nb_training_obs=xtrain.shape[0]
+    if nb_training_obs>xtest.shape[0]:
+        nb_training_obs=xtest.shape[0]
+        
+    epoch_shuffler=np.arange(xtrain.shape[0]) 
+    np.random.shuffle(epoch_shuffler)
+    mini_batch_observations = epoch_shuffler[:nb_training_obs]
+    var_X_batch = Variable(xtrain[mini_batch_observations,:]).float()
+    var_y_batch = Variable(ytrain[mini_batch_observations])
+    y_pred_batch = current_model(var_X_batch)
+    curr_loss = self.loss(y_pred_batch, var_y_batch)
+
+    self.history['loss'].append(curr_loss.item())
+    self.history['accuracy'].append( float( (torch.argmax(y_pred_batch, dim= 1) == var_y_batch).float().mean()) )
+    
+    #convergence information on the test data 
+    var_X_batch = Variable(xtest[:,:]).float()
+    var_y_batch = Variable(ytest[:])
+    y_pred_batch = current_model(var_X_batch)
+    curr_loss = self.loss(y_pred_batch, var_y_batch)
+
+    self.history['val_loss'].append(curr_loss.item())
+    self.history['val_accuracy'].append( float( (torch.argmax(y_pred_batch, dim= 1) == var_y_batch).float().mean()) )
+
+
+
+class convergence_history_MSELoss:
+  def __init__(self):
+    """
+    Class to save the training converge properties
+    """
+    self.loss = nn.MSELoss()
+    self.MAE_loss = nn.L1Loss()
+    self.history={}                #Save convergence measures in the end of each epoch
+    self.history['loss']=[]        #value of the cost function on training data
+    self.history['mae']=[]         #mean absolute error on training data
+    self.history['val_loss']=[]    #value of the cost function on validation data
+    self.history['val_mae']=[]     #mean absolute error on validation data
+  
+  def update(self,current_model,xtrain,ytrain,xtest,ytest):
+    
+    #convergence information on the training data 
+    nb_training_obs=xtrain.shape[0]
+    if nb_training_obs>xtest.shape[0]:
+        nb_training_obs=xtest.shape[0]
+        
+    epoch_shuffler=np.arange(xtrain.shape[0]) 
+    np.random.shuffle(epoch_shuffler)
+    mini_batch_observations = epoch_shuffler[:nb_training_obs]
+    var_X_batch = Variable(xtrain[mini_batch_observations,:]).float()
+    var_y_batch = Variable(ytrain[mini_batch_observations]).float()
+    y_pred_batch = current_model(var_X_batch)
+    curr_loss = self.loss(y_pred_batch.view(-1), var_y_batch.view(-1))
+
+    self.history['loss'].append(curr_loss.item())
+    self.history['mae'].append(self.MAE_loss(y_pred_batch.view(-1), var_y_batch.view(-1)).item())
+    
+    #convergence information on the test data 
+    var_X_batch = Variable(xtest[:,:]).float()
+    var_y_batch = Variable(ytest[:]).float()
+    y_pred_batch = current_model(var_X_batch)
+    curr_loss = self.loss(y_pred_batch.view(-1), var_y_batch.view(-1))
+
+    self.history['val_loss'].append(curr_loss.item())
+    self.history['val_mae'].append(self.MAE_loss(y_pred_batch.view(-1), var_y_batch.view(-1)).item())

DCGAN.Lightning/01-DCGAN-PL.ipynb

+%% Cell type:markdown id: tags:
+
+<img width="800px" src="../fidle/img/header.svg"></img>
+
+# <!-- TITLE --> [PLSHEEP3] - A DCGAN to Draw a Sheep, using Pytorch Lightning
+<!-- DESC --> "Draw me a sheep", revisited with a DCGAN, using Pytorch Lightning
+<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
+
+## Objectives :
+ - Build and train a DCGAN model with the Quick Draw dataset
+ - Understanding DCGAN
+
+The [Quick draw dataset](https://quickdraw.withgoogle.com/data) contains about 50.000.000 drawings, made by real people...
+We are using a subset of 117.555 of Sheep drawings
+To get the dataset : [https://github.com/googlecreativelab/quickdraw-dataset](https://github.com/googlecreativelab/quickdraw-dataset)
+Datasets in numpy bitmap file : [https://console.cloud.google.com/storage/quickdraw_dataset/full/numpy_bitmap](https://console.cloud.google.com/storage/quickdraw_dataset/full/numpy_bitmap)
+Sheep dataset : [https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/sheep.npy](https://storage.googleapis.com/quickdraw_dataset/full/numpy_bitmap/sheep.npy) (94.3 Mo)
+
+
+## What we're going to do :
+
+ - Have a look to the dataset
+ - Defining a GAN model
+ - Build the model
+ - Train it
+ - Have a look of the results
+
+%% Cell type:markdown id: tags:
+
+## Step 1 - Init and parameters
+#### Python init
+
+%% Cell type:code id: tags:
+
+``` python
+import os
+import sys
+import shutil
+
+import numpy as np
+import torch
+from lightning import Trainer
+from lightning.pytorch.callbacks                        import ModelCheckpoint
+from lightning.pytorch.loggers.tensorboard              import TensorBoardLogger
+
+import fidle
+
+from modules.QuickDrawDataModule import QuickDrawDataModule
+
+from modules.GAN                 import GAN
+from modules.WGANGP              import WGANGP
+from modules.Generators          import *
+from modules.Discriminators      import *
+
+# Init Fidle environment
+run_id, run_dir, datasets_dir = fidle.init('PLSHEEP3')
+```
+
+%% Cell type:markdown id: tags:
+
+#### Few parameters
+scale=1, epochs=20 : Need 22' on a V100
+
+%% Cell type:code id: tags:
+
+``` python
+latent_dim          = 128
+
+gan_name            = 'WGANGP'
+generator_name      = 'Generator_2'
+discriminator_name  = 'Discriminator_3'
+
+scale               = 0.001
+epochs              = 4
+num_workers         = 2
+lr                  = 0.0001
+b1                  = 0.5
+b2                  = 0.999
+lambda_gp           = 10
+batch_size          = 64
+num_img             = 48
+fit_verbosity       = 2
+
+dataset_file        = datasets_dir+'/QuickDraw/origine/sheep.npy'
+data_shape          = (28,28,1)
+```
+
+%% Cell type:markdown id: tags:
+
+Override parameters (batch mode) - Just forget this cell
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.override('latent_dim', 'gan_name', 'generator_name', 'discriminator_name')
+fidle.override('epochs', 'lr', 'b1', 'b2', 'batch_size', 'num_img', 'fit_verbosity')
+fidle.override('dataset_file', 'data_shape', 'scale', 'num_workers' )
+```
+
+%% Cell type:markdown id: tags:
+
+#### Cleaning
+
+%% Cell type:code id: tags:
+
+``` python
+# You can comment these lines to keep each run...
+shutil.rmtree(f'{run_dir}/figs', ignore_errors=True)
+shutil.rmtree(f'{run_dir}/models', ignore_errors=True)
+shutil.rmtree(f'{run_dir}/tb_logs', ignore_errors=True)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 2 - Get some nice data
+
+%% Cell type:markdown id: tags:
+
+#### Get a Nice DataModule
+Our DataModule is defined in [./modules/QuickDrawDataModule.py](./modules/QuickDrawDataModule.py)
+This is a [LightningDataModule](https://pytorch-lightning.readthedocs.io/en/stable/data/datamodule.html)
+
+%% Cell type:code id: tags:
+
+``` python
+dm = QuickDrawDataModule(dataset_file, scale, batch_size, num_workers=num_workers)
+dm.setup()
+```
+
+%% Cell type:markdown id: tags:
+
+#### Have a look
+
+%% Cell type:code id: tags:
+
+``` python
+dl         = dm.train_dataloader()
+batch_data = next(iter(dl))
+
+fidle.scrawler.images( batch_data.reshape(-1,28,28), indices=range(batch_size), columns=12, x_size=1, y_size=1,
+                       y_padding=0,spines_alpha=0, save_as='01-Sheeps')
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 3 - Get a nice GAN model
+
+Our Generators are defined in [./modules/Generators.py](./modules/Generators.py)
+Our Discriminators are defined in [./modules/Discriminators.py](./modules/Discriminators.py)
+
+
+Our GANs are defined in :
+ - [./modules/GAN.py](./modules/GAN.py)
+ - [./modules/WGANGP.py](./modules/WGANGP.py)
+
+%% Cell type:markdown id: tags:
+
+#### Retrieve class by name
+To be very flexible, we just specify class names as parameters.
+The code below retrieves classes from their names.
+
+%% Cell type:code id: tags:
+
+``` python
+module=sys.modules['__main__']
+Generator_     = getattr(module, generator_name)
+Discriminator_ = getattr(module, discriminator_name)
+GAN_           = getattr(module, gan_name)
+```
+
+%% Cell type:markdown id: tags:
+
+#### Basic test - Just to be sure it (could) works... ;-)
+
+%% Cell type:code id: tags:
+
+``` python
+generator     = Generator_(     latent_dim=latent_dim, data_shape=data_shape )
+discriminator = Discriminator_( latent_dim=latent_dim, data_shape=data_shape )
+
+print('\nFew tests :\n')
+z = torch.randn(batch_size, latent_dim)
+print('z size        : ',z.size())
+
+fake_img = generator.forward(z)
+print('fake_img      : ', fake_img.size())
+
+p = discriminator.forward(fake_img)
+print('pred fake     : ', p.size())
+
+print('batch_data    : ',batch_data.size())
+
+p = discriminator.forward(batch_data)
+print('pred real     : ', p.size())
+
+print('\nShow fake images :')
+nimg = fake_img.detach().numpy()
+fidle.scrawler.images( nimg.reshape(-1,28,28), indices=range(batch_size), columns=12, x_size=1, y_size=1,
+                       y_padding=0,spines_alpha=0, save_as='01-Sheeps')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+print('Fake images : ', fake_img.size())
+print('Batch size  : ', batch_data.size())
+e = torch.distributions.uniform.Uniform(0, 1).sample([batch_size,1])
+e = e[:None,None,None]
+i = fake_img * e + (1-e)*batch_data
+
+print('\ninterpolate images :')
+nimg = i.detach().numpy()
+fidle.scrawler.images( nimg.reshape(-1,28,28), indices=range(batch_size), columns=12, x_size=1, y_size=1,
+                       y_padding=0,spines_alpha=0, save_as='01-Sheeps')
+```
+
+%% Cell type:markdown id: tags:
+
+#### GAN model
+To simplify our code, the GAN class is defined separately in the module [./modules/GAN.py](./modules/GAN.py)
+Passing the classe names for generator/discriminator by parameter allows to stay modular and to use the PL checkpoints.
+
+%% Cell type:code id: tags:
+
+``` python
+gan = GAN_( data_shape          = data_shape,
+            lr                  = lr,
+            b1                  = b1,
+            b2                  = b2,
+            lambda_gp           = lambda_gp,
+            batch_size          = batch_size,
+            latent_dim          = latent_dim,
+            generator_name      = generator_name,
+            discriminator_name  = discriminator_name)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 5 - Train it !
+#### Instantiate Callbacks, Logger & co.
+More about :
+- [Checkpoints](https://pytorch-lightning.readthedocs.io/en/stable/common/checkpointing_basic.html)
+- [modelCheckpoint](https://pytorch-lightning.readthedocs.io/en/stable/api/pytorch_lightning.callbacks.ModelCheckpoint.html#pytorch_lightning.callbacks.ModelCheckpoint)
+
+%% Cell type:code id: tags:
+
+``` python
+
+# ---- for tensorboard logs
+#
+logger       = TensorBoardLogger(       save_dir       = f'{run_dir}',
+                                        name           = 'tb_logs'  )
+
+log_dir = os.path.abspath(f'{run_dir}/tb_logs')
+print('To access the logs with tensorboard, use this command line :')
+print(f'tensorboard --logdir {log_dir}')
+
+# ---- To save checkpoints
+#
+callback_checkpoints = ModelCheckpoint( dirpath        = f'{run_dir}/models',
+                                        filename       = 'bestModel',
+                                        save_top_k     = 1,
+                                        save_last      = True,
+                                        every_n_epochs = 1,
+                                        monitor        = "g_loss")
+```
+
+%% Cell type:markdown id: tags:
+
+#### Train it
+
+%% Cell type:code id: tags:
+
+``` python
+
+trainer = Trainer(
+    accelerator        = "auto",
+    max_epochs         = epochs,
+    callbacks          = [callback_checkpoints],
+    log_every_n_steps  = batch_size,
+    logger             = logger
+)
+
+trainer.fit(gan, dm)
+```
+
+%% Cell type:markdown id: tags:
+
+## Step 6 - Reload our best model
+Note :
+
+%% Cell type:code id: tags:
+
+``` python
+gan = GAN.load_from_checkpoint(f'{run_dir}/models/bestModel.ckpt')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+nb_images = 96
+
+z = torch.randn(nb_images, latent_dim)
+print('z size        : ',z.size())
+
+if torch.cuda.is_available(): z=z.cuda()
+
+fake_img = gan.generator.forward(z)
+print('fake_img      : ', fake_img.size())
+
+nimg = fake_img.cpu().detach().numpy()
+fidle.scrawler.images( nimg.reshape(-1,28,28), indices=range(nb_images), columns=12, x_size=1, y_size=1,
+                       y_padding=0,spines_alpha=0, save_as='01-Sheeps')
+```
+
+%% Cell type:code id: tags:
+
+``` python
+fidle.end()
+```
+
+%% Cell type:markdown id: tags:
+
+---
+<img width="80px" src="../fidle/img/logo-paysage.svg"></img>

DCGAN.Lightning/modules/Discriminators.py

+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|                         GAN / Generators
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI - https://fidle.cnrs.fr
+# ------------------------------------------------------------------
+# JL Parouty (Mars 2024)
+
+
+import numpy as np
+import torch.nn as nn
+
+
+# -----------------------------------------------------------------------------
+# -- Discriminator n°1
+# -----------------------------------------------------------------------------
+#
+class Discriminator_1(nn.Module):
+    '''
+    A basic DNN discriminator, usable with classic GAN
+    '''
+
+    def __init__(self, latent_dim=None, data_shape=None):
+    
+        super().__init__()
+        self.img_shape = data_shape
+        print('init discriminator 1     : ',data_shape,' to sigmoid')
+
+        self.model = nn.Sequential(
+
+            nn.Flatten(),
+            nn.Linear(int(np.prod(data_shape)), 512),
+            nn.ReLU(),
+            
+            nn.Linear(512, 256),
+            nn.ReLU(),
+
+            nn.Linear(256, 1),
+            nn.Sigmoid(),
+        )
+
+    def forward(self, img):
+        validity = self.model(img)
+
+        return validity
+
+
+
+# -----------------------------------------------------------------------------
+# -- Discriminator n°2
+# -----------------------------------------------------------------------------
+#
+class Discriminator_2(nn.Module):
+    '''
+    A more efficient discriminator,based on CNN, usable with classic GAN
+    '''
+
+    def __init__(self, latent_dim=None, data_shape=None):
+    
+        super().__init__()
+        self.img_shape = data_shape
+        print('init discriminator 2     : ',data_shape,' to sigmoid')
+
+        self.model = nn.Sequential(
+
+            nn.Conv2d(1, 32, kernel_size = 3, stride = 2, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(32),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(32, 64, kernel_size = 3, stride = 1, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(64),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(64, 128, kernel_size = 3, stride = 1, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(128),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(128, 256, kernel_size = 3, stride = 2, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(256),
+            nn.Dropout2d(0.25),
+
+            nn.Flatten(),
+            nn.Linear(12544, 1),
+            nn.Sigmoid(),
+        )
+
+    def forward(self, img):
+        img_nchw = img.permute(0, 3, 1, 2) # reformat from NHWC to NCHW
+        validity = self.model(img_nchw)
+
+        return validity
+
+
+   
+# -----------------------------------------------------------------------------
+# -- Discriminator n°3
+# -----------------------------------------------------------------------------
+#     
+class Discriminator_3(nn.Module):
+    '''
+    A CNN discriminator, usable with a WGANGP.
+    This discriminator has no sigmoid and returns a critical and not a probability
+    '''
+
+    def __init__(self, latent_dim=None, data_shape=None):
+    
+        super().__init__()
+        self.img_shape = data_shape
+        print('init discriminator 3     : ',data_shape,' to sigmoid')
+
+        self.model = nn.Sequential(
+
+            nn.Conv2d(1, 32, kernel_size = 3, stride = 2, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(32),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(32, 64, kernel_size = 3, stride = 1, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(64),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(64, 128, kernel_size = 3, stride = 1, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(128),
+            nn.Dropout2d(0.25),
+
+            nn.Conv2d(128, 256, kernel_size = 3, stride = 2, padding = 1),
+            nn.ReLU(),
+            nn.BatchNorm2d(256),
+            nn.Dropout2d(0.25),
+
+            nn.Flatten(),
+            nn.Linear(12544, 1),
+            nn.Sigmoid(),
+
+        )
+
+    def forward(self, img):
+        img_nchw = img.permute(0, 3, 1, 2) # reformat from NHWC to NCHW
+        validity = self.model(img_nchw)
+
+        return validity
\ No newline at end of file

DCGAN.Lightning/modules/GAN.py

+
+# ------------------------------------------------------------------
+#     _____ _     _ _
+#    |  ___(_) __| | | ___
+#    | |_  | |/ _` | |/ _ \
+#    |  _| | | (_| | |  __/
+#    |_|   |_|\__,_|_|\___|                Basic GAN LigthningModule
+# ------------------------------------------------------------------
+# Formation Introduction au Deep Learning  (FIDLE)
+# CNRS/MIAI - https://fidle.cnrs.fr
+# ------------------------------------------------------------------
+# JL Parouty (Mars 2024)
+
+
+
+import sys
+import numpy as np
+import torch
+import torch.nn.functional as F
+import torchvision
+from lightning import LightningModule
+
+
+class GAN(LightningModule):
+
+    # -------------------------------------------------------------------------
+    # Init
+    # -------------------------------------------------------------------------
+    #
+    def __init__(
+        self,
+        data_shape          = (None,None,None),
+        latent_dim          = None,
+        lr                  = 0.0002,
+        b1                  = 0.5,
+        b2                  = 0.999,
+        batch_size          = 64,
+        generator_name      = None,
+        discriminator_name  = None,
+        **kwargs,
+    ):
+        super().__init__()
+
+        print('\n---- GAN initialization --------------------------------------------')
+
+        # ---- Hyperparameters
+        #
+        # Enable Lightning to store all the provided arguments under the self.hparams attribute.
+        # These hyperparameters will also be stored within the model checkpoint.
+        #
+        self.save_hyperparameters()
+
+        print('Hyperarameters are :')
+        for name,value in self.hparams.items():
+            print(f'{name:24s} : {value}')
+
+        # ---- Because we have more than one optimizer
+        #
+        self.automatic_optimization = False
+
+        # ---- Generator/Discriminator instantiation
+        #
+        print('Submodels :')
+        module=sys.modules['__main__']
+        class_g = getattr(module, generator_name)
+        class_d = getattr(module, discriminator_name)
+        self.generator     = class_g( latent_dim=latent_dim, data_shape=data_shape)
+        self.discriminator = class_d( latent_dim=latent_dim, data_shape=data_shape)
+
+        # ---- Validation and example data
+        #
+        self.validation_z        = torch.randn(8, self.hparams.latent_dim)
+        self.example_input_array = torch.zeros(2, self.hparams.latent_dim)
+
+
+    def forward(self, z):
+        return self.generator(z)
+
+
+    def adversarial_loss(self, y_hat, y):
+        return F.binary_cross_entropy(y_hat, y)
+
+
+    def training_step(self, batch, batch_idx):
+        imgs       = batch
+        batch_size = batch.size(0)
+
+        optimizer_g, optimizer_d = self.optimizers()
+
+        # ---- Get some latent space vectors
+        #      We use type_as() to make sure we initialize z on the right device (GPU/CPU).
+        #
+        z = torch.randn(batch_size, self.hparams.latent_dim)
+        z = z.type_as(imgs)
+
+        # ---- Train generator ------------------------------------------------
+        #      Generator use optimizer #0
+        #      We try to generate false images that could mislead the discriminator
+        #
+        self.toggle_optimizer(optimizer_g)
+        
+        # Generate fake images
+        self.fake_imgs = self.generator.forward(z)
+
+        # Assemble labels that say all images are real, yes it's a lie ;-)
+        # put on GPU because we created this tensor inside training_loop
+        misleading_labels = torch.ones(batch_size, 1)
+        misleading_labels = misleading_labels.type_as(imgs)
+
+        # Adversarial loss is binary cross-entropy
+        y_hat = self.discriminator.forward(self.fake_imgs)
+        # print(y_hat)
+        g_loss = self.adversarial_loss(self.discriminator.forward(self.fake_imgs), misleading_labels)
+        
+        self.log("g_loss", g_loss, prog_bar=True)
+        
+        # Backward loss
+        self.manual_backward(g_loss)
+        
+        optimizer_g.step()
+        optimizer_g.zero_grad()
+        
+        self.untoggle_optimizer(optimizer_g)
+
+        # ---- Train discriminator --------------------------------------------
+        #      Discriminator use optimizer #1
+        #      We try to make the difference between fake images and real ones 
+        #
+        self.toggle_optimizer(optimizer_d)
+
+        # These images are reals
+        real_labels = torch.ones(batch_size, 1)
+        real_labels = real_labels.type_as(imgs)
+        pred_labels = self.discriminator.forward(imgs)
+
+        real_loss   = self.adversarial_loss(pred_labels, real_labels)
+
+        # These images are fake
+        fake_imgs   = self.generator.forward(z)
+        fake_labels = torch.zeros(batch_size, 1)
+        fake_labels = fake_labels.type_as(imgs)
+
+        fake_loss   = self.adversarial_loss(self.discriminator(fake_imgs.detach()), fake_labels)
+
+        # Discriminator loss is the average
+        d_loss = (real_loss + fake_loss) / 2
+        
+        self.log("d_loss", d_loss, prog_bar=True)
+ 
+        # Backward
+        self.manual_backward(d_loss)
+        
+        optimizer_d.step()
+        optimizer_d.zero_grad()
+        
+        self.untoggle_optimizer(optimizer_d)
+ 
+
+
+    def configure_optimizers(self):
+
+        lr = self.hparams.lr
+        b1 = self.hparams.b1
+        b2 = self.hparams.b2
+
+        # With a GAN, we need 2 separate optimizer.
+        # opt_g = torch.optim.Adam(self.generator.parameters(),     lr=lr, betas=(b1, b2))
+        # opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr, betas=(b1, b2),)
+        opt_g = torch.optim.Adam(self.generator.parameters(),     lr=lr)
+        opt_d = torch.optim.Adam(self.discriminator.parameters(), lr=lr)
+        return [opt_g, opt_d], []
+
+
+    def on_train_epoch_end(self):
+
+        # ---- Log Graph
+        #
+        if(self.current_epoch==1):
+            sampleImg=torch.rand((1,28,28,1))
+            sampleImg=sampleImg.type_as(self.generator.model[0].weight)
+            self.logger.experiment.add_graph(self.discriminator,sampleImg)
+
+        # ---- Log some of these images
+        #
+        z = torch.randn(self.hparams.batch_size, self.hparams.latent_dim)
+        z = z.type_as(self.generator.model[0].weight)
+        sample_imgs = self.generator(z)
+        sample_imgs = sample_imgs.permute(0, 3, 1, 2) # from NHWC to NCHW
+        grid = torchvision.utils.make_grid(tensor=sample_imgs, nrow=12, )
+        self.logger.experiment.add_image(f"Generated images", grid,self.current_epoch)