Skip to content
Snippets Groups Projects

Compare revisions

Changes are shown as if the source revision was being merged into the target revision. Learn more about comparing revisions.

Source

Select target project
No results found

Target

Select target project
  • daconcea/fidle
  • bossardl/fidle
  • Julie.Remenant/fidle
  • abijolao/fidle
  • monsimau/fidle
  • karkars/fidle
  • guilgautier/fidle
  • cailletr/fidle
  • talks/fidle
9 results
Show changes
Hide whitespace changes
Inline Side-by-side
Showing
with 1892 additions and 3688 deletions
IMDB/02-Prediction.ipynb deleted 100644 → 0
View file @ a434f008
%% Cell type:markdown id: tags:
<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
# <!-- TITLE --> [IMDB2] - Text embedding with IMDB - Reloaded
<!-- DESC --> Example of reusing a previously saved model
<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
## Objectives :
- The objective is to guess whether film reviews are **positive or negative** based on the analysis of the text.
- For this, we will use our **previously saved model**.
Original dataset can be find **[there](http://ai.stanford.edu/~amaas/data/sentiment/)**
Note that [IMDb.com](https://imdb.com) offers several easy-to-use [datasets](https://www.imdb.com/interfaces/)
For simplicity's sake, we'll use the dataset directly [embedded in Keras](https://www.tensorflow.org/api_docs/python/tf/keras/datasets)
## What we're going to do :
- Preparing the data
- Retrieve our saved model
- Evaluate the result
%% Cell type:markdown id: tags:
## Step 1 - Init python stuff
%% Cell type:code id: tags:
``` python
import numpy as np
import tensorflow as tf
import tensorflow.keras as keras
import tensorflow.keras.datasets.imdb as imdb
import matplotlib.pyplot as plt
import matplotlib
import seaborn as sns
import pandas as pd
import os,sys,h5py,json,re
from importlib import reload
sys.path.append('..')
import fidle.pwk as pwk
datasets_dir = pwk.init('IMDB2')
```
%% Output
**FIDLE 2020 - Practical Work Module**
Version : 0.6.1 DEV
Notebook id : IMDB2
Run time : Friday 18 December 2020, 18:21:49
TensorFlow version : 2.0.0
Keras version : 2.2.4-tf
Datasets dir : /home/pjluc/datasets/fidle
Running mode : full
Update keras cache : False
Save figs : True
Path figs : ./run/figs
%% Cell type:markdown id: tags:
## Step 2 : Preparing the data
### 2.1 - Our reviews :
%% Cell type:code id: tags:
``` python
reviews = [ "This film is particularly nice, a must see.",
"Some films are great classics and cannot be ignored.",
"This movie is just abominable and doesn't deserve to be seen!"]
```
%% Cell type:markdown id: tags:
### 2.2 - Retrieve dictionaries
Note : This dictionary is generated by [01-Embedding-Keras](01-Embedding-Keras.ipynb) notebook.
%% Cell type:code id: tags:
``` python
with open('./data/word_index.json', 'r') as fp:
word_index = json.load(fp)
index_word = {index:word for word,index in word_index.items()}
```
%% Cell type:markdown id: tags:
### 2.3 - Clean, index and padd
%% Cell type:code id: tags:
``` python
max_len = 256
vocab_size = 10000
nb_reviews = len(reviews)
x_data = []
# ---- For all reviews
for review in reviews:
# ---- First index must be <start>
index_review=[1]
# ---- For all words
for w in review.split(' '):
# ---- Clean it
w_clean = re.sub(r"[^a-zA-Z0-9]", "", w)
# ---- Not empty ?
if len(w_clean)>0:
# ---- Get the index
w_index = word_index.get(w,2)
if w_index>vocab_size : w_index=2
# ---- Add the index if < vocab_size
index_review.append(w_index)
# ---- Add the indexed review
x_data.append(index_review)
# ---- Padding
x_data = keras.preprocessing.sequence.pad_sequences(x_data, value = 0, padding = 'post', maxlen = max_len)
```
%% Cell type:markdown id: tags:
### 2.4 - Have a look
%% Cell type:code id: tags:
``` python
def translate(x):
return ' '.join( [index_word.get(i,'?') for i in x] )
for i in range(nb_reviews):
imax=np.where(x_data[i]==0)[0][0]+5
print(f'\nText review :', reviews[i])
print( f'x_train[{i:}] :', list(x_data[i][:imax]), '(...)')
print( 'Translation :', translate(x_data[i][:imax]), '(...)')
```
%% Output
Text review : This film is particularly nice, a must see.
x_train[0] : [1, 2, 22, 9, 572, 2, 6, 215, 2, 0, 0, 0, 0, 0] (...)
Translation : <start> <unknown> film is particularly <unknown> a must <unknown> <pad> <pad> <pad> <pad> <pad> (...)
Text review : Some films are great classics and cannot be ignored.
x_train[1] : [1, 2, 108, 26, 87, 2239, 5, 566, 30, 2, 0, 0, 0, 0, 0] (...)
Translation : <start> <unknown> films are great classics and cannot be <unknown> <pad> <pad> <pad> <pad> <pad> (...)
Text review : This movie is just abominable and doesn't deserve to be seen!
x_train[2] : [1, 2, 20, 9, 43, 2, 5, 152, 1833, 8, 30, 2, 0, 0, 0, 0, 0] (...)
Translation : <start> <unknown> movie is just <unknown> and doesn't deserve to be <unknown> <pad> <pad> <pad> <pad> <pad> (...)
%% Cell type:markdown id: tags:
## Step 2 - Bring back the model
%% Cell type:code id: tags:
``` python
model = keras.models.load_model('./run/models/best_model.h5')
```
%% Cell type:markdown id: tags:
## Step 4 - Predict
%% Cell type:code id: tags:
``` python
y_pred = model.predict(x_data)
```
%% Cell type:markdown id: tags:
#### And the winner is :
%% Cell type:code id: tags:
``` python
for i in range(nb_reviews):
print(f'\n{reviews[i]:<70} =>',('NEGATIVE' if y_pred[i][0]<0.5 else 'POSITIVE'),f'({y_pred[i][0]:.2f})')
```
%% Output
This film is particularly nice, a must see. => POSITIVE (0.56)
Some films are great classics and cannot be ignored. => POSITIVE (0.63)
This movie is just abominable and doesn't deserve to be seen! => NEGATIVE (0.35)
%% Cell type:code id: tags:
``` python
pwk.end()
```
%% Output
End time is : Friday 18 December 2020, 18:21:50
Duration is : 00:00:01 555ms
This notebook ends here
%% Cell type:markdown id: tags:
---
<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>
IMDB/03-LSTM-Keras.ipynb deleted 100644 → 0
View file @ a434f008
%% Cell type:markdown id: tags:
<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
# <!-- TITLE --> [IMDB3] - Text embedding/LSTM model with IMDB
<!-- DESC --> Still the same problem, but with a network combining embedding and LSTM
<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
## Objectives :
- The objective is to guess whether film reviews are **positive or negative** based on the analysis of the text.
- Use of a model combining embedding and LSTM
Original dataset can be find **[there](http://ai.stanford.edu/~amaas/data/sentiment/)**
Note that [IMDb.com](https://imdb.com) offers several easy-to-use [datasets](https://www.imdb.com/interfaces/)
For simplicity's sake, we'll use the dataset directly [embedded in Keras](https://www.tensorflow.org/api_docs/python/tf/keras/datasets)
## What we're going to do :
- Retrieve data
- Preparing the data
- Build a Embedding/LSTM model
- Train the model
- Evaluate the result
%% Cell type:markdown id: tags:
## Step 1 - Init python stuff
%% Cell type:code id: tags:
``` python
import numpy as np
import tensorflow as tf
import tensorflow.keras as keras
import tensorflow.keras.datasets.imdb as imdb
import matplotlib.pyplot as plt
import matplotlib
import os,sys,h5py,json
from importlib import reload
sys.path.append('..')
import fidle.pwk as pwk
datasets_dir = pwk.init('IMDB3')
```
%% Output
**FIDLE 2020 - Practical Work Module**
Version : 0.6.1 DEV
Notebook id : IMDB3
Run time : Friday 18 December 2020, 19:52:57
TensorFlow version : 2.0.0
Keras version : 2.2.4-tf
Datasets dir : /home/pjluc/datasets/fidle
Running mode : full
Update keras cache : False
Save figs : True
Path figs : ./run/figs
%% Cell type:markdown id: tags:
## Step 2 - Retrieve data
IMDb dataset can bet get directly from Keras - see [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/datasets)
Note : Due to their nature, textual data can be somewhat complex.
### 2.1 - Data structure :
The dataset is composed of 2 parts:
- **reviews**, this will be our **x**
- **opinions** (positive/negative), this will be our **y**
There are also a **dictionary**, because words are indexed in reviews
```
<dataset> = (<reviews>, <opinions>)
with : <reviews> = [ <review1>, <review2>, ... ]
<opinions> = [ <rate1>, <rate2>, ... ] where <ratei> = integer
where : <reviewi> = [ <w1>, <w2>, ...] <wi> are the index (int) of the word in the dictionary
<ratei> = int 0 for negative opinion, 1 for positive
<dictionary> = [ <word1>:<w1>, <word2>:<w2>, ... ]
with : <wordi> = word
<wi> = int
```
%% Cell type:markdown id: tags:
### 2.2 - Get dataset
For simplicity, we will use a pre-formatted dataset - See [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/datasets/imdb/load_data)
However, Keras offers some usefull tools for formatting textual data - See [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/preprocessing/text)
**Load dataset :**
%% Cell type:code id: tags:
``` python
vocab_size = 10000
# ----- Retrieve x,y
# Uncomment this if you want to load dataset directly from keras (small size <20M)
#
(x_train, y_train), (x_test, y_test) = imdb.load_data( num_words = vocab_size,
skip_top = 0,
maxlen = None,
seed = 42,
start_char = 1,
oov_char = 2,
index_from = 3, )
# To load a h5 version of the dataset :
#
# with h5py.File(f'{datasets_dir}/IMDB/origine/dataset_imdb.h5','r') as f:
# x_train = f['x_train'][:]
# y_train = f['y_train'][:]
# x_test = f['x_test'][:]
# y_test = f['y_test'][:]
```
%% Output
/home/pjluc/anaconda3/envs/fidle/lib/python3.7/site-packages/tensorflow_core/python/keras/datasets/imdb.py:129: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
x_train, y_train = np.array(xs[:idx]), np.array(labels[:idx])
/home/pjluc/anaconda3/envs/fidle/lib/python3.7/site-packages/tensorflow_core/python/keras/datasets/imdb.py:130: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
x_test, y_test = np.array(xs[idx:]), np.array(labels[idx:])
%% Cell type:markdown id: tags:
**About this dataset :**
%% Cell type:code id: tags:
``` python
print(" Max(x_train,x_test) : ", pwk.rmax([x_train,x_test]) )
print(" x_train : {} y_train : {}".format(x_train.shape, y_train.shape))
print(" x_test : {} y_test : {}".format(x_test.shape, y_test.shape))
print('\nReview example (x_train[12]) :\n\n',x_train[12])
```
%% Output
Max(x_train,x_test) : 9999
x_train : (25000,) y_train : (25000,)
x_test : (25000,) y_test : (25000,)
Review example (x_train[12]) :
[1, 14, 22, 1367, 53, 206, 159, 4, 636, 898, 74, 26, 11, 436, 363, 108, 7, 14, 432, 14, 22, 9, 1055, 34, 8599, 2, 5, 381, 3705, 4509, 14, 768, 47, 839, 25, 111, 1517, 2579, 1991, 438, 2663, 587, 4, 280, 725, 6, 58, 11, 2714, 201, 4, 206, 16, 702, 5, 5176, 19, 480, 5920, 157, 13, 64, 219, 4, 2, 11, 107, 665, 1212, 39, 4, 206, 4, 65, 410, 16, 565, 5, 24, 43, 343, 17, 5602, 8, 169, 101, 85, 206, 108, 8, 3008, 14, 25, 215, 168, 18, 6, 2579, 1991, 438, 2, 11, 129, 1609, 36, 26, 66, 290, 3303, 46, 5, 633, 115, 4363]
%% Cell type:markdown id: tags:
### 2.3 - Have a look for humans (optional)
When we loaded the dataset, we asked for using \<start\> as 1, \<unknown word\> as 2
So, we shifted the dataset by 3 with the parameter index_from=3
**Load dictionary :**
%% Cell type:code id: tags:
``` python
# ---- Retrieve dictionary {word:index}, and encode it in ascii
#
word_index = imdb.get_word_index()
# ---- Shift the dictionary from +3
#
word_index = {w:(i+3) for w,i in word_index.items()}
# ---- Add <pad>, <start> and unknown tags
#
word_index.update( {'<pad>':0, '<start>':1, '<unknown>':2} )
# ---- Create a reverse dictionary : {index:word}
#
index_word = {index:word for word,index in word_index.items()}
# ---- Add a nice function to transpose :
#
def dataset2text(review):
return ' '.join([index_word.get(i, '?') for i in review])
```
%% Cell type:markdown id: tags:
**Have a look :**
%% Cell type:code id: tags:
``` python
print('\nDictionary size : ', len(word_index))
for k in range(440,455):print(f'{k:2d} : {index_word[k]}' )
pwk.subtitle('Review example :')
print(x_train[12])
pwk.subtitle('After translation :')
print(dataset2text(x_train[12]))
```
%% Output
Dictionary size : 88587
440 : hope
441 : entertaining
442 : she's
443 : mr
444 : overall
445 : evil
446 : called
447 : loved
448 : based
449 : oh
450 : several
451 : fans
452 : mother
453 : drama
454 : beginning
<br>**Review example :**
[1, 14, 22, 1367, 53, 206, 159, 4, 636, 898, 74, 26, 11, 436, 363, 108, 7, 14, 432, 14, 22, 9, 1055, 34, 8599, 2, 5, 381, 3705, 4509, 14, 768, 47, 839, 25, 111, 1517, 2579, 1991, 438, 2663, 587, 4, 280, 725, 6, 58, 11, 2714, 201, 4, 206, 16, 702, 5, 5176, 19, 480, 5920, 157, 13, 64, 219, 4, 2, 11, 107, 665, 1212, 39, 4, 206, 4, 65, 410, 16, 565, 5, 24, 43, 343, 17, 5602, 8, 169, 101, 85, 206, 108, 8, 3008, 14, 25, 215, 168, 18, 6, 2579, 1991, 438, 2, 11, 129, 1609, 36, 26, 66, 290, 3303, 46, 5, 633, 115, 4363]
<br>**After translation :**
<start> this film contains more action before the opening credits than are in entire hollywood films of this sort this film is produced by tsui <unknown> and stars jet li this team has brought you many worthy hong kong cinema productions including the once upon a time in china series the action was fast and furious with amazing wire work i only saw the <unknown> in two shots aside from the action the story itself was strong and not just used as filler to find any other action films to rival this you must look for a hong kong cinema <unknown> in your area they are really worth checking out and usually never disappoint
%% Cell type:markdown id: tags:
### 2.4 - Have a look for NN
%% Cell type:code id: tags:
``` python
sizes=[len(i) for i in x_train]
plt.figure(figsize=(16,6))
plt.hist(sizes, bins=400)
plt.gca().set(title='Distribution of reviews by size - [{:5.2f}, {:5.2f}]'.format(min(sizes),max(sizes)),
xlabel='Size', ylabel='Density', xlim=[0,1500])
pwk.save_fig('01-stats-sizes')
plt.show()
```
%% Output
%% Cell type:markdown id: tags:
## Step 3 - Preprocess the data (padding)
In order to be processed by an NN, all entries must have the **same length.**
We chose a review length of **review_len**
We will therefore complete them with a padding (of \<pad\>\)
%% Cell type:code id: tags:
``` python
review_len = 256
x_train = keras.preprocessing.sequence.pad_sequences(x_train,
value = 0,
padding = 'post',
maxlen = review_len)
x_test = keras.preprocessing.sequence.pad_sequences(x_test,
value = 0 ,
padding = 'post',
maxlen = review_len)
pwk.subtitle('After padding :')
print(x_train[12])
pwk.subtitle('In real words :')
print(dataset2text(x_train[12]))
```
%% Output
<br>**After padding :**
[ 1 14 22 1367 53 206 159 4 636 898 74 26 11 436
363 108 7 14 432 14 22 9 1055 34 8599 2 5 381
3705 4509 14 768 47 839 25 111 1517 2579 1991 438 2663 587
4 280 725 6 58 11 2714 201 4 206 16 702 5 5176
19 480 5920 157 13 64 219 4 2 11 107 665 1212 39
4 206 4 65 410 16 565 5 24 43 343 17 5602 8
169 101 85 206 108 8 3008 14 25 215 168 18 6 2579
1991 438 2 11 129 1609 36 26 66 290 3303 46 5 633
115 4363 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0]
<br>**In real words :**
<start> this film contains more action before the opening credits than are in entire hollywood films of this sort this film is produced by tsui <unknown> and stars jet li this team has brought you many worthy hong kong cinema productions including the once upon a time in china series the action was fast and furious with amazing wire work i only saw the <unknown> in two shots aside from the action the story itself was strong and not just used as filler to find any other action films to rival this you must look for a hong kong cinema <unknown> in your area they are really worth checking out and usually never disappoint <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad> <pad>
%% Cell type:markdown id: tags:
**Save dataset and dictionary (For future use but not mandatory)**
%% Cell type:code id: tags:
``` python
# ---- Write dataset in a h5 file, could be usefull
#
output_dir = './data'
pwk.mkdir(output_dir)
with h5py.File(f'{output_dir}/dataset_imdb.h5', 'w') as f:
f.create_dataset("x_train", data=x_train)
f.create_dataset("y_train", data=y_train)
f.create_dataset("x_test", data=x_test)
f.create_dataset("y_test", data=y_test)
with open(f'{output_dir}/word_index.json', 'w') as fp:
json.dump(word_index, fp)
with open(f'{output_dir}/index_word.json', 'w') as fp:
json.dump(index_word, fp)
print('Saved.')
```
%% Output
Saved.
%% Cell type:markdown id: tags:
## Step 4 - Build the model
Few remarks :
- We'll choose a dense vector size for the embedding output with **dense_vector_size**
- **GlobalAveragePooling1D** do a pooling on the last dimension : (None, lx, ly) -> (None, ly)
In other words: we average the set of vectors/words of a sentence
- L'embedding de Keras fonctionne de manière supervisée. Il s'agit d'une couche de *vocab_size* neurones vers *n_neurons* permettant de maintenir une table de vecteurs (les poids constituent les vecteurs). Cette couche ne calcule pas de sortie a la façon des couches normales, mais renvois la valeur des vecteurs. n mots => n vecteurs (ensuite empilés par le pooling)
Voir : [Explication plus détaillée (en)](https://stats.stackexchange.com/questions/324992/how-the-embedding-layer-is-trained-in-keras-embedding-layer)
ainsi que : [Sentiment detection with Keras](https://www.liip.ch/en/blog/sentiment-detection-with-keras-word-embeddings-and-lstm-deep-learning-networks)
More documentation about this model functions :
- [Embedding](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Embedding)
- [GlobalAveragePooling1D](https://www.tensorflow.org/api_docs/python/tf/keras/layers/GlobalAveragePooling1D)
%% Cell type:code id: tags:
``` python
def get_model(dense_vector_size=128):
model = keras.Sequential()
model.add(keras.layers.Embedding(input_dim = vocab_size,
output_dim = dense_vector_size,
input_length = review_len))
model.add(keras.layers.LSTM(128, dropout=0.2, recurrent_dropout=0.2))
model.add(keras.layers.Dense(1, activation='sigmoid'))
model.compile(optimizer = 'adam',
loss = 'binary_crossentropy',
metrics = ['accuracy'])
return model
```
%% Cell type:markdown id: tags:
## Step 5 - Train the model
### 5.1 - Get it
%% Cell type:code id: tags:
``` python
model = get_model(32)
model.summary()
```
%% Output
Model: "sequential"
_________________________________________________________________
Layer (type) Output Shape Param #
=================================================================
embedding (Embedding) (None, 256, 32) 320000
_________________________________________________________________
lstm (LSTM) (None, 128) 82432
_________________________________________________________________
dense (Dense) (None, 1) 129
=================================================================
Total params: 402,561
Trainable params: 402,561
Non-trainable params: 0
_________________________________________________________________
%% Cell type:markdown id: tags:
### 5.2 - Add callback
%% Cell type:code id: tags:
``` python
os.makedirs('./run/models', mode=0o750, exist_ok=True)
save_dir = "./run/models/best_model.h5"
savemodel_callback = tf.keras.callbacks.ModelCheckpoint(filepath=save_dir, verbose=0, save_best_only=True)
```
%% Cell type:markdown id: tags:
### 5.1 - Train it
GPU : batch_size=512 : 6' 30s
CPU : batch_size=512 : 12' 57s
%% Cell type:code id: tags:
``` python
%%time
n_epochs = 10
batch_size = 512
history = model.fit(x_train,
y_train,
epochs = n_epochs,
batch_size = batch_size,
validation_data = (x_test, y_test),
verbose = 1,
callbacks = [savemodel_callback])
```
%% Output
Train on 25000 samples, validate on 25000 samples
Epoch 1/10
25000/25000 [==============================] - 74s 3ms/sample - loss: 0.6924 - accuracy: 0.5083 - val_loss: 0.6919 - val_accuracy: 0.5006
Epoch 2/10
25000/25000 [==============================] - 80s 3ms/sample - loss: 0.6668 - accuracy: 0.5982 - val_loss: 0.6759 - val_accuracy: 0.5417
Epoch 3/10
25000/25000 [==============================] - 81s 3ms/sample - loss: 0.6737 - accuracy: 0.5459 - val_loss: 0.6750 - val_accuracy: 0.5363
Epoch 4/10
25000/25000 [==============================] - 80s 3ms/sample - loss: 0.6569 - accuracy: 0.5696 - val_loss: 0.6373 - val_accuracy: 0.5694
Epoch 5/10
25000/25000 [==============================] - 80s 3ms/sample - loss: 0.6409 - accuracy: 0.6388 - val_loss: 0.6431 - val_accuracy: 0.6351
Epoch 6/10
25000/25000 [==============================] - 79s 3ms/sample - loss: 0.6149 - accuracy: 0.6703 - val_loss: 0.6466 - val_accuracy: 0.6623
Epoch 7/10
25000/25000 [==============================] - 81s 3ms/sample - loss: 0.5660 - accuracy: 0.7337 - val_loss: 0.5943 - val_accuracy: 0.7200
Epoch 8/10
25000/25000 [==============================] - 79s 3ms/sample - loss: 0.5467 - accuracy: 0.7518 - val_loss: 0.5087 - val_accuracy: 0.7856
Epoch 9/10
25000/25000 [==============================] - 79s 3ms/sample - loss: 0.5587 - accuracy: 0.7446 - val_loss: 0.6065 - val_accuracy: 0.6842
Epoch 10/10
25000/25000 [==============================] - 80s 3ms/sample - loss: 0.5798 - accuracy: 0.7039 - val_loss: 0.5688 - val_accuracy: 0.7133
CPU times: user 58min 6s, sys: 9min 21s, total: 1h 7min 27s
Wall time: 13min 12s
%% Cell type:markdown id: tags:
## Step 6 - Evaluate
### 6.1 - Training history
%% Cell type:code id: tags:
``` python
pwk.plot_history(history, save_as='02-history')
```
%% Output
%% Cell type:markdown id: tags:
### 6.2 - Reload and evaluate best model
%% Cell type:code id: tags:
``` python
model = keras.models.load_model('./run/models/best_model.h5')
# ---- Evaluate
score = model.evaluate(x_test, y_test, verbose=0)
print('x_test / loss : {:5.4f}'.format(score[0]))
print('x_test / accuracy : {:5.4f}'.format(score[1]))
values=[score[1], 1-score[1]]
pwk.plot_donut(values,["Accuracy","Errors"], title="#### Accuracy donut is :", save_as='03-donut')
# ---- Confusion matrix
y_sigmoid = model.predict(x_test)
y_pred = y_sigmoid.copy()
y_pred[ y_sigmoid< 0.5 ] = 0
y_pred[ y_sigmoid>=0.5 ] = 1
pwk.display_confusion_matrix(y_test,y_pred,labels=range(2))
pwk.plot_confusion_matrix(y_test,y_pred,range(2), figsize=(8, 8),normalize=False, save_as='04-confusion-matrix')
```
%% Output
x_test / loss : 0.5087
x_test / accuracy : 0.7856
#### Accuracy donut is :
#### Confusion matrix is :
%% Cell type:code id: tags:
``` python
pwk.end()
```
%% Output
End time is : Friday 18 December 2020, 19:53:06
Duration is : 00:00:09 281ms
This notebook ends here
%% Cell type:markdown id: tags:
---
<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>
IRIS/01-Simple-Perceptron.ipynb deleted 100644 → 0
View file @ a434f008
%% Cell type:markdown id: tags:
<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
# <!-- TITLE --> [PER57] - Perceptron Model 1957
<!-- DESC --> A simple perceptron, with the IRIS dataset.
<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
## Objectives :
- Implement a historical linear classifier with a historical dataset !
- The objective is to predict the type of Iris from the size of the leaves.
- Identifying its limitations
The [IRIS dataset](https://archive.ics.uci.edu/ml/datasets/Iris) is probably one of the oldest datasets, dating back to 1936 .
## What we're going to do :
- Retrieve the dataset, via scikit learn
- training and classifying
## Step 1 - Import and init
%% Cell type:code id: tags:
``` python
import numpy as np
from sklearn.datasets import load_iris
from sklearn.linear_model import Perceptron
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib
import os,sys
sys.path.append('..')
import fidle.pwk as pwk
datasets_dir = pwk.init('PER57')
```
%% Output
**FIDLE 2020 - Practical Work Module**
Version : 0.6.1 DEV
Notebook id : PER57
Run time : Wednesday 16 December 2020, 21:01:59
TensorFlow version : 2.0.0
Keras version : 2.2.4-tf
Datasets dir : ~/datasets/fidle
Update keras cache : False
Save figs : True
Path figs : ./run/figs
%% Cell type:markdown id: tags:
## Step 2 - Prepare IRIS Dataset
Retrieve a dataset : http://scikit-learn.org/stable/modules/classes.html#module-sklearn.datasets
About the datesets : http://scikit-learn.org/stable/datasets/index.html
Data fields (X) :
- 0 : sepal length in cm
- 1 : sepal width in cm
- 2 : petal length in cm
- 3 : petal width in cm
Class (y) :
- 0 : class 0=Iris-Setosa, 1=Iris-Versicolour, 2=Iris-Virginica
### 2.1 - Get dataset
%% Cell type:code id: tags:
``` python
x0,y0 = load_iris(return_X_y=True)
x = x0[:, (2,3)] # We only keep fields 2 and 3
y = y0.copy()
y[ y0==0 ] = 1 # 1 = Iris setosa
y[ y0>=1 ] = 0 # 0 = not iris setosa
df=pd.DataFrame.from_dict({'Length (x1)':x[:,0], 'Width (x2)':x[:,1], 'Setosa {0,1} (y)':y})
display(df)
print(f'x shape : {x.shape}')
print(f'y shape : {y.shape}')
```
%% Output
x shape : (150, 2)
y shape : (150,)
%% Cell type:markdown id: tags:
### 2.2 - Train and test sets
%% Cell type:code id: tags:
``` python
x,y = pwk.shuffle_np_dataset(x, y)
n=int(len(x)*0.8)
x_train = x[:n]
y_train = y[:n]
x_test = x[n:]
y_test = y[n:]
print(f'x_train shape : {x_train.shape}')
print(f'y_train shape : {y_train.shape}')
print(f'x_test shape : {x_test.shape}')
print(f'y_test shape : {y_test.shape}')
```
%% Output
x_train shape : (120, 2)
y_train shape : (120,)
x_test shape : (30, 2)
y_test shape : (30,)
%% Cell type:markdown id: tags:
## Step 3 - Get a perceptron, and train it
%% Cell type:code id: tags:
``` python
pct = Perceptron(max_iter=100, random_state=82, tol=0.01, verbose=1)
pct.fit(x_train, y_train)
```
%% Output
-- Epoch 1
Norm: 1.56, NNZs: 2, Bias: 3.000000, T: 120, Avg. loss: 0.176917
Total training time: 0.00 seconds.
-- Epoch 2
Norm: 1.56, NNZs: 2, Bias: 3.000000, T: 240, Avg. loss: 0.000000
Total training time: 0.00 seconds.
-- Epoch 3
Norm: 1.56, NNZs: 2, Bias: 3.000000, T: 360, Avg. loss: 0.000000
Total training time: 0.00 seconds.
-- Epoch 4
Norm: 1.56, NNZs: 2, Bias: 3.000000, T: 480, Avg. loss: 0.000000
Total training time: 0.00 seconds.
-- Epoch 5
Norm: 1.56, NNZs: 2, Bias: 3.000000, T: 600, Avg. loss: 0.000000
Total training time: 0.00 seconds.
-- Epoch 6
Norm: 1.56, NNZs: 2, Bias: 3.000000, T: 720, Avg. loss: 0.000000
Total training time: 0.00 seconds.
-- Epoch 7
Norm: 1.56, NNZs: 2, Bias: 3.000000, T: 840, Avg. loss: 0.000000
Total training time: 0.00 seconds.
Convergence after 7 epochs took 0.00 seconds
Perceptron(max_iter=100, random_state=82, tol=0.01, verbose=1)
%% Cell type:markdown id: tags:
## Step 4 - Prédictions
%% Cell type:code id: tags:
``` python
y_pred = pct.predict(x_test)
df=pd.DataFrame.from_dict({'Length (x1)':x_test[:,0], 'Width (x2)':x_test[:,1], 'y_test':y_test, 'y_pred':y_pred})
display(df[:15])
```
%% Output
%% Cell type:markdown id: tags:
## Step 5 - Visualisation
%% Cell type:code id: tags:
``` python
def plot_perceptron(x_train,y_train,x_test,y_test):
a = -pct.coef_[0][0] / pct.coef_[0][1]
b = -pct.intercept_ / pct.coef_[0][1]
box=[x.min(axis=0)[0],x.max(axis=0)[0],x.min(axis=0)[1],x.max(axis=0)[1]]
mx=(box[1]-box[0])/20
my=(box[3]-box[2])/20
box=[box[0]-mx,box[1]+mx,box[2]-my,box[3]+my]
fig, axs = plt.subplots(1, 1)
fig.set_size_inches(10,6)
axs.plot(x_train[y_train==1, 0], x_train[y_train==1, 1], "o", color='tomato', label="Iris-Setosa")
axs.plot(x_train[y_train==0, 0], x_train[y_train==0, 1], "o", color='steelblue',label="Autres")
axs.plot(x_test[y_pred==1, 0], x_test[y_pred==1, 1], "o", color='lightsalmon', label="Iris-Setosa (pred)")
axs.plot(x_test[y_pred==0, 0], x_test[y_pred==0, 1], "o", color='lightblue', label="Autres (pred)")
axs.plot([box[0], box[1]], [a*box[0]+b, a*box[1]+b], "k--", linewidth=2)
axs.set_xlabel("Petal length (cm)", labelpad=15) #, fontsize=14)
axs.set_ylabel("Petal width (cm)", labelpad=15) #, fontsize=14)
axs.legend(loc="lower right", fontsize=14)
axs.set_xlim(box[0],box[1])
axs.set_ylim(box[2],box[3])
pwk.save_fig('01-perceptron-iris')
plt.show()
plot_perceptron(x_train,y_train, x_test,y_test)
```
%% Output
%% Cell type:code id: tags:
``` python
pwk.end()
```
%% Output
End time is : Wednesday 16 December 2020, 21:02:00
Duration is : 00:00:01 537ms
This notebook ends here
%% Cell type:markdown id: tags:
---
<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>
LinearReg/01-Linear-Regression.ipynb
View file @ 1cff5b5f
%% Cell type:markdown id: tags:
<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
<img width="800px" src="../fidle/img/header.svg"></img>
# <!-- TITLE --> [LINR1] - Linear regression with direct resolution
<!-- DESC --> Direct determination of linear regression
<!-- DESC --> Low-level implementation, using numpy, of a direct resolution for a linear regression
<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
## Objectives :
- Just one, the illustration of a direct resolution :-)
## What we're going to do :
Equation : $ Y = X.\theta + N$
Equation : $$Y = X.\theta + N$$
Where N is a noise vector
and $\theta = (a,b)$ a vector as y = a.x + b
%% Cell type:markdown id: tags:
## Step 1 - Import and init
%% Cell type:code id: tags:
``` python
import numpy as np
import math
import matplotlib
import matplotlib.pyplot as plt
import sys
import fidle
sys.path.append('..')
import fidle.pwk as pwk
datasets_dir = pwk.init('LINR1')
# Init Fidle environment
run_id, run_dir, datasets_dir = fidle.init('LINR1')
```
%% Output
**FIDLE 2020 - Practical Work Module**
Version : 0.6.1 DEV
Notebook id : LINR1
Run time : Thursday 17 December 2020, 16:30:24
TensorFlow version : 2.1.0
Keras version : 2.2.4-tf
Datasets dir : /gpfswork/rech/mlh/uja62cb/datasets
Running mode : full
Update keras cache : False
%% Cell type:markdown id: tags:
## Step 2 - Retrieve a set of points
%% Cell type:code id: tags:
``` python
# ---- Paramètres
nb = 100 # Nombre de points
xmin = 0 # Distribution / x
xmax = 10
a = 4 # Distribution / y
b = 2 # y= a.x + b (+ bruit)
noise = 7 # bruit
theta = np.array([[a],[b]])
# ---- Vecteur X (1,x) x nb
# la premiere colonne est a 1 afin que X.theta <=> 1.b + x.a
Xc1 = np.ones((nb,1))
Xc2 = np.random.uniform(xmin,xmax,(nb,1))
X = np.c_[ Xc1, Xc2 ]
# ---- Noise
# N = np.random.uniform(-noise,noise,(nb,1))
N = noise * np.random.normal(0,1,(nb,1))
# ---- Vecteur Y
Y = (X @ theta) + N
# print("X:\n",X,"\nY:\n ",Y)
```
%% Cell type:markdown id: tags:
### Show it
%% Cell type:code id: tags:
``` python
width = 12
height = 6
fig, ax = plt.subplots()
fig.set_size_inches(width,height)
ax.plot(X[:,1], Y, ".")
ax.tick_params(axis='both', which='both', bottom=False, left=False, labelbottom=False, labelleft=False)
ax.set_xlabel('x axis')
ax.set_ylabel('y axis')
pwk.save_fig('01-set_of_points')
fidle.scrawler.save_fig('01-set_of_points')
plt.show()
```
%% Output
%% Cell type:markdown id: tags:
## Step 3 - Direct calculation of the normal equation
We'll try to find an optimal value of $\theta$, minimizing a cost function.
The cost function, classically used in the case of linear regressions, is the **root mean square error** (racine carré de l'erreur quadratique moyenne):
$RMSE(X,h_\theta)=\sqrt{\frac1n\sum_{i=1}^n\left[h_\theta(X^{(i)})-Y^{(i)}\right]^2}$
$$RMSE(X,h_\theta)=\sqrt{\frac1n\sum_{i=1}^n\left[h_\theta(X^{(i)})-Y^{(i)}\right]^2}$$
With the simplified variant : $MSE(X,h_\theta)=\frac1n\sum_{i=1}^n\left[h_\theta(X^{(i)})-Y^{(i)}\right]^2$
With the simplified variant : $$MSE(X,h_\theta)=\frac1n\sum_{i=1}^n\left[h_\theta(X^{(i)})-Y^{(i)}\right]^2$$
The optimal value of regression is : $ \hat{ \theta } =( X^{-T} .X)^{-1}.X^{-T}.Y$
The optimal value of regression is : $$\hat{ \theta } =( X^{T} .X)^{-1}.X^{T}.Y$$
Démontstration : https://eli.thegreenplace.net/2014/derivation-of-the-normal-equation-for-linear-regression
%% Cell type:code id: tags:
``` python
theta_hat = np.linalg.inv(X.T @ X) @ X.T @ Y
print("Theta :\n",theta,"\n\ntheta hat :\n",theta_hat)
```
%% Output
Theta :
[[4]
[2]]
theta hat :
[[6.57350113]
[1.53229674]]
%% Cell type:markdown id: tags:
### Show it
%% Cell type:code id: tags:
``` python
Xd = np.array([[1,xmin], [1,xmax]])
Yd = Xd @ theta_hat
fig, ax = plt.subplots()
fig.set_size_inches(width,height)
ax.plot(X[:,1], Y, ".")
ax.plot(Xd[:,1], Yd, "-")
ax.tick_params(axis='both', which='both', bottom=False, left=False, labelbottom=False, labelleft=False)
ax.set_xlabel('x axis')
ax.set_ylabel('y axis')
pwk.save_fig('02-regression-line')
fidle.scrawler.save_fig('02-regression-line')
plt.show()
```
%% Output
%% Cell type:code id: tags:
``` python
pwk.end()
fidle.end()
```
%% Output
End time is : Thursday 17 December 2020, 16:30:38
Duration is : 00:00:13 289ms
This notebook ends here
%% Cell type:markdown id: tags:
---
<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>
<img width="80px" src="../fidle/img/logo-paysage.svg"></img>
......
LinearReg/02-Gradient-descent.ipynb
View file @ 1cff5b5f
source diff could not be displayed: it is too large. Options to address this: view the blob.
LinearReg/03-Polynomial-Regression.ipynb
View file @ 1cff5b5f
%% Cell type:markdown id: tags:
<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
<img width="800px" src="../fidle/img/header.svg"></img>
# <!-- TITLE --> [POLR1] - Complexity Syndrome
<!-- DESC --> Illustration of the problem of complexity with the polynomial regression
<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
## Objectives :
- Visualizing and understanding under and overfitting
## What we're going to do :
We are looking for a polynomial function to approximate the observed series :
$ y = a_n\cdot x^n + \dots + a_i\cdot x^i + \dots + a_1\cdot x + b $
## Step 1 - Import and init
%% Cell type:code id: tags:
``` python
import numpy as np
import math
import random
import matplotlib
import matplotlib.pyplot as plt
import sys
import fidle
sys.path.append('..')
import fidle.pwk as pwk
datasets_dir = pwk.init('POLR1')
# Init Fidle environment
run_id, run_dir, datasets_dir = fidle.init('POLR1')
```
%% Output
**FIDLE 2020 - Practical Work Module**
Version : 0.6.1 DEV
Notebook id : POLR1
Run time : Wednesday 16 December 2020, 17:48:01
TensorFlow version : 2.0.0
Keras version : 2.2.4-tf
Datasets dir : ~/datasets/fidle
Update keras cache : False
Save figs : True
Path figs : ./run/figs
%% Cell type:markdown id: tags:
## Step 2 - Dataset generation
%% Cell type:code id: tags:
``` python
# ---- Parameters
n = 100
xob_min = -5
xob_max = 5
deg = 7
a_min = -2
a_max = 2
noise = 2000
# ---- Train data
# X,Y : data
# X_norm,Y_norm : normalized data
X = np.random.uniform(xob_min,xob_max,(n,1))
# N = np.random.uniform(-noise,noise,(n,1))
N = noise * np.random.normal(0,1,(n,1))
a = np.random.uniform(a_min,a_max, (deg,))
fy = np.poly1d( a )
Y = fy(X) + N
# ---- Data normalization
#
X_norm = (X - X.mean(axis=0)) / X.std(axis=0)
Y_norm = (Y - Y.mean(axis=0)) / Y.std(axis=0)
# ---- Data visualization
width = 12
height = 6
nb_viz = min(2000,n)
def vector_infos(name,V):
m=V.mean(axis=0).item()
s=V.std(axis=0).item()
print("{:8} : mean={:+12.4f} std={:+12.4f} min={:+12.4f} max={:+12.4f}".format(name,m,s,V.min(),V.max()))
pwk.display_md('#### Generator :')
fidle.utils.display_md('#### Generator :')
print(f"Nomber of points={n} deg={deg} bruit={noise}")
pwk.display_md('#### Datasets :')
fidle.utils.display_md('#### Datasets :')
print(f"{nb_viz} points visibles sur {n})")
plt.figure(figsize=(width, height))
plt.plot(X[:nb_viz], Y[:nb_viz], '.')
plt.tick_params(axis='both', which='both', bottom=False, left=False, labelbottom=False, labelleft=False)
plt.xlabel('x axis')
plt.ylabel('y axis')
pwk.save_fig("01-dataset")
fidle.scrawler.save_fig("01-dataset")
plt.show()
pwk.display_md('#### Before normalization :')
fidle.utils.display_md('#### Before normalization :')
vector_infos('X',X)
vector_infos('Y',Y)
pwk.display_md('#### After normalization :')
fidle.utils.display_md('#### After normalization :')
vector_infos('X_norm',X_norm)
vector_infos('Y_norm',Y_norm)
```
%% Output
#### Generator :
Nomber of points=100 deg=7 bruit=2000
#### Datasets :
100 points visibles sur 100)
#### Before normalization :
X : mean= +0.3946 std= +2.7857 min= -4.7119 max= +4.9613
Y : mean= +944.8316 std= +2687.1875 min= -4221.2450 max= +11553.9320
#### After normalization :
X_norm : mean= -0.0000 std= +1.0000 min= -1.8331 max= +1.6393
Y_norm : mean= +0.0000 std= +1.0000 min= -1.9225 max= +3.9480
%% Cell type:markdown id: tags:
## Step 3 - Polynomial regression with NumPy
### 3.1 - Underfitting
%% Cell type:code id: tags:
``` python
def draw_reg(X_norm, Y_norm, x_hat,fy_hat, size, save_as):
plt.figure(figsize=size)
plt.plot(X_norm, Y_norm, '.')
x_hat = np.linspace(X_norm.min(), X_norm.max(), 100)
plt.plot(x_hat, fy_hat(x_hat))
plt.tick_params(axis='both', which='both', bottom=False, left=False, labelbottom=False, labelleft=False)
plt.xlabel('x axis')
plt.ylabel('y axis')
pwk.save_fig(save_as)
fidle.scrawler.save_fig(save_as)
plt.show()
```
%% Cell type:code id: tags:
``` python
reg_deg=1
a_hat = np.polyfit(X_norm.reshape(-1,), Y_norm.reshape(-1,), reg_deg)
fy_hat = np.poly1d( a_hat )
print(f'Nombre de degrés : {reg_deg}')
draw_reg(X_norm[:nb_viz],Y_norm[:nb_viz], X_norm,fy_hat, (width,height), save_as='02-underfitting')
```
%% Output
Nombre de degrés : 1
%% Cell type:markdown id: tags:
### 3.2 - Good fitting
%% Cell type:code id: tags:
``` python
reg_deg=5
a_hat = np.polyfit(X_norm.reshape(-1,), Y_norm.reshape(-1,), reg_deg)
fy_hat = np.poly1d( a_hat )
print(f'Nombre de degrés : {reg_deg}')
draw_reg(X_norm[:nb_viz],Y_norm[:nb_viz], X_norm,fy_hat, (width,height), save_as='03-good_fitting')
```
%% Output
Nombre de degrés : 5
%% Cell type:markdown id: tags:
### 3.3 - Overfitting
%% Cell type:code id: tags:
``` python
reg_deg=24
a_hat = np.polyfit(X_norm.reshape(-1,), Y_norm.reshape(-1,), reg_deg)
fy_hat = np.poly1d( a_hat )
print(f'Nombre de degrés : {reg_deg}')
draw_reg(X_norm[:nb_viz],Y_norm[:nb_viz], X_norm,fy_hat, (width,height), save_as='04-over_fitting')
```
%% Output
Nombre de degrés : 24
%% Cell type:code id: tags:
``` python
pwk.end()
fidle.end()
```
%% Output
End time is : Wednesday 16 December 2020, 17:48:03
Duration is : 00:00:01 491ms
This notebook ends here
%% Cell type:markdown id: tags:
---
<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>
<img width="80px" src="../fidle/img/logo-paysage.svg"></img>
......
LinearReg/04-Logistic-Regression.ipynb
View file @ 1cff5b5f
source diff could not be displayed: it is too large. Options to address this: view the blob.
LinearReg/modules/RegressionCooker.py
View file @ 1cff5b5f
......@@ -20,19 +20,17 @@ import matplotlib
import matplotlib.pyplot as plt
from IPython.display import display,Markdown,HTML
sys.path.append('..')
import fidle.pwk as pwk
class RegressionCooker():
pwk = None
fidle = None
version = '0.1'
def __init__(self, pwk):
self.pwk = pwk
pwk.subtitle('FIDLE 2020 - Regression Cooker')
def __init__(self, fidle):
self.fidle = fidle
fidle.utils.subtitle('FIDLE 2020 - Regression Cooker')
print('Version :', self.version)
print('Run time : {}'.format(time.strftime("%A %-d %B %Y, %H:%M:%S")))
print('Run time : {}'.format(time.strftime("%A %d %B %Y, %H:%M:%S")))
@classmethod
......@@ -101,7 +99,7 @@ class RegressionCooker():
print(f"X shape : {X.shape} Y shape : {Y.shape} plot : {nb_viz} points")
plt.figure(figsize=(width, height))
plt.plot(X[:nb_viz], Y[:nb_viz], '.')
self.pwk.save_fig('01-dataset')
self.fidle.scrawler.save_fig('01-dataset')
plt.show()
self.vector_infos('X',X)
self.vector_infos('Y',Y)
......@@ -189,13 +187,13 @@ class RegressionCooker():
# ---- Visualization
pwk.subtitle('Visualization :')
self.pwk.save_fig('02-basic_descent')
self.fidle.utils.subtitle('Visualization :')
self.fidle.scrawler.save_fig('02-basic_descent')
plt.show()
pwk.subtitle('Loss :')
self.fidle.utils.subtitle('Loss :')
self.__plot_loss(loss)
self.pwk.save_fig('03-basic_descent_loss')
self.fidle.scrawler.save_fig('03-basic_descent_loss')
plt.show()
return theta
......@@ -268,14 +266,14 @@ class RegressionCooker():
# ---- Visualization
pwk.subtitle('Visualization :')
self.pwk.save_fig('04-minibatch_descent')
self.fidle.utils.subtitle('Visualization :')
self.fidle.scrawler.save_fig('04-minibatch_descent')
plt.show()
pwk.subtitle('Loss :')
self.fidle.utils.subtitle('Loss :')
self.__plot_loss(loss)
self.pwk.save_fig('05-minibatch_descent_loss')
self.fidle.scrawler.save_fig('05-minibatch_descent_loss')
plt.show()
return theta
\ No newline at end of file
return theta
MNIST.Keras3/01-DNN-MNIST.ipynb 0 → 100644
View file @ 1cff5b5f
%% Cell type:markdown id: tags:
<img width="800px" src="../fidle/img/header.svg"></img>
# <!-- TITLE --> [K3MNIST1] - Simple classification with DNN
<!-- DESC --> An example of classification using a dense neural network for the famous MNIST dataset
<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
## Objectives :
- Recognizing handwritten numbers
- Understanding the principle of a classifier DNN network
- Implementation with Keras
The [MNIST dataset](http://yann.lecun.com/exdb/mnist/) (Modified National Institute of Standards and Technology) is a must for Deep Learning.
It consists of 60,000 small images of handwritten numbers for learning and 10,000 for testing.
## What we're going to do :
- Retrieve data
- Preparing the data
- Create a model
- Train the model
- Evaluate the result
%% Cell type:markdown id: tags:
## Step 1 - Init python stuff
%% Cell type:code id: tags:
``` python
import os
os.environ['KERAS_BACKEND'] = 'torch'
import keras
import numpy as np
import matplotlib.pyplot as plt
import sys,os
from importlib import reload
# Init Fidle environment
import fidle
run_id, run_dir, datasets_dir = fidle.init('K3MNIST1')
```
%% Cell type:markdown id: tags:
Verbosity during training : 0 = silent, 1 = progress bar, 2 = one line per epoch
%% Cell type:code id: tags:
``` python
fit_verbosity = 1
```
%% Cell type:markdown id: tags:
Override parameters (batch mode) - Just forget this cell
%% Cell type:code id: tags:
``` python
fidle.override('fit_verbosity')
```
%% Cell type:markdown id: tags:
## Step 2 - Retrieve data
MNIST is one of the most famous historic dataset.
Include in [Keras datasets](https://keras.io/datasets)
%% Cell type:code id: tags:
``` python
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
print("x_train : ",x_train.shape)
print("y_train : ",y_train.shape)
print("x_test : ",x_test.shape)
print("y_test : ",y_test.shape)
```
%% Cell type:markdown id: tags:
## Step 3 - Preparing the data
%% Cell type:code id: tags:
``` python
print('Before normalization : Min={}, max={}'.format(x_train.min(),x_train.max()))
xmax=x_train.max()
x_train = x_train / xmax
x_test = x_test / xmax
print('After normalization : Min={}, max={}'.format(x_train.min(),x_train.max()))
```
%% Cell type:markdown id: tags:
### Have a look
%% Cell type:code id: tags:
``` python
fidle.scrawler.images(x_train, y_train, [27], x_size=5,y_size=5, colorbar=True, save_as='01-one-digit')
fidle.scrawler.images(x_train, y_train, range(5,41), columns=12, save_as='02-many-digits')
```
%% Cell type:markdown id: tags:
## Step 4 - Create model
About informations about :
- [Optimizer](https://keras.io/api/optimizers)
- [Activation](https://keras.io/api/layers/activations)
- [Loss](https://keras.io/api/losses)
- [Metrics](https://keras.io/api/metrics)
%% Cell type:code id: tags:
``` python
hidden1 = 100
hidden2 = 100
model = keras.Sequential([
keras.layers.Input((28,28)),
keras.layers.Flatten(),
keras.layers.Dense( hidden1, activation='relu'),
keras.layers.Dense( hidden2, activation='relu'),
keras.layers.Dense( 10, activation='softmax')
])
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
```
%% Cell type:markdown id: tags:
## Step 5 - Train the model
%% Cell type:code id: tags:
``` python
batch_size = 512
epochs = 16
history = model.fit( x_train, y_train,
batch_size = batch_size,
epochs = epochs,
verbose = fit_verbosity,
validation_data = (x_test, y_test))
```
%% Cell type:markdown id: tags:
## Step 6 - Evaluate
### 6.1 - Final loss and accuracy
%% Cell type:code id: tags:
``` python
score = model.evaluate(x_test, y_test, verbose=0)
print('Test loss :', score[0])
print('Test accuracy :', score[1])
```
%% Cell type:markdown id: tags:
### 6.2 - Plot history
%% Cell type:code id: tags:
``` python
fidle.scrawler.history(history, figsize=(6,4), save_as='03-history')
```
%% Cell type:markdown id: tags:
### 6.3 - Plot results
%% Cell type:code id: tags:
``` python
#y_pred = model.predict_classes(x_test) Deprecated after 01/01/2021 !!
y_sigmoid = model.predict(x_test, verbose=fit_verbosity)
y_pred = np.argmax(y_sigmoid, axis=-1)
fidle.scrawler.images(x_test, y_test, range(0,200), columns=12, x_size=1, y_size=1, y_pred=y_pred, save_as='04-predictions')
```
%% Cell type:markdown id: tags:
### 6.4 - Plot some errors
%% Cell type:code id: tags:
``` python
errors=[ i for i in range(len(x_test)) if y_pred[i]!=y_test[i] ]
errors=errors[:min(24,len(errors))]
fidle.scrawler.images(x_test, y_test, errors[:15], columns=6, x_size=2, y_size=2, y_pred=y_pred, save_as='05-some-errors')
```
%% Cell type:code id: tags:
``` python
fidle.scrawler.confusion_matrix(y_test,y_pred,range(10),normalize=True, save_as='06-confusion-matrix')
```
%% Cell type:code id: tags:
``` python
fidle.end()
```
%% Cell type:markdown id: tags:
<div class="todo">
A few things you can do for fun:
<ul>
<li>Changing the network architecture (layers, number of neurons, etc.)</li>
<li>Display a summary of the network</li>
<li>Retrieve and display the softmax output of the network, to evaluate its "doubts".</li>
</ul>
</div>
%% Cell type:markdown id: tags:
---
<img width="80px" src="../fidle/img/logo-paysage.svg"></img>
MNIST.Keras3/02-CNN-MNIST.ipynb 0 → 100644
View file @ 1cff5b5f
%% Cell type:markdown id: tags:
<img width="800px" src="../fidle/img/header.svg"></img>
# <!-- TITLE --> [K3MNIST2] - Simple classification with CNN
<!-- DESC --> An example of classification using a convolutional neural network for the famous MNIST dataset
<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
## Objectives :
- Recognizing handwritten numbers
- Understanding the principle of a classifier DNN network
- Implementation with Keras
The [MNIST dataset](http://yann.lecun.com/exdb/mnist/) (Modified National Institute of Standards and Technology) is a must for Deep Learning.
It consists of 60,000 small images of handwritten numbers for learning and 10,000 for testing.
## What we're going to do :
- Retrieve data
- Preparing the data
- Create a model
- Train the model
- Evaluate the result
%% Cell type:markdown id: tags:
## Step 1 - Init python stuff
%% Cell type:code id: tags:
``` python
import os
os.environ['KERAS_BACKEND'] = 'torch'
import keras
import numpy as np
import matplotlib.pyplot as plt
import sys,os
from importlib import reload
# Init Fidle environment
import fidle
run_id, run_dir, datasets_dir = fidle.init('K3MNIST2')
```
%% Cell type:markdown id: tags:
Verbosity during training : 0 = silent, 1 = progress bar, 2 = one line per epoch
%% Cell type:code id: tags:
``` python
fit_verbosity = 1
```
%% Cell type:markdown id: tags:
Override parameters (batch mode) - Just forget this cell
%% Cell type:code id: tags:
``` python
fidle.override('fit_verbosity')
```
%% Cell type:markdown id: tags:
## Step 2 - Retrieve data
MNIST is one of the most famous historic dataset.
Include in [Keras datasets](https://keras.io/datasets)
%% Cell type:code id: tags:
``` python
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
x_train = x_train.reshape(-1,28,28,1)
x_test = x_test.reshape(-1,28,28,1)
print("x_train : ",x_train.shape)
print("y_train : ",y_train.shape)
print("x_test : ",x_test.shape)
print("y_test : ",y_test.shape)
```
%% Cell type:markdown id: tags:
## Step 3 - Preparing the data
%% Cell type:code id: tags:
``` python
print('Before normalization : Min={}, max={}'.format(x_train.min(),x_train.max()))
xmax=x_train.max()
x_train = x_train / xmax
x_test = x_test / xmax
print('After normalization : Min={}, max={}'.format(x_train.min(),x_train.max()))
```
%% Cell type:markdown id: tags:
### Have a look
%% Cell type:code id: tags:
``` python
fidle.scrawler.images(x_train, y_train, [27], x_size=5,y_size=5, colorbar=True, save_as='01-one-digit')
fidle.scrawler.images(x_train, y_train, range(5,41), columns=12, save_as='02-many-digits')
```
%% Cell type:markdown id: tags:
## Step 4 - Create model
About informations about :
- [Optimizer](https://keras.io/api/optimizers)
- [Activation](https://keras.io/api/layers/activations)
- [Loss](https://keras.io/api/losses)
- [Metrics](https://keras.io/api/metrics)
%% Cell type:code id: tags:
``` python
model = keras.models.Sequential()
model.add( keras.layers.Input((28,28,1)) )
model.add( keras.layers.Conv2D(8, (3,3), activation='relu') )
model.add( keras.layers.MaxPooling2D((2,2)))
model.add( keras.layers.Dropout(0.2))
model.add( keras.layers.Conv2D(16, (3,3), activation='relu') )
model.add( keras.layers.MaxPooling2D((2,2)))
model.add( keras.layers.Dropout(0.2))
model.add( keras.layers.Flatten())
model.add( keras.layers.Dense(100, activation='relu'))
model.add( keras.layers.Dropout(0.5))
model.add( keras.layers.Dense(10, activation='softmax'))
```
%% Cell type:code id: tags:
``` python
model.summary()
model.compile(optimizer='adam',
loss='sparse_categorical_crossentropy',
metrics=['accuracy'])
```
%% Cell type:markdown id: tags:
## Step 5 - Train the model
%% Cell type:code id: tags:
``` python
batch_size = 512
epochs = 16
history = model.fit( x_train, y_train,
batch_size = batch_size,
epochs = epochs,
verbose = fit_verbosity,
validation_data = (x_test, y_test))
```
%% Cell type:markdown id: tags:
## Step 6 - Evaluate
### 6.1 - Final loss and accuracy
Note : With a DNN, we had a precision of the order of : 97.7%
%% Cell type:code id: tags:
``` python
score = model.evaluate(x_test, y_test, verbose=0)
print(f'Test loss : {score[0]:4.4f}')
print(f'Test accuracy : {score[1]:4.4f}')
```
%% Cell type:markdown id: tags:
### 6.2 - Plot history
%% Cell type:code id: tags:
``` python
fidle.scrawler.history(history, figsize=(6,4), save_as='03-history')
```
%% Cell type:markdown id: tags:
### 6.3 - Plot results
%% Cell type:code id: tags:
``` python
#y_pred = model.predict_classes(x_test) Deprecated after 01/01/2021 !!
y_sigmoid = model.predict(x_test, verbose=fit_verbosity)
y_pred = np.argmax(y_sigmoid, axis=-1)
fidle.scrawler.images(x_test, y_test, range(0,200), columns=12, x_size=1, y_size=1, y_pred=y_pred, save_as='04-predictions')
```
%% Cell type:markdown id: tags:
### 6.4 - Plot some errors
%% Cell type:code id: tags:
``` python
errors=[ i for i in range(len(x_test)) if y_pred[i]!=y_test[i] ]
errors=errors[:min(24,len(errors))]
fidle.scrawler.images(x_test, y_test, errors[:15], columns=6, x_size=2, y_size=2, y_pred=y_pred, save_as='05-some-errors')
```
%% Cell type:code id: tags:
``` python
fidle.scrawler.confusion_matrix(y_test,y_pred,range(10),normalize=True, save_as='06-confusion-matrix')
```
%% Cell type:code id: tags:
``` python
fidle.end()
```
%% Cell type:markdown id: tags:
<div class="todo">
A few things you can do for fun:
<ul>
<li>Changing the network architecture (layers, number of neurons, etc.)</li>
<li>Display a summary of the network</li>
<li>Retrieve and display the softmax output of the network, to evaluate its "doubts".</li>
</ul>
</div>
%% Cell type:markdown id: tags:
---
<img width="80px" src="../fidle/img/logo-paysage.svg"></img>
MNIST.Lightning/01-DNN-MNIST_Lightning.ipynb 0 → 100644
View file @ 1cff5b5f
%% Cell type:markdown id:86fe2213-fb44-4bd4-a371-a541cba6a744 tags:
<img width="800px" src="../fidle/img/header.svg"></img>
# <!-- TITLE --> [LMNIST1] - Simple classification with DNN
<!-- DESC --> An example of classification using a dense neural network for the famous MNIST dataset, using PyTorch Lightning
<!-- AUTHOR : MBOGOL Touye Achille (AI/ML Engineer EFELIA-MIAI/SIMAP Lab) -->
## Objectives :
- Recognizing handwritten numbers
- Understanding the principle of a classifier DNN network
- Implementation with pytorch lightning
The [MNIST dataset](http://yann.lecun.com/exdb/mnist/) (Modified National Institute of Standards and Technology) is a must for Deep Learning.
It consists of 60,000 small images of handwritten numbers for learning and 10,000 for testing.
## What we're going to do :
- Retrieve data
- Preparing the data
- Create a model
- Train the model
- Evaluate the result
%% Cell type:markdown id:7f16101a-6612-4e02-93e9-c45ce1ac911d tags:
## Step 1 - Init python stuff
%% Cell type:code id:743c77d3-0983-491c-90be-ef2219861a47 tags:
``` python
import pandas as pd
import numpy as np
import torch
import torch.nn as nn
import lightning.pytorch as pl
import torch.nn.functional as F
import torchvision.transforms as T
import sys,os
import multiprocessing
from torchvision import datasets
from torchmetrics.functional import accuracy
from torch.utils.data import Dataset, DataLoader
from lightning.pytorch import loggers as pl_loggers
from modules.progressbar import CustomTrainProgressBar
from lightning.pytorch.loggers.tensorboard import TensorBoardLogger
# Init Fidle environment
import fidle
run_id, run_dir, datasets_dir = fidle.init('LMNIST1')
```
%% Cell type:markdown id:df10dcda-aa63-476b-8665-9b1610fe51c6 tags:
## Step 2 Retrieve data
MNIST is one of the most famous historic dataset include in torchvision Datasets. `torchvision` provides many built-in datasets in the `torchvision.datasets`.
%% Cell type:code id:6668e50c-f0c6-43cf-b733-9ac29d6a3900 tags:
``` python
# Load data sets
train_dataset = datasets.MNIST(root="data", train=True, download=True, transform=None)
test_dataset = datasets.MNIST(root="data", train=False, download=True, transform=None)
```
%% Cell type:code id:b543b885-6336-461d-abbe-6d3171405771 tags:
``` python
# print info for train data
print(train_dataset)
print()
# print info for test data
print(test_dataset)
```
%% Cell type:code id:44a489f5-3e53-4a2b-8069-f265b2814dc0 tags:
``` python
# See the shape of train data and test data
print("x_train : ",train_dataset.data.shape)
print("y_train : ",train_dataset.targets.shape)
print()
print("x_test : ",test_dataset.data.shape)
print("y_test : ",test_dataset.targets.shape)
print()
# print number of labels or class
print("Number of Targets :",len(np.unique(train_dataset.targets)))
print("Targets Values :", np.unique(train_dataset.targets))
print("\nRemark that we work with torch tensors and not numpy array, not tensorflow tensor")
print(" -> x_train.dtype = ",train_dataset.data.dtype)
print(" -> y_train.dtype = ",train_dataset.targets.dtype)
```
%% Cell type:markdown id:b418adb7-33ea-450c-9793-3cdce5d5fa8c tags:
## Step 3 - Preparing your data for training with DataLoaders
The Dataset retrieves our dataset’s features and labels one sample at a time. While training a model, we typically want to pass samples in `minibatches`, reshuffle the data at every epoch to reduce model overfitting, and use Python’s multiprocessing to speed up data retrieval. DataLoader is an iterable that abstracts this complexity for us in an easy API.
%% Cell type:code id:8af0bc4c-acb3-46d9-aae2-143b0327d970 tags:
``` python
# Before normalization:
x_train=train_dataset.data
print('Before normalization : Min={}, max={}'.format(x_train.min(),x_train.max()))
# After normalization:
## T.Compose creates a pipeline where the provided transformations are run in sequence
transforms = T.Compose(
[
# This transforms takes a np.array or a PIL image of integers
# in the range 0-255 and transforms it to a float tensor in the
# range 0.0 - 1.0
T.ToTensor()
]
)
train_dataset = datasets.MNIST(root="data", train=True, download=True, transform=transforms)
test_dataset = datasets.MNIST(root="data", train=False, download=True, transform=transforms)
# print image and label After normalization.
## iter() followed by next() is used to get some images and label.
image,label=next(iter(train_dataset))
print('After normalization : Min={}, max={}'.format(image.min(),image.max()))
```
%% Cell type:markdown id:35d50a57-8274-4660-8765-d0f2bf7214bd tags:
### Have a look
%% Cell type:code id:a172ebc5-8858-4f30-8e2c-1e9c123ae0ee tags:
``` python
x_train=train_dataset.data
y_train=train_dataset.targets
```
%% Cell type:code id:5a487760-b43a-4f7c-bfd8-1ce2c9652769 tags:
``` python
fidle.scrawler.images(x_train, y_train, [27], x_size=5, y_size=5, colorbar=True, save_as='01-one-digit')
fidle.scrawler.images(x_train, y_train, range(5,41), columns=12, save_as='02-many-digits')
```
%% Cell type:code id:ca0a63ae-e6d6-4940-b8ff-9b11cb2737bb tags:
``` python
# train bacth data
train_loader= DataLoader(
dataset=train_dataset,
shuffle=True,
batch_size=512,
num_workers=2
)
# test batch data
test_loader= DataLoader(
dataset=test_dataset,
shuffle=False,
batch_size=512,
num_workers=2
)
# print image and label after normalization.
image, label=next(iter(train_loader))
print('Shape of first training data batch after use pytorch dataloader :\nbatch images = {} \nbatch labels = {}'.format(image.shape,label.shape))
```
%% Cell type:markdown id:51bf21ee-76ca-42fa-b67f-066dbd239a72 tags:
## Step 4 - Create Model
About informations about :
- [Optimizer](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers)
- [Activation](https://www.tensorflow.org/api_docs/python/tf/keras/activations)
- [Loss](https://www.tensorflow.org/api_docs/python/tf/keras/losses)
- [Metrics](https://www.tensorflow.org/api_docs/python/tf/keras/metrics)
`Note :` PyTorch provides losses such as the cross-entropy loss (`nn.CrossEntropyLoss`) usually use for classification problem. we're using the softmax function to predict class probabilities. With a softmax output, you want to use cross-entropy as the loss. To actually calculate the loss, we need to pass in the raw output of our network into the loss, not the output of the softmax function. This raw output is usually called the *logits* or *scores*. because in pytorch the cross entropy contain softmax function already.
%% Cell type:code id:16701119-71eb-4f59-a50a-f153b07a74ae tags:
``` python
class MyNet(nn.Module):
def __init__(self,num_class=10):
super().__init__()
self.num_class = num_class
self.model = nn.Sequential(
# Input vector:
nn.Flatten(), # convert each 2D 28x28 image into a contiguous array of 784 pixel values
# first hidden layer
nn.Linear(in_features=1*28*28, out_features=100),
nn.ReLU(),
nn.Dropout1d(0.1), # Combat overfitting
# second hidden layer
nn.Linear(in_features=100, out_features=100),
nn.ReLU(),
nn.Dropout1d(0.1), # Combat overfitting
# logits outpout
nn.Linear(100, num_class)
)
# forward pass
def forward(self, x):
return self.model(x)
```
%% Cell type:code id:37abf99b-f8ec-4048-a65d-f173ee18b234 tags:
``` python
class LitModel(pl.LightningModule):
def __init__(self, MyNet):
super().__init__()
self.MyNet = MyNet
# forward pass
def forward(self, x):
return self.MyNet(x)
def configure_optimizers(self):
# optimizer
optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
return optimizer
def training_step(self, batch, batch_idx):
# defines the train loop
x, y = batch
# forward pass
y_hat = self.MyNet(x)
# computes the cross entropy loss between input logits and target
loss = F.cross_entropy(y_hat, y)
# accuracy metrics
acc = accuracy(y_hat, y,task="multiclass",num_classes=10)
metrics = {"train_loss": loss,
"train_acc" : acc
}
# logs metrics for each training_step
self.log_dict(metrics,
on_step = False,
on_epoch = True,
prog_bar = True,
logger = True
)
return loss
def validation_step(self, batch, batch_idx):
# defines the valid loop.
x, y = batch
# forward pass
y_hat = self.MyNet(x)
# computes the cross entropy loss between input logits and target
loss = F.cross_entropy(y_hat, y)
# accuracy metrics
acc = accuracy(y_hat, y,task="multiclass",num_classes=10)
metrics = {"test_loss": loss,
"test_acc": acc
}
# logs metrics for each validation_step
self.log_dict(metrics,
on_step = False,
on_epoch = True,
prog_bar = True,
logger = True
)
return metrics
def predict_step(self, batch, batch_idx):
# defnie the predict loop
x, y = batch
# forward pass
y_hat = self.MyNet(x)
return y_hat
```
%% Cell type:code id:7546b27e-d492-420a-8d5d-109201b47830 tags:
``` python
# print summary model
model=LitModel(MyNet())
print(model)
```
%% Cell type:markdown id:fb32e85d-bd92-4ca5-a3dc-ddb5ed50ba6b tags:
## Step 5 - Train Model
%% Cell type:code id:96f0e087-f21a-4afc-85c5-3a3c0c353fe1 tags:
``` python
# loggers data
os.makedirs(f'{run_dir}/logs', mode=0o750, exist_ok=True)
logger= TensorBoardLogger(save_dir=f'{run_dir}/logs',name="DNN_logs")
```
%% Cell type:code id:ce975c03-d05d-40c4-92ff-0cc90699c13e tags:
``` python
# train model
# trainer= pl.Trainer(accelerator='auto',
# max_epochs=20,
# logger=logger,
# num_sanity_val_steps=0,
# callbacks=[CustomTrainProgressBar()]
# )
trainer= pl.Trainer(accelerator='auto',
max_epochs=20,
logger=logger,
num_sanity_val_steps=0
)
trainer.fit(model=model, train_dataloaders=train_loader, val_dataloaders=test_loader,)
```
%% Cell type:markdown id:a1191f05-4454-415c-a5ed-e63d9ae56651 tags:
## Step 6 - Evaluate
### 6.1 - Final loss and accuracy
%% Cell type:code id:9f45316e-0d2d-4fc1-b9a8-5fb8aaf5586a tags:
``` python
score=trainer.validate(model=model,dataloaders=test_loader, verbose=False)
print('x_test / acc : {:5.4f}'.format(score[0]['test_acc']))
print('x_test / loss : {:5.4f}'.format(score[0]['test_loss']))
```
%% Cell type:markdown id:e352e48d-b473-4162-a1aa-72d6d4f7aa38 tags:
## 6.2 - Plot history
To access logs with tensorboad :
- Under **Docker**, from a terminal launched via the jupyterlab launcher, use the following command:<br>
```tensorboard --logdir <path-to-logs> --host 0.0.0.0```
- If you're **not using Docker**, from a terminal :<br>
```tensorboard --logdir <path-to-logs>```
**Note:** One tensorboard instance can be used simultaneously.
%% Cell type:markdown id:f00ded6b-a7db-4c5d-b1b2-72264db20bdb tags:
### 6.3 - Plot results
%% Cell type:code id:e387a70d-9c23-4d16-8ef7-879aec7791e2 tags:
``` python
# logits outpout by batch size
y_logits=trainer.predict(model=model,dataloaders=test_loader)
# Concat into single tensor
y_logits=torch.cat(y_logits)
# output probabilities values
y_pred_values=F.softmax(y_logits,dim=1)
# Returns the indices of the maximum output probabilities values
y_pred=torch.argmax(y_pred_values,dim=-1).numpy()
```
%% Cell type:code id:fb2b2eeb-fcd8-453c-93ef-59a960a8bbd5 tags:
``` python
x_test=test_dataset.data
y_test=test_dataset.targets
```
%% Cell type:code id:71187fa9-2ad3-4b23-94b9-1846045bd070 tags:
``` python
fidle.scrawler.images(x_test, y_test, range(0,200), columns=12, x_size=1, y_size=1, y_pred=y_pred, save_as='04-predictions')
```
%% Cell type:markdown id:2fc7b2b9-9115-4848-9aae-2798bf7aa79a tags:
### 6.4 - Plot some errors
%% Cell type:code id:e55f17c4-fce7-423a-9adf-f2511c534ef5 tags:
``` python
errors=[ i for i in range(len(x_test)) if y_pred[i]!=y_test[i] ]
errors=errors[:min(24,len(errors))]
fidle.scrawler.images(x_test, y_test, errors[:15], columns=6, x_size=2, y_size=2, y_pred=y_pred, save_as='05-some-errors')
```
%% Cell type:code id:fea1b396-70ca-4b00-851d-0538a4b347fb tags:
``` python
fidle.scrawler.confusion_matrix(y_test,y_pred,range(10),normalize=True, save_as='06-confusion-matrix')
```
%% Cell type:code id:e982c032-cce8-4c71-8cdc-2af4b31b2914 tags:
``` python
fidle.end()
```
%% Cell type:markdown id:233838c2-c97f-4489-8c79-9247d7b7456b tags:
<div class="todo">
A few things you can do for fun:
<ul>
<li>Changing the network architecture (layers, number of neurons, etc.)</li>
<li>Display a summary of the network</li>
<li>Retrieve and display the softmax output of the network, to evaluate its "doubts".</li>
</ul>
</div>
%% Cell type:markdown id:51b87aa0-d4e9-48bb-8205-4b583f4b0b61 tags:
---
<img width="80px" src="../fidle/img/logo-paysage.svg"></img>
MNIST.Lightning/02-CNN-MNIST_Lightning.ipynb 0 → 100644
View file @ 1cff5b5f
%% Cell type:markdown id:86fe2213-fb44-4bd4-a371-a541cba6a744 tags:
<img width="800px" src="../fidle/img/header.svg"></img>
## <!-- TITLE --> [LMNIST2] - Simple classification with CNN
<!-- DESC --> An example of classification using a convolutional neural network for the famous MNIST dataset, using PyTorch Lightning
<!-- AUTHOR : MBOGOL Touye Achille (AI/ML Engineer MIAI/SIMaP) -->
## Objectives :
- Recognizing handwritten numbers
- Understanding the principle of a classifier DNN network
- Implementation with pytorch lightning
The [MNIST dataset](http://yann.lecun.com/exdb/mnist/) (Modified National Institute of Standards and Technology) is a must for Deep Learning.
It consists of 60,000 small images of handwritten numbers for learning and 10,000 for testing.
## What we're going to do :
- Retrieve data
- Preparing the data
- Create a model
- Train the model
- Evaluate the result
%% Cell type:markdown id:7f16101a-6612-4e02-93e9-c45ce1ac911d tags:
## Step 1 - Init python stuff
%% Cell type:code id:743c77d3-0983-491c-90be-ef2219861a47 tags:
``` python
import pandas as pd
import numpy as np
import torch
import torch.nn as nn
import lightning.pytorch as pl
import torch.nn.functional as F
import torchvision.transforms as T
import sys,os
import multiprocessing
import matplotlib.pyplot as plt
from torchvision import datasets
from torchmetrics.functional import accuracy
from torch.utils.data import Dataset, DataLoader
from modules.progressbar import CustomTrainProgressBar
from lightning.pytorch.loggers import TensorBoardLogger
# Init Fidle environment
import fidle
run_id, run_dir, datasets_dir = fidle.init('LMNIST2')
```
%% Cell type:markdown id:df10dcda-aa63-476b-8665-9b1610fe51c6 tags:
## Step 2 Retrieve data
MNIST is one of the most famous historic dataset include in torchvision Datasets. `torchvision` provides many built-in datasets in the `torchvision.datasets`.
%% Cell type:code id:6668e50c-f0c6-43cf-b733-9ac29d6a3900 tags:
``` python
# Load data sets
train_dataset = datasets.MNIST(root="data", train=True, download=True, transform=None)
test_dataset= datasets.MNIST(root="data", train=False, download=False, transform=None)
```
%% Cell type:code id:a14d6fc2-b913-4eaa-9cde-5ca6785bfa12 tags:
``` python
# print info for train data
print(train_dataset)
print()
# print info for test data
print(test_dataset)
```
%% Cell type:code id:44a489f5-3e53-4a2b-8069-f265b2814dc0 tags:
``` python
# See the shape of train data and test data
print("x_train : ",train_dataset.data.shape)
print("y_train : ",train_dataset.targets.shape)
print()
print("x_test : ",test_dataset.data.shape)
print("y_test : ",test_dataset.targets.shape)
print()
# print number of targets and values targets
print("Number of Targets :",len(np.unique(train_dataset.targets)))
print("Targets Values :", np.unique(train_dataset.targets))
print()
print("Remark that we work with torch tensors and not numpy array, not tensorflow tensor")
print(" -> x_train.dtype = ",train_dataset.data.dtype)
print(" -> y_train.dtype = ",train_dataset.targets.dtype)
```
%% Cell type:markdown id:b418adb7-33ea-450c-9793-3cdce5d5fa8c tags:
## Step 3 - Preparing your data for training with DataLoaders
The Dataset retrieves our dataset’s features and labels one sample at a time. While training a model, we typically want to pass samples in `minibatches`, reshuffle the data at every epoch to reduce model overfitting, and use Python’s multiprocessing to speed up data retrieval. DataLoader is an iterable that abstracts this complexity for us in an easy API.
%% Cell type:code id:8af0bc4c-acb3-46d9-aae2-143b0327d970 tags:
``` python
# Before normalization:
x_train=train_dataset.data
print('Before normalization : Min={}, max={}'.format(x_train.min(),x_train.max()))
# After normalization:
## T.Compose creates a pipeline where the provided transformations are run in sequence
transforms = T.Compose(
[
# This transforms takes a np.array or a PIL image of integers
# in the range 0-255 and transforms it to a float tensor in the
# range 0.0 - 1.0
T.ToTensor(),
]
)
train_dataset = datasets.MNIST(root="data", train=True, download=True, transform=transforms)
test_dataset= datasets.MNIST(root="data", train=False, download=True, transform=transforms)
# print image and label After normalization.
# iter() followed by next() is used to get some images and label
image,label=next(iter(train_dataset))
print('After normalization : Min={}, max={}'.format(image.min(),image.max()))
```
%% Cell type:markdown id:35d50a57-8274-4660-8765-d0f2bf7214bd tags:
### Have a look
%% Cell type:code id:a172ebc5-8858-4f30-8e2c-1e9c123ae0ee tags:
``` python
x_train=train_dataset.data
y_train=train_dataset.targets
```
%% Cell type:code id:5a487760-b43a-4f7c-bfd8-1ce2c9652769 tags:
``` python
fidle.scrawler.images(x_train, y_train, [27], x_size=5,y_size=5, colorbar=True, save_as='01-one-digit')
fidle.scrawler.images(x_train, y_train, range(5,41), columns=12, save_as='02-many-digits')
```
%% Cell type:code id:ca0a63ae-e6d6-4940-b8ff-9b11cb2737bb tags:
``` python
# train bacth data
train_loader= DataLoader(
dataset=train_dataset,
shuffle=True,
batch_size=512,
num_workers=2
)
# test batch data
test_loader= DataLoader(
dataset=test_dataset,
shuffle=False,
batch_size=512,
num_workers=2
)
# print image and label After normalization and batch_size.
image, label=next(iter(train_loader))
print('Shape of first training data batch after use pytorch dataloader :\nbatch images = {} \nbatch labels = {}'.format(image.shape,label.shape))
```
%% Cell type:markdown id:51bf21ee-76ca-42fa-b67f-066dbd239a72 tags:
## Step 4 - Create Model
About informations about :
- [Optimizer](https://www.tensorflow.org/api_docs/python/tf/keras/optimizers)
- [Activation](https://www.tensorflow.org/api_docs/python/tf/keras/activations)
- [Loss](https://www.tensorflow.org/api_docs/python/tf/keras/losses)
- [Metrics](https://www.tensorflow.org/api_docs/python/tf/keras/metrics)
`Note :` PyTorch provides losses such as the cross-entropy loss (`nn.CrossEntropyLoss`) usually use for classification problem. we're using the softmax function to predict class probabilities. With a softmax output, you want to use cross-entropy as the loss. To actually calculate the loss, we need to pass in the raw output of our network into the loss, not the output of the softmax function. This raw output is usually called the *logits* or *scores*. because in pytorch the cross entropy contain softmax function already.
%% Cell type:code id:16701119-71eb-4f59-a50a-f153b07a74ae tags:
``` python
class MyNet(nn.Module):
def __init__(self,num_class=10):
super().__init__()
self.num_class=num_class
self.model = nn.Sequential(
# first convolution
nn.Conv2d(in_channels=1, out_channels=8, kernel_size=3, stride=1, padding=0),
nn.ReLU(),
nn.MaxPool2d((2,2)),
nn.Dropout2d(0.1), # Combat overfitting
# second convolution
nn.Conv2d(in_channels=8, out_channels=16, kernel_size=3, stride=1, padding=0),
nn.ReLU(),
nn.MaxPool2d((2,2)),
nn.Dropout2d(0.1), # Combat overfitting
nn.Flatten(), # convert feature map into feature vectors
# MLP network
nn.Linear(16*5*5,100),
nn.ReLU(),
nn.Dropout1d(0.1), # Combat overfitting
nn.Linear(100, num_class), # logits outpout
)
def forward(self, x):
x=self.model(x) # forward pass
return x
```
%% Cell type:code id:37abf99b-f8ec-4048-a65d-f173ee18b234 tags:
``` python
class LitModel(pl.LightningModule):
def __init__(self, MyNet):
super().__init__()
self.MyNet = MyNet
# forward pass
def forward(self, x):
return self.MyNet(x)
# optimizer
def configure_optimizers(self):
optimizer = torch.optim.Adam(self.parameters(), lr=1e-3)
return optimizer
def training_step(self, batch, batch_idx):
# defines the train loop
x, y = batch
# forward pass
y_hat = self.MyNet(x)
# loss function
loss= F.cross_entropy(y_hat, y)
# metrics accuracy
acc=accuracy(y_hat, y,task="multiclass",num_classes=10)
metrics = {"train_loss": loss, "train_acc": acc}
# logs metrics for each training_step
self.log_dict(metrics,
on_step=False ,
on_epoch=True,
prog_bar=True,
logger=True)
return loss
def validation_step(self, batch, batch_idx):
# defines the valid loop.
x, y = batch
# forward pass
y_hat= self.MyNet(x)
# loss function MSE
loss = F.cross_entropy(y_hat, y)
# metrics accuracy
acc=accuracy(y_hat, y, task="multiclass", num_classes=10)
metrics = {"test_loss": loss, "test_acc": acc}
# logs metrics for each validation_step
self.log_dict(metrics,
on_step = False,
on_epoch = True,
prog_bar = True,
logger = True
)
return metrics
def predict_step(self, batch, batch_idx):
# defnie the predict loop
x, y = batch
# forward pass
y_hat = self.MyNet(x)
return y_hat
```
%% Cell type:code id:489af62f-8f7c-4d1b-a6d0-5a0417e79869 tags:
``` python
# print summary model
model=LitModel(MyNet())
print(model)
```
%% Cell type:markdown id:fb32e85d-bd92-4ca5-a3dc-ddb5ed50ba6b tags:
## Step 5 - Train Model
%% Cell type:code id:756d5e19-6a10-42b8-8971-411389f7d19c tags:
``` python
# loggers data
os.makedirs(f'{run_dir}/logs', mode=0o750, exist_ok=True)
logger= TensorBoardLogger(save_dir=f'{run_dir}/logs',name="CNN_logs")
```
%% Cell type:code id:ce975c03-d05d-40c4-92ff-0cc90699c13e tags:
``` python
# train model
# trainer = pl.Trainer(accelerator='auto',
# max_epochs=16,
# logger=logger,
# num_sanity_val_steps=0,
# callbacks=[CustomTrainProgressBar()]
# )
trainer = pl.Trainer(accelerator='auto',
max_epochs=16,
logger=logger,
num_sanity_val_steps=0
)
trainer.fit(model=model, train_dataloaders=train_loader, val_dataloaders=test_loader)
```
%% Cell type:markdown id:a1191f05-4454-415c-a5ed-e63d9ae56651 tags:
## Step 6 - Evaluate
### 6.1 - Final loss and accuracy
Note : With a DNN, we had a precision of the order of : 97.7%
%% Cell type:code id:9f45316e-0d2d-4fc1-b9a8-5fb8aaf5586a tags:
``` python
# evaluate your model
score=trainer.validate(model=model,dataloaders=test_loader, verbose=False)
print('x_test / acc : {:5.4f}'.format(score[0]['test_acc']))
print('x_test / loss : {:5.4f}'.format(score[0]['test_loss']))
```
%% Cell type:code id:5cfe9bd6-654b-42e0-b430-5f3b816526b0 tags:
``` python
score=trainer.validate(model=model,dataloaders=test_loader, verbose=False)
```
%% Cell type:markdown id:e352e48d-b473-4162-a1aa-72d6d4f7aa38 tags:
## 6.2 - Plot history
To access logs with tensorboad :
- Under **Docker**, from a terminal launched via the jupyterlab launcher, use the following command:<br>
```tensorboard --logdir <path-to-logs> --host 0.0.0.0```
- If you're **not using Docker**, from a terminal :<br>
```tensorboard --logdir <path-to-logs>```
**Note:** One tensorboard instance can be used simultaneously.
%% Cell type:markdown id:f00ded6b-a7db-4c5d-b1b2-72264db20bdb tags:
### 6.3 - Plot results
%% Cell type:code id:e387a70d-9c23-4d16-8ef7-879aec7791e2 tags:
``` python
# logits outpout by batch size
y_logits= trainer.predict(model=model,dataloaders=test_loader)
# Concat into single tensor
y_logits= torch.cat(y_logits)
# output probabilities values
y_pred_values=F.softmax(y_logits,dim=1)
# Returns the indices of the maximum output probabilities values
y_pred=torch.argmax(y_pred_values,dim=-1)
```
%% Cell type:code id:fb2b2eeb-fcd8-453c-93ef-59a960a8bbd5 tags:
``` python
x_test=test_dataset.data
y_test=test_dataset.targets
```
%% Cell type:code id:71187fa9-2ad3-4b23-94b9-1846045bd070 tags:
``` python
fidle.scrawler.images(x_test, y_test, range(0,200), columns=12, x_size=1, y_size=1, y_pred=y_pred, save_as='04-predictions')
```
%% Cell type:markdown id:2fc7b2b9-9115-4848-9aae-2798bf7aa79a tags:
### 6.4 - Plot some errors
%% Cell type:code id:e55f17c4-fce7-423a-9adf-f2511c534ef5 tags:
``` python
errors=[ i for i in range(len(x_test)) if y_pred[i]!=y_test[i] ]
errors=errors[:min(24,len(errors))]
fidle.scrawler.images(x_test, y_test, errors[:15], columns=6, x_size=2, y_size=2, y_pred=y_pred, save_as='05-some-errors')
```
%% Cell type:code id:fea1b396-70ca-4b00-851d-0538a4b347fb tags:
``` python
fidle.scrawler.confusion_matrix(y_test,y_pred,range(10),normalize=True, save_as='06-confusion-matrix')
```
%% Cell type:code id:e982c032-cce8-4c71-8cdc-2af4b31b2914 tags:
``` python
fidle.end()
```
%% Cell type:markdown id:233838c2-c97f-4489-8c79-9247d7b7456b tags:
<div class="todo">
A few things you can do for fun:
<ul>
<li>Changing the network architecture (layers, number of neurons, etc.)</li>
<li>Display a summary of the network</li>
<li>Retrieve and display the softmax output of the network, to evaluate its "doubts".</li>
</ul>
</div>
%% Cell type:markdown id:51b87aa0-d4e9-48bb-8205-4b583f4b0b61 tags:
---
<img width="80px" src="../fidle/img/logo-paysage.svg"></img>
MNIST.Lightning/modules/progressbar.py 0 → 100644
View file @ 1cff5b5f
# ------------------------------------------------------------------
# _____ _ _ _
# | ___(_) __| | | ___
# | |_ | |/ _` | |/ _ \
# | _| | | (_| | | __/
# |_| |_|\__,_|_|\___|
# ------------------------------------------------------------------
# Formation Introduction au Deep Learning (FIDLE)
# CNRS/SARI/DEVLOG 2023
# ------------------------------------------------------------------
# 2.0 version by Achille Mbogol Touye (EFELIA-MIAI/SIMAP¨), sep 2023
from tqdm import tqdm as _tqdm
from lightning.pytorch.callbacks import TQDMProgressBar
# Créez un callback de barre de progression pour afficher les métriques d'entraînement
class CustomTrainProgressBar(TQDMProgressBar):
def __init__(self):
super().__init__()
self._val_progress_bar = _tqdm()
self._predict_progress_bar = _tqdm()
def init_predict_tqdm(self):
bar=super().init_test_tqdm()
bar.set_description("Predicting")
return bar
def init_train_tqdm(self):
bar=super().init_train_tqdm()
bar.set_description("Training")
return bar
@property
def val_progress_bar(self):
if self._val_progress_bar is None:
raise ValueError("The `_val_progress_bar` reference has not been set yet.")
return self._val_progress_bar
@property
def predict_progress_bar(self) -> _tqdm:
if self._predict_progress_bar is None:
raise TypeError(f"The `{self.__class__.__name__}._predict_progress_bar` reference has not been set yet.")
return self._predict_progress_bar
def on_validation_start(self, trainer, pl_module):
# Désactivez l'affichage de la barre de progression de validation
self.val_progress_bar.disable = True
def on_predict_start(self, trainer, pl_module):
# Désactivez l'affichage de la barre de progression de validation
self.predict_progress_bar.disable = True
\ No newline at end of file
MNIST.PyTorch/01-DNN-MNIST_PyTorch.ipynb 0 → 100644
View file @ 1cff5b5f
%% Cell type:markdown id: tags:
<img width="800px" src="../fidle/img/header.svg"></img>
# <!-- TITLE --> [PMNIST1] - Simple classification with DNN
<!-- DESC -->Example of classification with a fully connected neural network, using Pytorch
<!-- AUTHOR : Laurent Risser (CNRS/IMT) -->
## Objectives :
- Recognizing handwritten numbers
- Understanding the principle of a classifier DNN network
- Implementation with PyTorch
The [MNIST dataset](http://yann.lecun.com/exdb/mnist/) (Modified National Institute of Standards and Technology) is a must for Deep Learning.
It consists of 60,000 small images of handwritten numbers for learning and 10,000 for testing.
## What we're going to do :
- Retrieve data
- Preparing the data
- Create a model
- Train the model
- Evaluate the result
%% Cell type:markdown id: tags:
## Step 1 - Init python stuff
%% Cell type:code id: tags:
``` python
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.autograd import Variable
import torchvision #to get the MNIST dataset
import numpy as np
import matplotlib.pyplot as plt
import sys,os
import fidle
from modules.fidle_pwk_additional import convergence_history_CrossEntropyLoss
# Init Fidle environment
run_id, run_dir, datasets_dir = fidle.init('PMNIST1')
```
%% Cell type:markdown id: tags:
## Step 2 - Retrieve data
MNIST is one of the most famous historic dataset.
Include in [torchvision datasets](https://pytorch.org/vision/stable/datasets.html)
%% Cell type:code id: tags:
``` python
#get and format the training set
mnist_trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=None)
x_train=mnist_trainset.data.type(torch.DoubleTensor)
y_train=mnist_trainset.targets
#get and format the test set
mnist_testset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=None)
x_test=mnist_testset.data.type(torch.DoubleTensor)
y_test=mnist_testset.targets
#check data shape and format
print("Size of the train and test observations")
print(" -> x_train : ",x_train.shape)
print(" -> y_train : ",y_train.shape)
print(" -> x_test : ",x_test.shape)
print(" -> y_test : ",y_test.shape)
print("\nRemark that we work with torch tensors and not numpy arrays:")
print(" -> x_train.dtype = ",x_train.dtype)
print(" -> y_train.dtype = ",y_train.dtype)
```
%% Cell type:markdown id: tags:
## Step 3 - Preparing the data
%% Cell type:code id: tags:
``` python
print('Before normalization : Min={}, max={}'.format(x_train.min(),x_train.max()))
xmax=x_train.max()
x_train = x_train / xmax
x_test = x_test / xmax
print('After normalization : Min={}, max={}'.format(x_train.min(),x_train.max()))
```
%% Cell type:markdown id: tags:
### Have a look
%% Cell type:code id: tags:
``` python
np_x_train=x_train.numpy().astype(np.float64)
np_y_train=y_train.numpy().astype(np.uint8)
fidle.scrawler.images(np_x_train,np_y_train , [27], x_size=5,y_size=5, colorbar=True)
fidle.scrawler.images(np_x_train,np_y_train, range(5,41), columns=12)
```
%% Cell type:markdown id: tags:
## Step 4 - Create model
About informations about :
- [Optimizer](https://pytorch.org/docs/stable/optim.html)
- [Basic neural-network blocks](https://pytorch.org/docs/stable/nn.html)
- [Loss](https://pytorch.org/docs/stable/nn.html#loss-functions)
%% Cell type:code id: tags:
``` python
class MyModel(nn.Module):
"""
Basic fully connected neural-network
"""
def __init__(self):
hidden1 = 100
hidden2 = 100
super(MyModel, self).__init__()
self.hidden1 = nn.Linear(784, hidden1)
self.hidden2 = nn.Linear(hidden1, hidden2)
self.hidden3 = nn.Linear(hidden2, 10)
def forward(self, x):
x = x.view(-1,784) #flatten the images before using fully-connected layers
x = self.hidden1(x)
x = F.relu(x)
x = self.hidden2(x)
x = F.relu(x)
x = self.hidden3(x)
x = F.softmax(x, dim=0)
return x
model = MyModel()
```
%% Cell type:markdown id: tags:
## Step 5 - Train the model
### 5.1 - Stochastic gradient descent strategy to fit the model
%% Cell type:code id: tags:
``` python
def fit(model,X_train,Y_train,X_test,Y_test, EPOCHS = 5, BATCH_SIZE = 32):
loss = nn.CrossEntropyLoss()
optimizer = torch.optim.Adam(model.parameters(),lr=1e-3) #lr is the learning rate
model.train()
history=convergence_history_CrossEntropyLoss()
history.update(model,X_train,Y_train,X_test,Y_test)
n=X_train.shape[0] #number of observations in the training data
#stochastic gradient descent
for epoch in range(EPOCHS):
batch_start=0
epoch_shuffler=np.arange(n)
np.random.shuffle(epoch_shuffler) #remark that 'utilsData.DataLoader' could be used instead
while batch_start+BATCH_SIZE < n:
#get mini-batch observation
mini_batch_observations = epoch_shuffler[batch_start:batch_start+BATCH_SIZE]
var_X_batch = Variable(X_train[mini_batch_observations,:,:]).float() #the input image is flattened
var_Y_batch = Variable(Y_train[mini_batch_observations])
#gradient descent step
optimizer.zero_grad() #set the parameters gradients to 0
Y_pred_batch = model(var_X_batch) #predict y with the current NN parameters
curr_loss = loss(Y_pred_batch, var_Y_batch) #compute the current loss
curr_loss.backward() #compute the loss gradient w.r.t. all NN parameters
optimizer.step() #update the NN parameters
#prepare the next mini-batch of the epoch
batch_start+=BATCH_SIZE
history.update(model,X_train,Y_train,X_test,Y_test)
return history
```
%% Cell type:markdown id: tags:
### 5.2 - Fit the model
%% Cell type:code id: tags:
``` python
batch_size = 512
epochs = 128
history=fit(model,x_train,y_train,x_test,y_test,EPOCHS=epochs,BATCH_SIZE = batch_size)
```
%% Cell type:markdown id: tags:
## Step 6 - Evaluate
### 6.1 - Final loss and accuracy
%% Cell type:code id: tags:
``` python
var_x_test = Variable(x_test[:,:,:]).float()
var_y_test = Variable(y_test[:])
y_pred = model(var_x_test)
loss = nn.CrossEntropyLoss()
curr_loss = loss(y_pred, var_y_test)
val_loss = curr_loss.item()
val_accuracy = float( (torch.argmax(y_pred, dim= 1) == var_y_test).float().mean() )
print('Test loss :', val_loss)
print('Test accuracy :', val_accuracy)
```
%% Cell type:markdown id: tags:
### 6.2 - Plot history
%% Cell type:code id: tags:
``` python
fidle.scrawler.history(history, figsize=(6,4))
```
%% Cell type:markdown id: tags:
### 6.3 - Plot results
%% Cell type:code id: tags:
``` python
y_pred = model(var_x_test)
np_y_pred_label = torch.argmax(y_pred, dim= 1).numpy().astype(np.uint8)
np_x_test=x_test.numpy().astype(np.float64)
np_y_test=y_test.numpy().astype(np.uint8)
fidle.scrawler.images(np_x_test, np_y_test, range(0,60), columns=12, x_size=1, y_size=1, y_pred=np_y_pred_label)
```
%% Cell type:markdown id: tags:
### 6.4 - Plot some errors
%% Cell type:code id: tags:
``` python
errors=[ i for i in range(len(np_y_test)) if np_y_pred_label[i]!=np_y_test[i] ]
errors=errors[:min(24,len(errors))]
fidle.scrawler.images(np_x_test, np_y_test, errors[:15], columns=6, x_size=2, y_size=2, y_pred=np_y_pred_label)
```
%% Cell type:code id: tags:
``` python
fidle.scrawler.confusion_matrix(np_y_test,np_y_pred_label, range(10))
```
%% Cell type:markdown id: tags:
<div class="todo">
A few things you can do for fun:
<ul>
<li>Changing the network architecture (layers, number of neurons, etc.)</li>
<li>Display a summary of the network</li>
<li>Retrieve and display the softmax output of the network, to evaluate its "doubts".</li>
</ul>
</div>
%% Cell type:markdown id: tags:
---
<img width="80px" src="../fidle/img/logo-paysage.svg"></img>
MNIST/01-DNN-MNIST.ipynb deleted 100644 → 0
View file @ a434f008
source diff could not be displayed: it is too large. Options to address this: view the blob.
MNIST_PyTorch/.gitkeep deleted 100644 → 0
View file @ a434f008
MNIST_PyTorch/01-DNN-MNIST_PyTorch.ipynb deleted 100644 → 0
View file @ a434f008
source diff could not be displayed: it is too large. Options to address this: view the blob.
Misc/Numpy.ipynb → Misc/00-Numpy.ipynb
View file @ 1cff5b5f
%% Cell type:markdown id: tags:
<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
<img width="800px" src="../fidle/img/header.svg"></img>
# <!-- TITLE --> [NP1] - A short introduction to Numpy
<!-- DESC --> Numpy is an essential tool for the Scientific Python.
<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
## Objectives :
- Comprendre les grands principes de Numpy et son potentiel
- Understand the main principles of Numpy and its potential
Note : This notebook is strongly inspired by the UGA Python Introduction Course
See : **https://gricad-gitlab.univ-grenoble-alpes.fr/python-uga/py-training-2017**
%% Cell type:markdown id: tags:
## Step 1 - Numpy the beginning
Code using `numpy` usually starts with the import statement
%% Cell type:code id: tags:
``` python
import numpy as np
```
%% Cell type:markdown id: tags:
NumPy provides the type `np.ndarray`. Such array are multidimensionnal sequences of homogeneous elements. They can be created for example with the commands:
%% Cell type:code id: tags:
``` python
# from a list
l = [10.0, 12.5, 15.0, 17.5, 20.0]
np.array(l)
```
%% Output
array([10. , 12.5, 15. , 17.5, 20. ])
%% Cell type:code id: tags:
``` python
# fast but the values can be anything
np.empty(4)
```
%% Output
array([ 6.93990061e-310, 6.95333088e-310, -1.90019324e+120,
6.93987701e-310])
%% Cell type:code id: tags:
``` python
# slower than np.empty but the values are all 0.
np.zeros([2, 6])
```
%% Output
array([[0., 0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0., 0.]])
%% Cell type:code id: tags:
``` python
# multidimensional array
a = np.ones([2, 3, 4])
print(a.shape, a.size, a.dtype)
a
```
%% Output
(2, 3, 4) 24 float64
array([[[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]],
[[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]]])
%% Cell type:code id: tags:
``` python
# like range but produce 1D numpy array
np.arange(4)
```
%% Output
array([0, 1, 2, 3])
%% Cell type:code id: tags:
``` python
# np.arange can produce arrays of floats
np.arange(4.)
```
%% Output
array([0., 1., 2., 3.])
%% Cell type:code id: tags:
``` python
# another convenient function to generate 1D arrays
np.linspace(10, 20, 5)
```
%% Output
array([10. , 12.5, 15. , 17.5, 20. ])
%% Cell type:markdown id: tags:
A NumPy array can be easily converted to a Python list.
%% Cell type:code id: tags:
``` python
a = np.linspace(10, 20 ,5)
list(a)
```
%% Output
[10.0, 12.5, 15.0, 17.5, 20.0]
%% Cell type:code id: tags:
``` python
# Or even better
a.tolist()
```
%% Output
[10.0, 12.5, 15.0, 17.5, 20.0]
%% Cell type:markdown id: tags:
## Step 2 - Access elements
Elements in a `numpy` array can be accessed using indexing and slicing in any dimension. It also offers the same functionalities available in Fortan or Matlab.
### 2.1 - Indexes and slices
For example, we can create an array `A` and perform any kind of selection operations on it.
%% Cell type:code id: tags:
``` python
A = np.random.random([4, 5])
A
```
%% Output
array([[0.14726334, 0.90799321, 0.67130094, 0.23978162, 0.96444415],
[0.26039418, 0.06135763, 0.35856793, 0.73366941, 0.50698925],
[0.39557097, 0.55950866, 0.70056205, 0.65344863, 0.90891062],
[0.19049184, 0.56355734, 0.71701494, 0.66035889, 0.06400119]])
%% Cell type:code id: tags:
``` python
# Get the element from second line, first column
A[1, 0]
```
%% Output
0.26039417830707656
%% Cell type:code id: tags:
``` python
# Get the first two lines
A[:2]
```
%% Output
array([[0.14726334, 0.90799321, 0.67130094, 0.23978162, 0.96444415],
[0.26039418, 0.06135763, 0.35856793, 0.73366941, 0.50698925]])
%% Cell type:code id: tags:
``` python
# Get the last column
A[:, -1]
```
%% Output
array([0.96444415, 0.50698925, 0.90891062, 0.06400119])
%% Cell type:code id: tags:
``` python
# Get the first two lines and the columns with an even index
A[:2, ::2]
```
%% Output
array([[0.14726334, 0.67130094, 0.96444415],
[0.26039418, 0.35856793, 0.50698925]])
%% Cell type:markdown id: tags:
### 2.2 - Using a mask to select elements validating a condition:
%% Cell type:code id: tags:
``` python
cond = A > 0.5
print(cond)
print(A[cond])
```
%% Output
[[False True True False True]
[False False False True True]
[False True True True True]
[False True True True False]]
[0.90799321 0.67130094 0.96444415 0.73366941 0.50698925 0.55950866
0.70056205 0.65344863 0.90891062 0.56355734 0.71701494 0.66035889]
%% Cell type:markdown id: tags:
The mask is in fact a particular case of the advanced indexing capabilities provided by NumPy. For example, it is even possible to use lists for indexing:
%% Cell type:code id: tags:
``` python
# Selecting only particular columns
print(A)
A[:, [0, 1, 4]]
```
%% Output
[[0.14726334 0.90799321 0.67130094 0.23978162 0.96444415]
[0.26039418 0.06135763 0.35856793 0.73366941 0.50698925]
[0.39557097 0.55950866 0.70056205 0.65344863 0.90891062]
[0.19049184 0.56355734 0.71701494 0.66035889 0.06400119]]
array([[0.14726334, 0.90799321, 0.96444415],
[0.26039418, 0.06135763, 0.50698925],
[0.39557097, 0.55950866, 0.90891062],
[0.19049184, 0.56355734, 0.06400119]])
%% Cell type:markdown id: tags:
## Step 3 - Perform array manipulations
### 3.1 - Apply arithmetic operations to whole arrays (element-wise):
%% Cell type:code id: tags:
``` python
(A+5)**2
```
%% Output
array([[26.49431985, 34.90438372, 32.16365436, 27.45531142, 35.57459401],
[27.67174691, 25.61734103, 28.71425022, 32.87496493, 30.32693058],
[29.11218606, 30.9081365 , 32.49640767, 31.96148136, 34.91522467],
[26.94120557, 30.95317031, 32.68425986, 32.03966276, 25.6441081 ]])
%% Cell type:markdown id: tags:
### 3.2 - Apply functions element-wise:
%% Cell type:code id: tags:
``` python
np.exp(A) # With numpy arrays, use the functions from numpy !
```
%% Output
array([[1.15865904, 2.47934201, 1.95678132, 1.27097157, 2.62332907],
[1.29744141, 1.0632791 , 1.43127825, 2.08270892, 1.66028496],
[1.48523197, 1.74981253, 2.01488485, 1.92215822, 2.48161763],
[1.2098445 , 1.75691133, 2.04830976, 1.93548684, 1.06609367]])
%% Cell type:markdown id: tags:
### 3.3 - Setting parts of arrays
%% Cell type:code id: tags:
``` python
A[:, 0] = 0.
print(A)
```
%% Output
[[0. 0.90799321 0.67130094 0.23978162 0.96444415]
[0. 0.06135763 0.35856793 0.73366941 0.50698925]
[0. 0.55950866 0.70056205 0.65344863 0.90891062]
[0. 0.56355734 0.71701494 0.66035889 0.06400119]]
%% Cell type:code id: tags:
``` python
# BONUS: Safe element-wise inverse with masks
cond = (A != 0)
A[cond] = 1./A[cond]
print(A)
```
%% Output
[[ 0. 1.10132983 1.48964487 4.17046144 1.03686668]
[ 0. 16.29789234 2.78887186 1.36301171 1.97242842]
[ 0. 1.78728245 1.42742531 1.53034219 1.1002182 ]
[ 0. 1.77444232 1.39467107 1.51432807 15.62470834]]
%% Cell type:markdown id: tags:
## Step 4 - Attributes and methods of `np.ndarray` (see the [doc](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.html#numpy.ndarray))
%% Cell type:code id: tags:
``` python
for i,v in enumerate([s for s in dir(A) if not s.startswith('__')]):
print(f'{v:16}', end='')
if (i+1) % 6 == 0 :print('')
```
%% Output
T all any argmax argmin argpartition
argsort astype base byteswap choose clip
compress conj conjugate copy ctypes cumprod
cumsum data diagonal dot dtype dump
dumps fill flags flat flatten getfield
imag item itemset itemsize max mean
min nbytes ndim newbyteorder nonzero partition
prod ptp put ravel real repeat
reshape resize round searchsorted setfield setflags
shape size sort squeeze std strides
sum swapaxes take tobytes tofile tolist
tostring trace transpose var view
%% Cell type:code id: tags:
``` python
# Ex1: Get the mean through different dimensions
print(A)
print('Mean value', A.mean())
print('Mean line', A.mean(axis=0))
print('Mean column', A.mean(axis=1))
```
%% Output
[[ 0. 1.10132983 1.48964487 4.17046144 1.03686668]
[ 0. 16.29789234 2.78887186 1.36301171 1.97242842]
[ 0. 1.78728245 1.42742531 1.53034219 1.1002182 ]
[ 0. 1.77444232 1.39467107 1.51432807 15.62470834]]
Mean value 2.818696254398785
Mean line [0. 5.24023674 1.77515328 2.14453585 4.93355541]
Mean column [1.55966056 4.48444087 1.16905363 4.06162996]
%% Cell type:code id: tags:
``` python
# Ex2: Convert a 2D array in 1D keeping all elements
print(A)
print(A.shape)
A_flat = A.flatten()
print(A_flat, A_flat.shape)
```
%% Output
[[ 0. 1.10132983 1.48964487 4.17046144 1.03686668]
[ 0. 16.29789234 2.78887186 1.36301171 1.97242842]
[ 0. 1.78728245 1.42742531 1.53034219 1.1002182 ]
[ 0. 1.77444232 1.39467107 1.51432807 15.62470834]]
(4, 5)
[ 0. 1.10132983 1.48964487 4.17046144 1.03686668 0.
16.29789234 2.78887186 1.36301171 1.97242842 0. 1.78728245
1.42742531 1.53034219 1.1002182 0. 1.77444232 1.39467107
1.51432807 15.62470834] (20,)
%% Cell type:markdown id: tags:
### 4.1 - Remark: dot product
%% Cell type:code id: tags:
``` python
b = np.linspace(0, 10, 11)
c = b @ b
# before 3.5:
# c = b.dot(b)
print(b)
print(c)
```
%% Output
[ 0. 1. 2. 3. 4. 5. 6. 7. 8. 9. 10.]
385.0
%% Cell type:markdown id: tags:
### 4.2 - For Matlab users
| ` ` | Matlab | Numpy |
| ------------- | ------ | ----- |
| element wise | `.*` | `*` |
| dot product | `*` | `@` |
%% Cell type:markdown id: tags:
`numpy` arrays can also be sorted, even when they are composed of complex data if the type of the columns are explicitly stated with `dtypes`.
%% Cell type:markdown id: tags:
### 4.3 - NumPy and SciPy sub-packages:
We already saw `numpy.random` to generate `numpy` arrays filled with random values. This submodule also provides functions related to distributions (Poisson, gaussian, etc.) and permutations.
%% Cell type:markdown id: tags:
To perform linear algebra with dense matrices, we can use the submodule `numpy.linalg`. For instance, in order to compute the determinant of a random matrix, we use the method `det`
%% Cell type:code id: tags:
``` python
A = np.random.random([5,5])
print(A)
np.linalg.det(A)
```
%% Output
[[0.33277412 0.18065847 0.10352574 0.48095553 0.97748505]
[0.20756676 0.33166777 0.00808192 0.18868636 0.1722338 ]
[0.94092977 0.21755657 0.52045179 0.45008315 0.1751413 ]
[0.27404121 0.53531168 0.41209088 0.22503687 0.50026306]
[0.23077516 0.99886616 0.74286904 0.40849416 0.57970741]]
-0.026288777656342802
%% Cell type:code id: tags:
``` python
squared_subA = A[1:3, 1:3]
print(squared_subA)
np.linalg.inv(squared_subA)
```
%% Output
[[0.33166777 0.00808192]
[0.21755657 0.52045179]]
array([[ 3.0460928 , -0.04730175],
[-1.27331197, 1.94118039]])
%% Cell type:markdown id: tags:
### 4.4 - Introduction to Pandas: Python Data Analysis Library
Pandas is an open source library providing high-performance, easy-to-use data structures and data analysis tools for Python.
[Pandas tutorial](https://pandas.pydata.org/pandas-docs/stable/10min.html)
[Grenoble Python Working Session](https://github.com/iutzeler/Pres_Pandas/)
[Pandas for SQL Users](http://sergilehkyi.com/translating-sql-to-pandas/)
[Pandas Introduction Training HPC Python@UGA](https://gricad-gitlab.univ-grenoble-alpes.fr/python-uga/training-hpc/-/blob/master/ipynb/11_pandas.ipynb)
%% Cell type:markdown id: tags:
---
<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>
<img width="80px" src="../fidle/img/logo-paysage.svg"></img>
......