{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"German Traffic Sign Recognition Benchmark (GTSRB)\n",
"=================================================\n",
"---\n",
"Introduction au Deep Learning (IDLE) - S. Aria, E. Maldonado, JL. Parouty - CNRS/SARI/DEVLOG - 2020\n",
"\n",
"## Episode 3 : Tracking, visualizing and save models\n",
"\n",
"Our main steps:\n",
" - Monitoring and understanding our model training \n",
" - Add recovery points\n",
" - Analyze the results \n",
" - Restore and run recovery pont\n",
"\n",
"## 1/ Import and init"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import tensorflow as tf\n",
"from tensorflow import keras\n",
"from tensorflow.keras.callbacks import TensorBoard\n",
"\n",
"import numpy as np\n",
"import h5py\n",
"\n",
"from sklearn.metrics import confusion_matrix\n",
"\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sn\n",
"import os, time, random\n",
"\n",
"import idle.pwk as ooo\n",
"from importlib import reload\n",
"\n",
"ooo.init()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 2/ Reload dataset\n",
"Dataset is one of the saved dataset: RGB25, RGB35, L25, L35, etc. \n",
"First of all, we're going to use a smart dataset : **set-24x24-L** \n",
"(with a GPU, it only takes 35'' compared to more than 5' with a CPU !)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%time\n",
"\n",
"dataset ='set-24x24-L'\n",
"\n",
"# ---- Read dataset\n",
"#\n",
"filename='./data/'+dataset+'.h5'\n",
"with h5py.File(filename) as f:\n",
" x_train = f['x_train'][:]\n",
" y_train = f['y_train'][:]\n",
" x_test = f['x_test'][:]\n",
" y_test = f['y_test'][:]\n",
" x_meta = f['x_meta'][:]\n",
" y_meta = f['y_meta'][:]\n",
"\n",
"# ---- Dataset shape\n",
"#\n",
"(n,lx,ly,lz) = x_train.shape\n",
"data_shape = (lx, ly, lz)\n",
"\n",
"# ---- done\n",
"print('Dataset loaded ({:.1f} Mo)\\n'.format(os.path.getsize(filename)/(1024*1024)))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 3/ Have a look to the dataset\n",
"Note: Data must be reshape for matplotlib"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"print(\"x_train : \", x_train.shape)\n",
"print(\"y_train : \", y_train.shape)\n",
"print(\"x_test : \", x_test.shape)\n",
"print(\"y_test : \", y_test.shape)\n",
"\n",
"ooo.plot_images(x_train, y_train, range(6), columns=3, x_size=4, y_size=3)\n",
"ooo.plot_images(x_train, y_train, range(36), columns=12, x_size=1, y_size=1)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 4/ Create model\n",
"Some hyperparameters :"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"batch_size = 64\n",
"num_classes = 43\n",
"epochs = 8"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"My models :"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"\n",
"def get_model_v1():\n",
" model = keras.models.Sequential()\n",
" model.add( keras.layers.Conv2D(96, (3,3), activation='relu', input_shape=(lx,ly,lz)))\n",
" model.add( keras.layers.MaxPooling2D((2, 2)))\n",
" model.add( keras.layers.Conv2D(192, (3, 3), activation='relu'))\n",
" model.add( keras.layers.MaxPooling2D((2, 2)))\n",
" model.add( keras.layers.Flatten()) \n",
" model.add( keras.layers.Dense(500, activation='relu'))\n",
" model.add( keras.layers.Dense(500, activation='relu'))\n",
" model.add( keras.layers.Dense(43, activation='softmax'))\n",
" return model\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# ---- The model I want to test..\n",
"#\n",
"model = get_model_v1()\n",
"model.summary()\n",
"model.compile(optimizer='adam',\n",
" loss='sparse_categorical_crossentropy',\n",
" metrics=['accuracy'])\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5/ Add callbacks\n",
"Nous allons ajouter 2 callbacks : \n",
" - **TensorBoard** \n",
"Training logs, which can be visualised with Tensorboard. \n",
"`#tensorboard --logdir ./run/logs` \n",
"IMPORTANT : Relancer tensorboard à chaque run\n",
" - **Model backup** \n",
" It is possible to save the model each xx epoch or at each improvement. \n",
" The model can be saved completely or partially (weight). \n",
" For full format, we can use HDF5 format."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# To clean old logs and saved model, run this cell\n",
"#\n",
"!/bin/rm -r ./run/logs ./run/models 2>/dev/null\n",
"!/bin/ls -l ./run"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ooo.mkdir('./run/models')\n",
"ooo.mkdir('./run/logs')\n",
"\n",
"# ---- Callback tensorboard\n",
"log_dir = \"./run/logs/tb_\" + ooo.tag_now()\n",
"tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)\n",
"\n",
"# ---- Callback ModelCheckpoint - Save best model\n",
"save_dir = \"./run/models/best-model.h5\"\n",
"bestmodel_callback = tf.keras.callbacks.ModelCheckpoint(filepath=save_dir, verbose=0, monitor='accuracy', save_best_only=True)\n",
"\n",
"# ---- Callback ModelCheckpoint - Save model each epochs\n",
"save_dir = \"./run/models/model-{epoch:04d}.h5\"\n",
"savemodel_callback = tf.keras.callbacks.ModelCheckpoint(filepath=save_dir, verbose=0, save_freq=2000*5)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 5/ Run model"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"%%time\n",
"\n",
"# ---- Shuffle train data\n",
"x_train,y_train=ooo.shuffle_np_dataset(x_train,y_train)\n",
"\n",
"# ---- Train\n",
"# Note: To be faster in our example, we take only 2000 values\n",
"# but in the real world, we'd take the whole dataset!\n",
"#\n",
"history = model.fit( x_train[:2000], y_train[:2000],\n",
" batch_size=batch_size,\n",
" epochs=epochs,\n",
" verbose=1,\n",
" validation_data=(x_test[:200], y_test[:200]),\n",
" callbacks=[tensorboard_callback, bestmodel_callback, savemodel_callback] )\n",
"\n",
"model.save('./run/models/last-model.h5')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 6/ History\n",
"The return of model.fit() returns us the learning history"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"ooo.plot_history(history)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 7/ Evaluation and confusion"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"reload(ooo)\n",
"y_pred = model.predict_classes(x_test)\n",
"conf_mat = confusion_matrix(y_test,y_pred, normalize=\"true\", labels=range(43))\n",
"\n",
"ooo.plot_confusion_matrix(conf_mat)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## 8/ Restore and evaluate\n",
"### 8.1/ List saved models :"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"!find ./run/models/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 8.2/ Restore a model :"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"loaded_model = tf.keras.models.load_model('./run/models/best-model.h5')\n",
"# best_model.summary()\n",
"print(\"Loaded.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 8.3/ Evaluate it :"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"score = loaded_model.evaluate(x_test, y_test, verbose=0)\n",
"\n",
"print('Test loss : {:5.4f}'.format(score[0]))\n",
"print('Test accuracy : {:5.4f}'.format(score[1]))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 8.4/ Make a prediction :"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# ---- Get a random image\n",
"#\n",
"i = random.randint(1,len(x_test))\n",
"x,y = x_test[i], y_test[i]\n",
"\n",
"# ---- Do prediction\n",
"#\n",
"predictions = loaded_model.predict( np.array([x]) )\n",
"\n",
"# ---- A prediction is just the output layer\n",
"#\n",
"print(\"\\nOutput layer from model is (x100) :\\n\")\n",
"with np.printoptions(precision=2, suppress=True, linewidth=95):\n",
" print(predictions*100)\n",
"\n",
"# ---- Graphic visualisation\n",
"#\n",
"print(\"\\nGraphically :\\n\")\n",
"plt.figure(figsize=(12,2))\n",
"plt.bar(range(43), predictions[0], align='center', alpha=0.5)\n",
"plt.ylabel('Probability')\n",
"plt.ylim((0,1))\n",
"plt.xlabel('Class')\n",
"plt.title('Trafic Sign prediction')\n",
"plt.show()\n",
"\n",
"# ---- Predict class\n",
"#\n",
"p = np.argmax(predictions)\n",
"\n",
"# ---- Show result\n",
"#\n",
"print(\"\\nPrediction on the left, real stuff on the right :\\n\")\n",
"ooo.plot_images([x,x_meta[y]], [p,y], range(2), columns=3, x_size=3, y_size=2)\n",
"\n",
"if p==y:\n",
" print(\"YEEES ! that's right!\")\n",
"else:\n",
" print(\"oups, that's wrong ;-(\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"That's all folks !"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.5"
}
},
"nbformat": 4,
"nbformat_minor": 4
}