05-Full-convolutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n",
    "\n",
    "# <!-- TITLE --> [GTS5] - CNN with GTSRB dataset - Full convolutions \n",
    "<!-- DESC --> Episode 5 : A lot of models, a lot of datasets and a lot of results.\n",
    "<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->\n",
    "\n",
    "## Objectives :\n",
    "  - Try multiple solutions\n",
    "  - Design a generic and batch-usable code\n",
    "  \n",
    "The German Traffic Sign Recognition Benchmark (GTSRB) is a dataset with more than 50,000 photos of road signs from about 40 classes.  \n",
    "The final aim is to recognise them !  \n",
    "Description is available there : http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset\n",
    "\n",
    "\n",
    "## What we're going to do :\n",
    "\n",
    "Our main steps:\n",
    " - Try n models with n datasets\n",
    " - Save a Pandas/h5 report\n",
    " - Write to be run in batch mode\n",
    "\n",
    "## Step 1 - Import and init"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       "\n",
       "div.warn {    \n",
       "    background-color: #fcf2f2;\n",
       "    border-color: #dFb5b4;\n",
       "    border-left: 5px solid #dfb5b4;\n",
       "    padding: 0.5em;\n",
       "    font-weight: bold;\n",
       "    font-size: 1.1em;;\n",
       "    }\n",
       "\n",
       "\n",
       "\n",
       "div.nota {    \n",
       "    background-color: #DAFFDE;\n",
       "    border-left: 5px solid #92CC99;\n",
       "    padding: 0.5em;\n",
       "    }\n",
       "\n",
       "div.todo:before { content:url(data:image/svg+xml;base64,PHN2ZyB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciIHdpZHRoPSI1My44OTEyIiBoZWlnaHQ9IjE0My4zOTAyIiB2aWV3Qm94PSIwIDAgNTMuODkxMiAxNDMuMzkwMiI+PHRpdGxlPjAwLUJvYi10b2RvPC90aXRsZT48cGF0aCBkPSJNMjMuNDU2OCwxMTQuMzAxNmExLjgwNjMsMS44MDYzLDAsMSwxLDEuODE1NywxLjgyNEExLjgyMDksMS44MjA5LDAsMCwxLDIzLjQ1NjgsMTE0LjMwMTZabS0xMC42NjEyLDEuODIyQTEuODI3MiwxLjgyNzIsMCwxLDAsMTAuOTgsMTE0LjMsMS44MiwxLjgyLDAsMCwwLDEyLjc5NTYsMTE2LjEyMzZabS03LjcwNyw0LjU4NzR2LTVzLjQ4NjMtOS4xMjIzLDguMDIxNS0xMS45Njc1YTE5LjIwODIsMTkuMjA4MiwwLDAsMSw2LjA0ODYtMS4yNDU0LDE5LjE3NzgsMTkuMTc3OCwwLDAsMSw2LjA0ODcsMS4yNDc1YzcuNTM1MSwyLjgzNDcsOC4wMTc0LDExLjk2NzQsOC4wMTc0LDExLjk2NzR2NS4wMjM0bC4wMDQyLDcuNjgydjIuNGMuMDE2Ny4xOTkyLjAzMzYuMzkyMS4wMzM2LjU4NzEsMCwuMjEzOC0uMDE2OC40MTA5LS4wMzM2LjYzMzJ2LjA1ODdoLS4wMDg0YTguMzcxOSw4LjM3MTksMCwwLDEtNy4zNzM4LDcuNjU0N3MtLjk5NTMsMy42MzgtNi42OTMzLDMuNjM4LTYuNjkzNC0zLjYzOC02LjY5MzQtMy42MzhhOC4zNyw4LjM3LDAsMCwxLTcuMzcxNi03LjY1NDdINS4wODQzdi0uMDU4N2MtLjAxODktLjIyLS4wMjk0LS40MTk0LS4wMjk0LS42MzMyLDAtLjE5MjkuMDE2Ny0uMzgzNy4wMjk0LS41ODcxdi0yLjRtMTguMDkzNy00LjA0YTEuMTU2NSwxLjE1NjUsMCwxLDAtMi4zMTI2LDAsMS4xNTY0LDEuMTU2NCwwLDEsMCwyLjMxMjYsMFptNC4wODM0LDBhMS4xNTk1LDEuMTU5NSwwLDEsMC0xLjE2MzYsMS4xN0ExLjE3NSwxLjE3NSwwLDAsMCwyNy4yNjE0LDEyNC4zNzc5Wk05LjM3MzksMTE0LjYzNWMwLDMuMTA5MywyLjQxMzIsMy4zMSwyLjQxMzIsMy4zMWExMzMuOTI0MywxMzMuOTI0MywwLDAsMCwxNC43MzQ4LDBzMi40MTExLS4xOTI5LDIuNDExMS0zLjMxYTguMDc3Myw4LjA3NzMsMCwwLDAtMi40MTExLTUuNTUxOWMtNC41LTMuNTAzMy05LjkxMjYtMy41MDMzLTE0Ljc0MTEsMEE4LjA4NTEsOC4wODUxLDAsMCwwLDkuMzczOSwxMTQuNjM1WiIgc3R5bGU9ImZpbGw6IzAxMDEwMSIvPjxjaXJjbGUgY3g9IjMzLjE0MzYiIGN5PSIxMjQuNTM0IiByPSIzLjgzNjMiIHN0eWxlPSJmaWxsOiMwMTAxMDEiLz48cmVjdCB4PSIzNS42NjU5IiB5PSIxMTIuOTYyNSIgd2lkdGg9IjIuMDc3IiBoZWlnaHQ9IjEwLjU0NTgiIHRyYW5zZm9ybT0idHJhbnNsYXRlKDIxLjYgMjQxLjExMjEpIHJvdGF0ZSgtMTU1Ljc0NikiIHN0eWxlPSJmaWxsOiMwMTAxMDEiLz48Y2lyY2xlIGN4PSIzOC44NzA0IiBjeT0iMTEzLjQyNzkiIHI9IjIuNDA4NSIgc3R5bGU9ImZpbGw6IzAxMDEwMSIvPjxjaXJjbGUgY3g9IjUuMjI0OCIgY3k9IjEyNC41MzQiIHI9IjMuODM2MyIgc3R5bGU9ImZpbGw6IzAxMDEwMSIvPjxyZWN0IHg9IjEuNDE2NCIgeT0iMTI0LjYzMDEiIHdpZHRoPSIyLjA3NyIgaGVpZ2h0PSIxMC41NDU4IiB0cmFuc2Zvcm09InRyYW5zbGF0ZSg0LjkwOTcgMjU5LjgwNikgcm90YXRlKC0xODApIiBzdHlsZT0iZmlsbDojMDEwMTAxIi8+PGNpcmNsZSBjeD0iMi40MDkxIiBjeT0iMTM3LjA5OTYiIHI9IjIuNDA4NSIgc3R5bGU9ImZpbGw6IzAxMDEwMSIvPjxwYXRoIGQ9Ik0xOC4wNTExLDEwMC4xMDY2aC0uMDE0NlYxMDIuNjFoMi4zdi0yLjQyNzlhMi40MjI5LDIuNDIyOSwwLDEsMC0yLjI4NTQtLjA3NTVaIiBzdHlsZT0iZmlsbDojMDEwMTAxIi8+PHBhdGggZD0iTTM5LjQyMTQsMjcuMjU4djEuMDVBMTEuOTQ1MiwxMS45NDUyLDAsMCwwLDQ0LjU5NTQsNS43OWEuMjQ0OS4yNDQ5LDAsMCwxLS4wMjM1LS40MjI3TDQ2Ljc1LDMuOTUxNWEuMzg5Mi4zODkyLDAsMCwxLC40MjYyLDAsMTQuODQ0MiwxNC44NDQyLDAsMCwxLTcuNzU0MywyNy4yNTkxdjEuMDY3YS40NS40NSwwLDAsMS0uNzA0Ny4zNzU4bC0zLjg0MTktMi41MWEuNDUuNDUsMCwwLDEsMC0uNzUxNmwzLjg0MTktMi41MWEuNDUuNDUsMCwwLDEsLjY5NDYuMzc1OFpNNDMuMjMsMi41ODkyLDM5LjM4NzguMDc5NGEuNDUuNDUsMCwwLDAtLjcwNDYuMzc1OHYxLjA2N2ExNC44NDQyLDE0Ljg0NDIsMCwwLDAtNy43NTQzLDI3LjI1OTEuMzg5LjM4OSwwLDAsMCwuNDI2MSwwbDIuMTc3Ny0xLjQxOTNhLjI0NS4yNDUsMCwwLDAtLjAyMzUtLjQyMjgsMTEuOTQ1MSwxMS45NDUxLDAsMCwxLDUuMTc0LTIyLjUxNDZ2MS4wNWEuNDUuNDUsMCwwLDAsLjcwNDYuMzc1OGwzLjg1NTMtMi41MWEuNDUuNDUsMCwwLDAsMC0uNzUxNlpNMzkuMDUyMywxNC4yNDU4YTIuMTIwNiwyLjEyMDYsMCwxLDAsMi4xMjA2LDIuMTIwNmgwQTIuMTI0LDIuMTI0LDAsMCwwLDM5LjA1MjMsMTQuMjQ1OFptNi4wNzMyLTQuNzc4MS44MjU0LjgyNTVhMS4wNTY4LDEuMDU2OCwwLDAsMSwuMTE3NSwxLjM0MjFsLS44MDIsMS4xNDQyYTcuMTAxOCw3LjEwMTgsMCwwLDEsLjcxMTQsMS43MTEybDEuMzc1Ny4yNDE2YTEuMDU2OSwxLjA1NjksMCwwLDEsLjg3NTcsMS4wNHYxLjE2NDNhMS4wNTY5LDEuMDU2OSwwLDAsMS0uODc1NywxLjA0bC0xLjM3MjQuMjQxNkE3LjExLDcuMTEsMCwwLDEsNDUuMjcsMTkuOTNsLjgwMTksMS4xNDQyYTEuMDU3LDEuMDU3LDAsMCwxLS4xMTc0LDEuMzQyMmwtLjgyODguODQ4OWExLjA1NywxLjA1NywwLDAsMS0xLjM0MjEuMTE3NGwtMS4xNDQyLS44MDE5YTcuMTMzOCw3LjEzMzgsMCwwLDEtMS43MTEzLjcxMTNsLS4yNDE2LDEuMzcyNGExLjA1NjgsMS4wNTY4LDAsMCwxLTEuMDQuODc1N0gzOC40Njg0YTEuMDU2OCwxLjA1NjgsMCwwLDEtMS4wNC0uODc1N2wtLjI0MTYtMS4zNzI0YTcuMTM1NSw3LjEzNTUsMCwwLDEtMS43MTEzLS43MTEzbC0xLjE0NDEuODAxOWExLjA1NzEsMS4wNTcxLDAsMCwxLTEuMzQyMi0uMTE3NGwtLjgzNTUtLjgyNTVhMS4wNTcsMS4wNTcsMCwwLDEtLjExNzQtMS4zNDIxbC44MDE5LTEuMTQ0MmE3LjEyMSw3LjEyMSwwLDAsMS0uNzExMy0xLjcxMTJsLTEuMzcyNC0uMjQxNmExLjA1NjksMS4wNTY5LDAsMCwxLS44NzU3LTEuMDRWMTUuNzgyNmExLjA1NjksMS4wNTY5LDAsMCwxLC44NzU3LTEuMDRsMS4zNzU3LS4yNDE2YTcuMTEsNy4xMSwwLDAsMSwuNzExNC0xLjcxMTJsLS44MDItMS4xNDQyYTEuMDU3LDEuMDU3LDAsMCwxLC4xMTc1LTEuMzQyMmwuODI1NC0uODI1NEExLjA1NjgsMS4wNTY4LDAsMCwxLDM0LjMyNDUsOS4zNmwxLjE0NDIuODAxOUE3LjEzNTUsNy4xMzU1LDAsMCwxLDM3LjE4LDkuNDUxbC4yNDE2LTEuMzcyNGExLjA1NjgsMS4wNTY4LDAsMCwxLDEuMDQtLjg3NTdoMS4xNjc3YTEuMDU2OSwxLjA1NjksMCwwLDEsMS4wNC44NzU3bC4yNDE2LDEuMzcyNGE3LjEyNSw3LjEyNSwwLDAsMSwxLjcxMTIuNzExM0w0My43NjY2LDkuMzZBMS4wNTY5LDEuMDU2OSwwLDAsMSw0NS4xMjU1LDkuNDY3N1ptLTIuMDMsNi44OTg3QTQuMDQzMyw0LjA0MzMsMCwxLDAsMzkuMDUyMywyMC40MWgwQTQuMDQ2NSw0LjA0NjUsMCwwLDAsNDMuMDk1NSwxNi4zNjY0WiIgc3R5bGU9ImZpbGw6I2UxMjIyOSIvPjxwb2x5Z29uIHBvaW50cz0iMzkuNDEzIDM0Ljc1NyAzOS41MzcgMzQuNzU3IDM5LjY3NSAzNC43NTcgMzkuNjc1IDEwOS41MSAzOS41MzcgMTA5LjUxIDM5LjQxMyAxMDkuNTEgMzkuNDEzIDM0Ljc1NyAzOS40MTMgMzQuNzU3IiBzdHlsZT0iZmlsbDpub25lO3N0cm9rZTojOTk5O3N0cm9rZS1saW5lY2FwOnJvdW5kO3N0cm9rZS1taXRlcmxpbWl0OjEwO3N0cm9rZS13aWR0aDowLjMwODg1NDQ1MDU2MDE2MThweDtmaWxsLXJ1bGU6ZXZlbm9kZCIvPjwvc3ZnPg==);\n",
       "    float:left;\n",
       "    margin-right:20px;\n",
       "    margin-top:-20px;\n",
       "    margin-bottom:20px;\n",
       "}\n",
       "div.todo{\n",
       "    font-weight: bold;\n",
       "    font-size: 1.1em;\n",
       "    margin-top:40px;\n",
       "}\n",
       "div.todo ul{\n",
       "    margin: 0.2em;\n",
       "}\n",
       "div.todo li{\n",
       "    margin-left:60px;\n",
       "    margin-top:0;\n",
       "    margin-bottom:0;\n",
       "}\n",
       "\n",
       "div .comment{\n",
       "    font-size:0.8em;\n",
       "    color:#696969;\n",
       "}\n",
       "\n",
       "\n",
       "\n",
       "</style>\n",
       "\n"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/markdown": [
       "**FIDLE 2020 - Practical Work Module**"
      ],
      "text/plain": [
       "<IPython.core.display.Markdown object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version              : 0.6.1 DEV\n",
      "Notebook id          : GTS5\n",
      "Run time             : Thursday 17 December 2020, 22:07:09\n",
      "TensorFlow version   : 2.1.0\n",
      "Keras version        : 2.2.4-tf\n",
      "Datasets dir         : /gpfswork/rech/mlh/uja62cb/datasets\n",
      "Running mode         : full\n",
      "Update keras cache   : False\n",
      "Save figs            : True\n",
      "Path figs            : ./run/figs\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "import numpy as np\n",
    "import h5py\n",
    "import sys,os,time,json\n",
    "import random\n",
    "from IPython.display import display\n",
    "sys.path.append('..')\n",
    "import fidle.pwk as pwk\n",
    "\n",
    "VERSION='1.6'\n",
    "\n",
    "datasets_dir = pwk.init('GTS5')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Full Convolutions Notebook :\n",
      "  Version            : 1.6\n",
      "  Now is             : Thursday 17 December 2020 - 22h07m09s\n",
      "  OAR id             : ??\n",
      "  SLURM id           : 1874675\n",
      "  Tag id             : 002079\n",
      "  Working directory  : /gpfsdswork/projects/rech/mlh/uja62cb/fidle/GTSRB\n",
      "  Output  directory  : ./run\n",
      "  for tensorboard    : --logdir /gpfsdswork/projects/rech/mlh/uja62cb/fidle/GTSRB/run/logs_002079\n"
     ]
    }
   ],
   "source": [
    "random.seed(time.time())\n",
    "\n",
    "# ---- Where I am ?\n",
    "now    = time.strftime(\"%A %d %B %Y - %Hh%Mm%Ss\")\n",
    "here   = os.getcwd()\n",
    "tag_id = '{:06}'.format(random.randint(0,99999))\n",
    "\n",
    "# ---- Who I am ?\n",
    "oar_id   = os.getenv(\"OAR_JOB_ID\",  \"??\")\n",
    "slurm_id = os.getenv(\"SLURM_JOBID\", \"??\")\n",
    "\n",
    "print('Full Convolutions Notebook :')\n",
    "print('  Version            : {}'.format(VERSION))\n",
    "print('  Now is             : {}'.format(now))\n",
    "print('  OAR id             : {}'.format(oar_id))\n",
    "print('  SLURM id           : {}'.format(slurm_id))\n",
    "print('  Tag id             : {}'.format(tag_id))\n",
    "print('  Working directory  : {}'.format(here))\n",
    "print('  Output  directory  : ./run')\n",
    "print('  for tensorboard    : --logdir {}/run/logs_{}'.format(here,tag_id))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ---- Uncomment for batch tests\n",
    "#\n",
    "# print(\"\\n\\n*** Test mode - Exit before making big treatments... ***\\n\\n\")\n",
    "# sys.exit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Dataset loading"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_dataset(dataset_dir, name):\n",
    "    '''Reads h5 dataset from dataset_dir\n",
    "    Args:\n",
    "        dataset_dir : datasets dir\n",
    "        name        : dataset name, without .h5\n",
    "    Returns:    x_train,y_train,x_test,y_test data'''\n",
    "    # ---- Read dataset\n",
    "    filename = f'{dataset_dir}/GTSRB/enhanced/{name}.h5'\n",
    "    size     = os.path.getsize(filename)/(1024*1024)\n",
    "\n",
    "    with  h5py.File(filename,'r') as f:\n",
    "        x_train = f['x_train'][:]\n",
    "        y_train = f['y_train'][:]\n",
    "        x_test  = f['x_test'][:]\n",
    "        y_test  = f['y_test'][:]\n",
    "\n",
    "    # ---- done\n",
    "    return x_train,y_train,x_test,y_test,size"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Models collection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# A basic model\n",
    "#\n",
    "def get_model_v1(lx,ly,lz):\n",
    "    \n",
    "    model = keras.models.Sequential()\n",
    "    \n",
    "    model.add( keras.layers.Conv2D(96, (3,3), activation='relu', input_shape=(lx,ly,lz)))\n",
    "    model.add( keras.layers.MaxPooling2D((2, 2)))\n",
    "    model.add( keras.layers.Dropout(0.2))\n",
    "\n",
    "    model.add( keras.layers.Conv2D(192, (3, 3), activation='relu'))\n",
    "    model.add( keras.layers.MaxPooling2D((2, 2)))\n",
    "    model.add( keras.layers.Dropout(0.2))\n",
    "\n",
    "    model.add( keras.layers.Flatten()) \n",
    "    model.add( keras.layers.Dense(1500, activation='relu'))\n",
    "    model.add( keras.layers.Dropout(0.5))\n",
    "\n",
    "    model.add( keras.layers.Dense(43, activation='softmax'))\n",
    "    return model\n",
    "    \n",
    "# A more sophisticated model\n",
    "#\n",
    "def get_model_v2(lx,ly,lz):\n",
    "    model = keras.models.Sequential()\n",
    "\n",
    "    model.add( keras.layers.Conv2D(64, (3, 3), padding='same', input_shape=(lx,ly,lz), activation='relu'))\n",
    "    model.add( keras.layers.Conv2D(64, (3, 3), activation='relu'))\n",
    "    model.add( keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add( keras.layers.Dropout(0.2))\n",
    "\n",
    "    model.add( keras.layers.Conv2D(128, (3, 3), padding='same', activation='relu'))\n",
    "    model.add( keras.layers.Conv2D(128, (3, 3), activation='relu'))\n",
    "    model.add( keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add( keras.layers.Dropout(0.2))\n",
    "\n",
    "    model.add( keras.layers.Conv2D(256, (3, 3), padding='same',activation='relu'))\n",
    "    model.add( keras.layers.Conv2D(256, (3, 3), activation='relu'))\n",
    "    model.add( keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add( keras.layers.Dropout(0.2))\n",
    "\n",
    "    model.add( keras.layers.Flatten())\n",
    "    model.add( keras.layers.Dense(512, activation='relu'))\n",
    "    model.add( keras.layers.Dropout(0.5))\n",
    "    model.add( keras.layers.Dense(43, activation='softmax'))\n",
    "    return model\n",
    "\n",
    "def get_model_v3(lx,ly,lz):\n",
    "    model = keras.models.Sequential()\n",
    "    model.add(tf.keras.layers.Conv2D(32, (5, 5), padding='same',  activation='relu', input_shape=(lx,ly,lz)))\n",
    "    model.add(tf.keras.layers.BatchNormalization(axis=-1))      \n",
    "    model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(tf.keras.layers.Dropout(0.2))\n",
    "\n",
    "    model.add(tf.keras.layers.Conv2D(64, (5, 5), padding='same',  activation='relu'))\n",
    "    model.add(tf.keras.layers.BatchNormalization(axis=-1))\n",
    "    model.add(tf.keras.layers.Conv2D(128, (5, 5), padding='same', activation='relu'))\n",
    "    model.add(tf.keras.layers.BatchNormalization(axis=-1))\n",
    "    model.add(tf.keras.layers.MaxPooling2D(pool_size=(2, 2)))\n",
    "    model.add(tf.keras.layers.Dropout(0.2))\n",
    "\n",
    "    model.add(tf.keras.layers.Flatten())\n",
    "    model.add(tf.keras.layers.Dense(512, activation='relu'))\n",
    "    model.add(tf.keras.layers.BatchNormalization())\n",
    "    model.add(tf.keras.layers.Dropout(0.4))\n",
    "\n",
    "    model.add(tf.keras.layers.Dense(43, activation='softmax'))\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 - Multiple datasets, multiple models ;-)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def multi_run(datasets_dir, datasets, models, datagen=None,\n",
    "              train_size=1, test_size=1, batch_size=64, epochs=16, \n",
    "              verbose=0, extension_dir='last'):\n",
    "    \"\"\"\n",
    "    Launches a dataset-model combination\n",
    "    args:\n",
    "        datasets_dir   : Directory of the datasets\n",
    "        datasets       : List of dataset (whitout .h5)\n",
    "        models         : List of model like { \"model name\":get_model(), ...}\n",
    "        datagen        : Data generator or None (None)\n",
    "        train_size     : % of train dataset to use. 1 mean all. (1)\n",
    "        test_size      : % of test dataset to use.  1 mean all. (1)\n",
    "        batch_size     : Batch size (64)\n",
    "        epochs         : Number of epochs (16)\n",
    "        verbose        : Verbose level (0)\n",
    "        extension_dir  : postfix for logs and models dir (_last)\n",
    "    return:\n",
    "        report        : Report as a dict for Pandas.\n",
    "    \"\"\"\n",
    "    # ---- Logs and models dir\n",
    "    #\n",
    "    os.makedirs(f'./run/logs_{extension_dir}',   mode=0o750, exist_ok=True)\n",
    "    os.makedirs(f'./run/models_{extension_dir}', mode=0o750, exist_ok=True)\n",
    "    \n",
    "    # ---- Columns of output\n",
    "    #\n",
    "    output={}\n",
    "    output['Dataset'] = []\n",
    "    output['Size']    = []\n",
    "    for m in models:\n",
    "        output[m+'_Accuracy'] = []\n",
    "        output[m+'_Duration'] = []\n",
    "\n",
    "    # ---- Let's go\n",
    "    #\n",
    "    for d_name in datasets:\n",
    "        print(\"\\nDataset : \",d_name)\n",
    "\n",
    "        # ---- Read dataset\n",
    "        x_train,y_train,x_test,y_test, d_size = read_dataset(datasets_dir, d_name)\n",
    "        output['Dataset'].append(d_name)\n",
    "        output['Size'].append(d_size)\n",
    "        \n",
    "        # ---- Get the shape\n",
    "        (n,lx,ly,lz) = x_train.shape\n",
    "        n_train = int( x_train.shape[0] * train_size )\n",
    "        n_test  = int( x_test.shape[0]  * test_size  )\n",
    "\n",
    "        # ---- For each model\n",
    "        for m_name,m_function in models.items():\n",
    "            print(\"    Run model {}  : \".format(m_name), end='')\n",
    "            # ---- get model\n",
    "            try:\n",
    "                model=m_function(lx,ly,lz)\n",
    "                # ---- Compile it\n",
    "                model.compile(optimizer='adam', loss='sparse_categorical_crossentropy', metrics=['accuracy'])\n",
    "                # ---- Callbacks tensorboard\n",
    "                log_dir = f\"./run/logs_{extension_dir}/tb_{d_name}_{m_name}\"\n",
    "                tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir, histogram_freq=1)\n",
    "                # ---- Callbacks bestmodel\n",
    "                save_dir = f\"./run/models_{extension_dir}/model_{d_name}_{m_name}.h5\"\n",
    "                bestmodel_callback = tf.keras.callbacks.ModelCheckpoint(filepath=save_dir, verbose=0, monitor='accuracy', save_best_only=True)\n",
    "                # ---- Train\n",
    "                start_time = time.time()\n",
    "                if datagen==None:\n",
    "                    # ---- No data augmentation (datagen=None) --------------------------------------\n",
    "                    history = model.fit(x_train[:n_train], y_train[:n_train],\n",
    "                                        batch_size      = batch_size,\n",
    "                                        epochs          = epochs,\n",
    "                                        verbose         = verbose,\n",
    "                                        validation_data = (x_test[:n_test], y_test[:n_test]),\n",
    "                                        callbacks       = [tensorboard_callback, bestmodel_callback])\n",
    "                else:\n",
    "                    # ---- Data augmentation (datagen given) ----------------------------------------\n",
    "                    datagen.fit(x_train)\n",
    "                    history = model.fit(datagen.flow(x_train, y_train, batch_size=batch_size),\n",
    "                                        steps_per_epoch = int(n_train/batch_size),\n",
    "                                        epochs          = epochs,\n",
    "                                        verbose         = verbose,\n",
    "                                        validation_data = (x_test[:n_test], y_test[:n_test]),\n",
    "                                        callbacks       = [tensorboard_callback, bestmodel_callback])\n",
    "                    \n",
    "                # ---- Result\n",
    "                end_time = time.time()\n",
    "                duration = end_time-start_time\n",
    "                accuracy = max(history.history[\"val_accuracy\"])*100\n",
    "                #\n",
    "                output[m_name+'_Accuracy'].append(accuracy)\n",
    "                output[m_name+'_Duration'].append(duration)\n",
    "                print(f\"Accuracy={accuracy:.2f} and Duration={duration:.2f}\")\n",
    "            except:\n",
    "                output[m_name+'_Accuracy'].append('0')\n",
    "                output[m_name+'_Duration'].append('999')\n",
    "                print('-')\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6 - Run !"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "---- Run --------------------------------------------------\n",
      "\n",
      "Dataset :  set-24x24-L\n",
      "    Run model v1  : WARNING:tensorflow:Method (on_train_batch_end) is slow compared to the batch update (0.218721). Check your callbacks.\n",
      "Accuracy=88.99 and Duration=8.01\n",
      "    Run model v2  : Accuracy=87.77 and Duration=4.63\n",
      "    Run model v3  : Accuracy=88.80 and Duration=5.25\n",
      "\n",
      "Dataset :  set-24x24-RGB\n",
      "    Run model v1  : Accuracy=89.98 and Duration=5.12\n",
      "    Run model v2  : Accuracy=89.35 and Duration=4.56\n",
      "    Run model v3  : Accuracy=86.70 and Duration=5.22\n",
      "\n",
      "Dataset :  set-48x48-RGB\n",
      "    Run model v1  : Accuracy=88.64 and Duration=18.32\n",
      "    Run model v2  : Accuracy=89.71 and Duration=10.17\n",
      "    Run model v3  : Accuracy=92.16 and Duration=11.10\n",
      "\n",
      "Report saved as  ./run/report_002079.json\n",
      "Duration : 77.23 s\n",
      "-----------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "start_time = time.time()\n",
    "\n",
    "print('\\n---- Run','-'*50)\n",
    "\n",
    "# --------- Datasets, models, and more.. -----------------------------------\n",
    "#\n",
    "# ---- For tests\n",
    "datasets   = ['set-24x24-L', 'set-24x24-RGB', 'set-48x48-RGB']\n",
    "models     = {'v1':get_model_v1, 'v2':get_model_v2, 'v3':get_model_v3}\n",
    "batch_size = 64\n",
    "epochs     = 5\n",
    "train_size = 0.2\n",
    "test_size  = 0.2\n",
    "with_datagen = False\n",
    "verbose      = 0\n",
    "#\n",
    "# ---- All possibilities\n",
    "# datasets     = ['set-24x24-L', 'set-24x24-RGB', 'set-48x48-L', 'set-48x48-RGB', 'set-24x24-L-LHE', 'set-24x24-RGB-HE', 'set-48x48-L-LHE', 'set-48x48-RGB-HE']\n",
    "# models       = {'v1':get_model_v1, 'v2':get_model_v2, 'v3':get_model_v3}\n",
    "# batch_size   = 64\n",
    "# epochs       = 16\n",
    "# train_size   = 1\n",
    "# test_size    = 1\n",
    "# with_datagen = False\n",
    "# verbose      = 0\n",
    "#\n",
    "# ---- Data augmentation\n",
    "# datasets     = ['set-48x48-RGB']\n",
    "# models       = {'v2':get_model_v2}\n",
    "# batch_size   = 64\n",
    "# epochs       = 20\n",
    "# train_size   = 1\n",
    "# test_size    = 1\n",
    "# with_datagen = True\n",
    "# verbose      = 0\n",
    "#\n",
    "# ---------------------------------------------------------------------------\n",
    "\n",
    "# ---- Data augmentation\n",
    "#\n",
    "if with_datagen :\n",
    "    datagen = keras.preprocessing.image.ImageDataGenerator(featurewise_center=False,\n",
    "                                                           featurewise_std_normalization=False,\n",
    "                                                           width_shift_range=0.1,\n",
    "                                                           height_shift_range=0.1,\n",
    "                                                           zoom_range=0.2,\n",
    "                                                           shear_range=0.1,\n",
    "                                                           rotation_range=10.)\n",
    "else:\n",
    "    datagen=None\n",
    "    \n",
    "# ---- Run\n",
    "#\n",
    "output = multi_run(datasets_dir,\n",
    "                   datasets, \n",
    "                   models,\n",
    "                   datagen       = datagen,\n",
    "                   train_size    = train_size,\n",
    "                   test_size     = test_size,\n",
    "                   batch_size    = batch_size,\n",
    "                   epochs        = epochs,\n",
    "                   verbose       = verbose,\n",
    "                   extension_dir = tag_id)\n",
    "\n",
    "# ---- Save report\n",
    "#\n",
    "report={}\n",
    "report['output']=output\n",
    "report['description'] = f'train_size={train_size} test_size={test_size} batch_size={batch_size} epochs={epochs} data_aug={with_datagen}'\n",
    "\n",
    "report_name=f'./run/report_{tag_id}.json'\n",
    "\n",
    "with open(report_name, 'w') as file:\n",
    "    json.dump(report, file)\n",
    "\n",
    "print('\\nReport saved as ',report_name)\n",
    "end_time = time.time()\n",
    "duration = end_time-start_time\n",
    "print(f'Duration : {duration:.2f} s')\n",
    "print('-'*59)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7 - That's all folks.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "End time is : Thursday 17 December 2020, 22:08:26\n",
      "Duration is : 00:01:17 312ms\n",
      "This notebook ends here\n"
     ]
    }
   ],
   "source": [
    "pwk.end()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<img width=\"80px\" src=\"../fidle/img/00-Fidle-logo-01.svg\"></img>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}