{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img width=\"800px\" src=\"../fidle/img/header.svg\"></img>\n",
    "\n",
    "# <!-- TITLE --> [PMNIST1] - Simple classification with DNN\n",
    "<!-- DESC -->Example of classification with a fully connected neural network, using Pytorch\n",
    "<!-- AUTHOR : Laurent Risser (CNRS/IMT) -->\n",
    "\n",
    "## Objectives :\n",
    " - Recognizing handwritten numbers\n",
    " - Understanding the principle of a classifier DNN network \n",
    " - Implementation with PyTorch \n",
    "\n",
    "\n",
    "The [MNIST dataset](http://yann.lecun.com/exdb/mnist/) (Modified National Institute of Standards and Technology) is a must for Deep Learning.  \n",
    "It consists of 60,000 small images of handwritten numbers for learning and 10,000 for testing.\n",
    "\n",
    "\n",
    "## What we're going to do :\n",
    "\n",
    " - Retrieve data\n",
    " - Preparing the data\n",
    " - Create a model\n",
    " - Train the model\n",
    " - Evaluate the result\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1 - Init python stuff"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "from torch.autograd import Variable\n",
    "import torchvision  #to get the MNIST dataset\n",
    "\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import sys,os\n",
    "\n",
    "import fidle\n",
    "from modules.fidle_pwk_additional import convergence_history_CrossEntropyLoss\n",
    "\n",
    "# Init Fidle environment\n",
    "run_id, run_dir, datasets_dir = fidle.init('PMNIST1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2 - Retrieve data\n",
    "MNIST is one of the most famous historic dataset.  \n",
    "Include in [torchvision datasets](https://pytorch.org/vision/stable/datasets.html)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#get and format the training set\n",
    "mnist_trainset = torchvision.datasets.MNIST(root='./data', train=True, download=True, transform=None)\n",
    "x_train=mnist_trainset.data.type(torch.DoubleTensor)\n",
    "y_train=mnist_trainset.targets\n",
    "\n",
    "\n",
    "#get and format the test set\n",
    "mnist_testset = torchvision.datasets.MNIST(root='./data', train=False, download=True, transform=None)\n",
    "x_test=mnist_testset.data.type(torch.DoubleTensor)\n",
    "y_test=mnist_testset.targets\n",
    "\n",
    "#check data shape and format\n",
    "print(\"Size of the train and test observations\")\n",
    "print(\" -> x_train : \",x_train.shape)\n",
    "print(\" -> y_train : \",y_train.shape)\n",
    "print(\" -> x_test  : \",x_test.shape)\n",
    "print(\" -> y_test  : \",y_test.shape)\n",
    "\n",
    "print(\"\\nRemark that we work with torch tensors and not numpy arrays:\")\n",
    "print(\" -> x_train.dtype = \",x_train.dtype)\n",
    "print(\" -> y_train.dtype = \",y_train.dtype)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3 - Preparing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Before normalization : Min={}, max={}'.format(x_train.min(),x_train.max()))\n",
    "\n",
    "xmax=x_train.max()\n",
    "x_train = x_train / xmax\n",
    "x_test  = x_test  / xmax\n",
    "\n",
    "print('After normalization  : Min={}, max={}'.format(x_train.min(),x_train.max()))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Have a look"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np_x_train=x_train.numpy().astype(np.float64)\n",
    "np_y_train=y_train.numpy().astype(np.uint8)\n",
    "\n",
    "fidle.scrawler.images(np_x_train,np_y_train , [27],  x_size=5,y_size=5, colorbar=True)\n",
    "fidle.scrawler.images(np_x_train,np_y_train, range(5,41), columns=12)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4 - Create model\n",
    "About informations about : \n",
    " - [Optimizer](https://pytorch.org/docs/stable/optim.html)\n",
    " - [Basic neural-network blocks](https://pytorch.org/docs/stable/nn.html)\n",
    " - [Loss](https://pytorch.org/docs/stable/nn.html#loss-functions)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class MyModel(nn.Module):\n",
    "    \"\"\"\n",
    "    Basic fully connected neural-network\n",
    "    \"\"\"\n",
    "    def __init__(self):\n",
    "        hidden1     = 100\n",
    "        hidden2     = 100\n",
    "        super(MyModel, self).__init__()\n",
    "        self.hidden1 = nn.Linear(784, hidden1)\n",
    "        self.hidden2 = nn.Linear(hidden1, hidden2)\n",
    "        self.hidden3 = nn.Linear(hidden2, 10)\n",
    "\n",
    "    def forward(self, x):\n",
    "        x = x.view(-1,784)   #flatten the images before using fully-connected layers\n",
    "        x = self.hidden1(x)\n",
    "        x = F.relu(x)\n",
    "        x = self.hidden2(x)\n",
    "        x = F.relu(x)\n",
    "        x = self.hidden3(x)\n",
    "        x = F.softmax(x, dim=0)\n",
    "        return x\n",
    "\n",
    "    \n",
    "    \n",
    "model = MyModel()\n",
    "    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5 - Train the model\n",
    "\n",
    "### 5.1 - Stochastic gradient descent strategy to fit the model\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit(model,X_train,Y_train,X_test,Y_test, EPOCHS = 5, BATCH_SIZE = 32):\n",
    "    \n",
    "    loss = nn.CrossEntropyLoss()\n",
    "    optimizer = torch.optim.Adam(model.parameters(),lr=1e-3) #lr is the learning rate\n",
    "    model.train()\n",
    "    \n",
    "    history=convergence_history_CrossEntropyLoss()\n",
    "    \n",
    "    history.update(model,X_train,Y_train,X_test,Y_test)\n",
    "    \n",
    "    n=X_train.shape[0] #number of observations in the training data\n",
    "    \n",
    "    #stochastic gradient descent\n",
    "    for epoch in range(EPOCHS):\n",
    "        \n",
    "        batch_start=0\n",
    "        epoch_shuffler=np.arange(n) \n",
    "        np.random.shuffle(epoch_shuffler) #remark that 'utilsData.DataLoader' could be used instead\n",
    "        \n",
    "        while batch_start+BATCH_SIZE < n:\n",
    "            #get mini-batch observation\n",
    "            mini_batch_observations = epoch_shuffler[batch_start:batch_start+BATCH_SIZE]\n",
    "            var_X_batch = Variable(X_train[mini_batch_observations,:,:]).float() #the input image is flattened\n",
    "            var_Y_batch = Variable(Y_train[mini_batch_observations])\n",
    "            \n",
    "            #gradient descent step\n",
    "            optimizer.zero_grad()               #set the parameters gradients to 0\n",
    "            Y_pred_batch = model(var_X_batch)   #predict y with the current NN parameters\n",
    "            \n",
    "            curr_loss = loss(Y_pred_batch, var_Y_batch)  #compute the current loss\n",
    "            curr_loss.backward()                         #compute the loss gradient w.r.t. all NN parameters\n",
    "            optimizer.step()                             #update the NN parameters\n",
    "            \n",
    "            #prepare the next mini-batch of the epoch\n",
    "            batch_start+=BATCH_SIZE\n",
    "            \n",
    "        history.update(model,X_train,Y_train,X_test,Y_test)\n",
    "    \n",
    "    return history\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5.2 - Fit the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "batch_size  = 512\n",
    "epochs      = 128\n",
    "\n",
    "\n",
    "history=fit(model,x_train,y_train,x_test,y_test,EPOCHS=epochs,BATCH_SIZE = batch_size)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6 - Evaluate\n",
    "### 6.1 - Final loss and accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "var_x_test = Variable(x_test[:,:,:]).float()\n",
    "var_y_test = Variable(y_test[:])\n",
    "y_pred = model(var_x_test)\n",
    "\n",
    "loss = nn.CrossEntropyLoss()\n",
    "curr_loss = loss(y_pred, var_y_test)\n",
    "\n",
    "val_loss = curr_loss.item()\n",
    "val_accuracy  = float( (torch.argmax(y_pred, dim= 1) == var_y_test).float().mean() )\n",
    "\n",
    "\n",
    "print('Test loss     :', val_loss)\n",
    "print('Test accuracy :', val_accuracy)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.2 - Plot history"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fidle.scrawler.history(history, figsize=(6,4))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.3 - Plot results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model(var_x_test)\n",
    "np_y_pred_label = torch.argmax(y_pred, dim= 1).numpy().astype(np.uint8)\n",
    "\n",
    "np_x_test=x_test.numpy().astype(np.float64)\n",
    "np_y_test=y_test.numpy().astype(np.uint8)\n",
    "\n",
    "fidle.scrawler.images(np_x_test, np_y_test, range(0,60), columns=12, x_size=1, y_size=1, y_pred=np_y_pred_label)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6.4 - Plot some errors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "errors=[ i for i in range(len(np_y_test)) if np_y_pred_label[i]!=np_y_test[i] ]\n",
    "errors=errors[:min(24,len(errors))]\n",
    "fidle.scrawler.images(np_x_test, np_y_test, errors[:15], columns=6, x_size=2, y_size=2, y_pred=np_y_pred_label)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fidle.scrawler.confusion_matrix(np_y_test,np_y_pred_label, range(10))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"todo\">\n",
    "    A few things you can do for fun:\n",
    "    <ul>\n",
    "        <li>Changing the network architecture (layers, number of neurons, etc.)</li>\n",
    "        <li>Display a summary of the network</li>\n",
    "        <li>Retrieve and display the softmax output of the network, to evaluate its \"doubts\".</li>\n",
    "    </ul>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "<img width=\"80px\" src=\"../fidle/img/logo-paysage.svg\"></img>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.9.2 ('fidle-env')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.2"
  },
  "vscode": {
   "interpreter": {
    "hash": "b3929042cc22c1274d74e3e946c52b845b57cb6d84f2d591ffe0519b38e4896d"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}