{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<img width=\"800px\" src=\"../fidle/img/00-Fidle-header-01.svg\"></img>\n",
"\n",
"# <!-- TITLE --> [GTSRB6] - Full convolutions as a batch\n",
"<!-- DESC --> Episode 6 : To compute bigger, use your notebook in batch mode\n",
"<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->\n",
"\n",
"## Objectives :\n",
" - Run a notebook code as a **job**\n",
" - Follow up with Tensorboard\n",
" \n",
"The German Traffic Sign Recognition Benchmark (GTSRB) is a dataset with more than 50,000 photos of road signs from about 40 classes. \n",
"The final aim is to recognise them ! \n",
"Description is available there : http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset\n",
"\n",
"\n",
"## What we're going to do :\n",
"Our main steps:\n",
" - Run Full-convolution.ipynb as a batch :\n",
" - Notebook mode\n",
" - Script mode \n",
" - Tensorboard follow up"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 1 - How to run a notebook as a batch ?\n",
"\n",
"Two simple solutions are possible :-)\n",
"\n",
" - **Option 1 - As a notebook ! (a good choice)**\n",
"\n",
" Very simple. \n",
" The result is the executed notebook, so we can retrieve all the cell'soutputs of the notebook : \n",
" ```jupyter nbconvert (...) --to notebook --execute <notebook>``` \n",
"\n",
" Example : \n",
" ```jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --execute my_notebook.ipynb'``` \n",
" The result will be a notebook: 'my_notebook.nbconvert.ipynb'.\n",
" \n",
" See: [nbconvert documentation](https://nbconvert.readthedocs.io/en/latest/usage.html#convert-notebook)\n",
"\n",
" - **Option 2 - As a script**\n",
"\n",
" Very simple too, but with some constraints on the notebook. \n",
" We will convert the notebook to a Python script (IPython, to be precise) : \n",
" ```jupyter nbconvert --to script <notebook>``` \n",
" \n",
" Then we can execute this script : \n",
" ```ipython <script>```\n",
" \n",
" See: [nbconvert documentation](https://nbconvert.readthedocs.io/en/latest/usage.html#executable-script)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 2 - Run as a script\n",
"\n",
"Maybe not always the best solution, but this solution is very rustic ! \n",
"\n",
"### 2.1 - Convert to IPython script :"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[NbConvertApp] Converting notebook 05-Full-convolutions.ipynb to script\n",
"[NbConvertApp] Writing 12984 bytes to 05-full_convolutions.py\n",
"-rw-r--r-- 1 uja62cb mlh 12984 Jan 28 11:35 05-full_convolutions.py\n"
]
}
],
"source": [
"! jupyter nbconvert --to script --output='05-full_convolutions' '05-Full-convolutions.ipynb'\n",
"! ls -l *.py"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### 2.2 - Batch submission\n",
"\n",
" See the two examples of bash launch script :\n",
" - [batch_slurm.sh](batch_slurm.sh) using Slurm (like at IDRIS)\n",
" - [batch_oar.sh](batch_oar.sh) using OAR (like at GRICAD)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Example at IDRIS\n",
"\n",
"On the frontal :\n",
"```bash\n",
"# hostname\n",
"jean-zay2\n",
"\n",
"# sbatch $WORK/fidle/GTSRB/batch_slurm.sh \n",
"Submitted batch job 249794\n",
"\n",
"#squeue -u $USER\n",
" JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)\n",
" 249794 gpu_p1 GTSRB Fu uja62cb PD 0:00 1 (Resources)\n",
"\n",
"# ls -l _batch/\n",
"total 32769\n",
"-rw-r--r-- 1 uja62cb gensim01 13349 Sep 10 11:32 GTSRB_249794.err\n",
"-rw-r--r-- 1 uja62cb gensim01 489 Sep 10 11:31 GTSRB_249794.out\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Example at GRICAD\n",
"\n",
"Have to be done on the frontal :\n",
"```bash\n",
"# hostname\n",
"f-dahu\n",
"\n",
"# pwd\n",
"/home/paroutyj\n",
"\n",
"# oarsub -S ~/fidle/GTSRB/batch_oar.sh\n",
"[GPUNODE] Adding gpu node restriction\n",
"[ADMISSION RULE] Modify resource description with type constraints\n",
"\n",
"#oarstat -u\n",
"Job id S User Duration System message\n",
"--------- - -------- ---------- ------------------------------------------------\n",
"5878410 R paroutyj 0:19:56 R=8,W=1:0:0,J=I,P=fidle,T=gpu (Karma=0.005,quota_ok)\n",
"5896266 W paroutyj 0:00:00 R=8,W=1:0:0,J=B,N=Full convolutions,P=fidle,T=gpu\n",
"\n",
"# ls -l\n",
"total 8\n",
"-rw-r--r-- 1 paroutyj l-simap 0 Feb 28 15:58 batch_oar_5896266.err\n",
"-rw-r--r-- 1 paroutyj l-simap 5703 Feb 28 15:58 batch_oar_5896266.out\n",
"```\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"----\n",
"**What you can do :**\n",
"\n",
"(If you have a calculation environment with a scheduler...)\n",
"Your mission if you accept it: Run our full_convolution code in batch mode.<br>\n",
" For that :\n",
" - Validate the full_convolution notebook on short tests</li>\n",
" - Submit it in batch mode for validation</li>\n",
" - Modify the notebook for a full run and submit it :-)</li>\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"---\n",
"<img width=\"80px\" src=\"../fidle/img/00-Fidle-logo-01.svg\"></img>"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.9"
}
},
"nbformat": 4,
"nbformat_minor": 4
}