Skip to content
Snippets Groups Projects
Commit a8e84281 authored by Jean-Luc Parouty's avatar Jean-Luc Parouty
Browse files

Update presentation

parent b0e3ba38
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
GTSRB/06-Notebook-as-a-batch.ipynb
+ 16
− 147
View file @ a8e84281
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img> <img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
# <!-- TITLE --> [GTSRB6] - Full convolutions as a batch # <!-- TITLE --> [GTSRB6] - Full convolutions as a batch
<!-- DESC --> Episode 6 : To compute bigger, use your notebook in batch mode <!-- DESC --> Episode 6 : To compute bigger, use your notebook in batch mode
<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) --> <!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
## Objectives : ## Objectives :
- Run a notebook code as a **job** - Run a notebook code as a **job**
- Follow up with Tensorboard - Follow up with Tensorboard
The German Traffic Sign Recognition Benchmark (GTSRB) is a dataset with more than 50,000 photos of road signs from about 40 classes. The German Traffic Sign Recognition Benchmark (GTSRB) is a dataset with more than 50,000 photos of road signs from about 40 classes.
The final aim is to recognise them ! The final aim is to recognise them !
Description is available there : http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset Description is available there : http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset
## What we're going to do : ## What we're going to do :
Our main steps: Our main steps:
- Run Full-convolution.ipynb as a batch : - Run Full-convolution.ipynb as a batch :
- Notebook mode - Notebook mode
- Script mode - Script mode
- Tensorboard follow up - Tensorboard follow up
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
### Step 1 - Import and init ## Step 1 - How to run a notebook as a batch ?
Not really useful here ;-)
%% Cell type:code id: tags:
``` python
import sys
sys.path.append('..')
import fidle.pwk as pwk
datasets_dir = pwk.init('GTSRB6')
```
%% Output
<br>**FIDLE 2020 - Practical Work Module**
Version : 1.2b1 DEV
Notebook id : GTSRB6
Run time : Thursday 7 January 2021, 15:41:17
TensorFlow version : 2.2.0
Keras version : 2.3.0-tf
Datasets dir : /home/pjluc/datasets/fidle
Run dir : ./run
Update keras cache : False
Save figs : True
Path figs : ./run/figs
%% Cell type:markdown id: tags:
## Step 2 - How to run a notebook as a batch ?
Two simple solutions are possible :-) Two simple solutions are possible :-)
- **Option 1 - As a notebook ! (a good choice)** - **Option 1 - As a notebook ! (a good choice)**
Very simple. Very simple.
The result is the executed notebook, so we can retrieve all the cell'soutputs of the notebook : The result is the executed notebook, so we can retrieve all the cell'soutputs of the notebook :
```jupyter nbconvert (...) --to notebook --execute <notebook>``` ```jupyter nbconvert (...) --to notebook --execute <notebook>```
Example : Example :
```jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --execute my_notebook.ipynb'``` ```jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --execute my_notebook.ipynb'```
The result will be a notebook: 'my_notebook.nbconvert.ipynb'. The result will be a notebook: 'my_notebook.nbconvert.ipynb'.
See: [nbconvert documentation](https://nbconvert.readthedocs.io/en/latest/usage.html#convert-notebook) See: [nbconvert documentation](https://nbconvert.readthedocs.io/en/latest/usage.html#convert-notebook)
- **Option 2 - As a script** - **Option 2 - As a script**
Very simple too, but with some constraints on the notebook. Very simple too, but with some constraints on the notebook.
We will convert the notebook to a Python script (IPython, to be precise) : We will convert the notebook to a Python script (IPython, to be precise) :
```jupyter nbconvert --to script <notebook>``` ```jupyter nbconvert --to script <notebook>```
Then we can execute this script : Then we can execute this script :
```ipython <script>``` ```ipython <script>```
See: [nbconvert documentation](https://nbconvert.readthedocs.io/en/latest/usage.html#executable-script) See: [nbconvert documentation](https://nbconvert.readthedocs.io/en/latest/usage.html#executable-script)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
## Step 3 - Run as a script ## Step 2 - Run as a script
Maybe not always the best solution, but this solution is very rustic ! Maybe not always the best solution, but this solution is very rustic !
### 3.1 - Convert to IPython script : ### 2.1 - Convert to IPython script :
%% Cell type:code id: tags: %% Cell type:code id: tags:
``` python ``` python
! jupyter nbconvert --to script --output='05-full_convolutions' '05-Full-convolutions.ipynb' ! jupyter nbconvert --to script --output='05-full_convolutions' '05-Full-convolutions.ipynb'
! ls -l *.py ! ls -l *.py
``` ```
%% Output %% Output
[NbConvertApp] Converting notebook 05-Full-convolutions.ipynb to script [NbConvertApp] Converting notebook 05-Full-convolutions.ipynb to script
[NbConvertApp] Writing 12854 bytes to 05-full_convolutions.py [NbConvertApp] Writing 12984 bytes to 05-full_convolutions.py
-rw-r--r-- 1 pjluc pjluc 12854 Jan 7 15:41 05-full_convolutions.py -rw-r--r-- 1 uja62cb mlh 12984 Jan 28 11:35 05-full_convolutions.py
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
### 2.2 - Batch submission ### 2.2 - Batch submission
See the two examples of bash launch script : See the two examples of bash launch script :
- `batch_slurm.sh` using Slurm (like at IDRIS) - [batch_slurm.sh](batch_slurm.sh) using Slurm (like at IDRIS)
- `batch_oar.sh` using OAR (like at GRICAD) - [batch_oar.sh](batch_oar.sh) using OAR (like at GRICAD)
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
#### Example at IDRIS #### Example at IDRIS
On the frontal : On the frontal :
```bash ```bash
# hostname # hostname
jean-zay2 jean-zay2
# sbatch $WORK/fidle/GTSRB/batch_slurm.sh # sbatch $WORK/fidle/GTSRB/batch_slurm.sh
Submitted batch job 249794 Submitted batch job 249794
#squeue -u $USER #squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
249794 gpu_p1 GTSRB Fu uja62cb PD 0:00 1 (Resources) 249794 gpu_p1 GTSRB Fu uja62cb PD 0:00 1 (Resources)
# ls -l _batch/ # ls -l _batch/
total 32769 total 32769
-rw-r--r-- 1 uja62cb gensim01 13349 Sep 10 11:32 GTSRB_249794.err -rw-r--r-- 1 uja62cb gensim01 13349 Sep 10 11:32 GTSRB_249794.err
-rw-r--r-- 1 uja62cb gensim01 489 Sep 10 11:31 GTSRB_249794.out -rw-r--r-- 1 uja62cb gensim01 489 Sep 10 11:31 GTSRB_249794.out
``` ```
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
#### Example at GRICAD #### Example at GRICAD
Have to be done on the frontal : Have to be done on the frontal :
```bash ```bash
# hostname # hostname
f-dahu f-dahu
# pwd # pwd
/home/paroutyj /home/paroutyj
# oarsub -S ~/fidle/GTSRB/batch_oar.sh # oarsub -S ~/fidle/GTSRB/batch_oar.sh
[GPUNODE] Adding gpu node restriction [GPUNODE] Adding gpu node restriction
[ADMISSION RULE] Modify resource description with type constraints [ADMISSION RULE] Modify resource description with type constraints
#oarstat -u #oarstat -u
Job id S User Duration System message Job id S User Duration System message
--------- - -------- ---------- ------------------------------------------------ --------- - -------- ---------- ------------------------------------------------
5878410 R paroutyj 0:19:56 R=8,W=1:0:0,J=I,P=fidle,T=gpu (Karma=0.005,quota_ok) 5878410 R paroutyj 0:19:56 R=8,W=1:0:0,J=I,P=fidle,T=gpu (Karma=0.005,quota_ok)
5896266 W paroutyj 0:00:00 R=8,W=1:0:0,J=B,N=Full convolutions,P=fidle,T=gpu 5896266 W paroutyj 0:00:00 R=8,W=1:0:0,J=B,N=Full convolutions,P=fidle,T=gpu
# ls -l # ls -l
total 8 total 8
-rw-r--r-- 1 paroutyj l-simap 0 Feb 28 15:58 batch_oar_5896266.err -rw-r--r-- 1 paroutyj l-simap 0 Feb 28 15:58 batch_oar_5896266.err
-rw-r--r-- 1 paroutyj l-simap 5703 Feb 28 15:58 batch_oar_5896266.out -rw-r--r-- 1 paroutyj l-simap 5703 Feb 28 15:58 batch_oar_5896266.out
``` ```
%% Cell type:code id: tags:
``` python
pwk.end()
```
%% Output
End time is : Thursday 7 January 2021, 15:41:18
Duration is : 00:00:01 101ms
This notebook ends here
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
---- ----
<div class='todo'> **What you can do :**
Your mission if you accept it: Run our full_convolution code in batch mode.<br>
For that :
<ul>
<li>Validate the full_convolution notebook on short tests</li>
<li>Submit it in batch mode for validation</li>
<li>Modify the notebook for a full run and submit it :-)</li>
</ul>
</div> (If you have a calculation environment with a scheduler...)
Your mission if you accept it: Run our full_convolution code in batch mode.<br>
For that :
- Validate the full_convolution notebook on short tests</li>
- Submit it in batch mode for validation</li>
- Modify the notebook for a full run and submit it :-)</li>
%% Cell type:markdown id: tags: %% Cell type:markdown id: tags:
--- ---
<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img> <img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment