Skip to content
Snippets Groups Projects
Commit 659137cd authored by Jean-Luc Parouty's avatar Jean-Luc Parouty
Browse files

Merge branch 'dev' of gricad-gitlab.univ-grenoble-alpes.fr:talks/fidle into dev

parents be9e742b a8e84281
No related branches found
No related tags found
No related merge requests found
Hide whitespace changes
Inline Side-by-side
GTSRB/06-Notebook-as-a-batch.ipynb
+ 16
− 147
View file @ 659137cd
%% Cell type:markdown id: tags:
<img width="800px" src="../fidle/img/00-Fidle-header-01.svg"></img>
# <!-- TITLE --> [GTSRB6] - Full convolutions as a batch
<!-- DESC --> Episode 6 : To compute bigger, use your notebook in batch mode
<!-- AUTHOR : Jean-Luc Parouty (CNRS/SIMaP) -->
## Objectives :
- Run a notebook code as a **job**
- Follow up with Tensorboard
The German Traffic Sign Recognition Benchmark (GTSRB) is a dataset with more than 50,000 photos of road signs from about 40 classes.
The final aim is to recognise them !
Description is available there : http://benchmark.ini.rub.de/?section=gtsrb&subsection=dataset
## What we're going to do :
Our main steps:
- Run Full-convolution.ipynb as a batch :
- Notebook mode
- Script mode
- Tensorboard follow up
%% Cell type:markdown id: tags:
### Step 1 - Import and init
Not really useful here ;-)
%% Cell type:code id: tags:
``` python
import sys
sys.path.append('..')
import fidle.pwk as pwk
datasets_dir = pwk.init('GTSRB6')
```
%% Output
<br>**FIDLE 2020 - Practical Work Module**
Version : 1.2b1 DEV
Notebook id : GTSRB6
Run time : Thursday 7 January 2021, 15:41:17
TensorFlow version : 2.2.0
Keras version : 2.3.0-tf
Datasets dir : /home/pjluc/datasets/fidle
Run dir : ./run
Update keras cache : False
Save figs : True
Path figs : ./run/figs
%% Cell type:markdown id: tags:
## Step 2 - How to run a notebook as a batch ?
## Step 1 - How to run a notebook as a batch ?
Two simple solutions are possible :-)
- **Option 1 - As a notebook ! (a good choice)**
Very simple.
The result is the executed notebook, so we can retrieve all the cell'soutputs of the notebook :
```jupyter nbconvert (...) --to notebook --execute <notebook>```
Example :
```jupyter nbconvert --ExecutePreprocessor.timeout=-1 --to notebook --execute my_notebook.ipynb'```
The result will be a notebook: 'my_notebook.nbconvert.ipynb'.
See: [nbconvert documentation](https://nbconvert.readthedocs.io/en/latest/usage.html#convert-notebook)
- **Option 2 - As a script**
Very simple too, but with some constraints on the notebook.
We will convert the notebook to a Python script (IPython, to be precise) :
```jupyter nbconvert --to script <notebook>```
Then we can execute this script :
```ipython <script>```
See: [nbconvert documentation](https://nbconvert.readthedocs.io/en/latest/usage.html#executable-script)
%% Cell type:markdown id: tags:
## Step 3 - Run as a script
## Step 2 - Run as a script
Maybe not always the best solution, but this solution is very rustic !
### 3.1 - Convert to IPython script :
### 2.1 - Convert to IPython script :
%% Cell type:code id: tags:
``` python
! jupyter nbconvert --to script --output='05-full_convolutions' '05-Full-convolutions.ipynb'
! ls -l *.py
```
%% Output
[NbConvertApp] Converting notebook 05-Full-convolutions.ipynb to script
[NbConvertApp] Writing 12854 bytes to 05-full_convolutions.py
-rw-r--r-- 1 pjluc pjluc 12854 Jan 7 15:41 05-full_convolutions.py
[NbConvertApp] Writing 12984 bytes to 05-full_convolutions.py
-rw-r--r-- 1 uja62cb mlh 12984 Jan 28 11:35 05-full_convolutions.py
%% Cell type:markdown id: tags:
### 2.2 - Batch submission
See the two examples of bash launch script :
- `batch_slurm.sh` using Slurm (like at IDRIS)
- `batch_oar.sh` using OAR (like at GRICAD)
See the two examples of bash launch script :
- [batch_slurm.sh](batch_slurm.sh) using Slurm (like at IDRIS)
- [batch_oar.sh](batch_oar.sh) using OAR (like at GRICAD)
%% Cell type:markdown id: tags:
#### Example at IDRIS
On the frontal :
```bash
# hostname
jean-zay2
# sbatch $WORK/fidle/GTSRB/batch_slurm.sh
Submitted batch job 249794
#squeue -u $USER
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
249794 gpu_p1 GTSRB Fu uja62cb PD 0:00 1 (Resources)
# ls -l _batch/
total 32769
-rw-r--r-- 1 uja62cb gensim01 13349 Sep 10 11:32 GTSRB_249794.err
-rw-r--r-- 1 uja62cb gensim01 489 Sep 10 11:31 GTSRB_249794.out
```
%% Cell type:markdown id: tags:
#### Example at GRICAD
Have to be done on the frontal :
```bash
# hostname
f-dahu
# pwd
/home/paroutyj
# oarsub -S ~/fidle/GTSRB/batch_oar.sh
[GPUNODE] Adding gpu node restriction
[ADMISSION RULE] Modify resource description with type constraints
#oarstat -u
Job id S User Duration System message
--------- - -------- ---------- ------------------------------------------------
5878410 R paroutyj 0:19:56 R=8,W=1:0:0,J=I,P=fidle,T=gpu (Karma=0.005,quota_ok)
5896266 W paroutyj 0:00:00 R=8,W=1:0:0,J=B,N=Full convolutions,P=fidle,T=gpu
# ls -l
total 8
-rw-r--r-- 1 paroutyj l-simap 0 Feb 28 15:58 batch_oar_5896266.err
-rw-r--r-- 1 paroutyj l-simap 5703 Feb 28 15:58 batch_oar_5896266.out
```
%% Cell type:code id: tags:
``` python
pwk.end()
```
%% Output
End time is : Thursday 7 January 2021, 15:41:18
Duration is : 00:00:01 101ms
This notebook ends here
%% Cell type:markdown id: tags:
----
<div class='todo'>
Your mission if you accept it: Run our full_convolution code in batch mode.<br>
For that :
<ul>
<li>Validate the full_convolution notebook on short tests</li>
<li>Submit it in batch mode for validation</li>
<li>Modify the notebook for a full run and submit it :-)</li>
</ul>
**What you can do :**
</div>
(If you have a calculation environment with a scheduler...)
Your mission if you accept it: Run our full_convolution code in batch mode.<br>
For that :
- Validate the full_convolution notebook on short tests</li>
- Submit it in batch mode for validation</li>
- Modify the notebook for a full run and submit it :-)</li>
%% Cell type:markdown id: tags:
---
<img width="80px" src="../fidle/img/00-Fidle-logo-01.svg"></img>
......
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment