README.md 6.65 KB
Newer Older
paugier's avatar
paugier committed
1
# Training Python HPC at UGA
Eric Maldonado's avatar
Eric Maldonado committed
2

paugier's avatar
paugier committed
3 4 5 6
This repository contains documents for a course on using Python for High
Performance Computing.

Although this is not classic for an interpreted language, we will see that we
7
can take advantage of many Python features to go toward high performances.
paugier's avatar
paugier committed
8 9

Before considering performances, we will address the problem of evaluating
10
performance, a.k.a. profiling Python code.
paugier's avatar
paugier committed
11

12

13
Formated output of the notebooks is available here: https://python-uga.gricad-pages.univ-grenoble-alpes.fr/training-hpc 
14 15

## Format et dates
paugier's avatar
paugier committed
16 17

sur 2 semaines
Eric Maldonado's avatar
Eric Maldonado committed
18 19 20 21 22

- 2 jours contigus (intro / numpy / matplot / pandas)
- 1 jour doc et package
- 2 jours contigus calcul intensif et optimisation

23
#### Dates sessions
paugier's avatar
paugier committed
24

25
- 12-13/06/2019 et 17-19/06/2019
Eric Maldonado's avatar
Eric Maldonado committed
26

27
## Prerequisites
paugier's avatar
paugier committed
28

29 30
- Coding experience with Python (see [UGA Python
  training](https://gricad-gitlab.univ-grenoble-alpes.fr/python-uga/py-training-2017))
Eric Maldonado's avatar
Eric Maldonado committed
31

32
- Algorithmics (loops, conditions)
paugier's avatar
paugier committed
33

34
- Knowledge of basic Linux command-line instructions (`ls`, `cd`)
Eric Maldonado's avatar
Eric Maldonado committed
35

36
## Planning
paugier's avatar
paugier committed
37

38 39
- Introduction, rappels langage Python et gestion de src [1j : 12/06] (Pierre,
Raphaël et Loïc)
Eric Maldonado's avatar
Eric Maldonado committed
40

41
- Numpy / Scipy [0.5j : 13/06] (Loïc et Raphaël)
paugier's avatar
paugier committed
42 43

  - fancy indexing, masking
paugier's avatar
paugier committed
44
  - presentation of the example "dtw_cort_dist"
paugier's avatar
paugier committed
45
  - broadcast
paugier's avatar
paugier committed
46
  - vectorizing
paugier's avatar
paugier committed
47

paugier's avatar
paugier committed
48
- Visualization - Matplotlib and other tools [0.25j : 13/06] (Loïc et Raphaël)
paugier's avatar
paugier committed
49

paugier's avatar
paugier committed
50 51
  - matplotlib object oriented (axes, features...) (Loïc)
  - figures et animations (Loïc)
52 53
  - voir graph plotting python jackc (Raphaël)
  - point bokeh, plotly, seaborn (Raphaël)
Eric Maldonado's avatar
Eric Maldonado committed
54

paugier's avatar
paugier committed
55
- Pandas [0.25j : 13/06] (Eric - Raphaël)
paugier's avatar
paugier committed
56
  - load, query, display, export
Eric Maldonado's avatar
Eric Maldonado committed
57

58
- Packaging, documentation and testing [0.5j : 17/06] (Raphaël et Pierre)
paugier's avatar
paugier committed
59 60 61 62 63 64 65 66 67 68 69 70 71 72 73

- General introduction HPC with Python [0.25j] (Pierre)

- Profiling [0.25j] (Pierre et Franck)

  See the book "High performance Python".

  - perf
  - cprofile (pstat, SnakeViz)
  - pyperf
  - line_profiler
  - kcachegrind
  - Memory (memory_profiler, ...)

- Wrapping low level code and/or compiled code [0.5j] (Franck et Loic)
74

paugier's avatar
paugier committed
75
  Mainly for legacy code...
Eric Maldonado's avatar
Eric Maldonado committed
76

paugier's avatar
paugier committed
77
  ctypes, cffi, cython, cppyy, pybind11, f2py, pyo3, swig
Eric Maldonado's avatar
Eric Maldonado committed
78

paugier's avatar
paugier committed
79
- Different ways to speed up native Python code [0.5j] (Pierre - Franck)
Eric Maldonado's avatar
Eric Maldonado committed
80

81 82 83 84
  - using on the fly compiler (like e.g. numba)
  - using static compilers (like e.g. pythran)
  - using cython (évocation juste)
  - transonic
85

paugier's avatar
paugier committed
86
- Tools for parallelisation [0.25j]
Eric Maldonado's avatar
Eric Maldonado committed
87

paugier's avatar
paugier committed
88
  threading / multiprocessing / concurrent.futures / OpenMP / mpi4py / GPU /joblib / dask (Pierre - Franck)
Eric Maldonado's avatar
Eric Maldonado committed
89

paugier's avatar
paugier committed
90
- Déploiement sur cluster et mésocentre (lien singularity, charlie cloud) [0.25j] (Franck - Raphaël)
Eric Maldonado's avatar
Eric Maldonado committed
91

paugier's avatar
paugier committed
92
- Travaux sur projets personnels [0.5j au moins]
Franck Thollard's avatar
Franck Thollard committed
93 94


paugier's avatar
paugier committed
95
### Présence formateurs
96

paugier's avatar
paugier committed
97 98
- m: matin
- a: après-midi
99

paugier's avatar
paugier committed
100 101
|         | m 12 | a 12 | m 13 | a 13 | m 17 | a 17 | m 18 | a 18 | m 19 | a 19 |
|---------|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|:----:|
Eric Maldonado's avatar
Eric Maldonado committed
102
| Eric    | (x)  | (x)  | (x)  | (x)  | (x)  |      | (x)  | (x)  | (x)  | (x)  |
paugier's avatar
paugier committed
103
| Franck  |      |      |      |      |      |  x   |  x   |  x   |  x   |  x   |
104
| Loïc    |      |      |  x   |  x   |      |  x   |      |      |      |      |
Raphael Bacher's avatar
Raphael Bacher committed
105 106
| Raphaël |      |      |  x   |  x   |  x   |      |      |      |      |  x   |
| Pierre  |  x   |   x  |      |      |  x   |  x   | (x)  |  x   |  x   | (x)  |
107 108 109 110 111 112 113 114
| Cyrille |      |      |      |      |      |      |      |      |      |      |

## To be done before the first day of the training session

We will have to use Python 3 (with Miniconda3), a good Python IDE (either
Spyder or VSCode), Jupyter and a version-control tool (Mercurial, or Git if you
know it and really like it).

115 116 117 118
In the following, we give indications about how to install these tools and how
to get the repository of this training locally on your computer. Please, try to
do this before the training and tell us if you encounter problems. You can fill
an issue
119 120 121 122
[here](https://gricad-gitlab.univ-grenoble-alpes.fr/python-uga/training-hpc/issues)
to explain what you did and what are the errors (please copy / paste the error
log).

123 124 125 126 127 128
Moreover, let's add that the best OS for HPC (and HPC with Python) is
Linux/GNU. Windows (at least without
[WLS](https://en.wikipedia.org/wiki/Windows_Subsystem_for_Linux)) and even
macOS are less adapted for this particular application. Python is a
cross-platform language but nevertheless, you will get a better experience for
HPC with Python on Linux. Therefore, we encourage the participant to work
paugier's avatar
paugier committed
129 130 131 132
during this training with a (real or
[virtual](https://www.virtualbox.org/wiki/Downloads)) Linux machine. Of course,
if you can't or don't want to use Linux, come with your computer on Windows or
macOS.
133 134 135
We offer a linux virtual machine ready to use (via virtualbox 6.X) for training.
It can be downloaded [here](https://filesender.renater.fr/?s=download&token=62c5f530-7def-8c32-4e3c-d205299a5201)

136 137 138 139 140 141 142 143 144 145 146 147 148

#### Install Python and utilities

The first steps are to install miniconda3 (see
[here](https://docs.conda.io/en/latest/miniconda.html)) and VSCode (see
[here](https://code.visualstudio.com/download)).

On Windows, all commands have to be run in the conda prompt.

You will need to activate the conda channel `conda-forge` with:

```conda config --add channels conda-forge```

paugier's avatar
paugier committed
149
With miniconda, it should be simple to install Spyder and Jupyter either with
paugier's avatar
paugier committed
150
the command `conda install spyder jupyterlab` or using the graphical tool
paugier's avatar
paugier committed
151
[Anaconda Navigator](https://docs.anaconda.com/anaconda/navigator/).
152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172

To install and setup Mercurial, you can follow the instructions
[here](https://fluiddyn.readthedocs.io/en/latest/mercurial_bitbucket.html).

Warning: Mercurial **has to be correctly setup**! Since we will use [the Gitlab
instance of UGA](https://gricad-gitlab.univ-grenoble-alpes.fr), the Mercurial
extension hg-git has to be activated so the line `hggit =` in the configuration
file is mandatory.

#### Clone this repository

Once everything is installed and setup, you should be able to clone the
repository of the training with:

```hg clone https://gricad-gitlab.univ-grenoble-alpes.fr/python-uga/training-hpc.git```

Please tell us before the training if it does not work.

#### Install few packages in your base conda environment

```
173
conda install --file requirements.txt
174 175 176 177 178 179 180 181 182 183
```

#### Check your environment

Once the repository is cloned you should have a new directory `training-hpc`
and you should be able to enter into it with `cd training-hpc`.

Finally, you can check that your Python environment is all fine with:

```python check_env.py```
paugier's avatar
paugier committed
184 185 186

#### Build the presentations

187 188 189
One needs Jupyter, rst2html5 (installable with `pip install rst2html5`), plus
`make` and few other Unix tools. Therefore, it is not easy to build the website
on Windows.
paugier's avatar
paugier committed
190

paugier's avatar
paugier committed
191 192
On Unix-like systems, `make presentations` should build the presentations and
`make serve` should start a server to visualize them in a browser.