Commit 0ba606a5 authored by Franck Thollard's avatar Franck Thollard
Browse files

Merge branch 'master' of gricad-gitlab.univ-grenoble-alpes.fr:python-uga/training-hpc

parents ee47b450 ab5a9f67
......@@ -824,7 +824,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.3"
"version": "3.7.2"
}
},
"nbformat": 4,
......
......@@ -40,6 +40,53 @@
"- IO bounded (wait for IO)"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Different parallel strategies"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"### IO bounded: one process + `async`/`await`\n",
"\n",
"<p class=\"small\"><br></p>\n",
"\n",
"Cooperative concurrency\n",
"\n",
"<p class=\"small\"><br></p>\n",
"\n",
"Functions able to pause\n",
"\n",
"<p class=\"small\"><br></p>\n",
"\n",
"`asyncio`, [`trio`](https://trio.readthedocs.io)"
]
},
{
"cell_type": "raw",
"metadata": {
"slideshow": {
"slide_type": "-"
}
},
"source": [
"<div align=\"middle\">\n",
"<img src=\"./fig/logo_trio.png\" style=\"width: 15%\">\n",
"</div>"
]
},
{
"cell_type": "markdown",
"metadata": {
......
%% Cell type:markdown id: tags:
# Parallel computing (CPU bounded)
Pierre Augier (LEGI), Cyrille Bonamy (LEGI), Eric Maldonado (Irstea), Franck Thollard (ISTerre), Christophe Picard (LJK), Loïc Huder (ISTerre)
%% Cell type:markdown id: tags:
# Parallelism
## Hardware:
- Multicore CPU
- Multi nodes super computers (MPI)
- GPU (Nvidia: Cuda, Cupy) / Intel Xeon Phi
<p class="small"><br></p>
## Different problems
- CPU bounded (need to use cores at the same time)
- IO bounded (wait for IO)
%% Cell type:markdown id: tags:
## Different parallel strategies
%% Cell type:markdown id: tags:
### IO bounded: one process + `async`/`await`
<p class="small"><br></p>
Cooperative concurrency
<p class="small"><br></p>
Functions able to pause
<p class="small"><br></p>
`asyncio`, [`trio`](https://trio.readthedocs.io)
%% Cell type:raw id: tags:
<div align="middle">
<img src="./fig/logo_trio.png" style="width: 15%">
</div>
%% Cell type:markdown id: tags:
## Different parallel strategies
### One process split in light subprocesses called threads 👩🏼‍🔧 👨🏼‍🔧👩🏼‍🔧 👨🏼‍🔧
- handled by the OS
- share memory and can use at the same time different CPU cores
How?
- OpenMP (Natively in C / C++ / Fortran. For Python: Pythran, Cython, ...)
- In Python: `threading` and `concurrent.futures`
%% Cell type:markdown id: tags:
**⚠️ in Python, one interpreter per process (~) and the Global Interpreter Lock (GIL)...**
- In a Python program, different threads can run at the same time (and take advantage of multicore)
- But... the Python interpreter runs the Python bytecodes sequentially !
- *Terrible* 🐌 for CPU bounded *if the Python interpreter is used a lot* !
- *No problem* for IO bounded !
- *No problem* for extensions when the interpreter is not used !
%% Cell type:markdown id: tags:
## Different parallel strategies
### One program, $n$ processes 👩🏼‍🔧 👨🏼‍🔧👩🏼‍🔧 👨🏼‍🔧
### Exchange data:
Very efficient and no problem with Python!
- With MPI: mpi4py and h5py parallel
- multiprocessing
- [ZeroMQ](http://zeromq.org/)
%% Cell type:markdown id: tags:
## Other packages for parallel computing with Python
- IPython
- dask
- joblib
%% Cell type:markdown id: tags:
# Code examples
## Target algorithms :
- computes the first prime number (with a **very** brute force algorithm)
- given a set of number, computes the distance map between each number
- computes for a set of number, its factors
## Solutions
- multiprocessing (map)
- multiprocessing (starmap)
- concurent.futures
- ...
......
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Port Python 2 code to Python 3"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once upon a time, it was very difficult to:\n",
"\n",
"- convert code written in Python 2\n",
"\n",
"- write code compatible for py2 and py3\n",
"\n",
"Dependencies + strong language incompatibilities (between Python 2.6 and 3.3).\n",
"\n",
"\n",
"Now, it much simpler!\n",
"\n",
"- main packages all work in Python 3\n",
"\n",
"- smaller language incompatibility (between Python 2.7 and >3.4) (for example `from __future__ import print_function, division`)\n",
"\n",
"- good tools to write compatible code ([six](https://pypi.org/project/six/) and [future](https://python-future.org))\n",
"\n",
"But... still some differences... For example for us in sciences:\n",
"\n",
"- division `int / int`\n",
"\n",
"- str are unicode in Python 3 but not in Python 2\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"## Steps to port Python 2 code to Python 3\n",
"\n",
"- Solidify the Python 2 code (write tests!)\n",
"\n",
"- Use [pyupgrade](https://github.com/asottile/pyupgrade) to modernize the code to Python 2.7 ➡️ test!\n",
"\n",
"- Use the automatic tool `futurize -w mymodule.py` (provided by [future](https://python-future.org)) ➡️ test!\n",
"\n",
"Today, it can make sense to stop supporting Python 2.7 (and even Python 3.5). See https://python3statement.org/. \n",
"\n",
"Main advantage: One can use nice modern features, for example:\n",
"\n",
"- f-strings\n",
"\n",
"- `from pathlib import Path`\n",
"\n",
"It can be done automatically with pyupgrade:\n",
"\n",
"`find -name \"*.py\" | xargs pyupgrade --py36-plus`\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
%% Cell type:markdown id: tags:
# Port Python 2 code to Python 3
%% Cell type:markdown id: tags:
Once upon a time, it was very difficult to:
- convert code written in Python 2
- write code compatible for py2 and py3
Dependencies + strong language incompatibilities (between Python 2.6 and 3.3).
Now, it much simpler!
- main packages all work in Python 3
- smaller language incompatibility (between Python 2.7 and >3.4) (for example `from __future__ import print_function, division`)
- good tools to write compatible code ([six](https://pypi.org/project/six/) and [future](https://python-future.org))
But... still some differences... For example for us in sciences:
- division `int / int`
- str are unicode in Python 3 but not in Python 2
%% Cell type:markdown id: tags:
## Steps to port Python 2 code to Python 3
- Solidify the Python 2 code (write tests!)
- Use [pyupgrade](https://github.com/asottile/pyupgrade) to modernize the code to Python 2.7 ➡️ test!
- Use the automatic tool `futurize -w mymodule.py` (provided by [future](https://python-future.org)) ➡️ test!
Today, it can make sense to stop supporting Python 2.7 (and even Python 3.5). See https://python3statement.org/.
Main advantage: One can use nice modern features, for example:
- f-strings
- `from pathlib import Path`
It can be done automatically with pyupgrade:
`find -name "*.py" | xargs pyupgrade --py36-plus`
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Developer tools\n",
"\n",
"## IDE (open-source)\n",
"\n",
"- Spyder\n",
"\n",
"- Visual Studio Code\n",
"\n",
"- ...\n",
"\n",
"\n",
"## Checker and formatter\n",
"\n",
"- [pylint](https://www.pylint.org/): check Python code\n",
"\n",
"- [black](https://github.com/python/black): The uncompromising Python code formatter\n",
"\n",
"- [mypy](http://mypy-lang.org/): Optional Static Typing for Python\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
%% Cell type:markdown id: tags:
# Developer tools
## IDE (open-source)
- Spyder
- Visual Studio Code
- ...
## Checker and formatter
- [pylint](https://www.pylint.org/): check Python code
- [black](https://github.com/python/black): The uncompromising Python code formatter
- [mypy](http://mypy-lang.org/): Optional Static Typing for Python
{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Port a Matlab code to Python\n",
"\n",
"⚠️ ⚠️ Difficult !! ⚠️ ⚠️\n",
"\n",
"### Converter (with runtime dependence)\n",
"\n",
"- [Smop](https://github.com/victorlei/smop), Small Matlab to Python compiler: convert Matlab code to Python code\n",
"\n",
"See https://stackoverflow.com/a/17535694\n",
"\n",
"### Other approach\n",
"\n",
"- fluidmat2py, provided by [fluiddyn](https://fluiddyn.readthedocs.io)\n",
"\n",
"```\n",
"usage: fluidmat2py [-h] [-c] path\n",
"\n",
"Utility to produce a strange code which is no longer Matlab and not yet Python.\n",
"\n",
"positional arguments:\n",
" path str indicating which file or directory has to be used.\n",
"\n",
"optional arguments:\n",
" -h, --help show this help message and exit\n",
" -c, --clean Only clean the Matlab code.\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.7.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}
%% Cell type:markdown id: tags:
# Port a Matlab code to Python
⚠️ ⚠️ Difficult !! ⚠️ ⚠️
### Converter (with runtime dependence)
- [Smop](https://github.com/victorlei/smop), Small Matlab to Python compiler: convert Matlab code to Python code
See https://stackoverflow.com/a/17535694
### Other approach
- fluidmat2py, provided by [fluiddyn](https://fluiddyn.readthedocs.io)
```
usage: fluidmat2py [-h] [-c] path
Utility to produce a strange code which is no longer Matlab and not yet Python.
positional arguments:
path str indicating which file or directory has to be used.
optional arguments:
-h, --help show this help message and exit
-c, --clean Only clean the Matlab code.
```
%% Cell type:markdown id: tags:
......@@ -30,8 +30,9 @@ module dtw_cort
end do
! Fill the dtw_matrix
do i = 2, len_s1
do j = 2, len_s2
do j = 2, len_s2
do i = 2, len_s1
dist = dabs(s1(i) - s2(j))
dtw_mat(i, j) = dist + dmin1(dtw_mat(i - 1, j), &
dtw_mat(i, j - 1), &
......
all:
python3 -m numpy.f2py -c "dtw_cort.f90" -m distances_fort
python3 -m numpy.f2py --opt="-O3 -march=native" -c "dtw_cort.f90" -m distances_fort
clean:
rm -f distances_fort.*.so *.mod
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment