Commit 2a43f63a authored by paugier's avatar paugier
Browse files

Add notebooks Python language

parent dbf67fcf
*.pyd*
*.pyc
*~
*tmp
**/tmp
*.so
*.o
*.pstats
......
......@@ -31,6 +31,7 @@ help:
clean:
rm -f ipynb/*.slides.html
rm -f ipynb/index.html
rm -rf ipynb/tmp
lab:
jupyter-lab
......
%% Cell type:markdown id: tags:
# Python for High Performance Computing
**A training to scale up Python code **
Pierre Augier (LEGI), Cyrille Bonamy (LEGI), Eric Maldonado (Irstea), Franck Thollard (ISTerre), Christophe Picard (LJK), Loïc Huder (ISTerre)
- Day 0: Python language
- Day 1: Numpy / Scipy / Pandas + Visualization
- Day 2: Packaging, documentation, intro HPC, profiling
- Day 3 and 4: Wrapping, Python accelerators, parallel computing + personal projects
%% Cell type:markdown id: tags:
# Python training UGA 2017
**A training to acquire strong basis in Python to use it efficiently**
Pierre Augier (LEGI), Cyrille Bonamy (LEGI), Eric Maldonado (Irstea), Franck Thollard (ISTerre), Christophe Picard (LJK), Loïc Huder (ISTerre)
# First steps
%% Cell type:markdown id: tags:
## Why Python ?
- A very nice and well-thought programming language.
- **Generalist**: good for several applications, [used with success in many fields](http://brochure.getpython.info/).
- For academic purposes (teaching, research): **great scientific ecosystem**.
%% Cell type:markdown id: tags:
## Python: a language, an interpreter
Python is a programming language.
It is first an **interpreted** language (like Bash, Matlab, and in contrast with Fortran, C or C++), so we need an interpreter, i.e. a binary parsing Python code and running computer instructions.
Open a terminal. There are usually different python commands: `python`, `python2` and `python3`.
%% Cell type:markdown id: tags:
### Remark Python 2 / Python 3
2 languages, 2 interpreters, 2 commands
%% Cell type:markdown id: tags:
## Python distributions
- On Linux, there are the system python binaries.
- On Windows, Mac (and Linux), you can install and use
* https://www.python.org/downloads/
* [Anaconda or Miniconda](https://www.continuum.io/downloads)
([advice on how to install a good scientific Python environment](http://fluiddyn.readthedocs.io/en/latest/get_good_Python_env.html))
- In some systems (cluster), there are modules:
```bash
module load python
```
%% Cell type:markdown id: tags:
### Do it yourself: execute a script with the command `python3`
Go into the root directory of the repository of the training.
Run a script with `python3` (pyfiles/helloworld.py is *just* a text file):
```bash
cat pyfiles/helloworld.py
python3 pyfiles/helloworld.py
```
(`cat` is a Unix command that print the content of a text file.)
%% Cell type:markdown id: tags:
### Do it yourself: work interactively with [ipython](https://ipython.org/)
The command `ipython3` launch the program [ipython](https://ipython.org/), which is used for interactive python.
- In ipython, you can execute a first interactive instruction:
%% Cell type:code id: tags:
``` python
2 + 2
```
%%%% Output: execute_result
4
%% Cell type:code id: tags:
``` python
3 / 4
```
%%%% Output: execute_result
0.75
%% Cell type:markdown id: tags:
- Run the script from ipython.
%% Cell type:code id: tags:
``` python
run ../pyfiles/helloworld.py
```
%% Cell type:code id: tags:
``` python
# name is defined in ../pyfiles/helloworld.py
print(name)
```
%% Cell type:markdown id: tags:
- Help on an object can be displayed with the question mark (try it):
```ipython
name?
```
- Try to type "name." and to press on tab... The tab key is used to tell you how what you typed can be completed.
- Try to use the top and bottom arrows...
%% Cell type:markdown id: tags:
### Python in an IDE ([Spyder](https://pythonhosted.org/spyder/))
Launch the application Spyder, a Python IDE (Integrated development environment).
- a quite good code editor with:
- syntax coloring,
- code analysis powered by pyflakes and pylint,
- introspection capabilities such as code completion.
- Ipython console
- variable inspector
- ...
Remarks:
- it is very important to use a good editor to code in Python (for example Spyder, emacs or vi with [a good setup](https://bitbucket.org/fluiddyn/fluid_emacs.d), or [Visual Studio Code](https://code.visualstudio.com/docs/languages/python)).
%% Cell type:markdown id: tags:
### Python in the browser ([Jupyter](https://jupyter.readthedocs.io))
The presentations of this python training are made with Jupyter (demonstration).
This is a very powerful tool to present results (see these [examples](http://nbviewer.jupyter.org/)).
%% Cell type:markdown id: tags:
# Python training UGA 2017
**A training to acquire strong basis in Python to use it efficiently**
Pierre Augier (LEGI), Cyrille Bonamy (LEGI), Eric Maldonado (Irstea), Franck Thollard (ISTerre), Christophe Picard (LJK), Loïc Huder (ISTerre)
# Characteristics of the Python language
%% Cell type:markdown id: tags:
## Characteristics of the Python language
- Open-source language, interpreters and ecosystem
- Interpreted (but there are tools to compile Python code)
- Automatic memory management
- Dynamically strongly typed
%% Cell type:markdown id: tags:
## Dynamically and strongly typed: objects and variables
The function `type` returns the type of an **object**:
%% Cell type:code id: tags:
``` python
type('hello')
```
%%%% Output: execute_result
str
%% Cell type:code id: tags:
``` python
type(2)
```
%%%% Output: execute_result
int
%% Cell type:code id: tags:
``` python
type(2.)
```
%%%% Output: execute_result
float
%% Cell type:code id: tags:
``` python
type(2 + 2)
```
%%%% Output: execute_result
int
%% Cell type:code id: tags:
``` python
type(2 + 2.)
```
%%%% Output: execute_result
float
%% Cell type:code id: tags:
``` python
type(True)
```
%%%% Output: execute_result
bool
%% Cell type:markdown id: tags:
## Dynamically and strongly typed: objects and variables
**Variables** are just tags pointing towards objects. New variables can be used when needed. They are not associated with a type but only with an object (which has a type)...
%% Cell type:code id: tags:
``` python
myvar = 1
print(myvar, type(myvar))
```
%% Cell type:code id: tags:
``` python
myvar = 'hello'
print(myvar, type(myvar))
```
%% Cell type:markdown id: tags:
## Spaces for objects and variables (names)
Objects and variables (names) are two very different concepts:
- Objects live in one "object space". They have an address in the memory.
- Names live in namespaces.
It is often interesting to represent the execution of a Python program in an **"object space - namespaces" diagram**.
The Zen of Python says "Namespaces are one honking great idea -- let's do more of those!". A namespace is created for every module (file) and for every function execution,
%% Cell type:markdown id: tags:
## Characteristics of the Python language
- Open-source language, interpreters and ecosystem
- Interpreted (but there are tools to compile Python code)
- Automatic memory management
- Dynamically strongly typed
- Gradual learning curve
- A philosophy: the [Zen of Python](https://www.python.org/dev/peps/pep-0020/)
- Very clean and readable
- Indentation defines the blocks
- Style coding is important: [pep8](https://www.python.org/dev/peps/pep-0008/)
%% Cell type:markdown id: tags:
## PEP8 (Python Extension Proposal)
https://www.python.org/dev/peps/pep-0008/
- Code layout
- Imports
- Whitespace in expressions and statements
- Comments
- Documentation strings
- Naming conventions
- Programming recommendations
%% Cell type:markdown id: tags:
## PEP8: examples of bad and good style practices
%% Cell type:code id: tags:
``` python
# bad (spaces between operator)
number=0
# ok
number = 0
```
%% Cell type:code id: tags:
``` python
# bad (indentation with 2 spaces, has to be 4)
if number == 0:
number = 1
# ok
if number == 0:
number = 1
```
%% Cell type:code id: tags:
``` python
# bad (space after ,)
mylist = [1,2,3]
# ok
mylist = [1, 2, 3]
```
%% Cell type:markdown id: tags:
## You have to use a good editor!!
Python:
- (most of the time) interpreted (no compilation)
- very dynamic
- pep8 (style).
- indentation defines the blocks
The editor can and has to help you!!
- syntax coloring
- automatic indentation
- code analysis (code analysis powered by pyflakes and pylint)
- introspection capabilities such as code completion
%% Cell type:markdown id: tags:
## Do it yourself: PEP8 checker with Spyder editor
In Spyder, open the file `./pyfiles/wrong.py` and see what append.
Activate the "Real time code style analysis (PEP 8)" (Preferences -> Editor -> Code Introspection/Analysis).
Correct the file to remove the warnings.
%% Cell type:markdown id: tags:
## Characteristics of the Python language
- Open-source language, interpreters and ecosystem
- Interpreted (but there are tools to compile Python code)
- Automatic memory management
- Dynamically strongly typed
- Gradual learning curve
- A philosophy: the [Zen of Python](https://www.python.org/dev/peps/pep-0020/)
- Very clean and readable
- Indentation defines the blocks
- Style coding is important: [pep8](https://www.python.org/dev/peps/pep-0008/)
- Only few [keywords](https://hg.python.org/cpython/file/3.6/Lib/keyword.py) and [built-in functions](https://docs.python.org/3/library/functions.html)
%% Cell type:markdown id: tags:
## Keywords
%% Cell type:code id: tags:
``` python
help("keywords")
```
%% Cell type:markdown id: tags:
plus 69 [built-in functions](https://docs.python.org/3/library/functions.html)... plus few [built-in constants](https://docs.python.org/3/library/constants.html)... plus [built-in exceptions](https://docs.python.org/3/library/exceptions.html)!
%% Cell type:markdown id: tags:
#### [Built-in functions](https://docs.python.org/3/library/functions.html) and [built-in exceptions](https://docs.python.org/3/library/exceptions.html):
%% Cell type:code id: tags:
``` python
types = [type, type(sum)]
for type_ in types:
names = [
name for name in dir(__builtins__)
if isinstance(getattr(__builtins__, name), type_)
]
print(f'\n {len(names)} "{type_.__name__}" object(s): {names}')
```
%% Cell type:markdown id: tags:
## Characteristics of the Python language
- Open-source language, interpreters and ecosystem
- Interpreted (but there are tools to compile Python code)
- Automatic memory management
- Dynamically strongly typed
- Gradual learning curve
- A philosophy: the [Zen of Python](https://www.python.org/dev/peps/pep-0020/)
- Very clean and readable
- Indentation defines the blocks
- Style coding is important: [pep8](https://www.python.org/dev/peps/pep-0008/)
- Only few [keywords](https://hg.python.org/cpython/file/3.6/Lib/keyword.py) and [built-in functions](https://docs.python.org/3/library/functions.html)
- Errors should never pass silently
- Multi-paradigm (sequential, object-oriented, functional)
- "Batteries Included": [the standard library](https://docs.python.org/3/tutorial/stdlib.html)
This diff is collapsed.
This diff is collapsed.
%% Cell type:markdown id: tags:
# Python training UGA 2017
**A training to acquire strong basis in Python to use it efficiently**
Pierre Augier (LEGI), Cyrille Bonamy (LEGI), Eric Maldonado (Irstea), Franck Thollard (ISTerre), Christophe Picard (LJK), Loïc Huder (ISTerre)
# Read / write files
%% Cell type:markdown id: tags:
## Read / write files
There are a lot of specialized tools to open specialized files (images, xml, csv, hdf5, netcdf, etc.). Here we focus on the low-level general method to open files.
### [`open` built-in function](https://docs.python.org/3/library/functions.html#open) and file object
%% Cell type:code id: tags:
``` python
file = open('../pyfiles/helloworld.py')
txt = file.read()
file.close()
print(txt)
```
%% Cell type:markdown id: tags:
But what if something weird happens when the file is open (e.g. a division by O) ?
-> Exception is raised that could be catch and run some code that is not aware of the opening
file.
-> The file remains open
%% Cell type:markdown id: tags:
### Context, `with` keyword
For such objects that need to be closed, it is a good practice to use the keyword `with` (THIS IS MUCH BETTER than using the close function, USE `with`!). Like this, we are sure that the file will be closed even if there is an error:
%% Cell type:code id: tags:
``` python
with open('../pyfiles/helloworld.py') as file:
txt = file.read()
print(txt)
```
%% Cell type:markdown id: tags:
### Loop over lines
%% Cell type:code id: tags:
``` python
with open('../pyfiles/helloworld.py') as file:
for line in file:
print(f'line ?: ' + line.strip())
```
%% Cell type:markdown id: tags:
And now using `enumerate` to get the index of the line:
%% Cell type:code id: tags:
``` python
with open('../pyfiles/helloworld.py') as file:
for i, line in enumerate(file):
print(f'line {i:2d}: {line.strip()}')
```
%% Cell type:markdown id: tags:
### Options of the built-in function open (read, write, append)
%% Cell type:code id: tags:
``` python
from pathlib import Path
Path("tmp").mkdir(exist_ok=True)
```
%% Cell type:code id: tags:
``` python
# write data in a file
with open('tmp/zoo.txt', 'w') as file_zoo:
file_zoo.write('sam;cat;2\n')
file_zoo.write('liloo;lion;2\n')
with open('tmp/zoo.txt', 'a') as file_zoo:
file_zoo.write('peter;panda;5\n')
with open('tmp/zoo.txt') as file_zoo:
print(file_zoo.read())
```
%% Cell type:code id: tags:
``` python
with open('tmp/zoo.txt', 'r') as file_zoo:
print(file_zoo.readline())
print(file_zoo.read())
```
%% Cell type:markdown id: tags:
Remark: difference `write` and `print`.
%% Cell type:markdown id: tags:
### Options of the built-in function open (binary file)
Until now, we have only written text files. It can of course be much more efficient to use binary format.
%% Cell type:code id: tags:
``` python
with open('/tmp/test', 'wb') as file:
file.write(b'a')
```
%% Cell type:markdown id: tags:
Remarks:
- In practice, saving data in binary file is most of the time a bad idea. There are much better solutions to do this (see for example h5py and h5netcdf).
- There are Python libraries to read and process many types for files (csv, xml, json, images, tabulars, etc.).
This diff is collapsed.