Commit da3bcb90 authored by Loic Huder's avatar Loic Huder
Browse files
parents e2a41778 8a9d9eb6
......@@ -5,6 +5,7 @@
*.so
*.o
*.pstats
*.mod
_build
build
......@@ -24,4 +25,6 @@ ipynb/index.rst
ipynb/index.html
pyfiles/dtw_cort_dist/V5_cython/*.c
pyfiles/dtw_cort_dist/V5_cython/*.html
\ No newline at end of file
pyfiles/dtw_cort_dist/V5_cython/*.html
**/V*/res_cort.npy
**/V*/res_dtw.npy
\ No newline at end of file
......@@ -169,7 +169,7 @@ Please tell us before the training if it does not work.
#### Install few packages in your base conda environment
```
conda install ipython spyder jupyter numpy scipy pandas matplotlib
conda install --file requirements.txt
```
#### Check your environment
......
......@@ -2,11 +2,11 @@
# Python for High Performance Computing
**A training to scale up Python code **
Pierre Augier (LEGI), Cyrille Bonamy (LEGI), Eric Maldonado (Irstea), Franck Thollard (ISTerre), Christophe Picard (LJK), Loïc Huder (ISTerre)
Pierre Augier (LEGI), Cyrille Bonamy (LEGI), Eric Maldonado (Irstea), Franck Thollard (ISTerre), Raphaël Bacher (GIPSA-LAB), Loïc Huder (ISTerre)
%% Cell type:markdown id: tags:
# Python for High Performance Computing
......
This diff is collapsed.
......@@ -6,10 +6,12 @@
Pierre Augier (LEGI), Cyrille Bonamy (LEGI), Eric Maldonado (Irstea), Franck Thollard (ISTerre), Christophe Picard (LJK), Loïc Huder (ISTerre)
# Object-oriented programming: inheritance
See https://docs.python.org/3/tutorial/classes.html
Python is also a Object-oriented language. Object-oriented programming is very useful and used in many libraries so it is very useful to understand how the simple Object-oriented mechanisms work in Python.
%% Cell type:markdown id: tags:
For some problems, Object-oriented programming is a very efficient paradigm. Many libraries use it so it is necessary to understand how it works in Python to really use these libraries.
......@@ -77,10 +79,15 @@
The first line states that instances of the class `AdultBee` will be Python objects. The class `AdultBee` inherits from the class `object`.
The first line could have been replaced by the less explicit `class AdultBee:`. Actually, in Python 3, the classes that do not inherit explicitly from another class automatically inherit from the class `object`.
- `kind` and `limit_age` are **class variables**,
- `mother`, `father`, `tag` and `living` are **instance variables**,
- `__init__` is a **"special" method**,
- `act_and_envolve` and `die` are **methods**.
%% Cell type:markdown id: tags:
### Instantiation of a class
......@@ -136,12 +143,13 @@
Attributes and methods whose names start with `_` are said to be "protected". It is just a name convention. It tells the users that they should not use these objects directly.
%% Cell type:markdown id: tags:
### Warning for C++ users
`__init__` is NOT the constructor. The real constructor is `__new__`. This method is called to really create the Python object and it really returns the object. Usually, we do not need to redefine it. Python `__init__` and C++ constructor have to be used in very different ways. Only the `__init__` method of the class is automatically called by Python during the instantiation. Nothing like the Russian dolls C++ mechanism.
### Warning for C++ developers
`__init__` is NOT the constructor. The real constructor is `__new__`. This method is called to really create the Python object and it really returns the object. Usually, we do not need to redefine it. Python `__init__` and C++ constructor have to be used in very different ways. Only the `__init__` method of the class is automatically called by Python during the instantiation. Nothing like the Russian dolls C++ mechanism. All methods in Python are effectively virtual.
%% Cell type:markdown id: tags:
### Use the objects (instances)
......@@ -170,11 +178,11 @@
## Inheritance
%% Cell type:markdown id: tags:
To indicate the dependency to an other class, we put the parent class in parenthesis in the definition. The class `QueenBee` inherits from the class `AdultBee`
To indicate the dependency to another class, we put the parent class in parenthesis in the definition. The class `QueenBee` inherits from the class `AdultBee`
%% Cell type:code id: tags:
``` python
class QueenBee(AdultBee):
......@@ -289,21 +297,22 @@
def __init__(self):
pass
class Student(Person):
role = 'student'
@classmethod
def show_role(cls):
print('The role for this class is ' +
cls.role + '.')
Student.show_role()
```
%% Cell type:markdown id: tags:
### "Static methods"
### "Static methods" and class variables
For some situation we don't even need to explicitly use the class or an instance. We can use static methods.
%% Cell type:code id: tags:
......@@ -321,233 +330,12 @@
```
%% Cell type:code id: tags:
``` python
p1 = IdPerson('Pierre')
p1 = IdPerson('Maya')
p2 = IdPerson('Cyrille')
p3 = IdPerson('Olivier')
p4 = IdPerson('Franck')
IdPerson.show_nb_person()
```
%% Cell type:markdown id: tags:
## Do it yourself
At the end of the last presentation, we asked the following question about our weather stations measuring wind and temperature:
> What if we now have a weather station that also measure humidity ? Do we have to rewrite everything ?
Give your own answer by doing the following tasks:
- Write a class `HumidWeatherStation` inheriting `WeatherStation` (code reproduced below) to implement a new attribute to store the humidity measurements.
- Write a function `humidity_at_max_temp` that returns the value of the humidity at the maximal temperature. Use the fact that `HumidWeatherStation` inherits from `WeatherStation` and therefore can use the method `arg_max_temp` previously implemented !
- *Advanced*: Overloadg the methods of `WeatherStation` to take humidity into account when computing percieved temperatures `Tp`. For simplicity, we will assume that `Tp = Tw + 5*humidity` with `Tw` the temperature computed with the wind chill effect.
- *Advanced*: Write tests for this new class
%% Cell type:code id: tags:
``` python
# Code to use for the DIY
class WeatherStation(object):
""" A weather station that holds wind and temperature """
def __init__(self, wind, temperature):
""" initialize the weather station.
Precondition: wind and temperature must have the same length
:param wind: any ordered iterable
:param temperature: any ordered iterable"""
self.wind = [x for x in wind]
self.temp = [x for x in temperature]
if len(self.wind) != len(self.temp):
raise ValueError(
"wind and temperature should have the same size"
)
def perceived_temp(self, index):
""" computes the perceived temp according to
https://en.wikipedia.org/wiki/Wind_chill
i.e. The standard Wind Chill formula for Environment Canada is:
apparent = 13.12 + 0.6215*air_temp - 11.37*wind_speed^0.16 + 0.3965*air_temp*wind_speed^0.16
:param index: the index for which the computation must be made
:return: the perceived temperature"""
air_temp = self.temp[index]
wind_speed = self.wind[index]
# Perceived temperature does not have a sense without wind...
if wind_speed == 0:
apparent_temp = air_temp
else:
apparent_temp = 13.12 + 0.6215*air_temp \
- 11.37*wind_speed**0.16 \
+ 0.3965*air_temp*wind_speed**0.16
# Let's round to the integer to avoid trailing decimals...
return round(apparent_temp,0)
def perceived_temperatures(self):
""" Returns an array of percieved temp computed from the temperatures and wind speed data """
apparent_temps = []
for index in range(len(self.wind)):
# Reusing the method perceived_temp defined above
apparent_temperature = self.perceived_temp(index)
apparent_temps.append(apparent_temperature)
return apparent_temps
def max_temp(self, perceived=False):
""" returns the maximum temperature record in the station"""
if perceived:
apparent_temp = self.perceived_temperatures()
return max(apparent_temp)
else:
return max(self.temp)
def arg_max_temp(self, perceived=False):
""" returns the index of (one of the) maximum temperature record in the station"""
if perceived:
temp_array_to_search = self.perceived_temperatures()
else:
temp_array_to_search = self.temp
return temp_array_to_search.index(self.max_temp(perceived))
```
%% Cell type:markdown id: tags:
### A Solution
%% Cell type:code id: tags:
``` python
class HumidWeatherStation(WeatherStation):
""" A weather station that holds wind, temperature and humidity. Inherits from WeatherStation """
def __init__(self, wind, temperature, humidity):
""" initialize the weather station.
Precondition: wind, temperature and humidity must have the same length
:param wind: any ordered iterable
:param temperature: any ordered iterable
:param humidity: any ordered iterable"""
# Delegate the initialisation of wind and temperature to the mother class constructor
super(HumidWeatherStation, self).__init__(wind, temperature)
# Or: super().init(wind, temperature)
# Add humidity treatement
self.humidity = [x for x in humidity]
if len(self.humidity) != len(self.temp):
raise ValueError("humidity and temperature should have the same size")
# If humidity and temp have the same size, humidity and wind do as well
# as len(temp) == len(wind) is enforced from the mother class constructor
def humidity_at_max_temp(self):
""" Returns the value of humidity at the maximal temperature
"""
index_max_temp = self.arg_max_temp()
return self.humidity[index_max_temp]
def perceived_temp(self, index):
""" Compute the perceived temperature according to wind_speed (wind-chill) and humidity
:param index: the index for which the computation must be made
:return: the perceived temperature"""
# Compute the wind-chilled temp from WeatherStation method
wind_chilled_temp = super().perceived_temp(index)
apparent_temp = wind_chilled_temp + 5*self.humidity[index]
return round(apparent_temp, 2)
def perceived_temps(self):
""" Returns an array of percieved temp computed from the temperatures, wind speed and humidity data """
apparent_temps = []
for index in range(len(self.temp)):
# This time, we use the perceived_temp method of HumidWeatherStation
apparent_temperature = self.perceived_temp(index)
apparent_temps.append(apparent_temperature)
return apparent_temps
singapore = HumidWeatherStation(wind=[11, 23, 23, 19, 18, 18],
temperature = [28, 33, 31, 32, 35, 34],
humidity = [0.78, 0.63, 0.61, 0.58, 0.5, 0.72])
print(singapore.humidity_at_max_temp()) #0.5 expected
print(singapore.max_temp()) #35 expected
# As we overloaded perceived_temp, the rest of the class features work with the new behaviour
print(singapore.perceived_temps())
print(singapore.max_temp(perceived=True))
```
%% Cell type:markdown id: tags:
In this case, we used inheritance to create the new object (`HumidWeatherStation`) to:
- Add new features (`humidity_at_max_temp`) to an existing object without rewritting the common features
- Define new behaviours for features already present (`perceived_temp`) that integrate well with the structure in place
For such needs, think about inheritance.
%% Cell type:markdown id: tags:
## Do it yourself (advanced)
- Write a class named `MovingObject` that has at least one attribute `position` and implements two functions `start()` and `stop()`. These 2 functions could just print for example "starting" and "stoping" (or they could do more funny things)...
- Write another class named `Car` that inherits `MovingObject` and overload start and stop functions.
- Use the classes (instantiate objects and use them).
- Options : add a static method in Car class that returns the number of cars.
%% Cell type:code id: tags:
``` python
# a solution
pollution = 0.
class MovingObject:
def __init__(self, position=0., max_speed=1., acceleration=1.):
self.position = position
self.max_speed = max_speed
self.acceleration = acceleration
def start(self):
print ("starting")
def stop(self):
print ("stoping")
class Car(MovingObject):
count = 0
def __init__(self, position=0., max_speed=1., acceleration=1., name='', pollution_start=0.5):
super(Car, self).__init__(position, max_speed, acceleration)
self.name = name
self.pollution_start = pollution_start
Car.count += 1
def start(self):
global pollution
print ('The car ' + self.name + ' starts (vrooum)')
pollution += self.pollution_start
def stop(self):
print ('The car ' + self.name + ' stops')
@staticmethod
def get_number_of_cars():
print ("There are " + str(Car.count) + " cars")
class Bike(MovingObject):
pass
ferrari = Car(name='Ferrari')
mybike = Bike()
mybike.name = 'blue bike'
porsche = Car(name='Porsche', pollution_start=0.8)
ferrari.start()
ferrari.stop()
porsche.start()
porsche.stop()
objs = [ferrari, porsche, mybike]
for f in objs:
f.start()
print(f'pollution at the end: {pollution}')
```
......
This diff is collapsed.
## Timing:
We use [invoke](https://www.pyinvoke.org/) (`pip install invoke`) to build and run the benchmarks. The full requirements can be installed with `pip install -r requirements.txt`.
We use [invoke](https://www.pyinvoke.org/) to build and run the benchmarks.
See [how to get autocompletion with invoke](http://docs.pyinvoke.org/en/1.2/invoke.html?highlight=completion#cmdoption-print-completion-script).
......
......@@ -19,8 +19,7 @@ def serie_pair_index_generator(number):
return (
(_idx_greater, _idx_lower)
for _idx_greater in range(number)
for _idx_lower in range(number)
if _idx_lower < _idx_greater
for _idx_lower in range(_idx_greater)
)
......@@ -39,7 +38,6 @@ def DTWDistance(s1, s2):
_dtw_mat[0, 0] = abs(s1[0] - s2[0])
# two special cases : filling first row and columns
for j in range(1, len_s2):
dist = abs(s1[0] - s2[j])
_dtw_mat[0, j] = dist + _dtw_mat[0, j - 1]
......
......@@ -5,7 +5,6 @@ from runpy import run_path
from pathlib import Path
import numpy as np
import math
util = run_path(Path(__file__).absolute().parent.parent / "util.py")
......@@ -20,8 +19,7 @@ def serie_pair_index_generator(number):
return (
(_idx_greater, _idx_lower)
for _idx_greater in range(number)
for _idx_lower in range(number)
if _idx_lower < _idx_greater
for _idx_lower in range(_idx_greater)
)
......@@ -68,14 +66,12 @@ def cort(s1, s2):
:rtype: float64
"""
slope_1 = s1[1:] - s1[:-1]
slope_2 = s2[1:] - s2[:-1]
num = np.sum(slope_1 * slope_2)
sum_square_x = np.sum(slope_1 * slope_1)
sum_square_y = np.sum(slope_2 * slope_2)
return num / (math.sqrt(sum_square_x * sum_square_y))
return num / (np.sqrt(sum_square_x * sum_square_y))
def compute(series, nb_series):
......
......@@ -30,8 +30,7 @@ def serie_pair_index_generator(number):
return (
(_idx_greater, _idx_lower)
for _idx_greater in range(number)
for _idx_lower in range(number)
if _idx_lower < _idx_greater
for _idx_lower in range(_idx_greater)
)
......
......@@ -27,8 +27,7 @@ def serie_pair_index_generator(number):
return (
(_idx_greater, _idx_lower)
for _idx_greater in range(number)
for _idx_lower in range(number)
if _idx_lower < _idx_greater
for _idx_lower in range(_idx_greater)
)
......
#!/usr/bin/python3
#!/usr/bin/env python3
import numpy as np
import sysconfig
......@@ -25,8 +25,11 @@ def serie_pair_index_generator(number):
:returns: pairs (lower, greater)
:rtype: a generator
"""
return ((_idx_greater, _idx_lower) for _idx_greater in range(number)
for _idx_lower in range(number) if _idx_lower < _idx_greater)
return (
(_idx_greater, _idx_lower)
for _idx_greater in range(number)
for _idx_lower in range(_idx_greater)
)
def cDTW(serie_a, serie_b):
......@@ -50,6 +53,7 @@ def cort(serie_a, serie_b):
ret = dllib.cort(a_ptr, b_ptr, len(serie_a))
return ret
def compute(series, nb_series):
gen = serie_pair_index_generator(nb_series)
......
dtw_cort.so: setup.py dtw_cort.pyx
python3 setup.py build_ext
mv build/lib.linux-x86_64-3.7/dtw_cort.cpython-37m-x86_64-linux-gnu.so dtw_cort.so
build: setup.py dtw_cort.pyx
python3 setup.py develop
run: dtw_cort.so
run:
python3 dtw_cort_dist_mat.py ../data.npy -s dists_cython.npz
test: dtw_cort.so
test:
python3 dtw_cort_dist_mat.py ../data.npy -s dists_cython.npz
python3 check.py
clean:
rm -rf build dtw_cort.c dtw_cort.so
rm -rf build dtw_cort.c dtw_cort.*.so
rm -f dists_cython.npz
rm -f dtw_cort.html
......@@ -10,6 +10,7 @@ from dtw_cort import cort, DTWDistance
util = run_path(Path(__file__).absolute().parent.parent / "util.py")
def serie_pair_index_generator(number):
""" generator for pair index (i, j) such that i < j < number
......@@ -20,8 +21,7 @@ def serie_pair_index_generator(number):
return (
(_idx_greater, _idx_lower)
for _idx_greater in range(number)
for _idx_lower in range(number)
if _idx_lower < _idx_greater
for _idx_lower in range(_idx_greater)
)
......
from distutils.core import setup
from setuptools import setup
from Cython.Build import cythonize
setup(
......
all:
transonic dtw_cort_dist_mat.py
transonic dtw_cort_dist_mat.py -pf "-march=native -O3 -DUSE_XSIMD"
clean:
rm -rf __pythran__
\ No newline at end of file
......@@ -21,8 +21,7 @@ def serie_pair_index_generator(number):
return (
(_idx_greater, _idx_lower)
for _idx_greater in range(number)
for _idx_lower in range(number)
if _idx_lower < _idx_greater
for _idx_lower in range(_idx_greater)
)
......@@ -62,23 +61,19 @@ def DTWDistance(s1, s2):
def cort(s1, s2):
""" Computes the cort between serie one and two (assuming they have the same length)
""" Computes the cort between series one and two (assuming they have the same length)
:param s1: the first serie (or any iterable over floats64)
:param s2: the second serie (or any iterable over floats64)
:param s1: the first series (or any iterable over floats64)
:param s2: the second series (or any iterable over floats64)
:returns: the cort distance
:rtype: float64
"""
num = 0.0
sum_square_x = 0.0
sum_square_y = 0.0
for t in range(len(s1) - 1):
slope_1 = s1[t + 1] - s1[t]
slope_2 = s2[t + 1] - s2[t]
num += slope_1 * slope_2
sum_square_x += slope_1 * slope_1
sum_square_y += slope_2 * slope_2
slope_1 = s1[1:] - s1[:-1]
slope_2 = s2[1:] - s2[:-1]
num = np.sum(slope_1 * slope_2)
sum_square_x = np.sum(slope_1 * slope_1)
sum_square_y = np.sum(slope_2 * slope_2)
return num / (np.sqrt(sum_square_x * sum_square_y))
......
......@@ -22,8 +22,7 @@ def serie_pair_index_generator(number):
return (
(_idx_greater, _idx_lower)
for _idx_greater in range(number)
for _idx_lower in range(number)
if _idx_lower < _idx_greater
for _idx_lower in range(_idx_greater)
)
......@@ -63,23 +62,19 @@ def DTWDistance(s1, s2):
def cort(s1, s2):
""" Computes the cort between serie one and two (assuming they have the same length)
""" Computes the cort between series one and two (assuming they have the same length)
:param s1: the first serie (or any iterable over floats64)
:param s2: the second serie (or any iterable over floats64)
:param s1: the first series (or any iterable over floats64)
:param s2: the second series (or any iterable over floats64)
:returns: the cort distance
:rtype: float64
"""
num = 0.0
sum_square_x = 0.0
sum_square_y = 0.0
for t in range(len(s1) - 1):
slope_1 = s1[t + 1] - s1[t]
slope_2 = s2[t + 1] - s2[t]
num += slope_1 * slope_2
sum_square_x += slope_1 * slope_1
sum_square_y += slope_2 * slope_2
slope_1 = s1[1:] - s1[:-1]
slope_2 = s2[1:] - s2[:-1]
num = np.sum(slope_1 * slope_2)
sum_square_x = np.sum(slope_1 * slope_1)
sum_square_y = np.sum(slope_2 * slope_2)
return num / (np.sqrt(sum_square_x * sum_square_y))