Add notebooks + better index page

.gitignore

@@ -16,4 +16,6 @@ ipynb/index.html
 .vscode
 
 TP/TP2_package/mypackuga/build
-TP/TP2_package/mypackuga/dist
\ No newline at end of file
+TP/TP2_package/mypackuga/dist
+
+ipynb/index.rst

Makefile

@@ -2,6 +2,9 @@ IPYNBDIR := ipynb
 IPYNBFILES := $(shell find $(IPYNBDIR) -name '*.ipynb' | grep -v ipynb_checkpoints)
 IPYNBPRES = $(addsuffix .slides.html, $(basename $(IPYNBFILES)))
 
+# TPL_FILE = ipynb/slides_reveal_wide.tpl
+TPL_FILE = ipynb/slides_reveal.tpl
+
 define STR_HELP
 This makefile can be used for
 
@@ -17,7 +20,7 @@ endef
 
 export STR_HELP
 
-.PHONY: help lab serve presentations
+.PHONY: help lab serve presentations ipynb/index.rst
 
 help:
 	@echo "$$STR_HELP"
@@ -25,18 +28,22 @@ help:
 clean:
 	rm -f ipynb/*.slides.html
 	rm -f ipynb/index.html
+	rm -rf ipynb/tmp
 
 lab:
 	jupyter-lab
 
+ipynb/index.rst:
+	python3 ipynb/make_index.py
+
 ipynb/index.html: ipynb/index.rst
 	cd $(IPYNBDIR) && rst2html5 index.rst > index.html
 
-%.slides.html: %.ipynb ipynb/slides_reveal_wide.tpl
-	jupyter-nbconvert $< --reveal-prefix='./reveal.js' --to slides --template ipynb/slides_reveal_wide.tpl
+%.slides.html: %.ipynb $(TPL_FILE)
+	jupyter-nbconvert $< --reveal-prefix='./reveal.js' --to slides --template $(TPL_FILE)
 
-presentations: $(IPYNBPRES) ipynb/index.html
-	echo $(IPYNBPRES)
+presentations: ipynb/index.html $(IPYNBPRES)
+	@echo $(IPYNBPRES)
 
 serve: $(IPYNBPRES) ipynb/index.html
-	cd $(IPYNBDIR) && python3 -m http.server 9000
+	cd $(IPYNBDIR) && python3 -m http.server

ipynb/22_intro_HPC.ipynb

+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": true,
+    "slideshow": {
+     "slide_type": "skip"
+    }
+   },
+   "outputs": [],
+   "source": [
+    "# this ipython command loads a special backend for notebooks\n",
+    "%matplotlib notebook\n",
+    "import matplotlib.pyplot as plt\n",
+    "import numpy as np\n",
+    "\n",
+    "from IPython.core.display import display as display_base, HTML\n",
+    "\n",
+    "# https://stackoverflow.com/questions/21356014/how-can-i-insert-a-new-tag-into-a-beautifulsoup-object\n",
+    "\n",
+    "# https://github.com/lepture/mistune\n",
+    "import mistune\n",
+    "markdown = mistune.Markdown()\n",
+    "\n",
+    "def display(code):\n",
+    "    return display_base(HTML(code))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Overview of the Python HPC landscape 🚀"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<div align=\"middle\"> Python, only a great glue language? </div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "<div align=\"middle\"> Pierre Augier </div>"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "metadata": {},
+   "source": [
+    "<div align=\"middle\">\n",
+    "<img src=\"./fig/logo_CNRS.jpg\"  style=\"width: 10%\"> &nbsp; &nbsp;\n",
+    "<img src=\"./fig/logo_LEGI.jpg\"  style=\"width: 15%\"> &nbsp; &nbsp;\n",
+    "<img src=\"./fig/logo-uga-vo-cmjn.jpg\"  style=\"width: 15%\">\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Python for High-Performance Computing?\n",
+    "\n",
+    "1. *Fast prototyping* (Numpy!)\n",
+    "\n",
+    "2. *Popular*:\n",
+    "\n",
+    "    - Well-known\n",
+    "    \n",
+    "    - Several great libraries\n",
+    "\n",
+    "3. *Share ideas* between developers / scientists\n",
+    "\n",
+    "    - Popularity counts\n",
+    "\n",
+    "    - Readability counts\n",
+    "\n",
+    "    - Expressivity counts\n",
+    "    \n",
+    "4. Anyway, one needs a good and well-known *scripting language* so yes!\n",
+    "\n",
+    "   (even considering Julia)\n",
+    "   \n",
+    "### *Where / when should we stop ?*\n",
+    "   "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Python & fast prototyping...\n",
+    "\n",
+    "## The software engineering method for scientists 👩‍🔬 👨‍🔬 and HPC\n",
+    "\n",
+    "1. Fast prototyping\n",
+    "\n",
+    "2. Solidify **as needed**\n",
+    "\n",
+    "    * ↗ code quality\n",
+    "    \n",
+    "    * [testing](https://docs.pytest.org) \n",
+    "    \n",
+    "    * refactoring    \n",
+    "    \n",
+    "    * [type hints](https://docs.python.org/3/library/typing.html) and [type checking](http://mypy-lang.org/)\n",
+    "    \n",
+    "    * ↗ performances for bottlenecks (profile ⏱ !)\n",
+    "    \n",
+    "Again and again: `(1, 2)`, `(1, 2)`, ...\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "## Python: a programming language, compromises ⚖️\n",
+    "\n",
+    "Designed for fast prototyping & \"glue\" codes together\n",
+    "\n",
+    "- Generalist + easy to learn ⇒ huge and diverse community 👨🏿‍🎓🕵🏼 👩🏼‍🎓 👩🏽‍🏫👨🏽‍💻👩🏾‍🔬 🎅🏼 🌎 🌍 🌏\n",
+    "\n",
+    "- Expressivity and readability\n",
+    "\n",
+    "- Not oriented towards high performance \n",
+    "\n",
+    "  (fast and easy dev, easy debug, correctness)\n",
+    "\n",
+    "    - Highly dynamic 🐒 + introspection (`inspect.stack()`)\n",
+    "\n",
+    "    - Automatic memory management 💾\n",
+    "\n",
+    "    - All objects encapsulated 🥥 (PyObject, C struct) \n",
+    "    \n",
+    "    - Objects accessible through \"references\" ➡️ \n",
+    "    \n",
+    "    - Usually interpreted \n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "## Python interpreters\n",
+    "\n",
+    "\n",
+    "### [CPython](https://github.com/python/cpython) <img src=\"./fig/logo_python.png\" align=\"left\" style=\"width: 7%; margin-bottom: 2px; margin-right: 5px;\">\n",
+    "\n",
+    "Interpreted (nearly) instruction per instruction, (nearly) no code optimization\n",
+    "\n",
+    "The numerical stack (Numpy, Scipy, Scikits, ...) based on the CPython C API (CPython implementation details)!\n",
+    "\n",
+    "<br>\n",
+    "\n",
+    "### [PyPy](https://pypy.org/) <img src=\"./fig/logo_pypy.png\" align=\"left\" style=\"width: 7%; margin-bottom: 0px; margin-right: 5px;\">\n",
+    "\n",
+    "Optimized implementation with tracing Just-In-Time compilation\n",
+    "\n",
+    "[\"Abstractions for free\"](https://www.youtube.com/watch?v=NQfpHQII2cU)\n",
+    "\n",
+    "The CPython C API is an issue! PyPy can't accelerate Numpy code!\n",
+    "\n",
+    "<br>\n",
+    "\n",
+    "### [Micropython](http://micropython.org/) <img src=\"./fig/logo_micropython.jpeg\" align=\"left\" style=\"width: 7%; margin-bottom: 0px; margin-right: 5px;\">\n",
+    "\n",
+    "For microcontrollers"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "## Python & performance\n",
+    "\n",
+    "### References and PyObjects"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "mylist = [1, 3, 5]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "`list`: array of references towards PyObjects"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### The C / Python border"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "4\n"
+     ]
+    }
+   ],
+   "source": [
+    "arr = 2 * np.arange(10)\n",
+    "print(arr[2])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "## Python & performance\n",
+    "\n",
+    "### Python interpreters bad at crunching numbers"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "-"
+    }
+   },
+   "source": [
+    "Pure Python terrible 🐢 (except with PyPy)..."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "from math import sqrt\n",
+    "my_const = 10.\n",
+    "result = [elem * sqrt(my_const * 2 * elem**2) for elem in range(1000)]"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "fragment"
+    }
+   },
+   "source": [
+    "but even this is not *very* efficient (temporary objects)..."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "a = np.arange(1000)\n",
+    "result = a * np.sqrt(my_const * 2 * a**2)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Even slightly worth with PyPy 🙁"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "## Is Python efficient enough?"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "-"
+    }
+   },
+   "source": [
+    "Python is known to be slow... But what does it mean?\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "### Efficiency / inefficiency: depends on tasks ⏱\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "### When is it inefficient? Especially for number crunching 🔢 ...\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "### Can we write efficient scientific code in 🐍 ?"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "## Book"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "metadata": {},
+   "source": [
+    "<div align=\"middle\">\n",
+    "<img src=\"./fig/high-perf-python-book.jpg\"  style=\"width: 35%; margin-top: 0px;\">\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "See also https://faster-cpython.readthedocs.io/"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Performance (generalities)\n",
+    "\n",
+    "### Measure ⏱, don't guess! Profile to find the bottlenecks.\n",
+    "\n",
+    "Cprofile (pstats, SnakeViz), line-profiler, [`perf`](https://pypi.org/project/perf/), [perf_events](https://perf.wiki.kernel.org)\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "### Do not optimize everything!\n",
+    "\n",
+    "- *\"Premature optimization is the root of all evil\"* (Donald Knuth)\n",
+    "\n",
+    "- 80 / 20 rule, efficiency important for expensive things and NOT for small things\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "### CPU or IO bounded problems\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "### Use the *right algorithms* and the right data structures!\n",
+    "\n",
+    "For example, using Numpy arrays instead of Python lists...\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "### Unittests before optimizing to maintain correctness!\n",
+    "\n",
+    "`unittest`, `pytest`"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "## \"Crunching numbers\" and computers architectures"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "metadata": {},
+   "source": [
+    "<div align=\"middle\">\n",
+    "<img src=\"./fig/modern-hardware-1.png\"  style=\"width: 30%; margin-top: -30px; margin-bottom: -50px\"> &nbsp;\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "-"
+    }
+   },
+   "source": [
+    "### CPU optimizations\n",
+    "\n",
+    "- pipelining, hyper-threading, vectorization, advanced instructions (simd), ...\n",
+    "\n",
+    "- important to get data aligned in memory (arrays)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Proper compilation needed for high efficiency !\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "## Compilation to virtual machine instructions\n",
+    "\n",
+    "What does CPython (`compile`, \"byte code\", nearly no optimization, see `dis` module)\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "## Compilation to machine instructions\n",
+    "\n",
+    "- Just-in-time \n",
+    "\n",
+    "  Has to be fast (warm up), can be hardware specific\n",
+    "\n",
+    "- Ahead-of-time \n",
+    "\n",
+    "  Can be slow, hardware specific or more general to distribute binaries\n",
+    "\n",
+    "*Compilers are usually good for optimizations!* Better than most humans...\n",
+    "\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "## Transpilation\n",
+    "\n",
+    "From one language to another language (for example Python to C++)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "# Parallelism\n",
+    "\n",
+    "## Hardware: \n",
+    "\n",
+    "- Multicore CPU\n",
+    "\n",
+    "- Multi nodes super computers (MPI)\n",
+    "\n",
+    "- GPU (Nvidia: Cuda, Cupy) / Intel Xeon Phi\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "## Different problems\n",
+    "\n",
+    "- CPU bounded (need to use cores at the same time)\n",
+    "\n",
+    "- IO bounded (wait for IO)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }
+   },
+   "source": [
+    "## Different parallel strategies"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "-"
+    }
+   },
+   "source": [
+    "### IO bounded: one process + `async`/`await`\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "Cooperative concurrency\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "Functions able to pause\n",
+    "\n",
+    "<p class=\"small\"><br></p>\n",
+    "\n",
+    "`asyncio`, [`trio`](https://trio.readthedocs.io)"
+   ]
+  },
+  {
+   "cell_type": "raw",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "-"
+    }
+   },
+   "source": [
+    "<div align=\"middle\">\n",
+    "<img src=\"./fig/logo_trio.png\"  style=\"width: 15%\">\n",
+    "</div>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "slideshow": {
+     "slide_type": "slide"
+    }