HySoP (Hybrid Simulation with Particles) is a library dedicated to high performance direct numerical simulation of fluid related problems based on semi-lagrangian particle methods, for hybrid architectures providing multiple compute devices including CPUs, GPUs or MICs. See https://particle_methods.gricad-pages.univ-grenoble-alpes.fr/hysop-doc for more information.

Basics

Download and install lastest hysop:

git clone git@gricad-gitlab.univ-grenoble-alpes.fr:particle_methods/hysop.git
cd hysop
mkdir build
cd build
cmake ..
make -j8
make install

By default, cmake will try to find your most up to date Python3 installation. The minimum required version is Python3.8. You can force the python version by using the following trick during the cmake configuration step:

PYTHON_EXECUTABLE="$(which python3.8)"
PYTHON_INCLUDE_DIR=$(${PYTHON_EXECUTABLE} -c "import sysconfig as sc; print(sc.get_paths()['include'])")  
PYTHON_LIBRARY=$(${PYTHON_EXECUTABLE} -c "import sysconfig as sc, os; print(os.path.normpath(os.path.sep.join(sc.get_config_vars('LIBDIR', 'INSTSONAME'))))")
cmake -DPython3_EXECUTABLE="${PYTHON_EXECUTABLE}" -DPython3_INCLUDE_DIR="${PYTHON_INCLUDE_DIR}" -DPython3_LIBRARY="${PYTHON_LIBRARY}" ..

Run Taylor-Green with 8 mpi processes:

mpirun -np 8 python3.9 hysop_examples/examples/taylor_green/taylor_green.py -impl fortran -V -cp fp64 -d 64

Check examples directory for complete simulation cases. Hysop is currently developped and tested with python3.9. It also works well with python3.8. You may want to export PYTHON_EXECUTABLE=python3.8 to run some of hysop scripts that look for python3.9 by default. Please note that python3.10 has not been tested yet but should work as well.

For each example you can get help with the -h option:

 python3.9 hysop_examples/examples/taylor_green/taylor_green.py -h

Dependencies:

To install hysop dependencies locally on an ubuntu machine (20.04 LTS):

sudo apt-get install -y expat unzip xz-utils automake libtool pkg-config cmake git ssh gcc g++ gfortran lsb-core cpio libnuma1 libpciaccess0 libreadline-dev libblas-dev liblapack-dev libgcc-10-dev libgfortran-10-dev libgmp-dev libmpfr-dev libmpc-dev python3.8-dev opencl-headers swig libgmp-dev libmpfr-dev libmpc-dev libcairo-dev libcairomm-1.0-dev python3.8-dev python3.8-tk libopenmpi-dev libhdf5-openmpi-dev libfftw3-dev libfftw3-mpi-dev
python3.9 -m pip install --upgrade numpy setuptools cffi wheel pytest pybind11 cython
python3.9 -m pip install --upgrade -r requirements.txt

Additionally you may want to provide a working OpenCL platform, HPTT, llvm/llvmlite/numba, clFFT/gpyFFT, flint/arb/python-flint and tbb/mklfft. See the docker files to install those packages (hysop/ci/docker_images/ubuntu). Alternatively your can run hysop in an isolated environment by using docker containers (see next section).

How to use docker images ?

Docker images can be pulled (resp. pushed) with ./ci/utils/pull_docker_image.sh [imgname] and ./ci/utils/push_docker_image.sh [imgname]. The docker images do not contain de hysop library and can be run with ./ci/utils/run_docker_image.sh [imgname]. This script mounts your local hysop directory (read only) to /hysop inside the docker container and prompt a shell. Images have to be downloaded (pulled) prior to be run with this script.

By default, [imgname] corresponds to the docker image used for gitlab continuous integration (currently jammy, which corresponds to Ubuntu 22.04 running with python3.9). Docker images ship an intel OpenCL platform that is compatible with intel CPUs. Cuda enabled images may be a bit out-of-date (see ./ci/docker_images/ubuntu/bionic_cuda to get started).

To quickly test and/or debug hysop inside the docker you can run ./ci/utils/run_debug.sh [imgname] which will build and install hysop inside the container and prompt a shell (read-only mount-bind /hysop directory is copied to read-write /tmp/hysop within the container). Alternatively if you just want to run tests inside docker, you can use the ./ci/utils/run_ci.sh [imgname] script instead.

Docker images can be build locally by using the ./ci/utils/build_docker_image.sh script. It is advised to build docker image only if the pull fails or if you want to add/update dependencies. Each build takes around one hour (36 cores @ 2GHz). By default, the build script will use all of your cores. At least 16GB of RAM is recommended.

Known problems within docker containers

1. OpenMPI complains about the program being run as root: You can solve this by defining two extra variables:

   export OMPI_ALLOW_RUN_AS_ROOT=1
   export OMPI_ALLOW_RUN_AS_ROOT_CONFIRM=1

   These variables are already defined during tests and in ./ci/utils/run_debug.sh.

2. OpenMPI complains about not having enough slots: This happens when you request more processes than you have physical cpu cores. To be able to run more processes you can define the following variables: export OMPI_MCA_rmaps_base_oversubscribe=1. This variable is already defined during tests and in ./ci/utils/run_debug.sh.

3. The program crashes with Bus error (Non-existant physical address): By default, docker does not enable ptrace which makes the vader BTL crash (vader is openmpi shared memory interprocess communicator). There are three independent solutions to this problem:

   • Enable ptrace when running docker (best solution): use docker run --cap-add=SYS_PTRACE to run an interactive container or docker create --cap-add=SYS_PTRACE to create a persistent container. This is already done in our scripts in ./ci/utils*.
   • Disable vader CMA (Cross Memory Attach) with OMPI_MCA_btl_vader_single_copy_mechanism=none: works but interprocess communication performance drops ~50%.
   • Disable vader completely: mpirun --mca btl ^vader: works but leads to really poor performances