WaveQC

Description

Quality for seismologic data at Resif

Visuals

Installation

First, ensure that git, python3, postgresql and rabbitmq are installed

sudo apt install git-core postgresql python3 rabbitmq-server redis npm

(Note that the minimum python version required is python 3.10 due to the syntax of type hints in waveqc code)

Then, install poetry. Poetry is a tool for dependency management and packaging.

curl -sSL https://install.python-poetry.org | python3 -
# put poetry in your $PATH
echo 'export PATH="$HOME/.local/bin:$PATH"' >> ~/.bashrc
# enable tab completion for poetry commands
poetry completions bash >> ~/.bash_completion

Then, clone the project from the Gitlab repository:

git clone git@gricad-gitlab.univ-grenoble-alpes.fr:OSUG/RESIF/waveqc.git
cd waveqc

Install python dependencies :

poetry install

# show info on the virtualenv created
poetry env info

Configuration

Environment Variables

Copy the dotenv sample and modify the credentials for PostgreSql and Amqp You may also change the directory where the mseed and stationxml files will be temporary stored DEBUG should be set to 0 (False) in a production environment

cp .env.sample .env

PostgreSQL

Create a PostGreSQL role and database dedicated to WaveQC with the right credentials

su - postgres
createuser --pwprompt <waveqc_user>
createdb -O <waveqc_user> waveqc

Get a dump of staging database and restore it on your local machine

# on your locale machine as the postgres user
su - postgres
pg_dump waveqc -h resif-pgpreprod.u-ga.fr -U waveqc --file waveqc.dump --format c
pg_restore --clean -d waveqc waveqc.dump

Migrations

Once the database is created, we can run project migrations

# activate virtualenv
poetry shell

# run migrations
alembic upgrade head

Static files

See Parcel docs Generate static files by running the following commands

# install npm dependencies
npm install

# generate static files
npm run build

RabbitMQ

See Celery Doc for rabbitmq (Installing a rabbitmq-server on your local machine is not mandatory. If one is allready available somewhere, define its dsn in .env file)

Create a RabbitMQ user, vhost, and permissions for that user on that vhost. Use the same credentials as the one you defined in WAVEQC_AMQP_DSN in your .env file

sudo rabbitmqctl add_user <waveqc_user> <waveqc_password>
sudo rabbitmqctl add_vhost <waveqc>
sudo rabbitmqctl set_user_tags <waveqc_user> <waveqc_tag>
sudo rabbitmqctl set_permissions -p <waveqc> <waveqc_user> ".*" ".*" ".*"

# And restart the rabbitmq-server
sudo systemctl restart rabbitmq-server.service

Redis

See Celery Doc for redis (Installing a redis-server on your local machine is not mandatory. If one is allready available somewhere, define its dsn in .env file)

sudo apt install redis

Celery

Using bash

For celery:

celery -A waveqc.tasks:app worker -l INFO

For celery-beat

celery -A waveqc.tasks:app beat -l INFO

Note that using this method, you'll need to run these commands each time you restart your computer In order to start celery and celery-beat along the system startup, you'll need to daemonize celery

Daemonization

In order to use Celery tasks, we need to run it as a system service. See Celery Daemonization Doc

First, we need to create all necessary directories

sudo cp conf/tmpfiles/celery.conf /etc/tmpfiles.d/
sudo systemd-tmpfiles --create

Move configuration files from local repository to their destination

sudo cp conf/celery /etc/conf.d/
sudo cp conf/systemd/celery.service /etc/systemd/system/
sudo cp conf/systemd/celerybeat.service /etc/systemd/system/

You'll need to edit those configuration files:

• in /etc/conf.d/celery replace CELERY_BIN with the path of the celery bin (inside the virtualenv created by poetry)
• in /etc/systemd/system/celery.service replace :
  • WorkingDirectory with the path where you cloned the waveqc repository
  • USER and GROUP with your user and your group
• in /etc/systemd/system/celerybeat.service replace :
  • WorkingDirectory with the path where you cloned the waveqc repository
  • USER and GROUP with your user and your group
• in /etc/tmpfiles.d/celery.conf replace :
  • USER and GROUP with your user and your group

Finally, don't forget to enable and start celery daemon

sudo systemctl daemon-reload
sudo systemctl enable celery.service
sudo systemctl enable celerybeat.service
sudo systemctl start celery.service
sudo systemctl start celerybeat.service

Usage

Local server

In order to launch WaveQC in your browser :

poetry shell
pserve app.ini --reload

The application will be available at this url : http://localhost:6543/

Pyramid Shell

A python shell is also available loaded with :

• models (Network, Station, etc...)
• the request object (which has a dbsession as attribute)
• some sqlalchemy functions (select, delete, func, etc...)

In order to launch Pyramid shell :

poetry shell
pshell app.ini

Buildah/Podman

Once buildah and podman installed, you can build the waveqc docker image with the following command

buildah bud -t waveqc --layers --build-arg SENTRY_RELEASE=123456789

... and run it with the following command (make sure you have the appropriate .env.docker env file)

podman run -p 8000:8000 --env-file=".env.docker" --rm waveqc

Support

Our issue tracker is available here

Roadmap

Roadmap for this project is available in the original specifications

Contributing

If you're willing to contribute to this project, here are some tools that we are using to lint, test and format code.

Pre-commit hooks

First of, install the pre-commit hooks.

To do so, activate the virtual env with the following command and run the install command for pre-commit:

poetry shell
pre-commit install

Now, each time you'll want to commit, several tasks will be run to ensure certain standards of code quality:

• ruff : a linter and formatter for python files
• djLint : a formatter for jinja templates
• poetry-checks : checks that python dependencies are not commited in a broken state

Configuration for pre-commit is located in .pre-commit-config.yaml file

Gitlab Ci jobs

In addition to pre-commit hooks, the following jobs are run after each push:

• pytest : unit tests
• mypy : a type checker for python files

Each of these tools can be run locally with the following commands : pytest and mypy

Two more jobs are also run :

• secret-detection : detects if password, secrets, ssh keys are present in repository
• semgrep-sast : Static Application Security Testing

For those two last jobs, reports are available in each merge requests as a json file

Tooling configuration

If you are curious about the configuration of all of these tools, please consult the pyproject.toml file

Authors and acknowledgment

• Philippe Bollard (Project manager)
• Simon Panay (Developments)
• Laurent Stehly (Scientific overview)

WaveQC is funded by ISTerre

License

This software is licenced under the AGPLv3 Licence

Project status

In Development