RESIF issueshttps://gricad-gitlab.univ-grenoble-alpes.fr/groups/OSUG/RESIF/-/issues2024-03-27T15:08:34+01:00https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/235Experiment on problematic stations2024-03-27T15:08:34+01:00Simon PanayExperiment on problematic stationshttps://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/233Add an ARCHITECTURE.md file2024-03-22T17:21:54+01:00Simon PanayAdd an ARCHITECTURE.md fileto have a quick description of the architecture of waveqc (the schedule, the celery workers, the webapp, etc...)
see an example at the end of this article : https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html?utm_campaign=Django%2...to have a quick description of the architecture of waveqc (the schedule, the celery workers, the webapp, etc...)
see an example at the end of this article : https://matklad.github.io/2021/02/06/ARCHITECTURE.md.html?utm_campaign=Django%2BNewsletter&utm_medium=web&utm_source=Django_Newsletter_221https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/229Use Rye as python project manager ?2024-03-22T13:57:22+01:00Simon PanayUse Rye as python project manager ?See https://rye-up.com/
This may change the way we manage dependencies
How would be the impact on docker image creation ? A lead can be found here : [https://micro.webology.dev/2024/03/14/pythons-uv-tool.html](https://micro.webology.de...See https://rye-up.com/
This may change the way we manage dependencies
How would be the impact on docker image creation ? A lead can be found here : [https://micro.webology.dev/2024/03/14/pythons-uv-tool.html](https://micro.webology.dev/2024/03/14/pythons-uv-tool.html?utm_campaign=Django%2BNewsletter&utm_medium=web&utm_source=Django_Newsletter_224)https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect/-/issues/16Lorsque redis redémarre, il y a un cas non géré.2024-03-18T12:13:32+01:00Jonathan SchaefferLorsque redis redémarre, il y a un cas non géré.Sentry Issue: [WSDATASELECT-2M](https://osug.sentry.io/issues/5079090354/?referrer=gitlab_integration)
```
Plug.Conn.WrapperError: ** (FunctionClauseError) no function clause matching in Plug.Conn.resp/3
File "lib/plug/conn.ex", line ...Sentry Issue: [WSDATASELECT-2M](https://osug.sentry.io/issues/5079090354/?referrer=gitlab_integration)
```
Plug.Conn.WrapperError: ** (FunctionClauseError) no function clause matching in Plug.Conn.resp/3
File "lib/plug/conn.ex", line 614, in Plug.Conn.resp/3
File "lib/plug/conn.ex", line 593, in Plug.Conn.send_resp/3
File "deps/plug/lib/plug/router.ex", line 246, in anonymous fn/4 in Wsdataselect.Router.dispatch/2
File "/app/deps/telemetry/src/telemetry.erl", line 321, in :telemetry.span/3
File "deps/plug/lib/plug/router.ex", line 242, in Wsdataselect.Router.dispatch/2
...
(3 additional frame(s) were not displayed)
```https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/228Use wildcards in channel multiselect in channel form2024-03-22T14:04:59+01:00Simon PanayUse wildcards in channel multiselect in channel formhttps://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/225Find a way to ignore Errors within a chord2024-03-25T16:23:19+01:00Jonathan SchaefferFind a way to ignore Errors within a chordSee
* https://github.com/celery/celery/issues/8211
Fixed in celery 5.4 ?
* https://github.com/celery/celery/pull/8798
Sentry Issue: [WAVEQC-W](https://osug.sentry.io/issues/4730150930/?referrer=gitlab_integration)
```
ChordError: ...See
* https://github.com/celery/celery/issues/8211
Fixed in celery 5.4 ?
* https://github.com/celery/celery/pull/8798
Sentry Issue: [WAVEQC-W](https://osug.sentry.io/issues/4730150930/?referrer=gitlab_integration)
```
ChordError: Dependency 4ed627ec-7d38-482c-ba36-2e2e128d733a raised WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV) Job: 4596.')
File "celery/backends/redis.py", line 528, in on_chord_part_return
resl = [unpack(tup, decode) for tup in resl]
File "celery/backends/redis.py", line 528, in <listcomp>
resl = [unpack(tup, decode) for tup in resl]
File "celery/backends/redis.py", line 434, in _unpack_chord_result
raise ChordError(f'Dependency {tid} raised {retval!r}')
Chord "93a7605a-84b6-447c-81e2-97a07cf34d33" raised: "ChordError(\"Dependency 4ed627ec-7d38-482c-ba36-2e2e128d733a raised WorkerLostError('Worker exited prematurely: signal 11 (SIGSEGV) Job: 4596.')\")"
```WorkerLostError Messhttps://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/207Validated data : listen to integration queue in order to process validated data2024-03-27T17:43:47+01:00Simon PanayValidated data : listen to integration queue in order to process validated datahttps://docs.celeryq.dev/en/latest/userguide/extending.html#custom-message-consumers
https://wiki.osug.fr/!isterre-geodata/resif/systemes/services/integration_n%C5%93ud_b
* [x] Fix "Received and deleted unknown message. Wrong destinati...https://docs.celeryq.dev/en/latest/userguide/extending.html#custom-message-consumers
https://wiki.osug.fr/!isterre-geodata/resif/systemes/services/integration_n%C5%93ud_b
* [x] Fix "Received and deleted unknown message. Wrong destination?!" problem. See : https://github.com/celery/django-celery/issues/415#issuecomment-315296673
* [ ] switch to production rabbitmq-server (resif-mq) instead of the staging server
* [ ] think how to organize exchanges/vhosts/etc...
* [ ] Make an example consumer project with pika
* [ ] Write some doc here : https://wiki.osug.fr/!isterre-geodata/resif/systemes/services/rabbitmq
* [ ] Publish messages on the exchange at the end of post-integration
* [ ] Use settings to configure custom consumer ?https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect/-/issues/15%{message: {#Port<0.146814>, {:data, <<48, 48, 48, 48, 52, 57, 68, 32, 49, 48...2024-02-14T09:26:45+01:00Jonathan Schaeffer%{message: {#Port<0.146814>, {:data, <<48, 48, 48, 48, 52, 57, 68, 32, 49, 48, 50, 32, 32, 48, 48...Sentry Issue: [WSDATASELECT-1Z](https://osug.sentry.io/issues/4735228088/?referrer=gitlab_integration)
```
%{message: {#Port<0.146814>, {:data, <<48, 48, 48, 48, 52, 57, 68, 32, 49, 48, 50, 32, 32, 48, 48, 66, 72, 90, 89, 84, 7, 209, 0,...Sentry Issue: [WSDATASELECT-1Z](https://osug.sentry.io/issues/4735228088/?referrer=gitlab_integration)
```
%{message: {#Port<0.146814>, {:data, <<48, 48, 48, 48, 52, 57, 68, 32, 49, 48, 50, 32, 32, 48, 48, 66, 72, 90, 89, 84, 7, 209, 0, 239, 5, 49, 25, 0, 10, 105, 7, 98, 0, 250, 255, 248, 2, 0, 0, 2, 255, 254, 181, 241, 0, ...>>}}, module: Bandit.HTTP1.Handler, name: #PID<0.224281.0>}
```
Il faudrait voir côté Bandit comment gérer cette erreur proprement.https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/179Should we use an unprivileged user in docker file ?2024-02-12T17:42:54+01:00Simon PanayShould we use an unprivileged user in docker file ?see : https://testdriven.io/blog/docker-best-practices/#use-unprivileged-containerssee : https://testdriven.io/blog/docker-best-practices/#use-unprivileged-containershttps://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/174Warning displayed during response removal2024-02-15T14:59:03+01:00Simon PanayWarning displayed during response removal```plaintext
WARNING (norm_resp): computed and reported sensitivities differ by more than 5 percent | Execution continuing
``````plaintext
WARNING (norm_resp): computed and reported sensitivities differ by more than 5 percent | Execution continuing
```https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/172Handle exception raised when celery is still processing checks2024-02-09T14:30:59+01:00Jonathan SchaefferHandle exception raised when celery is still processing checksSentry Issue: [WAVEQC-18](https://osug.sentry.io/issues/4939185816/?referrer=gitlab_integration)
```
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected s...Sentry Issue: [WAVEQC-18](https://osug.sentry.io/issues/4939185816/?referrer=gitlab_integration)
```
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (1274,) + inhomogeneous part.
(11 additional frame(s) were not displayed)
...
File "waveqc/views.py", line 453, in network_full_detail
return self.context(query, label, pk, items)
File "waveqc/views.py", line 310, in context
results=self.build_z_values(query),
File "waveqc/views.py", line 199, in build_z_values
return np.array(
```https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/169Buildah : should we use local caching or gitlab caching ?2024-03-26T14:39:52+01:00Simon PanayBuildah : should we use local caching or gitlab caching ?In other terms, should we remove --cache-from/to arguments from buildah bud ? And so improve execution time of the pipelines ?
for buidlkit cache : here is a successful pipeline where its working : https://gricad-gitlab.univ-grenoble-al...In other terms, should we remove --cache-from/to arguments from buildah bud ? And so improve execution time of the pipelines ?
for buidlkit cache : here is a successful pipeline where its working : https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/jobs/615334https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/148When a new network is added to the list of checked networks, do not fill in g...2023-12-18T17:30:11+01:00Simon PanayWhen a new network is added to the list of checked networks, do not fill in gaps in checksThe current behaviour is to fill up the checks from the beginning of the use of waveqc.
But the retry_failing_checks routine will try to rerun those checks. It could be a massive amount of checks that could :
* take a few hours
* over...The current behaviour is to fill up the checks from the beginning of the use of waveqc.
But the retry_failing_checks routine will try to rerun those checks. It could be a massive amount of checks that could :
* take a few hours
* overlap between two retry_failing_checks routine that run every hour
* crash the server by filling up the disk
Find a way to solve this situationhttps://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/145Exclude closed channels from processing of completeness, shortest segments, e...2024-01-16T17:45:23+01:00Simon PanayExclude closed channels from processing of completeness, shortest segments, etc... in network detail viewFor STBU, VOEL, data reads as the following : completeness : 0% - traces : 1 - shortest segments : 0%
But if we go to the station detail view, data is 100% complete.
It seems that we include closed channels in the summary of the networ...For STBU, VOEL, data reads as the following : completeness : 0% - traces : 1 - shortest segments : 0%
But if we go to the station detail view, data is 100% complete.
It seems that we include closed channels in the summary of the network detail viewhttps://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/130Retrieve Triggered/Continuous information for each channel2023-12-07T16:04:11+01:00Simon PanayRetrieve Triggered/Continuous information for each channelA field is present in stationxml : https://docs.fdsn.org/projects/stationxml/en/latest/reference.html#channel-type
And retrievable with Obspy in "types" field : https://docs.obspy.org/packages/autogen/obspy.core.inventory.channel.Chann...A field is present in stationxml : https://docs.fdsn.org/projects/stationxml/en/latest/reference.html#channel-type
And retrievable with Obspy in "types" field : https://docs.obspy.org/packages/autogen/obspy.core.inventory.channel.Channel.html#obspy.core.inventory.channel.Channel
But this field is likely to be removed in the future.
Furthermore, we should store this information in waveqc database and display triggered station accordingly in all visualizations (heatmaps and homepage)https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect/-/issues/11Renvoyer "nodata" si la donnée récupérée est vide2023-12-04T10:56:53+01:00Jonathan SchaefferRenvoyer "nodata" si la donnée récupérée est videIl est possible que dataselect renvoie un fichier vide dans le cas où la requête tombe dans un trou de données.
Il faut gérer ce cas.Il est possible que dataselect renvoie un fichier vide dans le cas où la requête tombe dans un trou de données.
Il faut gérer ce cas.Jonathan SchaefferJonathan Schaefferhttps://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/waveqc/-/issues/118Cleanup use of datetime, UTCDateTime and pendulum2023-12-04T16:02:52+01:00Simon PanayCleanup use of datetime, UTCDateTime and pendulumhttps://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect/-/issues/9Plug.Conn.WrapperError: ** (DBConnection.ConnectionError) connection not avai...2023-11-27T13:48:25+01:00Jonathan SchaefferPlug.Conn.WrapperError: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue aft...Sentry Issue: [WSDATASELECT-4](https://osug.sentry.io/issues/4542797060/?referrer=gitlab_integration)
```
Plug.Conn.WrapperError: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1777ms...Sentry Issue: [WSDATASELECT-4](https://osug.sentry.io/issues/4542797060/?referrer=gitlab_integration)
```
Plug.Conn.WrapperError: ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1777ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
1. Ensuring your database is available and that you can connect to it
2. Tracking down slow queries and making sure they are running fast enough
3. Increasing the pool_size (although this increases resource consumption)
4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
See DBConnection.start_link/2 for more information
File "lib/ecto/adapters/sql.ex", line 1054, in Ecto.Adapters.SQL.raise_sql_call_error/1
File "lib/ecto/adapters/sql.ex", line 952, in Ecto.Adapters.SQL.execute/6
File "lib/ecto/repo/queryable.ex", line 232, in Ecto.Repo.Queryable.execute/4
File "lib/ecto/repo/queryable.ex", line 19, in Ecto.Repo.Queryable.all/3
File "lib/ecto/repo/queryable.ex", line 154, in Ecto.Repo.Queryable.one/3
...
(3 additional frame(s) were not displayed)
(Plug.Conn.WrapperError ** (DBConnection.ConnectionError) connection not available and request was dropped from queue after 1777ms. This means requests are coming in and your connection pool cannot serve them fast enough. You can address this by:
1. Ensuring your database is available and that you can connect to it
2. Tracking down slow queries and making sure they are running fast enough
3. Increasing the pool_size (although this increases resource consumption)
4. Allowing requests to wait longer by increasing :queue_target and :queue_interval
See DBConnection.start_link/2 for more information
)
```https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect/-/issues/4File.Error: could not list directory "/spool/988a21258f20/files.*": no such f...2023-10-18T16:45:07+02:00Jonathan SchaefferFile.Error: could not list directory "/spool/988a21258f20/files.*": no such file or directorySentry Issue: [WSDATASELECT-K](https://osug.sentry.io/issues/4552988555/?referrer=gitlab_integration)
```
File.Error: could not list directory "/spool/988a21258f20/files.*": no such file or directory
File "lib/file.ex", line 1654, in ...Sentry Issue: [WSDATASELECT-K](https://osug.sentry.io/issues/4552988555/?referrer=gitlab_integration)
```
File.Error: could not list directory "/spool/988a21258f20/files.*": no such file or directory
File "lib/file.ex", line 1654, in File.ls!/1
File "lib/wsdataselect/backend.ex", line 370, in Wsdataselect.Backend.send_statistics/3
File "lib/task/supervised.ex", line 101, in Task.Supervised.invoke_mfa/2
(File.Error could not list directory "/spool/988a21258f20/files.*": no such file or directory)
```https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect/-/issues/3Plug.Conn.WrapperError: ** (File.Error) could not make directory "/spool/7b44...2023-10-13T15:04:52+02:00Jonathan SchaefferPlug.Conn.WrapperError: ** (File.Error) could not make directory "/spool/7b44f2a3059c": read-only file systemSentry Issue: [WSDATASELECT-F](https://osug.sentry.io/issues/4545191812/?referrer=gitlab_integration)
```
Plug.Conn.WrapperError: ** (File.Error) could not make directory "/spool/7b44f2a3059c": read-only file system
File "lib/file.ex"...Sentry Issue: [WSDATASELECT-F](https://osug.sentry.io/issues/4545191812/?referrer=gitlab_integration)
```
Plug.Conn.WrapperError: ** (File.Error) could not make directory "/spool/7b44f2a3059c": read-only file system
File "lib/file.ex", line 255, in File.mkdir!/1
File "lib/wsdataselect/backend.ex", line 298, in Wsdataselect.Backend.dataselect/2
File "deps/plug/lib/plug/router.ex", line 246, in anonymous fn/4 in Wsdataselect.Router.dispatch/2
File "/app/deps/telemetry/src/telemetry.erl", line 321, in :telemetry.span/3
File "deps/plug/lib/plug/router.ex", line 242, in Wsdataselect.Router.dispatch/2
...
(3 additional frame(s) were not displayed)
(Plug.Conn.WrapperError ** (File.Error) could not make directory "/spool/7b44f2a3059c": read-only file system)
```