Commit 92fa23ba authored by Millian Poquet's avatar Millian Poquet
Browse files

Merge branch 'master' into coalloc_to_master

parents 92c53d9b f24c2b30
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
Upstream-Name: batsim
Upstream-Contact: Millian Poquet <millian.poquet@gmail.com>
Source: https://gitlab.inria.fr/batsim/batsim
Files: *
Copyright:
2015-2017, Olivier Richard <olivier.richard@imag.fr>
2015-2017, Millian Poquet <millian.poquet@inria.fr>
2015-2016, David Glesser <david.glesser@imag.fr>
2015-2017, Pierre-François Dutot <pfdutot@imag.fr>
2016-2017, Henri Casanova <henric@hawaii.edu>
2015-2017, Michael Mercier <michael.mercier@inria.fr>
2017 , Steffen Lackner <lackner@cs.tu-darmstadt.de>
2017 , Adrien Faure <adrien.faure2@gmail.com>
License: LGPL-3.0
Files: src/docopt/*
Copyright: 2013 Jared Grubb. All rights reserved.
License: Expat
Comment: See details in the source directory
Files: src/pugixml-1.7/*
Copyright: 2006-2017 Arseny Kapoulkine
License: Expat
Files: cmake/Modules/Findrapidjson.cmake
Copyright:
2011 Milo Yip (miloyip@gmail.com)
2013 Rafal Jeczalik (rjeczalik@gmail.com)
License: Expat
Batsim
======
Batsim is a Batch Scheduler Simulator. A Batch scheduler -- AKA Resources and
Jobs Management System (RJMS) -- is a system that manages resources in
large-scale computing centers, notably by scheduling and placing jobs, and by
setting up energy policies.
Batsim is a Batch scheduler simulator.
A batch scheduler -- AKA Resources and Jobs Management System (RJMS) --
is a system that manages resources in large-scale computing centers,
notably by scheduling and placing jobs, and by setting up energy policies.
Batsim is open source and distributed under LGPL-3.0 license.
See [COPYING](COPYING) for more details.
Batsim simulates the computing center behaviour. It is made such that any
event-based scheduling algorithm can be plugged to it. Thus, it permits to
compare decision algorithms coming from production and academics worlds.
![Batsim overview figure]
Here is an overview of how Batsim works in comparison with real executions.
![Batsim vs. real]
And here is a Batsim simulation overview from a different point of view.
![Batsim archi layer]
Batsim simulates a computing center behavior.
It is made such that any event-based scheduling algorithm can be plugged to it.
Thus, it allows to compare decision algorithms coming from production and
academics worlds.
Quick links
-----------
- The [contribute](doc/contribute.md) page tells how to contribute to Batsim
- The [changelog](doc/changelog.md) summarizes information about the project
evolution.
- Tutorials shows how to use Batsim and how it works:
- The [usage tutorial](doc/tuto_usage.md) explains how to execute a Batsim
simulation, and how to setup a development docker environment
......@@ -71,24 +70,26 @@ External References
* Batsim internal documentation can be found
[there](http://batsim.gforge.inria.fr/).
Build status
Build and code status
------------
| [master][master-link] | [upstream_sg (recent SimGrid)][upstream_sg-link] |
| :-------------------: | :----------------------------------------------: |
| ![master-badge] | ![upstream_sg-badge] |
| [master][master-link] | [upstream_sg (recent SimGrid)][upstream_sg-link] | [codacy][codacy-link] |
| :-------------------: | :----------------------------------------------: | :-------------------: |
| ![master-badge] | ![upstream_sg-badge] | ![codacy-badge] |
[master-badge]: https://gitlab.inria.fr/batsim/batsim/badges/master/build.svg "Gitlab CI build status (master branch)"
[master-link]: https://gitlab.inria.fr/batsim/batsim/pipelines "Gitlab CI build status"
[upstream_sg-badge]: https://gitlab.inria.fr/batsim/batsim/badges/upstream_sg/build.svg "Gitlab CI build status (upstream_sg branch)"
[upstream_sg-link]: https://gitlab.inria.fr/batsim/batsim/pipelines "Gitlab CI build status"
[codacy-badge]: https://api.codacy.com/project/badge/Grade/e5990f2e9abc4573b13a0b8c9d9e0f08 "Codacy code style"
[codacy-link]: https://www.codacy.com/app/mpoquet/batsim/dashboard
Batsim uses Gitlab CI as its continuous integration system.
Batsim uses Gitlab CI as its continuous integration system.
Build status of the different commits can be found
[there](https://gitlab.inria.fr/batsim/batsim/pipelines).
[there](https://gitlab.inria.fr/batsim/batsim/pipelines).
More information about our CI setup can be found
[there](./doc/continuous_integration.md)
[there](./doc/continuous_integration.md).
Development environment
-------------------------
......@@ -135,11 +136,15 @@ Also, some tools can be found in the [tools](./tools) directory:
Write your own scheduler (or adapt an existing one)
---------------------------------------------------
As Batsim is using a text-based protocol, your scheduler has to implement this
protocol: For more detail on the protocol, see [protocol description].
Schedulers must follow a text-based protocol to communicate with Batsim.
More details about the protocol can be found in the [protocol description].
A good starting point is Pybatsim which helps you to easily implement your
scheduling policy in Python. See the [pybatsim folder] for more details.
You may also base your work on existing Batsim-compatible schedulers:
- C++: [batsched][batsched gitlab]
- D: [datsched][datsched gitlab]
- Perl: [there][perl sched repo] (deprecated)
- Python: [pybatsim][pybatsim gitlab]
- Rust: [there][rust sched repo]
Installation
------------
......@@ -282,11 +287,15 @@ Other options:
If you want to run more complex scenarios, giving a look at our
[experiment tools](./tools/experiments) may save you some time!
[Batsim vs. real]: ./doc/batsim_overview.png
[Batsim archi layer]: ./doc/batsim_archi_layer.png
[Batsim overview figure]: ./doc/batsim_rjms_overview.png
[./publications/Batsim\_JSSPP\_2016.pdf]: ./publications/Batsim_JSSPP_2016.pdf
[Evalys]: https://github.com/oar-team/evalys
[Vite]: http://vite.gforge.inria.fr/
[protocol description]: ./doc/proto_description.md
[pybatsim folder]: ./schedulers/pybatsim/
[oar3]: https://github.com/oar-team/oar3
[pybatsim gitlab]: https://gitlab.inria.fr/batsim/pybatsim
[batsched gitlab]: https://gitlab.inria.fr/batsim/batsched
[datsched gitlab]: https://gitlab.inria.fr/batsim/datsched
[rust sched repo]: https://gitlab.inria.fr/adfaure/schedulers
[perl sched repo]: https://github.com/fernandodeperto/batch-simulator
* TODO test job rejection
* TODO test shared nodes
* TODO file I/O
* TODO evolving/moldable/malleable job support
* TODO VM
* TODO support for job creation during simulation
* TODO suspend/resume commands
* TODO kill job command
* TODO node sleep/wakeup
* DONE job rejection
* DONE allow the scheduler to ask when he wants to be NOPped
* DONE energy
* DONE smpi profile
* DONE msg_par_homogenous profile
* DONE job composed of sequence of profile [1, 5, 5]
* DONE delay job profile
* DONE profile composed
* DONE (VITE ISSUE) improve the Pajé trace generation (there are warnings when the trace are opened via vite)
Before editing any file, here is a small reminder of things missing from the Readme.
The cmake arguments I needed to compile batsim:
cmake -DREDOX_LIBRARY=/usr/local/lib64/libredox.so -DBoost_FILESYSTEM_LIBRARY_DEBUG=/usr/lib/x86_64-linux-gnu/libboost_filesystem.so.1.58.0 -DBoost_SYSTEM_LIBRARY_DEBUG=/usr/lib/x86_64-linux-gnu/libboost_system.so.1.58.0 -DSIMGRID_INCLUDE_DIR=/home/pfdutot/stow/include/ -DSIMGRID_LIBRARY=/home/pfdutot/stow/lib/libsimgrid.so ..
The installs I made:
sudo apt-get install libhiredis-dev
tar -xzvf redis-3.2.3.tar.gz
unzip redox-master.zip
sudo apt-get install libev-dev
sudo apt-get install python-sortedcontainers
sudo pip install redis
(the following are more related to evalys)
sudo apt-get install libfreetype6-dev
sudo apt-get install libpng-dev
sudo -H pip install git+https://github.com/oar-team/evalys.git
sudo pip install tox
sudo pip install pytest
sudo apt-get install python3-pip
sudo apt-get install libffi-dev
sudo /usr/bin/python3 -m pip install matplotlib
sudo /usr/bin/python3 -m pip install cairocffi
Current invocation in data-storage branch:
redis-server
build/batsim -p platforms/small_platform.xml -w workload_profiles/test_workload_profile.json
python2 schedulers/pybatsim/launcher.py fillerSched workload_profiles/test_workload_profile.json
The hack invocation:
In build/
./batsim -W ../workload_profiles/GENOME.d.351024866.5.dax -p ../platforms/small_platform.xml -w ../workload_profiles/test_workload_profile.json
In schedulers/pybatsim/
python2 launcher.py fillerSched ../../workload_profiles/test_workload_profile_hack.json
And then:
evalys --gantt --output /tmp/g.pdf out_jobs.csv
This diff is collapsed.
This diff is collapsed.
# Changelog
All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog][changelog].
Batsim adheres to [Semantic Versioning][semver] from version 1.0.0.
Batsim's public API includes:
- The Batsim's command-line interface.
- The Batsim input files format.
- The communication protocol with the decision-making component.
[//]: ==========================================================================
## [Unreleased]
[//]: ==========================================================================
## [1.0.0] - 2017-09-09
### SimGrid version to use:
- Commit ``587483ebe`` on ``https://github.com/mpoquet/simgrid.git``.
Please notice that energy consumption of parallel tasks does not work
as expected.
### Added
- Stated LGPL-3.0 license.
- Code cosmetics standards are now checked by Codacy.
- New PFS host. Associated with a new ``hpst-host`` command-line option.
- New protocol messages:
- The ``CHANGE_JOB_STATE`` allows the scheduler to change the state of jobs
in Batsim in-memory data structures.
- The ``submission_finished`` notification can be cancelled with a
``continue_submission`` notification.
- New data in existing protocol messages:
- ``SIMULATION_BEGINS``:
- ``allow_time_sharing`` boolean is now forwarded.
- ``resources_data`` gives information on the resources.
- ``hpst_host`` and ``lcst_host give information about the
parallel file system.
- ``JOB_COMPLETED``:
- ``job_state`` contains the job state (as stored by Batsim).
- ``kill_reason`` contains why the job has been killed (if relevant).
- New tests:
- ``demo`` corresponds to the documented demonstration.
- ``energy_minimal`` and ``energy_small`` test the energy plugin in detail.
- ``kill_multiple`` checks that Batsim can handle multiple kills concerning
the same jobs.
- ``fewer_resources`` checks that complex job mapping work
- ``reject`` tests whether rejection works
### Modified
- Improved and renamed parallel file system profiles.
- Improved code documentation.
- Improved the python scripts of the tools/ directory.
- Improved the python scripts of the test/ directory.
### Fixed
- Complex allocation mapping were not handled correctly
[//]: ==========================================================================
## 0.99 - 2017-05-26
### Changed
- The protocol is based on ZeroMQ instead of Unix Domain Sockets.
- The protocol messages are now JSON-formatted
[//]: ==========================================================================
[changelog]: http://keepachangelog.com/en/1.0.0/
[semver]: http://semver.org/spec/v2.0.0.html
[Unreleased]: https://github.com/oar-team/batsim/compare/v1.0.0...HEAD
[1.0.0]: https://github.com/oar-team/batsim/compare/v0.99...v1.0.0
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment