Commit 9243533b authored by Millian Poquet's avatar Millian Poquet
Browse files

Merge branch 'master' of gitlab.inria.fr:batsim/batsim

parents 43c55b07 3dfddcdb
......@@ -14,6 +14,36 @@ Here is an overview of how Batsim works compared to real executions.
![Batsim vs. real]
Run batsim example
------------------
*Important note*: It is highly recommended to use batsim with the provided
container. It use really up-to-date version of some packages (like boost)
that is not available on classic distribution yet.
To test simply test batsim you can directly run it though docker. First run
batsim in your container for a simple workload:
```bash
# launch a batsim container
docker run -ti --name batsim oarteam/batsim bash
# inside the container
cd /root/batsim
redis-server &
batsim -p platforms/small_platform.xml \
-w workload_profiles/test_workload_profile.json
```
Then in an *other terminal* execute the scheduler:
```bash
# Run an other bash in the same container
docker exec -ti batsim bash
# inside the container
cd /root/batsim
python2 schedulers/pybatsim/launcher.py fillerSched
```
External References
-------------------
* Batsim scientific publication pre-print is available on HAL:
......@@ -58,15 +88,6 @@ protocol: For more detail on the protocol, see [protocol description].
A good starting point is Pybatsim which helps you to easily implement your
scheduling policy in Python. See the [pybatsim folder] for more details.
Try it directly with Docker
----------------------------
You can use the docker container to try it without installation:
``` bash
docker run -ti oarteam/batsim bash
batsim
```
Installation
------------
......@@ -109,7 +130,7 @@ sudo make install
```
Batsim Use Cases
------
----------------
Simulating with Batsim involves at least two processes:
- Batsim itself
......@@ -122,46 +143,79 @@ simple experiments with Batsim.
Batsim usage can be shown by calling the Batsim program with the ``--help``
option. It should display something like this:
```
Usage: batsim [OPTION...] PLATFORM_FILE WORKLOAD_FILE
batsim --help
A tool to simulate (via SimGrid) the behaviour of scheduling algorithms.
-e, --export=FILENAME_PREFIX The export filename prefix used to generate
simulation output. Default value: 'out'
-h, --allow-space-sharing Allows space sharing: the same resource can
compute several jobs at the same time
-l, --limit-machine-count=M Allows to limit the number of computing
machines to use. If M == -1 (default), all the
machines described in PLATFORM_FILE are used (but
the master_host). If M >= 1, only the first M
machines will be used to comupte jobs.
-L, --limit-machine-count-by-worload
If set, allows to limit the number of computing
machines to use. This number is read from the
workload file. If both limit-machine-count and
limit-machine-count-by-worload are set, the
minimum of the two will be used.
-m, --master-host=NAME The name of the host in PLATFORM_FILE which will
run SimGrid scheduling processes and won't be used
to compute tasks. Default value: 'master_host'
-p, --energy-plugin Enables energy-aware experiments
-q, --quiet Shortcut for --verbosity=quiet
-s, --socket=FILENAME Unix Domain Socket filename. Default value:
'/tmp/bat_socket'
-t, --process-tracing Enables SimGrid process tracing (shortcut for
SimGrid options ----cfg=tracing:1
--cfg=tracing/msg/process:1)
-T, --disable-schedule-tracing If set, the tracing of the schedule is
disabled (the output file _schedule.trace will not
be exported)
-v, --verbosity=VERBOSITY_LEVEL
Sets the Batsim verbosity level. Available values
are : quiet, network-only, information (default),
debug.
-?, --help Give this help list
--usage Give a short usage message
Mandatory or optional arguments to long options are also mandatory or optional
for any corresponding short options.
Usage:
batsim -p <platform_file> [-w <workload_file>...]
[-W <workflow_file>...]
[--WS (<cut_workflow_file> <start_time>)...]
[options]
batsim --help
Input options:
-p --platform <platform_file> The SimGrid platform to simulate.
-w --workload <workload_file> The workload JSON files to simulate.
-W --workflow <workflow_file> The workflow XML files to simulate.
--WS --workflow-start (<cut_workflow_file> <start_time>)... The workflow XML
files to simulate, with the time at which
they should be started.
Most common options:
-m, --master-host <name> The name of the host in <platform_file>
which will be used as the RJMS management
host (thus NOT used to compute jobs)
[default: master_host].
-E --energy Enables the SimGrid energy plugin and
outputs energy-related files.
Execution context options:
-s, --socket <socket_file> The Unix Domain Socket filename
[default: /tmp/bat_socket].
--redis-hostname <redis_host> The Redis server hostname
[default: 127.0.0.1]
--redis-port <redis_port> The Redis server port [default: 6379].
Output options:
-e, --export <prefix> The export filename prefix used to generate
simulation output [default: out].
--enable-sg-process-tracing Enables SimGrid process tracing
--disable-schedule-tracing Disables the Pajé schedule outputting.
--disable-machine-state-tracing Disables the machine state outputting.
Platform size limit options:
--mmax <nb> Limits the number of machines to <nb>.
0 means no limit [default: 0].
--mmax-workload If set, limits the number of machines to
the 'nb_res' field of the input workloads.
If several workloads are used, the maximum
of these fields is kept.
Verbosity options:
-v, --verbosity <verbosity_level> Sets the Batsim verbosity level. Available
values: quiet, network-only, information,
debug [default: information].
-q, --quiet Shortcut for --verbosity quiet
Workflow options:
--workflow-jobs-limit <job_limit> Limits the number of possible concurrent
jobs for workflows. 0 means no limit
[default: 0].
--ignore-beyond-last-workflow Ignores workload jobs that occur after all
workflows have completed.
Other options:
--allow-time-sharing Allows time sharing: One resource may
compute several jobs at the same time.
--batexec If set, the jobs in the workloads are
computed one by one, one after the other,
without scheduler nor Redis.
--pfs-host <pfs_host> The name of the host, in <platform_file>,
which will be the parallel filesystem target
as data sink/source [default: pfs_host].
-h --help Shows this help.
--version Shows Batsim version.
```
#### Display information control
......@@ -171,125 +225,15 @@ The end-of-options delimiter "--" must then be used to use SimGrid options.
``` bash
# Display Batsim options/help
batsim PLATFORM WORKLOAD --help
batsim --help
```
``` bash
# Display SimGrid options/help
batsim PLATFORM WORKLOAD -- --help
```
``` bash
# Quiet run
batsim [batsim_options...] PLATFORM WORKLOAD -q
```
``` bash
# Display network information only
batsim [batsim_options...] PLATFORM WORKLOAD -vnetwork-only
```
``` bash
# Display debug information
batsim [batsim_options...] PLATFORM WORKLOAD -vdebug
```
``` bash
# Generate SimGrid processes' trace (can be useful to visualize what happens)
batsim [batsim_options...] PLATFORM WORKLOAD -t
```
### Simulating with Batsim
#### Simple Case
This case exhibits Batsim's default behaviour
##### Running Batsim
``` bash
batsim platforms/small_platform.xml \
workload_profiles/test_workload_profile.json
```
##### Running the (Stupid) Scheduler
``` bash
python2 schedulers/pybatsim/launcher.py fillerSched \
workload_profiles/test_workload_profile.json
```
#### Specifying Batsim Output and Socket
This case shows how to specify where Batsim should put its output file and which
socket to use, which can be useful to simulate different instances in parallel
##### Preparing the launch
``` bash
mkdir -p /tmp/batsim_output/
```
##### Running Batsim
``` bash
batsim platforms/small_platform.xml \
workload_profiles/test_workload_profile.json \
-e /tmp/batsim_output/out \
-s /tmp/batsim_output/socket
```
##### Running the (Stupid) scheduler
``` bash
python2 schedulers/pybatsim/launcher.py fillerSched \
workload_profiles/test_workload_profile.json \
-s /tmp/batsim_output/socket
```
#### Running Energy-Aware experiments
This case shows how to enable energy-aware experiments in Batsim, which allows:
- to know how much energy the system has been using over time
- to change the resources' power states (shutdown/DVFS) at simulation time
##### Running Batsim
``` bash
batsim platforms/small_platform.xml \
workload_profiles/test_workload_profile.json \
-e /tmp/batsim_output/out \
-s /tmp/batsim_output/socket \
-p
```
##### Running the (Stupid) Scheduler
``` bash
python2 schedulers/pybatsim/launcher.py fillerSched \
workload_profiles/test_workload_profile.json \
-s /tmp/batsim_output/socket
```
#### Executing Batsim with OAR
This case shows how to execute Batsim with the real RJMS OAR.
Refer to [oar3] site or bataar.py -h for more information.
##### Installing OAR
``` bash
git clone https://github.com/oar-team/oar3.git
cd oar3
sudo python setup.py install
```
##### Running Batsim
``` bash
batsim platforms/small_platform.xml \
workload_profiles/test_workload_profile.json \
-e /tmp/batsim_output/out \
-s /tmp/batsim_output/socket
```
##### Running the OAR (Kamelot) Scheduler
The scheduler is run via **bataar.py**, which is the OAR3 adaptor to interact
with Batsim.
``` bash
bataar workload_profiles/test_workload_profile.json \
-s /tmp/batsim_output/socket
batsim -- --help
```
#### Executing experiments
#### Executing complete experiments
If you want to run more complex scenarios, giving a look at our
[experiment tools](./tools/experiments) may save you some time!
......
:date: 2015-12-01
:author: Michael Mercier <michael_mercier@inria.fr>
Batsim environment recipes
==========================
This is a Kameleon recipe for Batsim experiments. It provides a complete
appliance with Batsim and Simgrid as simulator and with several scheduler
that plugs with it:
* the Perl scheduler from F.Wagner and the OAR scheduler
* the OAR python scheduler
It also provide a script to run the experiment and reproduce the results.
Usage
-----
First, you need to install Kameleon:
http://kameleon.imag.fr/installation.html
This is a Kameleon recipe for Batsim experiments. It provides several
recipes to build Docker images (can also be configured for other format
like VDI or QCOW2) to build, test and run Batsim.
Then, you have to open the recipe ``debian8_batsim.yaml`` with your favorite
text editor and chose the isolation mechanism that suits your environment.
To do so, just comment and uncomment the ``extend`` lines. You can also
look at the other global variables if you want to customize your appliance.
- [batsim_ci](batsim_ci.yaml) contains all the dependencies to build
batsim install it and run tests. This recipe is updated each time
dependencies changes. As it is used by the Continuous integration system,
it is always up-to-date. It is accessible on Docker Hub at
[oarteam/batsimi_ci](https://hub.docker.com/r/oarteam/batsim_ci/).
.. note:: You will need to install some specific tools depending on the
method you chose.
- [batsim](batsim.yaml) contains a complete appliance with batsim installed
and tested. It is accessible on Docker Hub at
[oarteam/batsim](https://hub.docker.com/r/oarteam/batsim/).
You can now run Kameleon::
Build images
------------
kameleon build debian8_batsim.yaml
It is recommended to run these images directly from docker, but you can build
them from scratch if necessary
You can find the generated images in ``build/debian8_batsim/``.
Run the image
-------------
You can use either VirtualBox or KVM or just a chroot to use this image
directly from the exported images that you can find in
``build/debian8_batsim/``.
First, you need to install Kameleon:
[http://kameleon.imag.fr/installation.html](http://kameleon.imag.fr/installation.html)
With Docker
~~~~~~~~~~~
You can now run Kameleon, for example:
```bash
kameleon build batsim.yaml
```
You can find the generated images in ``build/batsim/``.
If you want to use Docker, you can import and run the image using these
commands::
Run the image with docker
-------------------------
docker import build/debian8_batsim/debian8_batsim.tar.gz simctn
docker run -ti \
--env LANG=en_US.UTF-8 \
--volume $SSH_AUTH_SOCK:/ssh-agent --env SSH_AUTH_SOCK=/ssh-agent \
-p 8888:8888 \
simctn bash
To run the image with Docker, you have to import and run the image using these
commands:
```bash
docker import build/batsim/batsim.tar.gz my_batsim
docker run -ti -p 8888:8888 my_batsim bash
```
Now you should be in the container.
With Qemu/KVM
~~~~~~~~~~~~~
You can use virt-manager to have a nice GUI like VirtualBox or launch it by
and with this command::
sudo qemu-system-x86_64 -m 1024 --enable-kvm \
-redir tcp:2222::22 \
-redir tcp:8888::8888 \
build/debian8_batsim/debian8_batsim.qcow2
Then ssh into it::
ssh -A root@localhost -p2222
# password: root
Run the experiment
------------------
Login into the image. Default credential is **login: root** and **password:
root**. Now you can run the experiment like this::
cd /root/expe-batsim
./experiment.py
You can modify the experimentation parameters directly inside the
``experiment.py`` script.
Results
-------
To check the results of the experiment you can use ipython notebook (now
jupyther). To do so, run the server and connect to it via your local
browser::
ipython3 notebook --no-browser --ip 0.0.0.0
Development
-----------
......@@ -107,8 +57,7 @@ remote clones for any modified tools of the chain to save your work. In
order to push your work on a remote repository, SSH credentials need to be
imported using SSH forward Agent or Key import from your home. For docker
container the ssh-agent socket (SSH_AUTH_SOCK) can be mounted as volume and
exported in the environment. Note that the examples below already do this
for you.
exported in the environment.
References
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment