proto_description.md 10.8 KB
Newer Older
1
# Introduction
Millian Poquet's avatar
Millian Poquet committed
2
3
4
5
A Batsim simulation consists in two processes:
- Batsim itself, in charge of simulating what happens on the platform
- A *Decision Process* (or more simply scheduler), in charge of making decisions

6
7
8
9
10
The two processes communicate via a socket with the protocol explained in
the present document.  The protocol is synchronous and follows a simple
request-reply pattern.  Whenever an event which may require making decision
occurs in Batsim in the simulation, the following steps occur:

Millian Poquet's avatar
Millian Poquet committed
11
1. Batsim suspends the simulation
12
13
2. Batsim sends a request to the scheduler (telling it what happened on the
   platform)
14
3. Batsim waits for a reply from the scheduler
Millian Poquet's avatar
Millian Poquet committed
15
4. Batsim receives the reply
16
17
5. Batsim resumes the simulation, applying the decision which have been
   made
Millian Poquet's avatar
Millian Poquet committed
18

19
20
![protocol_overview_figure](protocol_img/request_reply.png)

21
22
ZeroMQ is used in both processes (Batsim uses a ZMQ REQ socket, the
scheduler a ZMQ REP one).
23

24
25
26
This protocol is used for synchronization purpose. Metadata associated to the
jobs are shared via Redis, as described [here](data_storage_description.md)

27
28
29
30
31
32
# Message Composition

It is a JSON object that looks like this:

```json
{
33
  "now": 1024.24,
34
35
36
  "events": [
    {
      "timestamp": 1000,
37
      "type": "EXECUTE_JOB",
38
39
40
41
42
43
44
      "data": {
        "job_id": "workload!job_1234",
        "alloc": "1 2 4-8",
      }
    },
    {
      "timestamp": 1012,
45
      "type": "EXECUTE_JOB",
46
47
48
49
50
51
52
53
54
55
      "data": {
        "job_id": "workload!job_1235",
        "alloc": "12-100",
      }
    }
  ]
}

```

Millian Poquet's avatar
Millian Poquet committed
56
The ``now`` field defines the current simulation time.
57
58
59
60
61
62
- If the message comes from Batsim, it means that the scheduler cannot make
  decisions before ```now``` (time travel is forbidden)
- If the message comes from the scheduler, it tells Batsim that the
  scheduler finished making its decisions at timestamp ```now```. It is
  used by Batsim to know when the scheduler will be available for making
  new decisions.
63
64
65
66


## Constraints

Millian Poquet's avatar
Millian Poquet committed
67
Constraints on the message format are defined here:
68

69
70
71
72
- the message timestamp ``now`` MUST be greater than or equal to every
  event ``timestamp``
- events timestamps MUST be in ascending order: event[i].timestamp <=
  event[i+1].timestamp
73
- mandatory fields:
74
75
76
77
78
79
80
    - ``now`` (type: float)
    - ``events``: (type array (can be empty))
        - ``timestamp`` (type: float)
        - ``type`` (type: string as defined below)
        - ``data`` (type: dict (can be empty))

---
81

Millian Poquet's avatar
Millian Poquet committed
82
## Bidirectional events
83

84
85
These events can be sent from Batsim to the scheduler, or in the opposite
direction.
86
87
88
89
90
91
```
BATSIM <---> DECISION
```

### NOP

92
The simplest message, stands either for: "nothing happened" if sent by
93
94
Batsim, or "do nothing" if sent by the scheduler. It means that the
events list is empty: ``"events": []``
Millian Poquet's avatar
Millian Poquet committed
95

96
- **data**: N/A
Millian Poquet's avatar
Millian Poquet committed
97
- **full message example**:
98
```json
Millian Poquet's avatar
Millian Poquet committed
99
100
101
102
{
  "now": 1024.24,
  "events": []
}
103
```
104
105
106

---

Millian Poquet's avatar
Millian Poquet committed
107
## Batsim to Scheduler events
108

Millian Poquet's avatar
Millian Poquet committed
109
These events are sent by Batsim to the scheduler.
110
111
112
113
```
BATSIM ---> DECISION
```

Michael Mercier's avatar
Michael Mercier committed
114
### SIMULATION_BEGINS
115
Sent at the beginning of the simulation. Once it has been sent,
Michael Mercier's avatar
Michael Mercier committed
116
and if redis is enabled, meta-information can be read from Redis.
Millian Poquet's avatar
Millian Poquet committed
117

118
119
120
121
122
Batsim configuration is sent through the ``config`` object (in ``data``).
Any custom information can be added into the
[Batsim configuration](./configuration.md), which gives a generic way to give
metainformation from Batsim to any scheduler at runtime.

Michael Mercier's avatar
Michael Mercier committed
123
- **data**: the number of resources
Millian Poquet's avatar
Millian Poquet committed
124
125
126
- **example**:
```json
{
127
  "timestamp": 0.0,
Michael Mercier's avatar
Michael Mercier committed
128
  "type": "SIMULATION_BEGINS",
Michael Mercier's avatar
Michael Mercier committed
129
  "data": {
130
131
    "nb_resources": 60,
    "config":{}
Michael Mercier's avatar
Michael Mercier committed
132
  }
Millian Poquet's avatar
Millian Poquet committed
133
134
135
136
137
138
139
140
141
142
}
```

### SIMULATION_ENDS
Sent once all jobs have been submitted and have completed.

- **data**: empty
- **example**:
```json
{
143
  "timestamp": 100.0,
Millian Poquet's avatar
Millian Poquet committed
144
145
146
147
148
  "type": "SIMULATION_ENDS",
  "data": {}
}
```

149
150
### JOB_SUBMITTED

151
152
153
Some jobs have been submitted within Batsim. It is sent whenever a job
coming from Batsim inputs (workloads and workflows) are submitted. It is
also sent as a reply to a ```SUBMIT_JOB``` message if and only if an
154
acknowledgement has been requested. Without Redis enabled the job
155
description and optionnaly the profile are also transmitted.
156
157

- **data**: list of job id
158
- **example with redis**:
159
160
161
162
163
164
165
```json
{
  "timestamp": 10.0,
  "type": "JOB_SUBMITTED",
  "data": {"job_id": "w0!1"}
}
```
166
- **example without redis**:
167
168
```json
{
169
  "timestamp": 10.0,
170
171
  "type": "JOB_SUBMITTED",
  "data": {
172
173
174
175
176
177
178
179
180
181
182
    "job_id": "dyn!my_new_job",
    "job": {
      "profile": "delay_10s",
      "res": 1,
      "id": "my_new_job",
      "walltime": 12.0
    },
    "profile":{
      "type": "delay",
      "delay": 10
    }
183
184
}
```
RICHARD Olivier's avatar
RICHARD Olivier committed
185

186
187
### JOB_COMPLETED

188
189
190
A job has completed its execution. It acknowledges that the actions coming
from a previous ```EXECUTE_JOB``` message have been done (successfully or
not, depending on whether the job completed without reaching timeout).
191
192
193
194
195

- **data**: a job id string with a status string (TIMEOUT, SUCCESS)
- **example**:
```json
{
196
  "timestamp": 10.0,
197
198
199
200
201
  "type": "JOB_COMPLETED",
  "data": {"job_id": "w0!1", "status": "SUCCESS"}
}
```

Millian Poquet's avatar
Millian Poquet committed
202
203
### JOB_KILLED

204
205
Some jobs have been killed. It acknowledges that the actions coming from a
previous ```KILL_JOB``` message have been done.
Millian Poquet's avatar
Millian Poquet committed
206
207
208
209
210

- **data**: A list of job ids
- **example**:
```json
{
211
  "timestamp": 10.0,
Millian Poquet's avatar
Millian Poquet committed
212
213
214
215
216
  "type": "JOB_KILLED",
  "data": {"job_ids": ["w0!1", "w0!2"]}
}
```

217
218
### RESOURCE_STATE_CHANGED

219
220
The state of some resources has changed. It acknowledges that the actions
coming from a previous ```SET_RESOURCE_STATE``` message have been done.
221
222
223
224
225

- **data**: an interval set of resource id and the new state
- **example**:
```json
{
226
  "timestamp": 10.0,
227
228
229
230
231
232
233
  "type": "RESOURCE_STATE_CHANGED",
  "data": {"resources": "1 2 3-5", "state": "42"}
}
```

### QUERY_REPLY

234
235
236
This is a reply to a ``QUERY_REQUEST`` message. It depends on the
configuration of Batsim: if ``"redis": { "enabled": true }`` the reply will
go in redis and only the key will be given. Otherwise, the response will be
Millian Poquet's avatar
Millian Poquet committed
237
238
put directly in the message.
See [Configuration documentation](./configuration.md) for more details.
239

240

Millian Poquet's avatar
Millian Poquet committed
241
- **data**: See [QUERY_REQUEST](#query_request) documentation
242
243
244
- **example**:
```json
{
245
  "timestamp": 10.0,
246
247
248
249
  "type": "QUERY_REPLY",
  "data": {"redis_keys": "/my/key/path0" }
}
```
250
251
252
253
254
or
```json
{
  "timestamp": 10.0,
  "type": "QUERY_REPLY",
Michael Mercier's avatar
Michael Mercier committed
255
  "data": {"consumed_energy": "12500" }
256
257
258
}
```

259
260
261
262
263
264
265
266
267
268
269
270
271
### REQUESTED_CALL
This message is a response to the [CALL_ME_LATER](#call_me_later) message.

- **data**: empty
- **example**:
```json
{
  "timestamp": 25.5,
  "type": "REQUESTED_CALL",
  "data": {}
}
```

272
---
273

Millian Poquet's avatar
Millian Poquet committed
274
275
## Scheduler to Batsim events
These events are sent by the scheduler to Batsim.
276
277
278
279
280
281
282
```
BATSIM <--- DECISION
```

### QUERY_REQUEST

This is a query sent to Batsim to get information about the simulation
283
state (or whatever you want to know...). The supported requests are:
Michael Mercier's avatar
Michael Mercier committed
284
- "consumed_energy" with no argument that asks Batsim about the total
285
286
  consumed energy (from time 0 to now) in Joules. Works only in energy
  mode.
287

288
- **data**: a dictionnary of requests.
289
290
291
- **example**:
```json
{
292
  "timestamp": 10.0,
293
294
  "type": "QUERY_REQUEST",
  "data": {
Michael Mercier's avatar
Michael Mercier committed
295
    "requests": {"consumed_energy": {}}
296
297
298
299
300
301
302
303
304
305
306
307
  }
}
```

### REJECT_JOB

Reject a job that was submitted before.

- **data**: A job id
- **example**:
```json
{
308
  "timestamp": 10.0,
309
310
311
312
313
314
315
316
317
  "type": "REJECT_JOB",
  "data": { "job_id": "w12!45" }
}
```

### EXECUTE_JOB

Execute a job on a given set of resources. An optional mapping can be
added to tell Batsim how to map executors to resources: where the
Millian Poquet's avatar
Millian Poquet committed
318
executors will be placed inside the allocation (resource numbers are shifted
319
320
321
to 0). It only works for SMPI for now.

The following example overrides the default round robin mapping to put the
Millian Poquet's avatar
Millian Poquet committed
322
323
first two ranks (0 and 1) on the first allocated machine (0, which stands for
resource id 2), and the last two ranks (2 and 3) on the second machine (1, which
324
325
326
327
328
329
stands for resource id 3).

- **data**: A job id, an allocation and a mapping (optional)
- **example**:
```json
{
330
  "timestamp": 10.0,
331
332
333
334
  "type": "EXECUTE_JOB",
  "data": {
    "job_id": "w12!45",
    "alloc": "2-3",
MERCIER Michael's avatar
MERCIER Michael committed
335
    "mapping": {"0": "0", "1": "0", "2": "1", "3": "1"}
336
337
338
339
340
341
  }
}
```

### CALL_ME_LATER

Millian Poquet's avatar
Millian Poquet committed
342
Asks Batsim to call the scheduler later on, at a given timestamp.
343
344
345
346
347

- **data**: future timestamp float
- **example**:
```json
{
348
  "timestamp": 10.0,
349
350
351
352
353
354
355
  "type": "CALL_ME_LATER",
  "data": {"timestamp": 25.5}
}
```


### KILL_JOB
Millian Poquet's avatar
Millian Poquet committed
356
357
358
359
360
361
Kills some jobs (almost instantaneously).

- **data**: A list of job ids
- **example**:
```json
{
362
  "timestamp": 10.0,
Millian Poquet's avatar
Millian Poquet committed
363
364
365
366
  "type": "KILL_JOB",
  "data": {"job_ids": ["w0!1", "w0!2"]}
}
```
367
368
369

### SUBMIT_JOB

370
371
Submits a job (from the scheduler). The submission is acknowledged by
default. See [Configuration documentation](./configuration.md) for more
372
373
374
375
details.

- **data**: A job id (job id duplication is forbidden), classical job and
  profile information (optional).
Millian Poquet's avatar
Millian Poquet committed
376

377
- **example with redis** : the job description, and the profile description if
378
379
  it unknown to Batsim yet, must have been pushed into redis by the
  scheduler before sending this message
Millian Poquet's avatar
Millian Poquet committed
380
381
```json
{
382
  "timestamp": 10.0,
Millian Poquet's avatar
Millian Poquet committed
383
384
385
386
387
388
  "type": "SUBMIT_JOB",
  "data": {
    "job_id": "w12!45",
  }
}
```
389
- **example without redis** : the whole job description goes through the protocol.
Millian Poquet's avatar
Millian Poquet committed
390
391
```json
{
392
  "timestamp": 10.0,
Millian Poquet's avatar
Millian Poquet committed
393
394
395
  "type": "SUBMIT_JOB",
  "data": {
    "job_id": "dyn!my_new_job",
396
    "job":{
Millian Poquet's avatar
Millian Poquet committed
397
398
399
400
401
      "profile": "delay_10s",
      "res": 1,
      "id": "my_new_job",
      "walltime": 12.0
    },
402
    "profile":{
Millian Poquet's avatar
Millian Poquet committed
403
404
405
406
407
408
409
      "type": "delay",
      "delay": 10
    }
  }
}
```

410
### SET_RESOURCE_STATE
Millian Poquet's avatar
Millian Poquet committed
411
412
413
414
415
416
417

Sets some resources into a state.

- **data**: an interval set of resource id, and the new state
- **example**:
```json
{
418
  "timestamp": 10.0,
Millian Poquet's avatar
Millian Poquet committed
419
420
421
422
  "type": "SET_RESOURCE_STATE",
  "data": {"resources": "1 2 3-5", "state": "42"}
}
```
Millian Poquet's avatar
Millian Poquet committed
423

424
### NOTIFY
425
426
427
The scheduler notify Batsim of something. For example, that job submission
from the scheduler is over, so Batsim is able to stop the simulation.  This
message **must** be sent if ``"scheduler_submission": {"enabled": false}``
428
is configured. See [Configuration documentation](./configuration.md) for more
429
details.
Millian Poquet's avatar
Millian Poquet committed
430
431
432
433
434

- **data**: empty
- **example**:
```json
{
435
436
437
  "timestamp": 42.0,
  "type": "NOTIFY",
  "data": { "type": "submission_finished" }
Millian Poquet's avatar
Millian Poquet committed
438
439
}
```
440
441
442
443
444

# Use cases
The way to do some operations with the protocol is shown in this section.

## Executing jobs
Millian Poquet's avatar
Millian Poquet committed
445
446
Depending on the [configuration](./configuration.md), job information might
either be transmitted through the protocol or Redis.
447
448
449
450
451
![executing_jobs_figure](protocol_img/job_submission_and_execution.png)

## Dynamic submission of jobs
Jobs are in most cases submitted within Batsim. It is however possible to submit
jobs from the scheduler. The following two figures described how it should be
Millian Poquet's avatar
Millian Poquet committed
452
453
454
455
done, depending on whether Redis is enabled in the
[configuration](./configuration.md). The acknowledgement of dynamic job
submissions can be enabled or disabled in the
[configuration](./configuration.md).
456
457
458
459
460
461

### Without Redis
![dynamic_submission](protocol_img/dynamic_job_submission.png)

### With Redis
![dynamic_submission_redis](protocol_img/dynamic_job_submission_redis.png)