Commit 52fb825b authored by Millian Poquet's avatar Millian Poquet
Browse files

[code,doc] Test+fix+test QUERY/ANSWER

Modifications are mainly described in doc/changelog.md.

In short:
- QUERY_REQUEST is now named QUERY
- QUERY_REPLY is now named ANSWER
the two messages are now bidirectional.

This feature was not working nor following the protocol.
This is now tested thanks to batsched new version.
parent 91263db2
Pipeline #2237 failed with stages
in 7 minutes and 52 seconds
......@@ -38,7 +38,7 @@ build_batsched:
# Get the code at the expected revision
- git clone https://gitlab.inria.fr/batsim/batsched.git /builds/batsim/batsim/schedulers/batsched
- cd /builds/batsim/batsim/schedulers/batsched
- git reset --hard fe0a6b2 # 2018-01-15
- git reset --hard fb92085 # 2018-01-16
# Build batsched
- rm -rf /builds/batsim/batsim/schedulers/batsched/build
- mkdir -p /builds/batsim/batsim/schedulers/batsched/build
......
......@@ -217,6 +217,12 @@ add_test(energy
-bod /tmp/batsim_tests/energy
-bwd ${CMAKE_SOURCE_DIR})
add_test(energy_query
${CMAKE_SOURCE_DIR}/tools/experiments/execute_instances.py
${CMAKE_SOURCE_DIR}/test/test_energy_query.yaml
-bod /tmp/batsim_tests/energy_query
-bwd ${CMAKE_SOURCE_DIR})
add_test(walltime
${CMAKE_SOURCE_DIR}/tools/experiments/execute_instances.py
${CMAKE_SOURCE_DIR}/test/test_walltime.yaml
......@@ -336,7 +342,7 @@ add_test(set_job_metadata
${CMAKE_SOURCE_DIR}/tools/experiments/execute_instances.py
${CMAKE_SOURCE_DIR}/test/test_set_job_metadata.yaml
-bod /tmp/batsim_tests/set_job_metadata
-bwd ${CMAKE_SOURCE_DIR})
-bwd ${CMAKE_SworkloadOURCE_DIR})
add_test(send_recv
${CMAKE_SOURCE_DIR}/tools/experiments/execute_instances.py
......
......@@ -19,16 +19,25 @@ Batsim's public API includes:
set metadata to jobs.
Such metadata is written in the ``_jobs.csv`` output file.
- The ``_schedule.csv`` output file now contains a batsim_version field.
- The ``energy_query`` test checks that ``QUERY``/``ANSWER`` work as expected
for the ``consumed_energy`` request.
### Changed
- The ``_jobs.csv`` output file is now written more cleanly.
The order of the columns within it may have changed.
Removal of the deprecated hacky_job_id.
- The ``QUERY_REQUEST`` and ``QUERY_REPLY`` messages have been respectively
renamed ``QUERY`` and ``ANSWER``. This pair of messages is now bidirectional
(Batsim can now ask information to the scheduler).
Redis interactions with this pair of messages is no longer in the protocol
(as it has never been implemented).
### Fixed
- Numeric sort should now work as expected (this is now tested).
- Power stace tracing now works when the number of machines is big.
- Output buffers now work even if incoming texts are bigger than the buffer.
- The ``QUERY_REQUEST``/``QUERY_REPLY`` messages were not respecting the
protocol definition (probably never tested since the JSON protocol update).
[//]: ==========================================================================
## [1.4.0] - 2017-10-07
......
......@@ -113,6 +113,48 @@ events list is empty: ``"events": []``
}
```
### QUERY
This message allows a peer to ask specific information to its counterpart:
- Batsim can ask information to the scheduler.
- The scheduler can ask information to Batsim.
The other peer should answer such QUERY via an [ANSWER](#answer).
For now, Batsim **queries** the following requests:
- None.
For now, Batsim **answers** to the following requests:
- "consumed_energy" with no argument: queries Batsim about the total
consumed energy (from time 0 to now) in Joules.
Only works in energy mode.
- **data**: a dictionnary of requests.
- **example**:
```json
{
"timestamp": 10.0,
"type": "QUERY",
"data": {
"requests": {"consumed_energy": {}}
}
}
```
### ANSWER
This is a reply to a [QUERY](#query) message.
- **data**: See [QUERY](#query) documentation
- **example**:
```json
{
"timestamp": 10.0,
"type": "ANSWER",
"data": {"consumed_energy": 12500.0}
}
```
---
## Batsim to Scheduler events
......@@ -415,33 +457,6 @@ been done.
}
```
### QUERY_REPLY
This is a reply to a [QUERY_REQUEST](#query_request) message. It depends on the
The message content depends on whether redis is enabled in the
[Batsim configuration](./configuration.md).
If ``{"redis": { "enabled": true }}``, the reply will
go in redis and only the key will be given. Otherwise, the response will be
put directly in the message.
- **data**: See [QUERY_REQUEST](#query_request) documentation
- **example**:
```json
{
"timestamp": 10.0,
"type": "QUERY_REPLY",
"data": {"redis_keys": "/my/key/path0" }
}
```
or
```json
{
"timestamp": 10.0,
"type": "QUERY_REPLY",
"data": {"consumed_energy": "12500" }
}
```
### REQUESTED_CALL
This message is a response to the [CALL_ME_LATER](#call_me_later) message.
......@@ -463,26 +478,6 @@ These events are sent by the scheduler to Batsim.
BATSIM <--- DECISION
```
### QUERY_REQUEST
This is a query sent to Batsim to get information about the simulation
state (or whatever you want to know...). The supported requests are:
- "consumed_energy" with no argument that asks Batsim about the total
consumed energy (from time 0 to now) in Joules. Works only in energy
mode.
- **data**: a dictionnary of requests.
- **example**:
```json
{
"timestamp": 10.0,
"type": "QUERY_REQUEST",
"data": {
"requests": {"consumed_energy": {}}
}
}
```
### REJECT_JOB
Rejects a job that has already been submitted.
......
batsched @ fb92085a
Subproject commit fe0a6b242c4c4371afc7545e730aa68ceee721c2
Subproject commit fb92085ac21563cb07bdc53f068654104ae893c3
......@@ -423,13 +423,13 @@ void JsonProtocolWriter::append_resource_state_changed(const MachineRange & reso
_events.PushBack(event, _alloc);
}
void JsonProtocolWriter::append_query_reply_energy(double consumed_energy,
double date)
void JsonProtocolWriter::append_answer_energy(double consumed_energy,
double date)
{
/* {
"timestamp": 10.0,
"type": "QUERY_REPLY",
"data": {"energy_consumed": "12500" }
"type": "ANSWER",
"data": {"consumed_energy": 12500.0}
} */
xbt_assert(date >= _last_date, "Date inconsistency");
......@@ -438,8 +438,8 @@ void JsonProtocolWriter::append_query_reply_energy(double consumed_energy,
Value event(rapidjson::kObjectType);
event.AddMember("timestamp", Value().SetDouble(date), _alloc);
event.AddMember("type", Value().SetString("QUERY_REPLY"), _alloc);
event.AddMember("data", Value().SetObject().AddMember("energy_consumed", Value().SetDouble(consumed_energy), _alloc), _alloc);
event.AddMember("type", Value().SetString("ANSWER"), _alloc);
event.AddMember("data", Value().SetObject().AddMember("consumed_energy", Value().SetDouble(consumed_energy), _alloc), _alloc);
_events.PushBack(event, _alloc);
}
......@@ -477,7 +477,7 @@ string JsonProtocolWriter::generate_current_message(double date)
JsonProtocolReader::JsonProtocolReader(BatsimContext *context) :
context(context)
{
_type_to_handler_map["QUERY_REQUEST"] = &JsonProtocolReader::handle_query_request;
_type_to_handler_map["QUERY"] = &JsonProtocolReader::handle_query;
_type_to_handler_map["REJECT_JOB"] = &JsonProtocolReader::handle_reject_job;
_type_to_handler_map["EXECUTE_JOB"] = &JsonProtocolReader::handle_execute_job;
_type_to_handler_map["CHANGE_JOB_STATE"] = &JsonProtocolReader::handle_change_job_state;
......@@ -544,37 +544,46 @@ void JsonProtocolReader::parse_and_apply_event(const Value & event_object,
handler_function(this, event_number, timestamp, data_object);
}
void JsonProtocolReader::handle_query_request(int event_number, double timestamp, const Value &data_object)
void JsonProtocolReader::handle_query(int event_number, double timestamp, const Value &data_object)
{
(void) event_number; // Avoids a warning if assertions are ignored
/* {
"timestamp": 10.0,
"type": "QUERY_REQUEST",
"type": "QUERY",
"data": {
"requests": {"consumed_energy": {}}
}
} */
xbt_assert(data_object.IsObject(), "Invalid JSON message: the 'data' value of event %d (QUERY_REQUEST) should be an object", event_number);
xbt_assert(data_object.MemberCount() > 0, "Invalid JSON message: the 'data' value of event %d (QUERY_REQUEST) cannot be empty (size=%d)", event_number, (int)data_object.MemberCount());
xbt_assert(data_object.IsObject(), "Invalid JSON message: the 'data' value of event %d (QUERY) should be an object", event_number);
xbt_assert(data_object.MemberCount() == 1, "Invalid JSON message: the 'data' value of event %d (QUERY) must be of size 1 (size=%d)", event_number, (int)data_object.MemberCount());
xbt_assert(data_object.HasMember("requests"), "Invalid JSON message: the 'data' value of event %d (QUERY) must have a 'requests' member", event_number);
const Value & requests = data_object["requests"];
xbt_assert(requests.IsObject(), "Invalid JSON message: the 'requests' member of the 'data' object of event %d (QUERY) must be an object", event_number);
xbt_assert(requests.MemberCount() > 0, "Invalid JSON message: the 'requests' object of the 'data' object of event %d (QUERY) must be non-empty", event_number);
for (auto it = data_object.MemberBegin(); it != data_object.MemberEnd(); ++it)
for (auto it = requests.MemberBegin(); it != requests.MemberEnd(); ++it)
{
const Value & key_value = it->name;
const Value & value_object = it->value;
(void) value_object; // Avoids a warning if assertions are ignored
xbt_assert(key_value.IsString(), "Invalid JSON message: a key within the 'data' object of event %d (QUERY_REQUEST) is not a string", event_number);
xbt_assert(key_value.IsString(), "Invalid JSON message: a key within the 'data' object of event %d (QUERY) is not a string", event_number);
string key = key_value.GetString();
xbt_assert(std::find(accepted_requests.begin(), accepted_requests.end(), key) != accepted_requests.end(), "Invalid JSON message: Unknown QUERY_REQUEST '%s' of event %d", key.c_str(), event_number);
xbt_assert(std::find(accepted_requests.begin(), accepted_requests.end(), key) != accepted_requests.end(), "Invalid JSON message: Unknown QUERY '%s' of event %d", key.c_str(), event_number);
xbt_assert(value_object.IsObject(), "Invalid JSON message: the value of '%s' inside 'data' object of event %d (QUERY_REQUEST) is not an object", key.c_str(), event_number);
xbt_assert(value_object.IsObject(), "Invalid JSON message: the value of '%s' inside the 'requests' object of the 'data' object of event %d (QUERY) is not an object", key.c_str(), event_number);
if (key == "consumed_energy")
{
xbt_assert(value_object.ObjectEmpty(), "Invalid JSON message: the value of '%s' inside 'data' object of event %d (QUERY_REQUEST) should be empty", key.c_str(), event_number);
xbt_assert(value_object.ObjectEmpty(), "Invalid JSON message: the value of '%s' inside the 'requests' object of the 'data' object of event %d (QUERY) should be empty", key.c_str(), event_number);
send_message(timestamp, "server", IPMessageType::SCHED_TELL_ME_ENERGY);
}
else
{
xbt_assert(0, "Invalid JSON message: in event %d (QUERY): request type '%s' is unknown", event_number, key.c_str());
}
}
}
......
......@@ -153,12 +153,12 @@ public:
double date) = 0;
/**
* @brief Appends a QUERY_REPLY (energy) event.
* @brief Appends an ANSWER (energy) event.
* @param[in] consumed_energy The total consumed energy in joules
* @param[in] date The event date. Must be greater than or equal to the previous event.
*/
virtual void append_query_reply_energy(double consumed_energy,
double date) = 0;
virtual void append_answer_energy(double consumed_energy,
double date) = 0;
/**
* @brief Appends a REQUESTED_CALL message.
......@@ -293,12 +293,12 @@ public:
double date);
/**
* @brief Appends a QUERY_REPLY (energy) event.
* @brief Appends an ANSWER (energy) event.
* @param[in] consumed_energy The total consumed energy in joules
* @param[in] date The event date. Must be greater than or equal to the previous event.
*/
void append_query_reply_energy(double consumed_energy,
double date);
void append_answer_energy(double consumed_energy,
double date);
/**
* @brief Appends a REQUESTED_CALL message.
......@@ -403,12 +403,12 @@ public:
/**
* @brief Handles a QUERY_REQUEST event
* @brief Handles a QUERY event
* @param[in] event_number The event number in [0,nb_events[.
* @param[in] timestamp The event timestamp
* @param[in] data_object The data associated with the event (JSON object)
*/
void handle_query_request(int event_number, double timestamp, const rapidjson::Value & data_object);
void handle_query(int event_number, double timestamp, const rapidjson::Value & data_object);
/**
* @brief Handles a REJECT_JOB event
......
......@@ -428,7 +428,7 @@ void server_on_sched_tell_me_energy(ServerData * data,
{
(void) task_data;
long double total_consumed_energy = data->context->machines.total_consumed_energy(data->context);
data->context->proto_writer->append_query_reply_energy(total_consumed_energy, MSG_get_clock());
data->context->proto_writer->append_answer_energy(total_consumed_energy, MSG_get_clock());
}
void server_on_wait_query(ServerData * data,
......
# This script should be called from Batsim's root directory
# If needed, the working directory of this script can be specified within this file
#base_working_directory: ~/proj/batsim
# If needed, the output directory of this script can be specified within this file
base_output_directory: /tmp/batsim_tests/energy_query
base_variables:
batsim_dir: ${base_working_directory}
implicit_instances:
implicit:
sweep:
platform :
- {"name":"homo128", "filename":"${batsim_dir}/platforms/energy_platform_homogeneous_no_net_128.xml"}
workload :
- {"name":"tiny", "filename":"${batsim_dir}/workload_profiles/test_workload_profile.json"}
- {"name":"medium", "filename":"${batsim_dir}/workload_profiles/batsim_paper_workload_example.json"}
algo:
- {"name":"watcher", "sched_name":"energy_watcher"}
generic_instance:
timeout: 10
working_directory: ${base_working_directory}
output_directory: ${base_output_directory}/results/${algo[name]}_${workload[name]}_${platform[name]}
batsim_command: ${BATSIM_BIN:=batsim} -p ${platform[filename]} -w ${workload[filename]} -E -e ${output_directory}/out --mmax-workload
sched_command: ${BATSCHED_BIN:=batsched} -v ${algo[sched_name]}
commands_before_instances:
- ${batsim_dir}/test/is_batsim_dir.py ${base_working_directory}
- ${batsim_dir}/test/clean_output_dir.py ${base_output_directory}
Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment