[code,doc] Test+fix+test QUERY/ANSWER

Modifications are mainly described in doc/changelog.md.

In short:
- QUERY_REQUEST is now named QUERY
- QUERY_REPLY is now named ANSWER
the two messages are now bidirectional.

This feature was not working nor following the protocol.
This is now tested thanks to batsched new version.

.gitlab-ci.yml

@@ -38,7 +38,7 @@ build_batsched:
     # Get the code at the expected revision
     - git clone https://gitlab.inria.fr/batsim/batsched.git /builds/batsim/batsim/schedulers/batsched
     - cd /builds/batsim/batsim/schedulers/batsched
-    - git reset --hard fe0a6b2 # 2018-01-15
+    - git reset --hard fb92085 # 2018-01-16
     # Build batsched
     - rm -rf   /builds/batsim/batsim/schedulers/batsched/build
     - mkdir -p /builds/batsim/batsim/schedulers/batsched/build

CMakeLists.txt

@@ -217,6 +217,12 @@ add_test(energy
          -bod /tmp/batsim_tests/energy
          -bwd ${CMAKE_SOURCE_DIR})
 
+add_test(energy_query
+         ${CMAKE_SOURCE_DIR}/tools/experiments/execute_instances.py
+         ${CMAKE_SOURCE_DIR}/test/test_energy_query.yaml
+         -bod /tmp/batsim_tests/energy_query
+         -bwd ${CMAKE_SOURCE_DIR})
+
 add_test(walltime
          ${CMAKE_SOURCE_DIR}/tools/experiments/execute_instances.py
          ${CMAKE_SOURCE_DIR}/test/test_walltime.yaml
@@ -336,7 +342,7 @@ add_test(set_job_metadata
          ${CMAKE_SOURCE_DIR}/tools/experiments/execute_instances.py
          ${CMAKE_SOURCE_DIR}/test/test_set_job_metadata.yaml
          -bod /tmp/batsim_tests/set_job_metadata
-         -bwd ${CMAKE_SOURCE_DIR})
+         -bwd ${CMAKE_SworkloadOURCE_DIR})
 
 add_test(send_recv
          ${CMAKE_SOURCE_DIR}/tools/experiments/execute_instances.py

doc/changelog.md

@@ -19,16 +19,25 @@ Batsim's public API includes:
   set metadata to jobs.
   Such metadata is written in the ``_jobs.csv`` output file.
 - The ``_schedule.csv`` output file now contains a batsim_version field.
+- The ``energy_query`` test checks that ``QUERY``/``ANSWER`` work as expected
+  for the ``consumed_energy`` request.
 
 ### Changed
 - The ``_jobs.csv`` output file is now written more cleanly.  
   The order of the columns within it may have changed.  
   Removal of the deprecated hacky_job_id.
+- The ``QUERY_REQUEST`` and ``QUERY_REPLY`` messages have been respectively
+  renamed ``QUERY`` and ``ANSWER``. This pair of messages is now bidirectional
+  (Batsim can now ask information to the scheduler).  
+  Redis interactions with this pair of messages is no longer in the protocol
+  (as it has never been implemented).
 
 ### Fixed
 - Numeric sort should now work as expected (this is now tested).
 - Power stace tracing now works when the number of machines is big.
 - Output buffers now work even if incoming texts are bigger than the buffer.
+- The ``QUERY_REQUEST``/``QUERY_REPLY`` messages were not respecting the
+  protocol definition (probably never tested since the JSON protocol update).
 
 [//]: ==========================================================================
 ## [1.4.0] - 2017-10-07

doc/proto_description.md

@@ -113,6 +113,48 @@ events list is empty: ``"events": []``
 }
 ```
 
+### QUERY
+
+This message allows a peer to ask specific information to its counterpart:
+- Batsim can ask information to the scheduler.
+- The scheduler can ask information to Batsim.
+
+The other peer should answer such QUERY via an [ANSWER](#answer).
+
+For now, Batsim **queries** the following requests:
+- None.
+
+For now, Batsim **answers** to the following requests:
+- "consumed_energy" with no argument: queries Batsim about the total
+  consumed energy (from time 0 to now) in Joules.
+  Only works in energy mode.
+
+- **data**: a dictionnary of requests.
+- **example**:
+```json
+{
+  "timestamp": 10.0,
+  "type": "QUERY",
+  "data": {
+    "requests": {"consumed_energy": {}}
+  }
+}
+```
+
+### ANSWER
+
+This is a reply to a [QUERY](#query) message.
+
+- **data**: See [QUERY](#query) documentation
+- **example**:
+```json
+{
+  "timestamp": 10.0,
+  "type": "ANSWER",
+  "data": {"consumed_energy": 12500.0}
+}
+```
+
 ---
 
 ## Batsim to Scheduler events
@@ -415,33 +457,6 @@ been done.
 }
 ```
 
-### QUERY_REPLY
-
-This is a reply to a [QUERY_REQUEST](#query_request) message. It depends on the
-The message content depends on whether redis is enabled in the
-[Batsim configuration](./configuration.md).
-If ``{"redis": { "enabled": true }}``, the reply will
-go in redis and only the key will be given. Otherwise, the response will be
-put directly in the message.
-
-- **data**: See [QUERY_REQUEST](#query_request) documentation
-- **example**:
-```json
-{
-  "timestamp": 10.0,
-  "type": "QUERY_REPLY",
-  "data": {"redis_keys": "/my/key/path0" }
-}
-```
-or
-```json
-{
-  "timestamp": 10.0,
-  "type": "QUERY_REPLY",
-  "data": {"consumed_energy": "12500" }
-}
-```
-
 ### REQUESTED_CALL
 This message is a response to the [CALL_ME_LATER](#call_me_later) message.
 
@@ -463,26 +478,6 @@ These events are sent by the scheduler to Batsim.
 BATSIM <--- DECISION
 ```
 
-### QUERY_REQUEST
-
-This is a query sent to Batsim to get information about the simulation
-state (or whatever you want to know...). The supported requests are:
-- "consumed_energy" with no argument that asks Batsim about the total
-  consumed energy (from time 0 to now) in Joules. Works only in energy
-  mode.
-
-- **data**: a dictionnary of requests.
-- **example**:
-```json
-{
-  "timestamp": 10.0,
-  "type": "QUERY_REQUEST",
-  "data": {
-    "requests": {"consumed_energy": {}}
-  }
-}
-```
-
 ### REJECT_JOB
 
 Rejects a job that has already been submitted.

batsched @ fb92085a

-Subproject commit fe0a6b242c4c4371afc7545e730aa68ceee721c2
+Subproject commit fb92085ac21563cb07bdc53f068654104ae893c3

src/protocol.cpp

@@ -423,13 +423,13 @@ void JsonProtocolWriter::append_resource_state_changed(const MachineRange & reso
     _events.PushBack(event, _alloc);
 }
 
-void JsonProtocolWriter::append_query_reply_energy(double consumed_energy,
-                                                   double date)
+void JsonProtocolWriter::append_answer_energy(double consumed_energy,
+                                              double date)
 {
     /* {
       "timestamp": 10.0,
-      "type": "QUERY_REPLY",
-      "data": {"energy_consumed": "12500" }
+      "type": "ANSWER",
+      "data": {"consumed_energy": 12500.0}
     } */
 
     xbt_assert(date >= _last_date, "Date inconsistency");
@@ -438,8 +438,8 @@ void JsonProtocolWriter::append_query_reply_energy(double consumed_energy,
 
     Value event(rapidjson::kObjectType);
     event.AddMember("timestamp", Value().SetDouble(date), _alloc);
-    event.AddMember("type", Value().SetString("QUERY_REPLY"), _alloc);
-    event.AddMember("data", Value().SetObject().AddMember("energy_consumed", Value().SetDouble(consumed_energy), _alloc), _alloc);
+    event.AddMember("type", Value().SetString("ANSWER"), _alloc);
+    event.AddMember("data", Value().SetObject().AddMember("consumed_energy", Value().SetDouble(consumed_energy), _alloc), _alloc);
 
     _events.PushBack(event, _alloc);
 }
@@ -477,7 +477,7 @@ string JsonProtocolWriter::generate_current_message(double date)
 JsonProtocolReader::JsonProtocolReader(BatsimContext *context) :
     context(context)
 {
-    _type_to_handler_map["QUERY_REQUEST"] = &JsonProtocolReader::handle_query_request;
+    _type_to_handler_map["QUERY"] = &JsonProtocolReader::handle_query;
     _type_to_handler_map["REJECT_JOB"] = &JsonProtocolReader::handle_reject_job;
     _type_to_handler_map["EXECUTE_JOB"] = &JsonProtocolReader::handle_execute_job;
     _type_to_handler_map["CHANGE_JOB_STATE"] = &JsonProtocolReader::handle_change_job_state;
@@ -544,37 +544,46 @@ void JsonProtocolReader::parse_and_apply_event(const Value & event_object,
     handler_function(this, event_number, timestamp, data_object);
 }
 
-void JsonProtocolReader::handle_query_request(int event_number, double timestamp, const Value &data_object)
+void JsonProtocolReader::handle_query(int event_number, double timestamp, const Value &data_object)
 {
     (void) event_number; // Avoids a warning if assertions are ignored
     /* {
       "timestamp": 10.0,
-      "type": "QUERY_REQUEST",
+      "type": "QUERY",
       "data": {
         "requests": {"consumed_energy": {}}
       }
     } */
 
-    xbt_assert(data_object.IsObject(), "Invalid JSON message: the 'data' value of event %d (QUERY_REQUEST) should be an object", event_number);
-    xbt_assert(data_object.MemberCount() > 0, "Invalid JSON message: the 'data' value of event %d (QUERY_REQUEST) cannot be empty (size=%d)", event_number, (int)data_object.MemberCount());
+    xbt_assert(data_object.IsObject(), "Invalid JSON message: the 'data' value of event %d (QUERY) should be an object", event_number);
+    xbt_assert(data_object.MemberCount() == 1, "Invalid JSON message: the 'data' value of event %d (QUERY) must be of size 1 (size=%d)", event_number, (int)data_object.MemberCount());
+    xbt_assert(data_object.HasMember("requests"), "Invalid JSON message: the 'data' value of event %d (QUERY) must have a 'requests' member", event_number);
+
+    const Value & requests = data_object["requests"];
+    xbt_assert(requests.IsObject(), "Invalid JSON message: the 'requests' member of the 'data' object  of event %d (QUERY) must be an object", event_number);
+    xbt_assert(requests.MemberCount() > 0, "Invalid JSON message: the 'requests' object of the 'data' object of event %d (QUERY) must be non-empty", event_number);
 
-    for (auto it = data_object.MemberBegin(); it != data_object.MemberEnd(); ++it)
+    for (auto it = requests.MemberBegin(); it != requests.MemberEnd(); ++it)
     {
         const Value & key_value = it->name;
         const Value & value_object = it->value;
         (void) value_object; // Avoids a warning if assertions are ignored
 
-        xbt_assert(key_value.IsString(), "Invalid JSON message: a key within the 'data' object of event %d (QUERY_REQUEST) is not a string", event_number);
+        xbt_assert(key_value.IsString(), "Invalid JSON message: a key within the 'data' object of event %d (QUERY) is not a string", event_number);
         string key = key_value.GetString();
-        xbt_assert(std::find(accepted_requests.begin(), accepted_requests.end(), key) != accepted_requests.end(), "Invalid JSON message: Unknown QUERY_REQUEST '%s' of event %d", key.c_str(), event_number);
+        xbt_assert(std::find(accepted_requests.begin(), accepted_requests.end(), key) != accepted_requests.end(), "Invalid JSON message: Unknown QUERY '%s' of event %d", key.c_str(), event_number);
 
-        xbt_assert(value_object.IsObject(), "Invalid JSON message: the value of '%s' inside 'data' object of event %d (QUERY_REQUEST) is not an object", key.c_str(), event_number);
+        xbt_assert(value_object.IsObject(), "Invalid JSON message: the value of '%s' inside the 'requests' object of the 'data' object of event %d (QUERY) is not an object", key.c_str(), event_number);
 
         if (key == "consumed_energy")
         {
-            xbt_assert(value_object.ObjectEmpty(), "Invalid JSON message: the value of '%s' inside 'data' object of event %d (QUERY_REQUEST) should be empty", key.c_str(), event_number);
+            xbt_assert(value_object.ObjectEmpty(), "Invalid JSON message: the value of '%s' inside the 'requests' object of the 'data' object of event %d (QUERY) should be empty", key.c_str(), event_number);
             send_message(timestamp, "server", IPMessageType::SCHED_TELL_ME_ENERGY);
         }
+        else
+        {
+            xbt_assert(0, "Invalid JSON message: in event %d (QUERY): request type '%s' is unknown", event_number, key.c_str());
+        }
     }
 }
 

src/protocol.hpp

@@ -153,12 +153,12 @@ public:
                                                double date) = 0;
 
     /**
-     * @brief Appends a QUERY_REPLY (energy) event.
+     * @brief Appends an ANSWER (energy) event.
      * @param[in] consumed_energy The total consumed energy in joules
      * @param[in] date The event date. Must be greater than or equal to the previous event.
      */
-    virtual void append_query_reply_energy(double consumed_energy,
-                                           double date) = 0;
+    virtual void append_answer_energy(double consumed_energy,
+                                      double date) = 0;
 
     /**
      * @brief Appends a REQUESTED_CALL message.
@@ -293,12 +293,12 @@ public:
                                        double date);
 
     /**
-     * @brief Appends a QUERY_REPLY (energy) event.
+     * @brief Appends an ANSWER (energy) event.
      * @param[in] consumed_energy The total consumed energy in joules
      * @param[in] date The event date. Must be greater than or equal to the previous event.
      */
-    void append_query_reply_energy(double consumed_energy,
-                                   double date);
+    void append_answer_energy(double consumed_energy,
+                              double date);
 
     /**
      * @brief Appends a REQUESTED_CALL message.
@@ -403,12 +403,12 @@ public:
 
 
     /**
-     * @brief Handles a QUERY_REQUEST event
+     * @brief Handles a QUERY event
      * @param[in] event_number The event number in [0,nb_events[.
      * @param[in] timestamp The event timestamp
      * @param[in] data_object The data associated with the event (JSON object)
      */
-    void handle_query_request(int event_number, double timestamp, const rapidjson::Value & data_object);
+    void handle_query(int event_number, double timestamp, const rapidjson::Value & data_object);
 
     /**
      * @brief Handles a REJECT_JOB event

src/server.cpp

@@ -428,7 +428,7 @@ void server_on_sched_tell_me_energy(ServerData * data,
 {
     (void) task_data;
     long double total_consumed_energy = data->context->machines.total_consumed_energy(data->context);
-    data->context->proto_writer->append_query_reply_energy(total_consumed_energy, MSG_get_clock());
+    data->context->proto_writer->append_answer_energy(total_consumed_energy, MSG_get_clock());
 }
 
 void server_on_wait_query(ServerData * data,

test/test_energy_query.yaml

+# This script should be called from Batsim's root directory
+
+# If needed, the working directory of this script can be specified within this file
+#base_working_directory: ~/proj/batsim
+
+# If needed, the output directory of this script can be specified within this file
+base_output_directory: /tmp/batsim_tests/energy_query
+
+base_variables:
+  batsim_dir: ${base_working_directory}
+
+implicit_instances:
+  implicit:
+    sweep:
+      platform :
+        - {"name":"homo128", "filename":"${batsim_dir}/platforms/energy_platform_homogeneous_no_net_128.xml"}
+      workload :
+        - {"name":"tiny", "filename":"${batsim_dir}/workload_profiles/test_workload_profile.json"}
+        - {"name":"medium", "filename":"${batsim_dir}/workload_profiles/batsim_paper_workload_example.json"}
+      algo:
+        - {"name":"watcher", "sched_name":"energy_watcher"}
+    generic_instance:
+      timeout: 10
+      working_directory: ${base_working_directory}
+      output_directory: ${base_output_directory}/results/${algo[name]}_${workload[name]}_${platform[name]}
+      batsim_command: ${BATSIM_BIN:=batsim} -p ${platform[filename]} -w ${workload[filename]} -E -e ${output_directory}/out --mmax-workload
+      sched_command: ${BATSCHED_BIN:=batsched} -v ${algo[sched_name]}
+
+commands_before_instances:
+  - ${batsim_dir}/test/is_batsim_dir.py ${base_working_directory}
+  - ${batsim_dir}/test/clean_output_dir.py ${base_output_directory}
+