Vous avez reçu un message "Your GitLab account has been locked ..." ? Pas d'inquiétude : lisez cet article https://docs.gricad-pages.univ-grenoble-alpes.fr/help/unlock/

README.md 24.3 KB
Newer Older
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
1
2
3
4
5
# QBind, a convenient and efficient (de)serialization mechanism on top of Qt

QBind was developed to demonstrate the feasibility and advantages of a novel (de)serialization mechanism on top of
existing [Qt](http://qt.io) data formats (Settings, QDataStream, Json, Cbor, Xml) and generic data types (containers, QVariant, QMetaObject, etc.).

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
6
**It would greatly simplify binding C++ data to Qt-supported [data formats](https://doc.qt.io/qt-5/topics-data-storage.html#)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
7
or [data model](https://doc.qt.io/qt-5/model-view-programming.html) while providing more performance than existing approaches.**
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
8
9
10

QBind requirements below are useful for most (de)serialization use cases. They would be mandatory to implement in Qt a tracing facility
allowing to analyse how software is used in the field and diagnose complex issues. Effectively:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
11
12
13
14
15
16
- QDebug is conveniently used for debugging. But it may not be enough time- or space- efficient for running software
  in the field, or require too much log parsing work to detect issues from multiple tracepoints or traces.
- LTTng/ETW (or Qt tracegen tool) allow analysing performance using a few statically-defined tracepoints with
  high-performance and structured data. But analysing unexpected uses and issues in the field requires a lot more 
  tracepoints than can be conveniently defined statically.

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
17
18
19
20
> **DISLAIMER** QBind is not currently unit-tested but provided with a [sample and benchmark](main.cpp). 
> Also, the implementation is written in a concise style that supported the multiple refactorings but does not help explaining it.
> This is especially true for the various IBind implementations. The README and [design](design.md) documentation should be read
> before trying to understand the implementation.
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
21

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
22
See:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
23
24
25
26
- [The requirements (read/write)](#the-requirements)
- [Some examples](#examples)
- [The results](#results)
- [Our conclusion](#conclusion)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
27
- [The design](design.md)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
28

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
29

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
30
## The requirements
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
31

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
32
33
34
35
36
37
38
39
40
41
* **RW1. Easily customizable** for user-defined and third-part types
  1. do not require writing template specializations for user-defined types
  2. use type system and code completion to guide the user for simple binds (see also W1)
  3. avoid most boiler-plate code, including redundant read/write code (as required with QDataStream << and >>)
  4. override existing bind with custom view types or lambda
* **RW2. Good support of Qt data**:
  1. almost all features of simple data (QJson..., QDataStream, QSettings)
  2. most features of complex data (QCbor..., QXml..., QMetaObject, QModel...)
* **RW3. Allow optional metadata for complex formats** (CBOR tags, XML tags and attributes, QModel* columnNames, etc.)
* **RW4. No restriction on data size**
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
42
43
  (some restrictions may apply with specific implementations that may, e.g. store context for each data structure levels, 
  cache out-of-order data, or even store all the data in memory)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
44

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
45
### Write (serialization)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
46

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
47
48
49
* **W1. Format can be changed at runtime**
  (a compiled tracepoint in a library must be able to generate Json or Cbor as desired by library user)
* **W2. Very fast**
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
50
51
52
  1. similar to QDataStream
  2. same order of magnitude as protobuf/lttng/etl
  3. potentially without dynamic memory allocation or thread locking
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
53
* **W3. Well-formed** data ensured (almost) with low impact on performance
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
54

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
55
### Read (deserialization)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
56
57
58
59
60
61
62
63
64
65
66

* **R1. Encourage standard and explicit formats like Json/Cbor** to:
  - avoid mismatches between actual data and the "schema" at hand (be it a data schema or just code)
  - favor interoperability
* **R2. Support simple, type-by-type, data-schema evolution** like:
  1. adding, deleting or moving named items in a record
  2. adding, or changing from required to optional an item with default value in a sequence
  3. *changing from optional or required to repeated an item (provided it is not itself a sequence) (TBD)*
* **R3. Allow reporting all errors** and mismatches between what was expected and what is read (unless the data format was implicit as below)
* **R4. Allow implicit formats like QDataStream** when the reader knows exactly what to read

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
67
### A notable non-requirement
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
68

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
69
70
71
**QBind does not _automatically_ support (de)serialization of graphs unlike boost::serialization**. Pointers will be followed and in
case of a graph with a loop, the operation will result in a stack overflow. Hence, the user has to choose an appropriate encoding for his
graph data using some kind of "reference" values. We argue this makes the model translatable to much more data formats like Json since
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
72
73
74
75
76
it does not mandate native support for references. Moreover, QBind supports metadata as an optional way to encode such special values 
for data formats supporting it like [CBOR value sharing tags](http://cbor.schmorp.de/value-sharing) and XML.

## Examples

77
78
### Extending QBind to C++ types

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
79
One can rely on Qt reflection to bind a Q_OBJECT or Q_GADGET stored properties using `QBIND_GADGET_WITH_METAOBJECT` macro
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
80
```cpp
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
81
82
83
84
85
86
87
class Phone {
    Q_GADGET
    Q_PROPERTY(Type    type   MEMBER _t)
    Q_PROPERTY(QString number MEMBER _n)
public:
    QBIND_GADGET_WITH_METAOBJECT // making it (de)serializable and printable using reflection

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
88
    enum Type { Unknown=0, Mobile, Home, Office };
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
89
90
91
92
93
94
    Q_ENUM(Type)

    Phone(Type t=Unknown, QString n=QString()) : _t(t), _n(n) {}
private:
    //...
};
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
95
96
```

97
98
One can be much faster and get more control implementing a custom `bind` method using a convenient fluent interface for which 
smart editors will provide code completion similar to that of an XML/JSON editor:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
99
100
101
102
103
104
105
```cpp
struct Person
{
    QString firstName, lastName; double height; int age; QVector<QString> phones; QString comments; QList<Person> children;

    Cursor bind(Val<Cursor>&& value) { // works with value->mode()==Read as well as Write
        return value
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
106
107
            .record("Person") // aggregates a usually small number of named items in any order (Person is an optional meta-name for the record that may be ignored)
                .sequence("names") // aggregates any number of items in a fixed order, just to illustrate a basic mapping between struct Person and a more general data schema
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
108
109
110
                    .bind(firstName)
                    .bind( lastName)
                    .out()
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
111
                .bind("height"  ,height  ) // remember named items such as "names" and "height" may appear in any order in a record()
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
112
                .bind("age"     ,age  ,-1) // reads null() or missing values as the default value: -1
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
113
114
                .bind("phones"  ,phones  ) // recursively calls QBind to take care of that part
                .bind("comments",comments) // directly calls IBind since it natively supports QString
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
115
116
117
118
119
                .bind("children",children)
                ; // automagically closes opened record()
    }
};
```
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
120
121

The value argument provides a fluent interface allowing to declaratively bind C++ data to a choice of:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
122
123
* `sequence` of data items in a fixed (meaningful) order, possibly infinite
* `record` of named data items in any order
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
124
125
126
* Atomic values like `QString firstName`, `double height`, `int age`
* Generically supported T values for which a specialized QBind<T> is defined like `QVector<QString> phones`, `QList<E> children`

127
<details>
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
<summary>The same bind can read/write data in various formats with the same data (but varying metadata support)</summary>

```xml
<Person>
	<names>
		<string>John</string>
		<string>Doe</string>
        </names>
	<height>1.75</height>
	<age>18</age>
	<phones><Phone><Home>2</Home><number>+44 1234567</number></Phone></phones>
	<comments>...</comments>
	<children/>
</Person>
```
```json
{
	"names":[
		"John",
		"Doe"
		],
	"height":1.75,
	"age":18,
	"phones":[{"type":2,"number":"+44 1234567"}],
	"comments":"...",
	"children":[]
}
```

</details>
<details>
<summary>This code can be compared with the minimum required to produce the same output using QCborStreamWriter...</summary>
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178

```cpp
QCborStreamWriter s(&b);
s.startMap();
    s.append("names"); s.startArray();
        s.append(person.firstName);
        s.append(person. lastName);
        s.endArray();
    s.append("height"  ); s.append(person.height  );
    s.append("age"     ); s.append(person.age     );
    s.append("phones"  );
        int size = person.phones.size();
        s.startArray(quint64(size));
        for (int i=0; i < size; i++) {
            s.append(person.phones[i]);
        };
        s.endArray();
    s.append("comments"); s.append(person.comments.toUtf8().constData());
    s.append("children"); s.startArray();
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
179
        //...
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
        s.endArray();
s.endMap();
```
*NB: QCborValue would be more convenient than QCborStreamWriter but it is much less efficient than QBind*

</details>
<details>
<summary>Or the equivalent for reading the exact same output (without regard to variable length collections)...</summary>

```cpp
QCborStreamReader reader(roundtrip.buffer());
if (reader.isMap() &&
    reader.enterContainer() &&
    reader.hasNext() &&
    reader.isString() &&
    reader.readString().data=="names" &&
    reader.readString().status==QCborStreamReader::EndOfString &&
    reader.isArray() &&
    reader.enterContainer() &&
    reader.hasNext() &&
    reader.isString())
    p.firstName=reader.readString().data;
if (reader.readString().status==QCborStreamReader::EndOfString &&
    reader.hasNext() &&
    reader.isString())
    p.lastName=reader.readString().data;
if (reader.readString().status==QCborStreamReader::EndOfString &&
    !reader.hasNext() &&
    reader.leaveContainer() &&
    reader.hasNext() &&
    reader.isString() &&
    reader.readString().data=="height" &&
    reader.readString().status==QCborStreamReader::EndOfString &&
    reader.isDouble())
    p.height=reader.toDouble();
if (reader.next() &&
    reader.hasNext() &&
    reader.isString() &&
    reader.readString().data=="age" &&
    reader.readString().status==QCborStreamReader::EndOfString &&
    !reader.isUnsignedInteger() &&
    reader.isNegativeInteger())
    p.age=int(reader.toInteger());
if (reader.next() &&
    reader.hasNext() &&
    reader.isString() &&
    reader.readString().data=="phones" &&
    reader.readString().status==QCborStreamReader::EndOfString &&
    reader.isArray() &&
    reader.enterContainer() &&
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
230
231
232
    reader.hasNext() /*&& ...*/)
    //...
if (/*... &&*/
233
234
235
236
237
238
239
240
241
242
243
244
245
246
    reader.leaveContainer() &&
    reader.hasNext() &&
    reader.isString() &&
    reader.readString().data=="comments" &&
    reader.readString().status==QCborStreamReader::EndOfString &&
    reader.isString())
    p.comments=reader.readString().data;
    reader.readString().status==QCborStreamReader::EndOfString &&
    !reader.hasNext() &&
    reader.leaveContainer();
```

</details>

247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
It is also possible to define missing QBind specializations for any C++ type using external QBind class specializations.
Each specialization must implement at least the bind() method for lvalue references. It may also overload bind() for rvalue
references to support temporaries, and for const lvalue reference to efficiently support copyable types.
```cpp
template<> struct QBind<QColor> {
    static Cursor bind(Val<Cursor>&& v, QColor&& c) { QColor copy(c); return bind(std::move(v),copy); } // supports writing temporaries and const QColor&
    static Cursor bind(Val<Cursor>&& v, QColor&  c) {
        if (!c.isValid()) {
            return v.null();
        }
        Rec<Cursor> r = v.record();
        switch(c.spec()) {
            case QColor::Spec::Rgb : r = r.sequence("RGB" ).bind(c.red   ()).bind(c.green        ()).bind(c.blue     ()); break;
            case QColor::Spec::Hsl : r = r.sequence("HSL" ).bind(c.hslHue()).bind(c.hslSaturation()).bind(c.lightness()); break;
            //...
        }
        if (c.alpha()<255) { r = r.bind("alpha",quint8(c.alpha())); }
        return r.bind("base", 255);
    }
};
```
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
268

269
### Extending QBind to custom formats
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
270

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
271
Defining a custom format for, say, console output only requires implementing a few `IWriter` abstract methods:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
272
```cpp
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
273
class MyTextWriter : public IWriter
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
274
275
276
{
    //...
protected:
277
278
279
280
    bool _sequence(quint32* =nullptr) { ba->append("[")                      ; return true; }
    bool _record  (quint32* =nullptr) { ba->append("[")                      ; return true; }
    bool _null    (                 ) {                                        return true; }
    bool _bind    (   const char* u8) { ba->append( u8)                      ; return true; }
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
281

282
283
284
    bool _item (QIdentifierLiteral n) { ba->append(" ").append(n).append(":"); return true; }
    bool _item (                    ) { ba->append(" ")                      ; return true; }
    bool _out  (                    ) { ba->append("]")                      ; return true; }
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
285
286
287
288
    //...
};
```

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
289
With the addition of a shortcut `operator<<`, MyTextWriter can mimic QDebug and replace it in existing code:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
290
291
```cpp
// 1.33333 3.14159 ascii characters are common in QDebug false QColor(ARGB 1, 0.176471, 0, 0.729412)
292
293
QDebug      (&s ) << 1.333333333333f << PI << ascii << false << color ;
MyTextWriter(&ba) << 1.333333333333f << PI << ascii << false << color ;
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
294
295
296
// [ 1.33333 3.14159 ascii characters are common in QDebug false [ RGB:[ 45 0 186] base:255]
```

297
298
299
300
301
Implementing a custom reader requires implementing a few `IReader` methods. This is always more complex because reading
needs to perform much more checks and may have to report transient errors instead of returning `true` as `MyTextWriter` does.
Both `IWriter` and `IReader` are convenient base classes which implement most of `IBind` interface used by QBind<T> with
generic text representations of boolean, numbers, etc.

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
302
303
304
305
306
307
308
Since QBind supports Val<_> on the right-hand side, translating between generic data formats or structures is a one-liner:
```cpp
// baIn = {"names":["John","Doe"],"height":1.7500000000000002,"age":-1,"phones":["+44 1234567","+44 2345678"],"comments":"","children":[]}
QJsonReader(&baIn).bind(QCborWriter(&baOut).value());
// baOut = 0x bf656e616d65739f644a6f686e63446f65ff66686569676874fa3fe0000063616765206670686f6e65739f6b2b343420313233343536376b2b34342032333435363738ff68636f6d6d656e747360686368696c6472656e9fffff
```

309
Last but not least, providing in advance some `meta` data allows binding deep C++ data structures in custom ways for use with multi-dimensional Q...View classes:
310
311
```cpp
QStandardItemModel tree, table, matrix; 
312
QModelWriter<>(&matrix).meta(qmSizes   ,"4,3"      ).bind(transform);
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
313
QModelWriter<>(&  flat).sequence()                  .with(persons, flatten); // recursive bind function
314
315
QModelWriter<>(&  tree).meta(qmChildren,"children" ).bind(persons);
QModelWriter<>(& table).meta(qmColumns ,"names,age").bind(persons);
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
316
//...
317
```
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
318
![Q...View](qstandardmodel.PNG)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
319

320
321
322
323
324
325
What meta does is guide the way QModelWriter will translate nested `data` structures into the [QAbstractItemModel](https://doc.qt.io/Qt-5/qabstractitemmodel.html#details).
By convention:
- `qmChildren` defines a named item corresponding to a sequence of data with the same type and same [parent](https://doc.qt.io/Qt-5/qabstractitemmodel.html#parent)
- `qmColumns`  defines the ordered set of named items that should be bound to their respective [columns](https://doc.qt.io/Qt-5/qmodelindex.html#column)
- `qmSizes`    defines the bounds of successive dimensions of a N-dimensional row- or column-wise array (where N=2 to account for Q...View limitations)
- `qmName`     allows naming data items for, e.g. XML root element and sequence items
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
326
- ...
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
327

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
328
The latest customized binds require using ad-hoc std::function like `flatten` or lambda below (mimicking Python list comprehensions):
329
330
```cpp
QStandardItemModel custom; 
331
QModelWriter<>(&custom).sequence().with([&](Seq<Cursor>&& s) {
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
332
333
    for (auto&& person : persons) {
        s = s.record()
334
335
336
337
338
339
                .item("first name")
                    .meta(qmColor, person.age >= 42 ? "green" : "blue")
                    .bind(person.firstName)
                .item("office phone")
                    .with([&](Val<Cursor>&& v) {
                        for (auto&& phone : person.phones) {
340
                            if (phone._t == Phone::Office) {
341
                                return v.bind(phone._n);
342
343
                            }
                        }
344
                        return v.null();
345
346
347
                    })
                .out();
    }
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
348
    return std::move(s);
349
350
351
});
```

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
352
## Results
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
353

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
The [samples](samples.txt) shows all data formats written along with error reporting. Here are a few of them:

### "builtin"

QDebug (with only a few decimals), Json, Xml, Cbor (can be decoded on http://cbor.me)
```
1.33333 3.14159 ascii characters are common in QDebug false QColor(ARGB 1, 0.176471, 0, 0.729412) 
```
```json
[1.33333337,3.1415926535897931,"ascii characters are common in QDebug",false,{"RGB":[45,0,186],"base":255}]
```
```xml
<sequence><decimal>1.33333337</decimal><decimal>3.1415926535897931</decimal><string>ascii characters are common in QDebug</string><boolean>false</boolean><record><RGB><integer>45</integer><integer>0</integer><integer>186</integer></RGB><base>255</base></record>
```
```
85fa3faaaaabfb400921fb54442d187825617363696920636861726163746572732061726520636f6d6d6f6e20696e20514465627567f4bf635247429f182d0018baff646261736518ffff
```

### "doubles"

QDebug (with only a few decimals), Json, Xml, Cbor (can be decoded on http://cbor.me)
```
0.333333 0.666667 0.333333 1 0.666667 0.333333 0.666667 1 0.333333 0.666667 0.333333 1 0 0 0 1
```
```json
[0.33333333333333331,0.66666666666666663,0.33333333333333331,1,0.66666666666666663,0.33333333333333331,0.66666666666666663,1,0.33333333333333331,0.66666666666666663,0.33333333333333331,1,0,0,0,1]
```
```xml
<sequence><decimal>0.33333333333333331</decimal><decimal>0.66666666666666663</decimal><decimal>0.33333333333333331</decimal><decimal>1</decimal><decimal>0.66666666666666663</decimal><decimal>0.33333333333333331</decimal><decimal>0.66666666666666663</decimal><decimal>1</decimal><decimal>0.33333333333333331</decimal><decimal>0.66666666666666663</decimal><decimal>0.33333333333333331</decimal><decimal>1</decimal><decimal>0</decimal><decimal>0</decimal><decimal>0</decimal><decimal>1</decimal></sequence>
```
```
90fb3fd5555555555555fb3fe5555555555555fb3fd5555555555555fb3ff0000000000000fb3fe5555555555555fb3fd5555555555555fb3fe5555555555555fb3ff0000000000000fb3fd5555555555555fb3fe5555555555555fb3fd5555555555555fb3ff0000000000000fb0000000000000000fb0000000000000000fb0000000000000000fb3ff0000000000000
```

### "Person"

QDebug (with only a few decimals), Json, Xml, Cbor (can be decoded on http://cbor.me)
```
Person("John", "Doe", 1.75, 18, QVector(), "unicode is likely U+01 \u0001 + U+1F \u001F + U+A4 � U+B0 � U+D8 � U+FF �", ())
```
```json
{"names":["John","Doe"],"height":1.75,"age":18,"phones":[],"comments":"unicode is likely U+01 \u0001 + U+1F \u001F + U+A4 � U+B0 � U+D8 � U+FF �","children":[]}
```
```xml
<Person><names><string>John</string><string>Doe</string></names><height>1.75</height><age>18</age><phones/><comments>unicode is likely U+01  + U+1F  + U+A4 � U+B0 � U+D8 � U+FF �</comments><children/></Person>
```
```
bf656e616d65739f644a6f686e63446f65ff66686569676874fb3ffc00000000000063616765126670686f6e65738068636f6d6d656e74737843756e69636f6465206973206c696b656c7920552b30312001202b20552b3146201f202b20552b413420c2a420552b423020c2b020552b443820c39820552b464620c3bf686368696c6472656e80ff
```

### "Phone"

QDebug (with only a few decimals), Json, Xml, Cbor (can be decoded on http://cbor.me)
```
Phone(Phone::Home, "+44 1234567")
```
```json
{"type":2,"number":"+44 1234567"}
```
```xml
<Phone><Home>2</Home><number>+44 1234567</number></Phone>
```
```
bf647479706502666e756d6265726b2b34342031323334353637ff
```

420
421
422
423
424
425
> WARNING: The QBind ability to adequately bind C++ types and formats is only limited by the set of natively supported C++ types and
> the support of meta() by each format. `IBind` natively supports QDataStream::operator<<() types and could be extended to include
> native types which can have efficient binary representations like QUuid. Alternatively, the fluent interface currently contains an
> optional TImpl parameter that could be used to define another set of natively supported C++ types, or removed to simplify the fluent
> interface.

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
426
427
### Write performance

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
428
Overall, QBind demonstrates write performance superior to existing Qt classes except QDataStream (which does not meet our W1
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
429
430
431
432
433
and read requirements). Not surprisingly, the performance depends on the kind of C++ data and data format used. Here are some
explanations about the best results:
- QByteArray obviously fails almost all our requirements since it does not even have a global version number as QDataStream does
  but it represents the minimum cost of serializing each dataset as it is equivalent to a few memcpy into a reserved buffer 
  (QByteArray results are similar to protobuf serialization part without dataset construction)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
434
435
436
- QDataStream performs very well and can offer data schema evolution using a global data schema version regularly updated for Qt internal
  needs but users do not control it and will not be able to detect errors on read, so they should refuse to read new schema versions
  (this is the main reason why we are advocating more explicit data formats)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
437
- The fluent interface (which brings convenience and well-formedness guarantees) costs around 20% for non trivial datasets as can 
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
438
  bes seen between Data and QDataStream but this cost is usually compensated by other factors like below
439
- Cbor obtains better results than QCborWriter because the fluent interface cost is more than compensated by working 
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
440
  directly on a QByteArray instead of a QIODevice (around 50% slower)
441
442
443
- Cbor can be up to 10x faster than QDebug for the "builtin" dataset and even 20x faster for the "doubles" dataset (which is not a 
  surprise), but it suffers from the absence of utf16 encoding when more QString are added like in the "Person" dataset (this may be
  solved by defining a utf16 Cbor tag)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
444
- The cost of `Bindable` which allows to compile a tracepoint and choose the trace format at runtime is hardly measurable.
445
- Regarding text formats suitable for display on a console providing more metadata (such as the name of classes and enum values), our
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
446
447
448
449
450
451
452
453
  toy example "TextWriter" obtains better performance than QDebug essentially because:
  1. it works with all source code literals in utf8 instead of QString utf16
  2. it does not use QIODevice to write small chunks (although each tracepoint could send written data to a QIODevice)
  3. it relies on the fluent interface static well-formedness guarantee instead of passing along a format state that users are responsible
     to maintain

![Benchmark Write results](write.PNG)

454
Regarding the worst performances:
455
- Variant which translates C++ types to an in-memory data structure of QVariantList and QVariantMap cannot perform very well
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
456
  (without even storing metadata at all) because of the numerous small allocations that may cost much from time to time
457
458
459
460
461
462
- QJsonValue, QCborValue bad performance is explained the same way as Variant above
- More generally, it seems clear that pointer-based generic data structures cannot be as efficient as QBind to offer runtime 
  choice of final data format
- The worst write results are obtained with the "Phone" dataset which conveniently uses QMetaObject stored properties and gathers 
  QMetaEnum names to make values more explicit in Text and Xml formats. It also impacts negatively overall Cbor performance that 
  would be 6x faster than QDebug without taking into account this use case (and even faster than QCborStreamWriter)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
463
464

![Benchmark Read/Write results](readwrite.PNG)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
465
466
467

## Conclusion

468
469
QBind is a (de)serialization mechanism that can be used on top of Qt, enabling convenient and efficient data transfers without
loss among a lot of Qt data types and formats including Json, Cbor, QVariant..., QModel... QMetaObject, replacing **a lot** of
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
470
471
format-specific code with a few `T::bind()` or `QBind<T>::bind()` methods (where T can be defined by user, Qt, or a third-part).
It can reuse QDebug and QDataStream << and >> operators, although this requires more visible templates and loses the ability 
472
to switch format at runtime. It can also evolve to something akin to Python list comprehensions for ad-hoc bind operations.
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
473
474
475

Integrated to Qt, QBind would enable the addition of a new kind of tracing facility, as dynamic and convenient as QDebug,
and practically as efficient as Qt tracegen tool. Many existing QDebug tracepoints could be switched transparently
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
476
to the new facility. This would allow going beyond debugging or performance analysis, and tackling complex software issues
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
477
in the field involving multiple tracepoints or traces with adequate tools like Python, or the more academic
478
"Parametric Trace Properties" language [ParTraP](https://gricad-gitlab.univ-grenoble-alpes.fr/modmed/partrap/blob/master/README.md).