Vous avez reçu un message "Your GitLab account has been locked ..." ? Pas d'inquiétude : lisez cet article https://docs.gricad-pages.univ-grenoble-alpes.fr/help/unlock/

README.md 25.2 KB
Newer Older
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
1
# QTransmogrifier, a convenient and efficient (de)serialization mechanism on top of Qt
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
2

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
3
QTransmogrifier was developed to demonstrate the feasibility and advantages of a novel (de)serialization mechanism on top of
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
4
5
6
existing [Qt](http://qt.io) [data formats](https://doc.qt.io/qt-5/topics-data-storage.html#) (Settings, QDataStream, Json, Cbor, Xml) 
or [data model](https://doc.qt.io/qt-5/model-view-programming.html) and generic data types (containers, QVariant, QMetaObject, etc.).
It would greatly simplify writing/reading custom C++ data to/from them while providing more performance than existing approaches.
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
7

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
8
QTransmogrifier requirements below are useful for most (de)serialization use cases. They would be mandatory to implement in Qt a tracing facility
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
9
allowing to analyse how software is used in the field and diagnose complex issues. Effectively:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
10
11
12
13
14
15
- QDebug is conveniently used for debugging. But it may not be enough time- or space- efficient for running software
  in the field, or require too much log parsing work to detect issues from multiple tracepoints or traces.
- LTTng/ETW (or Qt tracegen tool) allow analysing performance using a few statically-defined tracepoints with
  high-performance and structured data. But analysing unexpected uses and issues in the field requires a lot more 
  tracepoints than can be conveniently defined statically.

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
16
> **DISCLAIMER:** QTransmogrifier is not currently unit-tested but provided with a [sample and benchmark](main.cpp). 
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
17
> Also, the implementation is written in a concise style that supported the multiple refactorings but does not help explaining it.
18
> This is especially true for the various QAbstractValue implementations. The README and [DESIGN](DESIGN.md) documentation should be read
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
19
> before trying to understand the implementation.
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
20

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
21
See:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
22
23
24
25
- [The requirements (read/write)](#the-requirements)
- [Some examples](#examples)
- [The results](#results)
- [Our conclusion](#conclusion)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
26
- [The design](DESIGN.md)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
27

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
28
## The requirements
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
29

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
30
31
32
33
34
35
36
37
38
39
* **RW1. Easily customizable** for user-defined and third-part types
  1. do not require writing template specializations for user-defined types
  2. use type system and code completion to guide the user for simple binds (see also W1)
  3. avoid most boiler-plate code, including redundant read/write code (as required with QDataStream << and >>)
  4. override existing bind with custom view types or lambda
* **RW2. Good support of Qt data**:
  1. almost all features of simple data (QJson..., QDataStream, QSettings)
  2. most features of complex data (QCbor..., QXml..., QMetaObject, QModel...)
* **RW3. Allow optional metadata for complex formats** (CBOR tags, XML tags and attributes, QModel* columnNames, etc.)
* **RW4. No restriction on data size**
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
40
41
  (some restrictions may apply with specific implementations that may, e.g. store context for each data structure levels, 
  cache out-of-order data, or even store all the data in memory)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
42

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
43
### Write (serialization)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
44

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
45
46
47
* **W1. Format can be changed at runtime**
  (a compiled tracepoint in a library must be able to generate Json or Cbor as desired by library user)
* **W2. Very fast**
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
48
49
50
  1. similar to QDataStream
  2. same order of magnitude as protobuf/lttng/etl
  3. potentially without dynamic memory allocation or thread locking
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
51
* **W3. Well-formed** data ensured (almost) with low impact on performance
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
52

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
53
### Read (deserialization)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
54
55
56
57
58
59
60
61
62
63
64

* **R1. Encourage standard and explicit formats like Json/Cbor** to:
  - avoid mismatches between actual data and the "schema" at hand (be it a data schema or just code)
  - favor interoperability
* **R2. Support simple, type-by-type, data-schema evolution** like:
  1. adding, deleting or moving named items in a record
  2. adding, or changing from required to optional an item with default value in a sequence
  3. *changing from optional or required to repeated an item (provided it is not itself a sequence) (TBD)*
* **R3. Allow reporting all errors** and mismatches between what was expected and what is read (unless the data format was implicit as below)
* **R4. Allow implicit formats like QDataStream** when the reader knows exactly what to read

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
65
### A notable non-requirement
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
66

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
67
**QTransmogrifier does not _automatically_ support (de)serialization of graphs unlike boost::serialization**. Pointers will be followed and in
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
68
69
case of a graph with a loop, the operation will result in a stack overflow. Hence, the user has to choose an appropriate encoding for his
graph data using some kind of "reference" values. We argue this makes the model translatable to much more data formats like Json since
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
70
it does not mandate native support for references. Moreover, QTransmogrifier supports metadata as an optional way to encode such special values 
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
71
72
73
74
for data formats supporting it like [CBOR value sharing tags](http://cbor.schmorp.de/value-sharing) and XML.

## Examples

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
75
### Extending QTransmogrifier to C++ types
76

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
77
One can rely on Qt reflection to bind a Q_OBJECT or Q_GADGET stored properties using `QBIND_GADGET_WITH_METAOBJECT` macro
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
78
```cpp
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
79
80
81
82
83
84
85
class Phone {
    Q_GADGET
    Q_PROPERTY(Type    type   MEMBER _t)
    Q_PROPERTY(QString number MEMBER _n)
public:
    QBIND_GADGET_WITH_METAOBJECT // making it (de)serializable and printable using reflection

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
86
    enum Type { Unknown=0, Mobile, Home, Office };
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
87
88
89
90
91
92
    Q_ENUM(Type)

    Phone(Type t=Unknown, QString n=QString()) : _t(t), _n(n) {}
private:
    //...
};
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
93
94
```

95
96
One can be much faster and get more control implementing a custom `bind` method using a convenient fluent interface for which 
smart editors will provide code completion similar to that of an XML/JSON editor:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
97
98
99
100
101
```cpp
struct Person
{
    QString firstName, lastName; double height; int age; QVector<QString> phones; QString comments; QList<Person> children;

102
    QValueEnd zap(QValue&& value) { // works with value->mode()==Read as well as Write
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
103
        return value
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
104
105
            .record("Person") // aggregates a usually small number of named items in any order (Person is an optional meta-name for the record that may be ignored)
                .sequence("names") // aggregates any number of items in a fixed order, just to illustrate a basic mapping between struct Person and a more general data schema
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
106
107
108
                    .bind(firstName)
                    .bind( lastName)
                    .out()
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
109
                .bind("height"  ,height  ) // remember named items such as "names" and "height" may appear in any order in a record()
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
110
                .bind("age"     ,age  ,-1) // reads null() or missing values as the default value: -1
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
111
                .bind("phones"  ,phones  ) // recursively calls QTransmogrifier to take care of that part
112
                .bind("comments",comments) // directly calls QAbstractValue since it natively supports QString
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
113
114
115
116
117
                .bind("children",children)
                ; // automagically closes opened record()
    }
};
```
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
118
119

The value argument provides a fluent interface allowing to declaratively bind C++ data to a choice of:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
120
121
* `sequence` of data items in a fixed (meaningful) order, possibly infinite
* `record` of named data items in any order
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
122
* Atomic values like `QString firstName`, `double height`, `int age`
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
123
* Generically supported T values for which a specialized QTransmogrifier<T> is defined like `QVector<QString> phones`, `QList<E> children`
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
124

125
<details>
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
<summary>The same bind can read/write data in various formats with the same data (but varying metadata support)</summary>

```xml
<Person>
	<names>
		<string>John</string>
		<string>Doe</string>
        </names>
	<height>1.75</height>
	<age>18</age>
	<phones><Phone><Home>2</Home><number>+44 1234567</number></Phone></phones>
	<comments>...</comments>
	<children/>
</Person>
```
```json
{
	"names":[
		"John",
		"Doe"
		],
	"height":1.75,
	"age":18,
	"phones":[{"type":2,"number":"+44 1234567"}],
	"comments":"...",
	"children":[]
}
```

</details>
<details>
<summary>This code can be compared with the minimum required to produce the same output using QCborStreamWriter...</summary>
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176

```cpp
QCborStreamWriter s(&b);
s.startMap();
    s.append("names"); s.startArray();
        s.append(person.firstName);
        s.append(person. lastName);
        s.endArray();
    s.append("height"  ); s.append(person.height  );
    s.append("age"     ); s.append(person.age     );
    s.append("phones"  );
        int size = person.phones.size();
        s.startArray(quint64(size));
        for (int i=0; i < size; i++) {
            s.append(person.phones[i]);
        };
        s.endArray();
    s.append("comments"); s.append(person.comments.toUtf8().constData());
    s.append("children"); s.startArray();
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
177
        //...
178
179
180
        s.endArray();
s.endMap();
```
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
181
*NB: QCborValue would be more convenient than QCborStreamWriter but it is much less efficient than QTransmogrifier*
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227

</details>
<details>
<summary>Or the equivalent for reading the exact same output (without regard to variable length collections)...</summary>

```cpp
QCborStreamReader reader(roundtrip.buffer());
if (reader.isMap() &&
    reader.enterContainer() &&
    reader.hasNext() &&
    reader.isString() &&
    reader.readString().data=="names" &&
    reader.readString().status==QCborStreamReader::EndOfString &&
    reader.isArray() &&
    reader.enterContainer() &&
    reader.hasNext() &&
    reader.isString())
    p.firstName=reader.readString().data;
if (reader.readString().status==QCborStreamReader::EndOfString &&
    reader.hasNext() &&
    reader.isString())
    p.lastName=reader.readString().data;
if (reader.readString().status==QCborStreamReader::EndOfString &&
    !reader.hasNext() &&
    reader.leaveContainer() &&
    reader.hasNext() &&
    reader.isString() &&
    reader.readString().data=="height" &&
    reader.readString().status==QCborStreamReader::EndOfString &&
    reader.isDouble())
    p.height=reader.toDouble();
if (reader.next() &&
    reader.hasNext() &&
    reader.isString() &&
    reader.readString().data=="age" &&
    reader.readString().status==QCborStreamReader::EndOfString &&
    !reader.isUnsignedInteger() &&
    reader.isNegativeInteger())
    p.age=int(reader.toInteger());
if (reader.next() &&
    reader.hasNext() &&
    reader.isString() &&
    reader.readString().data=="phones" &&
    reader.readString().status==QCborStreamReader::EndOfString &&
    reader.isArray() &&
    reader.enterContainer() &&
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
228
229
230
    reader.hasNext() /*&& ...*/)
    //...
if (/*... &&*/
231
232
233
234
235
236
237
238
239
240
241
242
243
244
    reader.leaveContainer() &&
    reader.hasNext() &&
    reader.isString() &&
    reader.readString().data=="comments" &&
    reader.readString().status==QCborStreamReader::EndOfString &&
    reader.isString())
    p.comments=reader.readString().data;
    reader.readString().status==QCborStreamReader::EndOfString &&
    !reader.hasNext() &&
    reader.leaveContainer();
```

</details>

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
245
It is also possible to define missing QTransmogrifier specializations for any C++ type using external QTransmogrifier class specializations.
246
247
248
Each specialization must implement at least the bind() method for lvalue references. It may also overload bind() for rvalue
references to support temporaries, and for const lvalue reference to efficiently support copyable types.
```cpp
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
249
template<> struct QTransmogrifier<QColor> {
250
251
    static QValueEnd zap(QValue&& v, QColor&& c) { QColor copy(c); return bind(std::move(v),copy); } // supports writing temporaries and const QColor&
    static QValueEnd zap(QValue&& v, QColor&  c) {
252
253
254
        if (!c.isValid()) {
            return v.null();
        }
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
255
        QRecord r = v.record();
256
257
258
259
260
261
262
263
264
265
        switch(c.spec()) {
            case QColor::Spec::Rgb : r = r.sequence("RGB" ).bind(c.red   ()).bind(c.green        ()).bind(c.blue     ()); break;
            case QColor::Spec::Hsl : r = r.sequence("HSL" ).bind(c.hslHue()).bind(c.hslSaturation()).bind(c.lightness()); break;
            //...
        }
        if (c.alpha()<255) { r = r.bind("alpha",quint8(c.alpha())); }
        return r.bind("base", 255);
    }
};
```
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
266

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
267
### Extending QTransmogrifier to custom formats
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
268

269
Defining a custom format for, say, console output only requires implementing a few `QAbstractValueWriter` abstract methods:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
270
```cpp
271
class MyTextWriter : public QAbstractValueWriter
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
272
273
274
{
    //...
protected:
275
276
    bool trySequence(quint32* =nullptr) { ba->append("[")                      ; return true; }
    bool tryRecord  (quint32* =nullptr) { ba->append("[")                      ; return true; }
277
    bool tryAny     (                 ) { ba->append("*")                      ; return true; }
278
279
280
281
282
283
    bool tryNull    (                 ) {                                        return true; }
    bool tryBind    (   const char* u8) { ba->append( u8)                      ; return true; }

    bool tryItem (QIdentifierLiteral n) { ba->append(" ").append(n).append(":"); return true; }
    bool tryItem (                    ) { ba->append(" ")                      ; return true; }
    bool tryOut  (                    ) { ba->append("]")                      ; return true; }
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
284
285
286
287
    //...
};
```

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
288
With the addition of a shortcut `operator<<`, MyTextWriter can mimic QDebug and replace it in existing code:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
289
290
```cpp
// 1.33333 3.14159 ascii characters are common in QDebug false QColor(ARGB 1, 0.176471, 0, 0.729412)
291
292
QDebug      (&s ) << 1.333333333333f << PI << ascii << false << color ;
MyTextWriter(&ba) << 1.333333333333f << PI << ascii << false << color ;
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
293
294
295
// [ 1.33333 3.14159 ascii characters are common in QDebug false [ RGB:[ 45 0 186] base:255]
```

296
Implementing a custom reader requires implementing a few `QAbstractValueReader` methods. This is always more complex because reading
297
needs to perform much more checks and may have to report transient errors instead of returning `true` as `MyTextWriter` does.
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
298
Both `QAbstractValueWriter` and `QAbstractValueReader` are convenient base classes which implement most of `QAbstractValue` interface used by QTransmogrifier<T> with
299
300
generic text representations of boolean, numbers, etc.

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
301
Since QTransmogrifier supports QVal<_> on the right-hand side, translating between generic data formats or structures is a one-liner:
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
302
303
304
305
306
307
```cpp
// baIn = {"names":["John","Doe"],"height":1.7500000000000002,"age":-1,"phones":["+44 1234567","+44 2345678"],"comments":"","children":[]}
QJsonReader(&baIn).bind(QCborWriter(&baOut).value());
// baOut = 0x bf656e616d65739f644a6f686e63446f65ff66686569676874fa3fe0000063616765206670686f6e65739f6b2b343420313233343536376b2b34342032333435363738ff68636f6d6d656e747360686368696c6472656e9fffff
```

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
308
309
310
311
### Customizing bind operations

Last but not least, one can customize the bind operation to suit specific needs in various ways.
First, Providing in advance some `meta` data allows binding deep C++ data structures in custom ways for use with multi-dimensional Q...View classes:
312
```cpp
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
313
QStandardItemModel matrix, flat, tree, table; 
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
314
315
316
317
QModelWriter<>(&matrix).meta(qmSizes   ,"4,3"      ).bind(QVector<double>{1./3, 2./3, 1./3, 1.,
                                                                          2./3, 1./3, 2./3, 1.,
                                                                          1./3, 2./3, 1./3, 1.,
                                                                          0.  , 0.  , 0.  , 1.});
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
318
QModelWriter<>(&  flat).sequence()                  .with(persons, flatten); // recursive bind function
319
320
QModelWriter<>(&  tree).meta(qmChildren,"children" ).bind(persons);
QModelWriter<>(& table).meta(qmColumns ,"names,age").bind(persons);
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
321
//...
322
```
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
323
![Q...View](qstandardmodel.PNG)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
324

325
326
327
328
329
330
What meta does is guide the way QModelWriter will translate nested `data` structures into the [QAbstractItemModel](https://doc.qt.io/Qt-5/qabstractitemmodel.html#details).
By convention:
- `qmChildren` defines a named item corresponding to a sequence of data with the same type and same [parent](https://doc.qt.io/Qt-5/qabstractitemmodel.html#parent)
- `qmColumns`  defines the ordered set of named items that should be bound to their respective [columns](https://doc.qt.io/Qt-5/qmodelindex.html#column)
- `qmSizes`    defines the bounds of successive dimensions of a N-dimensional row- or column-wise array (where N=2 to account for Q...View limitations)
- `qmName`     allows naming data items for, e.g. XML root element and sequence items
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
331
- ...
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
332

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
333
One can also customize binds using ad-hoc std::function like [flatten](tests/QTransmogrifier/main.cpp#L996) or lambda below (mimicking Python list comprehensions):
334
335
```cpp
QStandardItemModel custom; 
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
336
QModelWriter<>(&custom).sequence().with([&](QSequence&& s) {
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
337
338
    for (auto&& person : persons) {
        s = s.record()
339
340
341
342
                .item("first name")
                    .meta(qmColor, person.age >= 42 ? "green" : "blue")
                    .bind(person.firstName)
                .item("office phone")
343
                    .with([&](QValue&& v) {
344
                        for (auto&& phone : person.phones) {
345
                            if (phone._t == Phone::Office) {
346
                                return v.bind(phone._n);
347
348
                            }
                        }
349
                        return v.null();
350
351
352
                    })
                .out();
    }
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
353
    return std::move(s);
354
355
356
});
```

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
357
## Results
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
358

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
The [samples](samples.txt) shows all data formats written along with error reporting. Here are a few of them:

### "builtin"

QDebug (with only a few decimals), Json, Xml, Cbor (can be decoded on http://cbor.me)
```
1.33333 3.14159 ascii characters are common in QDebug false QColor(ARGB 1, 0.176471, 0, 0.729412) 
```
```json
[1.33333337,3.1415926535897931,"ascii characters are common in QDebug",false,{"RGB":[45,0,186],"base":255}]
```
```xml
<sequence><decimal>1.33333337</decimal><decimal>3.1415926535897931</decimal><string>ascii characters are common in QDebug</string><boolean>false</boolean><record><RGB><integer>45</integer><integer>0</integer><integer>186</integer></RGB><base>255</base></record>
```
```
85fa3faaaaabfb400921fb54442d187825617363696920636861726163746572732061726520636f6d6d6f6e20696e20514465627567f4bf635247429f182d0018baff646261736518ffff
```

### "doubles"

QDebug (with only a few decimals), Json, Xml, Cbor (can be decoded on http://cbor.me)
```
0.333333 0.666667 0.333333 1 0.666667 0.333333 0.666667 1 0.333333 0.666667 0.333333 1 0 0 0 1
```
```json
[0.33333333333333331,0.66666666666666663,0.33333333333333331,1,0.66666666666666663,0.33333333333333331,0.66666666666666663,1,0.33333333333333331,0.66666666666666663,0.33333333333333331,1,0,0,0,1]
```
```xml
<sequence><decimal>0.33333333333333331</decimal><decimal>0.66666666666666663</decimal><decimal>0.33333333333333331</decimal><decimal>1</decimal><decimal>0.66666666666666663</decimal><decimal>0.33333333333333331</decimal><decimal>0.66666666666666663</decimal><decimal>1</decimal><decimal>0.33333333333333331</decimal><decimal>0.66666666666666663</decimal><decimal>0.33333333333333331</decimal><decimal>1</decimal><decimal>0</decimal><decimal>0</decimal><decimal>0</decimal><decimal>1</decimal></sequence>
```
```
90fb3fd5555555555555fb3fe5555555555555fb3fd5555555555555fb3ff0000000000000fb3fe5555555555555fb3fd5555555555555fb3fe5555555555555fb3ff0000000000000fb3fd5555555555555fb3fe5555555555555fb3fd5555555555555fb3ff0000000000000fb0000000000000000fb0000000000000000fb0000000000000000fb3ff0000000000000
```

### "Person"

QDebug (with only a few decimals), Json, Xml, Cbor (can be decoded on http://cbor.me)
```
Person("John", "Doe", 1.75, 18, QVector(), "unicode is likely U+01 \u0001 + U+1F \u001F + U+A4 � U+B0 � U+D8 � U+FF �", ())
```
```json
{"names":["John","Doe"],"height":1.75,"age":18,"phones":[],"comments":"unicode is likely U+01 \u0001 + U+1F \u001F + U+A4 � U+B0 � U+D8 � U+FF �","children":[]}
```
```xml
<Person><names><string>John</string><string>Doe</string></names><height>1.75</height><age>18</age><phones/><comments>unicode is likely U+01  + U+1F  + U+A4 � U+B0 � U+D8 � U+FF �</comments><children/></Person>
```
```
bf656e616d65739f644a6f686e63446f65ff66686569676874fb3ffc00000000000063616765126670686f6e65738068636f6d6d656e74737843756e69636f6465206973206c696b656c7920552b30312001202b20552b3146201f202b20552b413420c2a420552b423020c2b020552b443820c39820552b464620c3bf686368696c6472656e80ff
```

### "Phone"

QDebug (with only a few decimals), Json, Xml, Cbor (can be decoded on http://cbor.me)
```
Phone(Phone::Home, "+44 1234567")
```
```json
{"type":2,"number":"+44 1234567"}
```
```xml
<Phone><Home>2</Home><number>+44 1234567</number></Phone>
```
```
bf647479706502666e756d6265726b2b34342031323334353637ff
```

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
425
> WARNING: The QTransmogrifier ability to adequately bind C++ types and formats is only limited by the set of natively supported C++ types and
426
> the support of meta() by each format. `QAbstractValue` natively supports QDataStream::operator<<() types and could be extended to include
427
428
429
430
> native types which can have efficient binary representations like QUuid. Alternatively, the fluent interface currently contains an
> optional TImpl parameter that could be used to define another set of natively supported C++ types, or removed to simplify the fluent
> interface.

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
431
432
### Write performance

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
433
Overall, QTransmogrifier demonstrates write performance superior to existing Qt classes except QDataStream (which does not meet our W1
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
434
435
436
437
438
and read requirements). Not surprisingly, the performance depends on the kind of C++ data and data format used. Here are some
explanations about the best results:
- QByteArray obviously fails almost all our requirements since it does not even have a global version number as QDataStream does
  but it represents the minimum cost of serializing each dataset as it is equivalent to a few memcpy into a reserved buffer 
  (QByteArray results are similar to protobuf serialization part without dataset construction)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
439
440
441
- QDataStream performs very well and can offer data schema evolution using a global data schema version regularly updated for Qt internal
  needs but users do not control it and will not be able to detect errors on read, so they should refuse to read new schema versions
  (this is the main reason why we are advocating more explicit data formats)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
442
- The fluent interface (which brings convenience and well-formedness guarantees) costs around 20% for non trivial datasets as can 
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
443
  bes seen between Data and QDataStream but this cost is usually compensated by other factors like below
444
- Cbor obtains better results than QCborWriter because the fluent interface cost is more than compensated by working 
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
445
  directly on a QByteArray instead of a QIODevice (around 50% slower)
446
447
448
- Cbor can be up to 10x faster than QDebug for the "builtin" dataset and even 20x faster for the "doubles" dataset (which is not a 
  surprise), but it suffers from the absence of utf16 encoding when more QString are added like in the "Person" dataset (this may be
  solved by defining a utf16 Cbor tag)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
449
- The cost of `Bindable` which allows to compile a tracepoint and choose the trace format at runtime is hardly measurable.
450
- Regarding text formats suitable for display on a console providing more metadata (such as the name of classes and enum values), our
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
451
452
453
454
455
456
457
458
  toy example "TextWriter" obtains better performance than QDebug essentially because:
  1. it works with all source code literals in utf8 instead of QString utf16
  2. it does not use QIODevice to write small chunks (although each tracepoint could send written data to a QIODevice)
  3. it relies on the fluent interface static well-formedness guarantee instead of passing along a format state that users are responsible
     to maintain

![Benchmark Write results](write.PNG)

459
Regarding the worst performances:
460
- Variant which translates C++ types to an in-memory data structure of QVariantList and QVariantMap cannot perform very well
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
461
  (without even storing metadata at all) because of the numerous small allocations that may cost much from time to time
462
- QJsonValue, QCborValue bad performance is explained the same way as Variant above
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
463
- More generally, it seems clear that pointer-based generic data structures cannot be as efficient as QTransmogrifier to offer runtime 
464
465
466
467
  choice of final data format
- The worst write results are obtained with the "Phone" dataset which conveniently uses QMetaObject stored properties and gathers 
  QMetaEnum names to make values more explicit in Text and Xml formats. It also impacts negatively overall Cbor performance that 
  would be 6x faster than QDebug without taking into account this use case (and even faster than QCborStreamWriter)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
468
469

![Benchmark Read/Write results](readwrite.PNG)
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
470
471
472

## Conclusion

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
473
QTransmogrifier is a (de)serialization mechanism that can be used on top of Qt, enabling convenient and efficient data transfers without
474
loss among a lot of Qt data types and formats including Json, Cbor, QVariant..., QModel... QMetaObject, replacing **a lot** of
475
format-specific code with a few `T::zap()` or `QTransmogrifier<T>::zap()` methods (where T can be defined by user, Qt, or a third-part).
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
476
It can reuse QDebug and QDataStream << and >> operators, although this requires more visible templates and loses the ability 
477
to switch format at runtime. It can also evolve to something akin to Python list comprehensions for ad-hoc bind operations.
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
478

EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
479
Integrated to Qt, QTransmogrifier would enable the addition of a new kind of tracing facility, as dynamic and convenient as QDebug,
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
480
and practically as efficient as Qt tracegen tool. Many existing QDebug tracepoints could be switched transparently
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
481
to the new facility. This would allow going beyond debugging or performance analysis, and tackling complex software issues
EXT Arnaud Clère's avatar
EXT Arnaud Clère committed
482
in the field involving multiple tracepoints or traces with adequate tools like Python, or the more academic
483
"Parametric Trace Properties" language [ParTraP](https://gricad-gitlab.univ-grenoble-alpes.fr/modmed/partrap/blob/master/README.md).