* Previously each read and write would 1:1 become a send or recv call.
* Now we buffer writes and send in batch (or when a chunk finishes),
and every time we read we try to non-blocking read more data to fill
the buffer, to allow batching reads where possible without blocking
on data that will never come.
* Previously once we started loading a capture we'd blindly continue
until we loaded it (and then it's assumed to be successful), or we
crash.
* Now errors can be reported during serialisation and bubbled up to
abort the file load process. The next steps are to add error checking
in each function serialise before doing any replay calls to the API
with potentially corrupt data, and on top of that catching API-only
errors when the serialisation is (seemingly) fine, and propagating
those in a reasonable way.
* We also harden the serialisation a bit so that if it reads an
obviously invalid byte length for a buffer or array count, it won't
continue. It's still not perfect as the sizes could still be large and
invalid but within range, but it should catch the worst cases.
* fflush is a no-op wrapper but nice to have for consistency.
* ftruncateat will take a file read-write handle and truncate it at a
given length. Useful if re-writing a file and the new data ends up
being shorter than the oldd ata.
* Move in addition to Copy, and success/fail bool return on both.
* If the external code has a NULL pointer and doesn't want to allocate
but we are exporting buffer data, we still need to read the data
somewhere so make a temporary allocation to read it into.
* When opening a capture file, a format is now available to allow
easy import from another format without a completely different
interface. Only rdc files can be replayed, but any other file can
load and access structured data through the same interface.
* The replay initialisation and capture writing interfaces also use the
RDCFile instead of passing filenames or Serialisers around directly.
Driver initialisation parameters are now entirely private, and don't
need to be exposed - any agnostic metadata like thumbnail, driver, etc
are all accessed via the RDCFile container itself.
* Callstack resolution is now part of the container file, not the
back-end via way of its Serialiser.
* Importers/Exporters to other non-RDC formats are registered in a
similar way to replay/remote drivers.
* It is also then possible to construct an RDC file from thin air, by
creating an empty RDCFile container and filling it with data, then
requesting it to be written to disk.
* The new system contains the ability to export serialised data to a
structured form in memory - and conversion back to serialised bytes.
* This will allow offline transformations/visualisation of capture files
as well as more rich representations of API calls in the UI.
* Likewise it enables a number of optimisations such as the ability to
write straight from mapped API memory to disk via a compressor,
without any intermediate copies.
* The new serialiser system will not handle the container file itself,
so we've separated it out to deal in sections and stream I/O to read
from and write to sections.
* We also add the ability to choose the compression scheme for a section
instead of always using LZ4. LZ4 will still be used on the fly for
fast write performance, but then RDC files can be re-compressed for
savings offline.
* This I/O will form the basis of the new serialiser - it will simply
read to or write from one of these I/O streams. Then that stream can
come from a file, go to a memory buffer, or go through a compressor
or decompressor transparently.
* It also allows a unified way of writing over sockets instead of
needing special socket helper functions.
* With this commit, the code isn't used aside from in tests.
* Note that while this is public and uses std::string, because it's a
template with specialisations in a .inl the string never crosses a
module boundary - each including module has its own implementation.
* This will be used as part of the upcoming serialisation refactor.
* Some POD structs are still given ToStr implementations as we haven't
yet switched over the serialisation system to expect all structs to
have serialise functions.
* Since these types are more prevalent than originally designed, it
makes more sense to remove the namespace for ease of typing/naming.
* Also add a specialised type 'bytebuf' for an array of bytes.
* This makes mapping easier to SWIG since there's no special casing for
namespaced arrays. Especially so for nested cases like
rdctype::array<rdctype::str> -> rdcarray<rdcstr>
* For the most part the interface is stl-compatible, but we have a few
little changes of our own for convenience.
* This class is still needed after deleting the C# UI, because we don't
want to pass C++ stl structs over module boundaries and possibly run
into hard to diagnose incompatibilities.
* The +1 for internal NULL terminator must be done internally, otherwise
we end up with "foobar" being a 7-character string of "foobar\0". If
this is then re-serialised we add more and more null terminators.
* There's still an issue lurking that the compressed size is stored as
32-bit on disk, but at least now we don't store the uncompressed size
as 64-bit but track it as 32-bit.
* The serialisation is going to change Soon(tm) anyway so this
workaround should be enough for now, since hopefully compressed size
is comfortably below uncompressed size.
* This gives a little nicer syntax, a bit better type safety, and also
reflects better for SWIG bindings. Overall it's a minor change but
better.
* We don't update the C# UI at all, since it's soon to be removed and
not worth the effort/code churn.
* For now so we're ABI compatible with C#, all enums are uint32_t, but
that is an obvious optimisation in future to reduce struct packing.
* We avoid 'None' as an enum value, because it's a reserved word in
python so will cause problems generating bindings.
baldurk