OD-Format Module

The OD-Format module was designed by RobMacLachlan and implements the Object Description Format, a binary format for storing persistent objects. This is used by the *.lib.du files emitted by d2c, though it might be potentially useful elsewhere.

The file gd/src/d2c/compiler/base/od-format.dylan contains an extensive discussion the module's goals and data format.

The basic requirement of the "Object Description Format" (ODF) is to allow an open-ended variety of objects to be dumped into a file, where:

In order to make the design interesting, the ODF attempts to satisfy various incompatible requirements:

We should consider making this into a standalone library at some point in the future.

Things that can be the name of a data unit.

<symbol>

<data-unit-name> is a <symbol> because is must be comparible with \==.

A constant for begin-dumping

0

This is currently the only type of OD format allowed. To build another format (e.g. DOOD), a new exported constant must be added to OD-Format and the file extension must be added to $unit-type-strings (an internal constant to OD-Format).

Returns a dump state used by other OD functions.

name	An instance of <data-unit-name>. The thing to dump.

type	An instance of <integer>. The "unit type code"

where:

An instance of false-or(<string>). Dump file name.

dump-state

An instance of <dump-state>.

Creates a state (which acts very much like a buffer) for other dumping functions to use to build the dump file in memory. The end-dumping function actually writes the built information to file.

Flushes the dump state to file.

state

An instance of <dump-state>.

None

Actually output the file.

Write an object to the dump buffer.

object

An instance of <object>. The object to dump.

buffer

An instance of <dump-buffer>. The in-memory buffer into which the object should be dumped.

None

This routine is overloaded to dump individual types objects. Methods for primitive Dylan ™ types can be found in the section called “The Dylan-Dump Module”.

Holder of the in-progress object dump.

<dump-buffer>

None

Contains all information about the current object-dump in progress. It is illegal to use a dump-state after it has been passed to end-dumping. All instances of <dump-state> are created by calls to begin-dumping.

Binary to in-memory instance transformer.

<object>

initialize-from:

An instance of <object>. Unless #f, initialize instance's dispatch information from this dispatcher.

A dispatcher is an object that tells the loader how to convert binary data into Dylan ™ objects.

Default dispatcher used for initialization.

<dispatcher>

Dylan-dump loads this variable with dispatching information for several commonly-used Dylan ™ types (see the section called “The Dylan-Dump Module”).

Paths to du files.

<sequence>

#[]

This is used by find-data-unit to locate the du file to load.

A hint of where to find a data unit.

<byte-string>

This is currently an absolute(?) pathname.

A portable, full-sized <integer>.

<integer>

It seems mindy has a limited size for <integer>, so, for mindy, <word> is a <general-integer> to allow for proper bit manipulations that do not clobber the value via overflow.

Locates then loads a du file.

name	An instance of <data-unit-name>.

type	An instance of <integer>. (Currently only $library-summary-unit-type allowed).

location-hint:

An instance of false-or(<location-hint>). File name for the dump file; used if the data unit cannot be found in the current *data-unit-search-path*.

check-hash:

An instance of false-or(<word>). Can be used to detect version mismatches between data units (used internally as expected-hash)

dispatcher:

An instance of <dispatcher>. Provides the dispatcher to be used for the loading process.

res	An instance of <data-unit>. Contains the resulting transformed objects from the du file.

Find-data-unit, when given a data unit name and type, returns the corresponding data unit. This is the main entry for causing loading to happen. Also used recursively.

To write a dumper for a class, one needs to think of a name for the class (usually just take the <>'s off the class name), register the name with register-object-id, and define a dumper. If your data structure contains any object sharing or circular usages, also see the section called “Circular Usage” and the section called “Object Sharing”. For automatic generation of object dumpers and loaders, see the section called “Autodump”.

Associates a number for a type for the OD dumping functions.

name	An instance of <symbol>. The name of the type.

id	An instance of <integer>. The associated integral value.

None

Equate the name with the id. This function is not exported; rather, all register-object-id calls must occur in the OD-Format module.

Of course, this requirement makes it really tough for extending OD-Format for user defined types ...

Register a method for reading a dumped object back into memory.

dispatcher

An instance of <dispatcher>. When loading .lib.du files, this is the *default-dispatcher*

name	An instance of <symbol>. The method will be activated when the loader encounters the type registered with register-object-id in the du file.

func	An instance of <function>. The loader method.

None

This function is the “other half” of dump-od. It is called to register a function for reloading dumped objects, which has the signature:

method(state :: <load-state>) => res :: <object-type-read-in>
	       

If you use add-od-loader, you will need to define a corresponding method on dump-od (see the section called “Dumping Objects”).

Convenience method to add both a loader and dumper.

name	An instance of <symbol>. Type name.

dispatcher

An instance of <dispatcher>. Where the loader and dumper are being added.

obj-class

An instance of <class>. Must be the direct class of the instances we are to dump (and of course, must also be instantiable via make.)

slots

An instance of <list>. The slots' descriptions of the class.

load-external:

An instance of <boolean>.

dumper-only:

An instance of <boolean>. If #t, do not create the loader.

load-side-effect:

An instance of false-or(<function>).

dump-side-effect:

An instance of false-or(<function>).

None

A convenient way to define both a loader and a dumper for objects of class obj-class. Slots is a list which is in this format:

#(slot-accessor-function-1, init-keyword-1, setter-function-1,
  slot-accessor-function-2, init-keyword-2, setter-function-2,
  ...)
	      

Note that the stuff is implicitly grouped in clumps of 3. Either of the init-keyword or the setter may be #f, but not both. The "make dumper" requests backpatching when it encounters an unresolved forward reference (due to circularities). Any slot which might have a forward reference must have a setter function (it can also have an init-keyword, which will be used if possible).

A "make dumper" may be used when an object can be reconstructed from some accessor values by reinstatiating the object with make and backpatching some slots. This is pretty general, since the accessor and setter function are not required to be slot accessors and setters.

[Description of Autodump coming soon]

In the dump files, objects contain the following parts: a header, raw data, sub-objects, and an end-of-object marker. The header is a single machine word which contains the class of the object. (Actually, it contains the object-id, which is found by registry look-up.) The header also indicates if there is any raw data and sub-objects. The raw data section is zero or more machine words which only have meaning to the object's loader. If raw data is present, it is preceded by a byte-count (which is a single machine word). If there are sub-objects, they come next. The last object in a sub-object list is a magical $end-object, which denotes both the end of the sub-object list and the last word of the current object.

For more detail on the binary representation and on issues such as endian-ness and machine word size, see the section called “Binary Representation”.

When an object is loaded, the loading code reads in the header. The object class id is extracted from the header. The dispatcher then provides a load function based on the class id, which reads the remainder of the data associated with the object and returns the reconstructed object.

When adding a method to dump-od use the methods described here to push out the objects correctly.

Dump an object definition header.

name	An instance of <symbol>.

buf	An instance of <dump-buffer>. Probably best to use the <dump-state> instance passed into dump-od

subobjects:

An instance of <object>. #t means this object has subobjects

raw-data:

An instance of <object>. The Raw-Format field in an object-definition header encodes information needed to byte/word format translation, and in the degenerate case, indicates there is no raw data at all. Allowable format codes are $odf-no-raw-data-format, $odf-byte-raw-data-format, $odf-16bit-raw-data-format, $odf-32bit-raw-data-format, $odf-64bit-raw-data-format, $odf-untranslatable-raw-data-format, and $odf-word-raw-data-format (which is 32 or 64 as appropriate, used for <word> in format pseudo-objects).

None

The first dump call in dump-op for simple objects, this function dumps the object's header to the buffer. If one is dumping compound objects, first save the position of the buf before calling this method with some code like this:

define method dump-od(obj :: <ratio>, buf :: <dump-state>) => ();
  let start-pos = buf.current-pos;
  dump-definition-header(#"ratio", buf, subobjects: #t);
  dump-od(obj.numerator, buf);
  dump-od(obj.denominator, buf);
  dump-end-entry(start-pos, buf);
end method dump-od;
	     

This is necessary so that dump-end-entry (see the next documented method) writes out the terminal correctly.

Dump an "end of object" marker. This is only necessary if there are subobjects.

start-posn

An instance of <integer>. Where the object began.

buf	An instance of <dump-buffer>. Usually the <dump-state> instance passed into dump-od.

None

Before dumping the compound object, be sure to save the buffer's current position (as in the sample code for dump-definition-header) because this method needs that information.

Dump an object that has only a constant number of subobjects (no raw data).

name	An instance of <symbol>. The instance's class name.

buf	An instance of <dump-buffer>. Usually the <dump-state> instance passed into dump-od.

slots

An instance of <object>. The values of the slots of this object.

None

This method provides a simple way to dump objects of known size, so using this method, writing dump-od for <ratio> would be:

define method dump-od(obj :: <ratio>, buf :: <dump-state>) => ();
  dump-simple-object(#"ratio", buf, obj.numerator, obj.denominator);
end method dump-od;
	   

Dump an integer into the buffer as one word.

obj	An instance of <word>. The integer to dump.

buf	An instance of <dump-buffer>.

None

This dumps an <integer> as part of of an object. If you want to dump an <integer> as an <integer>, use the dump-od on <integer> in the section called “The Dylan-Dump Module”.

Dumps a collection into the buffer.

obj	An instance of <collection>. The collection to dump.

bsize

An instance of <integer>. The number of elements in the collection.

buf	An instance of <dump-buffer>.

None

This take a collection, obj, and dumps its contents to buf. This method is usually called in connection with dumping a complex object, for example, dump-od for <byte-string> in the section called “The Dylan-Dump Module” dumps strings thusly:

define method dump-od(obj :: <byte-string>, buf :: <dump-buffer>) => ();
  dump-definition-header(#"byte-string", buf,
  			 raw-data: $odf-byte-raw-data-format);
  dump-raw-data(obj, obj.size, buf);
end method;
	     

The basic concept of writing a loader is to read in the information you wrote out with the method on dump-od. This process (writing loaders) is a bit different than writing dumpers, though. Here, you provide a method to add-od-loader (described in the section called “Defining Simple Dumpers and Loaders”) That is activated by a <dispatcher> instance when it encounters the object in the dump file to load.

A loader on <byte-string> (the corresponding dump-od method is found in the section called “Functions for Writing Dumpers”) would look like this:

add-od-loader(*default-dispatcher*, #"byte-string", 
  method (state :: <load-state>)
   => res :: <byte-string>;
    let next = state.od-next;
    let bsize = buffer-word(state.od-buffer, next);
    state.od-next := next + $word-bytes;
    let res = make(<byte-string>, size: bsize);
    load-raw-data(res, bsize, state);
    res;
  end method
);
	 

(In this case, the call to add-od-loader is made at the top-level in dylan-dump.dylan.) The functions used to build the dispatch method are described below.

A load-state is an object that describes the current state of the loading process. Among other things, it contains an input <stream> and the buffer to the stream.

<object>

stream:

An instance of <stream>. The stream we're reading from. Only used for refilling the buffer. This slot is accessed via the od-stream method.

buffer:

An instance of <buffer>. State of the stream buffer, which we hold for the duration of the load. It is accessed via the od-buffer method.

next:

An instance of <buffer-index>. The index of the state's stream; it is accessed via the od-next method and can be set (changed) using the od-next-setter method.

end:	An instance of <buffer-index>. The end-index for the buffer and stream. This slot is accessed via the od-end method.

position-offset:

An instance of <integer>. When added to the od-next pointer, this offset yields the byte offset of our current position from the data-unit start. This must be updated whenever we refill the buffer.

dispatcher:

An instance of <dispatcher>.

A <load-state> object is passed into the dispatching method to reconstruct a dumped object (so, you do not need to make a <load-state> instance). Use that object in conjuction with the functions described below.

An end-of-subobjects marker.

<end-object>

A magical object which is always the last sub-object in an object.

Read an object header and invoke the appropriate loader. Return the object that the loader returns.

state

An instance of <load-state>.

res	An instance of <object>. The reconstructed object.

Start loading some objects from a load-state. Dispatches to an appropriate loader method depending on the dispatcher and the entry type. Returns the object.

For example, to load a <ratio> instance, build a dispatching method like this:

add-od-loader(*default-dispatcher*, #"ratio", 
  method (state :: <load-state>) => res :: <ratio>;
    
    let npart = load-object-dispatch(state);
    let dpart = load-object-dispatch(state);
    assert-end-object(state);
    ratio(npart, dpart);
  end method
);
	     

Loads data into a parameter.

res	An instance of <byte-string>. A pre-allocated buffer where the raw data gets loaded.

elsize

An instance of <integer>. Size of the buffer (or the maximum bytes to load).

state

An instance of <load-state>.

nnext

An instance of <buffer-index>. The position in the <load-state>'s buffer after loading the raw data.

Loads raw data into res. It is up to the dispatching method to interpret that loaded raw data in order to reconstruct the object's state.

Usually a call to load-raw-data is to load a string, so reconstruction is easy (see the example at the top of this section). One thing of note is that this method is not functional: res is not returned, but is a destructively modified parameter.

Returns a vector of an object's subobjects.

state

An instance of <load-state>.

size-hint

An instance of false-or(<integer>). If a number, then the number of subobjects MUST match.

res	An instance of <simple-object-vector>. The subobjects of the object.

Loads the dumped subobjects into a vector. Note that the subobjects may be unresolved when loaded with this function ... it is the responsibility of the user of this function to resolve those objects (see the section called “Circular Usage” for more information on functions for object resolution such as obj-resolved?, actual-obj, and request-backpatch).

So, getting a <vector> from this function still requires programming effort, e.g.:

add-od-loader(*default-dispatcher*, #"simple-object-vector", 
  method (state :: <load-state>) => res :: <simple-object-vector>;
    let contents = load-subobjects-vector(state, #f);
    let rsize = contents.size;
    let res = make(<simple-object-vector>, size: rsize);
    for (i from 0 below rsize)
      let el = contents[i];
      if (el.obj-resolved?)
        res[i] := el.actual-obj;
      else
        res[i] := el;
	request-backpatch(el, method (actual) res[i] := actual end);
      end;
    end;
    res;
  end method
);
	     

Fortunately, the Dylan-dump module provides this dispatching method. See the section called “The Dylan-Dump Module”.

Loads the only subobject of this object.

state

An instance of <load-state>.

res	An instance of <object>. The resolved subobject.

Load a one-slot object. The value is returned, and we check that there was in fact only one subobject.

Reads from the buffer, and signals an error if the next thing read isn't an end-of-object marker.

state

An instance of <load-state>.

None

Ensure the subobject list ends with the end-of-object token.

To deal with circular object usage graphs, several mechanisms are employed. Objects can be dumped as forward references and defined later. The loader for the object containing the forward reference has the option of delaying object creation until the forward referenced object is loaded, or requesting that the object be backpatched when the forward referenced object is finally loaded.

The following descriptions TBD:

By default, it is assumed that the object being dumped has no object sharing, i.e., for all sub-objects the ref-count is one. For any object that has a ref-count greater than one, that object must be a sub-class of <identity-preserving-mixin>. (If it is ok to dump the object multiple times and load it back as separate objects, then this superclass may be omitted.)

The following descriptions TBD:

To allow efficiency as well as potential interchange, objects are described by a word sequence which vaguely resembles the in-core representation. The word size is implementation defined, and dumper/loader methods dump/load data as words having the native byte ordering. The format is defined so that any necessary byte-swapping and word-size adjustment can be done as a pre-pass filter to the actual loader. The format is described in detail at the top of od-format.dylan.

The implication of the above is that data-unit files produced from OD-Format are not portable.

    0: <Header? = #b1, Etype = #b00, Subobjects? = #b0, Raw-Format = #x1,
	Class-ID = byte-string-odf-id>
    1: string byte-count
    2[ceiling(byte-count, word-bytes)]: 
       <string-chars: *>
	   

    #x81000666
    #x00000003
    #x466F6F00
	   

    0: <Header? = #b1, Etype = #b00, Subobjects? = #b1, Raw-Format = #x0,
        Raw-Size = #x00, Class-ID = list-odf-id>

    1[N]: <header of subobject 0, subobject data>
       ...

    M: <Header? = #b1, Etype = #b01, Start-Offset: * = M>
	   

    0: #x90000667
    1: #x81000666
    2: #x00000003
    4: #x466F6F00
    5: #x81000666
    6: #x00000003
    9: #x42617200
    7: #xA0000007