About DIME...
"Direct
Internet Message Encapsulation"
(DIME) is "... a lightweight, binary message format designed to encapsulate one
or more application-defined payloads into a single message construct."
DIME lends itself to the packaging of a SOAP message with "attachments", or
additional data payloads that may be referenced from within the SOAP message.
The convention that the White Mesa SOAP Implementation follows for this purpose
is defined in the internet draft
WS-Attachments
.
The primary reason for using attachments is to send binary data outside of the
SOAP message envelope so that the overhead of encoding/decoding the data (e.g.
base64) can be avoided.
This is an experimental implementation!
Highlights:
-
DIME support is integrated into the SOAP server, and DIME capable C++
clients are furnished.
-
DIME attachments are supported for RPC or document style operations.
-
DIME packaged request messages are detected and parsed automatically by the
SOAP server, with the attachments made available as appropriate to the
application. The presence of DIME attachments placed in the response message by
the application is automatically detected and results in the generation of a
DIME packaged response. The reverse holds for the DIME client.
-
Generation of href/id attributes is automatic in the case of RPC operations.
-
Attachments may be referenced from the SOAP message Body or from Header entry
elements.
-
Attachments may be of any size, although disk space availability imposes a
practical limit.
-
There is no dependency on any WSDL extensions for DIME at this time, although
support is planned for the proposed
WSDL Extension for SOAP in DIME in the future.
-
Support is provided for DIME "options", but is limited to doc/literal
applications only. Clients and service implementations may set "options" on
payloads they send, and read options on payloads they receive. Options are set
on a per payload basis, and are read from/written to the first DIME record of
the payload only.
-
A reusable C++ DIME parser/packager implementation
is included.
-
Demonstration SOAP/DIME service implementations and
a test client are provided. The test client and its
matching service are configured and ready to run upon installation of the White
Mesa SOAP Server.
So how does it work?
HTTP request messages received by the White Mesa SOAP server are dispatched on
the basis of their media type, found in the HTTP "Content-Type" header. Those
with type "application/dime" are assumed to be SOAP messages packaged in DIME
per the "Encapsulating SOAP in DIME" draft, and are fed into a DIME parser. The
SOAP message envelope, found in the primary DIME payload, is processed in the
normal way, with any attachments stored for use by the SOAP application, which
consists of a Body processor and zero or more Header processors. It is
important to note that data to be sent as a DIME attachment must be typed as
XML Schema "base64Binary" or "hexBinary" types in the WSDL service description
or supporting schema. Also, there is no symmetry requirement, that is, a normal
SOAP request message may result in a DIME packaged response, and vice versa.
In the case of SOAP Body processing, the manner in which the DIME payloads are
made available to the application depends on whether the SOAP operation is
using the RPC convention, or document/literal messaging:
-
RPC Body
In this case any [in] or [in, out] method parameters representing attachments
will contain an IUnknown interface pointer. This pointer is used to
access the IStorage interface of a COM compound storage object which
contains the attachment octet stream. The SOAP engine handles the task of
marshalling this storage object pointer onto the call stack automatically, by
resolving the href value contained in the accessor element for the
parameter in the SOAP message. Similarly, if the Body handler wishes to pass an
[out] parameter value or the return value as a DIME attachment, the data is
passed out in the form of another IStorage interface pointer. In this
case the SOAP engine will automatically assign an id value (a uuid) to
the attachment and reference it from the accessor element in the SOAP response
envelope. DIME packaging will then be used for the response message. If no
attachments are passed out, then the SOAP response message is not DIME
packaged. Binary data passed out as SAFEARRAY of bytes will not be
treated as a DIME attachment. In fact the application may choose at runtime
whether to return binary data as a DIME attachment or not, the choice dictating
whether an IUnknown or a SAFEARRAY pointer is passed back in the
return value or an [out] parameter. An example of an RPC application using DIME
attachments is found in this demo app source
code. The DIMEPayload class
described below may be used to simplify the task of working with attachments;
it serves as a wrapper for IStorage interface pointers.
-
Document/literal Body
In this case it is the responsibility of the Body processor to parse the SOAP
message and extract any SOAP attachment id values. The mechanism by
which attachments are referenced is application defined in this case. Having
obtained any needed id values, the application may then query the Message
object for the payloads, invoking the GetDIMEAttachmentReq() method.
Given an id value, this method returns the following if a DIME payload is found
with this id:
-
The type of the payload.
-
The type name format identifier for the payload.
-
An IUnknown pointer.
-
A SAFEARRAY containing any DIME "options" contained in the first record of the
payload.
As before, the IUnknown pointer is used to obtain an IStorage interface
for the storage object containing the data. The DIMEPayload
class described below may be used to simplify the task of working with
attachments; it serves as a wrapper for IStorage interface pointers. If
the application wishes to include DIME payloads in the response message, it
would typically prepare a response message in the usual way, including any
references to attachments. The attachment data is then loaded into a storage
object whose IStorage interface pointer is passed to the Message object
in an invocation of its AddDIMEAttachmentResp() method. Additional
arguments to this method are the id value, the type name value, type name
format value, and a SAFEARRAY containing any DIME "options". An example of this
type of processing is found in this demo app
source code.
In the case of Header processing, the manner in which DIME payloads are made
available to the application depends on whether the Header entry is SOAP
encoded or is literal XML:
-
SOAP encoded header entries
In this case the Header processor will query the Message object in the usual
way for the header entry of interest, invoking the GetHeaderElem() method.
If the header entry is found, a VARIANT is returned containing its deserialized
value. If this represents a DIME attachment, the value will be an IUnknown
pointer which is used to access the storage object containing the data. Binary
data passed out as SAFEARRAY of bytes will not be treated as a DIME
attachment. In fact the Header processor may choose at runtime whether to
return binary data as a DIME attachment or not, the choice dictating whether an IUnknown
or a SAFEARRAY pointer is passed back in the return value or an [out]
parameter. The DIMEPayload class described
below may be used to simplify the task of working with attachments; it serves
as a wrapper for IStorage interface pointers. Similarly, if the header
processor wishes to insert a SOAP encoded header entry whose data is passed
back as a DIME attachment, the data is passed out in the form of another IStorage
interface pointer. This is done by placing the pointer into a VARIANT which is
then passed out in an invocation of the Message objects AddHeaderElem() method.
An example of this type of processing is found in this
demo header processor source code.
-
Literal XML header entries
In this case it is the responsibility of the Header processor to parse the SOAP
Header entry and extract any SOAP attachment id values. The mechanism by
which attachments are referenced is application defined in this case. Having
obtained any such values, the application may then query the Message object for
the payloads, invoking the GetDIMEAttachmentReq() method. This works as
described above in the document/literal body processing case. If the header
processor wishes to insert a literal XML header entry whose data is passed back
as a DIME attachment, it would typically prepare a response header entry in the
usual way, including any references to attachments, and add it to the response
message using the Message object method AddHeaderElemLit(). The
attachment data is then loaded into a storage object whose IStorage interface
pointer is passed to the Message object in an invocation of its AddDIMEAttachmentResp()
method. Additional arguments to this method are the id value, the type name
value, and type name format value. As usual, the DIMEPayload
class described below may be used to simplify the task of working with
attachments; it serves as a wrapper for IStorage interface pointers.
DIME Parser/Packager Implementation
The White Mesa DIME support is based on the use of a DIME parser which is
responsible for reading and generating DIME messages. The parser is an instance
of the DIMEParser class. It consists of:
-
A container for DIME payloads.
-
A serializer for generating a DIME package from one or more payloads.
-
A deserializer for reading a DIME package and storing the payloads it
contains.
The serializer is implemented as a state machine, and is an instance of the
DIMESerializer class. The serializer is DIME "record" oriented,
intended for writing into a fixed size buffer. Typically the record size
specified when the serializer is initialized is matched to the buffer size that
will be used. Records are then pulled from it one at at time until
serialization is complete. Any payloads whose size cannot be accomodated in a
single record will be sent "chunked", divided into a series of records, as
allowed by the DIME spec. If desired, the total size of the DIME package can be
obtained through a call to "CalcSerialLength()" in advance of serialization.
The deserializer is implemented as a state machine, and is an instance of the
DIMEDeserializer class. The deserializer operates on blocks of input
data passed to it (of any size), and stores received payloads as they are
extracted from the DIME package. It is intended for use in applications
involving event driven I/O processing.
DIME payloads are contained in instances of the DIMEPayload
class. This serves as a wrapper around a COM IStorage interface pointer,
and also contains these DIME specific attributes:
-
The id of the payload.
-
The type name format of the payload.
-
The type name of the payload, e.g. "text/xml".
This class is very useful in application code for wrapping IStorage interface
pointers. Methods are provided to simplify the operations of reading and
writing data to the storage object. The data is stored in an anonymous stream
object which is accessed via the IStorage interface. The storage object
is associated with a temporary compound storage file and is programmed to
delete the file at the end of its lifetime. So far, this has proven to be a
reasonably efficient arrangement, with a huge data capacity.
To create a DIME package, a sender will typically perform the following steps:
-
Create an instance of the DIMEParser class.
-
Initialize it with the desired record size, e.g. 4096 bytes. This is done by a
call to the DIME parsers "SerializeInit()" method.
-
Add payloads using one of the DIME parsers "AddPayload()" methods. There are 3
such methods, in one an IStorage interface pointer is passed in, in the second
a buffer of bytes is passed in, and in the third a disk file name is passed in.
-
Serialize the DIME package into a buffer, one record at a time. This is done by
repeated calls to the DIME parsers "Serialize()" method. The buffer must be
large enough to hold a complete record, whose size was specified in the
initialization call. The records are then passed along to whatever output
stream is in use, e.g. passed into a TCP socket "send" call.
To process an incoming message stream, a receiver will typically perform the
following steps:
-
Create an instance of the DIMEParser class.
-
Initialize it with a call to the "DeserializeInit()" method.
-
Read data from the stream and pass it into a call to the DIMEParsers
"Deserialize()" method.
-
To test whether or not the DIME package has been received complete, a call to
the "DeserializationFinished()" method may be used. It is harmless to make
calls to "Deserialize()" after the DIME package deserialization is complete,
the data is ignored.
-
When processing is complete, access to the payloads container is obtained by a
call to the "GetPayloads()" method of the DIMEParser.
Here is a contrived example of sending:
string filename("c:\\someFile.txt");
string id("uuid:550BA703-B918-4E75-9BFF-2B8CC08390EC");
string typename("text/plain");
unsigned long tnf = 1; // media type format
DIMEParser p;
p.SerializeInit(4096);
p.AddPayload
p.AddPayload(filename,
id,
typename,
tnf);
char buffer[4096];
unsigned long numbytes;
bool unfinished = true;
while (unfinished)
{
try
{
unfinished = p.Serialize(buffer, 4096, &numbytes);
SendFarAway(buffer, numbytes);
}
catch (DIMEParserException *e)
{
// do something
}
}
And here is another for receiving:
DIMEParser p;
p.DeserializeInit();
bool finished = false;
while (!finished)
{
finished = ReadSomeData(buffer, &numbytes);
try
{
p.Deserialize(buffer, numbytes);
}
catch (DIMEParserException *e)
{
// do something
}
}
vector::iterator iter;
for (iter = p.GetPayloads().begin(); iter != p.GetPayloads().end(); iter++)
{
char buffer[4096];
unsigned long numbytes;
DIMEPayload *payload = *iter;
payload->OpenForRead();
do
{
payload->Read(buffer, 4096, numbytes);
DoSomethingWithData(buffer, numbytes);
} while (numbytes == 4096);
payload->Close();
}
Demo Services
Four demo services are supplied:
-
A document/literal service described in the WSDL document found
here. Note that this doc uses the proposed
WSDL Extension for SOAP in DIME.
This service implements the SOAPBuilders "Round 4" DIME testing doc/literal
interface. It echoes any DIME payloads received in request message back in the
response message, including any DIME "options" included in the first record of
the payload.
-
An RPC service described in the WSDL document found
here. Note that this doc uses the proposed
WSDL Extension for SOAP in DIME.
This service implements the SOAPBuilders "Round 4" DIME testing RPC interface.
It echoes any DIME payloads received in the request message back in the
response message.
-
The "echoFile" RPC service with various operations described in the WSDL
document found here.
The "echoDataFile" operation expects an input which is a struct of type
"DataFileStruct". This struct contains a filename member (of type string) and a
binary member, which is either a DIME attachment reference or base64 encoded
data element representing the file contents. The test client described below
invokes this operation.
Also, a header entry present in the request message with the name
"inputDataFile" will have its data echoed back in the response message wrapped
in a header entry named "outputDataFile". "inputDataFile" is also of type
"DataFileStruct". This demonstates the ability of the White Mesa implementation
to process DIME attachments referenced from SOAP header entries.
This service is configured and ready to run upon installation of the White Mesa
SOAP Server. Note that the test service ("/dime-rpc/echofile") configuration
specifies a header processor so that the header entries defined in the
WSDL document will be processed if present in messages.
Test Client
A GUI test client for Win32 ("echofile.exe") is supplied which invokes the
"echoDataFile" operation of the demo RPC service. It looks like this:
The client is pointed to a service endpoint by specifying the URL for its WSDL
document. It is configured to work "out of the box" in a typical White Mesa
SOAP Server installation. The user then browses for a disk file to be echoed,
and chooses whether its contents are sent as a DIME attachment or as base64
encoded data. This allows the efficiency of the two techniques to be compared
for a given file. When the response message containing the echoed file arrives,
a "File Save" dialog box appears so that the file can be saved and compared
with the original.
The size of the files that can be successfully round-tripped is limited by the
allowed connection time for both client and server. The connection timeout set
during installation is 5 minutes for the SOAP server and may be varied. The
test client is hard wired with a 5 minute timeout value. A little
experimentation with the test client will quickly reveal that that the use of
base64 encoding/decoding significantly reduces throughput.
