This document is an authoring guide for the eXtensible Data Description Language, or XDDL. It is not intended to be a complete reference. Rather the intent is to get you up and running quickly. The Documentation page provides an exhaustive XDDL element reference with examples for you to dive into.
XDDL is an open, XML based, unambiguous, verbose notation that describes data structures for representing and decoding binary messages. This includes message fields, sub-fields, records, conditional fields, repeating fields, etc. It is, in fact, a data structure schema.
XDDL is a human readable domain specific language.
A minimal understanding of XML is all that is required. Spending just a few minutes here at wikipedia should give you all the background you need.
If you have not yet, please download and install the IDM, and have your favorite text editor up and running. We'll be using the installed tool, idm, to validate your XDDL and parse messages via the command line.
Let's get started with an example. Below is a complete well-formed and valid XDDL file describing the "One Bit Protocol". It's a protocol of, well, just one bit.
<?xml version="1.0" encoding="UTF-8" ?>
<xddl>
<structure>
<field name="Bit" length="1"/>
</structure>
</xddl>
obp.xddl
Although a trivial example, the main structure of an XDDL document is illustrated. The <xddl> element is the root, and the <structure> element is where we define a message's bit pattern.
We can parse a message defined by the OBP with idm:
# idm obp.xddl @1
Name Length Value Hex Description
Bit 1 1 01
The output of idm is a table with the name of each field, the length, and the value. The value is also printed out in hexadecimal. We did not provide a description and that is why the description column is left blank. We will learn how to do that later.
If we chose to use the xidm tool instead of idm, the tool would be opened in windowed mode. This is what runs if you, for example, run the tool form the Windows Start menu. We'll stick with the command line version for this tutorial.
We defined our message using the "@" prefix to denote a binary number.
Without this prefix, the idm will interpret the number as hexadecimal.
You may also explicitly indicate the message is hex by using the optional
"#" prefix. The following are valid hex numbers: "04", "AA",
"1122334455AB3F" (or "#04", "#AA", "#1122334455AB3F"). Hex numbers
must be byte aligned (binary numbers do not have this restriction). So
"#3" is considered invalid by idm.
The idm command reference contains a complete list of options.
The <field> is the most basic element in XDDL. Let's look at the following file:
<?xml version="1.0" encoding="UTF-8" ?>
<xddl>
<structure>
<field name="A" length="8"/>
<field name="B" length="8"/>
<field name="C" length="8"/>
<field name="D" length="8"/>
</structure>
</xddl>
field.xddl
And use idm to parse the message, 01020304:
# idm field.xddl 01020304
Name Length Value Hex Description
A 8 1 01
B 8 2 02
C 8 3 03
D 8 4 04
We can think of the <field> elements as consuming the message. The
first byte, 01 is consumed by the A field, 02 is consumed by the B
field, and so on.
All the fields in the above definition are 8 bits long to make it easy for you to see how things are parsed. However, the length can be any number. Let's change the lengths of the fields and parse the same message.
<?xml version="1.0" encoding="UTF-8" ?>
<xddl>
<structure>
<field name="A" length="3"/>
<field name="B" length="7"/>
<field name="C" length="9"/>
<field name="D" length="13"/>
</structure>
</xddl>
lengths.xddl
# idm lengths.xddl 01020304
Name Length Value Hex Description
A 3 0 00
B 7 4 04
C 9 16 0010
D 13 772 0304
As you can see, you can easily slice and dice messages with no regard to byte or word alignment. XDDL takes care of it all for you.
By the way, that first line (the one with the question marks) in the XDDL files is called the XML Declaration. It's good to have, but optional. For the sake of brevity from now on we won't show it in the examples.
In this section we will briefly discuss all of the basic XDDL elements. These elements are used to define the dynamic structures of messages.
We have already seen the the <field> in the previous section. It is the most basic element. One important aspect of XDDL is the capability of dynamically sized fields. Consider the following:
<xddl>
<structure>
<field name="A" length="8"/>
<field name="B" length="A"/>
</structure>
</xddl>
dynamic.xddl
The length of the B field is determined by the value of the A field.
Here we will parse two messages with the above definition.
# idm dynamic.xddl 08FF 10FFAA
Name Length Value Hex Description
A 8 8 08
B 8 255 FF
A 8 16 10
B 16 65450 FFAA
The length can be any XDDL Expression -- a
simple, powerful expression evaluation mini-language embedded in XDDL.
Expressions are used throughout XDDL in most any place an integer
can be used.
Fields can be conditionally included based on the value of other fields. This is done using the <if> element. Here's an example:
<xddl>
<structure>
<field name="Included" length="8"/>
<if expr="Included">
<field name="A" length="8"/>
</if>
<field name="B" length="8"/>
</structure>
</xddl>
if.xddl
If the Included field is anything but zero, the A field will be
included in the message. Otherwise it will not be:
# idm if.xddl 0002 FF0102
Name Length Value Hex Description
Included 8 0 00
B 8 2 02
Included 8 255 FF
A 8 1 01
B 8 2 02
There is no else or else if construct in XDDL. in that case you can
have either a series of <if> elements, or use the <switch> statement,
described next.
The following file describes the pattern of a message type followed by the message. The fields after the message type are different depending on its value. We conditionally include fields by using the <switch> element:
<xddl>
<structure>
<field name="MessageType" length="8"/>
<switch expr="MessageType">
<case value="1">
<field name="A" length="4"/>
<field name="B" length="4"/>
</case>
<case value="2">
<field name="C" length="1"/>
<field name="D" length="7"/>
</case>
</switch>
</structure>
</xddl>
switch.xddl
As you can see, the first field is 8 bits long and its name is MessageType.
The <switch> element has an expr attribute. This expression is evaluated and
based on its value, the corresponding <case> is used to continue parsing the
message.
Here are some examples using the definition:
# idm switch.xddl 01AB 02FF
Name Length Value Hex Description
MessageType 8 1 01
A 4 10 0A
B 4 11 0B
MessageType 8 2 02
C 1 1 01
D 7 127 7F
As you can see, the <switch> element is similar to the switch statement in other programming languages such as C and C++, with a few exceptions. There is no corresponding break. The only way for parsing to automatically "fall through" to the next <case> is if the <case> is an empty element. The <switch> and <case> element references have all the details.
XDDL supports looping with the <repeat> element:
<xddl>
<structure>
<field name="Count" length="8"/>
<repeat name="Repeat" num="Count">
<field name="DataPoint" length="8"/>
</repeat>
</structure>
</xddl>
repeat.xddl
A Count field is read in, and then the <repeat> element's children are
repeated Count times.
# idm repeat.xddl 0401020304 01FF
Name Length Value Hex Description
Count 8 4 04
Repeat 32 01020304
record 8 01
DataPoint 8 1 01
record 8 02
DataPoint 8 2 02
record 8 03
DataPoint 8 3 03
record 8 04
DataPoint 8 4 04
Count 8 1 01
Repeat 8 FF
record 8 FF
DataPoint 8 255 FF
The <repeat> behavior can be modified extensively through the use of attributes. You can specify a minimum, maximum, or fixed repeat count, among other things.
Repeats are often used in conjunction with the <record> element.
The following file illustrates another common pattern among telecommunication protocols:
<xddl>
<structure>
<field name="RecordType" length="8"/>
<field name="RecordLength" length="8"/>
<record name="InformationFields" length="RecordLength * 8">
<repeat name="repeat">
<field name="One" length="8"/>
<field name="Two" length="8"/>
<field name="Three" length="8"/>
</repeat>
</record>
</structure>
</xddl>
record.xddl
Here we use a <record> element to encapsulate a block of other elements. The %record size can be bound, as in this case. The <repeat> element will be repeated until the entire record is consumed. Records are ideal for logically containing subfields, or for specifying frame sizes for frame based messages.
# idm record.xddl 0106010203010203
Name Length Value Hex Description
RecordType 8 1 01
RecordLength 8 6 06
InformationFields 48 010203010203
repeat 48 010203010203
record 24 010203
One 8 1 01
Two 8 2 02
Three 8 3 03
record 24 010203
One 8 1 01
Two 8 2 02
Three 8 3 03
Wow there is a lot of stuff there! Don't forget, the only elements that
consume data from the message are fields. The other things are just
structural containers of the fields. So in the above decode, the only
things that consume data is the RecordType, RecordLength, and the 4
Info fields.
XDDL supports <fragment> elements. They are similar to macro expansion in traditional programming languages. Any of the above XDDL sequences can be placed within a <fragment> element. Then fragments can be inserted inline. Here is a rewrite of the above example using a <fragment>:
<xddl>
<fragment id="Information"> <!-- define fragment -->
<record name="InformationFields" length="RecordLength * 8">
<repeat>
<field name="Info" length="8"/>
</repeat>
</record>
</fragment>
<structure>
<field name="RecordType" length="8"/>
<field name="RecordLength" length="8"/>
<fragment href="#Information"/> <!-- reference fragment -->
</structure>
</xddl>
fragment.xddl
As you can see, a fragment is defined by using a <fragment> element as a children of the <xddl> element and before the <structure> element.
These fragments are then inserted inline by using a <fragment> with an href
attribute. The href attribute supports complete local URI specifications.
This means a fragment from a different file can be referenced. For
example:
<xddl>
<structure>
<field name="RecordType" length="8"/>
<field name="RecordLength" length="8"/>
<fragment href="fragment.xddl#Information"/>
</structure>
</xddl>
insert.xddl
This file is identical in functionality to the previous one.
XDDL can not only capture a field's value, but also its meaning. It does this through the use of XDDL types.
Let's revisit the one bit protocol example above. Giving the <field> a type yields the following definition:
<xddl>
<type id="HelloType">
<enum>
<item key="0" value="Goodbye World!"/>
<item key="1" value="Hello World!"/>
</enum>
</type>
<structure>
<field name="Bit" length="1" type="#HelloType"/>
</structure>
</xddl>
hello.xddl
The type="#HelloType" attribute was added to the Bit <field>. This
attribute is a local URI to a <type> element. The attribute may have
referenced a type from a different file. This is convenient for XDDL
authors who wish to contain all the types used in one location.
Rerunning the command:
# idm hello.xddl @1 @0
Name Length Value Hex Description
Bit 1 1 01 Hello World!
Bit 1 0 00 Goodbye World!
We see that a description is now added to the output.
Message definitions may also declare XDDL Properties with the <prop> element. The property can be referenced in expressions just like fields. Unlike fields they do not consume data, and their value can subsequently be changed with the <setprop> element.
<xddl>
<structure>
<field name="A" length="8"/>
<if expr="A == 1">
<prop name="B" value="5"/>
</if>
</structure>
</xddl>
prop.xddl
Here are a couple example messages:
# idm prop.xddl 01 00
Name Length Value Hex Description
A 8 1 01
B 5
A 8 0 00
The above definition illustrates one reason to use properties. Some protocols have implied fields with default values. Properties let us create these fields when needed. If an expression references a field that was not previously declared, the XDDL processor will fail. A field must be declared before it is referenced. Using a property will satisfy this requirement (more on errors below).
And possibly the most important reason for using properties is for name aliasing.
Name aliasing is described in the next section for IDM Customization.
We now have the tools necessary to discuss another topic -- scoping. Each %record element has its own scope. This means fields or properties referenced in expressions are evaluated using the current scope, then the next higher scope, and so on. An example:
<xddl>
<structure>
<field length="8" name="Type"/>
<prop value="0" name="mini"/>
<record>
<prop value="0" name="mini"/>
<setprop name="mini" value="32"/> <!-- set the prop on previous line -->
</record>
<setprop name="mini" value="16"/> <!-- set the prop before record -->
</structure>
</xddl>
scope.xddl
Here we can see which mini property is affected by the two <setprop>
elements:
# idm scope.xddl 01
Name Length Value Hex Description
Type 8 1 01
mini 16
0
mini 32
mini 32
mini 16
Although this example is a bit contrived, it does illustrate the basic concept of scoping.
You may include an <oob> section as the first child of the <structure> element. This is a way of specifying Out-of-Band information. What is Out-of-Band? Well, in our case, OOB data is fields or properties that are required to decode the rest of the message, though technically not part of the message.
You can place these required fields in the <oob> section, and then configure the IDM (via the View/Preferences menu item) to not display the Out-of-Band data.
Any elements that can be placed within the <structure> element can also be placed in an <oob> element (except of course <oob> itself).
That was the conceptual reason. There is another more practical reason to have an OOB section. The IDM is configured to look in the OOB section for specific properties. These properties include "Name" and "Direction". The Name property is used for the name of the message, and the Direction is used for the arrow next to it.
Let's take a look at an example.
<xddl>
<type id="MessageType">
<enum>
<item key="0" value="Unknown"/>
<item key="1" value="General Page"/>
<item key="2" value="Access Parameters"/>
<item key="3" value="Service Connect"/>
</enum>
</type>
<structure>
<oob>
<prop name="Name" value="0"/>
</oob>
<field name="Message" length="8" type="#MessageType"/>
<setprop name="Name" value="Message" type="#MessageType"/>
<field name="B" length="8"/>
<field name="C" length="8"/>
<field name="D" length="8"/>
</structure>
</xddl>
magic.xddl
Now parse a message with the IDM in window mode using this definition:
xidm -x magic.xddl 01010203 02A3F501 03F004BD
So far, we have not talked about errors in messages. But if you played around with the idm commands, you may have created a message that did not align well with the XDDL. One easy way to do this is to decode a message with extra data, or without enough data. Let's take a look again at our field example presented earlier:
<xddl>
<structure>
<field name="A" length="8"/>
<field name="B" length="8"/>
<field name="C" length="8"/>
<field name="D" length="8"/>
</structure>
</xddl>
field.xddl
This one has too much data:
# idm field.xddl 0102030405
Name Length Value Hex Description
A 8 1 01
B 8 2 02
C 8 3 03
D 8 4 04
extra 8 5 05
And this one doesn't have enough:
# idm field.xddl 010203
Name Length Value Hex Description
A 8 1 01
B 8 2 02
C 8 3 03
D 0
Another type of error is when you have the correct amount of data, but an individual field's value is incorrect. This can be illustrated with the following:
<xddl>
<type id="ValidityType">
<enum>
<item key="0" value="Pass"/>
<item key="1" value="Fail"/>
<item key="2" value="Unknown"/>
</enum>
</type>
<structure>
<field name="Response Code" length="8" type="#ValidityType"/>
</structure>
</xddl>
error.xddl
Let's parse a few messages:
# idm error.xddl 00 01 02 03
Name Length Value Hex Description
Response Code 8 0 00 Pass
Response Code 8 1 01 Fail
Response Code 8 2 02 Unknown
Response Code 8 3 03 invalid value
As you can see, the last message is incorrect because the Response Code field
is set to 3, and 3 is not part of the ValidityType <enum>.
We have just a few more things to discuss:
As we've stated earlier <field> elements are interpreted as unsigned integers. We can specify floating point numbers with the <float> element. This element specifies a 32 bit single precision floating point number. Here is an example using the same value represented as a <field> and a <float>:
<xddl>
<structure>
<field name="uint" length="32"/>
<float name="float"/>
</structure>
</xddl>
float.xddl
Decoding a message with the number 40.875:
# idm float.xddl 4223800042238000
Name Length Value Hex Description
uint 32 1109622784 42238000
float 32 40.875 42238000
As you can see, using <float> just tells the IDM to interpret the bit pattern as a float instead of an unsigned integer.
Notice the <float> element does not have a length attribute; it's length is
fixed at 32 bits.
By default, XDDL treats all fields as
big endian.
However, sometimes our data is in the little endian format. We can
specify using the bigEndian boolean attribute of the <structure> element. The
default for the attribute is true. Here is a small example with and
without the attribute set to true:
Big:
<xddl>
<structure bigEndian="true">
<field name="Big" length="32"/>
</structure>
</xddl>
big.xddl
# idm big.xddl 00112233
Name Length Value Hex Description
Big 32 1122867 00112233
Little:
<xddl>
<structure bigEndian="false">
<field name="Little" length="32"/>
</structure>
</xddl>
little.xddl
# idm little.xddl 00112233
Name Length Value Hex Description
Little 32 857870592 00112233
This bigEndian attribute affects <float> and <field> elements that are byte
aligned. <field> elements that are not byte aligned are not
affected.
The <pad> element is used in order to pad a message for word or byte
alignment purposes. The mod attribute is used to customize this
behavior. Here's an example of a five bit <field>, followed by a <pad>,
followed by another <field>.
<xddl>
<structure>
<field name="A" length="5"/>
<pad/>
<field name="B" length="8"/>
</structure>
</xddl>
pad.xddl
# idm pad.xddl A014
Name Length Value Hex Description
A 5 20 14
pad 3 0 00
B 8 20 14
The <peek> element is used to actually look forward in the message. Some protocols require this for proper decoding. The <peek> reference provides examples and details some important restrictions to keep in mind when using this powerful feature.
We have briefly touched on every element in XDDL. All the examples in this document are short and concise. You can think of them as building blocks for more complicated and elaborate structures. A complete reference for XDDL can be found at the Documentation page. And you can always contact Intrig Professional Services for help with your particular needs.