API Documentation

Protocol

Define protocol handlers for different classes of protocols

These protocols all wrap some base message schema and provide all the necessary hooks for pushing in a stream of bytes and getting out packets in the order they were found. The protocol handlers will also provide notifications of error conditions (for instance, unexpected bytes or a bad checksum)

class suitcase.protocol.StreamProtocolHandler(message_schema, packet_callback)

Protocol handler that deals fluidly with a stream of bytes

The protocol handler is agnostic to the data source or methodology being used to collect the data (blocking reads on a socket to async IO on a serial port).

Here’s an example of what one usage might look like (very simple appraoch for parsing a simple tcp protocol:

from suitcase.protocol import StreamProtocolHandler
from suitcase.fields import LengthField, UBInt16, VariableRawPayload
from suitcase.struct import Structure
import socket

class SimpleFramedMessage(Structure):
    length = LengthField(UBInt16())
    payload = VariableRawPayload(length)

def packet_received(packet):
    print packet

def run_forever(host, port):
    protocol_handler = StreamProtocolHandler(SimpleFramedMessage,
                                             packet_received)
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    sock.connect(host, port)
    sock.setblocking(1)
    while True:
        bytes = sock.recv(1024)
        if len(bytes) == 0:
            print "Socket closed... exiting"
            return
        else:
            protocol_handler.feed(bytes)

Parameters:

• message_schema – The top-level message schema that defines the packets for the protocol to be used.
• packet_callback – A callback to be executed with the form callback(packet) when a fully-formed packet is detected.

feed(new_bytes)

Feed a new set of bytes into the protocol handler

These bytes will be immediately fed into the parsing state machine and if new packets are found, the packet_callback will be executed with the fully-formed message.

Parameters:new_bytes – The new bytes to be fed into the stream protocol handler.

reset()

Reset the internal state machine to a fresh state

If the protocol in use does not properly handle cases of possible desycnronization it might be necessary to issue a reset if bytes are being received but no packets are coming out of the state machine. A reset is issue internally whenever an unexpected exception is encountered while processing bytes from the stream.

Structure

class suitcase.structure.Packer(ordered_fields, crc_field)

Object responsible for packing/unpacking bytes into/from fields

unpack_stream(stream)

Unpack bytes from a stream of data field-by-field

In the most basic case, the basic algorithm here is as follows:

for _name, field in self.ordered_fields:
   length = field.bytes_required
   data = stream.read(length)
   field.unpack(data)

This logic is complicated somewhat by the handling of variable length greedy fields (there may only be one). The logic when we see a greedy field (bytes_required returns None) in the stream is to pivot and parse the remaining fields starting from the last and moving through the stream backwards. There is also some special logic present for dealing with checksum fields.

exception suitcase.structure.ParseError

Exception raied when there is an error parsing

class suitcase.structure.Structure

Base class for message schema declaration

Structure forms the core of the Suitcase library and allows for a declarative syntax for specifying packet schemas and associated methods for transforming these schemas into packed bytes (and vice-versa).

Here’s an example showing how one might specify the format for a UDP Datagram:

class UDPDatagram(Structure):
    source_port = UBInt16()
    destination_port = UBInt16()
    length = LengthField(UBInt16())
    checksum = UBInt16()
    data = VariableRawPayload(length)

From this we have a near-ideal form for packing and parsing packet data following the schema:

>>> dgram = UDPDatagram()
>>> dgram.source_port = 9110
>>> dgram.destination_port = 1001
>>> dgram.checksum = 27193
>>> dgram.data = "Hello, world!"
>>> dgram.pack()
'#\x96\x03\xe9\x00\rj9Hello, world!'
>>> dgram2 = UDPDatagram()
>>> dgram2.unpack(dgram.pack())
>>> print dgram2
UDPDatagram (
  source_port=9110,
  destination_port=1001,
  length=13,
  checksum=27193,
  data='Hello, world!',
)

classmethod from_data(data)

Create a new, populated message from some data

This factory method is identical to doing the following, it just takes one line instead of two and looks nicer in general:

m = MyMessage()
m.unpack(data)

Can be rewritten as just:

m = MyMessage.from_data(data)

class suitcase.structure.StructureMeta

Metaclass for all structure objects

When a class with this metaclass is created, we look for any FieldProperty instances associated with the class and record those for use later on.

Fields

class suitcase.fields.BaseField(*args, **kwargs)

Base class for all Field instances

Some magic is used whenever a field is created normally in order to allow for a declarative syntax without having to create explicit factory methods/classes everywhere. By default (for a normal call) when a field is instantiated, it actually ends up returning a FieldPlaceholder object containing information about the field that has been declared.

To get an actual instance of the field, the call to the construction must include a keyword argument instantiate that is set to some truthy value. Typically, this instantiation logic is done by calling FieldPlaceholder.create_instance(parent). This is the recommended way to construct a field as it ensure all invariants are in place (having a parent).

Parameters:

• instantiate – Create an actual instance instead of a placeholder
• parent – Specify the parent of this field (typically a Structure instace).

class suitcase.fields.BaseFixedByteSequence(make_format, size, **kwargs)

Base fixed-length byte sequence field

class suitcase.fields.BaseStructField(**kwargs)

Base for fields based very directly on python struct module formats

It is expected that this class will be subclassed and customized by definining FORMAT at the class level. FORMAT is expected to be a format string that could be used with struct.pack/unpack. It should include endianess information. If the FORMAT includes multiple elements, the default _unpack logic assumes that each element is a single byte and will OR these together. To specialize this, _unpack should be overriden.

class suitcase.fields.BitField(number_bits, field=None, **kwargs)

Represent a sequence of bytes broken down into bit segments

Bit segments may be any BitFieldField instance, with the two most commonly used fields being:

• BitBool(): A single bit flag that is either True or False

• BitNum(bits): A multi-bit field treated like a big-endian integral

  value.

Example Usage:

class TCPFrameHeader(Structure):
    source_address = UBInt16()
    destination_address = UBInt16()
    sequence_number = UBInt32()
    acknowledgement_number = UBInt32()
    options = BitField(16,
        data_offset=BitNum(4),
        reserved=BitNum(3),
        NS=BitBool(),
        CWR=BitBool(),
        ECE=BitBool(),
        URG=BitBool(),
        ACK=BitBool(),
        PSH=BitBool(),
        RST=BitBool(),
        SYN=BitBool(),
        FIN=BitBool()
    )
    window_size = UBInt16()
    checksum = UBInt16()
    urgent_pointer = UBInt16()

tcp_packet = TCPFrameHeader()
o = tcp_packet.options
o.data_offset = 3
o.NS = True
o.CWR = False
o.
# ... so on, so forth

class suitcase.fields.CRCField(field, algo, start, end, **kwargs)

Field representing CRC (Cyclical Redundancy Check) in a message

CRC checks and calculation frequently work quite differently from other fields in a protocol and as such are treated differently by the messag container. In particular, a CRCField requires special steps at either the beginning or end of the message pack/unpack process. For each CRCField, we specify the following:

Parameters:

• field – The underlying field defining how the checksum will be stored. This should match the size of the checksum in whatever algorithm is being used (16 bits for CRC16).
• algo – The algorithm to be used for performing the checksum. This is basically a function that takes a chunk of data and gives the checksum. Several of these are provided out of the box in suitcase.crc.
• start – The offset in the overall message at which we should start the checksum. This may be positive (from the start) or negative (from the end of the message).
• end – The offset in the overall message at which we should end the checksum algo. This may be positive (from the start) or negative (from the end of the message).

A quick example of a message with a checksum is in order:

class MyChecksummedMessage(Structure):
    soh = Magic('\x1f\x1f')
    message_id = UBInt16()
    sequence_number = UBInt8()
    payload_length = LengthField(UBInt16())
    payload = VariableRawPayload(payload_length)

    # crc starts after soh and ends before crc
    crc = CRCField(UBInt16(), crc16_ccitt, 2, -3)
    eof = Magic('~')

packed_checksum(data)

Given the data of the entire packet reutrn the checksum bytes

validate(data, offset)

Raises SuitcaseChecksumException if not valid

class suitcase.fields.ConditionalField(field, condition, **kwargs)

Field which may or may not be included depending on some condition

In some protocols, there exist fields which may or may not be present depending on the values of other fields that would have already been parsed at this point in time. Wrapping such fields in a ConditionalField allows us to define a function to examine that state and only have the field capture the bytes if the conditions are right.

Parameters:

• field – The field which should handle the parsing if the condition evaluates to true.
• condition – This is a function which is given access to the parent message that is expected to return a boolean value. If the value is true, then field will handle the bytes. If not, then the field will be skipped and left with its default value (None).

class suitcase.fields.DependentField(name, **kwargs)

Field populated by container packet at lower level

It is sometimes the case that information from another layer of a messaging protocol be needed at another higher-level of the protocol. The DependentField is a way of declaring that a message at some layer is dependent on a field with some name from the parent layer.

For instance, let’s suppose that my protocol had an option byte that I wanted to include in some logic handling packets at some higher layer. I could include that byte in my message as follows:

class MyDependentMessage(Structure):
    ll_options = DependentField('proto_options')
    data = UBInt8Sequence(16)

class LowerLevelProtocol(Structure):
    type = DispatchField(UBInt16())
    proto_options = UBInt8()
    body = DispatchTarget(None, type, {
        0x00: MyDependentMessage,
    })

Params name:The name of the field from the parent message that we would like brought into our message namespace.

class suitcase.fields.DispatchField(field, **kwargs)

Decorate a field as a dispatch byte (used as conditional)

A DispatchField is always used with a DispatchTarget within the same message (at some level).

Example:

class MyMessage(Structure):
    type = DispatchField(UBInt8())
    body = DispatchTarget(dispatch_field = type, dispatch_mapping={
        0x00: MessageType0,
        0x01: MessageType1,
    })

Parameters:field – The field containing the dispatch parameter. This is typically an integer but could be any hashable object.

class suitcase.fields.DispatchTarget(length_provider, dispatch_field, dispatch_mapping, **kwargs)

Represent a conditional branch on some DispatchField

Example:

class MyMessage(Structure):
    type = DispatchField(UBInt8())
    body = DispatchTarget(dispatch_field = type, dispatch_mapping={
        0x00: MessageType0,
        0x01: MessageType1,
    })

Parameters:

• length_provider – The field providing a length value binding this message (if any). Set this field to None to leave unconstrained.
• dispatch_field – The field being target for dispatch. In most protocols this is an integer type byte or something similar. The possible values for this field act as the keys for the dispatch.
• dispatch_mapping – This is a dictionary mapping dispatch_field values to associated message types to handle the remaining processing.

class suitcase.fields.FieldPlaceholder(cls, args, kwargs)

Internally used object that holds information about a field schema

A FieldPlaceholder is what is actually instantiated and stored with the message schema class when a message schema is declared. The placeholder stores all instantiation information needed to create the fields when the message object is instantiated

create_instance(parent)

Create an instance based off this placeholder with some parent

class suitcase.fields.FieldProperty(field, onget=None, onset=None, **kwargs)

Provide the ability to define “Property” getter/setters for other fields

This is useful and preferable and preferable to inheritance if you want to provide a different interface for getting/setting one some field within a message. Take the test case as an example:

# define the message
class MyMessage(Structure):
    _version = UBByteSequence(2)
    version = FieldProperty(_version,
                            onget=lambda v: "%d.%02d" % (v[0], v[1]),
                            onset=lambda v: tuple(int(x) for x
                                                  in v.split(".", 1)))

msg = MyMessage()
msg.unpack('')
self.assertEqual(msg._version,(16, 3))
self.assertEqual(msg.version, "16.03")

msg.version = "22.7"
self.assertEqual(msg._version, (22, 7))
self.assertEqual(msg.version, "22.07")

Parameters:

• field – The field that is wrapped by this FieldProperty. The value of this field is passed into or set by the associated get/set fns.
• onget – This is a function pointer to a function called to mutate the value returned on field access. The function receives a single argument containing the value of the wrapped field.

Oaram onset:

This is a functoin pointer to a function called to map between the property and the underlying field. The function takes a single parameter which is the value the property was set to. It should return the value that the underlying field expects (or raise an exception if appropriate).

class suitcase.fields.LengthField(length_field, multiplier=1, **kwargs)

Wraps an existing field marking it as a LengthField

This field wraps another field which is assumed to return an integer value. A LengthField can be pointed to by a variable length field and the two will be coupled appropriately.

Parameters:

• length_field – The field providing the actual length value to be used. This field should return an integer value representing the length (or a multiple of the length) as a return value
• multiplier – If specified, this multiplier is applied to the length. If I specify a multiplier of 8, I am saying that each bit in the length field represents 8 bytes of actual payload length. By default the multiplier is 1 (1 bit/byte).

class suitcase.fields.Magic(expected_sequence, **kwargs)

Represent Byte Magic (fixed, expected sequence of bytes)

class suitcase.fields.Payload(length_provider=None, **kwargs)

Variable length raw (byte string) field

This field is expected to be used with a LengthField. The variable length field provides a value to be used by the LengthField on message pack and vice-versa for unpack.

Parameters:length_provider – The LengthField with which this variable length payload is associated. If not included, it is assumed that the length_provider should consume the remainder of the bytes available in the string. This is only valid in cases where the developer knows that they will be dealing with a fixed sequence of bytes (already boxed).

class suitcase.fields.SBInt16(**kwargs)

Signed Big Endian 16-bit integer field

class suitcase.fields.SBInt32(**kwargs)

Signed Big Endian 32-bit integer field

class suitcase.fields.SBInt64(**kwargs)

Signed Big Endian 64-bit integer field

class suitcase.fields.SBInt8(**kwargs)

Signed Big Endian 8-bit integer field

class suitcase.fields.SLInt16(**kwargs)

Signed Little Endian 16-bit integer field

class suitcase.fields.SLInt32(**kwargs)

Signed Little Endian 32-bit integer field

class suitcase.fields.SLInt64(**kwargs)

Signed Little Endian 64-bit integer field

class suitcase.fields.SLInt8(**kwargs)

Signed Little Endian 8-bit integer field

class suitcase.fields.UBInt16(**kwargs)

Unsigned Big Endian 16-bit integer field

class suitcase.fields.UBInt24(**kwargs)

Unsigned Big Endian 24-bit integer field

class suitcase.fields.UBInt32(**kwargs)

Unsigned Big Endian 32-bit integer field

class suitcase.fields.UBInt64(**kwargs)

Unsigned Big Endian 64-bit integer field

class suitcase.fields.UBInt8(**kwargs)

Unsigned Big Endian 8-bit integer field

class suitcase.fields.ULInt16(**kwargs)

Unsigned Little Endian 16-bit integer field

class suitcase.fields.ULInt32(**kwargs)

Unsigned Little Endian 32-bit integer field

class suitcase.fields.ULInt64(**kwargs)

Unsigned Little Endian 64-bit integer field

class suitcase.fields.ULInt8(**kwargs)

Unsigned Little Endian 8-bit integer field

suitcase.fields.VariableRawPayload

alias of Payload

CRC Checksums

A collection of Suitcase CRCField compatible CRC algorithms

suitcase.crc.crc16_ccitt(data, crc=0)

Calculate the crc16 ccitt checksum of some data

A starting crc value may be specified if desired. The input data is expected to be a sequence of bytes (string) and the output is an integer in the range (0, 0xFFFF). No packing is done to the resultant crc value. To check the value a checksum, just pass in the data byes and checksum value. If the data matches the checksum, then the resultant checksum from this function should be 0.

suitcase.crc.crc16_kermit(data, crc=0)

Calculate/Updaet the Kermit CRC16 checksum for some data

Exceptions

Define an exception hierarchy used within Suitcase

The basic rule is that users of the Suitcase library should always be able to catch the base SuitcaseException and be confident that they will not get some other random exception. Catching more specific subclasses of this exception should then be possible.

There are still some cases (especially when constructing a field) that base exceptions (like TypeError, ValueError, or TypeError) might still be raised. In general, however, any case where things are dynamic, we do our best to raise a SuitcaseException with meaningful information.

exception suitcase.exceptions.SuitcaseChecksumException

Exception raised when a checksum does not match

exception suitcase.exceptions.SuitcaseException

Base exception acting as the root for other exceptions

exception suitcase.exceptions.SuitcasePackException

Raised when there is an error packing a message

exception suitcase.exceptions.SuitcasePackStructException(struct_exception)

Raised on an error packing a struct field

exception suitcase.exceptions.SuitcaseParseError

Raised when there is a problem parsing bytes into a suitcase schema

exception suitcase.exceptions.SuitcaseProgrammingError

Exception raised when somebody is doing something wrong