2
\$\begingroup\$

Preliminary

I need to read a binary file constructed by writing a Struct to it with Python. So I decided to write a simple wrapper using struct module.

Code

import struct


class structure:
    def __init__ ( self, little_endian = True ):
        self._little_endian = little_endian

        self._table  = dict() #stores field name, its format and value 
        self._pack  = list()  #stores field names in order fields must be read


    def add_field( self, field_name, field_format ):
        if field_name in self._table.keys():
            print "WARNING :: Field with name '%s' already exists in this structure" % field_name 
        elif field_format == "":
            print "WARNING :: Format cannot be an empty string"
        else:
            self._table[ field_name ] = { "format" : field_format, "value" : None }
            self._pack.append( field_name )
        #maybe error handling is required here


    def get_value( self, field_name ):
        try:
            return self._table[ field_name ][ "value" ]
        except KeyError:
            print "WARNING :: There is no such field named '%s' in this structure" % field_name 
            return None


    def get_format_string( self ):
        f_str = ""

        for key in self._pack:
            f_str += self._table[ key ][ "format" ]

        if (self._little_endian == True):
            f_str = "<" + f_str
        else:
            f_str = ">" + f_str

        return f_str


    def get_size( self ):
        return struct.calcsize( self.get_format_string() )


    def fill( self, file_ptr ):
        result = struct.Struct( self.get_format_string() ).unpack( file_ptr.read( self.get_size() ) )
        for (i,key) in enumerate(self._pack):
            self._table[ key ][ "value" ] = result[ i ]


    def show( self ):
        print "General info:"
        print "%20s = %s"  % ( "Format string", self.get_format_string() )
        print "%20s = %d" % ( "Size (bytes)", self.get_size() ) 
        print "Value:"
        for key in self._pack:
            print "%20s = " % key, self._table[ key ][ "value" ]

Usage

Usage is straightforward. I used the real binary produced by OWON oscilloscope here so some of the values might look strange (consist non-printable characters).

import structure


s = structure.structure( little_endian=True )
s.add_field( "header", "10s" )
s.add_field( "serial", "14s" )
s.add_field( "string", "18s" )
s.add_field( "trig_time", "f" )
s.add_field( "float1", "f" )
s.add_field( "float2", "f" )

with open( "./data/trigger.bin" ) as f:
    s.fill( f )

s.show()

Output

General info:
       Format string = <10s14s18sfff
        Size (bytes) = 54
Value:
              header =  SPBS01e�
              serial =  SDS60621715234
              string =  
           trig_time =  99.0
              float1 =  3.41239206136e-06
              float2 =  0.001953125

As always any suggestions, ideas and critics are appreciated.

\$\endgroup\$
3
\$\begingroup\$

A major drawback of this design is that you cannot reuse a single instance of structure.structure to read multiple records. Instead of s.fill(file_ptr), consider making a method s.read(f) that returns an OrderedDict of the deserialized data. (By the way, mentioning "pointer" in a parameter name isn't very Pythonic.) Note that that would also replace your .get_value() method with a dictionary lookup, which can be dealt with more idiomatically. (Or, at least implement dict-compatible iteration and subscripting in your structure class.)

Switching to a reusable .read() method would also eliminate a point of awkwardness in your code, which is that ._table is a dict of dicts, each having a "format" and a "value".

In your implementation of .fill(), why not call .unpack_from(file_ptr) instead of .unpack( file_ptr.read( self.get_size() ) )?

A more natural default endianness would be either big-endian (because that is the standard Network Byte Order) or host byte order (because that is a non-opinionated choice that also happens to be the path of least resistance).

Whatever you design, it should accommodate the opposite operation — serializing data to a file.

Consider allowing a structure to contain nested structures as fields.

The .show() method should probably be .__str__() instead.

I'm not convinced of the benefit of requiring multiple calls to .add_field() to set up the structure. I'd prefer to have the constructor accept the field specification as a list of 2-tuples. Not only would the calling code be cleaner, but the implementation of the structure class would be simpler as well if it is immutable.

I'm not a fan of your printed warnings. Utility code, such as this, has absolutely no business contaminating sys.stdout, and ideally shouldn't print to sys.stderr either. An end user would be completely clueless about what to do about these "warnings". A programmer would also be puzzled about where these warnings came from (if it's a large program) and what actually happened as a consequence. Rather, you should decide to either tolerate these situations gracefully or raise exceptions. In my opinion…

  • "Field with name … already exists" should raise an ArgumentError.
  • "Format cannot be an empty string" should be tolerated (or perhaps an assertion failure).
  • "There is no such field" would be eliminated altogether based on my suggestion to return an OrderedDict. If you had to offer a .get_value() method, then you should just let the KeyError propagate.
\$\endgroup\$
  • \$\begingroup\$ Nice answer! I will go through it gradually. First question: Why do you say I cannot reuse a single instance for reading multiple records? More over, I DO it in my algorithm successfully (at least I don't see any inconsistencies). What is a problem I call s.fill( f ) several times? Or am I missing something? \$\endgroup\$ – LRDPRDX Aug 18 at 7:58
  • \$\begingroup\$ Well, you can .fill() the structure again, but then you would lose the data that was previously read. \$\endgroup\$ – 200_success Aug 18 at 8:01
  • \$\begingroup\$ Ah. Of course. In this sense you right. \$\endgroup\$ – LRDPRDX Aug 18 at 8:09
  • \$\begingroup\$ Good catch on unpack_from method. I missed it. \$\endgroup\$ – LRDPRDX Aug 18 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.