2
\$\begingroup\$

This is my code to extract values from a binary stream. I am fairly happy with it, the only thing I can think should change is the fact it throws if there is an error. The only option for failure is boost::optional (limited to C++11 by target platform).

struct Token 
{
    enum TokenClass 
    {
        BYTE, WORD, INT, SZCHAR, DATA
    };

    TokenClass type;
    boost::variant<uint8_t, uint16_t, uint32_t, std::string, std::vector<uint8_t>> value;

    Token( TokenClass c, uint8_t b ) : type(c), value(b) { }
    Token( TokenClass c, uint16_t w ) : type(c), value(w) { }
    Token( TokenClass c, uint32_t i ) : type(c), value(i) { }
    Token( TokenClass c, std::string s ) : type(c), value(std::move(s)) { }
    Token( TokenClass c, std::vector<uint8_t> d ) : type(c), value(std::move(d)) { }
};


// tokenize a byte stream. returns a vector of tokens that can be tested using validate
std::vector<Token> tokenize( const std::vector<uint8_t>& message_data, const std::vector<Token::TokenClass>& expects ) 
{
    std::vector<Token> ts;

    auto it = std::begin( message_data );
    for ( auto&& expect : expects )
    {
        switch (expect)
        {
            case Token::BYTE:
            {
                if (it == std::end( message_data ))
                {
                    std::stringstream ss;
                    ss << "not enough data left in stream for byte";
                    throw std::out_of_range(ss.str());
                }
                ts.emplace_back( Token::BYTE, *it );
            }
            break;

            case Token::WORD:
            {
                if (std::distance(it, std::end( message_data )) < 2)
                {
                    std::stringstream ss;
                    ss << "not enough data left in stream for word";
                    throw std::out_of_range(ss.str());
                }
                uint16_t v = (*it++ << 8) + *it;
                ts.emplace_back( Token::WORD, v );
            }
            break;

            case Token::INT:
            {
                if (std::distance(it, std::end( message_data )) < 4)
                {
                    std::stringstream ss;
                    ss << "not enough data left in stream for dword";
                    throw std::out_of_range(ss.str());
                }
                uint32_t v = (*it++ << 24) + (*it++ << 16) + (*it++ << 8) + *it;
                ts.emplace_back( Token::INT, v );               
            }
            break;

            case Token::SZCHAR:
            {
                auto pos = std::find(std::begin(message_data), std::end(message_data), '\0');
                if (pos == std::end( message_data ))
                {
                    std::stringstream ss;
                    ss << "no terminating null byte found in stream";
                    throw std::out_of_range(ss.str());
                }
                auto e = it;
                for( ; e != pos and std::isprint(*e); e++ )
                    ;
                ts.emplace_back( Token::SZCHAR, std::string(it,e) );
                it = e;
            }
            break;

            default:
            break;
        }
        it++;
    }

    // collect the rest of the message and place in a data token
    if (it != std::end(message_data))
    {
        ts.emplace_back( Token::DATA, std::vector<uint8_t>(it,std::end(message_data)) );
    }

    return ts;
}

Usage is fairly simple. You provide an expected input sequence (for headers, etc) and then call tokenize to parse the data. Left over bytes at the end of the stream is placed into a DATA token for further parsing.

For example:

std::vector<Token::TokenClass> expects{ Token::SZCHAR, Token::WORD, Token::BYTE };
std::vector<uint8_t> input{ 'A', 'B', 'C', '\0', 0x1, 0xff, 'S', 0x12, 0x34 };
auto ts = tokenize( input, expects );

This takes a stream (input) and the tokens (expects) and the results are a null-terminated string ('ABC'), a word (0x1ff), a byte ('S'), and data (0x12, 0x34).

The next thing this needs is a validation function. You can have fixed values (for magic bytes that signify start of headers, etc.), a range of values where anything outside that range is an error (ie a percentage) or a series of fixed values (usually signifying an action) where anything that it does not understand is an error.

\$\endgroup\$
0
\$\begingroup\$

You force the user code to pass in a vector as the raw data. This is not always how data is read in. Provide an overload that takes in a begin and end pair of char* iterators and forward the vector overload to that.

memcpy is faster than scanning for a null byte. So consider changing the format to use a prefix length instead of the null terminator.

If the format is fixed then you will want to add the values of the enums explicitly. That way they don't depend on the declaration order and it's clear that that the values are chosen for a reason.

Exceptions are frankly fine, one thing I see is that you only throw an exception where more data can create a correct parse (either directly or by adding a null terminator). So you can add the remaining unparsed tokens to the last DATA token. That way you can partially parse the data and resume parsing when more data comes in.

\$\endgroup\$
  • \$\begingroup\$ Unfortunately I can't change the format - this was set in stone about 20 years ago and built upon since then. \$\endgroup\$ – graham.reeds Jul 10 '17 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.