Integer endianness types for protocol structures

Question

Motivation

When working with storage or wire protocols, we often read or write structures containing integers with specific byte-ordering (e.g. big-endian for Internet Protocol, or little-endian for USB).

It's common to use functions such as the htons() family to convert values, but it's easy to miss a conversion (particularly if the protocol is the same endianness as the development system). Instead, I prefer to use the type system to distinguish host integers from their representations' protocol byte-order.

Known limitations (non-goals):

• I haven't had a need to handle floating-point values, so only integers are handled here.
• It's not suitable for protocols such as IIOP or X11 where one of the hosts chooses at run-time which endianness to use.

Implementation

#ifndef ENDIAN_HPP
#define ENDIAN_HPP

#include <array>
#include <concepts>
#include <cstdint>
#include <ranges>
#include <type_traits>

namespace endian
{

    template<std::integral T, auto ReadView, auto WriteView>
    struct Endian
    {
        // We use unsigned T for bitwise operations
        using U = std::make_unsigned_t<T>;

        // The underlying storage
        std::array<unsigned char, sizeof (T)> data = {};

        // implicit conversion from T
        Endian(T value = 0)
        {
            auto uval = static_cast<U>(value);
            for (auto& c: data | WriteView) {
                c = static_cast<std::uint8_t>(uval);
                uval >>= 8;
            }
        }

        // implicit conversion to T
        operator T() const
        {
            U value = 0;
            for (auto c: data | ReadView) {
                value <<= 8;
                value |= c;
            }
            return static_cast<T>(value);
        }
    };

    template<std::integral T>
    using BigEndian = Endian<T, std::views::all, std::views::reverse>;

    template<std::integral T>
    using LittleEndian = Endian<T, std::views::reverse, std::views::all>;

}

#endif // ENDIAN_HPP

Tests

using endian::BigEndian;
using endian::LittleEndian;

#include <gtest/gtest.h>

// Ensure there's no padding
static_assert(sizeof (BigEndian<int>) == sizeof (int));
static_assert(sizeof (LittleEndian<int>) == sizeof (int));

TEST(big_endian, uint8)
{
    std::uint8_t x = 2;
    auto be = BigEndian{x};
    std::array<unsigned char, 1> expected{{2}};
    EXPECT_EQ(be.data,expected);

    for (auto& c: be.data) { ++c; }
    std::uint8_t y = be;
    EXPECT_EQ(y, 3);
}

TEST(little_endian, uint8)
{
    std::uint8_t x = 2;
    auto le = LittleEndian{x};
    std::array<unsigned char, 1> expected{{2}};
    EXPECT_EQ(le.data,expected);

    for (auto& c: le.data) { ++c; }
    std::uint8_t y = le;
    EXPECT_EQ(y, 3);
}

TEST(big_endian, uint16)
{
    std::uint16_t x = 0x1234;
    BigEndian be = x;
    std::array<unsigned char, 2> expected{{0x12, 0x34}};
    EXPECT_EQ(be.data,expected);

    for (auto& c: be.data) { ++c; }
    std::uint16_t y = be;
    EXPECT_EQ(y, 0x1335);
}

TEST(little_endian, uint16)
{
    std::uint16_t x =  0x1234;
    auto le = LittleEndian{x};
    std::array<unsigned char, 2> expected{{0x34, 0x12}};
    EXPECT_EQ(le.data,expected);

    for (auto& c: le.data) { ++c; }
    std::uint16_t y = le;
    EXPECT_EQ(y, 0x1335);
}

TEST(big_endian, uint32)
{
    std::uint32_t x = 0x12345678;
    auto be = BigEndian{x};
    std::array<unsigned char, 4> expected{{ 0x12, 0x34, 0x56, 0x78 }};
    EXPECT_EQ(be.data,expected);

    for (auto& c: be.data) { ++c; }
    std::uint32_t y = be;
    EXPECT_EQ(y, 0x13355779);
}

TEST(little_endian, uint32)
{
    std::uint32_t x = 0x12345678;
    auto le = LittleEndian{x};
    std::array<unsigned char, 4> expected{{ 0x78, 0x56, 0x34, 0x12 }};
    EXPECT_EQ(le.data,expected);

    for (auto& c: le.data) { ++c; }
    std::uint32_t y = le;
    EXPECT_EQ(y, 0x13355779);
}

Example usage

I'm currently using this when sending values over the wire. A simplified example, with all the error handling removed, looks something like:

struct Response {
    BigEndian<std::uint16_t> seq_no;
    BigEndian<std::uint16_t> sample_value;
};

We send by assigning (which implicitly converts our integer to BigEndian) and then writing the structure:

void send_result(std::uint16_t value)
{
    Response r;
    r.seq_no = counter++;
    r.sample_value = value;
    write(fd, &r, sizeof r);
}

On the receive side, we read the wire representation into the same structure (so it's bitwise identical to the sending side) and then use the conversion operator to access the data in native form:

std::uint16_t recv_result()
{
    Response r;
    read(fd, &r, sizeof r);
    // ignore seq_no, for now
    return r.sample_value;
}

Concerns

• I don't like having to pass two different view adapters, but I feel that reverse | reverse would add overhead that I don't want, and I do like the simple arithmetic when we read the most-significant byte first and write the least-significant first.
• I'd like optimised compilation to emit a simple load or store when converting to/from the host's native order if possible, but I'm not good enough at assembly languages to check that (I think that the result [on gobolt] implies that writing is well-optimised, but not reading, since the two read functions look so similar). And it may well be impossible, if the data are not aligned correctly for the integer type.
• For systems with CHAR_BIT > 8, am I doing the right thing by assuming we pack/unpack 8 bits per char for transmission? Will it even compile, given the likely absence of std::uint8_t, and should I be masking with 0xFF instead of truncating by casting?
• Should I declare a separate default constructor (using = default) instead of using the defaulted argument?

Answer 1

About your concerns

• I'd like optimised compilation to emit a simple load or store when converting to/from the host's native order if possible, but I'm not good enough at assembly languages to check that (I think that the result [on gobolt] implies that writing is well-optimised, but not reading, since the two read functions look so similar). And it may well be impossible, if the data are not aligned correctly for the integer type.

If the compiler can see it is well-aligned, then it will most likely result in optimal code (ie, no-op for native endianness, potential use of byte-swapping instructions for non-native endianness). However, if it cannot see this, that might be a problem. You can solve that by adding an alignas specifier to make sure your types have the same alignment as T would have.

You could also make the alignment requirement part of the template parameters, but default it to be the same as alignof(T).

• For systems with CHAR_BIT > 8, am I doing the right thing by assuming we pack/unpack 8 bits per char for transmission? Will it even compile, given the likely absence of std::uint8_t, and should I be masking with 0xFF instead of truncating by casting?

You have a big problem on such systems if you want to use it to store fixed-with types which are a multiple of 8 bits in size. It might not be possible; I have never heard of systems where CHAR_BITS != 8 that do support uint8_t. However, what is guaranteed in C is that you can at least tell the size of a type in the number of chars using sizeof. So you can still swap endianness, but for example on a 36-bit system with CHAR_BITS == 9, you would swap 4 9-bit characters. This might make sense if you have two 36-bit systems that have different endianness that you want to communicate between. To complicate matters even worse, there are some 32-bit systems where CHAR_BITS == 16 or even larger, as they physically cannot address anything smaller.

In any case, make sure you don't mix unsigned char and std::uint8_t like you did in your code: I recommend you stick with unsigned char, shift by CHAR_BITS, and not use any of the fixed-width types in your Endian class. If you do want to swap 8-bits at a time, then the size of the array data is wrong for CHAR_BITS > 8, it should then be (sizeof(T) * CHAR_BITS) / 8.

• Should I declare a separate default constructor (using = default)?

You already have an explicit default constructor. You could add a defaulted one if you remove the default value = 0 from the existing constructor. It probably makes sense to do this, to get the same behavior as with the built-in integral types.

More about endianness

There is more to endianness than little-endian and big-endian. There is also middle-endian. There are architectures were some types are in one endianness, others in another endianness (mostly floats vs. ints though). There are architectures where endianness is flexible, and can be changed per process or even per memory page. How are you going to handle that?

At the other extreme I would say that in practice, you nowadays don't need to worry about anything but little-endian systems where integer sizes are multiples of 8. So you only need a BigEndian class to deal with conversions to/from big-endian, nothing more. Anything else falls into the YAGNI category.

Make it constexpr

You can make the constructor and conversion operator constexpr.