5
\$\begingroup\$

Motivation

When working with storage or wire protocols, we often read or write structures containing integers with specific byte-ordering (e.g. big-endian for Internet Protocol, or little-endian for USB).

It's common to use functions such as the htons() family to convert values, but it's easy to miss a conversion (particularly if the protocol is the same endianness as the development system). Instead, I prefer to use the type system to distinguish host integers from their representations' protocol byte-order.

Known limitations (non-goals):

  • I haven't had a need to handle floating-point values, so only integers are handled here.
  • It's not suitable for protocols such as IIOP or X11 where one of the hosts chooses at run-time which endianness to use.

Example usage

I'm currently using this when sending values over the wire. A simplified example, with all the error handling removed, looks something like:

struct Response {
    BigEndian<std::uint16_t> seq_no;
    BigEndian<std::uint16_t> sample_value;
};

We send by assigning (which implicitly converts our integer value to big-endian byte sequence) and then writing the structure:

void send_result(std::uint16_t value)
{
    Response r;
    r.seq_no = counter++;
    r.sample_value = value;
    write(fd, &r, sizeof r);
}

On the receive side, we read the wire representation into the same structure (so it's bitwise identical to the sending side) and then use the conversion operator to access the data in native form:

std::uint16_t recv_result()
{
    Response r;
    read(fd, &r, sizeof r);
    // ignore seq_no, for now
    return r.sample_value;
}

Changes since previous version

Since version 1, I've changed the following:

  • Renamed the views which give us access to the storage in the desired order, and added comments to make it clearer. This was previously giving the impression that round-trip would byte-reverse the value, which is not the case.
  • Uniformly scatter/gather to native char size (in units of CHAR_BIT).
  • Added warning for non-8-bit platforms to encourage audit of octet-orientated code.
  • Optimised for reading/writing native-endian values directly with no wrapper class (will do the Right Thing for unicorn/dinosaur platforms such as mixed-endian or no-endian).
  • Split out the default constructor, since serialising zero is a no-op.
  • Added constexpr to conversion operations.
  • Demonstrate plain old round-trip in the tests, rather than modifying the storage. I think this makes the tests clearer.

Implementation

#ifndef ENDIAN_HPP
#define ENDIAN_HPP

#include <array>
#include <bit>
#include <climits>
#include <concepts>
#include <ranges>
#include <type_traits>

#ifndef ENDIAN_SUPPORT_NON_8BIT
static_assert(CHAR_BIT == 8,
              "This header splits into chars, not octets. "
              "Define ENDIAN_SUPPORT_NON_8BIT to enable.");
#endif

namespace endian
{
    namespace detail {
        template<std::integral T, // type to represent
                 auto BigFirst,    // view that presents MSB first
                 auto LittleFirst> // view that presents LSB first
        struct Endian
        {
            // We use unsigned T for bitwise operations
            using U = std::make_unsigned_t<T>;

            // The underlying storage
            std::array<unsigned char, sizeof (T)> data = {};

            constexpr Endian() = default;

            // implicit conversion from T
            constexpr Endian(T value)
            {
                // unpack value starting with the least-significant bits
                auto uval = static_cast<U>(value);
                for (auto& c: data | LittleFirst) {
                    c = static_cast<unsigned char>(uval);
                    uval >>= CHAR_BIT;
                }
            }

            // implicit conversion to T
            constexpr operator T() const
            {
                // compose value starting with most-significant bits
                U value = 0;
                for (auto c: data | BigFirst) {
                    value <<= CHAR_BIT;
                    value |= c;
                }
                return static_cast<T>(value);
            }
        };
    }

    template<std::integral T>
    using BigEndian =
        std::conditional_t<std::endian::native == std::endian::big,
                           T,   // no conversion needed
                           detail::Endian<T, std::views::all, std::views::reverse>>;

    template<std::integral T>
    using LittleEndian =
        std::conditional_t<std::endian::native == std::endian::little,
                           T,   // no conversion needed
                           detail::Endian<T, std::views::reverse, std::views::all>>;

}

#endif // ENDIAN_HPP

Unit Tests

using endian::BigEndian;
using endian::LittleEndian;

#include <gtest/gtest.h>

#include <cstdint>
#include <cstring>

// Ensure there's no padding
static_assert(sizeof (BigEndian<int>) == sizeof (int));
static_assert(sizeof (LittleEndian<int>) == sizeof (int));

// Helper function to inspect representation
template<typename T>
auto byte_array(const T& t)
{
    std::array<unsigned char, sizeof t> bytes;
    std::memcpy(bytes.data(), &t, sizeof t);
    return bytes;
}

// Now the tests themselves
TEST(big_endian, uint8)
{
    const std::uint8_t x = 2;
    auto be = BigEndian<std::uint8_t>{x};
    std::array<unsigned char, 1> expected{{2}};
    EXPECT_EQ(byte_array(be), expected);

    // round trip back to native
    std::uint8_t y = be;
    EXPECT_EQ(y, x);
}

TEST(little_endian, uint8)
{
    const std::uint8_t x = 2;
    auto le = LittleEndian<std::uint8_t>{x};
    std::array<unsigned char, 1> expected{{2}};
    EXPECT_EQ(byte_array(le), expected);

    std::uint8_t y = le;
    EXPECT_EQ(y, x);
}

TEST(big_endian, uint16)
{
    const std::uint16_t x = 0x1234;
    BigEndian<std::uint16_t> be = x;
    std::array<unsigned char, 2> expected{{0x12, 0x34}};
    EXPECT_EQ(byte_array(be), expected);

    std::uint16_t y = be;
    EXPECT_EQ(y, x);
}

TEST(little_endian, uint16)
{
    const std::uint16_t x =  0x1234;
    auto le = LittleEndian<std::uint16_t>{x};
    std::array<unsigned char, 2> expected{{0x34, 0x12}};
    EXPECT_EQ(byte_array(le), expected);

    std::uint16_t y = le;
    EXPECT_EQ(y, x);
}

TEST(big_endian, uint32)
{
    const std::uint32_t x = 0x12345678;
    auto be = BigEndian<std::uint32_t>{x};
    std::array<unsigned char, 4> expected{{ 0x12, 0x34, 0x56, 0x78 }};
    EXPECT_EQ(byte_array(be), expected);

    std::uint32_t y = be;
    EXPECT_EQ(y, x);
}

TEST(little_endian, uint32)
{
    const std::uint32_t x = 0x12345678;
    auto le = LittleEndian<std::uint32_t>{x};
    std::array<unsigned char, 4> expected{{ 0x78, 0x56, 0x34, 0x12 }};
    EXPECT_EQ(byte_array(le), expected);

    std::uint32_t y = le;
    EXPECT_EQ(y, x);
}
\$\endgroup\$

2 Answers 2

4
\$\begingroup\$

This looks very good!

About the limitations

  • I haven't had a need to handle floating-point values, so only integers are handled here.

If you did want to support floating-point values, and perhaps other types that might have an endianness, then your approach falls apart, or at least it would require some reinterpret_casting (or std::bit_casting since C++20) from the non-integer to the integer types, and it would still only work for types whose sizes match those of integers.

Another way to solve the issue would be to reinterpret the value as std::bytes, and just rearrange them into the std::array as necessary for the desired endianness.

  • It's not suitable for protocols such as IIOP or X11 where one of the hosts chooses at run-time which endianness to use.

You could add a template parameter that would switch between a statically chosen endianness and one that can be changed at run-time, analogous to std::span's std::dynamic_extent.

Add support for alignment

You did ask about this in your previous version, and I think it is still important to consider this: you probably want to make sure your Endian<T> has the same alignment as T. Consider that you have a struct like this:

struct Foo {
    char bar;
    uint16_t baz;
    uint32_t quux;
};

And now you want to ensure the endianness is fixed, so you just add your wrappers:

struct Foo {
    LittleEndian<char> bar;
    LittleEndian<uint16_t> baz;
    LittleEndian<uint32_t> quux;
};

Before, sizeof(Foo) == 8 and all variables have their natural alignment. Afterwards sizeof(Foo) == 7, and baz and quux are incorrectly aligned. The compiler will see that, and will no longer be able to optimize the loads and stores to these variables on a little-endian machine. So I would do something like:

template<std::integral T,   // type to represent
         auto BigFirst,     // view that presents MSB first
         auto LittleFirst,  // view that presents LSB first
         std::size_t Align> // alignment requirement
struct alignas(Align) Endian
{
    …
};

template<std::integral T, std::size_t Align = alignof(T)>
using LittleEndian =
    std::conditional_t<std::endian::native == std::endian::little,
                      T,   // no conversion needed
                      detail::Endian<T, std::views::reverse, std::views::all>,
                      Align>;
…

Improvements to the unit tests

I haven't used this myself, but I believe GoogleTest has some support for testing templated code, which allows you to get rid of a lot of code duplication.

Even before you do that, it might be a good exercise to rewrite the tests a bit so that the actual type you want to test is only mentioned once, and that you have a generic way to generate test values. For example:

TEST(little_endian, uint16_t)
{
    using T = std::uint16_t;
    constexpr T x = static_cast<T>(0x12345678);
    constexpr LittleEndian<T> le = x;
    std::array<unsigned_char, 4> expected_full({0x78, 0x56, 0x34, 0x12});
    decltype(byte_array(le)) expected;
    std::copy_n(expected.end() - sizeof(T), sizeof(T), expected.begin());
    EXPECT_EQ(byte_array(le), expected);
    constexpr T y = le;
    EXPECT_EQ(y, x);
}

Also consider adding support for 64-bit integers. You also want to test whether you can use your class correctly in a constexpr environment. If you add alignment, you should also test that.

\$\endgroup\$
1
  • \$\begingroup\$ I could probably default the Align parameter to alignof(T) - or perhaps omit it and just use alignas(T) in all cases. And thanks in particular for the suggested improvements to the tests; I really learnt something there! \$\endgroup\$ Feb 23, 2023 at 7:08
3
\$\begingroup\$

Adopt C++23

Since the code was written, C++23 introduced std::byteswap(), which enables a completely different implementation for the common case where native-endian is either big-endian or little-endian.

That enables us to use a T as underlying storage instead of the array, and we could assume that std::byteswap() is strongly tuned to the target architecture (it's possible a good optimiser will spot what we're doing, but best not to rely on that).

If we want to be ruthlessly portable (even to ancient mixed-endian platforms), then we'll need to fall back to the existing implementation using std::array for the storage.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.