5
\$\begingroup\$

Are the two function templates below well-formed for serializing/deserializing POD types? Will they work for all the different types that satisfy the constraint pod?

Here (live):

#include <type_traits>
#include <ostream>
#include <istream>
#include <fstream>
#include <print>
#include <stdfloat>


template <class T>
concept pod = std::is_standard_layout_v<T> && std::is_trivial_v<T>;

template <pod T>
[[ nodiscard ]] bool
put_pod( std::ostream& os, const T& value )
{
    const auto good { os.write( reinterpret_cast<const char*>( &value ), sizeof( value ) ).good( ) };
    return good;
}

template <pod T>
[[ nodiscard ]] bool
get_pod( std::istream& is, T& value )
{
    T temp_value;
    const auto good { is.read( reinterpret_cast<char*>( &temp_value ), sizeof( temp_value ) ).good( ) };
    if ( good )
        value = temp_value;
    return good;
}

int main( )
{
    {
        std::ofstream ofs { "data.bin", std::ios::binary | std::ios::app };
        if ( !ofs.is_open( ) ) return 1;

        const std::float32_t value { 7.00035f32 };
        const auto good { put_pod( ofs, value ) };
        if ( !good ) return 1;
        std::println( "{}", value );
    }

    std::println( "-----------------------------" );

    {
        std::ifstream ifs { "data.bin", std::ios::binary };
        if ( !ifs.is_open( ) ) return 1;

        while ( true )
        {
            std::float32_t value;
            const auto good { get_pod( ifs, value ) };
            if ( !good ) break;
            std::println( "{}", value );
        }
    }
}

Also what other constraints should I use? For instance, std::default_initializable and std::destructible might be useful for get_pod since it both default constructs an instance of type T and then destructs it. However, I'm not sure if it's necessary in this case.

\$\endgroup\$
1

2 Answers 2

6
\$\begingroup\$

This is not a good idea

To answer your questions directly:

Are the two function templates below well-formed for serializing/deserializing POD types?

Yes, they are “well-formed”.

Will they work for all the different types that satisfy the constraint pod?

No.

The C++ standard guarantees that serializing and de-serializing data the way you are doing will only round-trip correctly when you run the exact same program. I want to stress that again: the EXACT same program. (And, probably only under the same conditions, though the standard has nothing to say about what those conditions might be.)

That means:

  • This is not guaranteed to work across different hardware (x86 and ARM, for example).
  • This is not even guaranteed to work if the hardware uses the same ISA (two systems running ARMv8, for example).
  • This is not guaranteed to work on a single system using two different operating systems (if you dual boot on a single machine and try writing with Windows and reading with Linux, or use two different containers or VMs, for example).
  • This is not guaranteed to work even on the same platform, if different compilers are used (GCC for the writer, Clang for the reader, for example).
  • This is not guaranteed to work even on the same platform, with the same compiler, if different versions are used (GCC 12 and 13, for example).
  • This is not even guaranteed to work on the same platform, with the same version of the same compiler, if different compiler flags are used.
  • And this might blow your mind, but even if you take the exact same source code, compile it twice for the exact same platform, with the exact same compiler, with the exact same settings, the two executables are not guaranteed to be able to work together. As in, data serialized out of one might not be compatible as input for the other. It would be a truly evil compiler that did this deliberately; however… and this is the point I’m trying to hammer home… the standard does not promise that this WON’T happen. (And, it might happen accidentally, if some setting gets unknowingly flipped.)

And if all the above doesn’t scare you out of this idea, this next point should: IF there is something… anythingany incompatibility introduced by a difference in platform, compiler, settings, the phase of the moon, whatever… NOTHING will save you. There is no practical way to guarantee that any data serialized out can be de-serialized back in correctly. You will not be saved by the type system. You will not be saved by any clever programming; even the best-designed and -written types are vulnerable. You will not be given any warnings or errors or any indication of trouble. Your program will simply do something unexpected. THAT should terrify you.

I would say the only conceivable use for such a serialization facility would be for backing up/restoring data for a single program, and probably even for a single program run. It might be useful if, say your program will be going to sleep: you can dump the contents of memory, go to sleep, then wake back up and restore the program state. Or if the program is doing one expensive task, but has to stop to do another: it can dump the data for the first task, do the second, then restore the state of the first task and continue. Maaaaaybe it might be useful to save program state, shut down the program, then restart it later and restore the state to where you left off… but that would be highly dangerous; you would need some way to determine that the state is still safe, and there probably is no foolproof way to do that.

If you were thinking of using this serialization method to save files to be loaded on other machines, or even on the same machine by different programs, or even using this method to persist data for a single program long-term… dear god, no. Don’t.

Serializing data to binary format, for very fast save/load times, is possible. But not like this. Not just carelessly barfing raw memory into a file.

“POD” is a useless classification

This is particularly true for this use case on two levels.

First, let’s look at your concept:

template <class T>
concept pod = std::is_standard_layout_v<T> && std::is_trivial_v<T>;

The layout has literally no relevance to what you are trying to do.

It doesn’t matter if a type is standard layout or not if all you’re interested in is whether you can bit-blast data into it. Consider the following class:

class foo
{
public:
    int x;
private:
    int y;
};

This is not standard layout, but it is trivial. You could dump its memory representation and reload it, no problem. (Well, no problem aside from all the problems mentioned above. But let’s assume that’s a given.)

All you care about is whether the type is trivial, and… and I’ll get to this later… specifically if it’s just trivially copyable.

The second level of uselessness of “POD” is specific to what you’re doing.

Just because a type is a “POD” type doesn’t mean it can be serialized in the way you are trying to do. Take std::string_view, for example. std::string_view is technically not trivial because it does not have a trivial default constructor, but that isn’t relevant here. It is trivially-copyable, and it is standard-layout (at least in both libstdc++ and libc++, and it may be mandated to be so but I can’t say for sure off the top of my head). However, serializing then de-serializing a std::string_view by merely copying its bits would be a terrible idea.

This is why the answer to “Will they work for all the different types that satisfy the constraint pod?” is unequivocally “no”. There are many types that satisfy “POD” (or “standard layout”, or “trivially copyable”) that will not serialize/deserialize properly by merely bit-blasting their guts.

So, basically:

  • “POD” is unnecessarily strict for what you are doing; and
  • “POD” is excessively loose for what you are doing.

Or even more basically: “POD” is wrong for what you are doing.

But this isn’t really an issue with the “POD” classification specifically. NOTHING will work as a general concept for a single serialization strategy.

Trying to blindly suck an entire category of types into a common serialization strategy is doomed to fail. The only thing that knows whether and how it is safe to serialize a type is the type itself. At the very least, types should be have to opt-in to whether they are okay with being serialized by having their guts copied (or by having their data members individually serialized, or by any method).

Or, put another way, it is not so much that “POD” is wrong for what you are doing, as it is that what you are doing is wrong.

Don’t use IOStreams’s .good()

All of your error checking is done by using the .good() member function of input/output streams. That is not correct.

The reason is that .good() doesn’t just check the stream’s error state; it also checks for EOF. But EOF is not an error. It is possible for a stream’s EOF bit to be set even for a successful read. Here is a simple example. Note that the read was successful, but .good() returned false.

.good() has a very specific use case… and it’s not what you’re doing. The correct way to check for a read/write error is to use the stream’s bool conversion. Note that in the example above, the bool conversion correctly returns true for a successful read. It will ALWAYS return true for a successful read/write… and never return true if a read/write failed for any reason.

Your get_pod() is needlessly complex

So you want to confirm that the read was successful before actually changing value. That’s good. To do that, you use a temporary T as a buffer. That’s not so good.

Because you do this, you introduce the pointless necessity that T has to be default constructible, even though you already have a T (value).

Consider instead that while you do need a temporary place to put the input so you can verify the read succeeded before touching value… that temporary place does not need to be a T.

For example:

template <pod T>
[[nodiscard]] auto get_pod(std::istream& is, T& value) -> bool
{
    alignas(T) char buffer[sizeof(T)];

    if (is.read(buffer, sizeof(T)))
    {
        std::memcpy(&value, buffer, sizeof(T));
        return true;
    }

    return false;
}

The alignas(T) is technically not necessary, but if the buffer has the same alignment as the value, then there is a better chance of being able to copy the memory in chunks, rather than byte-by-byte.

Now that you no longer care whether the type is default constructible, you can loosen the constraints:

template <typename T>
requires std::is_trivially_copyable_v<T>
[[nodiscard]] auto put_pod(std::ostream& os, T const& value)
{
    return bool{os.write(reinterpret_cast<char const*>(&value), sizeof(T))};
}

template <typename T>
requires std::is_trivially_copyable_v<T>
[[nodiscard]] auto get_pod(std::istream& is, T& value)
{
    alignas(T) char buffer[sizeof(T)];

    if (is.read(buffer, sizeof(T)))
    {
        std::memcpy(&value, buffer, sizeof(T));
        return true;
    }

    return false;
}

But, again, this is not a good idea.

A recommendation for a serialization API

So you want a serialize arbitrary types. Fine, but you can’t just blindly assume that all types are serializable, or deserializable, or neither or both. And you can’t assume that all types will want to be serialized in the same way.

What I would recommend is to make a single function overload set for serializing, and another for de-serialzing.

For example:

namespace whatevs {

// Don't actually write these; just illustrating the API.
[[nodiscard]] auto serialize(std::ostream& o, T const& t) -> std::ostream&;
[[nodiscard]] auto deserialize(std::istream& o, T& t) -> std::istream&;

} // namespace whatevs

Then create overloads for the basic types:

namespace whatevs {

[[nodiscard]] auto serialize(std::ostream& o, int t) -> std::ostream&
{
    // whatever logic to serialize an int
}

[[nodiscard]] auto deserialize(std::istream& o, int& t) -> std::istream&
{
    // whatever logic to deserialize an int
}

// Repeat for all built-in types.
//
// Note that you don't *literally* need to write the functions above for
// every built-in type. There are ways to avoid duplicating work.
//
// You can also add types like std::string.

} // namespace whatevs

Now anyone who wants their type to be serializable can simply define this function for their type:

namespace indi {

struct my_type
{
    int x;
    double y;
};

} // namespace indi

[[nodiscard]] auto whatevs::serialize(std::ostream& o, indi::my_type const& t) -> std::ostream&
{
    whatevs::serialize(o, t.x);
    whatevs::serialize(o, t.y);
    return o;
}

[[nodiscard]] auto whatevs::deserialize(std::istream& o, indi::my_type& t) -> std::istream&
{
    // Or use temporaries to hold the data, and don't touch t
    // unless all reads succeed.
    whatevs::deserialize(i, t.x);
    whatevs::deserialize(i, t.y);
    return i;
}

There are ways you can make this easier and more ergonomic (you might want to make the actual (de)serialize function a neibloid, for example), but that’s the gist.

Now you’re probably thinking that it sucks that you can’t auto-generate the obvious default serialize/deserialize function. It does, but that capability is coming. It might look like this:

namespace whatevs {

template <typename T>
requires requires { T::default_serializable; }
[[nodiscard]] auto serialize(std::ostream& o, T const& t) -> std::ostream&
{
    template for (constexpr auto member : std::meta::nonstatic_data_members_of(^T))
    {
        if (not serialize(o, t.[:member:]))
            break;
    }

    return o;
}

} // namespace whatevs

And you’d opt into it like this:

namespace indi {

struct my_type
{
    static constexpr auto default_serializable = true;

    int x;
    double y;
};

} // namespace indi

Something like that, basically.

But that’s the future… possibly as early as C++26. For now, you’ll have to hand-roll the serialize/de-serialize functions.

\$\endgroup\$
20
  • 1
    \$\begingroup\$ Thanks for the insights. I got the idea of serialization/serialization in this way from a fairly new C++ book. It's a shame that almost none of the pitfalls you mentioned are not mentioned in that book. \$\endgroup\$
    – digito_evo
    Feb 29 at 5:47
  • \$\begingroup\$ "// whatever logic to serialize an int" so what exactly should this logic be? I'm not aiming to run the program on different machines. Is there a way I can assure that things will go well through some assertion? \$\endgroup\$
    – digito_evo
    Feb 29 at 5:50
  • 1
    \$\begingroup\$ Welp, it's time to look back at some of my old code and see if some of my old files are still readable 😅 \$\endgroup\$
    – hegel5000
    Feb 29 at 18:01
  • 1
    \$\begingroup\$ I would avoid using '|' as a seporator as event that can be in the string. My prefered way of serializign the string is to prefix it with the length then print it. <length>:<String> Then reading is: read the length (remove the colon) make sure the destination is large enough, read the specified number of characters from input into the destination. \$\endgroup\$ Mar 1 at 1:58
  • 1
    \$\begingroup\$ It is possible to design a portable binary format. Numerous file formats already work that way. It even makes sense to design that format so that, on your “preferred” platform, with the right compiler settings, you can just memcpy() stuff in and out. The fast memcpy() code will be highly unportable, and platform/compiler-specific, but it could work as special-case code, with portable generic code doing the I/O “properly” (albeit a bit slower). \$\endgroup\$
    – indi
    Mar 2 at 21:33
6
\$\begingroup\$

Personal opinion here (so take or leave as you like).

Putting the bool here on a separate line to the function makes it harder to parse for me. I always look for the return type as the first part of the function.

I know my view may be data with introduction of the auto return type and putting the return value on the right. So you can safely ignore this opinion.

template <pod T>
[[ nodiscard ]] bool
put_pod( std::ostream& os, const T& value )

Personally I wold write like this:

template <pod T>
[[ nodiscard ]]
bool put_pod(std::ostream& os, T const& value)

Looking at the syntax.

This seems like a very awkward way:

    const auto good { os.write( reinterpret_cast<const char*>( &value ), sizeof( value ) ).good( ) };

    // to write:

    const bool good = os.write( reinterpret_cast<const char*>( &value ), sizeof( value ));

Or even simpler. I would simply write:

    return os.write( reinterpret_cast<const char*>(&value), sizeof( value);

Looking at what the operation is actually doing:

 os.write( reinterpret_cast<const char*>(&value), sizeof( value);

This is exceedingly brittle.

The trouble with C++ is that very few things have standard size or layout. It is left to the compiler manufacturer and potentially compiler flags to define a bunch of things about size and layout of the object.

What I am saying is the resulting-serialized object is:

  1. Not human-readable (so serialization is a stretch).
  2. The size of any types will be dependent on your current system.
    OS/OS Version/Compiler/Compiler Version/Compiler flags.
  3. The layout of the object (padding) is not standardized.
  4. The format of objects is not standardized.
    The format of these is not defined: int, float, double etc.

NOTE: std::is_standard_layout_v<T> && std::is_trivial_v<T> covers structures as well as the standard POD types.

So this will work fine in the short term. On your current system. On two different machines with the exact same configuration. Etc. So great for testing and experimenting. But this is likely to break over time or with exposure to larger user bases where machine configuration can not be guaranteed.

Your best bet is to serialize each individual member in a way that you have defined. Then you know the format/size etc.


Same points for the get_pod() function.


Overall:

I don't think this matches standard C++ conventions. Where serialization is usually done via the << operator and deserialization is done via >> operator.

So where you have done:

bool good = put_pod( ofs, value );
if (!good) {
    return 1;
}

I would have expected:

if (!(ofs << put_pod(value))) {
    return 1;
}

How I would write it:

template<typename T>
struct PODSerializer
{
     T const& value;
};
template<typename T>
std::ostream& operator<<(std::ostream& stream, PODSerializer<T> const& object) {
    stream.write(reinterpret_cast<char const*>(&object.value), sizeof(object.value));
    return stream;
}
template<typename T>
PODSerializer<T> put_pod(const T& value)
{
    return PODSerializer<T>{value};
}
\$\endgroup\$
6
  • \$\begingroup\$ Returning a bool makes the error handling straightforward. I'm not sure if I understood the purpose of introducing the PODSerializer<T> struct. What advantage does that approach have over the simpler one (mine)? \$\endgroup\$
    – digito_evo
    Feb 28 at 18:52
  • \$\begingroup\$ Also the operator bool of std::istream is not the equivalent of good() because the former doesn't check the EOF flag. So in case that is set then the read operation has probably failed and my function returns false in that case. Operator bool would report success which is wrong. \$\endgroup\$
    – digito_evo
    Feb 28 at 19:00
  • \$\begingroup\$ You should use operator bool or !fail() on a stream to check for success. Note: If you try and read past the end of stream it will set fail and EOF flags and give you the correct state. EOF does not mean things have failed (that is why you distinguish between EOF and fail). \$\endgroup\$ Feb 28 at 20:33
  • \$\begingroup\$ Returning a bool makes the error handling straightforward. The streams are already set up to give you the correct boolean value when you check them. So all reads should be done in a boolean context: if (stream >> object) do some action if red worked. while (stream >> objet) loop continues while read succeedes. \$\endgroup\$ Feb 28 at 20:34
  • \$\begingroup\$ What advantage does that approach have over the simpler one. It allows you zero cost specialization of the operator<< and operator>>. \$\endgroup\$ Feb 28 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.