Basis of custom C++ serialization lib

Question

I know it's been done a million times already, but I couldn't find a serialization library to suit my needs. This is the very basis of what I came up with. I know the code is ugly and unstructured, so you don't have to tell me that.

My goal is to make a serialization/deserialization system that provides a minimal, optimally non-intrusive interface for all classes, but can also store data verbosely, including variable names and type names. I will also implement a linear search-like tool that allows you to quickly access only a certain variable inside a structure hierarchy without having to serialize your whole app.

I'm wondering if any of this can be done in a more simple way. I especially don't like that I need to hard-code the class name for my classes, but I couldn't find a more simple solution to do it in a compiler-independent way.

Serializer.h:

#ifndef jag_SERIALIZER_H_
#define jag_SERIALIZER_H_

#include "jag_config.h"

namespace jag {

  namespace Serializer {

    struct SerializedData {
        SerializedData(const std::string& n, const std::string& val = "", const std::string& t = "") :
            name(n), value(val), type_name(t) {
        }

        std::string name;
        std::string value;
        std::string type_name;

        std::list<SerializedData> collection_value;
    };

    template<class T>
    struct Type {
        static const std::string GetTypeName() {
          return T::GetTypeName();
        }
    };

#define PRIMITIVE_TYPENAME(type) \
    template<>  \
    struct Type<type> { \
    static const std::string GetTypeName() { \
      return #type; \
    } };

    PRIMITIVE_TYPENAME(int)
    PRIMITIVE_TYPENAME(float)
#undef PRIMITIVE_TYPENAME

#define COLLECTION_TYPENAME(type) \
  template<class T> \
  struct Type<std::type<T>> { \
    static const std::string GetTypeName() { \
    return std::string(#type) + " of " + Type<T>::GetTypeName(); \
} };

    COLLECTION_TYPENAME(list)

#undef COLLECTION_TYPENAME

    template<class T>
    struct EnumeratedValue {
        EnumeratedValue(T& val, const std::string& n) :
            name(n), value(val), type_name(Type<T>::GetTypeName()) {
        }
        std::string name;
        T& value;
        std::string type_name;
    };

    template<class T, class Collector>
    struct GetSer {
        static SerializedData GetSerializedData(const EnumeratedValue<T>& data) {
          SerializedData sd(data.name);
          Collector coll;
          data.value.EnumerateComponents(coll);
          sd.collection_value = coll.datas;
          sd.type_name = data.type_name;
          return sd;
        }
    };

#define LEXICAL_SPEC(type) \
template<class Collector> \
    struct GetSer<type, Collector> { \
        static SerializedData GetSerializedData(const EnumeratedValue<type>& data) { \
          return SerializedData(data.name, boost::lexical_cast<std::string>(data.value), data.type_name); \
        } \
    };

    LEXICAL_SPEC(float)
    LEXICAL_SPEC(int)

#undef LEXICAL_SPEC

    template<class T, class Collector>
    struct GetSer<std::list<T>, Collector> {
        static SerializedData GetSerializedData(const EnumeratedValue<std::list<T> >& data) {
          SerializedData all_sd(data.name);

          unsigned count = 0;

          for (auto& d : data.value) {
            Collector coll;
            d.EnumerateComponents(coll);
            SerializedData sd("#" + boost::lexical_cast<std::string>(count));
            sd.type_name = Type<T>::GetTypeName();
            sd.collection_value = coll.datas;
            all_sd.collection_value.push_back(sd);
            ++count;
          }
          all_sd.type_name = data.type_name;
          return all_sd;
        }
    };

    struct SerializerCollector {
        template<class T>
        SerializerCollector& operator<<(const EnumeratedValue<T>& en) {
          SerializedData sd = GetSer<T, SerializerCollector>::GetSerializedData(en);
          datas.push_back(sd);
          return *this;
        }

        std::list<SerializedData> datas;
    };

    inline std::string FlattenSerializedData(const std::list<SerializedData>& datas, const std::string& prefix) {
      std::stringstream ret;
      for (auto& d : datas) {
        ret << "\n";

        ret << prefix << d.type_name << " " << d.name << " =";
        if (!d.value.empty()) {
          ret << " " << d.value;
        } else {
          ret << "\n" << FlattenSerializedData(d.collection_value, prefix + " ");
        }
      }

      return ret.str();
    }

    template<class T>
    std::string Serialize(const EnumeratedValue<T>& val) {
      return val.type_name + " " + val.name + " = " + FlattenSerializedData(GetSer<T, SerializerCollector>::GetSerializedData(val).collection_value, " ");
    }

#define E(x) (jag::Serializer::EnumeratedValue<decltype(x)>(x, #x))

    struct Sample2 {
        int b;
        /*
         template<class T>
         void EnumerateComponents(T& e) {
         e << E(b);
         }*/
    };

    struct Sample {
        float a;
        Sample2 s;

        Sample() :
            a(4.2f) {
          s.b = 3;
        }

        template<class T>
        void EnumerateComponents(T& e) {
          e << E(a)<< E(s);
        }
      };

    template<class Collector>
    struct GetSer<Sample2, Collector> {
        static SerializedData GetSerializedData(const EnumeratedValue<Sample2>& data) {
          SerializedData sd(data.name);
          sd.value = "go away";
          return sd;
        }
    };

  }
}

#endif /* jag_SERIALIZER_H_ */

A simple class that uses it:

  namespace Math {

    template<class T>
    struct Triangle {
        explicit Triangle(const T& p1, const T& p2, const T& p3, bool validate = true) :
            p1(p1), p2(p2), p3(p3) {
          if (validate)
            Validate();
        }

        Triangle() {
        }

        template<class B>
        void EnumerateComponents(B& e) {
          e << E(p1)<< E(p2) << E(p3);
        }

        static const std::string GetTypeName() {
          return "Triangle of " + Serializer::Type<T>::GetTypeName();
        }

        inline bool operator==(const Triangle<T>& other) const {
          return p1 == other.p1 && p2 == other.p2 && p3 == other.p3;
        }

        inline void Validate() const {
          JAG_ASSERT(p1 != p2 && p2 != p3 && p3 != p1);
        }

        inline bool Contains(const vec3& coord) const {
          return p1 == coord || p2 == coord || p3 == coord;
        }

        inline T GetSurfaceNormalCCW() const {
          Validate();

          const T u = p2 - p1;
          const T v = p3 - p1;

          vec3 norm;

          norm.x = u.y * v.z - u.z * v.y;
          norm.y = u.z * v.x - u.x * v.z;
          norm.z = u.x * v.y - u.y * v.x;

          return Math::normalize(norm);
        }

        const T* data() const {
          return &p1;
        }

        T p1;
        T p2;
        T p3;
      };

As you can see I only had to add two functions - an enumerator and a typename function.

Here's a test function:

Math::Triangle<int> tri;
tri.p1 = 2;
tri.p2 = 3;
tri.p3 = 4;

std::list<Math::Triangle<int> > trilist;
trilist.push_back(tri);
tri.p3 = 666;
trilist.push_back(tri);

JAG_LOG<< "\n" << Serializer::Serialize(E(trilist));

And finally, the result:

list of Triangle of int trilist = 
 Triangle of int #0 =

  int p1 = 2
  int p2 = 3
  int p3 = 4
 Triangle of int #1 =

  int p1 = 2
  int p2 = 3
  int p3 = 666

Answer 1

The claim "As you can see I only had to add two functions - an enumerator and a typename function." will not generally be true for every type. Specifically, you'll also need to provide an appropriate operator<< for every type you'd like to use as a member in a serializable type.

When doing so, you'll also need to pay special attention to pointers and pointer-like things. Currently, if your program tries to use a deserialized pointer that was serialized by another instance of your program, things will likely blow up. You'll want to serialize the pointed-at thing as well. And you'll need to keep track of all those pointed-at things, so that multiple pointers can point to the same thing on both ends.

There are other types that will be problematic as well, such as mutexes. Basically, unless it employs pure value semantics, it's going to be hard to serialize reliably. Trying to handle all of this in the serializer is going to get really messy really quickly. So come up with some way for classes to say "Nuh-uh, I'm gonna do my own (de)serialization".

I also have some portability and maintainability concerns regarding the way you're storing type information. As the code currently stands, I'm not sure how you plan on identifying the correct type to instantiate when deserializing.

I'd also strongly recommend adding some protocol information at the start of any serialization stream. At the very least have a couple of magic bytes, so you can avoid blowing up when faced with something that's not actually a serialization stream. If you eventually want to support this code in the wild, I'd also suggest adding versioning to your protocol.