2
\$\begingroup\$

I am working on a data recorder in my library for pseudo-realtime systems. I have a default templated record() function to cover basic data types, and library users can specialize that function for custom data types (e.g. for images they could write a flag to the main data file then record the image to a directory somewhere else).

It is expected that data from sensors (e.g. gyro, accel, ...) could be coming as fast as 1kHz each and there will be several log lines for each measurement accounting for logging of algorithm intermediaries running against those sensors.

The default record function itself is quite simple:

template <typename T>
void DataRecorder::record(int sourceId, const T& value, const timepoint& time) {
    // TODO: Record in binary
    mStream << fixed << time_utils::as<seconds>(clock::now()) << " "; // time data was written to file (should be monotonic)
    mStream << fixed << time_utils::as<seconds>(time) << " "; // time data was created
    mStream << typeId<T>() << " ";
    mStream << sourceId << " ";
    mStream.precision(numeric_limits<T>::max_digits10);
    mStream << value << endl;
}

template <typename T>
constexpr int DataRecorder::typeId() const {
    static_assert(DataRecorderType<T>::enabled, "Data type not registered with recorder");
    return DataRecorderType<T>::id;
}

What I am working through now is how to register types so the DataRecorder knows about them. There are three basic requirements:

  1. Given a type that instantiated the record template, we need to get the id that goes in the log file (the typeId() function)

  2. The type id should be consistent (independent of builds, compilers, machines, etc)

  3. At some point during runtime, make a list of all types and type identifier strings to create an ids.txt file associating type id's with type names (e.g. type 1 is int, type 2 is float, ...)

with the preference being to do things at compile time to avoid the need to look through a map at runtime.

Below is what I have for the first two requirements, which doesn't seem too bad, but I wish I could find a way to avoid the macro.

template <typename T>
struct DataRecorderType {
    static constexpr bool enabled=false;
};

template <size_t N>
struct DataRecorderRecord {
    static constexpr bool enabled = false;;
    static constexpr char const *name = "";
};

constexpr int DATATYPES_MAX = 200;
constexpr int DATATYPES_MIN = -200;
#define REGISTER_DATARECORDER_TYPE(typeId, type) \
    static_assert(typeId <= DATATYPES_MAX, "Data type id must be <= 200"); \
    static_assert(typeId >= DATATYPES_MIN, "Data type id must be >= -200"); \
    template <> \
    struct DataRecorderType<type> { \
        static constexpr bool enabled = true; \
        static constexpr size_t id = typeId; \
    }; \
    template <> \
    struct DataRecorderRecord<typeId> { \
        static constexpr bool enabled = true; \
        static constexpr char const *name = #type; \
    };

REGISTER_DATARECORDER_TYPE(0, int);
REGISTER_DATARECORDER_TYPE(1, float);
REGISTER_DATARECORDER_TYPE(2, double);

The last requirement is where things get ugly relying on template macro programming (which I am not comfortable with) and this solution's template recursion is what places the limit on allowable data type ids (the [-200 200] range above). This would cleanup a bit with c++17's constexpr if but still doesn't seem the best.

template <size_t N>
typename std::enable_if<DataRecorderRecord<N>::enabled, std::true_type>::type IsEnabled() { return std::true_type{}; }

template <size_t N>
typename std::enable_if<!DataRecorderRecord<N>::enabled, std::false_type>::type IsEnabled() { return std::false_type{}; }

// protoype this so the true version can find it
template <size_t N> const std::string id_strings(std::false_type);
template <size_t N>
const std::string id_strings(std::true_type) {
    return std::to_string(N) + " " + std::string(DataRecorderRecord<N>::name) + "\n" + id_strings<N+1>( IsEnabled<N+1>() );
}

template <size_t N>
const std::string id_strings(std::false_type) {
    return id_strings<N+1>( IsEnabled<N+1>() );
}

template <>
const std::string id_strings<DATATYPES_MAX>(std::true_type) {
    return std::to_string(DATATYPES_MAX) + " " + std::string(DataRecorderRecord<DATATYPES_MAX>::name) + "\n";
}

template <>
const std::string id_strings<DATATYPES_MAX>(std::false_type) {
    return "";
}

const std::string id_strings() {
    return id_strings<DATATYPES_MIN>( IsEnabled<DATATYPES_MIN>() );
}

Given that, the id file is written via:

void DataRecorder::writeIdFile(const string& filename) const {
    ofstream file(filename);
    if(!file.is_open()) {
        cerr << "Failed to open file: " << filename << endl;
        return;
    }

    file << id_strings();
}

One other thing I have considered is to have the user basically start with the id file and parse that with a preprocessor tool to generate header files to compile against.

I am not 100% about this but it would also require having library users run the tool as well against their own files defining additional types.

\$\endgroup\$
  • \$\begingroup\$ "The type id should be consistent (independent of builds, compilers, machines, etc)" - this one's going to be difficult without specifying the value explicitly per class. The reason is that you can't guarantee link order and therefore registration order. \$\endgroup\$ – Richard Hodges Jan 14 '17 at 23:28
  • \$\begingroup\$ yeah, I realized that too. I've resigned myself to needing explicit registration commands for each type but it will fail at compile time if a registration is missing so shouldn't be too big a deal. It will be most painful for templated classes (like the eigen library) \$\endgroup\$ – ryan0270 Jan 15 '17 at 16:23
  • \$\begingroup\$ Is there perhaps another way to achieve what you want? \$\endgroup\$ – Richard Hodges Jan 15 '17 at 17:03
  • \$\begingroup\$ I'm sure there are and I'm open to suggestions. The biggest limitations may be skill levels: I'm not really much of a library writer but the tools I'm working on will be used across multiple projects of my own as well as other guys in my lab (engineers who happen to program) \$\endgroup\$ – ryan0270 Jan 16 '17 at 0:34
  • \$\begingroup\$ I mean is there another way to rethink marshalling the data? Protocol buffers, for example? \$\endgroup\$ – Richard Hodges Jan 16 '17 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.