4
\$\begingroup\$

Following up on my previous Column work:
Column File Format Stream

It is now available in github: https://github.com/Loki-Astari/ThorsStorage

I have incorporated the comment by @L.F. but have done a lot more work since then. Unfortunately the main code is now at 798 lines so too much for a single review. I will break it into a couple of logical chunks. Any feedback appreciated.

file.h

#ifndef THORSANVIL_FILESYSTEM_COLUMNFORMAT_FILE_H
#define THORSANVIL_FILESYSTEM_COLUMNFORMAT_FILE_H

#include "filesystem.h"
#include <ThorSerialize/Traits.h>
#include <string>
#include <tuple>

// See: https://codereview.stackexchange.com/questions/81922/macro-to-build-type-declaration
// For details about the Traits type and how it is built.

namespace ThorsAnvil::FileSystem::ColumnFormat
{
    /*
     * This header defines two main types.
     * FileBase and FileMembers
     *
     * The FileMembers class holds a tuple of all subfiles and mainly deals with distributing any call on the class to all submembers.
     * Just to prove all rules have exceptions, the FileMembers class also holds a state member that is the binary or of all the other
     * subfiles state flags. This makes checking the overall state of the object simler.
     *
     * The FileBase class acts as the logical File object.
     * It contains any buisness logic associated with a file (including its primary name).
     *
     * The user variants of FileBase are:
     *      IFile:      Input File
     *      OFile:      Output File
     *      File:       Can be either in or out or in/out just need to be specific in the open flag.
     */

    // Need to forward declare FileMembers as it is used in the Implementation details section.
    template<typename S, typename T>
    class FileMembers;

    namespace Impl
    {
        /*
         * This section contains some utility class to help in building types (there is no code here)
         * NamingConvention:
         *      XXXSelector:    A class partial specialization that helps select a type for XXX based on input type T.
         *      XXXBuilder:     A class partial specialization that builds a tuple type XXX based on input type Arg...
         *
         * Normally Selector and builder are used together to build a tuple of types.
         *
         * GetPointerMember:    Given a pointer to a member (Type). Returns the value being pointed at.
         * FileType:            Given a type T The type of file we will use to store it.
         *                      A: If T is a POD type this is type S (which will be one std::ifstream, std::ofstream, std::fstream)
         *                      B: If T is a std::string this type is a struct with S being used to hold data and std::fstream used to hold an index into the strings.
         *                         This is so given a position we can quickly seek to a position in the file where the string is held.
         *                      C: If T is an object mapped by ThorsAnvil_MakeTrait (see ThorsSerializer) then File<S, T>.
         *                         Thus we map a structure to multiple files one file for each members. If a member is another structure
         *                         this becomes a subdirectory with each of its memebrs mapped to a file in this subdirectory.
         * TupleFileType:       Given a class T; This becomes a std::tuple<FileType...> one member of the tuple for each member of the class.
         *
         * PreOpenState:        When opening a file we do a pre-scan to decide if any file will fail to open.
         *                      We track the state of how we are doing with this type so we can tidy up if we decide the open will fail.
         * OpenState:           Given a type T the type is used to store state for an attempted opening.
         *                      A: If T is POD or std::string then PreOpenState
         *                      B: If T is an object mapped by ThorsAnvil_MakeTrait (see ThorsSerializer) then std::tuple<OpenState...>
         * OpenStateTuple:      Given a class T; this becomes a std::tuple<OpenState...> one member of the tuple for each member of the class.
         * OpenMemberTuple:     A Utility to help.
         */

        // Get the type being pointed at by a pointer to member variable.
        template<typename P>
        struct GetPointerMember;

        template<typename R, typename T>
        struct GetPointerMember<std::pair<char const*, R T::*>>
        {
            using ReturnType = R;
        };

        template<typename P>
        using GetPointerMemberType = typename GetPointerMember<P>::ReturnType;

        /*
         * FileTypeSelector:    Use template specialization to define the stream class used.
         *                      For basic objects this is `std::fstream`
         *                      For Json::Map types use a FileMembers<S, T> types as this will recursively contain
         *                      File<M> or `std::fstream` types.
         */
        template<typename S, typename T, ThorsAnvil::Serialize::TraitType type = ThorsAnvil::Serialize::Traits<T>::type>
        struct FileTypeSelector;

        template<typename S, typename T>
        struct FileTypeSelector<S, T, ThorsAnvil::Serialize::TraitType::Value>
        {
            using FileType  = S;
        };

        template<typename S>
        struct FileTypeSelector<S, std::string, ThorsAnvil::Serialize::TraitType::Value>
        {
            using FileType  = struct FileTypeStruct
            {
                S               data;
                std::fstream    index;
            };
        };

        template<typename S, typename T>
        struct FileTypeSelector<S, T, ThorsAnvil::Serialize::TraitType::Map>
        {
            using FileType  = FileMembers<S, T>;
        };

        template<typename S, typename T>
        using FileType = typename FileTypeSelector<S, T>::FileType;

        /*
         * FileTupleBuilder:    Iterate over a tuple to get the stream types.
         */
        template<typename S, typename T, typename TUP = typename ThorsAnvil::Serialize::Traits<T>::Members>
        struct TupleFileTypeBuilder;

        template<typename S, typename T, typename... Args>
        struct TupleFileTypeBuilder<S, T, std::tuple<Args...>>
        {
            using TupleFileType = std::tuple<FileType<S, GetPointerMemberType<Args>>...>;
        };

        template<typename S, typename T>
        using TupleFileType = typename TupleFileTypeBuilder<S, T>::TupleFileType;

        /*
         * OpenStateSelector:   Select if we use PreOpenState (for std::fstream) or a struct (for FileMembers)
         */
        enum PreOpenState {NoAction, NoDir, DirExists};

        template<typename T, ThorsAnvil::Serialize::TraitType type = ThorsAnvil::Serialize::Traits<T>::type>
        struct OpenStateSelector;

        /*
         * OpenStateBuilder:   Build a tuple of (OpenStateSelector) for the underlying stream types.
         */
        template<typename T>
        struct OpenStateTupleBuilder;

        template<typename... Args>
        struct OpenStateTupleBuilder<std::tuple<Args...>>
        {
            using OpenStateTuple = std::tuple<typename OpenStateSelector<GetPointerMemberType<Args>>::OpenState...>;
        };

        template<typename T>
        using OpenStateTuple = typename OpenStateTupleBuilder<T>::OpenStateTuple;

        template<typename T>
        struct OpenStateSelector<T, ThorsAnvil::Serialize::TraitType::Value>
        {
            using OpenState = PreOpenState;
        };

        template<typename T>
        struct OpenStateSelector<T, ThorsAnvil::Serialize::TraitType::Map>
        {
            struct OpenState
            {
                using OpenMemberTuple = OpenStateTuple<typename ThorsAnvil::Serialize::Traits<T>::Members>;
                PreOpenState        base;
                OpenMemberTuple     members;
            };
        };

        /*
         * The types used after we have built it from the above
         */
        template<typename T>
        using OpenState      = typename OpenStateSelector<T>::OpenState;
        template<typename T>
        using OpenMemberTuple = typename OpenState<T>::OpenMemberTuple;

        // Forward declaration of FileAccessObjectType See file.tpp for details.
        template<typename F, typename T, ThorsAnvil::Serialize::TraitType type = ThorsAnvil::Serialize::Traits<T>::type>
        struct FileAccessObjectType;
    }

    using streampos = unsigned long;
    using streamoff = signed long;
    using seekdir   = std::ios_base::seekdir;

    template<typename S, typename T>
    class FileMembers
    {
        using Traits    = ThorsAnvil::Serialize::Traits<T>;
        using Members   = typename Traits::Members;
        using Index     = std::make_index_sequence<std::tuple_size<Members>::value>;

        using FileTuple = Impl::TupleFileType<S, T>;

        FileTuple       fileTuple;
        iostate         state;

        public:
             FileMembers();

            Impl::OpenState<T> doOpenTry(bool& ok, std::string const& path, openmode mode);
            void doOpenFin(bool ok, std::string const& path, openmode mode, Impl::OpenState<T> const& state);
            void close()                            {doCloseMembers(Index{});}
            void read(T& data)                      {readMembers(data, Index{});}
            void write(T const& data)               {writeMembers(data, Index{});}
            void setstate(iostate extraState)       {setstateLocalOnly(extraState); setstateMembers(extraState, Index{});}
            void clear(iostate newState = goodbit)  {clearLocalOnly(newState);      clearMembers(newState, Index{});}
            void seekg(streampos pos)               {seekgMembers(pos, Index{});}
            void seekp(streampos pos)               {seekpMembers(pos, Index{});}

            // https://en.cppreference.com/w/cpp/io/ios_base/iostate
            bool good()                             const   {return !(state & (eofbit | badbit | failbit));}
            bool eof()                              const   {return state & eofbit;}
            bool bad()                              const   {return state & badbit;}
            bool fail()                             const   {return state & (failbit | badbit);}
            operator bool()                         const   {return !fail();}
            bool operator!()                        const   {return !static_cast<bool>(*this);}
            iostate rdstate()                       const   {return state;}
        private:
            template<std::size_t... I>
            Impl::OpenMemberTuple<T> doOpenTryMembers(bool& ok, std::string const& path, openmode mode, std::index_sequence<I...>);
            template<std::size_t... I>
            void doOpenFinMembers(bool ok, std::string const& path, openmode mode, Impl::OpenMemberTuple<T> const& state, std::index_sequence<I...>);

            template<std::size_t... I>
            void doCloseMembers(std::index_sequence<I...>);

            template<std::size_t... I>
            void readMembers(T& data, std::index_sequence<I...>);

            template<std::size_t... I>
            void writeMembers(T const& data, std::index_sequence<I...>);

            template<std::size_t... I>
            void setstateMembers(iostate extraState, std::index_sequence<I...>);

            template<std::size_t... I>
            void clearMembers(iostate newState, std::index_sequence<I...>);

            template<std::size_t... I>
            void seekgMembers(streampos pos, std::index_sequence<I...>);

            template<std::size_t... I>
            void seekpMembers(streampos pos, std::index_sequence<I...>);

            template<std::size_t I>
            std::string getMemberFilePath(std::string const& path);

        protected:
            void setstateLocalOnly(iostate extraState)      {state |= extraState;}
            void clearLocalOnly(iostate newState)           {state = newState;}
    };

    template<typename S, typename T>
    class FileBase: public FileMembers<S, T>
    {
        bool            fileOpened;
        std::string     baseFileName;
        std::fstream    index;
        streampos       getPos;
        streampos       putPos;

        public:
            FileBase(std::string fileName = "", openmode mode = 0);

            void open(std::string fileName, openmode mode = 0);
            void close();

            void read(T& data);
            void write(T const& data);

            streampos   tellg() {return getPos;}
            streampos   tellp() {return putPos;}
            void        seekg(streampos pos);
            void        seekp(streampos pos);
            void        seekg(streamoff off, seekdir dir);
            void        seekp(streamoff off, seekdir dir);

            friend FileBase& operator>>(FileBase& file, T& data)
            {
                file.read(data);
                return file;
            }
            friend FileBase& operator<<(FileBase& file, T const& data)
            {
                file.write(data);
                return file;
            }
        private:
            void open(openmode mode);
    };

    template<typename T>
    class IFile: public FileBase<std::ifstream, T>
    {
        public:
            IFile(std::string fileName = "", openmode mode = std::ios::in)
                : FileBase<std::ifstream, T>(std::forward<std::string>(fileName), mode)
            {}
            void open(std::string fileName, openmode mode = std::ios::in)
            {
                return FileBase<std::ifstream, T>::open(std::forward<std::string>(fileName), mode);
            }
    };
    template<typename T>
    class OFile: public FileBase<std::ofstream, T>
    {
        public:
            OFile(std::string fileName = "", openmode mode = std::ios::out)
                : FileBase<std::ofstream, T>(std::forward<std::string>(fileName), mode)
            {}
            void open(std::string fileName, openmode mode = std::ios::out)
            {
                return FileBase<std::ofstream, T>::open(std::forward<std::string>(fileName), mode);
            }
    };
    template<typename T>
    class File: public FileBase<std::fstream, T>
    {
        public:
            File(std::string fileName = "", openmode mode = std::ios::in | std::ios::out)
                : FileBase<std::fstream, T>(std::forward<std::string>(fileName), mode)
            {}
            void open(std::string fileName, openmode mode = std::ios::in | std::ios::out)
            {
                return FileBase<std::fstream, T>::open(std::forward<std::string>(fileName), mode);
            }
    };

}


#endif

File.tpp

#ifndef THORSANVIL_FILESYSTEM_COLUMNFORMAT_FILE_TPP
#define THORSANVIL_FILESYSTEM_COLUMNFORMAT_FILE_TPP

#include "file.h"
#include <iostream>
#include <string_view>

namespace ThorsAnvil::FileSystem::ColumnFormat
{
    namespace Impl
    {
        /*
         * FileAccessObjectType:        Given a type T knows how to access the underlying File for the type.
         *                              Note the file type will be defined in file.h by Impl::FileType
         *
         *                              Note we use the template type F to represent the type as FileType is specialized
         *                              by S (the stream) which could be (std::ifstream, std::ofstream, std::stream).
         *
         * But there are three basic versions:
         *                      Given a type T The type of file we will use to store it.
         *                      A: If T is a POD type this is type S (which will be one std::ifstream, std::ofstream, std::fstream)
         *                              For most operations we simply pass on the call,
         *                      B: If T is a std::string this type is a struct with S being used to hold data and std::fstream used to hold an index into the strings.
         *                              For most operations we simply pass on the call.
         *                              For writes we add an additional write for the index to the start of the next line.
         *                              seekp() and seekg() read the index files before seeking in the data file.
         *                      C: If T is an object mapped by ThorsAnvil_MakeTrait (see ThorsSerializer) then File<S, T>.
         *                              For most operations we simply pass on the call,
         */

        // Default versions handles case C: the File type is File<S, T>
        template<typename F, typename T, ThorsAnvil::Serialize::TraitType type>
        struct FileAccessObjectType
        {
            F&  file;
            FileAccessObjectType(F& file)
                : file(file)
            {}

            OpenState<T> openTry(bool& ok, std::string&& path, openmode mode)
            {
                return file.doOpenTry(ok, std::move(path), mode);
            }
            void openFinalize(bool ok, std::string&& path, openmode mode, OpenState<T> const& state)
            {
                file.doOpenFin(ok, std::move(path), mode, state);
            }
            void close()                                {file.close();}
            void read(T& obj)                           {file.read(obj);}
            void write(T const& obj) const              {file.write(obj);}
            void setstate(iostate extraState)           {file.setstate(extraState);}
            void clear(iostate newState)                {file.clear(newState);}
            iostate rdstate() const                     {return file.rdstate();}
            void seekg(streampos pos)                   {file.seekg(pos);}
            void seekp(streampos pos)                   {file.seekp(pos);}
        };

        // This specialization for Value types handles all POD types and F is a standrard library strean
        template<typename F, typename T>
        struct FileAccessObjectType<F, T, ThorsAnvil::Serialize::TraitType::Value>
        {
            F&  file;
            FileAccessObjectType(F& file)
                : file(file)
            {}

            PreOpenState openTry(bool& ok, std::string const& path, openmode mode)
            {
                ok = ok && FileSystem::isFileOpenable(path, mode);
                return NoAction;
            }
            void openFinalize(bool ok, std::string const& path, openmode mode, PreOpenState const&)
            {
                if (ok)
                {
                    file.open(path.c_str(), mode);
                }
            }
            void close()                                {file.close();}
            void read(T& obj)                           {file.read(reinterpret_cast<char*>(&obj), sizeof(T));}
            void write(T const& obj) const              {file.write(reinterpret_cast<char const*>(&obj), sizeof(T));}
            void setstate(iostate extraState)           {file.setstate(extraState);}
            void clear(iostate newState)                {file.clear(newState);}
            iostate rdstate() const                     {return file.rdstate();}
            void seekg(streampos pos)                   {file.seekg(pos * sizeof(T));}
            void seekp(streampos pos)                   {file.seekp(pos * sizeof(T));}
        };

        // This specialization for std::string keeps track of the data and an index into the data.
        template<typename F>
        struct FileAccessObjectType<F, std::string, ThorsAnvil::Serialize::TraitType::Value>
        {
            F&  file;
            FileAccessObjectType(F& file)
                : file(file)
            {}

            PreOpenState openTry(bool& ok, std::string const& path, openmode mode)
            {
                ok = ok && FileSystem::isFileOpenable(path, mode);
                return NoAction;
            }
            void openFinalize(bool ok, std::string const& path, openmode mode, PreOpenState const&)
            {
                if (ok)
                {
                    file.data.open(path, mode);
                    {
                        std::ofstream touch(path + ".index", std::ios::app);
                    }
                    file.index.open(path + ".index", mode | std::ios_base::in | std::ios_base::out);
                }
            }
            void close()
            {
                file.data.close();
                file.index.close();
            }
            void read(std::string& obj)
            {
                std::getline(file.data, obj);
                std::transform(std::begin(obj), std::end(obj), std::begin(obj), [](char x){return x == '\0' ? '\n' : x;});
            }
            void write(std::string const& obj)
            {
                std::string::const_iterator start   = std::begin(obj);
                std::size_t                 used    = 0;

                for (std::size_t size = obj.find('\n'); size != std::string::npos; size = obj.find('\n', size + 1))
                {
                    size = (size == std::string::npos) ? (std::end(obj) - start) : size;
                    std::size_t len = (size - used);
                    file.data << std::string_view(&*start, size - used) << '\0';

                    start += (len + 1);
                    used  += (len + 1);
                }
                file.data << std::string_view(&*start) << "\n";
                streampos index = file.data.tellp();
                file.index.write(reinterpret_cast<char*>(&index), sizeof(streampos));
            }
            void setstate(iostate extraState)           {file.data.setstate(extraState);file.index.setstate(extraState);}
            void clear(iostate newState)                {file.data.clear(newState);file.index.clear(newState);}
            iostate rdstate() const                     {return file.data.rdstate() | file.index.rdstate();}
            void seekg(streampos pos)
            {
                if (pos == 0)
                {
                    file.index.seekg(0);
                    file.data.seekg(0);
                }
                else
                {
                    file.index.seekg(pos * sizeof(std::size_t) - sizeof(std::size_t));
                    streampos index;
                    file.index.read(reinterpret_cast<char*>(&index), sizeof(streampos));
                    file.data.seekg(index);
                }
            }
            void seekp(streampos pos)
            {
                if (pos == 0)
                {
                    file.index.seekp(0);
                    file.data.seekp(0);
                }
                else
                {
                    file.index.seekg(pos * sizeof(std::size_t) - sizeof(std::size_t));
                    streampos index;
                    file.index.read(reinterpret_cast<char*>(&index), sizeof(streampos));
                    file.index.seekp(pos * sizeof(std::size_t) - sizeof(std::size_t));
                    file.data.seekp(index);
                }
            }
        };

        template<typename S, typename T, std::size_t I>
        struct FileAccessObjectSelector
        {
            using Traits            = ThorsAnvil::Serialize::Traits<T>;
            using Members           = typename Traits::Members;
            using FileTuple         = TupleFileType<S, T>;

            using FileIndex         = std::tuple_element_t<I, FileTuple>;
            using PointerTypeIndex  = std::tuple_element_t<I, Members>;
            using DstIndex          = GetPointerMemberType<PointerTypeIndex>;

            using FileAccessObject  = FileAccessObjectType<FileIndex, DstIndex>;
        };
        // Given an S, T and an index I.
        template<typename S, typename T, std::size_t I>
        using FileAccessObject      = typename FileAccessObjectSelector<S, T, I>::FileAccessObject;

    }

    // ==== FileMembers ====

    template<typename S, typename T>
    FileMembers<S, T>::FileMembers()
        : state(failbit)
    {}

    // ---- Open ----

    // void FileBase<S, T>::open for a description of the open processes.
    template<typename S, typename T>
    Impl::OpenState<T> FileMembers<S, T>::doOpenTry(bool& ok, std::string const& path, openmode mode)
    {
        Impl::OpenState<T>  result;
        if (!ok)
        {
            result.base = Impl::NoAction;
            return result;
        }

        FileSystem::DirResult createDir = FileSystem::makeDirectory(path, 0'777);

        if (createDir == FileSystem::DirFailedToCreate)
        {
            ok          = false;
            result.base = Impl::NoAction;
            return result;
        }

        result.base = createDir == FileSystem::DirAlreadyExists ? Impl::DirExists : Impl::NoDir;
        result.members = doOpenTryMembers(ok, path, mode, Index{});
        return result;
    }

    template<typename S, typename T>
    template<std::size_t... I>
    Impl::OpenMemberTuple<T> FileMembers<S, T>::doOpenTryMembers(bool& ok, std::string const& path, openmode mode, std::index_sequence<I...>)
    {
        Impl::OpenMemberTuple<T> result = std::make_tuple([this, &ok, &path, mode]()
        {
            Impl::FileAccessObject<S, T, I>  fileAccess(std::get<I>(fileTuple));

            auto result = fileAccess.openTry(ok, getMemberFilePath<I>(path), mode);
            setstateLocalOnly(fileAccess.rdstate());
            return result;
        }()...);

        return result;
    }

    template<typename S, typename T>
    void FileMembers<S, T>::doOpenFin(bool ok, std::string const& path, openmode mode, Impl::OpenState<T> const& state)
    {
        if (state.base == Impl::NoAction)
        {
            return;
        }

        doOpenFinMembers(ok, path,  mode, state.members, Index{});

        if (!ok && state.base == Impl::NoDir)
        {
            FileSystem::removeFileOrDirectory(path);
            // We should probably log something if we fail to remove the directory.
            // I don't think an exception is appropriate at this point we have already failed
            // to create the file if this is the issue then we don't want to create in appropriate errors and a few
            // extra directories in the file system is not going to hurt
        }
    }

    template<typename S, typename T>
    template<std::size_t... I>
    void FileMembers<S, T>::doOpenFinMembers(bool ok, std::string const& path, openmode mode, Impl::OpenMemberTuple<T> const& state, std::index_sequence<I...>)
    {
        ([this, ok, &path, mode, &state]()
        {
            Impl::FileAccessObject<S, T, I>  fileAccess(std::get<I>(fileTuple));

            fileAccess.openFinalize(ok, getMemberFilePath<I>(path), mode, std::get<I>(state));
            setstateLocalOnly(fileAccess.rdstate());
        }(), ...);
    }

    // ---- Close ----

    template<typename S, typename T>
    template<std::size_t... I>
    void FileMembers<S, T>::doCloseMembers(std::index_sequence<I...>)
    {
        // Using fold expression and lambda.
        ([this]()
        {
            Impl::FileAccessObject<S, T, I>  fileAccess(std::get<I>(fileTuple));
            fileAccess.close();
            setstateLocalOnly(fileAccess.rdstate());
        }(), ...);
    }

    // ---- Read/Write ----

    template<typename S, typename T>
    template<std::size_t... I>
    void FileMembers<S, T>::readMembers(T& data, std::index_sequence<I...>)
    {
        // Using fold expression and lambda.
        ([this, &data]()
        {
            Impl::FileAccessObject<S, T, I>  fileAccess(std::get<I>(fileTuple));
            auto& members       = Traits::getMembers();
            auto& pointer       = std::get<I>(members).second;

            fileAccess.read(data.*pointer);
            setstateLocalOnly(fileAccess.rdstate());
        }(), ...);
    }

    template<typename S, typename T>
    template<std::size_t... I>
    void FileMembers<S, T>::writeMembers(T const& data, std::index_sequence<I...>)
    {
        // Using fold expression and lambda.
        ([this, &data]()
        {
            Impl::FileAccessObject<S, T, I>  fileAccess(std::get<I>(fileTuple));
            auto& members       = Traits::getMembers();
            auto& pointer       = std::get<I>(members).second;

            fileAccess.write(data.*pointer);
            setstateLocalOnly(fileAccess.rdstate());
        }(), ...);
    }

    // ---- Clear State Bits ----

    template<typename S, typename T>
    template<std::size_t... I>
    void FileMembers<S, T>::clearMembers(iostate newState, std::index_sequence<I...>)
    {
        // Using fold expression and lambda.
        ([this, newState]()
        {
            Impl::FileAccessObject<S, T, I>  fileAccess(std::get<I>(fileTuple));
            fileAccess.clear(newState);
        }(), ...);
    }

    // ---- Set State Bits ----

    template<typename S, typename T>
    template<std::size_t... I>
    void FileMembers<S, T>::setstateMembers(iostate extraState, std::index_sequence<I...>)
    {
        ([this, extraState]()
        {
            Impl::FileAccessObject<S, T, I>  fileAccess(std::get<I>(fileTuple));
            fileAccess.setstate(extraState);
        }(), ...);
    }

    // ---- seek ----
    template<typename S, typename T>
    template<std::size_t... I>
    void FileMembers<S, T>::seekgMembers(streampos pos, std::index_sequence<I...>)
    {
        ([this, pos]()
        {
            Impl::FileAccessObject<S, T, I>  fileAccess(std::get<I>(fileTuple));
            fileAccess.seekg(pos);
        }(), ...);
    }

    template<typename S, typename T>
    template<std::size_t... I>
    void FileMembers<S, T>::seekpMembers(streampos pos, std::index_sequence<I...>)
    {
        ([this, pos]()
        {
            Impl::FileAccessObject<S, T, I>  fileAccess(std::get<I>(fileTuple));
            fileAccess.seekp(pos);
        }(), ...);
    }

    // ---- Get Index Element Path Name from base ----

    template<typename S, typename T>
    template<std::size_t I>
    std::string FileMembers<S, T>::getMemberFilePath(std::string const& path)
    {
        std::string filePath = path;
        filePath += "/";
        filePath += std::get<I>(Traits::getMembers()).first;

        return filePath;
    }

    // ===== FileBase =========

    template<typename S, typename T>
    FileBase<S, T>::FileBase(std::string fileName, openmode mode)
        : fileOpened(false)
        , baseFileName(std::move(fileName))
        , getPos(0)
        , putPos(0)
    {
        open(mode);
    }

    // ---- Open ----
    // Open is complex:
    //  Only the first function here is public.
    //  The second is the main entry point called by the public open and the constructor.
    //  It performs the open in two two stages:
    //      Stage 1:    doOpenTry:
    //                      Create Directory if they don't exist.
    //                      Check if required files can be opened in the required mode in a directory.
    //      Stage 2:    doOpenFin:
    //                      If all files can be created then create all files.
    //                      If we can not create all the files then remove the directories we created in stage 1.

    template<typename S, typename T>
    void FileBase<S, T>::open(std::string fileName, openmode mode)
    {
        if (fileOpened)
        {
            FileMembers<S, T>::setstate(failbit);
            return;
        }
        baseFileName = std::move(fileName);
        open(mode);
    }

    template<typename S, typename T>
    void FileBase<S, T>::open(openmode mode)
    {
        if (baseFileName == "")
        {
            return;
        }
        fileOpened = true;
        FileMembers<S, T>::clear();

        Impl::OpenState<T> state = FileMembers<S, T>::doOpenTry(fileOpened, baseFileName, mode);
        FileMembers<S, T>::doOpenFin(fileOpened, baseFileName, mode, state);

        if (!fileOpened)
        {
            FileMembers<S, T>::setstate(failbit);
        }
        else
        {
            index.open(baseFileName + "/$index", mode);
            getPos  = index.tellg();
            putPos  = index.tellp();
        }
    }

    // ---- close ----

    template<typename S, typename T>
    void FileBase<S, T>::close()
    {
        if (!fileOpened)
        {
            return;
        }
        FileMembers<S, T>::close();
        FileBase<S, T>::setstateLocalOnly(failbit);
        fileOpened = false;
    }

    // ---- read/write ----

    template<typename S, typename T>
    void FileBase<S, T>::read(T& data)
    {
        if (!FileMembers<S, T>::good())
        {
            return;
        }
        FileMembers<S, T>::read(data);
        char mark;
        index.read(&mark, 1);
        ++getPos;
    }

    template<typename S, typename T>
    void FileBase<S, T>::write(T const& data)
    {
        if (!FileMembers<S, T>::good())
        {
            return;
        }
        FileMembers<S, T>::write(data);
        char mark = 'A';
        index.write(&mark, 1);
        ++putPos;
    }

    // ---- tell/seek ----
    template<typename S, typename T>
    void FileBase<S, T>::seekg(streampos pos)
    {
        index.seekg(pos);
        FileMembers<S, T>::seekg(pos);
        getPos  = pos;
    }

    template<typename S, typename T>
    void FileBase<S, T>::seekp(streampos pos)
    {
        index.seekp(pos);
        FileMembers<S, T>::seekp(pos);
        putPos = pos;
    }
    template<typename S, typename T>
    void FileBase<S, T>::seekg(streamoff off, seekdir dir)
    {
        index.seekg(off, dir);
        streampos pos = index.tellg();
        seekg(pos);
    }
    template<typename S, typename T>
    void FileBase<S, T>::seekp(streamoff off, seekdir dir)
    {
        index.seekp(off, dir);
        streampos pos = index.tellp();
        seekp(pos);
    }
}

#endif
\$\endgroup\$
4
  • \$\begingroup\$ (Off-topic question) On your GitHub repository what is "build @ d6da7a9" called? \$\endgroup\$
    – Peilonrayz
    Apr 19 '20 at 19:53
  • 1
    \$\begingroup\$ @Peilonrayz It is a submodule. If you run git submodule update --init --recursive it will load all the submodules. Or you can run the ./configure command that will do that and perform a bunch of checks to make sure you can build it. \$\endgroup\$ Apr 20 '20 at 6:24
  • 1
    \$\begingroup\$ @Peilonrayz Specifically build loads ThorMaker. This is the build tools I use for all my C++ projects. It is generalized to build anything I want with only minor configuration for any given project. See the Makefiles in this project for examples. See: src/Makefile for building sub modules. See: src/ThorsStorage/Makefile for a normal build file. \$\endgroup\$ Apr 20 '20 at 6:31
  • 1
    \$\begingroup\$ Wow, that's really cool. Thank you for the in-depth answer :D \$\endgroup\$
    – Peilonrayz
    Apr 20 '20 at 12:53
1
\$\begingroup\$
template<typename T>
class IFile: public FileBase<std::ifstream, T> { ... };

template<typename T>
class OFile: public FileBase<std::ofstream, T> { ... };

template<typename T>
class File: public FileBase<std::fstream, T> { ... };

Read and write functions are both accessible in the FileBase interface for all three of these anyway, right? So we could delete these classes, use std::fstream all the time, and just pass in the appropriate open mode. (Perhaps adding an enum class WriteMode { Read, Write, ReadWrite }; to pass into the FileBase constructor).


FileBase(std::string fileName = "", openmode mode = 0);

Not sure having default arguments here makes sense.


template<typename S, typename T>
class FileBase: public FileMembers<S, T>

Yiiiiikes. Are all the FileMembers functions publicly accessible to users of FileBase too? So we have three different open functions, a close function that hides the close function in the FileMembers class, etc. Seems like a recipe for confusion.

We should be using composition, not inheritance:

struct FileBase
{
public:
    FileBase(...);

    void open(...);
    void close();
    ...

private:
    IndexFile index;
    FileMembers members;
};

Maybe we have to forward some function calls from the FileBase interface, but it's much, much simpler.


    bool            fileOpened;

Seems unnecessary. We can just check if the files are opened, and not have to worry about updating a variable if something changes.

e.g.

template<typename S, typename T>
void FileBase<S, T>::open(openmode mode)
{
    if (baseFileName == "")
    {
        return;
    }
    fileOpened = true;
    FileMembers<S, T>::clear();

    Impl::OpenState<T> state = FileMembers<S, T>::doOpenTry(fileOpened, baseFileName, mode);
    FileMembers<S, T>::doOpenFin(fileOpened, baseFileName, mode, state);

    if (!fileOpened)
    {
        FileMembers<S, T>::setstate(failbit);
    }
    else
    {
        index.open(baseFileName + "/$index", mode);
        getPos  = index.tellg();
        putPos  = index.tellp();
    }
}

What if the index file doesn't open? We already set fileOpened to true...


    streampos       getPos;
    streampos       putPos;

Same again. Are these actually used anywhere? Can't we get them from the index file stream (tellg, tellp) whenever we need to?


Opening seems over-complicated.

I don't think there's much point in a "pre-check". It seems like optimizing the rare fail-case, and making our common best-case slower.

Our check can be wrong / obsolete immediately anyway. So just open it! If it fails, it fails.


It might be worth specifying a separate enum class MissingMode { Create, Fail } we can use when opening the database, separate from the WriteMode. So if we expect to open an existing database at a given point, and it's not there, we can avoid creating a new empty database if we want to.


There's a lot of state stuff based on the C++ std::fstream states:

        void setstate(iostate extraState)       {setstateLocalOnly(extraState); setstateMembers(extraState, Index{});}
        void clear(iostate newState = goodbit)  {clearLocalOnly(newState);      clearMembers(newState, Index{});}

        // https://en.cppreference.com/w/cpp/io/ios_base/iostate
        bool good()                             const   {return !(state & (eofbit | badbit | failbit));}
        bool eof()                              const   {return state & eofbit;}
        bool bad()                              const   {return state & badbit;}
        bool fail()                             const   {return state & (failbit | badbit);}
        operator bool()                         const   {return !fail();}
        bool operator!()                        const   {return !static_cast<bool>(*this);}
        iostate rdstate()                       const   {return state;}

Managing that state member flag is probably quite complicated and error prone. And is it really better to keep track of it, instead of calculating it on demand?

I suspect that some of these errors should only happen when the database implementation is buggy, or an underlying file is corrupted, or something is Seriously Wrong. In that case, I don't think providing separate eof(), bad(), fail() methods makes sense. i.e. That eof error happened in a single file; we don't know which, and we don't know why, we only really care that our database is borked.

We have no access to the underlying streams. The user probably doesn't even care that there are underlying file streams, let alone what state one of the files (and which one?) is in.

I'd either return boolean values on specific actions (e.g. did our read work?) or throw specific error messages (e.g. `throw ReadError("Unexpected eof reading member {Foo} at index {I}");') when something goes wrong.

So maybe two separate functions: bool tryRead(foo); and void read(foo);.


We can avoid a some complexity by not providing an open function. :)

We can provide:

  • Default constructor (just a closed database).
  • Value constructor (to open a database).
  • Move constructor and move assignment operator.

This means we don't have to care about logic for re-opening an already-open database. We just use the move assignment operator.


Perhaps all the iteration over tuples could be abstracted into a separate function somehow?

\$\endgroup\$
3
  • \$\begingroup\$ I don't understand the comment: Our check can be wrong / obsolete immediately anyway \$\endgroup\$ Apr 28 '20 at 13:19
  • \$\begingroup\$ Thanks for the detailed and thoughtful review. I will incorporate most of these into the code. :-) \$\endgroup\$ Apr 28 '20 at 13:23
  • \$\begingroup\$ A file might exist and be accessible, but that doesn't mean we'll succeed in opening it even a short time later. Someone could unplug a USB stick, or lose their network connection or whatever. So I'm not sure there's much point in checking. \$\endgroup\$
    – user673679
    Apr 28 '20 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.