A binary 3D model format for a game

Question

I've recently started working on my own game engine and I have been thinking about creating my own model format. Since text usually takes a long time to load and process I decided to make a binary file. After a little work, I already had the basic concept:

The concept

• The header:
  The header of the file contains the magic number and the current major and minor version of the file. The magic number I chose is 706D6600. An example header would like this: 706D6600 0100 0000
• The body
  I thought for a long time how I could make the body. In the end, I decided on the following: Models consist of vertices, texture coordinates, normals, and faces. Each type has a unique identifier and always the same size. This fixed size makes it easier for the file reader to read the file. Here is a list of all ID's:
  • EOF - 0000FFF1
  • Vertex - 0000FFF2 3x Float(12 bytes)
  • Texture Coordinate 0000FFF3 2x Float(8 bytes)
  • Normal 0000FFF4 3x Float(12 bytes)
  • Face 0000FFF5 9x Integer(36 bytes) (Sorry for not choosing FACE😀)

---

A cube would look like this:

The Reader

To test the file out, I wrote a simple reader which would print the file like a Wavefront Obj Model.

While writing, I didn't pay attention to performance: it was just to make the function of the file clearer. Here is the code:

pmf_types.h

#define PMF_TYPES_H

#include <stdint.h>

const unsigned char pmf_magicnumber[4] = {0x70, 0x6D, 0x66, 0x00};

const uint32_t pmf_min_major = 1;
const uint32_t pmf_min_minor = 0;

const uint32_t pmf_max_major = 1;
const uint32_t pmf_max_minor = 0;

const unsigned char pmf_eof_id[4] = {0x00, 0x00, 0xFF, 0xF1};
const unsigned char pmf_vertex_id[4] = {0x00, 0x00, 0xFF, 0xF2};
const unsigned char pmf_texture_coordinate_id[4] = {0x00, 0x00, 0xFF, 0xF3};
const unsigned char pmf_normal_id[4] = {0x00, 0x00, 0xFF, 0xF4};
const unsigned char pmf_face_id[4] = {0x00, 0x00, 0xFF, 0xF5};


#endif

main.c

#include "pmf_types.h"

int main() {
    FILE *file = fopen("example.pmf", "rb");
    fseek(file, 0, SEEK_END);
    size_t fileSize = ftell(file);
    fseek(file, 0, SEEK_SET);

    unsigned char magicnumber[4];

    fread(magicnumber, sizeof(magicnumber), 1, file);

    if(memcmp(magicnumber, pmf_magicnumber, sizeof(magicnumber)) != 0) {
        printf("Invalid Magic Number. (Perhaps wrong format)\n");
        return -1;
    }

    unsigned char version[4];

    fseek(file, 4, SEEK_SET);
    fread(version, sizeof(version), 1, file);

    uint32_t major, minor;

    memcpy(&major, (unsigned char[2]){version[0], version[1]}, 2);
    memcpy(&minor, (unsigned char[2]){version[2], version[3]}, 2);

    if(major < pmf_min_major || (major == pmf_min_major && minor < pmf_max_minor))  {
        printf("The file is outdated.\nSupported maximum version: %d.%d\n", pmf_max_major, pmf_max_minor);
        return -1;
    }

    if(major > pmf_max_major || (major == pmf_max_major && minor > pmf_max_minor)) {
        printf("The reader is outdated.\nSupported maximum version: %d.%d\n", pmf_max_major, pmf_max_minor);
        return -1;
    }

    int offset = 8;

    while(1) {
        if(offset>fileSize) {
            printf("Unexpected end of file.\n");
            break;
        }

        unsigned char identifier[4];

        fseek(file, offset, SEEK_SET);
        fread(&identifier, sizeof(identifier), 1, file);

        offset+=4;

        if(memcmp(identifier, pmf_vertex_id, sizeof(identifier)) == 0) {
            if(offset+12>fileSize) {
                printf("Marked as vertex but not enough values supplied.\n");
                break;
            }

            float x, y, z;

            fseek(file, offset, SEEK_SET);
            fread((void*)(&x), sizeof(x), 1, file);

            offset += 4;

            fseek(file, offset, SEEK_SET);
            fread((void*)(&y), sizeof(y), 1, file);

            offset += 4;

            fseek(file, offset, SEEK_SET);
            fread((void*)(&z), sizeof(z), 1, file);

            offset += 4;

            printf("Vertex %f %f %f\n", x, y, z);
        } else if(memcmp(identifier, pmf_texture_coordinate_id, sizeof(identifier)) == 0) {
            if(offset+8>fileSize) {
                printf("Marked as texture coordinate but not enough values supplied.\n");
                break;
            }

            float x, y;

            fseek(file, offset, SEEK_SET);
            fread((void*)(&x), sizeof(x), 1, file);

            offset+=4;

            fseek(file, offset, SEEK_SET);
            fread((void*)(&y), 4, 1, file);

            offset+=4;

            printf("Texture coordinate %f %f\n", x, y);
        } else if(memcmp(identifier, pmf_normal_id, sizeof(identifier)) == 0) {
            if(offset+12>fileSize) {
                printf("Marked as normal but not enough values supplied.\n");
                break;
            }

            float x=0, y=0, z=0;

            fseek(file, offset, SEEK_SET);
            fread(&x, 4, 1, file);

            offset+=4;

            fseek(file, offset, SEEK_SET);
            fread(&y, 4, 1, file);

            offset+=4;

            fseek(file, offset, SEEK_SET);
            fread(&z, 4, 1, file);

            offset+=4;

            printf("Normal %f %f %f\n", x, y, z);
        } else if(memcmp(identifier, pmf_face_id, sizeof(identifier)) == 0) {
            if(offset+36>fileSize) {
                printf("Marked as face but not enough values supplied.\n");
                break;
            }

            uint32_t vx, vy, vz, vtx, vty, vtz, vnx, vny, vnz;

            fseek(file, offset, SEEK_SET);
            fread(&vx, 4, 1, file);

            offset+=4;
            fseek(file, offset, SEEK_SET);
            fread(&vy, 4, 1, file);

            offset+=4;
            fseek(file, offset, SEEK_SET);
            fread(&vz, 4, 1, file);

            offset+=4;
            fseek(file, offset, SEEK_SET);
            fread(&vtx, 4, 1, file);

            offset+=4;
            fseek(file, offset, SEEK_SET);
            fread(&vty, 4, 1, file);

            offset+=4;
            fseek(file, offset, SEEK_SET);
            fread(&vtz, 4, 1, file);

            offset+=4;
            fseek(file, offset, SEEK_SET);
            fread(&vnx, 4, 1, file);

            offset+=4;
            fseek(file, offset, SEEK_SET);
            fread(&vny, 4, 1, file);

            offset+=4;
            fseek(file, offset, SEEK_SET);
            fread(&vnz, 4, 1, file);

            offset+=4;

            printf("Face %d/%d/%d %d/%d/%d %d/%d/%d\n", vx, vtx, vnx, vy, vty, vny, vz, vtz, vnz);
        } else if(memcmp(identifier, pmf_eof_id, sizeof(identifier)) == 0) {
            //End of File
            break;
        } else {
            printf("Unknown identifier!\n");
            break;
        }
    }
}

What I want

I'm not sure if it's a good idea to use such identifiers or if floats and integers can be stored in a better way.

Anyway, I want to get tips and criticism (Especially about my format).

Miscellaneous

Here is a link to the cube file: https://hastebin.com/tuqayebila.apache

The file extension of my file is pmf which stands for Plutonium File Format. (Plutonium is the name of my engine)

Answer 1

Anyway, I want to get tips and criticism (Especially about my format).

I think the fixed-size 4-byte things are great, much better than text. They're easy to use both for humans and computers, the identifiers are easy to identify in a hex view, and nothing misaligns the data stream so it stays in nice columns in hex view as well. However, I don't think it's good that the identifiers pepper the format so much. I'm not that worried about the size, though that is an aspect as well. One result of having so many identifers everywhere was that code like memcmp(identifier, pmf_vertex_id, sizeof(identifier)) == 0 is executed several times per element, which conflicts with a general rule of thumb I would use for file formats:

Ideally, design a file format to be read and written in chunks of significant size.

IO (even buffered IO) tends to like significant chunks. It doesn't have to be megabytes at once, but if you can fire off a read or write of several KB at once, that's good. Working in bigger chunks becomes even more important if you add compression. Generally that means arrays of things, with their length stored somewhere before them, such as in the header (enabling reading all lengths at once which is nice) or length-prefixed (which is suitable for more dynamic formats). Array-with-length also has an advantage compared to using an end-of-file marker, related to an other rule of thumb:

Ideally, design a file format to be readable in one pass without reallocation.

The issue didn't show itself in the example reader here, because the data is printed instead of kept around. More commonly though, it would be loaded into arrays. With the current file format, if you wanted to load a file into arrays, their length would not be known until the the end-of-file marker is found. We could do two passes, with a first pass to find the size and the second pass to actually load the data, but that's a waste. Seeking to the end is an option, but the file format does not actually promise that there is nothing after the first occurrence of the end-of-file marker (and if it did promise that, what would be the point of the marker?). Anyway, having the lengths up-front also solves that issue.

Arrays with known length have an other advantage, which I don't know whether it's important to you, but it might become interesting someday: they enable a reader to skip data they're not interested in.

---

By the way, I don't think it's bad that you're making custom formats like this. Having custom converters as part of your asset pipeline is not strange at all, and compatibility of the game files with common editors is not that useful: editing them directly is often banned anyway (or it should be), it bypasses the asset pipeline risking incorrect or improper results (out-of-sync lightmaps, unoptimized meshes, meta-data with dangling references etc). The formats of the development versions of assets are an other matter entirely, of course there you would want them to be editable by their respective editors, but even then it may be useful to write a small plugin.

Answer 2

(This didn't fit in the comment field, not really an answer, more like advice)

I get the allure of rolling your own format and writing the code, you wanna do it right, you wanna squeeze every last ounce of performance out of the CPU, you wanna create something awesome, maybe even learn something along the way, but here's some advice from a professional software engineer:

Use pre-existing, well known formats and libraries for these things.

The reason being simply that if you start rolling your own model formats, vector classes, UI Frameworks etc you'll spend all your time and energy writing those instead of the thing you actually wanted to create. You'll feel like you're not getting where you wanted to go and you'll lose motivation and then abandon the project and start another one. In addition, whatever you create will likely not be as performant and well thought out as the industry standard libraries etc because they have many times more hours of development and research in them then you on your lonesome can ever commit to on a hobby project. Not to mention that if you create your own formats, you're also committing to writing your own tools for working with these formats which makes it more tedious to get assets into your game engine, adding additional friction to getting a working product, increasing the chance that the project is going to fail.

I have the best intentions here, I've been down the path you're setting your foot on here, and I can tell you: you're much more likely to compete the project if you focus on making the thing you want instead of getting bogged down with creating all the things around it.

So my review, I guess, is: this code is not what you actually want, you should replace it with a common format, loaded by some library.

Sorry for sounding bitter, it's not my intent. I really wish that op success in their project which is why I'm giving this advice.

Answer 3

fread advances the file pointer. You may safely discard all those

        offset+=4;
        fseek(file, offset, SEEK_SET);

When you really need an offset, call ftell.

---

Always test what fread returns. BTW, it would eliminate a need in maintaining offset and comparing it with fileSize.

---

        float x, y, z;

        fread((void*)(&x), sizeof(x), 1, file);
        fread((void*)(&y), sizeof(y), 1, file);
        fread((void*)(&z), sizeof(z), 1, file);

is a long way to say

        float vertex[3];
        fread (vertex, sizeof(vertex[0]), 3, file);
     

---

Beware the endianness.

Answer 4

I'm not sure if it's a good idea to use such identifiers or if floats and integers can be stored in a better way.
Anyway, I want to get tips and criticism.

Declare, not define

pmf_types.h declares and defines various objects. It should only declare them.

// const uint32_t pmf_min_major = 1;
extern const uint32_t pmf_min_major;

and define them in one .c file

e.g. pmf_types.c

#include "pmf_types.h"
const uint32_t pmf_min_major = 1;

Use include guards

pmf_types.h missing include guards.

Answer 5

In this answer I'm skipping over most of the implementation and only talking about the file format itself.

How does your game consume this file? Your format should be designed for efficient reading.

I bet your game doesn't use the data in this format. I bet your game engine parses this format and creates a single array with interleaved vertices, normals and texture coordinates - because that's the format OpenGL wants - and then passes it to OpenGL. In this regard, your format is no better than a text-based OBJ. In fact your format looks exactly like a binary-ified version of OBJ.

If your model data is just getting passed to OpenGL, your format should be convertible to OpenGL with minimal work. In this case, that means: you read the size, you allocate the buffer, you read that much data from the file and pass it to OpenGL. No conversion. The script that creates your model files should already write it in this format. If you use index buffers, then you'll have two buffers - do the same thing twice.

Why? Because you shouldn't do work if you don't need to. Programs that achieve the same result with less work are better. Pointless extra work is pointless.

If you have some bits of information that don't go in the buffer - like the type of shape (GL_QUADS/GL_TRIANGLES/GL_TRIANGLE_STRIP) - then of course you don't read those into the buffer. Those can be part of the header, next to the buffer size.

There's no reason to make all fields are the same size if you are parsing them anyway. There is a reason to align the fields for efficiency if you are processing the data on the CPU (e.g. 4-byte fields should start at multiples of 4 bytes). You're right about fixed-size fields, but it doesn't mean all fields have to be the same size as other fields, just that all fields have to be the same size as themselves. E.g. a float is always 4 bytes.

The version number is useful, but there's no reason to have a major and minor version. It just adds confusion because every time you update the format you have to decide whether it's a major or minor update. Just use a single number. The advantage of having a version number is that whenever you change the format, you don't have to re-convert all your models. The game knows to load the old models the old way and the new models the new way. Of course, when you release the game, then you can re-convert all the models and delete the code for loading old versions.

About the code, though

• Why do you use this pattern?

    fseek(file, offset, SEEK_SET);
    fread(&vx, 4, 1, file);
    offset+=4;
  

  The file already knows which offset it's at. You can just do:

    fread(&vx, 4, 1, file);
  

  and it reads 4 bytes from the current offset, and adds 4 bytes to the current offset. There's no need to keep track of the offset yourself and there's no need to keep calling fseek.

• You can also read an entire struct at once:

  struct face {
      int vx, vy, vz;
      int vtx, vty, vtz;
      int vnx, vny, vnz;
  } f;
  
  fread(&f, sizeof f, 1, file);
  

  For more complicated structures, beware of compilers inserting padding - but that's good, the compiler does it so the CPU can access the data more efficiently, you just have to make sure it's consistent and account for it in the file format.

• Why do you call the vertices in a face X, Y and Z? I think this is confusing, because the coordinates in a vertex are also called X, Y and Z. I would name them v1, v2, v3 or a, b, c.

Answer 6

Code

I would remove the explicit if(offset+val>fileSize) { checks, as well as even getting fileSize in the first place. Instead, replace it with actual error handling, which you completely skip. This properly handles errors such as a disk issue while reading, and allows for flexibility in cases such as streaming a file over the network when you don't know its size before hand.

File format

It looks fine to me, but I would consider if "This fixed size makes it easier for the file reader to read the file" is really true compared to 1-byte identifiers. I would also remove the explicit EOF marker, and just look for the EOF errno instead.

For final use, I would also compress it. LZ4 is extremely fast at decompression, and can generally decompress 1 byte per clock cycle: e.g. a 4GHz CPU could decompress 4GB of data per second. Unless you're on a PCI-e gen 4 NVMe drive, the disk will be slower than that, but if you are, you can easily just use more threads for loading other assets in parallel on different cores. As an example, Nintendo Switch games are compressed with LZ4.

Answer 7

        float x;
        fread((void*)(&x), sizeof(x), 1, file);

All pointers are convertible to void*, so there's no need for the explicit cast:

        fread(&x, sizeof x, 1, file);

However, there's a bigger issue lurking here. In-memory representation of float can vary between platforms. Even if we limit ourselves to platforms using single-width IEEE-754 format for their float, then interpretation is often affected by byte order issues This seems a shame, given the care to use order-independent constants in the header; it means that your saved files can't safely be transported between hosts without some translation.

That's a good argument to use an existing serialisation library, and/or to support a text-based format for interchange (and debugging, incidentally).