Custom Serialization of Game World

Question

I'm working with libGDX for a game, and over the last few days I have tried a few different types of serialization before settling on a custom serialization implementation. XML had huge file sizes, JSON was a bit better but unfortunately gave me problems on iOS, and Serializable would not work because if I extended that with my classes, they would not compile for the GWT version of the game. Even though the HTML/GWT version of the game will not be able to have saves, I still wanted the same code to work on all platforms without any changes.

@SimonAndreForsberg linked this article to me about using static factory methods in Java, and I agree that they are important here to handle the exceptions that can be thrown during object construction if something goes wrong.

The serialization itself is very simple. I simply manually encode all the necessary information from each object, and manually decode it when loading the game. The only complicated part of this is that I pass the DataOutputStream and DataInputStream objects into the static factory methods of the classes in order to do the coding and decoding.

The major drawback of this serialization approach is that any small change to the classes (such as adding a new instance variable for a new feature) will require a change to the serialization, which will then break any old saves. I don't like this, but it may well be a necessary tradeoff. This serialization is working on all platforms, and the file size is 1/30th of what it was with XML or JSON.

I will start at the top and show a couple of levels of serialization. Each class that needs to be serialized passes the stream to any objects that it possesses that need to be serialized.

For reference the World contains 50x50 (2500) Regions, which each contain 11x7 (77) Tiles. After serialization the world file is about 1.2 megabytes.

This is called when the save button is pressed:

public void saveGame() throws IOException {
    switch(Gdx.app.getType()) {
       case Android:
       case Desktop:
       case Applet:
       case HeadlessDesktop:
       case iOS:
           this.saveWorld();
           this.savePlayer();
           this.saveAIPlayers();
           break;
       case WebGL:
           break;
    }
}

Here is the saveWorld() method:

private void saveWorld() throws IOException {
    DataOutputStream out = new DataOutputStream(Gdx.files.local("world.map").write(false));
    this.world.customEncode(out);
    out.close();
}

Now on to the implementation in the World class:

public static World decodedWorld(DataInputStream in) throws IOException {

    HashMap<MapPoint, Region> loadedWorldMap = new HashMap<MapPoint, Region>();
    int numRegions = in.readInt();
    for (int i = 0; i < numRegions; i++) {
        int regionBiomeType = in.read();
        if (regionBiomeType == BiomeType.NETHER.ordinal()) {
            Region loadedNetherRegion = Region.decodedRegion(BiomeType.NETHER, in);
            loadedWorldMap.put(loadedNetherRegion.worldPosition(), loadedNetherRegion);
        } else {
            Region loadedRegion = Region.decodedRegion(BiomeType.values()[regionBiomeType], in);
            loadedWorldMap.put(loadedRegion.worldPosition(), loadedRegion);
        }
    }

    return new World(loadedWorldMap);
}

public void customEncode(DataOutputStream out) throws IOException {
    out.writeInt(this.worldMap.values().size());
    for (Region region : this.worldMap.values()) {
        region.customEncode(out);
    }
}

I should note that I realize that there is not custom logic for the NetherRegions, so that conditional is not really necessary. However there may very well be in the future, and I use this pattern in other places as well when dealing with deserializing subclasses.

Here are the methods in the Region class:

public static Region decodedRegion(BiomeType type, DataInputStream in) throws IOException {
    Region region = new Region(type, new MapPoint(in.read(), in.read()));

    int nameLength = in.read();
    byte [] nameBytes = new byte[nameLength];
    in.read(nameBytes);
    region.name = new String(nameBytes, "UTF-8");
    int ownerNameLength = in.read();
    if (ownerNameLength > 0) {
        byte [] ownerNameBytes = new byte[ownerNameLength];
        in.read(ownerNameBytes);
        region.setOwnerName(new String(ownerNameBytes, "UTF-8"));
    }

    int numTiles = in.read();
    for (int i = 0; i < numTiles; i++) {
        int tileType = in.read();
        if (tileType == TileType.CAPITAL.ordinal()) {
            TileCapital loadedCapital = TileCapital.decodedTile(in);
            region.setTileForPosition(loadedCapital.position(), loadedCapital);
            //System.out.println(loadedCapital.type.toString());
        } else {
            Tile loadedTile = Tile.decodedTile(TileType.values()[tileType], in);
            region.setTileForPosition(loadedTile.position(), loadedTile);
            //System.out.println(loadedTile.type.toString());
        }
    }

    return region;
}

public void customEncode(DataOutputStream out) throws IOException {
    out.write(this.biomeType.ordinal());
    out.write(this.position.x);
    out.write(this.position.y);

    BZSerializationTools.serializeString(out, this.name);
    BZSerializationTools.serializeString(out, this.ownerName);

    out.write(this.regionMap.values().size());
    for (Tile tile : this.regionMap.values()) {
        tile.customEncode(out);
    }
}

Here is the method that actually sets the loaded Tile:

public void setTileForPosition(MapPoint position, Tile tile) {
    this.regionMap.put(position, tile);
}

Here is the serializeString method:

public static void serializeString(DataOutputStream out, String string) throws IOException {
    if (string != null) {
        byte[] stringAsBytes = string.getBytes();
        out.write(stringAsBytes.length);
        out.write(stringAsBytes);
    } else {
        out.write(0);
    }
}

It should not be necessary to go deeper than that. I follow this same pattern for each of my classes that I need serialized. Let me know if you would like to see any other code.

Answer 1

Serializing ordinal values

if (regionBiomeType == BiomeType.NETHER.ordinal()) {

When relying the ordinal value of an enum, remember that you cannot change the order of these enums, that will break things. It has happened before.

I'd personally give each enum value it's own int id, which you then can use for serializing and deserializing.

enum BiomeType {
    NORMAL(42), BIOME(23);
    private final int id;
    private BiomeType(int id) {
        this.id = id;
    }

    public static BiomeType forId(int id) {
        // loop through all the biomes and find the one with the matching id
    }
}

Code duplication for reading strings

You have a method to write a String to a DataOutputStream, that is a good one. _{Whoever suggested that is a genius.}

Why do you not also have a method to read a string? This is duplicated code:

if (ownerNameLength > 0) {
    byte [] ownerNameBytes = new byte[ownerNameLength];
    in.read(ownerNameBytes);
    region.setOwnerName(new String(ownerNameBytes, "UTF-8"));
}

That could be:

region.name = BZSerializationTools.readString(in);
region.setOwnerName(BZSerializationTools.readString(in));

setTileForPosition

First of all, does this really need to be public? I would make that private if possible, or default-visibility (package-private) if not possible.

public void setTileForPosition(MapPoint position, Tile tile) {
    this.regionMap.put(position, tile);
}

Additionally, if it is changed to private, you won't need it as a method at all, as you can change this:

region.setTileForPosition(loadedTile.position(), loadedTile);

to:

region.regionMap.put(loadedTile.position(), loadedTile);

Data structure for 2D

You are currently storing a region using a Map<MapPoint, Tile>, and serializing it by using the size of the region map:

out.write(this.regionMap.values().size());
for (Tile tile : this.regionMap.values()) {
    tile.customEncode(out);
}

As a region is always 2D in size, you can instead store it using a 2D array, Tile[][], and serialize it using the following approach:

• write size (width, height) of region
• loop on each tile in specific order:
  • write tile, except its position

When reading you can do:

• read size (width, height)
• loop on each index in the Tile[][] array:
  • read tile, except its position

This will save you some more bytes, as there is no real reason to write the position of each tile. Instead, loop over them in a specific order, the one I find most common, and would recommend, is:

for (int y = 0; y < height; y++) {
    for (int x = 0; x < width; x++) {
        // read or write tile[x][y] (or [y][x] depending on how you want to store that)
    }
}