FlexBuffers for Unity3D

A viable alternative for storing large configurations

If you are using JSON, CSV, XML, or even SQLite in your game to store data, you definitely should take a few minutes to read this blog post. If you don’t, you might still learn something useful 😀.

WTF is FlexBuffers?

FlexBuffers is a JSON comparable binary format, designed as a schema-less alternative in the FlatBuffers project, which was initially built at Google.

I got you at Google right?

If you are interested in the roots, please follow the link.

What do you mean by JSON comparable?

By this, I mean that everything you can represent with JSON you can also represent with FlexBuffers and vice versa. This is why FlexBuffers Unity provides an easy way to convert the JSON file to FlexBuffer file. And export a FlexBuffer file as a JSON file. But more on this topic a bit later.

Why should I use FlexBuffers instead of JSON?

The JSON file is a text file. In order to access data from a JSON file, we need to convert the text into a tree of C# objects. This means that we have to read the content of the file into memory, parse the text, and create the C# object tree. There are different JSON parser out there, which can make some of the steps more or less performant, but there is no way a JSON parser can avoid one of the steps.

When you are reading values out of a FlexBuffers file, there are only two steps:

1. Load content of the file into memory
2. Access the value in the byte array

With FlexBuffers we do not reflect the structure of your data as a C# object tree. We can directly read data out of a byte array using an expression as following:

var root = FlxValue.FromBytes(bytes);
Assert.AreEqual("Berlin", root["address"]["city"].AsString);

With this expression, we say that root has a property address and this property has a property of its own called city. We know that city is of type string and we ask to convert this value to an object of C# string type.

Traversal of the bytes based on the access semantics [] is quite fast, as we can see by examining the following profiler screenshot:

What you see here is me pressing a UI button which loads a 161KB FlexBuffer file into memory (see TextAsset.get_bytes() (0.38ms)) which contains a vector of 1000 entries and then we access 50 entries out of this vector:

var bytes = Resources.Load<TextAsset>("flx_data").bytes;
var root = FlxValue.FromBytes(bytes);
var names = new List<string>(50);
for (var i = 0; i < 50; i++)
{
    names.Add(root[i]["name"].AsString);
}
Debug.Log(names);

The 50 icicles in the middle show the time needed to access the name as string from the bytes array. Here is a screenshot where I zoom in on the icicles:

This proves that accessing a value by itself is a very light weight operation. (Which probably could be optimised even further).

Let's compare a similar process using Unities own JSONUtility:

The JSON file which contains the same data that occupies 246KB is loaded in 1.28ms. Which is due to the fact that JSON representation of the same data is 80KB (35%) bigger, but also that in the case of FlexBuffers, we need just a raw byte array, wherein in the case of JSON we need a UTF-16 encoded C# string object.

Then we also see that most of the time is spent in JsonUtility.FromJsonInternal() as we need to parse and convert the string into C# objects.

var jsonText = Resources.Load<TextAsset>("json_data2").text;
var cities = JsonUtility.FromJson<CityList>(jsonText);
var names = new List<string>(50);
for (var i = 0; i < 50; i++)
{
    names.Add(cities.list[i].name);
}
Debug.Log(names);

The actual accessing the values and adding them to a list of strings is completely negligible:

So the main performance bottleneck with JSON is the parsing step. In case you wonder if other JSON parsers are better than Unities JsonUtility. See for yourself.

Here is a test with using Utf8Json library, which proclaims itself as:

Definitely Fastest and Zero Allocation JSON Serializer for C#(NET, .NET Core, Unity, Xamarin)

The JsonSerializer.Deserialize() takes whooping 47.87ms to run

And the test code is not that different:

var jsonText = Resources.Load<TextAsset>("json_data2").text;
var cities = Utf8Json.JsonSerializer.Deserialize<CityList>(jsonText);
var names = new List<string>(50);
for (var i = 0; i < 50; i++)
{
    names.Add(cities.list[i].name);
}
Debug.Log(names);

If you think this is bad, have a look at this screenshot for Json.Net:

With Json.Net we manage to parse the JSON file in 447.67ms. So about 10x slower than Utf8Json and 100x slower than JsonUtility.

var jsonText = Resources.Load<TextAsset>("json_data").text;
var jArray = JsonConvert.DeserializeObject(jsonText) as JArray;
var names = new List<string>(50);
for (var i = 0; i < 50; i++)
{
    names.Add(jArray[i]["name"].ToString());
}
Debug.Log(names);

With Json.Net we also don’t convert the JSON text into C# class instances. We convert the structure into a generic JArray and JObject instances. Accessing values out of those instances produce some overhead, which is a bit lower than the one we saw for FlexBuffers:

Pay upfront vs. Pay as you go

As we can see from this short profiling session, with FlexBuffers we pay for loading data into memory, and for each value we need to access. With JSON we have a big pay upfront tax, due to the necessary parsing step. The tax is also based on the size of the file and not on the number of values you need to extract from the file.

This is why FlexBuffers makes total sense when it comes to configuration files. A configuration file often contains data for all stages of your game. But you will access only a small fraction of the data, based on the player's progression. A similar goes for localization files, complex level files, and other data collections. I know that some games ship SQLite in order to read “random” data at runtime. I believe in many cases FlexBuffers files would be a better solution.

Is FlexBuffers human-readable though?

FlexBuffers is not a textual format so you will not be able to open a FlexBuffers file with a text editor, however, FlexBuffer Unity has a FlexBuffer Browser window:

Here you can see that I was able to open a 127,1MB complex FlexBuffers file and navigate in it without any considerable lag. From my experience, no text editor will open up a 127MB file with comparable speed.

Can I search for values in a FlexBuffer file?

Currently, FlexBuffer Browser only has a path filter functionality:

As you can see we can define a path and vector ranges in order to reduce the number of tree children. A full search is currently not possible, but might be introduced in a future release 😉.

How do I create FlexBuffer file?

You can build a FlexBuffer programmatically, but this is probably not what you want to do for a large configuration. This is why I introduced the following menu points:

By selecting JSON as FlexBuffer... CSV as FlexBuffer... or XML as FlexBuffer... you will be presented with a file picker to select the source file and another file picker to select where you want to store the .bytes the file containing the FlexBuffer.

What if I want to change something in my existing FlexBuffers file?

Changing data directly in a FlexBuffers file is sadly not that trivial. However, what we can do is export the FlexBuffer file as JSON:

Open the JSON file in a text editor and edit it:

Click Import from JSON button:

How Can I try it out myself?

FlexBuffers Unity is based on FlexBuffers-CSharp. And also has a self suficient Github repository of its own:

mzaks/FlexBuffersUnity

Which is structured as a Unity Package. So if you want to use FlexBuffers Unity in your game, just put following line in your manifest.json file:

"com.mzaks.flexbuffers": "https://github.com/mzaks/FlexBuffersUnity.git",

Thank you for reading, and please let me know if you tried FlexBuffers Unity.