Systematic Gaming

July 2, 2009

Asset Management – Processing

Filed under: asset management, game programming — Tags: , , — systematicgaming @ 4:21 am

Last time we looked into the lowest levels of asset management, which can be handled by referencing counting and intelligent loading.  With the next level, the processing layer, we look at how we prepare assets for our runtime.  This is an important step and where a lot of optimization occurs.  We can break the processing layer down into a few distinct stages:

  • Asset referencing
  • Building (or baking)
  • Packing

Asset Referencing

Asset referencing is the act of having one asset use another.  A material using a texture for example.  We mentioned composite assets in the runtime layer but didn’t go into detail on how we should actually maintain those references, or links.

Some links are implicit and some are explicit.  Implicit links are ones that are assumed by the engine, for example a model named “monster3” might always use the material named “monster3”.  Implicit links are based on game or engine specific rules and are quite common.  Explicit links are specified by the asset itself and are the general way assets reference each other.

The naive approach for supporting references is to store asset file names for each link.  This is quite straightforward to implement, but has a number of drawbacks.  File names require us to store and manipulate strings, which is waste of CPU time and memory. It also means assets are tightly coupled to our file system, which doesn’t always make sense for generated assets.

One way is to improve this is to assign each asset a unique id and reference the id instead of a file name.  This means we can save memory and processing time since we only need to store the asset id for each reference.  This id can be be assigned externally (from some central database) or simply be a hash of the asset name.

For example we could uses a simple CRC hash such as CRC(“Texture:Monster3Head”).  Hashes are nice because we can generate them on the fly if needed.   Alternatively, with assigned asset ids, asset references can remain stable even if the linked asset’s name changes.  In addition we don’t have to worry about hash collisions.  However, we do need a way to assign unique ids.

Either method means that we can store references compactly and uniquely identify each asset.  We’ll go into more depth about asset ids when we talk about the higher levels of asset management.

Data Building

Data building is actually a very involved process, involving converting data from artist created formats to runtime ready data.  A detailed description of the actual data conversion process, sometimes called baking, is outside the scope of this article.  However there are some important optimizations and considerations that are influenced by asset management.

One important optimization in building data is the idea of a load and go format – creating data than can be loaded and used right away with no post-load processing.  This can really help reduce memory fragmentation and the CPU cost of loading data. However the type of asset we are building will influence our data layout.

Let’s take an animated model as our example.  Our model will consist of a hierarchy of animated joints (Node) each with geometry (Mesh) attached.  One way of organizing this data could be:

struct Model
   int numNodes;
   Node nodes[numNodes];

struct Node
   Transform transform;
   Mesh mesh;

struct Transform
   Vector3 position;
   Vector3 rotation;
   Vector3 scale;

struct Mesh { ... };

This is a fairly straightforward way of processing our model.  However, since our model is animated we will need a per-instance copy of each Node’s transforms.  We could walk through the array of nodes and make a copy of each transform when we create a new instance.  But it would make updating the transforms when we draw difficult and messy as well a causing us to walk a relatively large structure.

By simply reorganizing the model’s data structures we can make a data layout that’s more friendly to asset management:

struct Model
   int numNodes;
   Transform transforms[numNodes];
   Mesh meshes[numNodes];

All we’ve done is separated the transform and mesh data into separate arrays, but now the data we want to copy per-instance is cleanly separated.  Also, it makes instanced models easier to handle at runtime, because we’ve separated mesh data from transform data we can more easily structure our code to handle transforms and meshes separately.

This is just one example of how we need to build our data with asset management in mind.

Data Packing

In my series on load times I mentioned the importance of packing files together to reduce load times.  The processing layer of asset management is where we perform this optimization and more.

Proper asset management allows us to track which assets reference each other. This means we can easily group these assets together on disc or within the same packfile.  In some cases we can merge assets into a single data set.

This allows us to intelligently duplicate data on the disc if needed. If an asset, such as a texture, is shared by multiple assets it may make sense to place multiple copies of the shared asset near assets that reference it.  We would then load the closest copy when the shared asset is needed.

In a open world game, where assets are streamed continuously as the player moves, we can use our asset management system to layout data such that data is organized by in-game proximity.  Since we know the world layout and the position of assets in the world, we can put assets that are close in game, close on disc.  Reducing seeks and allowing more efficient streaming.  We can also use our knowledge of asset references to determine how much memory is required to load a section of the game.

Clearly asset management, especially asset references, has many useful applications we can apply during the data processing phase.

Next article we’ll examine the highest level of asset management – where tools and designers interact with our engine and pipelines.


Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Create a free website or blog at

%d bloggers like this: