Systematic Gaming

September 16, 2008

Load Times: Packfiles

Filed under: file management, game programming — Tags: , — systematicgaming @ 5:27 am

A packfile is simply a single file containing one or more other files. They’re very useful for load times, and can be used in a number of ways. We can put files that get loaded together into a single packfile and load them at once, such as putting the data for a single level into a single packfile and loading in a single read. We can also put our entire filesystem into one or more packfiles, which lets us handle compression cleanly as well as making distribution easier.

So what do we need to do to create a packfile? No much:

  • File name and path, usually relative to a specific root directory
  • File size and compressed size

In this article now we’ll look at how to implement a packfile system.

Let’s look at one possible structure for a packfile’s metadata

struct PackFileHeader
{
   int32_t version;        // packfile format version
   int32_t numFiles;       // number of files in this packfile
   PackFileEntry fileEntries[];
};

struct PackFileEntry
{
   uint32_t nameHash;       // hash of the full path of the file
   uint32_t size;           // size of the original file
   uint32_t sizeCompressed; // size of the compressed file, 0 if uncompressed
   uint32_t offset;         // offset of the file in the packfile
};

We don’t store the filename, only a hashed name. This makes the data small and checking if a file exists fast – since we can sort the PackFileEntry list by hash for quick lookup. This only supports 4GB packfiles, but that’s not a huge problem, we can have multiple packfiles if necessary. Also important is the version in the PackFileHeader – I highly recommend adding a version to your data files, not just for supporting backwards compatibility but also for detecting out of date data.

So how do we read from a packfile?

  1. Read the header
  2. Read the entry list
  3. Seek to the file entry’s offset
  4. Read the file

Hmm, 3 read operations to access a file? That’s not very good. If we cache the header and file entry list then we only have to seek and read. We can also open the packfile once and keep the file handle around, so we don’t need to open and close the file.

Of course we could just read the entire packfile at once if we know we’ll load all the files inside it. That only works if we’ve carefully organized our data into separate packfiles. It’s a very good idea, and ideally you should organize your data that way. However, if you have a very dynamic game, customized characters or an open world it’s not always feasible.

Using packfiles as part of your file system is straightforward. When you start your game register each packfile by loading the header data and open the packfile, keeping the file handle around. Then when a file is read, you check the list of packfile entries and if found read from the packfile.

But enough talk, lets see how we could integrate packfiles into our file manager:

class PackFile
{
public:
   PackFile(const char *packfile);
   ~PackFile();

   // returns index of file, -1 if not in packfile
   int FindFile(const char *filename);

   // initializes a fileoperation to point to the given index
   void OpenFile( int index, FileOperation &operation );

private:
   PackFileHeader *mHeader;  // pointer to the file header (& entries)
   HANDLE mHandle;           // open handle to packfile
};

int PackFile::FindFile(const char *filename)
{
   // get hash of file name, assumes Unix style path
   uint32_t nameHash = Hash(filename);

   // search entires, returns -1 if not found
   int index = BinarySearch( nameHash, mHeader->fileEntries, mHeader->numEntries );
   return index;
}

void OpenFile( int index, FileOperation &operation )
{
   assert( index > -1 && index < mHeader->numEntires );
   operation.mHandle = mHandle;
   operation.mPosition = mHeader->entires[ index ].offset;
   operation.mState = STATE_END;
   operation.mOperation = OP_OPEN;

   operation.mFileSize = mHeader->entires[ index ].size;
   operation.mCompressedSize = mHeader->entires[ index ].sizeCompressed;
}

We also need to change our LoadFile a little, to search through a list of registered packfiles

FileHandle FileManager::LoadFile(const wchar_t *filename, void **buffer, uint64_t *size)
{
   FileHandle file = GetFreeOperation();
   if ( INVALID_FILE_HANDLE == file ) return file;

   int packFileIndex = -1;
   for ( int = iPackFile = 0; iPackFile < mNumPackFiles; iPackFile++ )
   {
      packFileIndex = mPackFiles[iPackFile].FindFile( filename );
      if ( -1 != packFileIndex )
      {
         mPackFiles[iPackFile].OpenFile( packFileIndex, mOperations[file] );
      }
   }

   // normal file, open normally
   if ( -1 == packFileIndex )
   {
      mOperations[file].Open(filename);
   }

   mParams[file].mBufferPtr = buffer;
   mParams[file].mSizePtr = size;

   return file;
}

Then we need to add a seek stage, since we’re not always loading from the beginning a file. Also, we change compression to only work from packfiles, since magic numbers suck.

void FileManager::Update()
{
   for ( int i = 0; i < MAX_OPERATIONS; i++ )
   {
      switch ( mOperations[i].GetOperation() )
      {
         case OP_NONE:
            break;

         case OP_OPEN:
            int fileSize = mOperations[i].mFileSize;

            void *fileBuffer = new uint8_t[fileSize];
            *mParams[i].mBufferPtr = fileBuffer;
            *mParams[i].mSizePtr = fileSize;

            mOperations[i].Seek( mOperations[i].mPosition );
            break;

         case OP_SEEK:
            mOperations[i].Read( fileBuffer, fileSize );
            break;

         case OP_READ:
            // overlapped reads are asynchronous, so poll, if done close
            bool complete = GetOverlappedResult(
                  mOperations[i].mHandle,
                  &mOperations[i].mOverlapped, NULL, FALSE);            

            if ( complete )
            {
               mOperations[i].mState = STATE_END;
               mOperations[i].Close();

               // check for our magic number
               if ( mOperations[i].mCompressedSize )
               {
                  DecompressData( i );
               }
            }

         case OP_CLOSE:
            // nothing to do here
            break;
      }
   }
}

What we’ve done is implement a simple version of packfiles. There a number of other tricks we can do with packfiles, but in reality this system give you 90% of the advantages. You get cleaner compression support and the ability to load many files in a single read. An additional bonus is you’ll get smaller seek times since packfiles put the data physically closer than individual files.

Next, we’ll look at how to reduce seek times by optimizing the layout of our disc.

Advertisements

Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at WordPress.com.

%d bloggers like this: