Home For Builders Fan-Created Utility: Hak Updater

Fan-Created Utility: Hak Updater

Intended Audience:      [Printer Friendly Version]
Profile by Polyhedral Nov 19, 2002

Utility: Hak Updater
Author: Artur "Revinor" Biesiadowski
Download Locations:
- Revinor NWN Utilities
- Neverwinter Vault

What is the Hak Updater?

The Hak Updater utility was created by Revinor to overcome the issue that small changes to an existing Hakpak then required the entire Hakpak to be downloaded by the player again, instead of just that small bit that was changed. The Hak Updater solves this problem by automatically downloading only the files you do not already have. This is very good news for people who use a slower dial-up connection to the Internet.

Summary of How it Works

The Hak Updater maintains a list of all the servers and Hakpaks that a player uses. Prior to launching a game, it checks the listed site for any updates that might have been made to a Hakpak. If a difference is found between the server/module version of the Hakpak, and the version the player has stored locally it will download only the files that have changed, keeping the rest of the Hakpak unchanged and intact. This is of immense value since it means a small change to a very large Hakpak does not have to result in every player downloading the entire Hakpak again, but only the tiny portion of it that actually changed since the last time they logged on to the module.

Further Details

Warning : Technical Kung-fu May be Required

The Hak Updater operates by making MD5 checksum comparisons of every file a Hakpak contains ('MD5' is a standard algorithm used to verify data integrity). This means that it does not rely on file names and does, in fact, have a filename independent cache of data.

This means is that if you have multiple Hakpaks, each of which has a rural tileset, it will contain a single instance of all the data that exists in an identical form across each rural set. It will also contain multiple instances of the data that has changed between each rural set. So if each rural set has a 'waterfall.tga', and in each case the file is different, it will maintain each version in the cache.

When a new Hakpak is downloaded, it will do a checksum comparison of these files. If the new 'waterfall.tga' that is in the server's Hakpak matches the checksum of any of the waterfall files in your cache then it will pull that version of the file from the cache. If it does not find any checksum matches then it will download the file from the server. The server admin must maintain all the files on an http-accessible site for the Hak Updater to download from.

This means that each file a Hakpak contains must be available on the server outside of the Hakpak itself. While it requires a small amount of work on the server, these steps become more and more trivial as Revinor refines his utility.

Revinor is also working with Maximus (of NW Vault) and together they have created a Library URL system which will check NW Vault's excellent resources for all the necessary files. An amazing collaboration that no doubt makes the task of managing multiple Hakpaks a much simpler process for all involved.

Interview with Artur "Revinor" Biesiadowski

Q. What drives you to create such utilities?

A. Scratching an itch. I would like to do something, but there is no ready tool for that or one available has major shortcomings - I create my own. Of course, it is a problem of time/benefit. I'm not going to spend hundreds of hours to get only small improvement over something which is already available. But if I can do something useful, for me and other people, by spending few hours of free time on time, I'll do it.

Q. Your various tools have become very popular; did you expect to find such a wide audience for your work?

A. I have actually started with java3d model viewer. This was project for very small audience and was happy to see that few people are going to use and enjoy it.

With release of beta toolkit, things started to be a lot more interesting. We have got few proprietary file formats. I wanted to look inside files to develop my java3d model loader further - and decided that maybe somebody else would also like to use my tools, so I have done them in C, instead of java. They were all meant only as quick hacks before somebody else will do the 'proper' tools - this is one of the reasons why I have made source public domain from very start. They have outlived my expectations - first Zoligato has made NWNViewer, which rendered my unmod/unbif tools obsolete and lately, Torlack's model decompiler replaced my mdl2ascii. Both these tools do the job better – I was just quick and dirty, they have took some time and done it properly (basing bits of it on my work - which I'm happy to hear).

Q. How do you go about getting the file formats to make your utilities?

A. HexEdit and a lot of guessing. Most formats are really trivial - especially given the fact that you do not need to know everything for decoding (encoding is a lot harder as you have to get every bit covered).

Good example is binary model format. I have done it totally wrong. I was just comparing existing ASCII models with their binary counterparts and looking for same values. But I've got basic structure wrong - instead of decoding tags inside file, I have just assumed that entries always go in given order. This allowed me to decode 99% of files, but few missing ones where not possible to solve. We had to wait for Torlack's tool to get them really right.

Q. Is there anything you would like to see changed with regards to how Hakpaks are implemented?

A. This is a subject for entire discussion - I do not think there is enough place to cover it here... In short:

1) It is extremely important to get away from current way of assigning indices to 2da entries in load order. I mean, they can be indexed like this internally, but values which are used in toolkit files should be ones which are in index column in 2da file, covered to internal value on load time. This would allow 2^32 values for everything for free, enough for community to divide it per-author and stay globally unique. In ideal world, I would prefer to have everything referenced by label internally, instead of by number, but this is too much to ask. Please note that this change will not invalidate current files - because so far, load order happens to be same as explicit index number.

2) Having that, incremental 2da files would be great. Having appearance entry all by itself, or for group of monsters in separate files. This would make distribution of small, self-contained packs possible and merging them would be painless.

3) With point 2 done, it should be easy to get multiple Hakpak support.

If I would have to talk totally about the blue sky, having internal database of files similar to way in which HakUpdater works would be great. No more solid hak downloads, server would just send info, which versions of which files should be used at given server. But this is too much to ask at this point - I would be happy enough if points one and two above would be done.

Q. Are you developing any new tools?

A. Currently no. I'm still working on HakUpdater - I'm trying hard with Maximus to get shared files library running on NWVault, so people can outsource their downloads from their small webservers to a big site.

In future, I plan to do something on subject of persistent worlds. Current solutions are ugly, because they cannot be automated - they require manual copying data from some file to OnLoad script and recompiling inside toolkit. I'm thinking about something in lines of automatically porting module.ifo inside save games to a new version, or creating simple compiler which would generate OnLoad script containing all needed variable assignments.

I'm also thinking about doing something which would allow easy distributed creation of module files. Some kind of build system, which would allow to keep everything unpacked in cvs system, build only parts of module for faster testing and creating full-build when needed.

But both these things can wait until I can finish HakUpdater - I want to have it rock solid first.