Part 1: File Decompression and Recompression

EDIT: Note this tutorial is outdated and a lot more progress has been made since I first wrote this many years ago. The easiest way to get into modding these games is to use my Hack tools (currently mac only but the windows version is in beta testing in the discord server). 

We have a discord server for everything Pokemon Colosseum, XD and Battle Revolution hacking here: https://discord.gg/xCPjjnv

---------------------

I can finally start posting all the data I have so far on hacking Pokemon Colosseum and Pokemon XD. There is a lot of it so it will take a while. Here is the first part. This is very important as it is a prerequisite for almost anything you want to hack in these games.

Before I get into the details of the hacking I have some hints and tips that you should read.

=====================

Prerequisites

=====================

Some useful files for looking up data: 

• Abilities
• Common Values
• Item List
• Pokemon IDs
• Pokemon Moves

Especially useful is the list of pokemon ids since they aren't always the same as their pokedex order. Those used to hacking gen III may already be aware of this since it is unique to the gen III games.

The source code for my editing tools:

https://github.com/PekanMmd/Pokemon-Colosseum-Hacking-Resources

Some files you'll need:

• fsys extract and decompress script.txt
• pokemon lzss recompress script.txt

- I. Things you'll need -

1. Download Gamecube rebuilder (or similar program).
2. Download QuickBMS.
3. Any hexadecimal converter (or you can do it all in your head if you prefer). Any free online converter is enough. You may also want a binary converter for XD. [Edit by @evandixon: When set to Programmer mode, the built-in calculator in Windows will work.]
4. A hex editor. (No tools as of the time of writing except my iOS apps which won't be released for the reasons explained in part III)
5. Check out my git hub repository and download the quickBMS scripts.
6. If you want to edit the banner for the ISO download a program called GCBanner. I won't be explaining it here but there are tutorials somewhere online.
7. An emulator will be very useful for testing. Dolphin is really good these days. It plays Colosseum perfectly with the right settings but does have a few glitches with XD. It is convenient having the emulator on the computer you're hacking on because once you build the ISO you can test right then and there. However, computers can rarely produce good emulation speeds. I recently started using Nintendon't on my Wii U and it is brilliant. If you have a Wii or Wii U I recommend finding a tutorial online on installing Nintendon't as the consoles have built in GameCube emulation made by Nintendo themselves and so runs the games perfectly at full speed.

- II. Useful knowledge -

1. Learn basic hexadecimal to decimal conversion. As long as you understand what it is a hex converter will do the rest. I will not be covering this myself as there are many tutorials already. A very tiny bit of binary also comes up in XD. If you understand hex you probably get binary too and there are many converters online.
2. Experience hacking other gen III games is very valuable. Hacking other gen games will also be of slight use but gen III does have it's quirks.
3. My git hub repo also has some useful files for looking things up like the ids of each Pokemon so download those files also. ( I haven't put it up yet but will do soon)
4. Less importantly, since the game was originally coded by the Japanese, a lot of the file names are transliterations of the Japanese names. For example, all the Pokemon and NPC models are named by their Japanese names. I happen to speak Japanese and I've found it really useful because it allows me to more easily figure out what is inside a file. Sometimes the name makes it very obvious. Like in field_common.fsys there is a file called "bikkuri". I know this means surprise in Japanese so I was able to make the assumption that this file contains data for when an NPC trainer spots you and the little surprise animation pops up above their head. There are no Japanese characters anywhere so just understanding a bit of the language phonetically helps.

Some phrases that might come in handy:

Japanese	English
snatch ball	snag ball
snatch dan	team snagem
dark pokemon	shadow pokemon
gonza	gonzap
mirabo / mirrorbo	miror B.
esaba	pokespot	(literally feeding place)
waza	move/attack
usohachi	bonsly
gonbe	munchlax
lugia	lugia	( convenient )
runpappa	ludicolo
relive	purify
labo	lab
nakigoe	cry	(as in a pokemon's 'roar')
10manvolt	thunderbolt	(literally 100,000 volts)

- III. N.B. -

1. The GameCube files are in "big endian". Those used to hacking on other platforms will be used to reversing bytes but there is no need for that at any time on the GameCube. The downside is that this makes opening files on "little endian" computers (as far as I know that's about every computer in common use) a bit trickier. Programs may need to byte-swap when reading and writing values longer than 1 byte in length.
2. The amount of time spent compressing and recompressing files (not to mention the difficulty of emulating GameCube games) means testing can be a long and tedious process so not everything has been tested fully. Also I'll be teaching you 'safe' techniques. I haven't reached the level of repointing to expand the size of data regions. Hopefully when more automated tools are available more possibilities will open up.
3. The preparation section isn't very exciting but is necessary so pay attention. You may find yourself repeating these steps many times before programs are available. In order to minimize repeating tedious steps, try to edit the files as much as possible all in one go before recompressing. Don't forget to be conscious of increasing the file size as I'll mention in the preparation section.
4. All of my tools are iPad apps which I can't release because I doubt it'll be allowed. Even if Apple were to somehow accept them for the App store, Pokemon Company/Genius Sonority would probably complain. The programming was all done in a rush and the UI is pretty poor but I will be releasing the source code for everything so hopefully someone else will make some better tools.
5. Do check out my YouTube videos. They're a bit long but I do go into more depth about many of the concepts and techniques introduced in these tutorials. I especially recommend the fsys and compression video because that is a bit tricky to describe.
6. Since there are few automated processes and it is very easy to make mistakes, make sure you keep backups of your files. Changing just one byte incorrectly can ruin your files.
7. The two most interesting files are the Start.dol and common_rel.fdat (in common.fsys). These two files contain a lot of the important data. They are loaded into the game's ram when the game is loaded which means that anything that can be done in those files can, in theory, also be achieved through Action Replay codes or if you prefer you could do a ram dump, edit the ram directly and then load the edited ram.
   This includes editing pokemon stats, move data, TMs list, a little bit of text, starter pokemon and gift pokemon and trainer teams(colo only).
8. XD has the physical/special split! It's only used for shadow moves though. I would really like to find a way to incorporate it into all the moves. I'm really hoping someone can crack this.
9. I'm hacking the US versions of the two games. There may be differences if you're using a different region.

==================================

Part 1: Extraction,Decompression and Recompression

==================================

- I. File extraction -

This is how we get individual game files out of the ISO.

1. Run Gamecube Rebuilder.
2. Select: Image >> open. navigate to your gamecube ISO/GCM and open it.
3. In the file browser to the right, right click on 'root' and then click 'extract'. Choose a location to save the folder in on your computer. This will save all the game files in a folder called 'root' wherever you choose to save it. This can take a while but is useful if you want easy access to all the files. Alternatively, if you only want an individual file then you can right click on the specific file you want and extract it in the same way.
4. Edit any of the files, making sure to either keep them in the same folder or put them back later with the same file name.

- II. File reimporting and ISO rebuilding -

This is how we put the files back into the ISO after we've edited them. Try to make sure the file size is the same as the size of the original file you are replacing. If it is too large then you may mess up our ISO.

- importing an individual file-

1. open the ISO image as in part i.

2.  find the file you have edited, right click it, click 'import' and load the edited file.

- rebuilding whole ISO-

1. If GameCube Rebuilder is still open with an image loaded, select: image >> close.
2. Run GameCube Rebuilder.
3. Select: root >> open, navigate to the 'root' folder you made in part I.
4. Select: root >> save, choose where you want to save your new ISO/GCM and name it.
5. Select root >> rebuild. A new ISO/GCM will be saved as you specified in step 3.

- III. fsys extraction and decompression -

Most of the game files are compressed and archived in .fsys files (kind of like a .zip or .rar file). This is how we get them out and decompress them in order to be able to edit them. Some files haven't been archived and don't require decompression but most of the useful and interesting ones do.

1. Run quickBMS.
2. Select the "fsys extract and decompression script" (see attatched files or source code on github) and click 'open'.
3. Select your .fsys file and click 'open'. You can select multiple files at the same time and the program will extract from them one at a time.
4. Choose a folder to save the files in and then click 'save' (warning this will most likely extract multiple files so you may want to create a folder for them). You can stay in the same folder if you wish. This script automatically decompresses and adds a '.fdat' file extension to each file extracted. Open the .fdat files in a hex editor or if opening them in a program then you can read the bytes as binary data.
5. If the fsys archive contains multiple files with the same name (which is quite common actually), then simply type 'r' into the command line and then press the return/enter key on your keyboard quickBMS will automatically rename the files with sequential numbers.

- IV. File recompression -

Of course, once the files have finished being edited they must be recompressed so we can put them back in.

1. Open the file you want to recompress in your hex editor and record the exact number of bytes in the file. Any decent hex editor should show this value somewhere; if not automatically then highlight all the bytes (ctrl + a on windows) and then it should show you how many bytes are highlighted. Make sure you convert this number to decimal if it is in hexadecimal.
   
2. Open the "pokemon lzss recompress script" (see attatched files or source code on github) in any text editor. A simple one like notepad is best. At the end of the file you will see 2 identical numbers. Replace them with the file size you recorded in step 1 (again remember to convert it to decimal). You should now have two new identical numbers which represent the file size. Save the modified script.
   
   (If you are editing the same file a lot without changing the file size it may be convenient to save a separate script for the specific file with a different name so you can reuse that script and skip the first 2 steps.)
3. Run quickBMS and follow the steps in section III above, however, this time run it with the script from step 2 and then the file you are trying to compress. This script will save the file without a .LZSS extension.

- V. fsys reimporting -

Once the files have been recompressed they now need to be put back precisely where they were found in the archive so that they game can load our new edited data.

1. Make sure your compressed file size is smaller than the original compressed file size. In each section where I edit .fdat files I will share tips on how to effectively do this. In general, if you can make long rows of 00 bytes, for example by deleting text or removing items, EVs, or a whole Pokémon from a trainer, your compressed size will decrease. If you edit data and you aren't creating long rows of 00bytes then you are most likely going to increase the compressed file size. Such is life. If you want to learn more then research how the LZSS compression algorithm works and about 'entropy' in terms of data compression, which is similar to the concept of entropy in physics. In very basic terms, we are trying to decrease the randomness of the data so it compresses better.
2. Open the .LZSS file you want to reimport in your hex editor. Note the exact number of bytes in the file. If you had changed the uncompressed file size then also note that beforere compressing the .fdat file.
3. Open the .fsys file you originally extracted the .LZSS file from (as a .fdat file) in your hex editor. Note the exact number of bytes in the file (in hexadecimal this time).
4. I recommend checking out my youtube video on fsys and decompression as this step gets complicated.
   Bear with me, this step is hard to explain by typing. However, it is usually very easy to do. The .fsys file is an archive comprising multiple files (or sometimes one file) within it. the files in the .fsys are all separated by some rows of 00 bytes and each one starts with the magic bytes "LZSS" which you can see in the ASCII version of the data in your hex editor. Your task in this step is to figure out which file in the archive is the one you have edited and are trying to replace.
   Not far from the start of the .fsys file you have the file names of the files in the archive. Use this to figure out the position of your file in the archive. If the file was sequentially named by quickBMS because of name conflicts then the lower numbered ones were the first ones in the archive. You can also figure out which file you want by looking at the header for each file which starts with the "LZSS" and then has the compressed and uncompressed file sizes. Match these with the sizes you expect for your file. Also the data should start with the same bytes as your file since you probably didn't change much at the very beginning of the file so it decompresses the same way.
   The files I edit a lot "common_rel.fdat" which is the first file in common.fsys, and the text data files for the maps which is the third file in any of the .fsys files for a map. This is true for both games and makes this step easy to figure out.
   If you somehow got through all the text and managed to make sense of it then it will soon become obvious how to figure all this out. There is a very easily recognised pattern to the .fsys files.
   
5. If you scroll down just past the file names then you will find data separated by rows of 0x11 bytes (i.e. 11 11 11 11 11 11 11 11 11 11 11 11) in colosseum or just 0x00 (0x0000000000000000000) in XD. The blocks of data which precede each of these rows describes and individual file in the archive and this data is in the same order as the file names and the same order that the files in the archive are saved in. This block of data contains the offset within the .fsys file that the .LZSS file starts as well as both the compressed and uncompressed file sizes. If we have changed any of these values (usually the compressed file size) then we must update this information here or the game will crash even before we reach the title screen. Before you change the decompressed file size value you must add 16 bytes (0x10) to your value because there is also a 16 byte header which isn't part of the LZSS file we created but is in the .fsys file.
   
6. Scroll to the start of the file in question. The "LZSS" bytes at the start will let you know you're at the start of the file. The 16 bytes starting with "LZSS" are the file header. Again this has the compressed and decompressed file sizes for that file so if you have changed these then update these values once again (remembering to add 16 bytes to your decompressed file size). Always remember you need to update the decompressed file size in two places.
   
7. Copy all the bytes from your .LZSS file (ctrl + a to highlight all then ctrl + c to copy).
8. Starting from the byte after the header in the .fsys file, highlight as many bytes as the number of bytes you copied from the .LZSS file. Assuming your .LZSS is smaller there will still be some bytes left over from the previous file. Replace these extra bytes with 00 until the end of the previous file. If your file is too large then you willoverlap with the next file, hence the emphasis on keeping your file as small as possible.
   We can move the next file if necessary as long as we update the pointers in its block of data before the 111111 row corresponding to the next file. I've never actually tested this myself but it seems obvious enough. There are a few bytes between each of the files for tidiness but we can overwrite these if we want to since this won't increase the size of the .fsys file. Again, I haven't tested this but if you wanted to increase the total size of the .fsys file and repointed everything properly then you would probably need to update the ISOs .toc file with the new file size of the .fsys. I haven't done this before though so good luck if you attempt it. The details of the .toc file have probably been documented somewhere before as it is pretty much the same in every gamecube ISO.
   
9. That's it ! You've successfully extracted, decompressed, recompressed and reimported a game file from a .fsys file. This process is a lot easier to do than describe and not knowing how to do this was the only reason Pokémon Colosseum and XD hadn't been hacked before. Now we can research and edit the game files with ease.

Part 2 -