psy_commando

The PX compression format, for a lack of better name, is a custom format used notably by the Pokémon Mystery Dungeon : Explorers of Time/Darkness/Sky games. Most of the files using this compression method are contained within either PKDPX or AT4PX containers. Overview of the Format The format itself is nothing wildly complex. It revolve around using what we'll call a command byte. The command byte is what will tell the decompressor what to do with the data that comes after it. The command byte can hold information for up to 8 "operations" to do on the data that follows. Each operation is represented as a bit. And each operation may use one or more byte(s) of data after the command byte. If the highest bit is 1, we copy the first following byte as-is. If its 0, we compare the value of the high nybble of the following byte to the list of control flags. And depending on which flag match, or if none match, we'll know what to do next. Note that, those control flags are always "0n" where "n" is an hexadecimal value from 0 to F. Also, note that, these flags are computed on a file by file basis! They're "tailor-made" for each individual file. We discuss this more in details in the other sections below. Decompression To make explaining things easier, you could imagine that the bytes we're decompressing come from a FIFO queue. And also imagine that we put the decompressed bytes into a Double Ended Queue or deque for short. (Even if in practice those are probably too inefficient to use in this case!) To decode the command byte, we simply look at the value of all of its 8 BITS, one at a time. From the highest bit to the lowest. So lets say we're in a loop that isolate the value of a particular bit each turn. Something like: uint8 mask = 0x80 loop while( mask > 0 ) { uint8 bitval = mask & cmdbyte //Bitwise AND if( ! inqueue.isempty() ) //Make sure our input data is not empty, because its not guaranteed we { // won't reach the end of file while handling a command byte! ... // Handle the cases written below } else { break; //If we reach the end of file, stop the loop! } mask = mask >> 1 //Bitshift right by 1 } On each turn of that loop, we'll take the value of the current bit, and depending on its state, we'll decide what to do based on these criteria: If the Bit is 1 If the BIT we've isolated from the command byte is 1, then we pop the next BYTE from the input queue, and push it as-is at the back of the output deque! If the Bit is 0 Otherwise if the BIT we've isolated is 0, we'll pop the next BYTE from the input queue. Then, we'll try to find whether the high nybble of the BYTE we just read is the same value as one of our control flags. We also want to keep the value of the low nybble for later. We'll refer to the high nybble and low nybble we got here as "nbhigh" and "nblow" respectively. We got 2 possible cases : If one of our control flag match the high nybble "nbhigh" of the BYTE we just read, then it means we're inserting a pattern of 4 nybbles or 2 bytes into the output deque. Go to Inserting Byte Pattern. We'll refer to the index of the control flag we got as "ctrlflagindex". If none of our control flags match the high nybble "nbhigh" of the BYTE we just read, it means we're copying a sequence of bytes from our decompressed output so far. Go to Copying a Sequence. Inserting Byte Pattern Then its either one of these: byte1 = nblow << 4 BitwiseOR nblow byte2 = byte1 //Then we just push those two bytes, and we're done! If the index of the control flag "ctrlflagindex" is the first one in the control flag table, at index 0, we calculate the value of the 2 bytes we'll push at the back of the output deque this way: Otherwise, for any other "ctrlflagindex" flag index, we have a few other cases to take into account. They all have in common using the value of "nblow". Got to Inserting a Special Byte Pattern. We'll add a new variable called "basenybbleval" to make things easier to understand. And we'll put the value of "nblow" into it right away. Inserting a Special Byte Pattern First depending on the control flag index, do one of the following : basenybbleval = basenybbleval + 1 basenybbleval = basenybbleval - 1 If "ctrlflagindex" is 1. We increment the value of all 4 nybbles. Or in our example, the value we use to store the base value of all nybble "basenybbleval". Otherwise, if "ctrlflagindex" is 5. We decrement the value of all 4 nybbles. Or in our case, the value we use to store the base value of all nybble "basenybbleval". From here, lets put the value of "basenybbleval" into 4 separate variables, each holding the value of the individual 4 nybbles. Lets just name them, "nybble0", "nybble1", "nybble2", "nybble3", and put the current value of "basenybbleval" into all of them. (You should use an array or something, as they're only named like this for clarity's sake) Then, depending on the control flag index, do one of the following : If "ctrlflagindex" is between 1 and 4 inclusively. Then substract 1 from the nybble corresponding to the value of (ctrlflagindex - 1). For example, if (ctrlflagindex - 1) is 0, we subtract 1 from "nybble0". If its 3, we subtract 1 from "nybble3", and so on! Otherwise, we add 1 to the nybble corresponding with the value of (ctrlflagindex - 5). If (ctrlflagindex - 5) is 0 we add one to "nybble0" and so on! Now, all that is left to do is to assemble the 4 nybbles into 2 bytes, and push those 2 bytes to the back of the output deque! byte1 = nybble0 << 4 BitwiseOR nybble1 byte2 = nybble2 << 4 BitwiseOR nybble3 We just push those two to the back of the output stack, and we're done ! Copying a Sequence To figure out the offset from the end of our output deque where we begin reading the bytes sequence to copy, we'll need to pop the next byte from the input queue and do the following operations: int16 offset = (((-0x1000) + nblow) << 8) BitwiseOR inqueue.pop() "offset" will contain a negative integer value smaller than the current size of the output queue. Just go from the end of the output deque towards the beginning of it, by the absolute value of "offset". This will get you the beginning position of the sequence of bytes to append to the output later on. We'll call this position "seqbeg". Then, to get the amount of bytes to copy starting from "seqbeg", we take the value of "nbhigh" and add 3. Knowing this, we can now copy the sequence and push it into the back of the output deque. After Handling the Bit After going through those conditions, we know that the next BYTE we pop from the input queue will be a new command byte guaranteed! (Unless we reached the end of the file) So we just have to repeat this loop using this new command byte! However, keep in mind that, there are no guaranty that there are enough bytes in the input to decompress for each bits in a command byte! So its important to check if we reached the end of file each times we handle a new bit from the command byte ! This sums it pretty much! Compression Compression is a bit trickier. The byte patterns and the copy as-is operations are fairly easy to implement, but the string search and managing the control flags gets a little complicated! The tool that compressed the data originally used a sliding window algorithm, or something you could call a lookback buffer. This window/buffer is the range within which the compressor will look for matching strings. That ranges starts from the position of the byte the algorithm is currently handling, to either the beginning of the data, or at most 4096 bytes towards the beginning of the data. For the format of PX compression used in PMD2's files, the buffer's/window's range seems the fixed at 4096 bytes. A smaller buffer/window could be used, but not a bigger one, because of the way the data is stored in compressed file. The shortest matching string possible is 3 bytes, and the longest is 18 bytes. A word of warning about matching strings. Keep in mind that since the high nybble of the data byte is used both to possibly match a control flag or to indicate the length of the sequence to copy, you have a limited amount of possible lengths. You have to reserve 9 values out of the 15 possible for the control flags. Or else some sequence lengths will trigger a 4 bytes pattern replacement instead of a sequence copy, and the resulting decompressed data will be incorrect. So the decompressor/compressor can only use sequences/strings that have a length that matches one of 7 different lengths.. This means, you have to pick the most optimal values to indicate the possible lengths of sequences to copy, and only then pick the control flags from the remaining values ! It is suggested to at least keep 0x0 and 0xF reserved as sequence lengths, as in several cases they appear to be the most used/flexible. But this is in no way a rule. The algorithm used in this discussion will be a 2 pass algorithm. The first one builds a list of operations, then based on the results, the control flags are determined. After that, the operations are all encoded, and inserted into a PKDPX or AT4PX container. This makes it much easier to determine how to optimally use the limited value range to use for lengths of sequences to copy. 1st Pass First, its important to find a good way of storing all the required info for each operations, and preserve the order to process them in! This could be a simple vector/deque of little structures such as this one: /********************************************************************************* compOp Stores an operation to insert into the output buffer. *********************************************************************************/ struct compOp { ePXOperation type; //The operation to do uint8_t highnybble, //The value of the compressed high nybble if applicable lownybble, //The value of the compressed low nybble nextbytevalue; //value of the compressed next byte if applicable void reset() { type = ePXOperation::COPY_ASIS; highnybble = 0; lownybble = 0; nextbytevalue = 0; } }; Where the types of operations are represented using an enum such as this one: /********************************************************************************* All the possible operations that can be done to compress data! Entries 0 to 8 correspond to their respective ctrl flag indexes! *********************************************************************************/ static enum struct ePXOperation : int8_t { COPY_ASIS =-1, COPY_NYBBLE_4TIMES = 0, COPY_NYBBLE_4TIMES_EX_INCRALL_DECRNYBBLE0 = 1, COPY_NYBBLE_4TIMES_EX_DECRNYBBLE1 = 2, COPY_NYBBLE_4TIMES_EX_DECRNYBBLE2 = 3, COPY_NYBBLE_4TIMES_EX_DECRNYBBLE3 = 4, COPY_NYBBLE_4TIMES_EX_DECRALL_INCRNYBBLE0 = 5, COPY_NYBBLE_4TIMES_EX_INCRNYBBLE1 = 6, COPY_NYBBLE_4TIMES_EX_INCRNYBBLE2 = 7, COPY_NYBBLE_4TIMES_EX_INCRNYBBLE3 = 8, COPY_SEQUENCE = 9, }; Then, for the first pass, we simply iterate over each bytes. We'll refer to the byte we're currently iterating on as "curbyte" To determine the optimal operation to use, attempt to find which one out of the 3 methods of compression is the best in this case : Try finding a matching string of at least 3 bytes, starting from "curbyte", within the lookback buffer / sliding window. And from those matches, try to find those that match more than 3 bytes, up to the maximum of 18. Keep the first longest you find. And keep in mind you can only have sequences matching one out of the 7 different lengths you have. So pick those lengths wisely! Try finding if the 2 bytes starting from "curbyte" could be compressed by doing the opposite of one of these methods: Copy low nybble 4 times, increment the 4 nybbles by 1, subtract 1 from nybble#0, and make 2 bytes with those. Copy low nybble 4 times, subtract 1 from nybble#1, and make 2 bytes with those. Copy low nybble 4 times, subtract 1 from nybble#2, and make 2 bytes with those. Copy low nybble 4 times, subtract 1 from nybble#3, and make 2 bytes with those. Copy low nybble 4 times, decrement the 4 nybbles by 1, add 1 to nybble#0, and make 2 bytes with those. Copy low nybble 4 times, add 1 to nybble#1, and make 2 bytes with those. Copy low nybble 4 times, add 1 to nybble#2, and make 2 bytes with those. Copy low nybble 4 times, add 1 to nybble#3, and make 2 bytes with those. Try finding if the 2 bytes starting from "curbyte" contain the exact same nybble repeated 4 times. If all 3 tests fails, you should "copy as-is" the current byte. Do this for the whole file, and once it has been completely processed, look at the sequence lengths you've used, and pick the remaining values between 0x0 to 0xF for your control flags. For example, if you're using as your possible sequence lengths: 0x0, 0xF, 0x2, 0x3, 0x5, 0x7, 0xA Your control flags would be: 0x1, 0x4, 0x6, 0x8, 0x9, 0xB, 0xC, 0xD, 0xE We'll keep those in an array we'll refer to as "ctrlflags". 2nd Pass For this part, you want to handle operations by batches of 8. Look at your 8 operations, and make a command byte from those, setting each bit to 0, unless its a "Copy as-is" operation. Then encode the operation into one or two bytes, depending on the operation type. Finally just write the command byte followed by the encoded bytes in order! Extra Examples To make understanding the command byte, think of it as an array of boolean, where each index matches another array containing small arrays of bytes. This example might make things clearer: 0xFD, 0x53, 0x49, 0x52, 0x30, 0x24, 0x38, 0xE0, 0x30 //In this byte sequence, 0xFD is the command byte! 1111 1101 //Here's 0xFD's value in binary //Here are what bytes in the sequence each bit in the command byte correspond to: [1]111 1101 corresponds to 0x53 1111 110[1] corresponds to 0x30 1111 [1]101 corresponds to 0x24 Here's how this whole sequence is decoded(debug output from ppmd_unpx.exe ): -> Command Byte 0xfd Bit is 1 : Copy 0x53 as is! Bit is 1 : Copy 0x49 as is! Bit is 1 : Copy 0x52 as is! Bit is 1 : Copy 0x30 as is! Bit is 1 : Copy 0x24 as is! Bit is 1 : Copy 0x38 as is! Bit is 0 : appending 2 bytes.. 0 0 Bit is 1 : Copy 0x30 as is! Which results in : 53 49 52 30 24 38 00 00 30 Now take this one, which comes right after the one above: 0x0F, 0x0F, 0xFC, 0xE0, 0xE0, 0x1F, 0xFC, 0xEC, 0x01, 0xF4, 0x88, Which is decoded as such(debug output from ppmd_unpx.exe ): -> Command Byte 0xf Bit is 0 : appending sequence.. (highnybble: 0x0, lownybble: 0xf, calculatedoffset: 0xfffc( decimal -4 ) ) 38 00 00 Bit is 0 : appending 2 bytes.. 0 0 Bit is 0 : appending 2 bytes.. 0 0 Bit is 0 : appending sequence.. (highnybble: 0x1, lownybble: 0xf, calculatedoffset: 0xfffc( decimal -4 ) ) 00 00 00 00 Bit is 1 : Copy 0xec as is! Bit is 1 : Copy 0x1 as is! Bit is 1 : Copy 0xf4 as is! Bit is 1 : Copy 0x88 as is! And results in this : 38 00 00 00 00 00 00 00 00 00 00 EC 01 F4 88 Credits A big thanks to Zhorkenwho coined many of the terms used here, and figured out the header and the compression format!

The PKDPX format is used as a compressed container for generic data. Unlike its specialized equivalent the AT4PX format, the PKDPX format can contain any kind of data. Just like the AT4PX format, its not unusual to find PKDPX files wrapped inside a SIR0 container! Its content is compressed using a custom compression format dubbed PX Compression for the lack of a better name. File Structure The structure is very simple. A 20 bytes long header followed with the compressed data. Overview Offset Length Endianness Type Name Description 0x00 5 big Magic Number The magic number, made of the ASCII characters "PKDPX" {0x50, 0x4B, 0x44, 0x50, 0x58} 0x05 2 little uint16 Container Length The length from the beginning of the header to the end of the compressed data. 0x07 9 byte array Ctrl Flags Array A list of flags to be used in decompressing the container's content. More detail about their purpose on the PX Compression page. They're really nybbles stored in the lower half of a single byte each. 0x10 4 little uint32 Decompressed Data Length This is the length of the raw input data before it was compressed. Note how the length of this field differ from the AT4PX format's own Decompressed Data Length ! 0x14 ..The PX compressed data begins immediately here!.. Credits A big thanks to Zhorkenwho figured out most of the header and the compression format!

The AT4PX container is a format used to contain compressed image data exclusively, as opposed to PKDPX which is more of a generic compressed container. Its not unusual to find AT4PX containers wrapped itself by a SIR0 container. Its content is compressed using a custom compression format dubbed PX Compression for the lack of a better name. File Structure The structure is very simple. A 18 bytes long header followed with the compressed data. Overview Offset Length Endianness Type Name Description 0x00 5 big Magic Number The magic number, made of the ASCII characters "AT4PX" {0x41, 0x54, 0x34, 0x50, 0x58} 0x05 2 little uint16 Container Length The length from the beginning of the header to the end of the compressed data. 0x07 9 byte array Ctrl Flags Array A list of flags to be used in decompressing the container's content. More detail about their purpose on the PX Compression page. They're really nybbles stored in the lower half of a single byte each. 0x10 2 little uint16 Decompressed Data Length This is the length of the raw input data before it was compressed. Note how the length of this field differ from the PKDPX format's own Decompressed Data Length ! 0x12 ..The PX compressed data begins immediately here!.. Credits A big thanks to Zhorken who figured out the header and the compression format!

The SIR0 format is a pretty common wrapper file format with as primary function wrapping other file formats. It provides a pointer to the format's "entry point", along with a list of file offsets to the pointers which need to be translated to NDS memory when the file is loaded. When a SIR0 file gets loaded into memory, its magic number turns from SIR0 to SIRO, and all the pointers in the entire file are modified to be offset relative to the NDS's memory. The second pointer in the SIR header is also set to null when the file has been loaded and turned into a SIRO. File Structure Overview Offset Length Endianness Type Name Description 0x00 16 Header The SIR0 header After Header Varies Content Data The data wrapped by the SIR0. After Content Data Varies Content Padding Some 0xAA padding bytes inserted to align the next section on 16 bytes. May be omitted completely if not required. After Content Padding Varies Pointer Offsets List A list containing the offsets to every pointers in the entire file. After Pointer Offsets List Varies End of File Padding Some 0xAA padding bytes to make the file end on a size divisible by 16 bytes with no leftovers. Header SIR0 header. (Total length 16 bytes) Offset Length Endianness Type Name Description 0x00 4 big Magic Number The 4 ASCII characters for "SIR0" (0x53 0x49 0x52 0x30) 0x04 4 little uint32 Pointer to Content's Header A pointer to the header of the data the SIR0 contains. If there are no headers, it points to the first byte after the SIR0 header. 0x08 4 little uint32 Pointer to Pointer Offsets List A pointer to the Pointer Offsets List located after the contained data. 0x0C 4 Null 4 bytes of zeros. Pointer Offsets List This list is what makes the SIR0 container what it is. Its a "compressed" list of all the offsets of all the pointers stored in the file. This includes both the SIR0 structure, and the contained structure. The game use this list to change the value of each pointers in the file after it has been loaded in memory, so they're relative to NDS memory. The list will always begin with 04 04, as those are the encoded offsets of the 2 pointers in the SIR0 header. Each encoded offset values are relative to the previous offsets. Encoding If a byte has its highest bit (1000 0000) set to 1, then we have to "append" the next byte. Here are the 3 possible cases, using example values: 0x80 0x81 0x82 0x75 => (0x80 & 0x7F) << 21 | (0x81 & 0x7F) << 14 | (0x82 & 0x7F) << 7 | 0x75 0x80 0x81 0x12 => (0x80 & 0x7F) << 14 | (0x81 & 0x7F) << 7 | 0x12 0x80 0x06 => (0x80 & 0x7F) << 7 | 0x06 Note that, since the offsets are stored as 32 bits integer, chaining more than 4 bytes is impossible. Also note that, we always get rid of the highest bit's value using the bitmask 0x7F, after checking if its toggled on! If the byte's highest bit (1000 0000) is set to 0, we use the byte as is, still applying the 0x7F (0111 1111) bitmask. An encoded byte with a value of 0 indicates the end of the list, but only when the byte that came before didn't have its highest bit set to 1 (1000 0000). Also, each times we decode an offset using either of the above, we add its value to the sum of all the previous offsets to get the actual offset. This is why the list starts with 04 04, and not 04 08 for example. Because they're added to each others as we process them. Example: 04 04 92 0C 14 00 AA AA AA AA AA AA AA AA AA AA This list comes from the "/FONT/frame0.wte" file. We ignore all 0xAA bytes, as they are padding. Here are the calculations: 4 4 + 4 4 + 4 + ( (0x92 & 0x7F) << 7) | 0xC 4 + 4 + ( (0x92 & 0x7F) << 7) | 0xC + 0x14 And the results: 0x4 0x8 0x914 0x928 Those are the offsets relative to the beginning of the file "/FONT/frame0.wte", where pointers are stored. Some Code: Here's a little C++11 code snippet, to decode the string of byte. #include <vector> #include <cstdint> using namespace std; //... std::vector<uint32_t> DecodeSIR0PtrOffsetList( const std::vector<uint8_t> &ptroffsetslst ) { vector<uint32_t> decodedptroffsets( ptroffsetslst.size() ); //worst case scenario auto itcurbyte = ptroffsetslst.begin(); auto itlastbyte = ptroffsetslst.end(); uint32_t offsetsum = 0; //This is used to sum up all offsets and obtain the offset relative to the file, and not the last offset uint32_t buffer = 0; //temp buffer to assemble longer offsets uint8_t curbyte = *itcurbyte; bool LastHadBitFlag = false; //This contains whether the byte read on the previous turn of the loop had the bit flag indicating to append the next byte! decodedptroffsets.resize(0); //preserve alloc, and allow pushbacks while( itcurbyte != itlastbyte && ( LastHadBitFlag || (*itcurbyte) != 0 ) ) { curbyte = *itcurbyte; //Ignore the first bit, using the 0x7F bitmask, as its reserved. And append or assign the next byte's value to the buffer. buffer |= curbyte & 0x7Fu; if( (0x80u & curbyte) != 0 ) { LastHadBitFlag = true; //If first bit is 1, bitshift left the current buffer, to append the next byte. buffer <<= 7u; } else { LastHadBitFlag = false; //If we don't need to append, add the value of the current buffer to the offset sum this far, // and add that value to the output vector. Then clear the buffer. offsetsum += buffer; decodedptroffsets.push_back(offsetsum); buffer = 0; } ++itcurbyte; } //Avoid copying the vector by using std::move to explicitly call the move constructor return std::move(decodedptroffsets); } Here's another one to encode the pointers offsets just as they would be at the end of the SIR0 file. #include <vector> #include <cstdint> using namespace std; //... void EncodeSIR0PtrOffsetList( const std::vector<uint32_t> &listoffsetptrs, std::vector<uint8_t> & out_encoded ) { uint32_t offsetSoFar = 0; //used to add up the sum of all the offsets up to the current one for( const auto & anoffset : listoffsetptrs ) { uint32_t offsetToEncode = anoffset - offsetSoFar; bool hasHigherNonZero = false; //This tells the loop whether it needs to encode null bytes, if at least one higher byte was non-zero offsetSoFar = anoffset; //set the value to the latest offset, so we can properly subtract it from the next offset. //Encode every bytes of the 4 bytes integer we have to for( int32_t i = 4; i > 0; --i ) { uint8_t currentbyte = ( offsetToEncode >> (7 * (i - 1)) ) & 0x7Fu; if( i == 1 ) //the lowest byte to encode is special { //If its the last byte to append, leave the highest bit to 0 ! out_encoded.push_back( currentbyte ); } else if( currentbyte != 0 || hasHigherNonZero ) //if any bytes but the lowest one! If not null OR if we have encoded a higher non-null byte before! { //Set the highest bit to 1, to signifie that the next byte must be appended out_encoded.push_back( currentbyte | 0x80u ); hasHigherNonZero = true; } } } //Append the closing 0 out_encoded.push_back(0); }

Pack files are container containing a large amount of sub-files. They're fairly similar to the .sbin files found in Blue Rescue Team, except there are no filenames stored in the table of content(ToC). They usually share the ".bin" file extension with many other formats. The kind of files it contains can be pretty much anything. List of Known Pack Files Since all pack files end up the very common ".bin" file extension, it becomes hard to find which files are pack files, and which aren't. To help with this, here is a list of all the known files ending with a ".bin" file extension that happen to be pack files ! "/DUNGEON/dungeon.bin" "/EFFECT/effect.bin" "/MONSTER/m_attack.bin" "/MONSTER/m_ground.bin" "/MONSTER/monster.bin" "/BALANCE/m_level.bin" Those 6 files have been proven to be pack files. File Structure Overview A global overview of all the parts that make up a pack file. Offset Length Endianness Type Name Description 0x00 4 Null 4 bytes of zeros. 0x04 4 little uint32 NbFiles The number of sub-files / valid ToC entries. 0x08 ( NbFiles * 8 ) + 1 Table of Content This contains the offset of each sub-files, along with its length. After ToC Varies ToC Padding 0xFF bytes used as padding to align the sub-files data on 16 bytes. It should be noted that for some files there will be more padding bytes to align the beginning offset of the first sub-file to a precise address. More details in the Special Cases section! After ToC Padding Varies Sub-Files Data This block contains all the sub-files' data. Table of Content The Table of Content is fairly simple. It consists of a bunch of 8 bytes entries one after the other. Offset Length Endianness Type Name Description 0x00 8 ToC Entry 0 A single ToC entry. .. Rest of the ToC entries here .. Last ToC Slot 8 Last ToC Entry The last entry in the table of content is null, and filled with 8 bytes of zeros. ToC Entry A single entry in the table of content. Its made up of 2, 4 bytes integers. The first is a pointer to the offset of the sub-file data, the second is the length of the sub-file's data. Offset Length Endianness Type Name Description 0x00 4 little uint32 Pointer to Sub-File This value contains the offset of the sub-file within the Sub-Files Data block. 0x04 4 little uint32 Length of Sub-File This value contains the length in bytes of the sub-file pointed to by the "Pointer to Sub-File". Sub-Files Data This is where all the sub-files' data is piled one after the other, separated with padding when needed. Offset Length Endianness Type Name Description 0x00 Sub-File Length Sub-File Data The data for the sub-file. After Sub-File Data varies Sub-File Padding Some 0xFF bytes to align the next sub-file on 16 bytes. Special Cases The 3 pack files in the "/MONSTER/" folder are special. unlike the other pack files, the 3 of them must have their first sub-file stored at the very same address, 0x1300. A lot of padding is added after the ToC in all 3 files just to meet that requirement. If a single one of them isn't at the same address, the game will either black screen, or no sprites will be loaded, and the game will eventually freeze or crash.

The "/FONT/kaomado.kao" file contains character portraits for the Pokémon Mystery Dungeon: Explorers games. It consists of a table of content, also used to look-up portraits available to each characters, and a data section filed with the actual portraits. By default it contains enough ToC entries for 1,154 pokemon. Out of those, only 652 are in use, and many are placeholders containing duplicate data. File Structure Overview Offset Length Endianness Type Name Description 0x00 160 Null Entry The first entry in the table of content is entirely filled with zeros. 0xA0 ToC End Offset - 160 Table of Content The table of content goes from here, to the offset of the first valid pointer in the ToC. More details below in the Table of Content section. After ToC Varies Portraits Data This is where all the portrait data is stored one after the other. More details in the Portraits Data section below. Table of Content Each block of 160 bytes represent all possible portrait slots for a single pokemon. There are 40 potraits slots per pokemon, however not a single pokemon uses its 40 slots. Each of these slots are 32 bits signed integers containing the offset of the portrait data it refers to. Most of those slots are filled with a null value. The null value is particular in that, its not just 0. Its actually calculated by taking the end offset of the data pointed to by the last valid pointer we've encountered starting from offset 0, and then changing the sign of the end offset to negative. For example, if our last valid pointer's value is 0x030058, and the end offset of the data pointed to by that pointer is 0x03034C. Then the value for any subsequent null entries, until the next non-null one would be: -( 0x0003034C ) => 0xFFFCFCB4 Anatomy of a single ToC entry. (Total size 160 bytes) Offset Length Endianness Type Name Description 0x0 4 little int32 Portrait Pointer 0 The Table of Content is made up of 40 pointers such as this one. Their value is signed! ... Rest of the pointers here ... 0x9C 4 little int32 Portrait Pointer 39 This is the last entry for this pokemon's block ! Portraits Data Each portraits is made up of 2 things. A 16 color palette, followed immediately by a AT4PX compressed container containing the actual image data for the portrait. The portraits do not carry any information about their formats. However, we do know that they're all 4 bits per pixels indexed images, and have a resolution of 40 x 40 pixels. The images are also tiled, which means they're made up of smaller "images" called tiles. Each tile is 8 x 8 pixels itself. Each tiles are filled linearly with the pixels contained in the decompressed file. The portraits are made of 5 tiles on their width, and 5 on their height for a total of 25 tiles. Anatomy of a single portrait block Offset Length Endianness Type Name Description 0x00 48 - - Color Palette The first part of a portrait block is a 16 colors RGB 24 bits palette, stored on 3 bytes per color. The first color is transparent. 0x30 Varies - - Compressed Image This contains the actual image data for the portrait. Its a compressed AT4PX container that contains the raw pixels of the image. The image itself, once decompressed, is stored as an indexed 4 bits per pixels, 40x40, tiled image. Once decompressed each images has a length of 800 bytes. More Details So far, it seems that some of those 40 slots for portraits are re-used between pokemon. For example, slot 0 is the "standard" default image shown when a pokemon talk, or the image that appears in Chimecho's assembly for a pokemon specie. It seems that odd numbered slots are a relic from Pokémon Mystery Dungeon : Blue Rescue Team. They contain mirrored portraits for some characters. However, the PMD:Explorers games all support mirroring the portraits at runtime. And so, it appears that most pokemon never use odd numbered slots. Even numbered slots on the other hand contain the portraits of the pokemon facing right. List of what each slots/pointers in a single pokemon's block are for. slot# Name Description 0 Standard Default pokemon portrait (Blue-Green BG) 1 - - 2 Grin Smiling pokemon portrait (Yellow-Orange BG) 3 - - 4 Pained Pained/Discouraged pokemon portrait (Blue-LightBlue BG) 5 - - 6 Angry Angry pokemon portrait (Read-Pink with sharp white line BG) 7 - - 8 Worried Worried pokemon portrait (blue-lightblue BG) 9 - - 10 Sad Sad/Disapointed pokemon portrait (blue-lightblue BG) 11 - - 12 Crying Crying pokemon portrait (blue-lightblue BG) 13 - - 14 Shouting Shouting pokemon portrait (Blue with light yellow "sunrays" BG) 15 - - 16 Teary Eye Teary eyed pokemon portrait (Light pink-darker pink BG) 17 - - 18 Determined Determined pokemon portrait (pink-red pink BG) 19 - - 20 Joyous Joyous/Very Happy/Ecstatic pokemon portrait (light yellow-yellow BG) 21 - - 22 Inspired Inspired/Admirative/Amazed pokemon portrait (light yellow-yellow BG) 23 - - 24 Surprised Surprised/Shocked pokemon portrait (blue-light blue BG) 25 - - 26 Dizzy Dizzy/(Spiral-Eyed) pokemon portrait (green to yellowish green BG) 27 - - 28 - - 29 - - 30 - - 31 - - 32 Sigh Sigh/Embarassed/Relieved pokemon portrait (light yellow to yellow BG) 33 - - 34 Stunned Stunned/Disheartened/Unnerved pokemon portrait (dark blue to light blue BG) 35 - - 36 - Wigglytuff YOM-TAH ! 37 - - 38 - - 39 - - Credits A big thanks to Zhorken, who figured out the compression format, the emotion name corresponding to each slots, and to what Pokémon each blocks of 160 bytes in the ToC was associated to!