[PMD WiiWare] AT7 File Container format

[Platinum Lucario]

AT7 File Container Format

Documentation for...

Pokémon Mystery Dungeon: Keep Going! Blazing Adventure Squad
Pokémon Mystery Dungeon: Let's Go! Stormy Adventure Squad
Pokémon Mystery Dungeon: Go For It! Light Adventure Squad

Or by it's other names...

Pokémon Mystery Dungeon: Forward! Adventurers of Flame
Pokémon Mystery Dungeon: Let's Go! Adventurers of Storm
Pokémon Mystery Dungeon: Aspire! Adventurers of Light

All on WiiWare~

Hello everyone, I've actually been making some progress with the research into the AT7 File Container format, here's what I have currently documented:

As of now, I've done some huge research into the AT7 Container format
and I have made some progress (finally). The AT7 format is split into
multiple segments and sectors. Each segment has a 6-byte long header, both which
indicate it's ID and size of the segment. AT7P counts the header, where as
AT7X does not. Like AT4PX or PKDPX, the segment headers have control flags in
it's header, but it is only determined within the first AT7P segment header .AT7P segments
have only been examined with compressed data sizes of less than 49,158 bytes. It is unknown
if the AT7P segments can go higher than 49,158 bytes

AT7X segments have uncompressed data, but can only be 49,158 bytes long.

Now here's some examples of where you can find the AT7 Segments for data2_WPAJ.bin:

Address	Type	Segment Size
0x0 - 42F8	AT7P	42F9 bytes (Dec: 17,145 bytes)
0x42F9 - 87C1	AT7P	44C8 bytes (Dec: 17,608 bytes)
0x11589 - 1D58E	AT7X	C006 bytes (Dec: 49,158 bytes)
0x1D58F - 29594	AT7X	C006 bytes (Dec: 49,158 bytes)
0x29595 - 3559A	AT7X	C006 bytes (Dec: 49,158 bytes)
0x3559B - 415A0	AT7X	C006 bytes (Dec: 49,158 bytes)
..
0x14AAD78-14BFFF0	AT7E	15288 (Dec: 86,664 bytes)

(This would keep going on and on, can't post all of the offsets
since it would take too long, but I am posting about AT7E though)

Segments

AT7P segments can be any size! The size of the segment is always stated at offsets 0x4-5
in little endian. Where as AT7X is always at a fixed size and have uncompressed data,
therefore meaning that offsets 0x4-5 will only count the uncompressed data, but not the
header. The entire AT7 file container is split into sectors, containing 131,072 bytes
(20000h) each. The AT7E segment always indicates the end of the archived files and then
the free space comes after the AT7E header.

Compressed data works as follows: After the compressed size bytes, it begins with
each of the blocks. Each block starts with a flag byte. As it goes from MSB to LSB,
if the bit is set, it reads, takes the next byte from the input and moves it to the
output completely unchanged. If the bit is clear, it reads a 2-byte big-endian
control code. The top nybble is a repeated string length of 3. The remaining bits
are a 12-bit negative offset of the most recent occurrence of the string to repeat.

 

AT7P Segment Info

Offset	Name	Type	Endianness	Size
0x0-3	Text (AT7P)	-	Big Endian	4 bytes
0x4-5	AT7P Length*	uint16	Little Endian	2 bytes
0x6	Compressed data begins	-	-	-

* - These bytes count both the header and compressed data

 

As explained before, AT7X segments are a fixed size of 49,158 (C006) each, counting the
header as well. Unlike with AT7P, AT7X segments have uncompressed data, which means that
the data can be read without any need to decompress. Since AT7X can only hold data sizes
of 49,152 bytes, any data contained has to be exactly 49,152 bytes. This can often be
used for textures or certain images or any data that has no data that can be re-replicated
in any way, but must be exactly 49,152 bytes within parts of large files.

 

AT7X Segment Info**

Offset	Name	Type	Endianness	Size
0x0-3	Text (AT7X)	-	Big Endian	4 bytes
0x4-5	Data size***	uint16	Little Endian	2 bytes
0x6-C006	Uncompressed data	-	-	49,152 bytes (fixed size)

** - It always has a fixed length of 49,158 bytes (C006h) each
*** - The bytes are always 00 C0 (C000h), which means it is always 49,152 bytes long in decimal. Unlike AT7P, these bytes only count the compressed data,  but not the header

 

The AT7E segment marks the end of the container and will fill any remaining bytes within a
sector with byte values of "45" (or "E" in ASCII).

 

AT7E Segment Info****

Offset	Name	Type	Endianness	Size
0x0-3	Text (AT7E)	-	Big Endian	4 bytes
0x4	Free space*****	-	-	-

**** - This segment always indicates the end of the AT7 container, before the free space
***** - These bytes can be any size, the entire AT7 container is made of blocks that are 131,072 bytes (20000h) each, and all free space bytes have a value of "45". The size of the free space will depend on how much space the last block has left over. For example, the last block of data2_WPAJ.bin only has AD7C (44,412 bytes) of data left in it 20000h - AD7Ch = 15284h (86,660 bytes), which means data2_WPAJ.bin only has 86,660 bytes of free space with each byte value of "45"

 

Sectors

Sectors are sections of the container that make up the file, like how sectors work on a Hard Disk Drive. Each sector is 131,072 bytes long (20000h). There can be multiple sectors within an AT7 container, for example, data2_WPAJ.bin contains 166 sectors (A6 in hexadecimal). But some contain as little as just 1 sector (such as data1_WPAJ.bin). If the last sector of the AT7 container has free space, the end of the data will be flagged with the AT7E segment, beginning the start of the free space.

 

Special thanks to:

MegaMinerd - For discovering and explaining about how the compressed data works and explaining that AT7X data isn't compressed

Edited September 12, 2018 by Platinum Lucario
Updated with information from MegaMinerd

[kkzero]

On 2/15/2018 at 6:31 AM, Platinum Lucario said:

The top nybble is a repeated string length of 3. The remaining bits
are a 12-bit negative offset of the most recent occurrence of the string to repeat.

Just to clarify what this means, the top nybble is representative of how many bytes will be copied over from the negative offset. If it is zero, three bytes will be copied to the current decompressed position, but if it is anything higher it will take three plus this nybble's value of the bytes. If the nybble were 2, for example, it would take five bytes.

Additionally, the negative offset, being the distance the string is behind from the current decompressed position, is determined by 1000 in hex minus the 12-bit value. 

I was able to find this out through a little programming trial-and-error of my own.