psy_commando

Good work! A few of the Bulbasaur are a little odd, but unless there's something like in .bgp that controls an x or y flip (which I don't think would help), it may or may not be garbage. It may be beneficial to look at this site if you haven't already. The misaligned sprites didn't seem to appear on that sheet.

Thanks, and yeah, those are most likely the ones that are wider than their height, and that's the reason why I named the thing "broken" because I currently have no way of figuring out the resolution, other than guessing from the total amount of pixels.

The wan files in the FONT folder don't work with this tool, while in the GROUND folder, the aXXX.wan files store footprints (extracted as 4 images), and all the rest appear to be part of dungeons, which don't seem to make it all the way through this program, probably because of a slight file format variation (or something you haven't seen yet).

That's an interesting find!

That means that maybe comparing those with the others sprite files that works might reveal what part of the file mentions format specific info !

EDIT:

I just thought about it, but, is there any particular error messages that pops up when the program stops midway ?

Anyways, here's a not fully working version of the sprite extractor for those interested.

Once we find out a sure fire way of telling the images' resolution, I'll make an "official" non-broken release and put it in the first post !

It works just like the last utility I posted here. Pass an input path, and optionally an output folder. Or just drag and drop a sprite file contained inside an SIR0

(Those are currently extracted with the extension ".sir0" by my extractor(but a lot of non-sprite files can get a ".sir0" extension!), but elsewhere in the rom's file, they have a ".wan" file extension, and I think I even saw another with a different file extension.. But the utility will extract frames from those as well !)

Oh, and the images are exported as indexed 8bpp PNGs. And the palette is included in every resulting PNGs!

There you go:

If there are any bugs, and if anybody finds anything about the sprite files, please lets us know !

ppmd_sprex_broken.zip

Nevermind everything I said.. I just realized I mixed up a couple of things..

And the reason the 16x32 images weren't coming out right was because I put the wrong variable in the wrong place..

However, I found a new problem, it seems image resolution for sprites isn't determine by the image's byte/pixel total size alone. Because I stumbled on some 32x16 sprites, and they come out wrong..

If you want I can give you a copy of the program in its current flawed state, so you can try it out for yourself ? Because I can't figure how we're supposed to find info on the resolution..

Alright, so most of the code works this far, but I can't find anything on Handling 16x32 sprites..

Reading those in YY-CHR with 16x32 as size seems to work, but when I try to emulate the 4x8 pattern it seems to use, I don't get good results at all..

Anybody knows of a good documentation on handling sprites like that ? Or maybe just have a tip to handle these ?

So is it possible to turn certain Pokémon Shiny? Like all female Pikachu as shiny.

Not quite yet. In PMD, shiny Pokémon are considered different Pokémon

We currently don't know how to extract Pokémon sprites, let alone change them. But we're making progress.

Well, we can extract the sprites, I already did. I got code that can decode the individual rle encoded sprite frames and re-encode them easily.

Putting them back together into a sprite file is indeed a little more complicated.. Given we have to figure out the whole sprite file format to insert modified images/frames back into the sprite file without breaking anything..

Looking at MAP_BG, I have absolutely no idea what the format for .bga and .bma is, and it's not possible to render the image without knowing that.

.bpl is the palette, the same format as in the block in .bgp, and .bpc contains the chunks/tiles (whichever term you prefer) in the same format as in the block in the .bgp.

As for the Pokémon sprite files, I can't tell where one block starts and another begins. Tomorrow, I'll take another look and re-read through your notes in case I missed something.

The wan files would possibly be easier to figure out if I had some idea of what it actually contained.

I can take a look at those bga and bma, but if I'd have to guess, one of those might just be data on whether one tile can be walked on or not, or something like that.

The sprites use a very particular structure. You have to follow the pointers in the header and sub-header to find anything at all in there. Bookmarks and the highlighter tool are your friends. The data is very similar in many place, but yet not for the same purpose. The wan file have the exact same structure as the pokemon sprites, they only differ in content.

But, yeah.. The notes for the sprites are really a mess right now.. I wish I didn't have to use vague letters to identify every parts of the file.. If there's anything unclear, let me know and I'll try to fix it/explain it better.

If you want a quick idea of what the wan files contain, just run them through yy-chr or tiledggd, the images will be a little deformed, but you'll be able to see a little better what they contain. You can also locate and extract the palette, using my notes, and load it to get a slightly better result.

I could also write a quick little utility to export all frames from a sprite file? Most of the code is there, but I didn't really see the point until I could figure out how to extract the rest of the data.

I guess I'll just code that utility regardless.. I've been cleaning up my, early, sprite handling code anyways..

Each 8x8 chunk/tile has its own palette, 16 of the 256 colors, starting at a particular offset, possibly different for each chunk/tile.

The MAP_BG files appear to have all the data in different files. I have no idea why. I'll have to look at it further, but each file is probably similar to the blocks in bgp.

That's interesting! So each tile has its own 16 colors palette ?

That works with the 4bpp format then!

It would have taken me a while to figure that one out

You seem pretty handy with image data! If you get the chance, do you think you could compare the character sprites from either "/MONSTER/m_ground.bin", "/MONSTER/m_attack.bin" or "/MONSTER/monster.bin" to the sprites stored inside ".wan" files? (there are a bunch inside the "/FONT/" folder)

Both of those are using the same structure, stored within a SIR0 container. They differ slightly, probably because the pokemon sprites are animated I'd say, but I can't seem to figure out at all what a big part of the structure in those is supposed to be there for..

The SIR0 header's first pointer points to a little 12 bytes chunk at the very end of the file.

And there, the first 4 bytes is a pointer to a part that contains other pointers to a bunch of "tables" and details about those.

And the next 4 bytes is a pointer to another part that contains pointers to the actual frames table and palette, along with a couple of other values.

I go into more details in my notes, but, this first part that doesn't seems to have anything to do with the actual images data/frames is the part I just, can't figure out..

I've known about it for a long while, but I can't figure out what's going on in there

But feel free to focus on something else.

I just think that, having someone else take a look at it could maybe help solve that mystery!

Nice.

Though, I wouldn't call those ".rawimg". Its just a generic extension I gave to images extracted from AT4PX containers.

Given the image format stored in those may be different from one another.

Though, if I can have more details on the formats, like from those extracted from BGP files, I could probably handle the file extensions better.

And what do you mean by 64 possible palettes ?

I thought it looked like a regular single 256 colors palette. Unless I missed something?

What is strange about this, is that the portraits aren't using the same format..

Why store the palettes before the AT4PX if they could put them inside with that file structure?

Anyways, I guess they're not always doing very logical things..

EDIT:

For some reasons the beginning of my message disapeared

Anyways, that's some nice finds!

I took a quick look to see if I couldn't recognize anything in there:

I took a quick look at an extracted bgp.

There seems to be a header going from 0x0 to 0x20 exclusively.

The value at 0x00 seems to be the length of the header, or a pointer to the beginning of the palette data.

The value at 0x04 seems to be the length of the palette data.

The value at 0x08 seems to be the offset from the end of the header where the actual image starts. or maybe just the offset where the image starts

The value at 0x0C is a really big value. but still within the file's size.. In the middle of a ton of zeros though..

The value at 0x10 seems to point to the first position after the palette.

The value at 0x14 might be the image's length maybe ?

In n_logo :

The palette begins at 0x20 and continue for the amount of bytes specified at offset 0x10.

Its easy to tell its a palette because of the 0x80 byte its 4 bytes per color, with the last byte unused. Most likely RGBX stored in little endian order.

One odd thing is that the size of the section after the palette to the image pointer seems to be 0x800, one of the value in the header. And that value is also very close to the apparent length of the image data section.

Just some quick observations.

Good to know ! I was slightly worried I might have broken something while cleaning up

I'm fairly sure the palettes for that image are stored somewhere else, probably in the FONT folder.

Also, they're most likely 8 bpp.

But that's pretty much all I can say without messing with the files myself!

I'll give Codeplex a try I guess. I'll look at it tomorrow.

And C++ is not all that complex syntactically, so I'm guessing you probably shouldn't have too much problem to read it. In the worst cases, I don't mind explaining some things! Plus, I tend to leave myself a lot of notes and comments

Its probably easier to understand than the Python script Zhorken made. Simply because Python has a complex syntax and isn't strongly typed like C and C++ are, which is handy when writing code, but not when trying to read it ^^;

But now that this is done, I gotta refocus on a single goal right now.. Not sure on what I should focus though. I could try finding out how to re-compress things, figure out the sprite data, now that we got access to every kinds of sprites around, or add support for the Kaomado file maybe.. And of course, there's the music and sound effects that I'm really close to figuring out, but I'd need to focus on it to have a decent breakthrough..

I actually was able to get decent results from extracting sound effects from the game. Samples are compressed using a variant of ADPCM. But I can't find anything at all anywhere that deals with ADPCM extraction from NDS files.. I really should stop using google search..

Though, I opened the samples using audacity, importing them as raw data, then picking VOX ADPCM, mono, little endian, and the sample rate of the sample is written on 2 bytes in the "wavi" chunk of the swdl files that contains the sample data. But it usually ranges from 32000 to ~16000. Most of the time its closer to 22050 though. But yeah, that yielded some decent results. Even though I can't replicate it using the same VOX / Diaphonic ADPCM decoding algorithm I got on the web

Sounds like you guys don't really need me on this. With access to what you already have, I may be able to poke around the ROM and see what stores what, but there's not much I could do beyond that.

Alright then.

Are you familiar with C++ at least a little ? Because I could send you my latest compilable working copy of my library if you want ? Its pretty embarrassing right now, but if it could help poke around the rom, it might just be worth it!

You should consider using some sort of source control for the ROM editor. I recommend CodePlex, as it supports both Git and TFS. If you use Visual Studio, TFS is very convenient. We could also use the documentation features to document what stores what in the ROM, as well as file structures. Github may also be viable, if you'd prefer that. Using source control, we could all collaborate on each project, or at least provide testing without the need for releases.

Right now I'm using a local repository with tortoise svn on my pc for my code, and I don't know what Nerketur uses currently.

And is codeplex/github free and not too capricious with licenses ?

Because those were the issues I had with those kinds of sites..

DeSmuMe already has code to allow directly loading the Arm9 and Arm7, but I'd imagine that it's those binaries that are expecting the files to be in the ROM itself. Maybe if we had the ROM stored in RAM? A hack-ish DeSmuMe may be able to do that, decreasing load and run time a little.

From what I saw, it can only load .elf executables. And we don't have those for pmd2, we only got the obfuscated .bin..

Well, the thing is, the rom data is accessed via requesting data at specific addresses in the rom from the gamecart. And then the gamecart sends the data in groups of 4 bytes at a time. So that makes it all really problematic..

I guess our best bet would be to edit the rom directly. But I don't know if I should focus on something else than PMD2 right now.

Update:

Also, I finally got the thing fully working, and tested. Let me know if there are any bugs!

Though, just to be clear, you need to feed this thing files that begins with either the PKDPX header, or the AT4PX header, without a single byte before it, or it will refuse to do anything!

https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/ppmd_unpx_0.1.zip

I'll update the front page as well!

You know, we're actually working on a rom editor too ^^;

Some kind of "IDE", to edit as much aspect of the game as its practical to do so!

I mean, its all good if you make your own, but there's most likely a way we can spare everyone having to "re-invent the wheel" so to speak!

We have a Skype chatroom we're using for discussing our findings and etc, and if you'd be interested to join it, I could probably ask Nerketur about it. It would probably make research and etc much faster!

Yeah, its really insane how "powerful"/far-reaching the script engine is in this game !

And, as for running the rom immediately that's indeed a must-have feature for any rom editor! Though, what would be annoying is the repackaging time..

I have been considering for a while making a hack-ish Desmume build that would load the game straight from the disk in extracted form. But, given the way the NDS works its kinda problematic..

There would also be the option of writing some code to simulate "mounting" a rom as a nitro-fs partition and being able to edit the content of the partition on the fly, and shrink and grow it at will, without having to repack the rom. Pretty much like a virtual drive. But I'm not sure how to do this exactly.

And yep, we noticed the link between the filenames of the scripts and the other files throughout the game data. Nerketur actually made me realize this a short while ago. He already knew about it, but, it was quite enlightening for me to say the least

And, I have absolutely no objection about anyone re-using my tools or source code! So feel free to do whatever you want with it all!

And also, I'm not just making a bunch of utilities, I'm also writing a library to handle pretty much any file format from the game, and export them to more common formats, and rebuild them all back into the game. Well, besides scripts, because Nerketur is way better at it and far-ahead than me!

And it will also allow editing the files while loaded in memory as well, to avoid potential wastes of time tied with having to extract stuff to disk everytime! It will probably be in .dll and .so form so that pretty much any code can run it! Though its far from release, but it might be something to consider upgrading to, from the individual little utilities I'm making available here, in the long run!

Anyways, I've written most of the code for the utility now. Just need to fix some errors, thanks to the MSVC2012 compiler being almost C++11 illiterate..

Wow, thanks, that's awesome!

It would have taken me ages to do that !

Especially since I haven't finished the episodes or post-game completely and wanted to do so before spoiling the music ^^;

Do you mind if I add that to my notes ? Leaving a mention that you did all the work of course!

And as for the missing tracks, I know that Blue Rescue Team uses a more common music and sound format, SDAT and SSEQ I think. So its possible that they're either stored elsewhere, or maybe they're re-using a single smd with all those songs in it, and just play a specific part of it ?

And the guy I'm working with, Nerketur Kamashi has managed to actually script an entirely new cutscene, over the intro cutscene I think, into the game a while ago!

He's really good and familiar with the scripting system of the game! So we actually have already "edited the rom" more or less !

And I wanted to wait until I got the re-compression worked out before releasing a tool, but I can totally just post a quick little AT4PX and PKDPX extractor in the meantime! That might even actually help me find potential bugs with it ! I'll try to get this done by tomorrow or later this week. All the code is already done, its just a matter of improving the error handling!

Its a little odd to put a container within a container though

But anyways.

Update:

The big breakthrough we had was that we finally figured out the compression on PKDPX and AT4PX files !

Its all thanks to someone named Zhorken really!

That person made a script for extracting portraits from the "/FONT/kaomado.kao" file, and it turns out, the portraits are compressed within AT4PX containers each preceded by their 16 color RGB24 palette.

So the code from that script allowed us to figure out that all AT4PX file in PMD2 used that compression scheme. But then it gets better.

I noticed that the PKDPX files had a largely similar header, with only an extra 2 bytes at the end. Those two bytes are actually the continuation of the bytes containing the decompressed file size. It is contained on 2 bytes in the AT4PX header, but on 4 bytes in the PKDPX header for some very odd reasons..

Well it turns out that the decompression code for the AT4PX files works flawlessly on PKDPX files as well ! (by adjusting for the extra 2 bytes in the header of course!)

I spoke with Zhorken, and they said it was ok to, well, learn from what's written there and make tools from what we learned from that script. So I wrote my own version of the algorithm in C++, and I'm going to try to write a compressor at one point, unless someone else would like to help with that! Given that, it looks like it would be really complicated to compress data exactly like they did.. And I never wrote any lossless data compression algorithm in my entire life

However, there is a way to implement a minimal version of that compression if all else fails. However, it would actually increase the size of the "compressed" file by its size divided by 8.

Essentially :

    CompressedFileSize = FileSize + (FileSize / 8)

Which is kinda inefficient to say the least ^^;

Anyways, here is the python script Zhorken wrote:

https://github.com/Zhorken/kaomado/blob/master/kaomado.py

(Just keep in mind that Python has many little particularities that makes the code do things that aren't very obvious for people not used to its syntax.)

And once I clean up and optimize my C++ code I'll publish the source too, along with another utility for handling PKDPX, AT4PX, and the kaomado.kao file.

Also, for now, until we figure out the actual name of the compression, I'll be referring to it in my notes and code as "PX compression", mainly given the 2 main file formats using it both end up in "PX".

I added this to my notes on dropbox about it(the dropbox version is the most up to date!):

(Just be wary of typos and also that I might have written something that's completely wrong somewhere in there ! So be sure to check the Python script Zhorken made as well ! )

====================================================================================================
                                     "PX" Compression Format
====================================================================================================
A custom compression method which appears to be a fancy version of an RLE algorithm, with some
touches from LZ compression added in I guess..

Its used by both PKDPX and AT4PX file format. It relies on 9 nybble flags referred to as 
"control flags" listed in those headers for handling a couple of special cases, and for the rest it 
basically works by decoding a "command byte" that indicates what to do until the next 
"command byte". 

Note that, those nybble flags are always "0x" where "x" is an hexadecimal value 
from 0 to F. Also, note that, these flags are computed on a file by file basis! They're 
"tailor-made" for each individual file.

To make understanding things easier, you could imagine that the bytes that we read are read from
a FIFO queue, and then that we put the decompressed bytes into a FILO stack. 
(Even if in practice those are probably too slow to use in this case!)

To decode the command byte, we simply check the value of all of its 8 bits, one at a time. 
From the highest bit to the lowest. 
So lets say we're in a loop that isolate the value of a particular bit each turn. Something like:

   uint8 mask = 0x80
   loop while( mask > 0 )
   {
       uint8 bitval = mask bitwiseAND cmdbyte

       ... // Handle the cases written below

       mask = mask >> 1
   }

On each turns of that loop, we'll check the value of the current bit, and depending on its value,
we'll decide what to do based on these criteria:

   1)  If the BIT we've isolated in the command byte is 1, then we pop the next BYTE from the input 
       queue, and push it as-is into the output stack!

   2)  Otherwise if the BIT we've isolated is 0, we'll pop the next BYTE from the input queue.
       Then, we'll try to find whether the high nybble of the BYTE we just read is the same value
       one of our control flags. 
       We also want to keep the value of the low nybble for later.
       We'll refer to the high nybble and low nybble we got here as "nbhigh" and "nblow"
       respectively.

       2.1) If one of our control flag match the high nybble of the BYTE we just read named 
            "nbhigh", then it means we're inserting a pattern of 4 nybbles / 2 bytes into the 
            stream. 
            We'll refer to the index of the control flag we got as "ctrlflagindex".

           2.1.1) If the index of the control flag "ctrlflagindex" is the first one at index 0, 
                  we calculate the value of the 2 bytes we'll push into the output stack this way:

                      byte1 = nblow << 4 bitwiseOR nblow
                      byte2 = byte1

                  We just insert those, and we're done! 

           2.1.2)  Otherwise, for any other "ctrlflagindex" flag index, we have a few other cases
                   to take into account.
                   However, they all have in common using the value of "nblow". 
                   We'll add a new variable called "basenybbleval" to make things easier to 
                   understand. And we'll put the value of "nblow" into it right away.

                   A)  If "ctrlflagindex" is 1. We increment the value of all 4 nybbles. Or in our
                       case, the value we use to store the base value of all nybble 
                       "basenybbleval".

                           basenybbleval = basenybbleval + 1

                   B)  Otherwise, if "ctrlflagindex" is 5. We decrement the value of all 4 nybbles. 
                       Or in our case, the value we use to store the base value of all nybble 
                       "basenybbleval".

                           basenybbleval = basenybbleval - 1

                   From this point forward, lets put the value of "basenybbleval" into 4 separate
                   variables, each holding the value of the individual 4 nybbles.
                   Lets just name them, "nybble0", "nybble1", "nybble2", "nybble3", and put the 
                   current value of "basenybbleval" into all of them.
                   (You should use an array, as they're only named like this for clarity's sake)

                   A)  If "ctrlflagindex" is between 1 and 4 inclusively. Then substract 1 from 
                       the nybble corresponding to the value of (ctrlflagindex - 1). 
                       For example, if (ctrlflagindex - 1) is 0, we subtract 1 from "nybble0". If its
                       3, we subtract 1 from "nybble3", and so on!

                   B)  Otherwise, we add 1 to the nybble corresponding with the value of 
                       (ctrlflagindex - 5). If (ctrlflagindex - 5) is 0 we add one to "nybble0" 
                        and so on!

                   Now, all that is left to do is to assemble the 4 nybbles into 2 bytes, and push
                   those 2 bytes into the output stack!

                       byte1 = nybble0 << 4 bitwiseOR nybble1
                       byte2 = nybble2 << 4 bitwiseOR nybble3

                   We just push those two into the output stack, and we're done !

       2.2) If none of our control flags match the high nybble of the BYTE we just read named 
            "nbhigh", it means we're copying a sequence of bytes from our decompressed output.

            To figure out the offset from the end of our output "stack" where we begin reading 
            the bytes sequence to copy, we'll need to read the next byte from the input "queue" 
            and do the following operations:

               uint16 offset = -0x1000 + nblow << 8 bitwiseOR inqueue.pop()

            "offset" will contain a negative integer value smaller than the current size of the 
            output queue. 
            Just go from the end of the output queue towards the beginning of it, 
            by the absolute value of "offset". This will get you the beginning position of the 
            sequence of bytes to append to the output later on. 
            We'll call this position "seqbeg".

            Then, to get the amount of bytes to copy starting from "seqbeg", we take the value
            of "nbhigh" and add 3. 

            Knowing this, we can now copy the sequence.


After going through those conditions, we know that the next BYTE we pop from the input queue will be
a new command byte guaranteed! So we just have to repeat this loop using this new command byte!

This sums it pretty much!

To sum it all up, the reason this is so huge is that, figuring this compression just opened up the door to editing nearly anything in the game files !!! We just need to reverse simple file structures now!!!

But yeah, we seriously owe a lot to Zhorken for that find! So, if anyone uses this code and gets the chance, be sure to say thanks to Zhorken !

EDIT:

I forgot those might actually be really handy to understand the above ^^;

==========================================================================================
                                   AT4PX:
==========================================================================================
AT4PX containers seems to contain image data in a compressed form. Namely, character portraits, 
map tiles, backgrounds. They usually don't contain their image's color palette however, as those
are often stored outside of the AT4PX container, either alongside, elsewhere, or not stored at all
in the case of a non-indexed bitmap image. 
(I haven't seen any bitmap/raster non-indexed images yet though, just speculating)


Offset:            Length:            Endian:      Description:
-------            -------            -------      -----------------------------------
0x00               0x5                -            Magic Number "AT4PX" {0x41, 0x54, 0x34, 0x50, 0x58}
0x05               0x2                little       (FLEN)Total compressed file length
0x07               0x9                -            List of control flags (more on these below)
0x10               0x2                little       Byte size of the decompressed data.
0x12               (FLEN - 0x12)                   Compressed image data.
[
   0x0                                             "PX" compressed data here.
   ...
]
-------            -------            -------      -----------------------------------
Total:             (FLEN)

Control Flags:
---------------
Those tend to change with each files. They're 4 bits values stored in the lower nybble of each 
individual 9 bytes that are used to handle some special cases while decoding some "Command Byte".
The high nybble is always 0.
They're picked on a per-file basis.

==========================================================================================
                                   PKDPX:
==========================================================================================
Generic compressed data container. 
Uses "PX" compression, just like the AT4PX format, except that PKDPX can contain another
file/container as its compressed data.
Identical to AT4PX besides the magic number, and the extra 2 bytes for the header!



Offset:            Length:            Endian:      Description:
-------            -------            -------      -----------------------------------
0x0                0x5                Big          Magic Number {0x50, 0x4B, 0x44, 0x50, 0x58} "PKDPX"
0x05               0x2                little       (FLEN)Total compressed file length
0x07               0x9                -            List of control flags (more on these below)
0x10               0x4                little       Byte size of the decompressed data.
0x14               (FLEN - 0x14)                   Compressed image data.
[
   0x0                                             "PX" compressed data here.
   ...
]
-------            -------            -------      -----------------------------------
Total:             (FLEN)

Control Flags:
---------------
Those tend to change with each files. They're 4 bits values stored in the lower nybble of each 
individual 9 bytes that are used to handle some special cases while decoding some "Command Byte".
The high nybble is always 0.
They're picked on a per-file basis.

Interesting.

Too bad I got a fully updated 3DS.. I even just got the game on the eshop, because of the sale

I'm guessing that FARC header contains some kind of table of content.

No problems ! I really gotta update the thread though

The most up-to date info is in the text files I linked!

Me and a couple of others have been working on reversing a ton of formats from PMD2, and we even had a huge, huge, breakthrough recently, which I'm going to write more about here, once I get the time.

And that's interesting ! But probably that the other formats must be wildly different no ?

How did you get to the actual content of the game though ? I mean, I thought 3ds games were encrypted or something like that? (I'm not even sure about that.. Just random things I've heard ^^; )

Update !

Its been a while since I updated this thread, but I didn't have much to show.. Mainly failed attempts..

I updated all my existing notes posted in here, and will keep doing so. But besides that, here's what I've been doing.

Pokemon Portraits:

Not much with the portraits, beside having been able to extract them and their palette from the emulator memory, to confirm resolution and little format specific details.

I also compiled Desmume myself, and inserted a couple of lines of custom code to attempt logging everything that was read from the gamecart, and dump it all to a file, but it wasn't all that conclusive. What I did was, saving a state at the companion select screen and then changing the cursor from one pokemon to the other, causing the pokemon portrait to change. And I reloaded the state later on when I needed to do more tests.

However, even by doing that, I couldn't find a link between what was being read, and the pokemon portrait.. The game seemed to be loading mainly bits of scripts, various numbers, but no obvious image data, or obvious PKDPX file content.

I also logged the rom offsets being read, and used that to identify what file was being read. But most of the time, the location that were read were very random. Which I suspect might be because there's a different starting offset for the raw rom, however, using tinke, its kinda hard to tell whether it takes the rom header into account in the shown offset for each file, or if the offset is relative to the raw rom.. I'll probably investigate this further later on.. But I pretty much burned myself out on that issue, so I went on to try to understand other aspects of the game..

Here's a differential patch with the logging code I wrote. Its a quick and dirty singleton to log things and group them together as long as each reads are adjacent in the rom. Note that, the log file will be overwritten at each emulator restarts. But feel free to change it however you want ! You have to apply it on the file in "../src/addons/slot1_retail.cpp" in the source folder for desmume, and then compile it. You'll have to uncomment line 227, for the logger to actually log anything. And be warned that it slows the game down somewhat.

Here it is :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/slot1_retail.cpp.debugcartreads.diff

(if you don't have winmerge, or tortoisediff or anything like that, you can just copy the code lines starting with ">" at the line number placed above each subsections. Lines beginning with "<" means lines that were replaced, so don't copy those! )

Music:

I also began taking a look at the music format for PMD2, to hopefully write a 2 way converter eventually.

It uses the SMD format.. And the NDS community doesn't seem to have much tools supporting it.. So I analyzed the files from scratch, and I was able to get a few conclusions and hypothesis.

It seems SMD files are very close to your typical MIDI files in overall structure! Thus by drawing parallels with midi, its been surprisingly easy to guess a lot of things from these files. The tracks are basically a collection of events, much like midi events but much more optimized to not waste any space! The shortest event can fit on a single byte, while the longest can fit on 4 bytes. Each events can have a delta-time prefixed, or not. And each note playing events seems to use a "delta-pitch"(probably in terms of the octave to play the note in) of some sort between the last note and itself.

Most of the SWD files that share the name of an SMD file basically contain data about the samples the game needs to load to play this track. It doesn't seem to actually contain the samples themselves. It probably maps notes to samples and set samples loop points and etc..

It looks like its possible to map several samples to an instrument, like the drum track in the bgm0003.smd file shows(its the last track before the eoc chunk, I think).

However, the bgm.swd file seems to maybe contains the actual samples. Given its size, and generic name, and content. So I'm guessing smd files can contain references to other smd files, and/or sound samples.. But that's not sure..

I made a separate file with my notes on the subject here :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/PMD2_MusicAndSoundFormats.txt

They're not very detailed yet, and a WIP. Like all my previous notes basically

But I was able to manually edit tracks and music files to play different notes, change instrument(sample), change the track wide pitch, insert silences, insert extra notes, change the tempo, remove tracks, add tracks, etc..

As long as you update the size value for the track, and for the whole file in the header, along with the number of tracks, and put the padding between tracks correctly, the game will take pretty much any music track you throw at it ! With a varying amount of success of course.. Especially, if you set the instrument of a track to an instrument not defined in the accompanying SWD file. That would result in a very silent song

You can also replace any music file, with any other music file, simply by renaming the .smd, and its accompanying .swd with the name of the target file !

For example I deleted the original bgm0002.smd and .swd file. (bgm0002 is the track for the top menu, the place with the "new game" and/or "continue" menu appears)

Then copied and renamed the bgm0003.smd and bgm0003.swd files as bgm0002. smd and bgm0002.swd. And then repacked the rom, ran it, and the music that played at the top menu was the training dojo dungeon music !

This is very handy for testing changes made to a song, given you can play it directly at the top screen!

Anyways that's it for now ! I hope this info will be useful ! As usual if anyone finds anything interesting or has any tidbits of info to share, feel free to do so!!

I could really use the help with anything really!

Alright, so I haven't made any progress regarding compression/mangling on PKDPX files this far.

But, I've been doing some more messing around and read a couple of articles on writing homebrew for the NDS. That proved invaluable in understanding and finding some of the things I was looking for!

http://www.dev-scene.com/NDS/Tutorials (those tuts gives so much details on how memory is mirrored and etc ! Its awesome !)

Namely:

- Sprites for characters are probably always multiples of 8 like 16x16, 32x32, thus why there is no information stored in the sprites for that ! Just by checking if we have 256 or 512 bytes we can tell whether an image is 16x16 or32x32.

- The pointer table thing after each frame's image data is some form of RLE compression, to save a few bytes.(basically, it saves space by storing that the same byte occurs X times at some place in the image, on a static 12 bytes instead of more. So ideally, you want to have more than 24 pixels with a similar color to use this[2 pixels per byte], or else you're losing space instead of gaining any! )

Also, I found out that there is a lot to learn from memory dumps made with desmume 0.9.9 (not 0.9.10 because there's a glitch with the dumper that makes it dump bytes until your HDD is full.. I learned the hard way! it made a file of 238 GB on my HDD under an hour, freezing the whole system with various out of memory errors and even leaving the task manager unable to load from the lack of memory! And it took 30 minutes before the system was even remotely responsive again )

And I mean that, instead of having to blindly look at assembly, the memory dumps will help figuring out how some of the data is stored, and what is the purpose of it, and what to look for in the actual files, or during execution !

Also, I found out a couple more things about the file format for the sprites. Many things that invalidate some previous hypothesis. I've updated the text file with the relevant data. I'm also trying to rename a few parts of the files to a meaningful name as I find what that section is for. So don't be surprised if some things aren't named the same way:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/CommonFileFormats.txt

I also attempted seeing a little better how the different pointer structures were working together. If anyone is interested in trying to understand how some of it is all linked without doing it themselves ! I used file #453 from m_ground.bin as reference.

Trying to figure out how these things work together!

============
DATABLOCK_G
============

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
G0         0x4E10         0x8   
[
  OFF     0x0            0x4            0x4D6C       Points to H0.
  LEN     0x4            0x4            0x8            Length array H0   
]

G1         0x4E18         0x8
[
  NULL
]

G2         0x4E20         0x8
[
  NULL
]

G3         0x4E28         0x8
[
  NULL
]

G4         0x4E30         0x8
[
  NULL
]

G5         0x4E38         0x8
[
  NULL
]

G6         0x4E40         0x8
[
  OFF      0x0           0x4            0x4DA0        Points to H1
  LEN      0x4           0x4            0x8           Length array H1
]

G7         0x4E48         0x8
[
  OFF      0x0           0x4            0x4DC0        Points to H2
  LEN      0x1           0x4            0x8           Length array H2
]

G8         0x4E50         0x8
[
  NULL
]

G9         0x4E58         0x8
[
  NULL
]

G10         0x4E60        0x8
[
  NULL
]

G11         0x4E68        0x8
[
  NULL
]

G12         0x4E70        0x8
[
  OFF      0x0           0x4            0x4DF0        Points to H3
  LEN      0x1           0x4            0x8           Length array H3
]

============
DATABLOCK_H
============

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
H0         0x4D6C         G0.LEN         
[
  0       0x0            0x4            0x0330        Points to I0
  1       0x1            0x4            0x0378        Points to I1
  2       0x2            0x4            0x03C0        Points to I2
  3       0x3            0x4            0x0408        Points to I3
  4       0x4            0x4            0x0450        Points to I4
  5       0x5            0x4            0x0498        Points to I5
  6       0x6            0x4            0x04E0        Points to I6
  7       0x7            0x4            0x0528        Points to I7
]

H1         0x4DA0         G6.LEN
[
  0       0x0            0x4            0x0570        Points to P0
  1       0x1            0x4            0x0594        Points to P1
  2       0x2            0x4            0x05B8        Points to P2
  3       0x3            0x4            0x05DC        Points to P3
  4       0x4            0x4            0x0600        Points to P4
  5       0x5            0x4            0x0624        Points to P5
  6       0x6            0x4            0x0648        Points to P6
  7       0x7            0x4            0x066C        Points to P7
]

H2         0x4DC0         G7.LEN
[
  0       0x0            0x4            0x0690        Points to Q0
  1       0x1            0x4            0x06E4        Points to Q1
  2       0x2            0x4            0x0738        Points to Q2
  3       0x3            0x4            0x078C        Points to Q3
  4       0x4            0x4            0x07E0        Points to Q4
  5       0x5            0x4            0x0834        Points to Q5
  6       0x6            0x4            0x0888        Points to Q6
  7       0x7            0x4            0x08DC        Points to Q7
]

H3         0x4DF0         G12.LEN
[
  0       0x0            0x4            0x0930        Points to R0
  1       0x1            0x4            0x09A8        Points to R1
  2       0x2            0x4            0x0A20        Points to R2
  3       0x3            0x4            0x0A98        Points to R3
  4       0x4            0x4            0x0B10        Points to R4
  5       0x5            0x4            0x0B88        Points to R5
  6       0x6            0x4            0x0C00        Points to R6
  7       0x7            0x4            0x0C78        Points to R7
]



============
DATABLOCK_I (unknown chunks of data over there)
============   

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
I0         0x330          0x48
[
]

I1         0x378          0x48
[
]

I2         0x3C0          0x48
[
]

I3         0x408          0x48
[
]

I4         0x450          0x48
[
]

I5         0x498          0x48
[
]

I6         0x4E0          0x48
[
]

I7         0x528          0x48
[
]


============
DATABLOCK_P (unknown chunks of data over there) At OFF_P
============

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
P0         0x570          0x24
[
]

P1         0x594          0x24   
[
]

P2         0x5B8          0x24   
[
]

P3         0x5DC          0x24   
[
]

P4         0x600          0x24   
[
]

P5         0x624          0x24   
[
]

P6         0x648          0x24   
[
]

P7         0x66C         0x24   
[
]


============
DATABLOCK_Q (unknown chunks of data over there) Is at OFF_Q
============   

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
Q0         0x690          0x54
[
]

Q1         0x6E4          0x54
[
]


Q2         0x738          0x54
[
]

Q3         0x78C          0x54
[
]

Q4         0x7E0          0x54
[
]

Q5         0x834          0x54
[
]

Q6         0x888          0x54
[
]

Q7         0x8DC          0x54
[
]



============
DATABLOCK_R (unknown chunks of data over there) Is at OFF_R
============   

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
R0         0x930          0x78
[
]

R1         0x9A8          0x78
[
]

R2         0xA20          0x78
[
]

R3         0xA98          0x78
[
]

R4         0xB10          0x78
[
]

R5         0xB88          0x78
[
]

R6         0xC00          0x78
[
]

R7         0xC78          0x78
[
]

Oh, and for those of you using HexEdit 4.0, the one from hexedit dot com, I made a wip template for parsing a few parts of a sir0 container containing sprite data ! Its very basic right now, and the entries are not very well labeled, but it completely beats parsing completely manually every new sprite files you want to look at !

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/_sir0.xml

Just drop that xml file into your "%APPDATA%/ECSoftware/HexEdit/" folder then start hexedit, go to the menu "Template->Open File Type->SIR0 File" then go again to the menu "Template->Split Window" to open the file navigation sidebar.

For the last step, click on "Template->Design Mode" if its already checked, given design mode won't let you navigate the file.

With this you can navigate a list of elements that were automatically recognized by the template. Just double click on the elements with the little blue boxes next to them, and you'll be brought to the location in the file where that particular element is located !

I'll try to keep this thread updated as I find new information!

Alright, so I made some progress again.

After trying to beat up DDD on linux and gdb-multiarch to work with the gdb stub from DeSmuMe and do all I wanted to, I decided to go with some simpler methods.

I was able to work together a decent toolchain for debugging the game while its running, using the NDS emulator iDeaS, because it got a barebone debugger built-in which helps a lot, and using Cheat Engine to set data watchpoints and to search the memory.

I've found out that sprite files seems to undergo some sort of conversion when they're sent into memory. They keep their header mostly intact. However, the magic number changes from "SIR0"{0x53, 0x49, 0x52, 0x30} to "SIRO" {0x53, 0x49, 0x52, 0x4F}..

The offsets inside the header+file structure are all changed, so they're not relative to the start of the SIR header, but to the NDS RAM. And the files are oddly not aligned on 16 bytes anymore.. Though, it could be due to the emulator..

And the sprite file I was using actually grew after being loaded/processed into memory, from 0x4F50 bytes to 0x5230 bytes.. Thus, it probably gets processed and stored into a new, bigger file structure, for some really weird reasons.. It kinda doesn't make much sense... Hopefully there must be a good reason for that..

All the sprite files from pokemon used in a level are all stored one after the other, each with its own SIRO header and structure(the header is exactly similar to SIR0, but not sure about the rest of the structure, I haven't had the time to seriously look at it). They're fairly easy to find using cheat engine's array search and copy/pasting the values from an entire frame from the sprite files on disk.

yeah, the overlays are loaded into the ds's memory, not mapped there from the rom. you can see where each is loaded with crystaltile.

Yep, thanks, I got that part.

But, I was mainly wondering if as the overlays are loaded into memory, they trigger some code to map files from the nitro fs to the memory, or do it by themselves? Because, I read on some website that mentioned that overlays somehow mapped resources to the memory.. But to be honest it kinda left me a little perplex..

Its hard to tell at one point what's possible or not on hardware you got zero experience with..

if you need to load your own or load others or something, you can hook some code and call the overlay load and unload functions that are sitting in the arm9 binary.

Well honestly, I'm not there yet

Would it be as simple as replacing some instruction with a jump, or even inserting a jump instruction, that would jump to some function I'd add at the end of the arm9.bin binary ?

Honestly right now, what I'm really after right now is to find the functions that load stuff from the file system, and more importantly, what demangle/decompress files. But, outside of examining every single instructions in the binary, I have no real idea of how to achieve that. And even then, assembly isn't that obvious.. Well, there's the iDeaS debugger, but, its not that powerful.. It does have a "run to cursor" option though.

Do you know of any documents / tutorials for doing that kind of stuff ? Or did you just figure out how it works yourself with some prior knowledge of ARM processors ?

I'm willing to work for getting more knowledge thoug! And if you could just point me in the right direction I could stop abusing of your kindness ^^;

Sounds good ! And yeah, the stuff in overlay 11 really looked that way, especially the "script.c" string

Overlay 11 looks almost like its what drives most of the game.. Well, at least, I guess..

Also, some of the others look like they're used for menus. like the buy menu, the bank, the storage, the swap shop, even the footprints mini-game. ARM is such a weird architecture

Btw, are you familiar with those ?

If so, would you happen to know if overlays actually map files from the ROM to the DS memory ?

Because, I read a couple of conflicting articles on that.. Some say you have to send commands to the ROM's register, to get the files, some others said that overlays map resources directly into the DS's memory.. Its all kinda ambiguous.

Update:

First, on the topic of the sprites. I need to figure out how their height and width are stored.. I can't find anything as of now in the vicinity of any frames in the files. But then again, I never really worked with sprites before, I had barely an idea what palettes were for 2 weeks ago, and I can't tell if its possible their dimensions are implicit through the way they're stored ..

Then on a more encouraging note, I did a couple of string search on the executable binaries, and on the overlays, using "strings -a -e S -t x" on linux, and I got some interesting leads!

Overlays:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/interesting_stuff_overlays.txt

ARM9:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/interesting_offsets_arm9bin.txt

In the arm9 binaries, these are really interesting:

 92ae8 AT4PXj

 94b00 %sfile = '%s'  line = %5d
 94b1c %sProgPos info NULL
 94b34   Print  
 94b40 !!!!! Fatal !!!!!
 94b58 (NULL)
 94b64 EFFECT/effect.bin
 94b78 MONSTER/monster.bin
 94b8c BALANCE/m_level.bin
 94ba0 DUNGEON/dungeon.bin
 94bb4 MONSTER/m_attack.bin
 94bcc MONSTER/m_ground.bin
 94be4 file directory init %4d %4d %08x %s
 94d00 0123456789

"AT4PXj" at offset 0x92ae8, is one of the weird headers we found in some files.

And those are all mentions of files containing sub-files we think are compressed (PKDPX):

94b64 EFFECT/effect.bin

94b78 MONSTER/monster.bin

94b8c BALANCE/m_level.bin

94ba0 DUNGEON/dungeon.bin

94bb4 MONSTER/m_attack.bin

And then in overlay_0011.bin, there are a ton, and I mean A TON of function name strings of some sort! It could have to do with the internal scripting maybe ? Or maybe they're debugging leftovers ?

Here are a couple of examples (much more in the text file !):

 3a944 WaitSe
 3a94c se_Play
 3a954 Destroy
 3a95c se_Stop
 3a964 me_Play
 3a96c me_Stop
 3a974 WaitBgm
 3a97c item_Set
 3a988 MoveTurn
 3a994 WaitBgm2
 3a9a0 bgm_Stop
 3a9ac flag_Set
 3a9b8 CaseMenu
 3a9c4 script.c
 3a9d0 CaseText
 3a9dc bgm_Play
 3a9e8 SetBlink
 3a9f4 bgm2_Play
 3aa00 performer
 3aa0c SetHeight
 3aa18 CaseValue
 3aa24 bgm2_Stop
 3aa30 BranchBit
 3aa3c BranchSum
 3aa48 CancelCut
 3aa54 CaseMenu2
 3aa60 SetEffect
 3aa6c BranchEdit
 3aa78 WaitRandom
 3aa84 se_PlayPan
 3aa9c WaitEffect
 3aaa8 flag_Clear
 3aab4 MoveHeight
 3aac0 sound_Stop
 3aacc se_FadeOut
 3aad8 JumpCommon
 3aae4 CallCommon
 3aaf0 WaitFadeIn
 3aafc SwitchValue
 3ab08 MoveSpecial
 3ab14 DefaultText
 3ab20 BranchDebug
 3ab2c PauseEffect
 3ab38 SwitchLives
 3ab44 BranchValue

Now, I just have to figure out where the compressed files are loaded, and I still have no ideas how, given all the stuff I thought I learned about the way the NDS works seems to not really apply to this game... I also have no ideas what are overlays exactly, and I still haven't found anything about them on the web, besides what I already know.. I also cannot figure out a way to get, say, DeSmuMe's disassembler, even using the GDB stub, to break upon landing on an interesting piece of code, like the above strings..

And no, I don't have access to IDA.. Just in case someone would bring it up here as well

Again, any help with this would be very welcome ! It would probably take me a long time to figure out how to guess what kind of compression is being used on those if I have to figure it out myself, given my non-existant knowledge of ARM/THUMB assembly/structure and lack of experience with the DS !

On an unrelated note, I don't know if double posting in our own thread is tolerated here ? I've seen several others do it, and I haven't found any mention of it in the rules.

Because, I'm afraid if I continue to edit my previous post with new things, it will become huge, and it might just force me to annoy the mods every times it flags the post for moderation when I add or change a link, like it did several times up to now.. Just to clear things up.

I looked it up. They are. (More or less.)

Oh well, I guess I don't have to worry about 32bits support now !

Thanks for the info !

Aren't 32bit versions of Dolphin discontinued now?

I doubt it.. But, honestly I mainly use dolphin 64 bits now anyways, so... I wouldn't exactly shed a tear if it was the case.. Less work for me!

Should try out Dolphin's dev/debug version, which can allow you to see the RAM at actual offsets.

With that you can edit a PKM's data; structure is a few posts above the linked ^

I posted a link to that thread in my post..

And I used dolphin in debug mode before, while making this and writing those notes. Without it I couldn't have converted the offsets properly. I even wrote about it in the notes I linked..

Also, I'm not sure if you've noticed, but I kinda added a good amount of new findings to the research those guys made.. And, the point of the script is to be more convenient and quicker than looking at raw data in memory.. And also being a good starting point for anyone looking to make something similar to what I did.

I'm not sure why you're posting that, though.. I'm guessing you're trying to help, but on the other hand, the way you brought it sounds kinda patronizing.. As if implying something like : "silly you, these guys have the only actual, good offsets, and the debugger is much better than your stuff" :,(

Of course, I'm over-dramatizing, and I'm not pissed, sad, or anything, but I'll admit it kinda stung a little when I read that.. :tongue:

Hello again !

I recently did a lot of work on PMD2, but before that I was working on a script to make some deep modifications to Pokemon at runtime, directly in the RAM of the emulator.

It got put on hold, given I use cheat engine as backend for the lua script that powers the whole thing, and the UI editor is very finicky and keeps crashing when I'm editing list views, and given I'm using a ton of them, its getting a little tedious.. I'm still trying to find out whether I can still use Cheat Engine as backend but used from another language such as C# or something where I can make a separate UI easily..

Its really not polished and it still has its share of bugs. And this far the actual editor can only list pokemon in the party, and in the 8 first pc boxes(I doubt its even possible to ever use more than that tbh..). However, it does list a lot of data that is invisible to the player, such as PID, SID, IVs, EVs, etc..

Right now, it works only on a x64 setup, x64 Windows OS + x64 dolphin. x64 Cheat Engine.. But I was in the process of implementing a detection for that and to use the proper addresses for an x32 setup. However, its really not that complicated to change the addresses in the script to their x32 equivalents..

PPXD Memedit

Research Notes:

Alright ! I found the table that points to individual frames and how it works !

I was able to manually rip the first frame of Bulbasaur's downward run anim !

I opened it in YY-CHR and its all there. Still not aligned, but its all there !

They're in what I've called this far the PointerBlockJ in my notes :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/CommonFileFormats.txt

Here's are the pointers to follow:

PointerA -> OFF_A

then from

OFF_A+0x4 -> OFF_A-0x10 -> OFF_J

OFF_J is the first pointer to a frame block in the pointer array I dubbed "PointerBlockJ" this far. The number of pointer in this array is stored at OFF_A-0x2!

Each pointer in "PointerBlockJ" points to a position near the end of a frame block. I refer to it as FRM_IN for now. That position may or may not contain another pointer to the beginning of the frame block, which is BEFORE FRM_IN.

And, there is 84 bytes of data AFTER FRM_IN that belongs to the frame block, containing data, and a pointer to somewhere in the frame block before FRM_IN.

I'm still trying to figure out what is the actual frame data from the rest though. But that's some nice progress !

EDIT:

And thanks for clearing up the terms. But, the one for sprite is kinda weird to me.. I did some 3D and we used sprite and billboard interchangeably for some reasons and it kinda stuck

EDIT2:

Scratch the 84 bytes of data after FRM_IN.. It can actually have a different size depending on the pokemon it seems.. I still haven't found out where data about the length and about what is after FRM_IN could be stored..

But, I made some pictures with annotations to show a little how the data is arranged !:

Here's what I call the "Master Pointer Table" for a lack of better name:

This is the first frame pointed by the first pointer in the PointerBlockJ above:

And after extracting that frame above into its own file and loading it in YY-CHR it gave this :

EDIT3:

Alright, it seems that, what is at/after FRM_IN is actually a list of pointers on 4 bytes pointing to actual image data inside the frame block's bounds, each followed by a size(?) on 4 bytes.

It seems those pointer might be there to "stitch" pieces of image together. Which would explain why some images are completely misaligned.. But that's not 100% sure yet.. However, I can hardly see what other purpose those pointer would have..

Here's how each pointer+length groups looks like :

C8150000 00010000 
|------| |------|
Pointer  Length

Some pointers can be null for some reasons, and still have a length right after them. But there must be at least one pointer after FRM_IN. Each of those blocks seems to be separated from eachothers by 4bytes of zeros..

Its also worth noting that in some cases, such as with Eevee, there are several more lines than needed, and several small values on 4 bytes scattered seemingly randomly between the pointer + length blocks and the FRM_BEG of the following frame block..

Moreover, we hit a wall that's gonna be a pain in the butt to get past.. The file monster.bin contains data about sprites, namely which ones to display from the m_ground.bin and probably m_attack.bin files. It also would maybe contain the portraits for every pokemons as well.

But, the issue is that every PKDPX subfiles are compressed, and its hard to tell which compression was used..

If anyone has any knowledge of PKDPX compressed files, or compression in NDS games in general, it would be great if you could tell us what you know about it !

Hmm.. Oh well, I guess it can come in handy later on then ! Doing Red and Blue rescue team as well could be interesting ! Especially given there are far more tools and existing data to go from !

A fun fact about PMD:Red Rescue Team is that some subfiles begin with a SIRO magic number ! Which is kind of an odd way to name them given the newer format is SIR0.. Its really easy to confuse both of them, given O almost looks like 0..

And, I'm thinking the data between tiles/frames is probably padding. They really seem to like aligning everything on 16 bytes ! And besides the SIR0 files that contains all the sprite data for all pokemon seems to rely on pointer tables to indicate both the size and location of frames. At least that what I think so far, but its still not 100% sure. It could in fact actually be the size of each tiles..

I'm still trying to figure out what pointer tables points to what kind of data.

If you ever want to take a look by yourself, here are some of my notes :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/CommonFileFormats.txt

I store everything on dropbox, so they're updated as I'm writing to them.

I also know someone from the PokeCommunity forums that is working on a script editor for PMD, and he shared some of his notes too :

http://www.pokecommunity.com/showthread.php?p=8248902#8248902

He's much more advanced than I am in this ! And its always useful to have a second opinion !

I've also got some old notes laying around from my findings using Crystal Tiles 2. Mainly tiles/frames offset, which might be handy in figuring out where is what ! (Don't take the dimensions into account too seriously, they're most likely not accurate !) :

NOTE: the following where taken from opening the whole thing in crystal tile2, so the offset probably need to be moved appropriately !

Bulbasaur world sprites(in crystaltile2):
- Running :
	- Down  :
		- frame1 at		  0x4544
		- frame2 at		  0x46B8 (first + 0x177)
		- frame3 at		  0x482C (second + 0x174)
	- Down Left Run:
		- frame1 at		  0x49A0
		- frame2 at		  0x4B14 (? the frame is somehow cut and doesn't align properly with the rest at 16 px on the height)
		- frame3 at		  0x4C68
	- Left Run:
		- frame1 at		  0x4DDC
		- frame2 at		  0x4F50
		- frame3 at		  0x50A4 (? trash appearing at the top left)
	- Up Left Run:
		- frame1 at		  0x5224
		- frame2 at		  0x5398 (? the frame is somehow cut and doesn't align properly with the rest at 16 px on the height)
		- frame3 at		  0x54EC
	- Up Run:
		- frame1 at		  0x5660
		- frame2 at		  0x57D4
		- frame3 at		  0x5948
- Taking hit ( 24 x 32 )
	- Down :
		- frame at		  0x5ABC
	- Down Left : 
		- frame at 		  0x5CA8 ( ? the frame seems to be cut in 3 places and misaligned on all those 3 ! )
	- Left :
		- frame at		  0x5E70 ? (too badly aligned to tell...)
	- Left Up:
		- frame at 		  0x6084
	- Up:
		- frame at		  0x6264

- Sleeping / Yawning ? ( 32 x 24 )
	- Left:
		- frame1 at		  0x6450
		- frame2 at		  0x6758
		- frame3 at		  0x68FC
		- frame4 at		  0x6A80

- Cheering ? ( 24 x 24 )
	- Down:
		- frame1 at		  0x6C30

- Looking ? ( 24 x 24 )
	-Down Left : 
		- frame 1 at 	  0x6DA4
	- Down Right:
		- frame 1 at 	  0x6F18

- Rolling ??? ( 24 x 24 )
	- Up:
		- frame1 at		  0x708C
		- frame2 at		  0x7200
		- frame3 at		  0x7374
		- frame4 at		  0x74E8

- Unconscious ? ( 24 x 32 )
	- Down:
		- frame1 at		  0x763C

- Cheering ? ( 24 x 24 )
	- Down:
		- frame1 at		  0x7814 (misaligned)

- Falling/Rolling ? ( 32 x 32 ) 
	- Up:
		- frame1 at		  0x79B4 (misaligned)
		- frame2 at		  0x7AFC (misaligned)
		- frame3 at		  0x7D1C (misaligned)
		- frame4 at		  0x7F60

- Knocked out ? ( 32 x 32 )
	- Left:
		- frame1 at		  0x8074 ( image split in 3 misaligned stripes )
		- frame2 at		  0x8254 ( image lower stripe misaligned )

- Yawning again ? ( 24 x 24 )
	- Down:
		- frame1 at		  0x843C
		- frame2 at		  0x

- From the side looking down ( 24 x 24 )
	- Left:
		- frame1 at		  0x86BC

Poochyena's World Sprites:(near offset 0x953DB8)
- Running ( 16 x 24 and 24 x 24 ? )
	- Down:
		- frame1 at				0x954050
		- frame2 at 			0x954134
		- frame3(24x24) at 		0x9541F8 (misaligned)
	- Down Left: 
		- frame1(24x24) at		0x954318 (misaligned)
		- frame2(24x24) at 		0x9544C4 (misaligned)
		- frame3(24x24) at 		0x954638
	- Left:
		-frame1(24x32???)		0x954758(misaligned)
		-frame2(32x32) at		0x954870(extremely misaligned)
		-frame3(32x32) at		0x954A88(misaligned)
	-Up Left:
		-frame1(24x32) at		0x954C48(misaligned)
		-frame2(24x32) at		0x954DF4(misaligned)
		-frame3(24x32) at 		0x954FA0 or 0x954F40(misaligned)
	-Up:
		-frame2(16x32) at		0x955114
		-frame3(16x32) at 		0x95522C

- Getting Hit (32x32)
	-Down:
		-frame1 at 				0x955408(misaligned)
	-Down Left:
		-frame1 at 				0x9555BC
	-Left:
	-Up Left:
		-frame1 at
		-frame2 at
	-Up:
		-frame1 at 				0x9559D4
		-frame2 at 				0x955C38

-Idle (16 x 32)
	- Down:
		-frame1 at 				0x955D4C
		-frame2 at				0x955E70
	- DownLeft:
		-frame1(24x32) at		0x956150(misaligned)
		-frame2(24x32) at		0x9562FC(misaligned)
		-frame3(24x32) at		0x956468

-Attack stance(16x32)
	-Down:
		-frame1 at				0x955F34

Also, now I'm not sure what to call tiles, sprites, frames, images or what without confusing anyone