tool Pokemon Mystery Dungeon 2 - Psy_commando's Tools and research notes

[TruePikachu]

Okay, right now my leading theory is that this Vaporeon is beta content. I don't see any sprite for sleeping.

Gonna see about dumping a thumbnail for each of the m_ground SIR0s

EDIT: Nope, sleeping is hidden _somewhere_

[psy_commando]

Its not beta. Its just that the way sprites are stored is really weird..

A lot of the animations for each pokes are in m_ground.bin, but a lot more are in m_attack.bin.

The sleeping sprites for most pokes are in m_attack.bin.

You have to unpack those and decompress the content into a regular sir0+sprite ".wan" file.

monster.bin also contains frames.. And all pokemons are at the same indexes in all 3 files.

[TruePikachu]

ah

I just finished "scrubbing" Vappy's m_ground, the only referenced and undocumented data is that stuff at the end. Gonna extract those other two files now.

EDIT: <___< So yeah, I guess I'll either be trying to get that Python script working, or I'm going to have to write a PX decompressor. Sounds like fun.

[psy_commando]

Why not use the util I wrote for that ? Just run it using wine.

https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/ppmd_unpx_0.2.zip

Just give it the file to decompress as parameter.

Or if you really want to write an util, just grab all my source code and do what you want with it.

In the case you really want to do it yourself, I tried to explain the decompression process in words in this : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/PX_Compression.txt

EDIT: Just fixed a lot of typos and rewrote a couple of sentences in that txt file. I shouldn't write those things really late at night I guess ^^;

EDIT2: Here's a direct link to the important part for decompressing px data : https://ppmd2utilslib.codeplex.com/SourceControl/latest#PPMD_UtilsLib/px_compression.cpp

Its an adapted version of the algorithm in Zhorken's python script. Its not really optimized, because I still don't understand enough about it to really go at it..

(don't mind the code too much.. some old comments weren't updated, and I had an obsession with functors when I wrote that)

Edited November 5, 2014 by psy_commando

[TruePikachu]

Got my own decompressor working, did a bit more sprite analyses.

Gonna have to get some sort of aniPNG library set up, as well as find out what is with frame times of >512 frames.

EDIT: Think I should see about getting the compressor working? I have the framework for it in my dePXer code, just isn't implemented. Since I understand the decompression algorithm, it shouldn't be _too_ hard to write a working compressor (an efficient compressor is a different story altogether).

[TruePikachu]

Okay, just confirmed that the /BALANCE/m_level.bin file handles stat growth at level up for each PkMn. I don't know how it is indexed exactly, but 0000 is Bulbasaur.

When unpacked, unSIR0'd (dd can strip the first 16 bytes), and uncompressed, you get 100 12-byte structures; the first DWORD in the structure is the number of experience points for the level 1..100, and the either BYTE or WORD just past it is how much the HP increases after levelling up. The numbers for experience match with my book, and the L1->10 HP growth, as well as the 90->100, matches as well.

EDIT: /BALANCE/monster.md has some information, like the Lv1 stats and mappings to sprite ID numbers, but it doesn't look like it contains learnsets or possibly the evolution tree. It is a collection of 0x44=68-byte structures:

+0x00 WORD  index
+0x02 WORD  family
+0x04 WORD  dex
+0x10 WORD  spriteID
+0x14 BYTE  mainType
+0x15 BYTE  altType
+0x20 WORD baseHP
+0x24 BYTE baseATK
+0x25 BYTE baseSPATK
+0x26 BYTE baseDEF
+0x27 BYTE baseSPDEF
+0x32 WORD unevolvedForm
+0x34 WORD exclusiveItems[4]

A lot of the enums (like items and types) are already listed in http://apointlessplace.net/wms/research/item_p_Sky.xlsx as part of the WMS research.

Bulbasaur♂ is at 0x0004C, and Bulbasaur♀ is at 0x09FAC. You also have such gems as Nidoran♀♂ at 0x007BC, but its likely just a placeholder for easier access via index numbers.

Edited November 6, 2014 by TruePikachu

[TruePikachu]

Further research on monster.md. Gonna just toss my stuff from IDA here...

00000000 PkMn            struc ; (sizeof=0x44)
00000000 index           dw ?
00000002 unk_02          dw ?
00000004 dex             dw ?                    ; enum DEX_ID
00000006 unk_06          dw ?
00000008 evolveFrom      dw ?                    ; Weirdness exists here with how indexing is done
0000000A evolveMethod    dw ?                    ; enum EVOLVE_TYPE
0000000C evolveParam1    dw ?
0000000E unk_0E          db 2 dup(?)
00000010 spriteID        dw ?
00000012 gender          db ?                    ; enum GENDER
00000013 bodySize        db ?
00000014 mainType        db ?
00000015 altType         db ?
00000016 unk_16          db 8 dup(?)
0000001E recruitRate     dw ?                    ; Tenths of a %
00000020 baseHP          dw ?                    ; base 10
00000022 unk_22          dw ?                    ; base 10
00000024 baseATK         db ?                    ; base 10
00000025 baseSPATK       db ?                    ; base 10
00000026 baseDEF         db ?                    ; base 10
00000027 baseSPDEF       db ?                    ; base 10
00000028 unk_28          db 10 dup(?)
00000032 family          dw ?
00000034 XItem0          dw ?
00000036 XItem1          dw ?
00000038 XItem2          dw ?
0000003A XItem3          dw ?
0000003C unk_3C          dw ?
0000003E unk_3E          dw ?
00000040 unk_40          dw ?
00000042 unk_42          dw ?
00000044 PkMn            ends

FFFFFFFF ; enum EVOLVE_TYPE
FFFFFFFF BASE_FORM        = 0
FFFFFFFF EVOLVE_LEVEL     = 1

FFFFFFFF ; enum GENDER
FFFFFFFF DUMMY_ENTRY      = 0
FFFFFFFF MALE             = 1
FFFFFFFF FEMALE           = 2
FFFFFFFF GENDERLESS       = 3

Data structures actually start at +0x00008, making the overall file structure be something like

struct MonsterMdFile {
   char     magic[4];   // "MD\0\0"
   uint32_t nEntries;
   PkMn     entries[nEntries];
};

The 601th entry is NULL♀; first is NULL♂. This +600 for female is consistant with the Wonder Mail S encoding, but I'm not entirely sure that it is a plain difference of 600, due to some mismatches in the 'evolveFrom' field.

Furthermore, I believe that move information is stored in the /BALANCE/waza_p.bin and /BALANCE/waza_p2.bin files. I haven't looked inside them yet, but I'd think the learnsets are in one, and the move stats in the other.

[psy_commando]

Woah, slow down !

That's a lot of information all at once !

Today I actually decided to take the entire day to write the compressor. I wrote all the code and just need to test the whole thing !

But I'm calling it a day, since I'm at a point where I might just begin doing silly things again..

And its great that you've managed to make some sense of the stats ! We already knew m_level.bin and monster.md contained pokemon stats related data, but never had time to deal with the format yet !

And that's a good lead you got with the wondermail codes !

We're so close to knowing nearly everything about this game now! Its kinda exciting

[psy_commando]

It took really long, but I finally got something working to compress files into the PKDPX and AT4PX format !

I ran binary comparisons on the result of decompressing files compressed with the tool and I got exact matches everytime..

But its probably still buggy, I haven't had as much time as I would have wanted to test it..

**removed link because the utility was broken**

Oh, and I don't know yet if the game will complain about those ! I haven't had the time to test that.. But I really doubt there will be any issues, given the resulting files are relatively similar..

EDIT:

I made a major derp, and there was an odd case where the whole thing would crash when reaching the end of the file.. I fixed it for now, but I'll have to look into it a little more.

Here's the fixed utility ( Also, no more dll ! That was another mistake on my part ^^; ):

https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/ppmd_pxcomp_0.2.zip

Edited November 10, 2014 by psy_commando

[evandixon]

Before I only posted VB code for converting decompressed BGP files to PNG. Now Sky Editor can do it. Just open an Explorers of Sky ROM (or maybe Time/Darkness, but that's untested) just like you would any save file. Navigate to a BGP file in the tree view, and you'll get to see the image. This release uses psy_commando's ppmd_unpx.exe to decompress the file.

https://skyeditor.codeplex.com/releases/view/137679

I would add a way to import a PNG and convert it TO a BGP, but I'm not sure how to encode the palette the way it's done in that format. With the palette for each tile being defined as an offset of the full palette, I'd have to put some serious thought into how to reverse the process. If anyone has any ideas (or already has a tool to do it), that would be great.

[psy_commando]

Before I only posted VB code for converting decompressed BGP files to PNG. Now Sky Editor can do it. Just open an Explorers of Sky ROM (or maybe Time/Darkness, but that's untested) just like you would any save file. Navigate to a BGP file in the tree view, and you'll get to see the image. This release uses psy_commando's ppmd_unpx.exe to decompress the file.

https://skyeditor.codeplex.com/releases/view/137679

Nice! Its good to see that thing getting some use !

I would add a way to import a PNG and convert it TO a BGP, but I'm not sure how to encode the palette the way it's done in that format. With the palette for each tile being defined as an offset of the full palette, I'd have to put some serious thought into how to reverse the process. If anyone has any ideas (or already has a tool to do it), that would be great.

I haven't touched BGP files yet to be honest. I remember you made some discoveries about those, but I wasn't sure of the specifics. Maybe you'd be better off by exporting all palettes as a huge standalone palette, and just output the image as RGB ? And then you'd have to generate a palette for each tiles when importing the image though. Or maybe you could just take a palette as input ?

Or maybe you could also just export each tiles as a separate image ?

I've been having my share of problems with image libraries though ! There doesn't seem to be a decent PNG library that is actually C++, works 100%, and is not some jumbled mess of structs, redundant types, and function calls.. And has some actual documentation..

PNG++ looked promising, but for some reasons, reading indexed png is not supported.. Or at least it seems so, because there is literally no documentation, and the whole thing crashes when de-allocating, even though everything went smoothly..

And messing directly with libpng is crazy complicated ! It looks like its meant to make you forget to de-allocate something and get memory leaks..

I'm considering just exporting everything to BMP using EasyBMP given its really as easy as the name implies

But idk if I should go with a legacy format like that..

[evandixon]

I've successfully (mostly) implemented converting back to the raw image, with the exception of me not knowing how to store all required colors in the palette. It should work on images that aren't as full of color as the ones in the game... Which defeats the purpose somewhat.

[Edit]

If I simply ignore the problem (replacing it with color 0 palette 0), then I get images that look like this.

Each tile has a 16 color palette, and there's 16 palettes, giving 256 colors. I haven't figured out how to figure out the palettes.

[Edit again]

I've figured it out. I have to make room on the palette, starting with the tiles with the most colors, then going in descending order. I've successfully converted png's to bgp's and back again. Time to try to recompress edited ones into the game to see if it works.

[Edit 3]

I've (mostly) successfully reinserted an edited image into the game!

However, when using ppmd_pxcomp.exe to compress then ppmd_unpx.exe to uncompress the newly compressed file, I get this error:

Unhandled exception at 0x77C9EBD8 (ntdll.dll) in ppmd_unpx.exe: 0xC0000374: A heap has been corrupted (parameters: 0x77CBC338).

I've attached a copy of the most recent version of Sky Editor. Be careful when re-inserting images. Each 8x8 tile can have one of 16 16-color palettes, so lots of colors in one place will throw an error.

Sky Editor 3.2 (With ROM Editor) Alpha 2.zip

Edited November 12, 2014 by evandixon

[psy_commando]

Hmm, that might just be part of the problem I was trying to fix earlier..

Could you pm me the compressed file, and the decompressed file causing the issue ? Or at least, give me the exact filenames that are involved in this ?

I think I might have an idea of what might be the issue. I think I'll also enable logging via a command line option for non-debug builds, that might help figuring future issues.

This usually happens when a sequence to copy wasn't properly encoded.. Or it could be the header not being written correctly for some reasons.

Sorry for not getting back to you earlier, but I'm getting random BSODs when I run any web browsers now, so as a result I check my messages less often

[evandixon]

I was going through my code to try to reproduce the error, and it may have just been me expecting a different filename.

Your programs accept an input filename and an output directory. When decompressing an unedited bgp, it gives the extension .rawimg, which my program is expecting. When decompressing an edited bgp, it gives the extension .wan, and because my program expects .rawimg, the display is outdated, and who knows what else is off. When testing now, I didn't encounter an error (using the same version of the executables), so it's hard to tell. Could you make an option to output to a given filename, so I don't have to predict the extension? It would make debugging for possible bugs easier.

In the mean time, the version of Sky Editor I uploaded can be used to reinsert bgp files. I included a picture of an edited n_logo.bgp in an emulator. It's a little off, but it's a good start.

If you're having BSOD's with browsing, you may need to check for bad drivers. Or, bad hardware. My old desktop had a BSOD whenever I did any intense graphics. Turns out that I blew all but one of the capacitors on my graphics card. I could see the fiber coming out and everything. I don't think a network card would do that, but it's worth a look if your drivers are up to date (and you're not using a laptop, which would be hard to take apart).

[psy_commando]

I really thought I had fixed that..

Anyways, I'll make it so you can name the output file as you want! It actually makes a lot of sense too..

Also, I'm planning on adding a batch compression/decompression option to the command line. Something like adding a "+" in front of additional files to extract/compress, like some program already do. But now I'm wondering how I'd implement that with the possibility of specifying the output filename..

And, that's funny that it would give it a .wan extension, because it only gives that to files that got a SIR0 header and a structure vaguely similar to a sprite file..

I'll have to take a look at the BGP file structure I guess...

And about the BSOD, I checked the video card recently and it had no visible problems. And they're not instant BSOD, they happen after maybe 10-20 TDR in a row happens. If I can close the browser quickly enough I'll avoid it. And I have no issues running games or anything else really.

The issue began when windows updated my drivers when I had explicitly specified to not updated my drivers.. I got a geforce 550 Ti and Nvidia has had very poor support for these cards.. So it could be the drivers, but I tried several times to rollback and even clean completely the drivers, using a driver cleaner, and installing the last version that worked. But nothing helped..

Since I switched from XP to Windows 7 I've had a lot of BSOD that came by waves. And I even built a brand new computer thinking there was something wrong with it, but nothing helped.. Plus, nowadays everything related to networking is laggy and spotty on this computer.. I even installed another wired network adapter not mounted on the motherboard and nothing changed. I even tried booting linux and I still get the same network issues.. I'm seriously stumped at this point..

[psy_commando]

Sorry for the delay ! I got the improved unpx working, but I wanted to be sure it wouldn't crash, so I made a lot of testing, and tried to bulletproof things a little more !

Here's the new version:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/ppmd_unpx_0.3.zip

It support adding several files to the commandline by using the "+" prefix for additional inputs. You can specify a single file's output filename as second parameter.

And when dealing with several input files, you can use the option "-fext" followed with the desired file extension to force a file extension for all outputed files !

The output path can be either a folder or a file in every cases. But when several files are to be handled, the output filename and extension will be ignored, and the input files will be outputed in the output path's parent directory.

I hope you'll like it !

Let me know if there are any features you'd like !

I also tried to eradicate all opportunities for heap corruption I could find. But I was wondering if you could tell me a little more what were the parameters involved when it happened ?

Heap corruption issues are really scary, so I'd like to get rid of those if possible

Also, I figured I was doing something wrong with png++ when loading indexed images.

Its undocumented, but if you want to load an indexed png the only way to do so is to do something like this :

       png::image<png::index_pixel> input;
       input.read( filepath, png::require_color_space<png::index_pixel>() ); 

I also modified the way I read data to structs, and upgraded to iterators. No more dirty reinterpret_casts ! I think I've even noticed a speed up since then ^^;

Next I'm going to standardize and clean up my codebase because it needs it very much!

And then I'll try to implement TruePikachu's findings, and attempt to make sprite files from scratch !

[evandixon]

Using that new version, Sky Editor can now view edited bgp files.

https://skyeditor.codeplex.com/releases/view/138901

At last, a tangible ROM editor! It can't edit much, but still, it's a great start.

Good work everyone!

[psy_commando]

Yep, we're certainly going somewhere !

I'm almost done with Kaomado file support, so soon we'll be able to add new faces to pokemon that were missing some if we want to ! And sprites are coming along too.

We still have to figure out those things though:

- Where are the dungeon templates/map and how do they work.

- Where are the non-randomized maps, and how do they work.

- Figuring out how tilesets works, and the way collision/material(water,lava,etc) data is stored.

- Figuring out if we modify the script if we can store script variables, extra data, and etc, in a savegame.

- Figuring out how to add script/level/etc..

- Figuring out how to add pokemon to, and modify the quiz at the beginning.

- Figuring out the sound format.

And etc..

[psy_commando]

Just leaving this here, but I won't be posting updates for probably the rest of the week, because I got Alpha Sapphire :3

(And now I wish I had a 3ds with the old firmware to decrypt the game files, and see for myself instead of waiting for the others to datamine the game, and tell us if Ohmori basically lied about hiding mega evos for the players to find.. )

[psy_commando]

Alright, so I managed to rebuild a kaomado file correctly !

But now, it seems I'm doing something very wrong to the images !

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/test_kaomado_12_1159.avi

I've seen something like this happen before, but I can't remember why..

Could it be endianess ?

[evandixon]

Looks like a combination of pallete and transform issues. If it's like a bgp, some tiles can be flipped across the x or y axis, which is specified in two bits of one of the values. The rest of that integer is the palette offset.

Hope that helps.

[psy_commando]

OMG, it works now !

It was such a clusterf**k to deal with, but damn it was worth it

Here's a build of it : https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/ppmd_kaoutil_0.1.zip

It output 4bpp pngs, and it uses 4bpp pngs to rebuild a kaomado. There is no readme, because I'm a little too excited

I found so many problem with the way the whole thing was written.. And I may have figured out a really sneaky bug with the px compressor. I'm going to run some more tests and I'll see if I can release an update to it !

Here's a screenshot showing the new portrait for the "Grin" emotion I added to poochyena! I used a cheat to make it my hero character. Its a quick crappy edit, but it prove that we can add new faces to pokemon that are lacking some of them and they'll be automatically used during the game! :

Here's a little video showing the other edits I made:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/example/test_kaomado_06_9214.avi

I replaced all of bulbasaur's face portraits with images with the name of the associated emotion written on it !

And here's an xdelta patch for the rom, to get my edits, for those who want to see for themselves! I also joined a dsv savegame for desmume, right after forcing my party to faint to force the game to reload the sprites and character data! (You won't need a cheat to get poochyena on your team with this savegame!) :

EDIT:

Some more details:

- The exact format for the portrait as PNG is 40 x 40, 4 bits per pixel, 16 colors.

- The tool only outputs entries in the table of content that aren't null, and contain portraits.

- The name for the folders, and for the "emotions" are stored in the "pokenames.txt" and "facenames.txt" file that come with the utility. You can modify them to your liking. The line number in the text file correspond to the number each folders or images is given ! The line number in the text files begins at 0, not 1, so keep that in mind !

- To add new entries, you just have to put it in the corresponding pokemon folder and give it a number as its filename. No need to give it a specific name, but if you want to, it won't mind as long as the name begins with a number between 0 and 39 (in base 10) ! For example, if I wanted to add a new face portrait for poochyena, say the "grin" emotion, I'd add a png in its folder, and name it "0002.png" or even "2.png". 2 is the slot in the portrait list for all pokemon for the "grin" emotion.

- You can even add new pokemon, by simply making a new directory in the root of the extracted kaomado's directory and give it a unique number as name ! The kaomado.kao file has by default 1,154 slots for pokemon, and only 652 are being used, a bunch of those are dummy placeholders too !

And here are some examples of the output:

test_bulbapooch.zip

Edited December 4, 2014 by psy_commando
added some more details!

[evandixon]

That's great!

Now I'm interested in changing the starter Pokemon in the ROM, not the save. Unfortunately, the location for that isn't strsightforward. Maybe future research can help find it.

Maybe the kao file will help you understand the in world sprites better.

[psy_commando]

I think that you might be able to change the starter pokemon modding the script for the quizz at the beginning ! We'll have to investigate the scripting more I guess. Nerketur has a lot of info on these in his notes!

And we'll see. To be honest, the kaomado format was way simpler than the sprite's !

But now I'm going to investigate as many similar sprite formats as possible, comparing those should give some more hints on the whole thing.

Also, I found something else that might be problematic. It seems that what the second ptr in the SIR0 header points at isn't random trash after all.. There's consistent padding bytes across nearly all files with a SIR0 header I've found this far !

I can't figure out what its supposed to contain exactly.. The size vary so much over each files. And some files have exactly the same values in there!

This might be a problem when rebuilding any formats contained in a SIR0..

Besides that, I found a new compressed image format !

Its called "AT4PN".

It has a very short header, only the magic word and then a size that could be the compressed size.. After that is the compressed data.. It doesn't seem to be compressed the same way.. And we got no control flags, which could mean they have static control flags maybe ? Figuring those out might be annoying.. If its even LZ compression again!

And I also figured out a good chunk of the WTE file format. The WTE files, in the FONT folder, seems to be containing the frame sprite for the text and UI.

Here are my current WIP notes : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/SIR0_WTE.txt

[evandixon]

I think that you might be able to change the starter pokemon modding the script for the quizz at the beginning ! We'll have to investigate the scripting more I guess. Nerketur has a lot of info on these in his notes!

If it is a script, it's not SCRIPT\S08P01A\quiz01.ssb, since replacing it with some random script yielded no change to the personality test.

Most of the scripts have their text resources in the ssb file, but the personality test stuff is in the MESSAGE folder, which may mean the personality test is elsewhere. There's a chance the test is controlled by an overlay, but I'll have to look more thoroughly later.