tool Pokemon Mystery Dungeon 2 - Psy_commando's Tools and research notes

[psy_commando]

Alright. And that makes sense!

Also, I got the fixed pxcomp version : https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/ppmd_pxcomp_0.3.zip

I strongly recommend everyone updates as the previous versions incorrectly wrote the wrong compressed size in the AT4PX and PKDPX header.

I forgot to count the length of the header itself...

[evandixon]

I think the personality test is controlled by overlay 13. I haven't made any successful edits, but replacing it with overlay 35 makes the personality test not load, and when I changed a value that I thought was Skitty, it now flashes the "Choose the Pokémon you want for a partner" dialog. I'll play around with it more and see if I can find anything tangible.

[Edit]

I did something. I changed 3 values (one of them did nothing, then I changed 2 more), and now my partner is Grovyle.

https://dl.dropboxusercontent.com/u/2722532/current_18_7932.avi

https://dl.dropboxusercontent.com/u/2722532/current_26_8742.png

[Edit 2]

The data starts at about 0x1F4C. I'll post more findings as I get them.

[Edit 3]

It works with your player too, but for full effect, portraits will be needed to be added, and the str file in MESSAGE changed.

With me being a Luxio:

https://dl.dropboxusercontent.com/u/2722532/current_58_12765.png

(needs headshot and changed text)

https://dl.dropboxusercontent.com/u/2722532/current_58_13157.avi

(needs more animation)

[Edit 4]

My final data for the night (well, now it's morning for me ) is that there are 21 possible partner Pokémon, each stored in 2 bytes, starting at 0x1F4C, leaving the last occupied offset being 0x1F75, total of 42 (0x2A) bytes. I think the player data comes afterward, but I'll examine more tomorrow (well, later today). I speculate that there are offsets for each personality type for each gender, so some Pokémon will be there 3 times (partner, male, and female).

Edited December 7, 2014 by evandixon

[psy_commando]

Wow, awesome find !

I'll try taking a look later today too! I'm really curious about overlays. I've read about them, but I never really took the time to look at how they're actually made

I wonder if there isn't a way to redirect the pointer to all that to a new memory location, outside the WRAM, and cram a bunch more pokes in a separate array?..

Also, desmume support lua scripting and provide direct access to the emulator's RAM through the script interface.. Maybe there's something to be done with this? Like a button to load lists of pokemon on demand at runtime ?

And yeah, that animation glitch when the "lying down" anim should play is what I've come to call the "sleepwalking syndrome" XP

Even though all pokes have a sleeping anim, it defaults to the running animation when the player character is sleeping..

Its probably because the sleeping anim for playable pokes is much more elaborate, and got like 5+ frames for waking up and all! And the the standard sleeping anim is just a loop of 2-3 frames !

I haven't figured yet how come it defaults to that specific animation though..

Also, I fixed a couple of minor issues with the ppmd_kaoutils utility, and fixed the output path not working, along with fixing the insanely long allocation phase on startup! I was allocating a little too much memory, and a little too often I think.. Still nothing very alarming.. I also made some optimization and inlined a lot of functions! It still takes a little less than 3 seconds to export or repack the whole thing! But that's libpng and png++ slowing down the whole thing, according to the profiler..

I guess its not to bad considering nobody will batch process those!

https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/ppmd_kaoutil_0.2.zip

[evandixon]

Wow, awesome find !

I'll try taking a look later today too! I'm really curious about overlays. I've read about them, but I never really took the time to look at how they're actually made

I wonder if there isn't a way to redirect the pointer to all that to a new memory location, outside the WRAM, and cram a bunch more pokes in a separate array?..

Also, desmume support lua scripting and provide direct access to the emulator's RAM through the script interface.. Maybe there's something to be done with this? Like a button to load lists of pokemon on demand at runtime ?

Sadly, adding more partners isn't as simple as changing an offset in the overlay. Maybe it's possible in RAM, but that's beyond me.

Here's the order of the UInt16s. I hope to have a way to edit these in Sky Editor soon.

Bulbasaur
Charamander
Squirtle
Pikachu
Chikorita (F)
Cyndaquil
Totodile
Treeko
Torchic (F)
Mudkip
Turtwig
Chimchar
Piplup
Eevee (F)
Shinx
Riolu
Phanphy
Vulpix (F)
Skitty (F)
Meowth
Munchlax

0 (not an empty partner slot, probably just a separator)

Hardy
Torchic
Treeko (F)

Docile
Charamander
Bulbasaur (F)

Brave
Pikachu
Charamander (F)

Jolly
Totodile
Eevee (F)

Impish
Piplup
Charamander (F)

Naive
Chimchar
Skitty (F)

Timid
Cyndaquil
Turtwig (F)

Hasty
Shinx
Pikachu (F)

Sassy
Riolu
Totodile (F)

Relaxed
Phanphy
Vulpix (F)

Lonely
Bulbasaur
Mudkip (F)

Quirky
Squirtle
Piplup (F)

Quiet
Treeko
Chikorita (F)

Rash
Mudkip
Torchic (F)

Bold
Turtwig
Squirtle

Calm?

[psy_commando]

I mainly meant replacing the pointer in the overlay that points to the array containing those entries, with the address of another array we'd inject into the NDS memory.

Since overlays are basically a way to load "static" code and data structure into the WRAM, its possible to change values in there without caring about dynamic allocation issues and whatnot ! At least, from the ARM doc, that's what I understood of the overlays.

EDIT:

I missed part of the message..

But, can't you change the pokemon ID in the array ? I meant that, the lua script could change the pokemon code in that array while the game is running.

[evandixon]

I mainly meant replacing the pointer in the overlay that points to the array containing those entries, with the address of another array we'd inject into the NDS memory.

Since overlays are basically a way to load "static" code and data structure into the WRAM, its possible to change values in there without caring about dynamic allocation issues and whatnot ! At least, from the ARM doc, that's what I understood of the overlays.

EDIT:

I missed part of the message..

But, can't you change the pokemon ID in the array ? I meant that, the lua script could change the pokemon code in that array while the game is running.

I can edit the Pokémon ID's (even the gender), but I can't easily insert more IDs to the partner array.

The offset for the array isn't obviously in the overlay, so that's the end of what I can do.

What I can do is change the available partners (haven't tried removing one, only changing), and change the result of each personality type. For my sanity, I'll also look into how Grovyle91 edits the language file, because the results of the personality test are mentioned there too (hence the screenshot in edit 3 a few posts ago).

I'm working on adding a tab in Sky Editor to do this. In addition, I also plan to use your kaoutil to copy the standard portrait to every other portrait, so there won't be a weird absence of a headshot occasionally (again, see screenshot in edit 3 a few posts ago).

We're close to allowing for an actual ROM hack, with your allowing people to add more headshots, my allowing changing starting Pokémon, and possibly Nerketur's script editing (which is currently not entirely user friendly, but still amazing work). Add that to what TruePikachu found out about Pokémon data, and we've made significant progress.

[psy_commando]

I can edit the Pokémon ID's (even the gender), but I can't easily insert more IDs to the partner array.

The offset for the array isn't obviously in the overlay, so that's the end of what I can do.

I know you can't just add more after the previous entries. I just meant that we can probably mess with the code to get it to load something else instead of that list.

And, it would be odd to refer to data in an overlay from the outside.. At least, from what I understand.. I'll try running the no$gba nds debugger on it, and see what can be done with that !

What I can do is change the available partners (haven't tried removing one, only changing), and change the result of each personality type. For my sanity, I'll also look into how Grovyle91 edits the language file, because the results of the personality test are mentioned there too (hence the screenshot in edit 3 a few posts ago).

I'm working on adding a tab in Sky Editor to do this. In addition, I also plan to use your kaoutil to copy the standard portrait to every other portrait, so there won't be a weird absence of a headshot occasionally (again, see screenshot in edit 3 a few posts ago).

We're close to allowing for an actual ROM hack, with your allowing people to add more headshots, my allowing changing starting Pokémon, and possibly Nerketur's script editing (which is currently not entirely user friendly, but still amazing work). Add that to what TruePikachu found out about Pokémon data, and we've made significant progress.

Sounds good. Let me know if there are any issue with the tool ! I'd like to improve it as much as possible !

And I saw that issue plenty of times. I played through the entire game with a pokemon that wasn't meant to be a hero character ! I even recorded most of the cutscenes, so it might even help finding any things we'll need to watch out for !

Lets not forget Zhorken's script! Without that thing Idk if I'd have a decompressor / recompressor by now, and none of the face editing would be possible !

But yeah, we're getting close!

Just wondering though, do you think I should focus on the character sprites, the other loose image formats floting around for ui elements and etc, or the tilesets for the levels ?

Also, I'm going to spend some time on adding stuff to the wiki!

[evandixon]

Just wondering though, do you think I should focus on the character sprites, the other loose image formats floting around for ui elements and etc, or the tilesets for the levels ?

Also, I'm going to spend some time on adding stuff to the wiki!

I think that the images for the UI elements are less likely to be edited. After all, we still need to fix "sleepwalking syndrome"!

I finished editing the partners/nature results. Next I plan on working on things like kaoutil integration and possible editing the language strings (in the last test I saw "...will be a Riolu" when I became some other Pokémon on the beach).

It's interesting to note that choosing "????????" (ID 0) doesn't remove the available partner slot, and doesn't fail until entering the dungeon. The ??????? Pokémon shows as the first entity loaded on the map, or nothing if no entity was shown before. I would upload a video, but Youtube went too slow, and Desmume made very large video files, too big for drop box.

[Edit]

I can't seem to get kaoutil to work unless I copy the kaomado.kao file into the directory ppmd_kaoutil.exe is located in. I attached an image showing the full command I used. I'm next to positive all the files/paths exist.

Sky Editor overlay13editor.zip

Untitled.zip

[psy_commando]

Was there a white space in the path ?

EDIT:

Nevermind.. I found the screenshot..

That's strange..

Try running it on its own in a cmd window.

And then try it with the pokenames.txt and facenames.txt file in the same folder as the exe, and without the -pn and -fn switch.

This will probably help finding out which part of my argument parser is crashing.. It looks like its POCO throwing an exception, given all the exceptions it throws are "Unknown Exception" for some reasons..

EDIT2:

Does the folder "kaomado_unpack" exists ?

If it doesn't I think I know what's going on.

The output path is the parent directory where the file will be extracted to. The name of the resulting folder is the filename of the kaomado file.

I could change that behavior though. I just found it more practical this far.

[evandixon]

Was there a white space in the path ?

EDIT:

Nevermind.. I found the screenshot..

That's strange..

Try running it on its own in a cmd window.

And then try it with the pokenames.txt and facenames.txt file in the same folder as the exe, and without the -pn and -fn switch.

This will probably help finding out which part of my argument parser is crashing.. It looks like its POCO throwing an exception, given all the exceptions it throws are "Unknown Exception" for some reasons..

Dragging the kao file onto the executable in windows explorer doesn't work unless it's in the same folder as kaoutil. I'll try it in its own cmd window in just a bit, although I tried it without the fn and pn switches with no change. The text files are actually in the same directory.

Update on my end, I have a language text editor, so I can finally change the "will be a Shinx" message accompanying anything else. Like Dialga.

[psy_commando]

Dragging the kao file onto the executable in windows explorer doesn't work unless it's in the same folder as kaoutil. I'll try it in its own cmd window in just a bit, although I tried it without the fn and pn switches with no change. The text files are actually in the same directory.

Update on my end, I have a language text editor, so I can finally change the "will be a Shinx" message accompanying anything else. Like Dialga.

Yeah drag and drop won't work with most comandline program if there is a white space in the path provided. Unless you add "" around the path, but by default it seems windows doesn't do that..

That's a big problem because white spaces are the delimiter for command line arguments.

So there is no sane way to tell if its a path with a white space or if its 2 parameters

Especially if one parameter is a relative path while the other is an absolute one..

Have you seen my last edit btw ? :

EDIT2:

Does the folder "kaomado_unpack" exists ?

If it doesn't I think I know what's going on.

The output path is the parent directory where the file will be extracted to. The name of the resulting folder is the filename of the kaomado file.

I could change that behavior though. I just found it more practical this far.

[evandixon]

Yeah drag and drop won't work with most comandline program if there is a white space in the path provided. Unless you add "" around the path, but by default it seems windows doesn't do that..

That's a big problem because white spaces are the delimiter for command line arguments.

So there is no sane way to tell if its a path with a white space or if its 2 parameters

Especially if one parameter is a relative path while the other is an absolute one..

Have you seen my last edit btw ? :

The output directory exists. And I've put quotes around the parameters, which I think is what dragging and dropping does. I'll get more info for you later, I'm about to go to bed. I also have to debug why I can't debug my implementation of the text_e.str file.

[psy_commando]

I guess I should go to bed, I mixed everything up again :

The output path is really the path where everything gets extracted.. Not the parent of the folder where it gets extracted..

And, I wasn't able to reproduce the crash.

It would only crash if the output directory didn't exist. I tried giving two paths in two completely separate directories and got no issues.

EDIT:

I think I've found it. I put a trailing backslash in my output path while it was surrounded between double quotes, and it failed.

When there's a trailing backslash at the end of any paths you pass the program, it gets parsed as " instead of \. Because \" is the escape sequence for the " character.

I can't do much about it, because that's the argument I get from the main function's parameters..

Try not putting any trailing backslash. You can use slashes safely though, given they don't trigger escape sequences.

EDIT2:

Actually, I made a little hack, Idk if it might break something in the future, but lets try it anyways! I'm guessing others will probably do this mistake too in the future so, lets try to save some headaches to everyone.

I made it so the program detects a path ending with a trailing double quote, and just fix the path by itself, after nagging the user of course :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/ppmd_kaoutil_0.21.zip

Edited December 8, 2014 by psy_commando

[evandixon]

It works without the trailing "\".

I have code to insert all missing Pokémon portraits, using the standard one. I wanted to show that for this video, but I didn't realize that green Celebi has a grin face, and I don't want to make another video just yet.

In this video, I've successfully edited some text in the personality test.

https://dl.dropboxusercontent.com/u/2722532/current_44_13192.avi

[Edit]

Turns out the line endings in Windows (CR LF) are both being applied, so that's why the spacing is a little off. I also have more work to do, because the game won't progress past the point where my partner looks at the guild.

[Edit 2]

Turns out that the game not progressing was due to the cheat I was using to get a consistent personality test result. Seems to be working now.

I've attached the latest version. It includes:

-Editing results of personality test/partners

-Editing relevant text for personality test results

-Extracting/Importing Pokémon portraits, using ppmd_kaoutil

-Adding missing Pokémon portraits, using the standard face. There's no GUI for this, so to use this, load a ROM, use Debug -> Run Command, and type KaomadoPatch.

Sky Editor 12-8-2014.zip

Edited December 9, 2014 by evandixon

[psy_commando]

Alright. That's another problem out of the way !

Hopefully, I'll be able to work on sprites pretty soon..

On my end I've been stuck trying to figure out my way out of inheritance and templates, and throwing CRTP into it.. X_X

[evandixon]

I'm a little unsure what exact format headshots should be in when being inserted. I tried a normal PNG, and there was a black window for the portrait (with two grey pixels). I tried an indexed PNG with 16 colors and get the attached.

In case you ask, the underscore is there in the file system, it just doesn't show in the label for some reason.

[psy_commando]

I'm a little unsure what exact format headshots should be in when being inserted. I tried a normal PNG, and there was a black window for the portrait (with two grey pixels). I tried an indexed PNG with 16 colors and get the attached.

In case you ask, the underscore is there in the file system, it just doesn't show in the label for some reason.

I wrote the format in the readme.

Its 4 bpp or 8 bpp indexed, 16 colors. I had no issues in gimp creating one of those, but I don't know what you're using.

The grin image you attached seems to be in the right format..

And, I have no clue how you even got a regular PNG in there as PNG++ is set to only accept either 4bpp or 8bpp indexed images

The underscore is irrelevant, anything after the number in the name is ignored. I just put a name there to make it convenient.

You could name it "2_POTATO.png" and it would still work like it should!

Are you using version 0.21 ? And does it do this with all pokemon after re-packing ? Maybe I actually broke something..

EDIT:

I wasn't able to reproduce the issue on my end, everything gets put back in just fine..

Can you tell me what steps you went through ?

And did you try it manually, not through skyeditor ? (it could have to do with the way the program invokes it maybe ?)

You're not running this on a big endian machine ? (I'd really doubt it)

[evandixon]

I wrote the format in the readme.

Its 4 bpp or 8 bpp indexed, 16 colors. I had no issues in gimp creating one of those, but I don't know what you're using.

The grin image you attached seems to be in the right format..

And, I have no clue how you even got a regular PNG in there as PNG++ is set to only accept either 4bpp or 8bpp indexed images

The underscore is irrelevant, anything after the number in the name is ignored. I just put a name there to make it convenient.

You could name it "2_POTATO.png" and it would still work like it should!

Are you using version 0.21 ? And does it do this with all pokemon after re-packing ? Maybe I actually broke something..

Using 0.21, all portraits except the one I got from your screenshot work. I guess I just haven't encoded the png correctly. Once I figure it out, I may be able to make Sky Editor convert it.

[psy_commando]

Using 0.21, all portraits except the one I got from your screenshot work. I guess I just haven't encoded the png correctly. Once I figure it out, I may be able to make Sky Editor convert it.

What are you using for encoding the PNG ?

EDIT:

Just to be really sure I made another png from scratch, and imported it, but it seems to load fine

[psy_commando]

Just a quick update to let you guys know that my PC crapped out. So it might take a while before I'm back in action..

I tried to use system restore to fix a minor annoyance, and it decided to delete my NTUSER.DAT file.. And it turns out system restore had been completely ignoring the file for months.. So, I can't restore it !

I managed to make myself a new user profile and copy all my stuff that wasn't saved in the user registry, but now a lot of things work more or less correctly, so I might just have to format and reinstall windows, we'll see.. I know that visual studio didn't like it one bit, but I haven't tried compiling anything with it.. It just throws me error messages then starts.

EDIT:

Thankfully, I got VS to work again !

I got Netbean and Codebolck setup with mingw too in case anything goes wrong.. But I think I'll need to fix a couple of VS2012-only quirks with the code first..

Edited December 12, 2014 by psy_commando

[psy_commando]

Spoke too soon, Windows died, and I'm currently typing this on my work laptop.. I've spent the last few days making backups, and I'm re-installing everything from scratch. It has been barely a year since my last format+re-install ;_;

Hopefully, it shouldn't take me long to re-install my hacking setup!

And, evandixon, what happened with the png import issue you had earlier ? Do you still have it ? I'd like to know what's going on, so I can fix it.

[evandixon]

I haven't figured out how to encode the image properly or something. What program did you use to encode your portrait image you inserted?

[psy_commando]

I used Gimp 2.8.2 ( something in the 2.8.x anyways the one that doesn't have the screen refresh issues..) . You just have to make sure the image is set to indexed colors.

I think its in Image->Mode->Indexed. If its not in indexed color mode already it will let you either give it a custom palette, or generate one containing the amount of colors specified. Just be sure to set the "Maximum number of color" to 16, and it should work pretty well.

Then when exporting the png there isn't anything special to specify. Just be sure to use the "export as" menu item, not the "save as", because that's how gimp works.

It should work with any other program that supports indexed colors, but I have only tried with gimp, given I can't find anything that supports saving indexed colors pngs for some reasons.. I might just add indexed bmp support, given its really old, and probably more apps can handle indexed bmps..

EDIT: I got my PC back !

Edited December 19, 2014 by psy_commando

[evandixon]

Using this version of Sky Editor that has your tool integrated into it, I turn 0002_GRIN.png [the first attachment] into an indexed PNG [the second attachment], only 16 colors, using the exact version of Gimp as you are, and have the same problem. Because it works for you, I think it's either something weird with the original image I have, or maybe something off with the integration with ppmd_kaoutil (seems less likely).

Any ideas?

[psy_commando]

I tried the image you converted to indexed colors, and it displayed as a black box in-game at first.

Then I took a look at the insides of the png file and I noticed that there's something odd with the palette in the indexed image you posted. All my other images have 48 bytes of palette data in the PLTE chunk of the PNG image, but yours has only 33 bytes in there for some reasons.. And the color values don't seem to match. I verified and they're not just re-ordered differently. The bit depth and color types are exactly what they should be though.

So then, I tried to swap out the palette from your image with one from mine, and it worked! Its possible that the shorter palette is causing the program to silently skip the png, and leave an empty image in the resulting kaomado.. I think the issue must be that gimp, and other programs, don't find enough colors in the image to fill the 16 slots, and leave some black/empty.

Now, what's happening in my code could be that I'm validating the palette length at one point after png++ returns it and I might have accidentally flushed the exception.. I can't tell what PNG++ will return when I try to get the palette from an image with a palette shorter than 16 colors.. Because its 4bpp, it would implies 16 colors always.. But maybe PNG++ is returning a smaller color vector.. Its kind of an undefined case

I'll try to find out what's going exactly once I can get the thing to compile on VS2013.. I have to recompile the libraries, and for some reasons I can't get them to link properly...

Edited December 23, 2014 by psy_commando