tool Pokemon Mystery Dungeon 2 - Psy_commando's Tools and research notes

[psy_commando]

So, I was wondering.

What would be the most important thing to research/invest time in out of those ?:

* Props sprites support for GFXUtil.

* Font support for GFXUtil.

* UI sprite/effects support for GFXUtil

* Dungeon tiles support for GFXUtil

* Item stats support for StatsUtil

* Random dungeon param support for StatsUtil.

* Music/Sound effects support with a new utility.

* Researching what each missing fields in the pokemon stats and items stats are for.

* Or something else ?

I'm lacking focus right now, and I need a clear goal to focus on and to help getting a sense of progress. But given everything I could currently do seems just equally important, its kinda hard to decide..

I'd also love to get more feedback if possible. Feedback is pretty much what keeps me going. Otherwise it feels like I'm just stumbling aimlessly in the dark, and not actually making anything useful..

[kleenexfeu]

So, I was wondering.

What would be the most important thing to research/invest time in out of those ?:

* Props sprites support for GFXUtil.

* Font support for GFXUtil.

* UI sprite/effects support for GFXUtil

* Dungeon tiles support for GFXUtil

* Item stats support for StatsUtil

* Random dungeon param support for StatsUtil.

* Music/Sound effects support with a new utility.

* Researching what each missing fields in the pokemon stats and items stats are for.

* Or something else ?

I'm lacking focus right now, and I need a clear goal to focus on and to help getting a sense of progress. But given everything I could currently do seems just equally important, its kinda hard to decide..

I'd also love to get more feedback if possible. Feedback is pretty much what keeps me going. Otherwise it feels like I'm just stumbling aimlessly in the dark, and not actually making anything useful..

Yop. I recently found an interest in genIII hacking so that's why I'm not getting news

To me, the most important in any game is the possibility to manipulate stats pokemon, and for mystery dungeon, there's one thing more that is missing in PMD2: being able to catch arceus. So, possibility to insert sprite/icon would be cool.

For the music/sound etc I don't care at all because I always played without sound x)

As you said almost everything is important so.. I would say stats Pokemon first

[Kajur]

Psy, can you work on the dungeon parameters next (once you finish the pokemon stats, that is). I think being able to create custom dungeons is something we all would love to do. But i suppose we would need to replace existing dungeons to do that (because of the guild jobs and the effort one would need to make them unlockable).

The one reason i pretty much don't play this game anymore (despite it being one of my favorite games), is because i got sick and tired of how repetitive all the dungeons on the game are. I even finished ZIS and went past floor 50 on Destiny tower twice (lost once to Kecleon and was unlucky with items in the second time around) and i haven't felt like playing for a while.

And Kleenexfeu, you shouldn't play this game without sound. The PMD2 soundtrack is easily the best of all pokemon games (most of them are subpar at best though).

Edited April 29, 2015 by Kajur
Fixed my Engrish

[psy_commando]

Yop. I recently found an interest in genIII hacking so that's why I'm not getting news

To me, the most important in any game is the possibility to manipulate stats pokemon, and for mystery dungeon, there's one thing more that is missing in PMD2: being able to catch arceus. So, possibility to insert sprite/icon would be cool.

For the music/sound etc I don't care at all because I always played without sound x)

As you said almost everything is important so.. I would say stats Pokemon first

I actually made something already to replace sprites, and there's another for portraits. Its called ppmd_gfxutil, and the other is ppmd_kaoutil. There isn't a user-friendly frontend for ppmd_gfxutil yet though, so you need to work with the XML to set animations and etc, but you can replace a pokemon's sprite entirely using that. I know that evandixon showed me a neat little app he made to view animations using the data exported with my tool, but I don't know if he's planning to do anything with that. If not, someone else might just make one, or I might, but I need to fully support the wan format first, because I can't really deal with having to rewrite that thing every times I find something new.

Also, you seem to like Arceus very much, don't you ?

And by pokemon stats, you mean researching what they do, right ? Because, if people want to help with those, that might go much faster given I'm kinda sick of doing trial/error with those in particular right now ^^;

Its not all that complicated, given all the unknown values are exported by my tool in the XML data.

Psy, can you work on the dungeon parameters next (once you finish the pokemon stats, that is). I think being able to create custom dungeons is something we all would love to do. But i suppose we would need to replace existing dungeons to do that (because of the guild jobs and the effort one would need to make them unlockable).

The one reason i pretty much don't play this game anymore (despite it being one of my favorite games), is because i got sick and tired of how repetitive all the dungeons on the game are. I even finished ZIS and went past floor 50 on Destiny tower twice (lost once to Kecleon and was unlucky with items in the second time around) and i haven't felt like playing for a while.

And Kleenexfeu, you shouldn't play this game without sound. The PMD2 soundtrack is easily the best of all pokemon games (most of them are subpar at best though).

Of course ! Custom dungeons is something I really want to enable as soon as possible. And, there is a possibility you wouldn't need to actually replace dungeons, the game is very modular in design, and dungeons seems to depend heavily on their script, which is much, much easier than having to deal with hard-coded stuff and ARM byte code.

But, from what I know right now, in order to make dungeons fully editable or add custom ones I need to figure out:

- dungeon.bin. That file is so weird. Its a big pack file with tons of different types of files piled up arbitrarily. But figuring out what is what, and why its organized that way is the tricky part.

- The tile format used for the dungeon tiles.

- The sprite format differences for props. It uses mostly the same layout as pokemon sprites, but some differences make my utility crash or fail to extract/rebuild them properly. Mainly because I can't find where the image resolution for those is, and its hard to figure out just from the amount of pixels what format they're in.. I'm suspecting the resolution is possibly stored in the file that indicates where each props is positioned on the map, but I only got a vague idea where it could be.

- the mappa_* files in the balance folder. The people on the Gamefaq thread figured out a bunch about those, but there is still much unknown.

- The way the script engine work and interact with levels/dungeons. (Though, Nerketur figured out most of it already, which helps a lot)

So that might take a while

I just wish I could find someone willing to do repetitive trial and error stuff. Like corrupting individual files with random bytes and finding out when the game crashes/glitches..

And then, most likely that having an actual graphical editor for maps would help a lot. But a map editor like that is in itself is a huge project! (That's why I'm planning on just exporting all that data into XML and images so that anyone can use any language or script language(even web-based !) to make an actually good map editor.)

And I second that about the soundtrack ! Its awesome ! Though I still like a good chunk of the other games' soundtracks.

[Dialga22239]

How I can make the swd/smd to a MIDI or How I rip it rip? I'm not the best in rip stuff

[evandixon]

How I can make the swd/smd to a MIDI or How I rip it rip? I'm not the best in rip stuff

Unless there's a more recent version I missed, the best I know of is linked to in this post: http://projectpokemon.org/forums/showthread.php?40199-Pokemon-Mystery-Dungeon-2-Psy_commando-s-Tools-and-research-notes&p=192061&viewfull=1#post192061

[Dialga22239]

Thx for the program But where I can change Input/Output file? You Tube don't has a Tutorial...

If that not work, can someone from you give me the MIDI files of PMD 2 with the Intro theme of EoT/D?

I have Win 8 if you want to know it

Edit: I got the MIDI files from my friend Uhm has someone the DLS or the SF2 files from the game?

Edited May 9, 2015 by Dialga22239

[psy_commando]

Idk if posting those would be against the rule here or not.. So I suggest you look around the pokemusicverse forums instead, without being too specific.

Besides that, I'm having a hard time doing any work these days on anything really.. Not sure why.. Possibly because I've been feeling pretty bleh lately. And as a result, I got a lot of catchup to do in every single ones of my hobbies..

But anyways. I got a question I'm trying to find an answer to..

Does anyone here have any experience with palette sets ?

Because, apparently that they're using something like that for the dungeon tiles. I can't seem to figure them out though.. Its probably really obvious too..

How do you know what palette/color to use for what pixels ?

[evandixon]

In the bgp format, there's a section that tells what 16x16 (or was it 8x8?) tile goes with what portion of the screen. There's also a few bits that are the offset of the palette.

So there's 64 colors in the overall palette. Each tile can only use 16 colors. The offset bits say to use 16 consecutive colors after skipping the first x colors in the overall palette.

I'll try to make time today to look at my source code and document this better.

[Edit]

Here's a rough wiki article on it. If you have any questions, let me know.

http://projectpokemon.org/wiki/Pmd2_BGP

Edited May 10, 2015 by evandixon

[psy_commando]

Thanks !

It seems the map tiles are not using the exact same method though. But that's a good lead.

The map tiles files I was looking at are split into 3 files. bpc, bma, and bpl.

BPC contains the actual tileset. BPL contains color palettes . And BMA contains possibly info on what color palette to use where, I'd say..

And it seems not all the tiles are in there, but only filler type stuff, depending on the name. Things like seamless rocky walls tiles and etc.. Other than that, there are a lot of duplicates !

I guess most of the map tiles must be in that messy "dungeon.bin" file then

And I guess I could probably also implement BGP import/export support while I'm at it.

*EDIT*:

Speaking of the "DUNGEON/dungeon.bin file" I analyzed the content a couple of weeks ago, and I still can't figure out how to display the images in there..

Its hard to tell what palette goes with which image, and if I'm even displaying them in the right format..

If only I could just find a tileset from the beginning of the game, I'd have a good way of figuring out the format by selectively corrupting it..

Here are my notes : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/Dungeon_bin.txt

(The latest packfileutil also properly give a file extension to most of the content of unpacked file btw. If anyone wants to take a look for themselves.)

[evandixon]

Have you played around with the debug menu in Explorers of Time/Darkness? There's a map visualizer that may come in handy for this.

I've attached a picture of D00P01 as shown by the debug menu. I've looked at some of the others and for each map that's not an actual dungeon, one of these maps exist. It may be harder to reverse engineer this format because not only is there (probably) collision data, there's also some animated frames.

Looking at the bpc file, it looks like there's some extra data starting somewhere at the bottom that doesn't store image data (it doesn't display as nicely in YY-CHR). Maybe it's collision data, although I wouldn't know what that looks like.

In case you don't have it, here's the Debug Menu code for Explorers of Time:

0202A09C A3A01000
92FFFFA8 FE7F0000
0202A09C E3A01008
D2000000 00000000

The source wherever it was said "R + Down for Debug Menu", but I never have to hold R, just scroll down to the bottom of the main menu.

[psy_commando]

I didn't even remember there was a map viewer in the debug menu ^^;

But thanks for mentioning it ! This should make things easier, especially if EoS shares the exact same format!

And honestly, we figured a good chunk of the wan format(Except those damn orphaned frames..), which is probably the most complicated format in the game, so it shouldn't be too hard to deal with those in comparison.

Selective corruption is your friend

And from looking at the files, it seems they come in groups of three. 1 bpl, 1 bpc, 1 bma. But it also seems some have a bpa file instead of a bpc. Maybe that the 'a' is for animated ?

I also noticed that sometimes the header of the bpc/bpa files gets longer..

[Jellynater]

I did something for IQ skills!

Here is the document.

When putting this together, I noticed a pattern between the byte a skill affects, and the order each IQ skill appears in text_e.str. I checked several skills by hand before following the pattern

The italicized skills are weird. Nothing truly is nothing; there is no skill there. PP Checker is a remanent of Rescue Team, and the rest I have no idea about as they don't appear at all in Rescue Team or Shiren the Wanderer.

Do note that all RAM addresses are relative to the hero pokemon, or the leader in dungeon

[psy_commando]

I did something for IQ skills!

Here is the document.

When putting this together, I noticed a pattern between the byte a skill affects, and the order each IQ skill appears in text_e.str. I checked several skills by hand before following the pattern

The italicized skills are weird. Nothing truly is nothing; there is no skill there. PP Checker is a remanent of Rescue Team, and the rest I have no idea about as they don't appear at all in Rescue Team or Shiren the Wanderer.

Do note that all RAM addresses are relative to the hero pokemon, or the leader in dungeon

That's a pretty nice find !

Btw, have you ever stumbled on the type effectiveness table, or anything of the sort by any chances ?

[Jellynater]

That's a pretty nice find !

Btw, have you ever stumbled on the type effectiveness table, or anything of the sort by any chances ?

I have not, I'm sorry. Do you have any guesses as to where it may be?

[psy_commando]

I have not, I'm sorry. Do you have any guesses as to where it may be?

I talked with someone earlier this week who found it in Pokemon and Colosseum by looking in memory for references to the index of the strings for the type names in the text_e file. I haven't had the time yet to try it on my own, but it sounds like a good way of finding it.

[Jellynater]

I looked into text_e.str, and promptly got distracted.

So, I now have a list of every control code I could find in that file.

I also looked through the entirety of the SCRIPT folder because I could.

https://www.dropbox.com/s/3s9a2en7at0csxp/Control%20Codes.txt?dl=0

Also, [CS:%var%] changes the color of all printed text between it and [CR] according to the hand-checked variables

here.

For example, if we replace %var% with the character E, The line above will print as...

[color="#FFFFFF"]Also, [/color][color="#00FFFF"]changes the color of all printed text between it and[/color] [color="#FFFFFF"]according to the hand-checked variables here.[/color]

Inputting numbers do nothing, and using whatever character is before "A" crashes the game.

Finally, The RAM address 0x217B107 is friendly. That is the location of the first message in the message log.

From there, you can edit values in Desmume's memory viewer and see changes you made from a quick L+B.

[psy_commando]

So, I've been messing around with the SWD files, and I've figure out a lot more about the format. I can extract individual samples, and etc. And I updated my notes :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/PMD2_MusicAndSoundFormats.txt

Though, I still haven't found the proper ADPCM implementation they're using. Its possible its the official Nintendo implementation, since I noticed calls to the SWI ADPCM decoder. But I don't know how it works exactly.

Using the standard IMA ADPCM tables and implementation yielded pretty quiet and deformed samples..

Not all samples are ADPCM, some are PCM 16 bits. Its indicated in the sample table by a flag value.

Also, the swd files that accompany each bgm tracks refers to samples stored in the main bgm.swd bank via their location in a pointer array. Its hard to explain, but it might be clearer upon reading my notes.

I looked into text_e.str, and promptly got distracted.

So, I now have a list of every control code I could find in that file.

I also looked through the entirety of the SCRIPT folder because I could.

https://www.dropbox.com/s/3s9a2en7at0csxp/Control%20Codes.txt?dl=0

Also, [CS:%var%] changes the color of all printed text between it and [CR] according to the hand-checked variables

here.

For example, if we replace %var% with the character E, The line above will print as...

[color="#FFFFFF"]Also, [/color][color="#00FFFF"]changes the color of all printed text between it and[/color] [color="#FFFFFF"]according to the hand-checked variables here.[/color]

Inputting numbers do nothing, and using whatever character is before "A" crashes the game.

Finally, The RAM address 0x217B107 is friendly. That is the location of the first message in the message log.

From there, you can edit values in Desmume's memory viewer and see changes you made from a quick L+B.

Tags are pretty interesting. Have you tried each of those tags to see what they do ?

I tried out a bunch of them in the past and figured what some of them did : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/text__str.txt (the LS tag is really interesting)

And you can even use "ppmd_statsutil" to convert the text from the .str file into a .txt file and edit the text and then use it again to convert it back into a .str file.

[psy_commando]

Hey ! I made a lot of progress with the sounds files ! I finally realized that the ADPCM samples are actually really encoded using the standard IMA ADPCM format.. >_<

I was just handling the samples in the wrong order..

To celebrate, I'll put this little extremely barebone utility that can dump both pcm samples and adpcm samples from SWD files from most games that uses them that I know of

Just pass it a swd file that has samples in it, as parameter or via drag and drop. It will dump the wav in the same folder. (its not really meant to be practical)

I also tried to write the loop points in the wav file, but I'm not sure how they work yet..

Anyways, here it is :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/pswdldump_0_0.zip

I'll write a real utility eventually.

[bananagram]

Do you think these tools could be used to extract all the image data from PMD2 efficiently? Many of them are missing from the internet.

If all the move sprites were named and the sounds were ripped and associated with them, I think a complete clone of the game could be made. This would allow a fangame to be made, where the gameplay is expanded or a new story is written. I don't know if I could do that, but it would be cool.

[psy_commando]

That's pretty much the idea..

But there are still formats that we haven't figured out yet.

And with the tools you could also just make a romhack of the game, instead of writing one from scratch.

[evandixon]

Do you think these tools could be used to extract all the image data from PMD2 efficiently? Many of them are missing from the internet.

Out of curiosity, which images are you referring to?

[bananagram]

Out of curiosity, which images are you referring to?

A massive number of tiles and almost all the move animations. Some of the additional frames of the player/partner pokemon are missing, too, since the sprite rips usually only include the standard frames. All the overworld is missing too, though that could be stored as a large bitmap rather than as tiles because not many of them are reused (except for the water, which is animated).

Edited June 16, 2015 by bananagram

[psy_commando]

Character sprites are already exportable. And re-importable too, but some things still need to be figured out to import a few particular sprites properly.

Move animations are just regular sprites re-used and animated differently, and some particle effects are layered over them.

The cutscenes and some backdrops are stored as a single big image. Some animated backdrops are stored as several images in the same file. I've been adding support for that, but there are still some major things I want to figure out before updating gfxutil.

Everything else is stored as tiles. And I've been trying to figure out the tiles' format, but I'm a bit stumped for now. So I went to work on the game's audio instead, waiting to get some kind of big realization.

Edited June 20, 2015 by psy_commando

[WhatHappenedToStairs]

So I tried doing something with this (inserting portrait), but I am such a noob that I don't know how to use the program (inb4 I missed obvious tutorial/manual). It says about Input and Output, but...