Jump to content

Pokemon Mystery Dungeon 2 - Psy_commando's Tools and research notes


Recommended Posts

I think those are from Gates to Infinity (3DS), not Explorers (NDS). However, for the entire list of "code from rom", I'm calling BS on that for now, unless more context is given.

Oh, I got that part. Its just that I was wondering if there wasn't a link somewhere with what we were talking about that I'd have missed. Its interesting to see they actually store 3d models inside SIR0 files though xD

And, I don't know how you can say its bs, when I don't even know what those codes are for.. Are they file ids or hashes or AR codes or something like that ? And then, what do they do?

I haven't actually made any edits to the ROM, only read data from it. I know there are a huge number of pieces of research possible by modifying the source material (i.e. compiled binary) to see what effects it has on the output, but I've only really done things like that when I'm trying to make the software behave differently in a specific way.

I don't know why I never actually do it. Case in point, Dwarf Fortress (which is very open to modding) has all the creature and object data stored in plain text files, called raws. However, incorrect modding results in what is called "duped raws" or "duplicated raws", where the game goes crazy because something very weird happened with the resources, and I _mean_ _crazy_ (even by DF standards; you can get things like embarking as elephants with picks made of spit, on dirt made of flesh and stone which is actually bacon). I really want to experiment with duped raws at some point, but never get around to it.

You should definitely try. Though, better extract the files first, and then pack them back into a rom, because of the nitrofs's FAT.

EoS is pretty straightforward when you edit things. It usually black screen or crashes when you screw something up.

Link to comment
Share on other sites

And, I don't know how you can say its bs, when I don't even know what those codes are for.. Are they file ids or hashes or AR codes or something like that ? And then, what do they do?

Check PMs

EDIT:

psy_commando is confirming good faith. What exactly is that data? I had believed that to be a troll attempt because it doesn't look like proper data; it is groups of 8 alphanumeric characters, but NOT groups of 8 hex characters. Notably, it is sorted. Is it possibly a list of file IDs, stored in the ROM as a char[8]? Can't be AR codes, those (at least on NDS) are groups of pairs of DWORDs, unless Datel is trying to take the route the old Game Genie took with encrypted codes...

Hmmm...I'm going to go check something on Bulbapedia...

EDIT: Bulbapedia doesn't have a list of PkMn in GtI. Serebii did, 148-151 or something, so there goes that theory, unless alt forms are present and I only counted Kyurem's.

Edited by TruePikachu
Link to comment
Share on other sites

Just as a heads-up, I'm officially dropping support for pmd2mid in favor of pmDJ, "Pokémon Mystery DJ", a collection of tools which will pretty much do the same thing, as well as convert in the other direction (eventually)

I have the codebase, right now, as a clone of the useful things from pmd2mid; project does compile properly right now.

Codebase is now hosted at Github, https://github.com/TruePikachu/pmDJ, so check there for further updates to my tools.

EDIT: Same is going for my notes, now at https://github.com/TruePikachu/pmDJ/wiki

Edited by TruePikachu
Link to comment
Share on other sites

Update:

I made another commit of my code, this time with working code for the unpacker/packer. I also improved it a lot, its even faster now!

For some reasons though, packing is faster than unpacking..

200ms to pack vs 450ms and up to unpack. I don't know why I'm so obsessed with performances ^^;

I also fixed a couple of dumb things I did, like in a few loops I was using a "for( auto curentry : m_structure )" kind of loop, and I forgot to put a "&" between auto and the variable name..

The result was a mess of unneeded copies which I didn't even notice until now ^^;

Just as a heads-up, I'm officially dropping support for pmd2mid in favor of pmDJ, "Pokémon Mystery DJ", a collection of tools which will pretty much do the same thing, as well as convert in the other direction (eventually)

I have the codebase, right now, as a clone of the useful things from pmd2mid; project does compile properly right now.

Codebase is now hosted at Github, https://github.com/TruePikachu/pmDJ, so check there for further updates to my tools.

EDIT: Same is going for my notes, now at https://github.com/TruePikachu/pmDJ/wiki

Sounds good!

wonder mail code for us version.

That explains everything xD

Honestly, I never even used wonder mail before.. I didn't even attempt a friend rescue :/

So I wasn't too sure what those were.

What do those do in particular though ?

Link to comment
Share on other sites

I've uploaded the code for swd2dls to https://drive.google.com/folderview?id=0B8n9Q6oziVxuRVBNOTJ6djNMNEU&usp=sharing. All that it does at the moment is read the wavi chunk for offsets that it uses to extract samples from the pcmd chunk. There are two unknown 32-bit integers that I'm sure have something to do with the length and the loop point of the samples, but I haven't quite worked that part out yet...

Link to comment
Share on other sites

I've uploaded the code for swd2dls to https://drive.google.com/folderview?id=0B8n9Q6oziVxuRVBNOTJ6djNMNEU&usp=sharing. All that it does at the moment is read the wavi chunk for offsets that it uses to extract samples from the pcmd chunk. There are two unknown 32-bit integers that I'm sure have something to do with the length and the loop point of the samples, but I haven't quite worked that part out yet...

Sounds good!

You might want to take a look at the Wav format. Given it can contain loop points info as well, and would give you a good idea of how its commonly done, and what data needs to be stored!

On my side, I've been writing a sprite analyser tool. It might just help figure out the format!

At least I hope.. ^^;

Link to comment
Share on other sites

I came up with a theory with the internal filenames last night...

...need to find my DS Lite, ARDSi, and EoT cart...

(I know that there is a _lot_ of debug stuff in EoT/EoD, maybe filenames?)

...I wonder if dsLinux can get read access to the cart inserted into the ARDSi...simplifies things if I can grep from the system itself...

EDIT: :( nope, grep of B_DUN_BANNIN_01 results in only matching the bgm0056 SMD/SWD files

Link to comment
Share on other sites

Speculating, I'd say that the internal names are left over from development. Especially seeing that most of them are only a ~14 character substring of what would logically be the file name. It could be that upon compiling the ROM, the music names were replaced with numbers.

Link to comment
Share on other sites

Speculating, I'd say that the internal names are left over from development. Especially seeing that most of them are only a ~14 character substring of what would logically be the file name. It could be that upon compiling the ROM, the music names were replaced with numbers.

Which is exactly why I checked the EoT ROM, since EoT/EoD has a whole lot of debugging stuff that was removed from EoS.

Link to comment
Share on other sites

I'd say its just an unwanted/unneeded feature of the toolset they've used. There are really no need for meaningful filenames in there, given its the final copy that they ship to production.

Also, I've been discovering a few interesting things about sprites.. I still don't have a clue of where to find the actual resolution, but I think I might be close to figuring it out.

For those interested, I made myself a little program to analyze sprite files in depth, and fill a report on them :

ppmd_spranalyser_alpha.zip

It only works on uncompressed sprites right now, and is still kinda glitchy, but the reports are really useful ! I'm just too slow at drawing parallels based on the data I found in those.. So if anyone wanna run it on a bunch of sprites and try to figure out what is what, that could help ! There are no readme, but the program can print its own detailed readme, if ran without arguments.

And the source code was commited to the codeplex page for the library.

ppmd_spranalyser_alpha.zip

Link to comment
Share on other sites

Ok, so after looking at a ton of sprites, I think I might have found a few correlations here and there between the format of the sprite and the value of the 9 boolean values spread around the 3 parts of the sprite header.

But, I think that, I might need the opinion of others to definitely find out what it is exactly that I've found..

So lets compare 3 sprite files. First a regular character sprite, then a sprite for some UI element, then another sprite for some UI element but with some difference.

Character sprite "/MONSTER/m_ground.bin/file_0001":

Sprite Sub-Header :
------------------------
Offset img data.........: 0x00004004
Offset unk data.........: 0x00003FEC
Unknown value0..........: 0x0001
Unknown value1..........: 0x0000

Sprite Info Block :
------------------------
Offset table E..........: 0x00003968
Offset table F..........: 0x00003A68
Offset table G..........: 0x00003F0C
NbEntries table G.......: 13
Unknown.................: 4
Unknown #1..............: 0x0000
Unknown #2..............: 0x0000
Unknown #3..............: 0x0000
Unknown #4..............: 0x0000

Sprite Frame Data Block :
------------------------
Offset frame ptr table..: 0x00003F74
Offset palette end......: 0x00003958
Unknown #1..............: 0x0000
Unknown #2..............: 0x0000
Unknown #3..............: 0x0001
Nb frames...............: 30

The UI sprite ("/FONT/cursor.wan"):

Sprite Sub-Header :
------------------------
Offset img data.........: 0x00000958
Offset unk data.........: 0x00000940
Unknown value0..........: 0x0000
Unknown value1..........: 0x0000

Sprite Info Block :
------------------------
Offset table E..........: 0x00000908
Offset table F..........: 0x00000000
Offset table G..........: 0x00000928
NbEntries table G.......: 1
Unknown.................: 8
Unknown #1..............: 0x0000
Unknown #2..............: 0x0000
Unknown #3..............: 0x0001
Unknown #4..............: 0x0000

Sprite Frame Data Block :
------------------------
Offset frame ptr table..: 0x00000930
Offset palette end......: 0x000008F8
Unknown #1..............: 0x0001
Unknown #2..............: 0x0001
Unknown #3..............: 0x0001
Nb frames...............: 4

And an alternate version version of it ("/FONT/cursor16.wan"):

Sprite Sub-Header :
------------------------
Offset img data.........: 0x00000398
Offset unk data.........: 0x00000380
Unknown value0..........: 0x0000
Unknown value1..........: 0x0000

Sprite Info Block :
------------------------
Offset table E..........: 0x00000348
Offset table F..........: 0x00000000
Offset table G..........: 0x00000368
NbEntries table G.......: 1
Unknown.................: 4
Unknown #1..............: 0x0000
Unknown #2..............: 0x0000
Unknown #3..............: 0x0000
Unknown #4..............: 0x0000

Sprite Frame Data Block :
------------------------
Offset frame ptr table..: 0x00000370
Offset palette end......: 0x00000338
Unknown #1..............: 0x0001
Unknown #2..............: 0x0000
Unknown #3..............: 0x0001
Nb frames...............: 4

Just compare the values that aren't pointers/offset or 32bits integers.

Its a pretty odd way of indicating the format of the file no ? I mean, in those 20 bytes they could have fitted a lot more details about the sprite data..

Which kinda makes me doubt a little of what I found here.. Could these be only coincidences ? Or are those really indicating details about the format ?

EDIT:

Here are a few more that differ:

"/GROUND/d01p11b2.wan"

Sprite Sub-Header :
------------------------
Offset img data.........: 0x00005D4C
Offset unk data.........: 0x00005D34
Unknown value0..........: 0x0000
Unknown value1..........: 0x0000

Sprite Info Block :
------------------------
Offset table E..........: 0x00005854
Offset table F..........: 0x00000000
Offset table G..........: 0x00005CE0
NbEntries table G.......: 1
Unknown.................: 6
Unknown #1..............: 0x0000
Unknown #2..............: 0x0000
Unknown #3..............: 0x0000
Unknown #4..............: 0x0000

Sprite Frame Data Block :
------------------------
Offset frame ptr table..: 0x00005CE8
Offset palette end......: 0x00005844
Unknown #1..............: 0x0000
Unknown #2..............: 0x0000
Unknown #3..............: 0x0001
Nb frames...............: 19

"/GROUND/d79p41a.wan"

Sprite Sub-Header :
------------------------
Offset img data.........: 0x00002050
Offset unk data.........: 0x00002038
Unknown value0..........: 0x0000
Unknown value1..........: 0x0000

Sprite Info Block :
------------------------
Offset table E..........: 0x00001F8C
Offset table F..........: 0x00000000
Offset table G..........: 0x00001FCC
NbEntries table G.......: 1
Unknown.................: 15
Unknown #1..............: 0x0000
Unknown #2..............: 0x0000
Unknown #3..............: 0x0000
Unknown #4..............: 0x0000

Sprite Frame Data Block :
------------------------
Offset frame ptr table..: 0x00001FD4
Offset palette end......: 0x00001F7C
Unknown #1..............: 0x0000
Unknown #2..............: 0x0000
Unknown #3..............: 0x0003
Nb frames...............: 25

"/GROUND/as001.wan"

Sprite Sub-Header :
------------------------
Offset img data.........: 0x00000A48
Offset unk data.........: 0x00000A30
Unknown value0..........: 0x0000
Unknown value1..........: 0x0000

Sprite Info Block :
------------------------
Offset table E..........: 0x00000A10
Offset table F..........: 0x00000000
Offset table G..........: 0x00000A18
NbEntries table G.......: 1
Unknown.................: 16
Unknown #1..............: 0x0000
Unknown #2..............: 0x0000
Unknown #3..............: 0x0000
Unknown #4..............: 0x0000

Sprite Frame Data Block :
------------------------
Offset frame ptr table..: 0x00000A20
Offset palette end......: 0x00000A00
Unknown #1..............: 0x0000
Unknown #2..............: 0x0000
Unknown #3..............: 0x0004
Nb frames...............: 4

There are a lot more unusual values here !

Edited by psy_commando
put the wrong report for the first sprite..
Link to comment
Share on other sites

I'm taking research requests right now, if you want, I can see about checking all the sprites and trying to convert them to something simple like .bmp . I've been really bored recently, and keep dying in NetHack...

Alternatively, if someone has another good suggestion for me to try to research, if I deem it "interesting" (something I would have use for), I'll work on it.

Are the backgrounds, etc. researched enough to extract?

EDIT: I should see about getting `ndsfs` set up on my VM...

EDIT: I can't pick up any sort of magic number off my m_ground.bin:

$ hexdump -C m_ground.bin | head
00000000  00 00 00 00 57 02 00 00  00 13 00 00 f0 b3 00 00  |....W...........|
00000010  f0 c6 00 00 30 41 00 00  20 08 01 00 10 3a 00 00  |....0A.. ....:..|
00000020  30 42 01 00 10 3a 00 00  40 7c 01 00 c0 ad 00 00  |0B...:..@|......|
00000030  00 2a 02 00 a0 44 00 00  a0 6e 02 00 30 3d 00 00  |.*...D...n..0=..|
00000040  d0 ab 02 00 e0 a8 00 00  b0 54 03 00 d0 3d 00 00  |.........T...=..|
00000050  80 92 03 00 d0 41 00 00  50 d4 03 00 80 31 00 00  |.....A..P....1..|
00000060  d0 05 04 00 a0 31 00 00  70 37 04 00 c0 37 00 00  |.....1..p7...7..|
00000070  30 6f 04 00 90 2b 00 00  c0 9a 04 00 70 22 00 00  |0o...+......p"..|
00000080  30 bd 04 00 10 38 00 00  40 f5 04 00 00 2b 00 00  |0....8..@....+..|
00000090  40 20 05 00 30 2f 00 00  70 4f 05 00 c0 34 00 00  |@ ..0/..pO...4..|

EDIT: Figured that out, wrote my own unpacker, working on understanding SIR0

Edited by TruePikachu
Link to comment
Share on other sites

You didn't have to do all that though ^^;

Pack files are well known by now. I've had the extractor and re-packer tool coded and on the front page for months now.

SIR0 also has been figured out for a long while, its all in the common format notes on the first page of the thread.

You might want to check those out, and go from there. The sprite analyser I posted a few post ago is all based on that knowledge.

EDIT:

And, we've already got a tool to extract sprites to indexed pngs.(I had bmp support, but its just redundant at this point given png does exactly the same thing and has good compatibility)

But the thing is, right now we have to guess the resolution of each frames in the sprite, given they can all have different dimensions. 16x32, 32x32, 8x16, name it.. And they can be laid out on the width or on the height..

So there must be something in the sprite file that we didn't see that indicates the proper image resolution.. Maybe something indicating the amount of tiles on the width and height, or simply indicating the resolution in pixels ?

Plus, we also found out that some sprite can be in another format than 4bpp... And so far the only obvious differences I spotted using my tool were a couple of 16bits integer.

If we can figure resolution out, and then the way animations are stored, we'll be pretty much able to rebuild a sprite from scratch !

Link to comment
Share on other sites

Well, that's all good.

But I'm just saying that it might be better not to burn yourself out on things like that before getting to the core of the problem!

I mean, I'm probably stuck in a way of thinking that keeps me to see the very obvious solution to the issue basically. And maybe that anyone else looking 5 minutes at it could find out what I missed ^^;

Its not the first time.. Remember those signed values I mistook for something else in the smd files? xD

Link to comment
Share on other sites

Well I might have found the pattern, actually, looking only at the Bulbasaur SIR0. Working on coding it up to check, but it looks like the SIR0 includes the actual animations as well (of which the frame sizes would be encoded inside).

EDIT: DBS_ENTRY+0x0 is a frame ID, DBS_ENTRY+0x4 & 0xC0 looks to determine the width of the sprite. Further research is required.

Link to comment
Share on other sites

Yep, they most likely contain animation data. There are 4 places I know of that could actually contain those though. But DBS is a pretty good candidate.

Datablock F was pretty high on my list, because in the UI sprites and anything that wasn't a pokemon, the entire block isn't even there at all.. But the data is too short and random to be a sequence of frames...

And you might not want to call those SIR0, because SIR0 is a generic container. Like, it can contain PKDPX files and literally anything you want. Most loose files that have the SIR0 header and contain sprites, in the FONT and GROUND folders, have a .wan file extension. Some have another extension though.

I'd speculate that they possibly might use 2 nybbles to indicate the amount of vertical and horizontal tiles, given at 40x40 we're not yet past the nybble's max value in terms of tiles.

And if you want to take a look, here's a detailed report on Bulbasaur's sprite:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/file_0000_0x1300.log

It kinda structures/organizes the data a little. Maybe it will help.

Link to comment
Share on other sites

Oh, so you cross-referenced DBG,DBH,DBI with DBS ? That looks pretty solid to me! Nicely done! :D

I can't believe I didn't try to do that.. Even though I'm hardly surprised I didn't given my track record ^^;

You should try it on "FONT/cursor.wan" and "FONT/cursor16.wan", and then on a sprite file with a lot of randomly sized frames, like vaporeon's, its file number 146 or something really close in "MONSTER/m_ground.bin" (eevee is 145).

Link to comment
Share on other sites

Looks like Vappy is done strangely:

42      realFrameID=26   val0..4=-39,1,-13,64,3072
SpriteFrame: (width=4):
................................
................................
................................
................................
................................
........................1.......
.......................181......
.......................1861.....

43      realFrameID=28   val0..4=-9,65,-13,-128,3072
SpriteFrame: (width=4):
........1189AA981...............
........19AAAA981...............
........199A9AA891..............
........19998AA981..............
........19988AAA891.............
.......189818AAA891.............
.......1991189AA98A1....11...1..
......11981.199AA8991...191.191.
......1981..1AAAA98A1...1A911A1.
.......111.19AAA8A8991..19A19A1.
...........189A91A98A1...198A91.
............119981A8991...19A1..
.............11111A98A1...1981..
..................1A8A1..1891...
..................1A88A11891....
..................18A8899981....

Care to confirm file frame numbers 26 and 28, data-wise? I wouldn't expect the sprite data to be chopped in this odd way, how I'm reading it.

I get similar oddness with Torterra, 0x0130=304

EDIT: Neither of those font .WANs give me any sprite data from animation, but four animations are detected for them.

Link to comment
Share on other sites

Yeah, I specified those particular cases because they completely threw me off !

The cursor16.wan has a different image format, or maybe its cursor.wan.. I mix them up all the time..

What do you mean data wise ? You mean like the report from my tool running on that sprite ?

I'm also pm-ing you the images I've extracted from that. AFAIK the frames are really "chopped" that way..

EDIT:

Here's what I got for Vaporeon:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/temp/file_0146_0x2d2e70.log

And here's what I got for cursor.wan and cursor16.wan :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/temp/cursor.log

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/temp/cursor16.log

( Just a word of caution: never trust the "average" values in those reports, there's something freaky going on with that.. Probably some implicit signed/unsigned conversion messing the result up.. )

Also, I'll have to sign off for a few hours, so it will take some time before I can reply.

Link to comment
Share on other sites

Wheee!

Rewrote my code to make it easier to maintain. Now the textual dumps of sprites will use ANSI coloring (based on the high bits of each component)...and I got animation playback implemented!

Vaporeon does have something different with its SIR0, compared to e.g. Eevee; spriteDump and playAnim will both fail with loading Vaporeon ("RESEARCH THIS!" is the error message).

Link to comment
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
×
×
  • Create New...