tool Pokemon Mystery Dungeon 2 - Psy_commando's Tools and research notes

psy_commando · May 26, 2014

Hi ! I'm fairly new around here, and I'm relatively new to rom hacking and to the NDS scene in general, but I'm working on several little projects for Pokemon Mystery Dungeon Explorers of Sky / Time / Darkness, and I thought it would nice to share it with others ! Especially with others with more technical knowledge than I !

I hope this will be useful ! :biggrin:

And be sure to report any bugs, or tidbits of information you have ! Those would prove invaluable !

Feel free to mention whether it worked for you, or not so much ! Any feedback is always great !

You might need 7zip or another 7z compatible program to open the compressed files on this page.

Looking for:

- Need people to write tutorials for some of the tools !
- Need feedback on the tools! 
- Need people to discover what some of the "unk" variables in the character sprites do !
- Need info on the NDS's DSP.

The Tools:

Compression/Decompression:

PPMD UnPX

Description: A quick little implementation of a "PX" compressed data decompressor. Its based on Zhorken's Python script for decompressing the character portraits, which happened to be stored as compressed "PX" files ! Just feed it an AT4PX or PKDPX file by drag and dropping, or use the command line argument for specifying the input file and output folder. The result will have an extension appended, or not, depending on whether its currently possible to guess the content. Again, many thanks to Zhorken for finding out about the compression scheme !
Donwload: Version 0.41 : ppmd_unpx_0_41.zip on Github

PPMD DoPX

Description: A compressor to compress files back into PX compressed files, either PKDPX or AT4PX files. Its still pretty new and experimental, so, watch out for bugs ! By default anything compressed is outputed to a PKDPX. To output to a AT4PX, the output filename has to be specified and to end with ".at4px" !
Donwload: Version 0.31 : ppmd_dopx_0_31.zip on Github

Packing/Unpacking:

PPMD Pack File Utility

Description: This tool unpacks and re-packs "pack files". Those are single file that contains many sub-files. Pack files often end with a .bin extension, and they can contain a lot of different types of files at the same time. Mostly for research purpose.
GUI Image:

Donwload: Version 0.53 + GUI 1.2 : ppmd_packfileutil_gui_1_2-0_53.zip on Github

Graphics:

PPMD KaoUtil

Description: This utility allows to both unpack the kaomado.kao file containing all the pokemon portraits, but also to rebuild the file! Basically, you can add new potraits into the game using this tool ! It just extract everything like a zip file if you want. And you just have to add a properly named 4 bits per pixels/16 colors png or bmp image to the folder matching the pokemon you want, before repacking the parent folder again into a ".kao" fle. The first color in the palette is reserved by the game and is forced to pure black, so you have really 15 colors.
Example: For example, here's a crappy little edit of Poochyena's basic portrait. I inserted it into slot 2, which correspond to the emotion named "Grin"! Poochyena by default doesn't have a portrait for this emotion, but by using the tool, I was able to insert this very easily !

GUI Image:

Donwload: Version 0.41 + GUI : ppmd_kaoutil_gui_1_0-0_41.zip on github
Dowload: Version 0.42 No GUI: ppmd_kaoutil_0.42 on github

PPMD GfxCrunch

Description : Handles converting various image formats from the game, including WAN sprites, BGP images, Compressed Pokemon sprites "*.pkdpx", and more to come! It exports animated sprites to XML for easier editing. It also can handle all at once the entire Pokemon sprites containers "monster.bin", "m_ground.bin", "m_attack.bin".
Currently Supported Graphics Formats:
- Individual Pokemon sprites "*.wan" file ( Type 1 ).
- Compressed Pokemon sprites "*.pkdpx"
- Entire Pokemon sprites container "monster.bin", "m_ground.bin", "m_attack.bin"
Currently Partially Supported Graphics Formats:
- Prop sprites with orphaned images "*.wan" file ( usually Type 0 or 3 ). Can extract, but wrong resolution and tile/pixel ordering, and can't be re-packed properly.
- WAT sprites "*.wat" file. (basically "*.wan" sprites with a different file extension and with their type byte set to type 3) Not tested.
- BGP Images. "*.bgp" files.
Currently Unsupported Graphics Formats:
- WTE Image container. "*.wte" file.
- WTU Color animation data file ? "*.wtu" file.
- kaomado.kao. (Use ppmd_kaoutil instead for now!)
- Raw at4px-compressed images. (This will never be perfect, because its raw data.)
- Raw images. (This will never be perfect, because its raw data.)
- Everything else..
Donwload: Alpha 0.13 : ppmd_gfxcrunch_0_13_alpha on GitHub

Stats/Text/etc..:

PPMD StatUtil

Description: A tool for exporting game statistics to XML, and importing them back into the game's files. Making it easy for third party tools developer to make more complete and user friendly utilities, can also be edited by humans directly and opened with anyone's favorite XML editor.(Notepad++ highly recommended) So far has limited support.
Currently supports editing: Move data, Pokemon Data, Item data, Generic game text, Game scripts + game dialog.
Donwload: Alpha 0.23 : ppmd_statsutil_0.23.2 on GitHub
Tutorial/Example: here, also include a few files and tools to make things easier!
Example Video for the Script Editing :

Audio:

PPMD AudioUtil

Description: A tool for exporting, and eventually importing/modifying, audio to/from games using the DSE audio driver, such as PMD2 ! Can output the samples to wav files, or the samples and music tracks to a soundfont and MIDIs, or to MIDIs only. You can specify your own conversion rules for exporting only MIDIs, such as what instruments the native DSE preset translate to when converted into MIDI ! This program REQUIRE you to read the readme btw. Its a console program only, for now. There is so much stuff going on in it that there won't be a GUI frontend for a while..
Please Note: Its still far from perfect!
Currently supports: Exporting Music to MIDI and soundfonts. Exporting MIDIs only. Exporting samples.
Donwload: Version 0.37 (2017/12/14) : latest ppmd_audioutil on Github

Source Code:

All source code for all utilities is available. Its not always pretty, because this codebase is also used for research, and messing around. But its much better than not sharing anything at all, I guess.

Main repository for the actual utilities : ppmdu
Secondary repository for the GUI frontends : ppmdu_gui_frontends

Research Notes:

For the most reliable, tested, and up to date info, check the Project Pokemon Wiki:

http://projectpokemon.org/wiki/Pok%C3%A9mon_Mystery_Dungeon_Explorers

All my current personal research notes files:

File Formats : https://www.dropbox.com/sh/8on92uax2mf79gv/AADCmlKOD9oC_NhHnRXVdmMSa?dl=0
General notes : https://www.dropbox.com/s/ucy2t9hqgtfn0y0/PMD2_GeneralNotes.txt?dl=0
PMD2_MusicAndSoundFormats: https://www.dropbox.com/s/bw0aym9rn22z4wu/PMD2_MusicAndSoundFormats.txt?dl=0
PMD2_MusicTracksAndInfo : https://www.dropbox.com/s/rwjq0s0fvtiamsv/PMD2_MusicTracksAndInfo.txt?dl=0
A bunch of jumbled notes on scripts : https://www.dropbox.com/s/1roqlwkhfqb9amz/PMD2_Scripts_And_Resources.txt?dl=0

Misc Unsupported Utilities

Here are a few little utilities I made to help with researching the ROMs. They're nothing special, and probably not very stable, but they're really handy:

SIR0PtrOffsetsDecoder : Use this to decode a SIR0 pointer offset list, which can help tremendously in finding out more about what a sir0 file contains! Just copy/paste the SIR0's ptr offset list only, and put it into its own separate file, and drag-and-drop that onto the .bat file, it will output the result in "decoded.txt". The program itself only output the result to the console, so use the batch script, or pipe the output!
IntegerListDecoder : Use this one to decode a list of encoded integers. Things like the stats growths in "m_level.bin", the move lists in "waza_p.bin". You can just put all the lists one after the other in a new file, and drag-and-drop that onto the .bat file and all the decoded lists will be outputed to "decoded.txt" !

Pages of Interest:

Todo

Edited December 14, 2017 by psy_commando
Updated dropbox links, fixed formating being messed up because of forum update

evandixon · May 26, 2014

It's great to see progress in editing this game. Good luck with your sprite tool. Based on what I saw in YY-CHR, it can't be straight forward. It displayed tiles side by side, and two tiles needed to be on top of the next 3 tiles..... yeah, can't be fun.

Besides the packing format, this might also be applied to Blue Rescue Team, which appears similar in YY-CHR. (Not to rush you or anything. I know this isn't easy.)

Now that I mentioned Blue Rescue Team, here's VB code to unpack the sbin files. Not that it's really related to the topic, but it's better than sitting there in the deep dark depths of my hard drive.

Module Module1

   Sub Main()
       unpack(IO.File.ReadAllBytes("C:\Users\[user]\Desktop\blue\data\monster.sbin"), "C:\Users\[user]\Desktop\blue\data\monster")
   End Sub
   Sub unpack(bytes As Byte(), directory As String)
       Dim x As New System.Text.ASCIIEncoding
       For count As Integer = 0 To bytes.Length - 1 Step 16
           Dim name = x.GetString(bytes, count, 8).Trim(vbNullChar)
           If name.Length > 0 Then
               Dim start = BitConverter.ToUInt32({bytes(count + 8), bytes(count + 9), bytes(count + 10), bytes(count + 11)}, 0)
               Dim last = BitConverter.ToUInt32({bytes(count + 12), bytes(count + 13), bytes(count + 14), bytes(count + 15)}, 0)
               Console.WriteLine("Extracting {0}", name)
               IO.File.WriteAllBytes(IO.Path.Combine(directory, name & ".bin"), GenericArrayOperations(Of Byte).CopyOfRange(bytes, start, start + last - 1))
           Else
               Exit For
           End If
       Next
   End Sub
End Module
Public Class GenericArrayOperations(Of T)
   Public Shared Function CopyOfRange(ByteArr As T(), Index As Integer, EndPoint As Integer) As T()
       Dim output(Math.Max(Math.Min(EndPoint, ByteArr.Length) - Index, 0)) As T
       For x As Integer = 0 To output.Length - 1
           output(x) = ByteArr(x + Index)
       Next
       Return output
   End Function
End Class

psy_commando · May 27, 2014

It's great to see progress in editing this game. Good luck with your sprite tool. Based on what I saw in YY-CHR, it can't be straight forward. It displayed tiles side by side, and two tiles needed to be on top of the next 3 tiles..... yeah, can't be fun.
Besides the packing format, this might also be applied to Blue Rescue Team, which appears similar in YY-CHR. (Not to rush you or anything. I know this isn't easy.)

Now that I mentioned Blue Rescue Team, here's VB code to unpack the sbin files. Not that it's really related to the topic, but it's better than sitting there in the deep dark depths of my hard drive.
Module Module1

   Sub Main()
       unpack(IO.File.ReadAllBytes("C:\Users\[user]\Desktop\blue\data\monster.sbin"), "C:\Users\[user]\Desktop\blue\data\monster")
   End Sub
   Sub unpack(bytes As Byte(), directory As String)
       Dim x As New System.Text.ASCIIEncoding
       For count As Integer = 0 To bytes.Length - 1 Step 16
           Dim name = x.GetString(bytes, count, 8).Trim(vbNullChar)
           If name.Length > 0 Then
               Dim start = BitConverter.ToUInt32({bytes(count + 8), bytes(count + 9), bytes(count + 10), bytes(count + 11)}, 0)
               Dim last = BitConverter.ToUInt32({bytes(count + 12), bytes(count + 13), bytes(count + 14), bytes(count + 15)}, 0)
               Console.WriteLine("Extracting {0}", name)
               IO.File.WriteAllBytes(IO.Path.Combine(directory, name & ".bin"), GenericArrayOperations(Of Byte).CopyOfRange(bytes, start, start + last - 1))
           Else
               Exit For
           End If
       Next
   End Sub
End Module
Public Class GenericArrayOperations(Of T)
   Public Shared Function CopyOfRange(ByteArr As T(), Index As Integer, EndPoint As Integer) As T()
       Dim output(Math.Max(Math.Min(EndPoint, ByteArr.Length) - Index, 0)) As T
       For x As Integer = 0 To output.Length - 1
           output(x) = ByteArr(x + Index)
       Next
       Return output
   End Function
End Class

Thanks !

And it shouldn't be all that too complicated. I was able to get some of them to display correctly in crystal tiles 2 before. Its just that the sprites seems to have some data in-between them that disalign them. Or at least, that's what I think it is..

Plus the sprites don't have always the same resolution, like the down facing ones are 16x32, the diagonal ones are 24x32, and the side facing ones are 32x32, so that doesn't help either !

But, what are sbin ? I don't remember seeing those..

evandixon · May 27, 2014

Thanks !
And it shouldn't be all that too complicated. I was able to get some of them to display correctly in crystal tiles 2 before. Its just that the sprites seems to have some data in-between them that disalign them. Or at least, that's what I think it is..

Plus the sprites don't have always the same resolution, like the down facing ones are 16x32, the diagonal ones are 24x32, and the side facing ones are 32x32, so that doesn't help either !

But, what are sbin ? I don't remember seeing those..

sbin is exclusive to Blue Rescue Team, and I haven't seen any in any Explorers game.

Anyway, I'd guess the data in between the tiles has something to do the size of each tile (not every Pokémon is the same size).

psy_commando · May 27, 2014

Hmm.. Oh well, I guess it can come in handy later on then ! Doing Red and Blue rescue team as well could be interesting ! Especially given there are far more tools and existing data to go from !

A fun fact about PMD:Red Rescue Team is that some subfiles begin with a SIRO magic number ! Which is kind of an odd way to name them given the newer format is SIR0.. Its really easy to confuse both of them, given O almost looks like 0..

And, I'm thinking the data between tiles/frames is probably padding. They really seem to like aligning everything on 16 bytes ! And besides the SIR0 files that contains all the sprite data for all pokemon seems to rely on pointer tables to indicate both the size and location of frames. At least that what I think so far, but its still not 100% sure. It could in fact actually be the size of each tiles..

I'm still trying to figure out what pointer tables points to what kind of data.

If you ever want to take a look by yourself, here are some of my notes :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/CommonFileFormats.txt

I store everything on dropbox, so they're updated as I'm writing to them.

I also know someone from the PokeCommunity forums that is working on a script editor for PMD, and he shared some of his notes too :

http://www.pokecommunity.com/showthread.php?p=8248902#8248902

He's much more advanced than I am in this ! And its always useful to have a second opinion !

I've also got some old notes laying around from my findings using Crystal Tiles 2. Mainly tiles/frames offset, which might be handy in figuring out where is what ! (Don't take the dimensions into account too seriously, they're most likely not accurate !) :

NOTE: the following where taken from opening the whole thing in crystal tile2, so the offset probably need to be moved appropriately !

Bulbasaur world sprites(in crystaltile2):
- Running :
	- Down  :
		- frame1 at		  0x4544
		- frame2 at		  0x46B8 (first + 0x177)
		- frame3 at		  0x482C (second + 0x174)
	- Down Left Run:
		- frame1 at		  0x49A0
		- frame2 at		  0x4B14 (? the frame is somehow cut and doesn't align properly with the rest at 16 px on the height)
		- frame3 at		  0x4C68
	- Left Run:
		- frame1 at		  0x4DDC
		- frame2 at		  0x4F50
		- frame3 at		  0x50A4 (? trash appearing at the top left)
	- Up Left Run:
		- frame1 at		  0x5224
		- frame2 at		  0x5398 (? the frame is somehow cut and doesn't align properly with the rest at 16 px on the height)
		- frame3 at		  0x54EC
	- Up Run:
		- frame1 at		  0x5660
		- frame2 at		  0x57D4
		- frame3 at		  0x5948
- Taking hit ( 24 x 32 )
	- Down :
		- frame at		  0x5ABC
	- Down Left : 
		- frame at 		  0x5CA8 ( ? the frame seems to be cut in 3 places and misaligned on all those 3 ! )
	- Left :
		- frame at		  0x5E70 ? (too badly aligned to tell...)
	- Left Up:
		- frame at 		  0x6084
	- Up:
		- frame at		  0x6264

- Sleeping / Yawning ? ( 32 x 24 )
	- Left:
		- frame1 at		  0x6450
		- frame2 at		  0x6758
		- frame3 at		  0x68FC
		- frame4 at		  0x6A80

- Cheering ? ( 24 x 24 )
	- Down:
		- frame1 at		  0x6C30

- Looking ? ( 24 x 24 )
	-Down Left : 
		- frame 1 at 	  0x6DA4
	- Down Right:
		- frame 1 at 	  0x6F18

- Rolling ??? ( 24 x 24 )
	- Up:
		- frame1 at		  0x708C
		- frame2 at		  0x7200
		- frame3 at		  0x7374
		- frame4 at		  0x74E8

- Unconscious ? ( 24 x 32 )
	- Down:
		- frame1 at		  0x763C

- Cheering ? ( 24 x 24 )
	- Down:
		- frame1 at		  0x7814 (misaligned)

- Falling/Rolling ? ( 32 x 32 ) 
	- Up:
		- frame1 at		  0x79B4 (misaligned)
		- frame2 at		  0x7AFC (misaligned)
		- frame3 at		  0x7D1C (misaligned)
		- frame4 at		  0x7F60

- Knocked out ? ( 32 x 32 )
	- Left:
		- frame1 at		  0x8074 ( image split in 3 misaligned stripes )
		- frame2 at		  0x8254 ( image lower stripe misaligned )

- Yawning again ? ( 24 x 24 )
	- Down:
		- frame1 at		  0x843C
		- frame2 at		  0x

- From the side looking down ( 24 x 24 )
	- Left:
		- frame1 at		  0x86BC

Poochyena's World Sprites:(near offset 0x953DB8)
- Running ( 16 x 24 and 24 x 24 ? )
	- Down:
		- frame1 at				0x954050
		- frame2 at 			0x954134
		- frame3(24x24) at 		0x9541F8 (misaligned)
	- Down Left: 
		- frame1(24x24) at		0x954318 (misaligned)
		- frame2(24x24) at 		0x9544C4 (misaligned)
		- frame3(24x24) at 		0x954638
	- Left:
		-frame1(24x32???)		0x954758(misaligned)
		-frame2(32x32) at		0x954870(extremely misaligned)
		-frame3(32x32) at		0x954A88(misaligned)
	-Up Left:
		-frame1(24x32) at		0x954C48(misaligned)
		-frame2(24x32) at		0x954DF4(misaligned)
		-frame3(24x32) at 		0x954FA0 or 0x954F40(misaligned)
	-Up:
		-frame2(16x32) at		0x955114
		-frame3(16x32) at 		0x95522C

- Getting Hit (32x32)
	-Down:
		-frame1 at 				0x955408(misaligned)
	-Down Left:
		-frame1 at 				0x9555BC
	-Left:
	-Up Left:
		-frame1 at
		-frame2 at
	-Up:
		-frame1 at 				0x9559D4
		-frame2 at 				0x955C38

-Idle (16 x 32)
	- Down:
		-frame1 at 				0x955D4C
		-frame2 at				0x955E70
	- DownLeft:
		-frame1(24x32) at		0x956150(misaligned)
		-frame2(24x32) at		0x9562FC(misaligned)
		-frame3(24x32) at		0x956468

-Attack stance(16x32)
	-Down:
		-frame1 at				0x955F34

Also, now I'm not sure what to call tiles, sprites, frames, images or what without confusing anyone

evandixon · May 27, 2014

Also, now I'm not sure what to call tiles, sprites, frames, images or what without confusing anyone

This is what each probably means:

Tile: static image that will appear beside a bunch of other static images (like the tiles in dungeons)

Sprite: character that moves around on the screen

Frame: individual image that's part of an animation

Image: static image that appears by itself (like a Pokémon's headshot)

psy_commando · May 27, 2014

Alright ! I found the table that points to individual frames and how it works !

I was able to manually rip the first frame of Bulbasaur's downward run anim !

I opened it in YY-CHR and its all there. Still not aligned, but its all there !

They're in what I've called this far the PointerBlockJ in my notes :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/CommonFileFormats.txt

Here's are the pointers to follow:

PointerA -> OFF_A

then from

OFF_A+0x4 -> OFF_A-0x10 -> OFF_J

OFF_J is the first pointer to a frame block in the pointer array I dubbed "PointerBlockJ" this far. The number of pointer in this array is stored at OFF_A-0x2!

Each pointer in "PointerBlockJ" points to a position near the end of a frame block. I refer to it as FRM_IN for now. That position may or may not contain another pointer to the beginning of the frame block, which is BEFORE FRM_IN.

And, there is 84 bytes of data AFTER FRM_IN that belongs to the frame block, containing data, and a pointer to somewhere in the frame block before FRM_IN.

I'm still trying to figure out what is the actual frame data from the rest though. But that's some nice progress !

EDIT:

And thanks for clearing up the terms. But, the one for sprite is kinda weird to me.. I did some 3D and we used sprite and billboard interchangeably for some reasons and it kinda stuck

EDIT2:

Scratch the 84 bytes of data after FRM_IN.. It can actually have a different size depending on the pokemon it seems.. I still haven't found out where data about the length and about what is after FRM_IN could be stored..

But, I made some pictures with annotations to show a little how the data is arranged !:

Here's what I call the "Master Pointer Table" for a lack of better name:

This is the first frame pointed by the first pointer in the PointerBlockJ above:

And after extracting that frame above into its own file and loading it in YY-CHR it gave this :

EDIT3:

Alright, it seems that, what is at/after FRM_IN is actually a list of pointers on 4 bytes pointing to actual image data inside the frame block's bounds, each followed by a size(?) on 4 bytes.

It seems those pointer might be there to "stitch" pieces of image together. Which would explain why some images are completely misaligned.. But that's not 100% sure yet.. However, I can hardly see what other purpose those pointer would have..

Here's how each pointer+length groups looks like :

C8150000 00010000 
|------| |------|
Pointer  Length

Some pointers can be null for some reasons, and still have a length right after them. But there must be at least one pointer after FRM_IN. Each of those blocks seems to be separated from eachothers by 4bytes of zeros..

Its also worth noting that in some cases, such as with Eevee, there are several more lines than needed, and several small values on 4 bytes scattered seemingly randomly between the pointer + length blocks and the FRM_BEG of the following frame block..

Moreover, we hit a wall that's gonna be a pain in the butt to get past.. The file monster.bin contains data about sprites, namely which ones to display from the m_ground.bin and probably m_attack.bin files. It also would maybe contain the portraits for every pokemons as well.

But, the issue is that every PKDPX subfiles are compressed, and its hard to tell which compression was used..

If anyone has any knowledge of PKDPX compressed files, or compression in NDS games in general, it would be great if you could tell us what you know about it !

Edited May 29, 2014 by psy_commando

psy_commando · June 3, 2014

Update:

First, on the topic of the sprites. I need to figure out how their height and width are stored.. I can't find anything as of now in the vicinity of any frames in the files. But then again, I never really worked with sprites before, I had barely an idea what palettes were for 2 weeks ago, and I can't tell if its possible their dimensions are implicit through the way they're stored ..

Then on a more encouraging note, I did a couple of string search on the executable binaries, and on the overlays, using "strings -a -e S -t x" on linux, and I got some interesting leads!

Overlays:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/interesting_stuff_overlays.txt

ARM9:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/interesting_offsets_arm9bin.txt

In the arm9 binaries, these are really interesting:

 92ae8 AT4PXj

 94b00 %sfile = '%s'  line = %5d
 94b1c %sProgPos info NULL
 94b34   Print  
 94b40 !!!!! Fatal !!!!!
 94b58 (NULL)
 94b64 EFFECT/effect.bin
 94b78 MONSTER/monster.bin
 94b8c BALANCE/m_level.bin
 94ba0 DUNGEON/dungeon.bin
 94bb4 MONSTER/m_attack.bin
 94bcc MONSTER/m_ground.bin
 94be4 file directory init %4d %4d %08x %s
 94d00 0123456789

"AT4PXj" at offset 0x92ae8, is one of the weird headers we found in some files.

And those are all mentions of files containing sub-files we think are compressed (PKDPX):

94b64 EFFECT/effect.bin

94b78 MONSTER/monster.bin

94b8c BALANCE/m_level.bin

94ba0 DUNGEON/dungeon.bin

94bb4 MONSTER/m_attack.bin

And then in overlay_0011.bin, there are a ton, and I mean A TON of function name strings of some sort! It could have to do with the internal scripting maybe ? Or maybe they're debugging leftovers ?

Here are a couple of examples (much more in the text file !):

 3a944 WaitSe
 3a94c se_Play
 3a954 Destroy
 3a95c se_Stop
 3a964 me_Play
 3a96c me_Stop
 3a974 WaitBgm
 3a97c item_Set
 3a988 MoveTurn
 3a994 WaitBgm2
 3a9a0 bgm_Stop
 3a9ac flag_Set
 3a9b8 CaseMenu
 3a9c4 script.c
 3a9d0 CaseText
 3a9dc bgm_Play
 3a9e8 SetBlink
 3a9f4 bgm2_Play
 3aa00 performer
 3aa0c SetHeight
 3aa18 CaseValue
 3aa24 bgm2_Stop
 3aa30 BranchBit
 3aa3c BranchSum
 3aa48 CancelCut
 3aa54 CaseMenu2
 3aa60 SetEffect
 3aa6c BranchEdit
 3aa78 WaitRandom
 3aa84 se_PlayPan
 3aa9c WaitEffect
 3aaa8 flag_Clear
 3aab4 MoveHeight
 3aac0 sound_Stop
 3aacc se_FadeOut
 3aad8 JumpCommon
 3aae4 CallCommon
 3aaf0 WaitFadeIn
 3aafc SwitchValue
 3ab08 MoveSpecial
 3ab14 DefaultText
 3ab20 BranchDebug
 3ab2c PauseEffect
 3ab38 SwitchLives
 3ab44 BranchValue

Now, I just have to figure out where the compressed files are loaded, and I still have no ideas how, given all the stuff I thought I learned about the way the NDS works seems to not really apply to this game... I also have no ideas what are overlays exactly, and I still haven't found anything about them on the web, besides what I already know.. I also cannot figure out a way to get, say, DeSmuMe's disassembler, even using the GDB stub, to break upon landing on an interesting piece of code, like the above strings..

And no, I don't have access to IDA.. Just in case someone would bring it up here as well

Again, any help with this would be very welcome ! It would probably take me a long time to figure out how to guess what kind of compression is being used on those if I have to figure it out myself, given my non-existant knowledge of ARM/THUMB assembly/structure and lack of experience with the DS !

On an unrelated note, I don't know if double posting in our own thread is tolerated here ? I've seen several others do it, and I haven't found any mention of it in the rules.

Because, I'm afraid if I continue to edit my previous post with new things, it will become huge, and it might just force me to annoy the mods every times it flags the post for moderation when I add or change a link, like it did several times up to now.. Just to clear things up.

Bond697 · June 3, 2014

the very top grouping and bottom 2 groupings in interesting_offsets_arm9bin.txt are from the sdk and the stuff from overlay 11 is function names and other stuff like that for their script interpreter. the overlay_0002 text is again sdk stuff and so is overlay_0000- both are for wireless/networking.

also, this: [sDK+NINTENDO:DWC3.1.30004.20081104.1700_DWC_3_1_PLUS4]

means it was build with sdk version 3.1 not useful for anything, but an interesting fact nonetheless.

psy_commando · June 3, 2014

Sounds good ! And yeah, the stuff in overlay 11 really looked that way, especially the "script.c" string

Overlay 11 looks almost like its what drives most of the game.. Well, at least, I guess..

Also, some of the others look like they're used for menus. like the buy menu, the bank, the storage, the swap shop, even the footprints mini-game. ARM is such a weird architecture

Btw, are you familiar with those ?

If so, would you happen to know if overlays actually map files from the ROM to the DS memory ?

Because, I read a couple of conflicting articles on that.. Some say you have to send commands to the ROM's register, to get the files, some others said that overlays map resources directly into the DS's memory.. Its all kinda ambiguous.

Bond697 · June 8, 2014

yeah, the overlays are loaded into the ds's memory, not mapped there from the rom. you can see where each is loaded with crystaltile. if you need to load your own or load others or something, you can hook some code and call the overlay load and unload functions that are sitting in the arm9 binary.

http://i.imgur.com/LC8fyuO.png

psy_commando · June 8, 2014

Alright, so I made some progress again.

After trying to beat up DDD on linux and gdb-multiarch to work with the gdb stub from DeSmuMe and do all I wanted to, I decided to go with some simpler methods.

I was able to work together a decent toolchain for debugging the game while its running, using the NDS emulator iDeaS, because it got a barebone debugger built-in which helps a lot, and using Cheat Engine to set data watchpoints and to search the memory.

I've found out that sprite files seems to undergo some sort of conversion when they're sent into memory. They keep their header mostly intact. However, the magic number changes from "SIR0"{0x53, 0x49, 0x52, 0x30} to "SIRO" {0x53, 0x49, 0x52, 0x4F}..

The offsets inside the header+file structure are all changed, so they're not relative to the start of the SIR header, but to the NDS RAM. And the files are oddly not aligned on 16 bytes anymore.. Though, it could be due to the emulator..

And the sprite file I was using actually grew after being loaded/processed into memory, from 0x4F50 bytes to 0x5230 bytes.. Thus, it probably gets processed and stored into a new, bigger file structure, for some really weird reasons.. It kinda doesn't make much sense... Hopefully there must be a good reason for that..

All the sprite files from pokemon used in a level are all stored one after the other, each with its own SIRO header and structure(the header is exactly similar to SIR0, but not sure about the rest of the structure, I haven't had the time to seriously look at it). They're fairly easy to find using cheat engine's array search and copy/pasting the values from an entire frame from the sprite files on disk.

yeah, the overlays are loaded into the ds's memory, not mapped there from the rom. you can see where each is loaded with crystaltile.

Yep, thanks, I got that part.

But, I was mainly wondering if as the overlays are loaded into memory, they trigger some code to map files from the nitro fs to the memory, or do it by themselves? Because, I read on some website that mentioned that overlays somehow mapped resources to the memory.. But to be honest it kinda left me a little perplex..

Its hard to tell at one point what's possible or not on hardware you got zero experience with..

if you need to load your own or load others or something, you can hook some code and call the overlay load and unload functions that are sitting in the arm9 binary.

Well honestly, I'm not there yet

Would it be as simple as replacing some instruction with a jump, or even inserting a jump instruction, that would jump to some function I'd add at the end of the arm9.bin binary ?

Honestly right now, what I'm really after right now is to find the functions that load stuff from the file system, and more importantly, what demangle/decompress files. But, outside of examining every single instructions in the binary, I have no real idea of how to achieve that. And even then, assembly isn't that obvious.. Well, there's the iDeaS debugger, but, its not that powerful.. It does have a "run to cursor" option though.

Do you know of any documents / tutorials for doing that kind of stuff ? Or did you just figure out how it works yourself with some prior knowledge of ARM processors ?

I'm willing to work for getting more knowledge thoug! And if you could just point me in the right direction I could stop abusing of your kindness ^^;

psy_commando · June 24, 2014

Alright, so I haven't made any progress regarding compression/mangling on PKDPX files this far.

But, I've been doing some more messing around and read a couple of articles on writing homebrew for the NDS. That proved invaluable in understanding and finding some of the things I was looking for!

http://www.dev-scene.com/NDS/Tutorials (those tuts gives so much details on how memory is mirrored and etc ! Its awesome !)

Namely:

- Sprites for characters are probably always multiples of 8 like 16x16, 32x32, thus why there is no information stored in the sprites for that ! Just by checking if we have 256 or 512 bytes we can tell whether an image is 16x16 or32x32.

- The pointer table thing after each frame's image data is some form of RLE compression, to save a few bytes.(basically, it saves space by storing that the same byte occurs X times at some place in the image, on a static 12 bytes instead of more. So ideally, you want to have more than 24 pixels with a similar color to use this[2 pixels per byte], or else you're losing space instead of gaining any! )

Also, I found out that there is a lot to learn from memory dumps made with desmume 0.9.9 (not 0.9.10 because there's a glitch with the dumper that makes it dump bytes until your HDD is full.. I learned the hard way! it made a file of 238 GB on my HDD under an hour, freezing the whole system with various out of memory errors and even leaving the task manager unable to load from the lack of memory! And it took 30 minutes before the system was even remotely responsive again )

And I mean that, instead of having to blindly look at assembly, the memory dumps will help figuring out how some of the data is stored, and what is the purpose of it, and what to look for in the actual files, or during execution !

Also, I found out a couple more things about the file format for the sprites. Many things that invalidate some previous hypothesis. I've updated the text file with the relevant data. I'm also trying to rename a few parts of the files to a meaningful name as I find what that section is for. So don't be surprised if some things aren't named the same way:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/CommonFileFormats.txt

I also attempted seeing a little better how the different pointer structures were working together. If anyone is interested in trying to understand how some of it is all linked without doing it themselves ! I used file #453 from m_ground.bin as reference.

Trying to figure out how these things work together!

============
DATABLOCK_G
============

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
G0         0x4E10         0x8   
[
  OFF     0x0            0x4            0x4D6C       Points to H0.
  LEN     0x4            0x4            0x8            Length array H0   
]

G1         0x4E18         0x8
[
  NULL
]

G2         0x4E20         0x8
[
  NULL
]

G3         0x4E28         0x8
[
  NULL
]

G4         0x4E30         0x8
[
  NULL
]

G5         0x4E38         0x8
[
  NULL
]

G6         0x4E40         0x8
[
  OFF      0x0           0x4            0x4DA0        Points to H1
  LEN      0x4           0x4            0x8           Length array H1
]

G7         0x4E48         0x8
[
  OFF      0x0           0x4            0x4DC0        Points to H2
  LEN      0x1           0x4            0x8           Length array H2
]

G8         0x4E50         0x8
[
  NULL
]

G9         0x4E58         0x8
[
  NULL
]

G10         0x4E60        0x8
[
  NULL
]

G11         0x4E68        0x8
[
  NULL
]

G12         0x4E70        0x8
[
  OFF      0x0           0x4            0x4DF0        Points to H3
  LEN      0x1           0x4            0x8           Length array H3
]

============
DATABLOCK_H
============

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
H0         0x4D6C         G0.LEN         
[
  0       0x0            0x4            0x0330        Points to I0
  1       0x1            0x4            0x0378        Points to I1
  2       0x2            0x4            0x03C0        Points to I2
  3       0x3            0x4            0x0408        Points to I3
  4       0x4            0x4            0x0450        Points to I4
  5       0x5            0x4            0x0498        Points to I5
  6       0x6            0x4            0x04E0        Points to I6
  7       0x7            0x4            0x0528        Points to I7
]

H1         0x4DA0         G6.LEN
[
  0       0x0            0x4            0x0570        Points to P0
  1       0x1            0x4            0x0594        Points to P1
  2       0x2            0x4            0x05B8        Points to P2
  3       0x3            0x4            0x05DC        Points to P3
  4       0x4            0x4            0x0600        Points to P4
  5       0x5            0x4            0x0624        Points to P5
  6       0x6            0x4            0x0648        Points to P6
  7       0x7            0x4            0x066C        Points to P7
]

H2         0x4DC0         G7.LEN
[
  0       0x0            0x4            0x0690        Points to Q0
  1       0x1            0x4            0x06E4        Points to Q1
  2       0x2            0x4            0x0738        Points to Q2
  3       0x3            0x4            0x078C        Points to Q3
  4       0x4            0x4            0x07E0        Points to Q4
  5       0x5            0x4            0x0834        Points to Q5
  6       0x6            0x4            0x0888        Points to Q6
  7       0x7            0x4            0x08DC        Points to Q7
]

H3         0x4DF0         G12.LEN
[
  0       0x0            0x4            0x0930        Points to R0
  1       0x1            0x4            0x09A8        Points to R1
  2       0x2            0x4            0x0A20        Points to R2
  3       0x3            0x4            0x0A98        Points to R3
  4       0x4            0x4            0x0B10        Points to R4
  5       0x5            0x4            0x0B88        Points to R5
  6       0x6            0x4            0x0C00        Points to R6
  7       0x7            0x4            0x0C78        Points to R7
]



============
DATABLOCK_I (unknown chunks of data over there)
============   

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
I0         0x330          0x48
[
]

I1         0x378          0x48
[
]

I2         0x3C0          0x48
[
]

I3         0x408          0x48
[
]

I4         0x450          0x48
[
]

I5         0x498          0x48
[
]

I6         0x4E0          0x48
[
]

I7         0x528          0x48
[
]


============
DATABLOCK_P (unknown chunks of data over there) At OFF_P
============

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
P0         0x570          0x24
[
]

P1         0x594          0x24   
[
]

P2         0x5B8          0x24   
[
]

P3         0x5DC          0x24   
[
]

P4         0x600          0x24   
[
]

P5         0x624          0x24   
[
]

P6         0x648          0x24   
[
]

P7         0x66C         0x24   
[
]


============
DATABLOCK_Q (unknown chunks of data over there) Is at OFF_Q
============   

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
Q0         0x690          0x54
[
]

Q1         0x6E4          0x54
[
]


Q2         0x738          0x54
[
]

Q3         0x78C          0x54
[
]

Q4         0x7E0          0x54
[
]

Q5         0x834          0x54
[
]

Q6         0x888          0x54
[
]

Q7         0x8DC          0x54
[
]



============
DATABLOCK_R (unknown chunks of data over there) Is at OFF_R
============   

ID         Offset         Length         Value         Description
---        ------         ------         ------        -----------
R0         0x930          0x78
[
]

R1         0x9A8          0x78
[
]

R2         0xA20          0x78
[
]

R3         0xA98          0x78
[
]

R4         0xB10          0x78
[
]

R5         0xB88          0x78
[
]

R6         0xC00          0x78
[
]

R7         0xC78          0x78
[
]

Oh, and for those of you using HexEdit 4.0, the one from hexedit dot com, I made a wip template for parsing a few parts of a sir0 container containing sprite data ! Its very basic right now, and the entries are not very well labeled, but it completely beats parsing completely manually every new sprite files you want to look at !

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/_sir0.xml

Just drop that xml file into your "%APPDATA%/ECSoftware/HexEdit/" folder then start hexedit, go to the menu "Template->Open File Type->SIR0 File" then go again to the menu "Template->Split Window" to open the file navigation sidebar.

For the last step, click on "Template->Design Mode" if its already checked, given design mode won't let you navigate the file.

With this you can navigate a list of elements that were automatically recognized by the template. Just double click on the elements with the little blue boxes next to them, and you'll be brought to the location in the file where that particular element is located !

I'll try to keep this thread updated as I find new information!

Edited June 24, 2014 by psy_commando

psy_commando · August 11, 2014

Update !

Its been a while since I updated this thread, but I didn't have much to show.. Mainly failed attempts..

I updated all my existing notes posted in here, and will keep doing so. But besides that, here's what I've been doing.

Pokemon Portraits:

Not much with the portraits, beside having been able to extract them and their palette from the emulator memory, to confirm resolution and little format specific details.

I also compiled Desmume myself, and inserted a couple of lines of custom code to attempt logging everything that was read from the gamecart, and dump it all to a file, but it wasn't all that conclusive. What I did was, saving a state at the companion select screen and then changing the cursor from one pokemon to the other, causing the pokemon portrait to change. And I reloaded the state later on when I needed to do more tests.

However, even by doing that, I couldn't find a link between what was being read, and the pokemon portrait.. The game seemed to be loading mainly bits of scripts, various numbers, but no obvious image data, or obvious PKDPX file content.

I also logged the rom offsets being read, and used that to identify what file was being read. But most of the time, the location that were read were very random. Which I suspect might be because there's a different starting offset for the raw rom, however, using tinke, its kinda hard to tell whether it takes the rom header into account in the shown offset for each file, or if the offset is relative to the raw rom.. I'll probably investigate this further later on.. But I pretty much burned myself out on that issue, so I went on to try to understand other aspects of the game..

Here's a differential patch with the logging code I wrote. Its a quick and dirty singleton to log things and group them together as long as each reads are adjacent in the rom. Note that, the log file will be overwritten at each emulator restarts. But feel free to change it however you want ! You have to apply it on the file in "../src/addons/slot1_retail.cpp" in the source folder for desmume, and then compile it. You'll have to uncomment line 227, for the logger to actually log anything. And be warned that it slows the game down somewhat.

Here it is :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/slot1_retail.cpp.debugcartreads.diff

(if you don't have winmerge, or tortoisediff or anything like that, you can just copy the code lines starting with ">" at the line number placed above each subsections. Lines beginning with "<" means lines that were replaced, so don't copy those! )

Music:

I also began taking a look at the music format for PMD2, to hopefully write a 2 way converter eventually.

It uses the SMD format.. And the NDS community doesn't seem to have much tools supporting it.. So I analyzed the files from scratch, and I was able to get a few conclusions and hypothesis.

It seems SMD files are very close to your typical MIDI files in overall structure! Thus by drawing parallels with midi, its been surprisingly easy to guess a lot of things from these files. The tracks are basically a collection of events, much like midi events but much more optimized to not waste any space! The shortest event can fit on a single byte, while the longest can fit on 4 bytes. Each events can have a delta-time prefixed, or not. And each note playing events seems to use a "delta-pitch"(probably in terms of the octave to play the note in) of some sort between the last note and itself.

Most of the SWD files that share the name of an SMD file basically contain data about the samples the game needs to load to play this track. It doesn't seem to actually contain the samples themselves. It probably maps notes to samples and set samples loop points and etc..

It looks like its possible to map several samples to an instrument, like the drum track in the bgm0003.smd file shows(its the last track before the eoc chunk, I think).

However, the bgm.swd file seems to maybe contains the actual samples. Given its size, and generic name, and content. So I'm guessing smd files can contain references to other smd files, and/or sound samples.. But that's not sure..

I made a separate file with my notes on the subject here :

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/PMD2_MusicAndSoundFormats.txt

They're not very detailed yet, and a WIP. Like all my previous notes basically

But I was able to manually edit tracks and music files to play different notes, change instrument(sample), change the track wide pitch, insert silences, insert extra notes, change the tempo, remove tracks, add tracks, etc..

As long as you update the size value for the track, and for the whole file in the header, along with the number of tracks, and put the padding between tracks correctly, the game will take pretty much any music track you throw at it ! With a varying amount of success of course.. Especially, if you set the instrument of a track to an instrument not defined in the accompanying SWD file. That would result in a very silent song

You can also replace any music file, with any other music file, simply by renaming the .smd, and its accompanying .swd with the name of the target file !

For example I deleted the original bgm0002.smd and .swd file. (bgm0002 is the track for the top menu, the place with the "new game" and/or "continue" menu appears)

Then copied and renamed the bgm0003.smd and bgm0003.swd files as bgm0002. smd and bgm0002.swd. And then repacked the rom, ran it, and the music that played at the top menu was the training dojo dungeon music !

This is very handy for testing changes made to a song, given you can play it directly at the top screen!

Anyways that's it for now ! I hope this info will be useful ! As usual if anyone finds anything interesting or has any tidbits of info to share, feel free to do so!!

I could really use the help with anything really!

Andibad · September 21, 2014

is great info , thanks!

Pokemon Mystery Dungeon - Gates to Infinity used SIR0 Format too ... is seems bit different but overall is still same format on nds one. And also i find interesting too on it.

psy_commando · September 21, 2014

No problems ! I really gotta update the thread though

The most up-to date info is in the text files I linked!

Me and a couple of others have been working on reversing a ton of formats from PMD2, and we even had a huge, huge, breakthrough recently, which I'm going to write more about here, once I get the time.

And that's interesting ! But probably that the other formats must be wildly different no ?

How did you get to the actual content of the game though ? I mean, I thought 3ds games were encrypted or something like that? (I'm not even sure about that.. Just random things I've heard ^^; )

Andibad · September 22, 2014

And that's interesting ! But probably that the other formats must be wildly different no ?

How did you get to the actual content of the game though ? I mean, I thought 3ds games were encrypted or something like that? (I'm not even sure about that.. Just random things I've heard ^^; )

yes, 3ds games is encrypted but right now you can decrypt that using a 3ds with firmware v4.x.

here some file on it :

psy_commando · September 23, 2014

Interesting.

Too bad I got a fully updated 3DS.. I even just got the game on the eshop, because of the sale

I'm guessing that FARC header contains some kind of table of content.

Andibad · September 25, 2014

Interesting.
Too bad I got a fully updated 3DS.. I even just got the game on the eshop, because of the sale

I'm guessing that FARC header contains some kind of table of content.

FARC file is like PT PF PC file on a/0/0/7 pokemon xy models, is a simple archive with 0x80 size header. but is still used SIR0 file for store text, sprite and some model. but some model stored on BCH iirc.

psy_commando · September 29, 2014

Its a little odd to put a container within a container though

But anyways.

Update:

The big breakthrough we had was that we finally figured out the compression on PKDPX and AT4PX files !

Its all thanks to someone named Zhorken really!

That person made a script for extracting portraits from the "/FONT/kaomado.kao" file, and it turns out, the portraits are compressed within AT4PX containers each preceded by their 16 color RGB24 palette.

So the code from that script allowed us to figure out that all AT4PX file in PMD2 used that compression scheme. But then it gets better.

I noticed that the PKDPX files had a largely similar header, with only an extra 2 bytes at the end. Those two bytes are actually the continuation of the bytes containing the decompressed file size. It is contained on 2 bytes in the AT4PX header, but on 4 bytes in the PKDPX header for some very odd reasons..

Well it turns out that the decompression code for the AT4PX files works flawlessly on PKDPX files as well ! (by adjusting for the extra 2 bytes in the header of course!)

I spoke with Zhorken, and they said it was ok to, well, learn from what's written there and make tools from what we learned from that script. So I wrote my own version of the algorithm in C++, and I'm going to try to write a compressor at one point, unless someone else would like to help with that! Given that, it looks like it would be really complicated to compress data exactly like they did.. And I never wrote any lossless data compression algorithm in my entire life

However, there is a way to implement a minimal version of that compression if all else fails. However, it would actually increase the size of the "compressed" file by its size divided by 8.

Essentially :

    CompressedFileSize = FileSize + (FileSize / 8)

Which is kinda inefficient to say the least ^^;

Anyways, here is the python script Zhorken wrote:

https://github.com/Zhorken/kaomado/blob/master/kaomado.py

(Just keep in mind that Python has many little particularities that makes the code do things that aren't very obvious for people not used to its syntax.)

And once I clean up and optimize my C++ code I'll publish the source too, along with another utility for handling PKDPX, AT4PX, and the kaomado.kao file.

Also, for now, until we figure out the actual name of the compression, I'll be referring to it in my notes and code as "PX compression", mainly given the 2 main file formats using it both end up in "PX".

I added this to my notes on dropbox about it(the dropbox version is the most up to date!):

(Just be wary of typos and also that I might have written something that's completely wrong somewhere in there ! So be sure to check the Python script Zhorken made as well ! )

====================================================================================================
                                     "PX" Compression Format
====================================================================================================
A custom compression method which appears to be a fancy version of an RLE algorithm, with some
touches from LZ compression added in I guess..

Its used by both PKDPX and AT4PX file format. It relies on 9 nybble flags referred to as 
"control flags" listed in those headers for handling a couple of special cases, and for the rest it 
basically works by decoding a "command byte" that indicates what to do until the next 
"command byte". 

Note that, those nybble flags are always "0x" where "x" is an hexadecimal value 
from 0 to F. Also, note that, these flags are computed on a file by file basis! They're 
"tailor-made" for each individual file.

To make understanding things easier, you could imagine that the bytes that we read are read from
a FIFO queue, and then that we put the decompressed bytes into a FILO stack. 
(Even if in practice those are probably too slow to use in this case!)

To decode the command byte, we simply check the value of all of its 8 bits, one at a time. 
From the highest bit to the lowest. 
So lets say we're in a loop that isolate the value of a particular bit each turn. Something like:

   uint8 mask = 0x80
   loop while( mask > 0 )
   {
       uint8 bitval = mask bitwiseAND cmdbyte

       ... // Handle the cases written below

       mask = mask >> 1
   }

On each turns of that loop, we'll check the value of the current bit, and depending on its value,
we'll decide what to do based on these criteria:

   1)  If the BIT we've isolated in the command byte is 1, then we pop the next BYTE from the input 
       queue, and push it as-is into the output stack!

   2)  Otherwise if the BIT we've isolated is 0, we'll pop the next BYTE from the input queue.
       Then, we'll try to find whether the high nybble of the BYTE we just read is the same value
       one of our control flags. 
       We also want to keep the value of the low nybble for later.
       We'll refer to the high nybble and low nybble we got here as "nbhigh" and "nblow"
       respectively.

       2.1) If one of our control flag match the high nybble of the BYTE we just read named 
            "nbhigh", then it means we're inserting a pattern of 4 nybbles / 2 bytes into the 
            stream. 
            We'll refer to the index of the control flag we got as "ctrlflagindex".

           2.1.1) If the index of the control flag "ctrlflagindex" is the first one at index 0, 
                  we calculate the value of the 2 bytes we'll push into the output stack this way:

                      byte1 = nblow << 4 bitwiseOR nblow
                      byte2 = byte1

                  We just insert those, and we're done! 

           2.1.2)  Otherwise, for any other "ctrlflagindex" flag index, we have a few other cases
                   to take into account.
                   However, they all have in common using the value of "nblow". 
                   We'll add a new variable called "basenybbleval" to make things easier to 
                   understand. And we'll put the value of "nblow" into it right away.

                   A)  If "ctrlflagindex" is 1. We increment the value of all 4 nybbles. Or in our
                       case, the value we use to store the base value of all nybble 
                       "basenybbleval".

                           basenybbleval = basenybbleval + 1

                   B)  Otherwise, if "ctrlflagindex" is 5. We decrement the value of all 4 nybbles. 
                       Or in our case, the value we use to store the base value of all nybble 
                       "basenybbleval".

                           basenybbleval = basenybbleval - 1

                   From this point forward, lets put the value of "basenybbleval" into 4 separate
                   variables, each holding the value of the individual 4 nybbles.
                   Lets just name them, "nybble0", "nybble1", "nybble2", "nybble3", and put the 
                   current value of "basenybbleval" into all of them.
                   (You should use an array, as they're only named like this for clarity's sake)

                   A)  If "ctrlflagindex" is between 1 and 4 inclusively. Then substract 1 from 
                       the nybble corresponding to the value of (ctrlflagindex - 1). 
                       For example, if (ctrlflagindex - 1) is 0, we subtract 1 from "nybble0". If its
                       3, we subtract 1 from "nybble3", and so on!

                   B)  Otherwise, we add 1 to the nybble corresponding with the value of 
                       (ctrlflagindex - 5). If (ctrlflagindex - 5) is 0 we add one to "nybble0" 
                        and so on!

                   Now, all that is left to do is to assemble the 4 nybbles into 2 bytes, and push
                   those 2 bytes into the output stack!

                       byte1 = nybble0 << 4 bitwiseOR nybble1
                       byte2 = nybble2 << 4 bitwiseOR nybble3

                   We just push those two into the output stack, and we're done !

       2.2) If none of our control flags match the high nybble of the BYTE we just read named 
            "nbhigh", it means we're copying a sequence of bytes from our decompressed output.

            To figure out the offset from the end of our output "stack" where we begin reading 
            the bytes sequence to copy, we'll need to read the next byte from the input "queue" 
            and do the following operations:

               uint16 offset = -0x1000 + nblow << 8 bitwiseOR inqueue.pop()

            "offset" will contain a negative integer value smaller than the current size of the 
            output queue. 
            Just go from the end of the output queue towards the beginning of it, 
            by the absolute value of "offset". This will get you the beginning position of the 
            sequence of bytes to append to the output later on. 
            We'll call this position "seqbeg".

            Then, to get the amount of bytes to copy starting from "seqbeg", we take the value
            of "nbhigh" and add 3. 

            Knowing this, we can now copy the sequence.


After going through those conditions, we know that the next BYTE we pop from the input queue will be
a new command byte guaranteed! So we just have to repeat this loop using this new command byte!

This sums it pretty much!

To sum it all up, the reason this is so huge is that, figuring this compression just opened up the door to editing nearly anything in the game files !!! We just need to reverse simple file structures now!!!

But yeah, we seriously owe a lot to Zhorken for that find! So, if anyone uses this code and gets the chance, be sure to say thanks to Zhorken !

EDIT:

I forgot those might actually be really handy to understand the above ^^;

==========================================================================================
                                   AT4PX:
==========================================================================================
AT4PX containers seems to contain image data in a compressed form. Namely, character portraits, 
map tiles, backgrounds. They usually don't contain their image's color palette however, as those
are often stored outside of the AT4PX container, either alongside, elsewhere, or not stored at all
in the case of a non-indexed bitmap image. 
(I haven't seen any bitmap/raster non-indexed images yet though, just speculating)


Offset:            Length:            Endian:      Description:
-------            -------            -------      -----------------------------------
0x00               0x5                -            Magic Number "AT4PX" {0x41, 0x54, 0x34, 0x50, 0x58}
0x05               0x2                little       (FLEN)Total compressed file length
0x07               0x9                -            List of control flags (more on these below)
0x10               0x2                little       Byte size of the decompressed data.
0x12               (FLEN - 0x12)                   Compressed image data.
[
   0x0                                             "PX" compressed data here.
   ...
]
-------            -------            -------      -----------------------------------
Total:             (FLEN)

Control Flags:
---------------
Those tend to change with each files. They're 4 bits values stored in the lower nybble of each 
individual 9 bytes that are used to handle some special cases while decoding some "Command Byte".
The high nybble is always 0.
They're picked on a per-file basis.

==========================================================================================
                                   PKDPX:
==========================================================================================
Generic compressed data container. 
Uses "PX" compression, just like the AT4PX format, except that PKDPX can contain another
file/container as its compressed data.
Identical to AT4PX besides the magic number, and the extra 2 bytes for the header!



Offset:            Length:            Endian:      Description:
-------            -------            -------      -----------------------------------
0x0                0x5                Big          Magic Number {0x50, 0x4B, 0x44, 0x50, 0x58} "PKDPX"
0x05               0x2                little       (FLEN)Total compressed file length
0x07               0x9                -            List of control flags (more on these below)
0x10               0x4                little       Byte size of the decompressed data.
0x14               (FLEN - 0x14)                   Compressed image data.
[
   0x0                                             "PX" compressed data here.
   ...
]
-------            -------            -------      -----------------------------------
Total:             (FLEN)

Control Flags:
---------------
Those tend to change with each files. They're 4 bits values stored in the lower nybble of each 
individual 9 bytes that are used to handle some special cases while decoding some "Command Byte".
The high nybble is always 0.
They're picked on a per-file basis.

Edited October 1, 2014 by psy_commando

evandixon · October 1, 2014

Good work! We may soon be able to actually edit the ROM. I look forward to the release of your tool, so we can find out what stores what.

In other news, I mapped out which song each file in SOUND/BGM contains. Each number below is the XXXX in "bgmXXXX.smd" and "bgmXXX.swd".

0000: Ambiance: Waves (high heavy)
0001: Explorers of Time Opening
0002: Menu
0003: Marowack Dojo
0004: Sentry Duty
0005: Mission Success
0006: Personality Test
0007: Wigglytuff's Guild
0008: Wigglytuff's Guild Remix
0009: Treasure Town
0010: Sharpedo Bluff
0011: Monster House
0012: Keckleon Shop in Dungeon
0013: Thief
0014: End of Dungeon
0015: Boss Battle
0016: Battle Against Dialga
0017: Battle Against Dusknoir
0018: Battle Vs. Regi's
0019: SE: Failure
0020: SE: Mission Success
0021: Beach Cave
0022: Drenched Bluff
0023: Mt. Bristle
0024: Waterfall Cave
0025: Apple Woods
0026: Craggy Coast
0027: Side Path
0028: Mt. Horn
0029: Foggy Forrest
0030: Steam Cave
0031: Upper Steam Cave
0032: Amp Plains
0033: Far Amp Plains
0034: Northern Desert
0035: Quicksand Cave
0036: Quicksand Pit
0037: Crystal Cave
0038: Crystal Crossing
0039: Chasm Cave
0040: Dark Hill
0041: Sealed Ruin
0042: Deep Sealed Ruin
0043: Dusk Forest
0044: Deep Dusk Forest
0045: Random Dungeon Music #2/Former Marowack Dojo
0046: Brine Cave
0047: Deep Brine Cave
0048: Hidden Land
0049: Hidden Highland?
0050: Temporal Tower
0051: Temporal Spire
0052: Mystifying Forest
0053: Blizzard Island Rescue Medley
0054: Surrounded Sea
0055: Random Dungeon Music #1
0056: Aegis Cave
0057: Concealed Ruins
0058: Mt. Travail
0059: The Nightmare
0060: Miracle Sea
0061: Treeshroud Forest
0062: Dark Crater
0063: Deep Dark Crater
0064: Explorers of Time/Darkness Opening
0065: SE: Success
0066: SE: Unused?
0067: SE: Unused?
0068: SE: Unused?  Failure?
0069: On the Beach at Dusk
0070: Goodnight
0071: Unused: Goodnight Remix
0072: At the End of the Day (Goodbye Dusknoir)
0073: Guildmaster Wigglytuff
0074: Growing Anxiety
0075: The Power of Darkness
0076: Oh No!
0077: Time Gear
0078: Time Gear Remix
0079: I Saw Something Again... (Dimensional Scream)
0080: In the Future (Jail Cell)
0081: Planet's Paralysis
0082: Through the Sea of Time
0083: In the Hands of Fate
0084: Like a Fountain/Time Restored
0085: Don't Ever Forget...
0086: A Wish for Peace
0087: On the Beach at Dusk (Again, but instruments sound minutely different, one of them is more echoey)
0088: Memories Returned
0089: Ending Theme Intro
0099: Lower Spring Cave
0100: Ambiance: Waves (Mid heavy)
0101: Ambiance: Heavy Thunderstorm
0102: Ambiance: Light Thunderstorm
0103: Ambiance: Heavy Tremors
0104: Ambiance: Light Tremors
0105: Ambiance: Heavy Tremors?
0106: Ambiance: Bubbling Water
0107: Ambiance: Temporal Pinnacle (Heavy Wind with Clock Ticking)
0108: Temporal Pinnacle
There is no 109 for some reason.
0110: SE: Stomping?
0111: SE: Constant Ringing (Unused maybe?)
0112: Ambiance: Electricity
0113: SE: "B_SE_MAIN19_HAD" (I have no idea what this is)
0114: Ambiance: Light Electricity/Remains of Passage of Time?
0115: Ambiance: Campfire
0116: Ambiance: Lava/Bigger Fire?
0117: Ambiance: Shimmer (Final Marowack Dojo/That shadow was Wigglytuff!)
0118: Ambiance: Rainbow Stoneship?
0119: Ambiance: Rainbow Stoneship Overload
0120: Ambiance: "B_EVENT_JIKUU_0" Low Pulsating?
0121: Have to Get Home
0122: Further Away
0123: Palkia's Onlslaught!
0124: "B_DUN_HASSAM_01" (Silence)
0125: Ambiance: Waterfall
0126: Ambiance: Eating
0127: The Dark Future Chord
0128: Ambiance: Thunderstorm at Sea
0129: Peliper Island
0130: Cut to Title (After beach cave)
0131: Heartwarming
0132: Down a Dark Path
0133: Growing Anxiety
0134: Team Skull
0135: Sympathy
0136: Beyond the Dream
0137: Air of Unease
0138: One for All, All for One!
0139: Boulder Quarry
0140: Spring Cave Depths
0141: Star Cave
0142: Deep Star Cave
0143: Limestone Cavern
0144: Deep Limestone Cavern
0145: Random Dungeon Theme #3
0146: Fortune Ravine
0147: Fortune Ravine Depths
0148: Barren Valley
0149: Dark Wasteland
0150: Spacial Cliffs
0151: Dark Ice Mountain
0152: Icicle Forrest
0153: Vast Ice Mountain
0154: Vast Ice Mountain Peak
0155: Sky Peak Forrest
0156: Sky Peak Prarie
0167: For a New Life
0168: Living Spirit
0169: Proud Accomplishment
0170: In the Morning Sun
0171: A New World
0172: Thoughts for Friends
0173: Life Goes On (Ending)
0174: It's Not a Miracle
0175: A Message on the Wind
0176: A Fun Exploration
0177: Shaymin Village
0178: Team Charm's Theme (Part 1: Suspense)
0179: Ambiance: Lava
0180: Ambiance: Lava (lower pitch)
0181: Ambiance: Waves
0182: Ambiance: Grass
0183: Ambiance: Wet Cave
0184: Ambiance: Campfire (again?)
0185: Ambiance: Campfire and Water
0186: Ambiance: Rain
0187: Ambiance: Wind
0188: Ambiance: "Source"?
0189: Ambiance: Ruins
0190: Ambiance: Jungle
0191: Ambiance: Crag
0192: Ambiance: Memory (The shadow was Wigglytuff!)
0193: Ambiance: Cave
There are no files from 194 to 198 for some reason
0199: Team Charm's Theme (Part 3: The Battle) [Redundant because of #201, although some instruments sound a little different]
0200: Team Charm's Theme (Two notes apparently taken out of context)
0201: Team Charm's Theme (Part 2-3: Lead AND The Battle)

I also have the internal names of each track:

00: B_ENV_BEACH_01
01: B_SYS_P3_OPENIN
02: B_SYS_MENU.SMD
03: B_SYS_TRAINING
04: B_SYS_MINIGAME.
05: B_SYS_EVENTCLEA
06: B_SYS_SHINDAN_0
07: B_MAP_GUILD_01.
08: B_MAP_GUILD_02.
09: B_MAP_TOWN_01.S
10: B_MAP_HOME_01.S
11: B_SYS_MONSTERHO
12: B_SYS_SHOP.SMD 
13: B_SYS_STEEL.SMD
14: B_EVENT_BOSSFLO
15: B_EVENT_BOSS_01
16: B_EVENT_BOSS_02
17: B_EVENT_BOSS_03
18: B_EVENT_BOSS_04
19: B_ME_GAMEOVER.S
20: B_ME_GAMECLEAR.
21: B_DUN_KAIGANNOD
22: B_DUN_SHIMETTAI
23: B_DUN_TOGETOGEY
24: B_DUN_TAKITSUBO
25: B_DUN_RINGONOMO
26: B_DUN_ENGANNOIW
27: B_DUN_YOKOANA_0
28: B_DUN_TSUNOYAMA
29: B_DUN_NOUMU_01.
30: B_DUN_NESSUI_01
31: B_DUN_NESSUI_02
32: B_DUN_EREKIHEIG
33: B_DUN_EREKIHEIG
34: B_DUN_NISHINOSA
35: B_DUN_RYUUSA_01
36: B_DUN_RYUUSA_02
37: B_DUN_SUISYOU_0
38: B_DUN_DAISUISYO
39: B_DUN_KUUKAN_01
40: B_DUN_KURAYAMI_
41: B_DUN_FUUINNOIW
42: B_DUN_FUUINNOIW
43: B_DUN_KURONOMOR
44: B_DUN_MORINOTAK
45: B_DUN_KIZAKINOM
46: B_DUN_ISONODOUK
47: B_DUN_ISONODOUK
48: B_DUN_MABOROSHI
49: B_DUN_MABOROSHI
50: B_DUN_JIGENNOTO
51: B_DUN_JIGENNOTO
52: B_DUN_SHINPI_01
53: B_DUN_HASSAM_01
54: B_DUN_TOZASARET
55: B_DUN_KISEKINOU
56: B_DUN_BANNIN_01
57: B_DUN_KAKUSARET
58: B_DUN_SYUGYOUYA
59: B_DUN_AKUMUNONA
60: B_DUN_SORANOSAK
61: B_DUN_SORANOSAK
62: B_DUN_YAMINOKAK
63: B_DUN_YAMINOKAK
64: B_SYS_TITLE_02.
65: B_ME_MINIGAME_0
66: B_ME_MINIGAME_0
67: B_ME_MINIGAME_0
68: B_ME_MINIGAME_0
69: B_EVENT_MEETING
70: B_EVENT_CALMLY_
71: B_EVENT_CALMLY_
72: B_EVENT_CALMLY_
73: B_EVENT_COMICAL
74: B_EVENT_FEAR_01
75: B_EVENT_FEAR_02
76: B_EVENT_TENSION
77: B_EVENT_TIMEWHE
78: B_EVENT_TIMEWHE
79: B_EVENT_JIKUU_0
80: B_EVENT_ROUGOKU
81: B_EVENT_ANKOKU_
82: B_EVENT_SEPARAT
83: B_EVENT_SEPARAT
84: B_EVENT_PEACEFU
85: B_EVENT_SEPARAT
86: B_EVENT_SEPARAT
87: B_EVENT_SEPARAT
88: B_EVENT_SEPARAT
89: B_EVENT_STAFFRO
90: B_EVENT_STAFFRO
91: B_EVENT_EPIROGU
92: B_EVENT_TITLECA
93: B_EVENT_TITLECA
94: B_DUN_P3_P1_CHI
95: B_DUN_P3_P1_GEN
96: B_DUN_P3_P1_MYS
97: B_DUN_P3_P1_GEN
98: B_DUN_P3_P1_CHI
99: B_DUN_P3_P1_CHI
100: B_ENV_BEACH_01.
101: B_ENV_STORM_01.
102: B_ENV_STORM_02.
103: B_ENV_QUAKE_01.
104: B_ENV_QUAKE_02.
105: B_ENV_QUAKE_03.
106: B_ENV_MAGMA_01.
107: B_ENV_LASTBOSS_
108: B_ENV_LASTBOSS_
110: B_SE_MAIN26_LIG
111: B_SE_MAIN26_LIG
112: B_EVENT_MIKARUG
113: B_SE_MAIN19_HAD
114: B_SE_MAIN19_ELE
115: B_EVENT_TAKIBI.
116: B_EVENT_TAKIGI.
117: B_SE_MAIN23_KAK
118: B_SE_MAIN25_ISH
119: B_SE_MAIN25_ISH
120: B_EVENT_JIKUU_0
121: B_EVENT_SEPARAT
122: B_EVENT_SEPARAT
123: B_EVENT_BOSS_05
124: B_DUN_HASSAM_01
125: B_ENV_WATERFALL
126: B_ENV_SYOKUJI_0
127: B_ENV_DRONE_01.
128: B_SE_MAIN05_DAK
129: B_MAP_ISLAND.SM
130: B_EVENT_TITLECA
131: B_EVENT_CALMLY_
132: B_EVENT_FEAR_03
133: B_EVENT_FEAR_04
134: B_EVENT_TENSION
135: B_EVENT_P3_CALM
136: B_EVENT_P3_IMAG
137: B_EVENT_P3_MYST
138: B_MAP_P3_GUILD_
139: B_DUN_P3_GANSEK
140: B_DUN_P3_GENSEN
141: B_DUN_P3_HOSHI_
142: B_DUN_P3_HOSHI_
143: B_DUN_P3_SHONYU
144: B_DUN_P3_SHONYU
145: B_DUN_P3_ZAIHO_
146: B_DUN_P3_ZAIHO_
147: B_DUN_P3_ZAIHO_
148: B_DUN_P3_KOKATS
149: B_DUN_P3_KURAGA
150: B_DUN_P3_KUUKAN
151: B_DUN_P3_KURAYA
152: B_DUN_P3_HYOCHU
153: B_DUN_P3_HYOUZA
154: B_DUN_P3_HYOUZA
155: B_DUN_P3_SORA_0
156: B_DUN_P3_SORA_0
157: B_DUN_P3_SORA_0
158: B_DUN_P3_SORA_0
159: B_MAP_P3_CAFE.S
160: B_EVENT_P3_DANC
161: B_EVENT_P3_WICK
162: B_EVENT_P3_PUPU
163: B_EVENT_P3_PUPU
164: B_EVENT_P3_PUPU
165: B_EVENT_P3_CHAR
166: B_EVENT_P3_CHAR
167: B_EVENT_P3_NEWL
168: B_EVENT_P3_SOUL
169: B_EVENT_P3_PRID
170: B_EVENT_P3_SUNR
171: B_EVENT_P3_NEWW
172: B_EVENT_P3_NOTM
173: B_EVENT_P3_S09S
174: B_EVENT_P3_DESI
175: B_EVENT_P3_WIND
176: B_EVENT_P3_ADVE
177: B_EVENT_P3_SHAY
178: B_EVENT_P3_CHAR
179: B_ENV_P3_LAVA_0
180: B_ENV_P3_LAVA_0
181: B_ENV_P3_WAVE_0
182: B_ENV_P3_GRASSY
183: B_ENV_P3_CAVE_0
184: B_ENV_P3_FIRESP
185: B_ENV_P3_FIREWA
186: B_ENV_P3_RAIN_0
187: B_ENV_P3_TOP_01
188: B_ENV_P3_SOURCE
189: B_ENV_P3_RUINS_
190: B_ENV_P3_JUNGLE
191: B_ENV_P3_CRAG_0
192: B_ENV_P3_MEMORY
193: B_ENV_P3_CAVE_0
199: B_EVENT_P3_CHAR
200: B_EVENT_P3_CHAR
201: B_EVENT_P3_CHAR

It appears that the recycled tracks from Red/Blue Rescue Team used in the special episodes are located elsewhere. The only recycled tracks in these lists have some kind of difference (be it medley, remix, etc). Judging purely from the files missing from Explorers of Time, it is possible these tracks are in SOUND/SE/ev_eXX.sed, but that's pure speculation. Also, in Explorers of Time, the last track is 134 (still skipping 109).

psy_commando · October 1, 2014

Wow, thanks, that's awesome!

It would have taken me ages to do that !

Especially since I haven't finished the episodes or post-game completely and wanted to do so before spoiling the music ^^;

Do you mind if I add that to my notes ? Leaving a mention that you did all the work of course!

And as for the missing tracks, I know that Blue Rescue Team uses a more common music and sound format, SDAT and SSEQ I think. So its possible that they're either stored elsewhere, or maybe they're re-using a single smd with all those songs in it, and just play a specific part of it ?

And the guy I'm working with, Nerketur Kamashi has managed to actually script an entirely new cutscene, over the intro cutscene I think, into the game a while ago!

He's really good and familiar with the scripting system of the game! So we actually have already "edited the rom" more or less !

And I wanted to wait until I got the re-compression worked out before releasing a tool, but I can totally just post a quick little AT4PX and PKDPX extractor in the meantime! That might even actually help me find potential bugs with it ! I'll try to get this done by tomorrow or later this week. All the code is already done, its just a matter of improving the error handling!

evandixon · October 2, 2014

I don't mind if you add that to my notes. We're all working toward a common goal: to be able to edit Pokémon Mystery Dungeon. That's partially why I've been tinkering with Sky Editor to make it a ROM editor too. That's how I managed to do all that research. It made one ROM per sound file for quick listening. (Which takes up about 16GB on my hard drive... should probably delete all that soon...) My current goal for Sky Editor is to make it be able to edit all known aspects of the ROM, despite me knowing how to edit almost none of it myself. Perhaps I can make it a suite of the future tools this research yields.

I'm sure I saw Nerketur's script editor before, but it was just yesterday that I really saw it and admired its potential. I think the biggest thing it's missing (beside being able to the script items in the left textbox - that's probably coming soon), is a button to run the ROM immediately, to test the changes just made. That's why I just added the ability for Sky Editor to repack a loaded ROM and run it using the emulator associated with .nds files. The only thing that may be missing would maybe be an AR code to immediately load a dungeon/the associated script, but that's probably beyond everyone's immediate area of focus.

You may or may not have seen this, but each folder in SCRIPT is the name of a dungeon, the same name that appears in MAP_BG and various other places. I speculate that the MAP_BG contains images of all the dungeons themselves, probably similar to the images found at The Spriter's Resource here and here. Perhaps your decompression tool will make it much simpler to view (and change) those.

Unless you have any objections or anything, when you release your tool, I'll probably wire it into Sky Editor and see what I can have the two of them extract. Since I already have ndstool wired in to extract all the files, it shouldn't be too much trouble. Looking forward to your release, and the potential it brings!

psy_commando · October 2, 2014

You know, we're actually working on a rom editor too ^^;

Some kind of "IDE", to edit as much aspect of the game as its practical to do so!

I mean, its all good if you make your own, but there's most likely a way we can spare everyone having to "re-invent the wheel" so to speak!

We have a Skype chatroom we're using for discussing our findings and etc, and if you'd be interested to join it, I could probably ask Nerketur about it. It would probably make research and etc much faster!

Yeah, its really insane how "powerful"/far-reaching the script engine is in this game !

And, as for running the rom immediately that's indeed a must-have feature for any rom editor! Though, what would be annoying is the repackaging time..

I have been considering for a while making a hack-ish Desmume build that would load the game straight from the disk in extracted form. But, given the way the NDS works its kinda problematic..

There would also be the option of writing some code to simulate "mounting" a rom as a nitro-fs partition and being able to edit the content of the partition on the fly, and shrink and grow it at will, without having to repack the rom. Pretty much like a virtual drive. But I'm not sure how to do this exactly.

And yep, we noticed the link between the filenames of the scripts and the other files throughout the game data. Nerketur actually made me realize this a short while ago. He already knew about it, but, it was quite enlightening for me to say the least

And, I have absolutely no objection about anyone re-using my tools or source code! So feel free to do whatever you want with it all!

And also, I'm not just making a bunch of utilities, I'm also writing a library to handle pretty much any file format from the game, and export them to more common formats, and rebuild them all back into the game. Well, besides scripts, because Nerketur is way better at it and far-ahead than me!

And it will also allow editing the files while loaded in memory as well, to avoid potential wastes of time tied with having to extract stuff to disk everytime! It will probably be in .dll and .so form so that pretty much any code can run it! Though its far from release, but it might be something to consider upgrading to, from the individual little utilities I'm making available here, in the long run!

Anyways, I've written most of the code for the utility now. Just need to fix some errors, thanks to the MSVC2012 compiler being almost C++11 illiterate..

evandixon · October 2, 2014

Sounds like you guys don't really need me on this. With access to what you already have, I may be able to poke around the ROM and see what stores what, but there's not much I could do beyond that.

You should consider using some sort of source control for the ROM editor. I recommend CodePlex, as it supports both Git and TFS. If you use Visual Studio, TFS is very convenient. We could also use the documentation features to document what stores what in the ROM, as well as file structures. Github may also be viable, if you'd prefer that. Using source control, we could all collaborate on each project, or at least provide testing without the need for releases.

DeSmuMe already has code to allow directly loading the Arm9 and Arm7, but I'd imagine that it's those binaries that are expecting the files to be in the ROM itself. Maybe if we had the ROM stored in RAM? A hack-ish DeSmuMe may be able to do that, decreasing load and run time a little.

[Edit]

If you haven't seen it already, Explorers of Time/Darkness's debug menu may prove useful during development/testing. While I don't think it can run whatever script we want it to, it can at least show the background of every map, among other things.

Edited October 2, 2014 by evandixon

Sign In

tool Pokemon Mystery Dungeon 2 - Psy_commando's Tools and research notes

Recommended Posts

Link to comment

Share on other sites

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

psy_commando

psy_commando

TyVogue64

Posted Images

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Create an account or sign in to comment

Create an account

Sign in