research Pokemon Super Mystery Dungeon And PMD:GTI Research And Utilities

psy_commando · August 16, 2015

Please be sure to hide any PSMD spoilers within spoiler tags, until the game is released in all regions !

About This Thread:

This thread is mostly for posting on-going research notes/progress and findings for the Pokemon Mystery Dungeon Gates to Infinity and Pokemon Super Mystery Dungeon games.

Both games use very similar formats, and work generally the same way, which is why they both share the same thread.

Information:

Here are links to interesting posts in the thread, and to external websites.

Notes:

Here are some links to notes, or to the wiki on the various file formats and etc of the games.

Utilities:

Here are some utilities for dealing with the file formats in PSMD/PMD:GTI :

...

Current Problematic Issues:

Here are some of the things that are currently holding us back right now:

No way to test modifications/investigate on a 3DS/emulator. My 3DS isn't unlocked. So having someone with one willing to try things out would be nice! (EDIT: Actually,thanks to ironhax I can run homebrew on it now But not much else )

This post will fill up as time goes.

Edited April 9, 2016 by psy_commando

psy_commando · August 16, 2015

So the first thing I'll contribute will be about the game's script engine.

The game uses regular compiled Lua 5.1 scripts!

Which means they can be decompiled using existing lua decompilers!

We don't know yet if we can recompile them, or if we even need to recompile them however.

Here's a nice little batch script to decompile all scripts in PMD:GTI's "script" folder:

[font=Fixedsys]
For /R .\ %%G in (*.lua) do (
java -jar unluac_2015_06_13.jar "%%G" > "%%~dG%%~pG%%~nG.txt"
)
[/font]

And here's a little batch script to delete the compiled original, and rename the decompiled .txt files into .lua !

[font=Fixedsys]
Echo Deleting lua files...
For /R .\ %%G in (*.lua) do (
Del "%%G"
)

Echo Renaming txt files...
For /R .\ %%G in (*.txt) do (
Ren "%%~G" "*.lua"
)
[/font]

Here's a little step-by step guide on how to use those:

0. Make sure you got Java installed, and have the rom extracted!

1. Get the unluac jar : http://sourceforge.net/projects/unluac/ (Be very careful on sourceforge! They may try to scam you with an automated downloader/installer. unluac has no installer and should come in a compressed archive as a .jar, not a .exe!)

2. Copy PMD:GTI's "script" folder somewhere else, to avoid deleting the original script files.

3. Go to that folder.

4. Create a text file, rename it "decompile.bat", and paste into it the first script above.

5. Create a text file, rename it "cleanandrename.bat" and paste into it the second script above.

6. Place unluac_2015_06_13.jar in the folder with the two batch script we just made.

7. Run the decompile.bat script

8. Run the cleanandrename.bat script

9. Done !

You should now have all the scripts decompiled ! Have a nice read

Edited September 20, 2015 by psy_commando

psy_commando · August 17, 2015

Oh, and I kinda forgot to specify how to extract the rom's data ^^;

I'm fairly sure this is not the best way to do it, but I haven't found an easier way. At least it gets the job done..

First you need to get a decrypted ROM. I got a decrypted eshop package, as a ".cia". You'll know that its decrypted if you find the string "paradise" in the cia file, by doing a a string search in an hex editor. If its encrypted, you won't find it, as it will be scrambled. (Careful because a lot of 3ds roms floating around are just useless encrypted wastes of spaces )
Next, the tools currently available won't be able to extract the rom from the ".cia", or at least, I wasn't able to. So we need to get our hands dirty and grab an hex editor to cut out the actual ROM from all that junk.
You'll need to search for the string "NCCH", or in bytes 0x4E, 0x43, 0x43, 0x48. That's the header we'll need to locate for the next part.
Since the NCCH header begins 256 bytes before the magic number, with a SHA-256 key, we need to start copying/exporting bytes from the offset of the NCCH header - 0x100.
(Details on the NCCH header : http://3dbrew.org/wiki/NCCH#NCCH_Header )
I use Hex Edit, so I just set my mark at the offset of the NCCH header - 0x100, went to the end of the entire file, and pressed CTRL+SHIFT+F9, this selected all the data past the beginning of the NCCH header. And then you can either copy paste this into another file, or if you use Hex Edit right click and in the menu click on "write selection..". Save the content to a file.
Next, open that file in the hex editor, go to offset 0x1B0, copy the 32bits integer value in there that's the offset of the romfs chunk. Then copy the 32bits integer at 0x1B4, that's the size of the romfs chunk. Since they're not in bytes, but in "media units" we multiply both value by 0x200, as one media unit is 0x200 bytes. We'll need both of them, so write them both down !
(Details on the romfs header :http://3dbrew.org/wiki/RomFS#Format)
Take the romfs offset we calculated earlier and press CTRL+G, enter that offset. If everything went right, the offset should bring you to the string "IVFC" in the file. That's the beginning of the romfs chunk we'll want to extract.
Then select all the bytes from that place, to the offset of the romfs + the size of the romfs. Put all those bytes into a new file named "pmd_romfs".
Now, we're almost done !
The next phase involves using ctrtool to extract the romfs. Ctrtool is pretty hard to get, and you need to compile it yourself with visual studio or with a build script. Its part of the ctr-sdk project: https://github.com/ctrdev/ctrsdk/tree/master/tools/ctrtool
(For convenience I compiled a version and zipped it here: )
Once you have ctrtool.exe, place it in the same folder as the "pmd_romfs" file, and open a command prompt.
(If you have anything newer than windows XP hold shift and right click on the background of the folder window with no file selected, select "open command window here". Otherwise, press Windows key + R, type in cmd, and then type cd followed with the path to the folder where "pmd_romfs" is. If you got linux, you probably already know how to do this )
In the command prompt, type:
```
[font=Fixedsys]
ctrtool.exe -t romfs --romfsdir=outputdir pmd_romfs[/font]
```

If all goes well, the program should extract the filesystem into the outputdir directory!

The binaries aren't included in the romfs though, they're in the exefs, and I haven't attempted to extract them yet.

ctrtool.zip

psy_commando · August 19, 2015

Portraits are stored uncompressed in face_graphic.bin.

Here are my notes on the file :

EDIT: The notes below are wrong/outdated! The FARC header mainly..

[font=Fixedsys]
face_graphic.bin

The file contains all the character portraits used in the game.

It contains several sub-files stored within a FARC container.

============================================================
Overview
============================================================
FARC chunks appear to be archive of some sort. They're fairly straightforward apparently.

Offset      Name                    Size            Type            Description
----------  ----                    ----            ----            -----------
FARCHeader                          vary            -               Header for the FARC archive format.
[
   0x00 magicn                        4            -               The 4 ASCII letters "FARC" { 0x46, 0x41, 0x52, 0x43 }
   0x04 unktbl                       28            -               Unknown content.
   []
   0x20 nbentries                     4            int32           The nb of entries in the subfiletbl ????
   0x24 subfiletbl       (nbentries * 4)           uint32[]        A table with offsets between all sub files. To get the actual offset to a subfile you need to add all the previous offset! They're relative to the beining of the file!
   [
       foffset                        4            uint32          Offset from last sub-file of a sub-file!
       ...
   ]
   padbytes                        vary            -               A certain amount of padding bytes. Must be a multiple of 4.
]
subfiledata                         vary            -               Contains the data for all sub-files mentioned in the subfiletbl.
[]

============================================================
Portrait Pointers Table
============================================================
This SIR0 chunk contains a list of 16bits character null terminated strings, 
along with a table associating those strings to the offsets of each portraits.

Offset    + Name                    Size            Type            Description
----------- ----                    ----            ----            -----------
SIR0Header                            16            -               The standard SIR0 header.
[
   0x00 magicn                        4            -               "SIR0"
   0x04 subhdrptr                     4            uint32          Pointer to the sub-header.
   0x08 ptroffsettblptr               4            uint32          Points to the SIR0 encoded pointer offset table.
   0x0C null                          4            -               Terminating zeroes.
]
strtable                            vary            -               A table containing 16 bits character strings. Each strings is the filename of a pokemon portrait.
[
   Entry
   [
       portraitstr                 vary            wchar[]         A null terminated wchar string.
       padchar                      0-2            wchar           If the string ends on an offset not 4 bytes aligned, a single wchar is added to align the next string on 4 bytes!
   ]
   ...
]
portraittbl             (nbportraits * 16)          -               A table containing details on all the portraits in the game.
[
   pentry
   [
       stroffset                      4            uint32          An offset to the name string for this portrait.
       portraitoffset                 4            uint32          An offset to the potrait's image data from the portrait data chunk's beginning.
       portraitsz                     4            uint32          The size of the portrait image in bytes.
       null                           4            -               4 bytes of zeroes.
   ]
   ...
]
subhdr                                16            -               The sub-header.
[
   0x00 portraittblptr                4            uint32          A pointer to the beginning of the portrait data table.
   0x04 nbportraits                   4            uint32          The nb of portrait entries in the portraittbl!
   0x08 null                          8            -               8 bytes of zeros.
]
ptroffsettbl                        vary            -               The standard SIR0 pointer offset table.
[]

============================================================
Portrait Image Data
============================================================
Stored in the sub-file after the SIR0 portrait table.
Images are 24bits, rgb, tiled. They're flipped vertically upside down.
I suspect they're 64x64 pixels. I can't figure the exact tile size right now. 8x8 does not works. 
[/font]

I can't figure the correct image format for some reasons.. I've had similar results when trying to figure out PXD's portraits formats. But for some reasons I can't get those to work..

Examples (Archen is the first portrait in the list for some reasons..):

As you can see, there is something odd going on here..

In that image the portrait was interpreted as a 64x64, rgb24, tiled 8x8, vertically flipped image.

But the rows are all weird...

Any suggestions ?

Edited September 20, 2015 by psy_commando

evandixon · August 19, 2015

Do other images have the exact same problem? If so, the tiles need to me manually shifted around. If not, there may be another piece of data controlling it.

Asia81 · August 19, 2015

This game is not even out yet, so how can you do a research?

evandixon · August 19, 2015

This game is not even out yet, so how can you do a research?

He's making the assumption that the engine is the same (or similar to) that of Gates to Infinity.

Current Problematic Issues:

Here are some of the things that are currently holding us back right now:

No way to test modifications/investigate on a 3DS/emulator. My 3DS isn't unlocked. So having someone with one willing to try things out would be nice! (EDIT: Actually,thanks to ironhax I can run homebrew on it now )

I have a Gateway, so I can test patched ROMs. I doubt IronHax would do that (since you'd probably need an ARM9 exploit).

I've made good progress on my NDS ROM Patcher. I'll try to fine tune that, then adapt it for 3DS ROMs too.

psy_commando · August 19, 2015

This game is not even out yet, so how can you do a research?

PMD:GTI has been out for a long while.

And well the title is basically :

Pokemon Super Mystery Dungeon AND PMD:GTI

I guess my intro text kinda overlooked that part though

But PSMD will most likely have the exact same engine with a few differences. Etrian Mystery Dungeon was largely the same as PMD:GTI, and its a completely different franchise that works differently than pokemon.

Chunsoft is still using the SIR0 format which they've been using since Red Rescue Team on the GBA. So something tells me they're not about to change everything

Besides, if the engine is actually all different, whatever we find can still be used for PMD:GTI.

I have a Gateway, so I can test patched ROMs. I doubt IronHax would do that (since you'd probably need an ARM9 exploit).

I've made good progress on my NDS ROM Patcher. I'll try to fine tune that, then adapt it for 3DS ROMs too.

I thought gateway only worked on a few firmwares ?

Isn't the ARM9 only for running NDS games, and doesn't it runs into its own little sandbox?

Just wondering, because up to date 3ds info is hard to come by

I know that ironhax is limited to usermode, but its really better than nothing.

Especially if people manage to find a kernel mode exploit, having at least access to user mode homebrew would probably be a plus.

And well, you can still access the SD card and etc. Maybe there's a chance we can find an exploit in PSMD or GTI to inject stuff in memory or run a basic debugger or something ?

Honestly, SIR0 files seem like something easily exploitable, because with the pointer translation table, you can literally manipulate pointers in the game's memory! That actually caused a huge bug in one of my utilities for a while, and it caused all kinds of really messed up effects in the game. In the past, overflow errors with c strings for example have been used as exploit vectors to write in memory in spots where they wanted to. Like that missingno glitch in red/blue with the trainer name and inventory and the old tutorial guy.

evandixon · August 19, 2015

I thought gateway only worked on a few firmwares ?

I have 4.5 (after downgrading months ago).

Isn't the ARM9 only for running NDS games, and doesn't it runs into its own little sandbox?
Just wondering, because up to date 3ds info is hard to come by

I know that ironhax is limited to usermode, but its really better than nothing.

Especially if people manage to find a kernel mode exploit, having at least access to user mode homebrew would probably be a plus.

And well, you can still access the SD card and etc. Maybe there's a chance we can find an exploit in PSMD or GTI to inject stuff in memory or run a basic debugger or something ?

From what I remember on various GBAtemp threads, ARM11 is for userland stuff, ARM9 handles lower-level hardware stuff (like game loading). Ironhax is an ARM11 exploit, and can't run ROMs without an ARM9 exploit.

Honestly, SIR0 files seem like something easily exploitable, because with the pointer translation table, you can literally manipulate pointers in the game's memory! That actually caused a huge bug in one of my utilities for a while, and it caused all kinds of really messed up effects in the game. In the past, overflow errors with c strings for example have been used as exploit vectors to write in memory in spots where they wanted to. Like that missingno glitch in red/blue with the trainer name and inventory and the old tutorial guy.

Unfortunately, in order to use edited SIR0 files, you'd already have to be running an exploited system.

psy_commando · August 19, 2015

I have 4.5 (after downgrading months ago).

Did you get cubic ninja to do that ? 0_o

From what I remember on various GBAtemp threads, ARM11 is for userland stuff, ARM9 handles lower-level hardware stuff (like game loading). Ironhax is an ARM11 exploit, and can't run ROMs without an ARM9 exploit.

Well that's a departure from how the NDS works..

Unfortunately, in order to use edited SIR0 files, you'd already have to be running an exploited system.

I didn't really mean to imply it could be used to exploit the system itself. But still that was kinda awkward on my part ^^;

I just can't think today.. I've spent 4 hours going step-by-step through the sound driver's code trying to figure out how to get an amplitude value out of the values in the SWD file, and my brain is like jello now..

But now that I think of it, do you think you could try to decompile one of the lua script file, for say, the top menu, and replace the compiled script with it ? And then repack the rom, and run it ?

I'd like to know if the lua interpreter in the game is fully capable of loading lua scripts that aren't compiled, or if it has been crippled or something. Because I doubt we'd be able to recompile the scripts properly.

And if you have time, if you go into "script\menu", there is a file named "menu_dungeon_debug.lua" with a function named "OpenDungeonDebugMenu()". I was wondering if you could try to put a call to that function somewhere during the gameplay ? And try to mess around with that ?

There are lua functions for literally everything in the game, and a lot of debug scripts are still lying around !

Nearly the whole game runs in the lua engine!

evandixon · August 20, 2015

Did you get cubic ninja to do that ? 0_o

Gateway's browser exploit on 9.2.

Well that's a departure from how the NDS works..

I thought it was exactly how it worked. ARM7 handled hardware stuff while ARM9 handled upper level logic. Right? I could be wrong.

And if you have time, if you go into "script\menu", there is a file named "menu_dungeon_debug.lua" with a function named "OpenDungeonDebugMenu()". I was wondering if you could try to put a call to that function somewhere during the gameplay ? And try to mess around with that ?

I'll have to try that! Might help with further research. I'll see if I can find time to do that tomorrow.

psy_commando · August 20, 2015

Gateway's browser exploit on 9.2.

hmm.. Well crap.. I thought I could buy one for myself, but I'm on 9.9 now.. >_<

I thought it was exactly how it worked. ARM7 handled hardware stuff while ARM9 handled upper level logic. Right? I could be wrong.

Well, from what I read on GBATek, the ARM7 is pretty much unused when running NDS software. Its mainly used while running GBA games.

Otherwise it apparently does some audio processing, and possibly a few things on the side.(Though, most of the audio processing stuff I saw was handled on the NDS9..) I think I read somewhere in a homebrew game tutorial that the NDS9 didn't have interrupts for all buttons, and that some button had to be accessed through the other processor, or something along those lines.. Also, apparently that the Nitrosdk wouldn't let developers run anything on the NDS7 themselves.

I'll have to try that! Might help with further research. I'll see if I can find time to do that tomorrow.

Sounds good ! If that doesn't work feel free to improvise

Any info we can get on the game would be very useful!

In other news, I figured out the format for the game strings:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/PMD_GTI/FileFormats/string_database.txt

Having the scripts decompiled helped a bunch! It made me realize that they're referring to strings through 32bits hashes/uid. (unluac interpret them as signed 32bits integer in the scripts)

I validated by looking at what string db file they were loading in a particular script, and tried to find the hash of a string from the script in the db file. I was able to find exactly the values I was looking for, so I guess it works

Edited August 20, 2015 by psy_commando

evandixon · August 20, 2015

Decompiled the lua files in /script/menu (leaving the jar and bat in place), and I got the "An error has occurred, forcing the software to close. The system will now restart." message after seeing the 3 color circle things after starting the game. Tried again after deleting the jar and bat files and I didn't even get that far. I think it's pretty safe to say that the game will only accept compiled lua files.

psy_commando · August 20, 2015

Well, have you tried with only a single decompiled file?

Because the decompiler might not be perfect. So if you only decompile a single one, you reduce the chances of that interfering.

The best thing to do would be to try it a few times with small files and see what happens.

EDIT:

Also, have you tried to rebuild the rom without changes and testing it ? maybe something went wrong during repacking?

EDIT2:

If that also doesn't work, here's a recompiled "menu_top.lua" file to try out if it works with recompiled script files :

script.zip

Edited August 21, 2015 by psy_commando

SciresM · August 25, 2015

Portraits are stored uncompressed in face_graphic.bin.
I can't figure the correct image format for some reasons.. I've had similar results when trying to figure out PXD's portraits formats. But for some reasons I can't get those to work..

Examples (Archen is the first portrait in the list for some reasons..):

[ATTACH=CONFIG]12496[/ATTACH]

As you can see, there is something odd going on here..

In that image the portrait was interpreted as a 64x64, rgb24, tiled 8x8, vertically flipped image.

But the rows are all weird...

Any suggestions ?

I already have a complete FARC extractor and .IMG converter...I made it forever ago when I uploaded all of GTI's portraits to vg-resource: http://www.spriters-resource.com/3ds/pokemonmysterydungeongatestoinfinity/sheet/63215/

psy_commando · August 25, 2015

I already have a complete FARC extractor and .IMG converter...I made it forever ago when I uploaded all of GTI's portraits to vg-resource: http://www.spriters-resource.com/3ds/pokemonmysterydungeongatestoinfinity/sheet/63215/

Nice!

Do you have notes or source code available ? Because, in addition to extracting them, we'll want to convert images back into the game as well.

And have you done any other work on PMD:GTI by any chances ?

evandixon · August 26, 2015

Well, have you tried with only a single decompiled file?
Because the decompiler might not be perfect. So if you only decompile a single one, you reduce the chances of that interfering.

The best thing to do would be to try it a few times with small files and see what happens.

EDIT:

Also, have you tried to rebuild the rom without changes and testing it ? maybe something went wrong during repacking?

EDIT2:

If that also doesn't work, here's a recompiled "menu_top.lua" file to try out if it works with recompiled script files :

[ATTACH]12497[/ATTACH]

Just tried it out, and the game loaded fine. Did you make any edits, because I didn't see any? (However, I didn't have my old gateway save on my SD, so it made a new game, which may have interfered if there were edits).

psy_commando · August 26, 2015

Just tried it out, and the game loaded fine. Did you make any edits, because I didn't see any? (However, I didn't have my old gateway save on my SD, so it made a new game, which may have interfered if there were edits).

I didn't edit anything I just recompiled the file after decompiling it. Just using the "Lua for windows" luac.exe.

I could try editing it though.

psy_commando · August 26, 2015

Here's another modified top menu script, I moved the coordinate of a bunch of UI elements this time before recompiling:

If it works it should be interesting!

script_messedupmenu.zip

evandixon · August 26, 2015

Here's another modified top menu script, I moved the coordinate of a bunch of UI elements this time before recompiling:
[ATTACH]12514[/ATTACH]

If it works it should be interesting!

Looks like it works.

psy_commando · August 27, 2015

Nice ! So that means we can modify the game's scripts very easily, without doing anything special !

Wanna try the a modified intro sequence + pokemon choice ? Just start a new game with those:

This one may crash at one point.. Idk if I added pokemons the correct way. But I messed around with the neck rotations and etc It should be much more interesting and conclusive than moving the menu elements around !

script_EditedIntro_and_pokechoice.zip

Andibad · August 29, 2015

Looks like it works.

wow, so is no needed any special with these script like harvest moon nut script file ... is interesting

psy_commando · August 29, 2015

wow, so is no needed any special with these script like harvest moon nut script file ... is interesting

Lua is very flexible

We even edited scripted scenes ! evandixon hasn't uploaded it yet, but I messed with the intro scene and I changed a lot of stuff. It was pretty funny to see Pikachu's head spinning at weird angles It was a bit unstable though. I wonder how accurate the decompiler was..

(Debugging modded scripts is going to be complicated though.. Unless we write a lua ui to place over everything else to display debug messages on, or something like that.. Or just manage to toggle on the debug mode that's mentioned in the scripts.)

And I'm sure we could find extra hooks and function to call from the scripts if we'd search the binaries! (Or maybe even some of the ones used in Etrian Mystery Dungeon were already in PMD GTI ?)

I'd need a RAM dump to have a better idea though.. It would really beat reading through a decompiled binary.

I also wonder if we can't load shared libraries through the lua script, since lua comes with a function for that.. However I don't know much about their ARM lua build.. I know the ARM11 can work with a form of shared libraries, but Idk if its taken in charge in their implementation..

I wish I had a gateway and a 3ds with an old firmware. I could do a lot more testing without pestering anyone

On another note, I'm looking for something like Doxygen, but for lua. Just to make a basic documentation of everything! So if anyone else has any suggestions, it would be much appreciated!

KashiToxicBlood · September 17, 2015

Lua is very flexible
We even edited scripted scenes ! evandixon hasn't uploaded it yet, but I messed with the intro scene and I changed a lot of stuff. It was pretty funny to see Pikachu's head spinning at weird angles It was a bit unstable though. I wonder how accurate the decompiler was..

(Debugging modded scripts is going to be complicated though.. Unless we write a lua ui to place over everything else to display debug messages on, or something like that.. Or just manage to toggle on the debug mode that's mentioned in the scripts.)

And I'm sure we could find extra hooks and function to call from the scripts if we'd search the binaries! (Or maybe even some of the ones used in Etrian Mystery Dungeon were already in PMD GTI ?)

I'd need a RAM dump to have a better idea though.. It would really beat reading through a decompiled binary.

I also wonder if we can't load shared libraries through the lua script, since lua comes with a function for that.. However I don't know much about their ARM lua build.. I know the ARM11 can work with a form of shared libraries, but Idk if its taken in charge in their implementation..

I wish I had a gateway and a 3ds with an old firmware. I could do a lot more testing without pestering anyone

On another note, I'm looking for something like Doxygen, but for lua. Just to make a basic documentation of everything! So if anyone else has any suggestions, it would be much appreciated!

I can be your test rat as I have two 3DS units on old firmware, I also have gateway but that isn't important as there is free cfw out there. the games got released today, im just waiting for a dump of the rom so I can help with data mining PSMD

psy_commando · September 17, 2015

Well, evandixon is helping with that already. But thanks, I'll keep that in mind.

I really need to somehow get an exploitable 3ds, because I just work better when I can test things immediately myself. I'm less afraid of trying things that might crash the game and make the process longer. Though I don't really plan on having any money to spend on that kind of hardware for at least 3 to 6 months..

(I wish I hadn't updated my firmware.. >_<)

I'm waiting for a dump too. (But I'll be really lucky if I even find a decrypted one..)

However, personally, I don't plan on doing any real datamining in PSMD until the game is released in november, because I doubt there is much interest from most in getting spoilers on a game that won't release for another 2 months, and I don't really want to ruin the game for me ^^;

I just want to confirm PSMD uses the "same" engine as GTI. And perhaps poke at the script engine, and any new file formats a bit. Then I can just use GTI and Etrian Mystery Dungeon to develop most things that will be needed for PSMD.

Sign In

research Pokemon Super Mystery Dungeon And PMD:GTI Research And Utilities

Recommended Posts

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Andibad

Guest TyVogue64

SirLoin4

Posted Images

Create an account or sign in to comment

Create an account

Sign in