Jump to content
This Topic

Pokemon Super Mystery Dungeon And PMD:GTI Research And Utilities

psy_commando
 Share

Recommended Posts

Andibad

Andibad

Posted
Posted (edited)

in short sir0 is an game configuration container, so that why each sir0 file is have different structure. FARC file is have 3 main part : Header, SIR0 (as FAT), and DAT , SIR0 in here used as FAT, which is have 2 type of them, 0x4 and 0x5. i already test my tool for most farc file on SMD, is working fine to find correct offset for each file. idk if is have more than 2 type SIRO FAT on it ... but i will check it later.

here for SIR0 / FARC or related on it structure :

  \ said:
0x0 - FARC (Magic Header)

0x20 - Type of FAT SIR0*

0x24 - first SIR0 Offset as FAT

0x28 - first SIR0 Length as FAT

0x2C - first DAT offset

0x30 - first DAT Length (with first SIR0 last padding)

each volume used 0x20 byte

  \ said:
0x0 : SIR0 magic header

0x4 : game config header offset

0x8 : pointer offset (is related on 3DS Virtual Memory)

  \ said:
0x0 : start Offset

0x4 : file count

0x8 : file name flag (0: with name, 1 : without name)

  \ said:
0x0 : filename offset

0x4 : data offset

0x8 : data lenght

0xc : 0x0

  \ said:
0x0 : filename offset

0x4 : data offset

0x8 : data lenght

well i try encode file on message.bin (packed with FARC container). well too much character is used as code xD

  \ said:
0x0 : file offset

0x4 : file name offset (is not file lenght)

0x8 : ?? (i think is attribute on text ...)

  Reveal hidden contents

here for tools : https://dl.dropboxusercontent.com/u/3122097/SIR0Reader.7z is still WIP, common FARC is can be extracted as raw file. but not working to read SIR0 Message on debug_message file since this tools used static method for read file offset on SIR0, not dynamic one (used Pointer Data).

Edited by Andibad
  • Replies 242
  • Created
  • Last Reply

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

Andibad
Andibad

so far is not posted in here , today got much free time and poking around PMDS Font. is using pretty simple format where img file is cte format (LA4 as texture format), and dic file as pointing font i

Guest TyVogue64
Guest TyVogue64

Welp, I know this is a bit of a bump, but I've managed to port both the Player and Partner modifier codes to the European version of the game, and I've been doing some testing with it. [ATTACH=CONFIG

SirLoin4
SirLoin4

Y'know, at very high levels, using a Void Shadow effectively curb-stomps most of the game given its stats and because of the Boss Pokemon thing. Drain heals most of the damage done, and Peripheral Att

Posted Images

psy_commando

psy_commando

Posted
  • Author
Posted
  Andibad said:
in short sir0 is an game configuration container, so that why each sir0 file is have different structure. FARC file is have 3 main part : Header, SIR0 (as FAT), and DAT , SIR0 in here used as FAT, which is have 2 type of them, 0x4 and 0x5. i already test my tool for most farc file on SMD, is working fine to find correct offset for each file. idk if is have more than 2 type SIRO FAT on it ... but i will check it later.

I seriously doubt that SIR0 is only for configuration though.

I've seen plenty of images, archives, compressed stats data, and etc that were wrapped into a SIR0 for no apparent reasons in the past. Especially in PMD:Explorers of Sky. It seems to me like its just a way to load any constant data structures at runtime directly into the 3ds's memory, and have the code use it as-is from an arbitrary memory position.

(And its really easy to crash the game and writing in a lot of places you're not supposed to with that file format xD A single error in the encoded pointer list is enough to corrupt most pointers after they're translated, and thus write pretty much anywhere in the allocated user space memory. I made a mistake once, and the game began writing into the OAM, or maybe even the video memory, and things started looking pretty crazy on-screen xD )

And about the different types of FAT, 2 types sounds about right. I've only seen 2 variations so far. (for models files at least)

But it depends on what you want to do. Are you writing a FARC extractor, or something else ? Because, there are probably a lot of variations of the SIR0's content around.

  Andibad said:

here for SIR0 / FARC or related on it structure :

well i try encode file on message.bin (packed with FARC container). well too much character is used as code xD

  Reveal hidden contents

here for tools : https://dl.dropboxusercontent.com/u/3122097/SIR0Reader.7z is still WIP, common FARC is can be extracted as raw file. but not working to read SIR0 Message on debug_message file since this tools used static method for read file offset on SIR0, not dynamic one (used Pointer Data).

Nice work with those FARC files !

Interestingly, the message_debug.bin archive seems to contain many "string database" files that were used in GTI. They all seem to match my notes : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/PMD_GTI/FileFormats/string_database.txt

But the message.bin seems to contain those too, only the strings that each pointers refer to isn't in the SIR0 container itself.. I wonder what this means.. (I only got the Japanese ROM right now though )

What's even more intriguing is that, the function used to load the game strings for menus still refers to the "message" directory, which is missing in PSMD. 

MENU:LoadMenuTextPool("message/staff.bin")

So, I'd guess, they modified that to pull text from somewhere else ? Possibly involving that message.bin file ?

EDIT:

Also, nvm about the strings missing from the string files within the message.bin file.. Its a bit too early for me to say I guess.. There's a possibility that UTF characters don't show up properly in my hex edtior, and well, I think I'll have an easier time once I can dump my NA rom.. ^^;

And the message.lst file seems to contain the file names for the "string database" files within the message.bin file ! Or at least, that's what I'm assuming.

Thunder

Thunder

Posted
Posted

I decrypted my US rom and extracted the romfs. I might not know much about 3DS hacking, but I hope I could be of some use. Judging from all the debug files, is there a chance that a usable debug menu exists? I hope so. :)

Andibad

Andibad

Posted
Posted

debug message is used 8 byte on header FAT, i just suspect pointer table is just table of length each file on header FAT. common.bin is have 0xc length each file which is filenameoffset, dataoffset and file lenght, on pointer table say 0xc too and pointer table length fit with file count (-1).

SIR0 Pointer table on common.bin :

04 04 9C 84 14 0C 0C 0C 0C 0C 0C 0C 0C 0C 0C 0C [end with 0x10 0x00]

i suspect 0x0c is not padding, but is seems length of header FAT on each file. italic one is seems an flag .. but i still not sure. sir0 on FARC is have 0x20 on it without 0xXX padding value. since i only have SMD us on my PC, maybe i need to extract GI too (us/jp version).

psy_commando

psy_commando

Posted
  • Author
Posted
  Andibad said:
debug message is used 8 byte on header FAT, i just suspect pointer table is just table of length each file on header FAT. common.bin is have 0xc length each file which is filenameoffset, dataoffset and file lenght, on pointer table say 0xc too and pointer table length fit with file count (-1).

SIR0 Pointer table on common.bin :

04 04 9C 84 14 0C 0C 0C 0C 0C 0C 0C 0C 0C 0C 0C [end with 0x10 0x00]

i suspect 0x0c is not padding, but is seems length of header FAT on each file. italic one is seems an flag .. but i still not sure. sir0 on FARC is have 0x20 on it without 0xXX padding value. since i only have SMD us on my PC, maybe i need to extract GI too (us/jp version).

The encoded pointer list must end on a 0. It will keep going until it hits one.

So, yep, that's not padding. All those 0xC are actually offsets to pointers within the SIR0 container.

http://projectpokemon.org/wiki/Pmd2_SIR0#Pointer_Offsets_List

Andibad

Andibad

Posted
Posted (edited)
  psy_commando said:

Also, nvm about the strings missing from the string files within the message.bin file.. Its a bit too early for me to say I guess.. There's a possibility that UTF characters don't show up properly in my hex edtior, and well, I think I'll have an easier time once I can dump my NA rom.. ^^;

And the message.lst file seems to contain the file names for the "string database" files within the message.bin file ! Or at least, that's what I'm assuming.

i was think is just an id for script for get an string or icon (type,item,etc). so is have weird unicode on it, so i advice just look it on hex on sir0 content (not converted text one, it might not correct to convert them).

example :

  Quote
Use it to teach the move \x65533{591}. But you can't use it on a Pokémon that has already learned or can't learn the move.

\x65533{0} is for call move string list with single value.

  Reveal hidden contents

is need more them to indexed all of them (also play this game too for make sure what string is...) xD or you will see a lot weird character (or code) on it. like adventure results window screen if you finish the dungeon ...

  Reveal hidden contents
  psy_commando said:
The encoded pointer list must end on a 0. It will keep going until it hits one.

So, yep, that's not padding. All those 0xC are actually offsets to pointers within the SIR0 container.

http://projectpokemon.org/wiki/Pmd2_SIR0#Pointer_Offsets_List

sometimes it filled by padding on sir0 used as FAT on FARC (type:5), and sometimes is have multiple set (ex: hint_database.bin). so far i found, is have filename set, data set, configuration set, other thing is depend on structure or none.

Pointer table :

04 04 D2 1C 04 04 30 04 04 30 04 04 30 04 04 30

[A]

header structure :

48 4E 54 00 24 29 00 00 B0 00 00 00 (sometimes [magic header] offset - count [ex attribute], or [magic header] count - offset [ex attribute]

recalculated [A] must same as (offset). (how to encode that just look on wiki). bold one is an magic header. i already done with calculate pointer stuff...

EDIT: after i see on EoT/EoD/EoS, 0xAA is padding, but on 3ds sir0 is used 0x0 as padding. but is not related to pointer table. always ending with 0x0 or 0x10 ...

Edited by Andibad
Silverhawke

Silverhawke

Posted
Posted (edited)

hi guys, i extracted the contents of pokemon_graphic.bin using the SIR0 reader in this thread (may wanna put that in the first post?), and got a lot of BIN files, most of which are BCH files (with some BGRS files interspersed which i believe ties the separated animation BCH files together with the models?). i wrote a little code to infer the name of the BCH files of pokemon models: http://pastebin.com/mmdZtU9k

looks like the numbers at the end indicate [form][gender difference]. starters get 0x and 1x (without/with scarf), and some has x0 and x1 for different genders (male/female) with some exceptions.

  Reveal hidden contents

put the exe in the same folder as the BINs, pipe the output to some text file so that you can see what gets renamed to what. i hope this can be used somehow!

  Reveal hidden contents
  psy_commando said:
Examples (Archen is the first portrait in the list for some reasons..):

when looking at the file names i realized that it's because Archen is "aaken" in japanese. since GTI lacks Ekans (aabo) and Arbok (aabokku), Archen gets to be first =p

Edited by Silverhawke
Andibad

Andibad

Posted
Posted
  Silverhawke said:
hi guys, i extracted the contents of pokemon_graphic.bin using the SIR0 reader in this thread (may wanna put that in the first post?), and got a lot of BIN files, most of which are BCH files (with some BGRS files interspersed which i believe ties the separated animation BCH files together with the models?). i wrote a little code to infer the name of the BCH files of pokemon models: http://pastebin.com/mmdZtU9k

looks like the numbers at the end indicate [form][gender difference]. starters get 0x and 1x (without/with scarf), and some has x0 and x1 for different genders (male/female) with some exceptions.

put the exe in the same folder as the BINs, pipe the output to some text file so that you can see what gets renamed to what. i hope this can be used somehow!

  Reveal hidden contents

filename of model are stored on pokemon_graphics_database.bin with header container 0x4 0x4 0x4c [filename_offset][?] [file_identification] and pokemon_motion_database.bin with header 0xc 0x4 0x4 0x4 [file_identification] ? ? ?. sir0 with own magicheader, [magic_header][offset][file_count] and must read pointer table to read them.

Silverhawke

Silverhawke

Posted
Posted
  Andibad said:
filename of model are stored on pokemon_graphics_database.bin with header container 0x4 0x4 0x4c [filename_offset][?] [file_identification] and pokemon_motion_database.bin with header 0xc 0x4 0x4 0x4 [file_identification] ? ? ?. sir0 with own magicheader, [magic_header][offset][file_count] and must read pointer table to read them.

I'm a bit confused. How does the offsets in these three files tie together? Is there already a tool to do this?

psy_commando

psy_commando

Posted
  • Author
Posted

So, I was able to rip my own cartridge of the game today, and I ran winmerge with the japanese rom and mine on the content, and then on the script. Here are the reports :

Its interesting that more than expected changed!

Especially the scripts.

I'm going to work on investigating the scripts and actor and placement data for now I think.

  Silverhawke said:

hi guys, i extracted the contents of pokemon_graphic.bin using the SIR0 reader in this thread (may wanna put that in the first post?), and got a lot of BIN files, most of which are BCH files (with some BGRS files interspersed which i believe ties the separated animation BCH files together with the models?). i wrote a little code to infer the name of the BCH files of pokemon models: http://pastebin.com/mmdZtU9k

looks like the numbers at the end indicate [form][gender difference]. starters get 0x and 1x (without/with scarf), and some has x0 and x1 for different genders (male/female) with some exceptions.

  Reveal hidden contents

put the exe in the same folder as the BINs, pipe the output to some text file so that you can see what gets renamed to what. i hope this can be used somehow!

  Reveal hidden contents

when looking at the file names i realized that it's because Archen is "aaken" in japanese. since GTI lacks Ekans (aabo) and Arbok (aabokku), Archen gets to be first =p

Which SIR0 reader ? There are 3 of them I think ^^;

Also, I'll only add something to the first post if the author ask me to.

And I don't think anyone seems to really know what BGRS files are for. At least it seems like that. How can you tell they're for animations ?

And that seems to make sense.

I guess that's possible its because the files were in alphabetical order xD

  Andibad said:

i was think is just an id for script for get an string or icon (type,item,etc). so is have weird unicode on it, so i advice just look it on hex on sir0 content (not converted text one, it might not correct to convert them).

example :

\x65533{0} is for call move string list with single value.

  Reveal hidden contents

is need more them to indexed all of them (also play this game too for make sure what string is...) xD or you will see a lot weird character (or code) on it. like adventure results window screen if you finish the dungeon ...

  Reveal hidden contents

I'm not really sure what you're trying to say here :/ And the image is too small to see anything.

  Andibad said:

EDIT: after i see on EoT/EoD/EoS, 0xAA is padding, but on 3ds sir0 is used 0x0 as padding. but is not related to pointer table. always ending with 0x0 or 0x10 ...

Yeah, the padding bytes change from game to game. But I'm not sure what that has to do with this? :/

evandixon

evandixon

Posted
Posted
  psy_commando said:

Its interesting that more than expected changed!

Especially the scripts.

I'd imagine that the parts of the scripts that changed are references to text items, the IDs of which are most likely generated when the scripts are compiled.

I think that because the placement data looks like it's the same, and there's only so much the scripts can do alone.

Silverhawke

Silverhawke

Posted
Posted (edited)
  psy_commando said:
Which SIR0 reader ? There are 3 of them I think ^^;

Also, I'll only add something to the first post if the author ask me to.

And I don't think anyone seems to really know what BGRS files are for. At least it seems like that. How can you tell they're for animations ?

And that seems to make sense.

I guess that's possible its because the files were in alphabetical order xD

ah, I have one which I think was provided by Andibad, i better look through the thread again.

that is only a guess though, since the BGRS files has plaintext in them that seems to refer to name of animations each model has. it also has the names of vertex points in the model (you can see this if you open the BCH file using Ohana3DS, exporting as SMD and opening it up in Blender)

  Reveal hidden contents

then again my filenames are guesses :P

for Andibad's adventure result screen, here's one result screen to help with understanding what the markers mean. a lot of those are formatting markers, apparently.

  Reveal hidden contents

just a quick look at the text files outputted by the SIR0 reader (in UTF-8 encoding) and i noticed that it uses 0x0A for new line within the dialogue box, 0xED 0x84 0x80 0x20 (aka 턀) for player's name, \x60160 for a pause mid-dialogue (y'know where you gotta press A and the text goes on again on the same dialogue), and 0xEC 0x88 0x80 (aka 숀) for centering formatting.

sorry if this is already known before ><

EDIT: updated program to rename files in pokemon_graphic.bin since i dunno how the database bin and the actual data ties together. now it renames all files! https://dl.dropboxusercontent.com/u/40407085/Stuff/binbchren_2.zip

Edited by Silverhawke
psy_commando

psy_commando

Posted
  • Author
Posted
  evandixon said:

I'd imagine that the parts of the scripts that changed are references to text items, the IDs of which are most likely generated when the scripts are compiled.

I think that because the placement data looks like it's the same, and there's only so much the scripts can do alone.

Well, it seems only a part of the decompiled scripts differs. And it would kinda defeat the purpose of using IDs if you'd need to change them with a different language. That's why I changed my mind on them being hashes a while ago.

After looking at the changed scripts, its just some things like using a localized sprite instead of another, and function name changes. All the strings IDs stayed the same.

Also, I'm making some progress messing with the game's internals. I got this messing with actors data :

[ATTACH=CONFIG]12796[/ATTACH]

( I'm so predictable xD )

It seems changing starters isn't as easy as it was in GTI though. But I got a good lead. Its just really annoying to have to rebuild the entire rom and copy it each times you change 3 tiny files.. Hopefully hans gets updated one day to work like game patches..

@Silverhawke:

Those animation names seems to match those in the scripts.

And those special utf characters are interesting. In previous games, the codes for doing that were between square brackets. Its strange to see they've changed their approach xD

And yet they're still using SIR0, ever since Red Rescue Team.. xD

Andibad

Andibad

Posted
Posted
  psy_commando said:

And those special utf characters are interesting. In previous games, the codes for doing that were between square brackets. Its strange to see they've changed their approach xD

And yet they're still using SIR0, ever since Red Rescue Team.. xD

unique code id is have 1 byte long, other byte is used as attribute. like Font Color started 0xC101 - 0xC10E, 0xC10F as close tag. text alignment like center (C200) per line (until 0x10 code inserted), icon replacement id (A1xx), linked page (CA00), menus, paragraph, and etc. some code is need more attribute on it like call an value like pokemon id, etc. and also in SMD still used square brackets code lol

psy_commando

psy_commando

Posted
  • Author
Posted
  PokemonWorldMaster1 said:

could you please show a tutorial of how you changed your starter?

Well, its not ready yet. That image looks good, but, the game is stuck in an infinite loop past that point.. xD

The personality test keeps popping back, which is the game's way of crashing. It did something similar in GTI.

I just have to find out what's going on.. It might have something to do with reloading the models, or maybe something else entirely..

  Andibad said:

unique code id is have 1 byte long, other byte is used as attribute. like Font Color started 0xC101 - 0xC10E, 0xC10F as close tag. text alignment like center (C200) per line (until 0x10 code inserted), icon replacement id (A1xx), linked page (CA00), menus, paragraph, and etc. some code is need more attribute on it like call an value like pokemon id, etc. and also in SMD still used square brackets code lol

Good to know ! xD

And sorry for not understanding what you meant earlier ^^;

PokemonWorldMaster1

PokemonWorldMaster1

Posted
Posted
  psy_commando said:
Well, its not ready yet. That image looks good, but, the game is stuck in an infinite loop past that point.. xD

The personality test keeps popping back, which is the game's way of crashing. It did something similar in GTI.

I just have to find out what's going on.. It might have something to do with reloading the models, or maybe something else entirely..

Good to know ! xD

And sorry for not understanding what you meant earlier ^^;

Thank you for responding, I really appreciate it. But would it be possible to change your species once your already in the game? Like changing some bits of code in the save file?

psy_commando

psy_commando

Posted
  • Author
Posted (edited)

I was able to find the correct fork of ctrtool to decompress the exefs's code.bin in PSMD !

And I ran strings on it.

And its like christmas morning! :P

https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/PSMD/PSMD_codebin_strings.txt

So much info in there ! Its like they left debug symbols ! I can see they're using the boost c++ libs, and also that jenkins is a very sloppy employe xD

I wonder how I can get radare2 to properly use those.. Any arm binary experts around here ?

There's also a lot of lua code inside strings literals xD

  PokemonWorldMaster1 said:
Thank you for responding, I really appreciate it. But would it be possible to change your species once your already in the game? Like changing some bits of code in the save file?

Possibly with a mod. But for save game stuff, you'll have to ask evandixon.

EDIT: Turns out, jenkins might just be an automated build server software :

https://wiki.jenkins-ci.org/display/JENKINS/Meet+Jenkins

Edited by psy_commando
evandixon

evandixon

Posted
Posted
  psy_commando said:
But for save game stuff, you'll have to ask evandixon.

Yeah, it's going to be a while before even money is editable with saves. Maybe once Gateway releases their in game cheat maker, we'll have cheats. But that save file is not straightforward.

psy_commando

psy_commando

Posted
  • Author
Posted (edited)

I managed to change the starter correctly this time ! :D

[ATTACH=CONFIG]12797[/ATTACH]

[ATTACH=CONFIG]12798[/ATTACH]

[ATTACH=CONFIG]12799[/ATTACH]

[ATTACH=CONFIG]12800[/ATTACH]

The move set is all wrong though.. And I'm not sure where the game gets the movesets it give each pokes..

[ATTACH=CONFIG]12802[/ATTACH]

And because of the really stupid way they're handling cutscenes, we may need a new cutscene actor for each new starters. Or maybe we can just re-use the same actor for any new starters.. We'll see..

EDIT:

Spotted some more weirdness:

[ATTACH=CONFIG]12803[/ATTACH]

The ability was also set to something else than it should be.

It seems like the level-up move list is working as expected though.

Edited by psy_commando
SirLoin4

SirLoin4

Posted
Posted

If no one minds my asking, would all of this editing happen to involve some sort of browser exploit? Or would some kind of tool be released to decrypt and modify the save file from an SD Card or PowerSaves file, assuming all of this goes that far.

evandixon

evandixon

Posted
Posted
  SirLoin4 said:
If no one minds my asking, would all of this editing happen to involve some sort of browser exploit? Or would some kind of tool be released to decrypt and modify the save file from an SD Card or PowerSaves file, assuming all of this goes that far.

This is research about editing the ROM, so no save editing for now. Once we've solidified things some more, you'll need a Gateway, CFW, or Homebrew (with Hans) to edit the ROM. If you have access to the browser exploit, you can install cfw and homebrew.

psy_commando

psy_commando

Posted
  • Author
Posted
  evandixon said:

This is research about editing the ROM, so no save editing for now. Once we've solidified things some more, you'll need a Gateway, CFW, or Homebrew (with Hans) to edit the ROM. If you have access to the browser exploit, you can install cfw and homebrew.

You can also get ocarina of time 3d and powersave, and install oot3d hax by simply uploading a hacked save game directly into the oot3d cart using powersave. It worked flawlessly for me, with no dumb browser hax.

SirLoin4

SirLoin4

Posted
Posted
  psy_commando said:
You can also get ocarina of time 3d and powersave, and install oot3d hax by simply uploading a hacked save game directly into the oot3d cart using powersave. It worked flawlessly for me, with no dumb browser hax.

Yes, I have access to OOT3dHAX. I suppose the only thing I can do now is wait, yes? And thanks for answering.

Andibad

Andibad

Posted
Posted

maybe this tool useful for someone for unpacking file from farc file, is still same as my first tool i share on here but with GUI. i will added source code on github later.

https://dl.dropboxusercontent.com/u/3122097/unFARC_build_2.7z for temporary build for testing.

for now can extract as RAW, Message Text (debug, non debug, and with code [you can add some definition on msg_code.json file, i will added it later]), potrait picture on SMD

i just wondering pokemon portrait is have a text on it O.o

961.png

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
 Share
Go to topic listing
×
×
  • Create New...