psy_commando

Alright, so I figured out how they're all linked together.. Its something I was told several times over, but I never really believed... The pokemons in kaomado.kao, and monster.md, aren't pokemon, but more like unique entities. And for every single pokemon, there is duplicate entity with for only difference its gender. For pokemons with a single gender or one that's genderless, that second entry has an invalid gender value. This basically means that, it would be possiblt to add face portraits for the females of any species if you've want to Also for people using the awesome Hex Edit hexeditor(I don't think I could have figured half of everything I did without that hexeditor! Rainbow hex data color scheme ftw! Source code is also available), I managed to implement integer decoding and sir0 offset decoding! Using this I was able to make a move data viewer, and a SIR0 pointer locator. It makes things so easy. So I'm going to share them here (Just put the xml files in your Hexedit appdata folder. Or wherever you put your template folder if you changed it from the default. By default, just open up a file explorer window and type this in the address bar : "%APPDATA%\ECSoftware\HexEdit" without quotes, and paste the xml files in there. ) And then to open them, just open the file you want to use this on, and go into the top menu bar "Template->Open Other->waza_p.bin" for the waza_p.bin file, or "Template->Open File Type->SIR0 File". If your SIR0 file has .sir0 file extension, hex edit will open it automatically once you open the template sidebar ! To open the template sidebar, just go into the "Template" menu, and check either "Split Window" or "Tabbed Window". To hide the template sidebar, just check "Hide Tree View". For people looking to find the pointer location in an unknown SIR0 file, this would probably come in handy : Template file here: https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/hexedit_templates/_sir0.xml And for those looking to investigate or modify move data in the waza_p.bin files : Template file here : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/hexedit_templates/waza_p.bin.xml By default however, Hex Edit will only display the first 20 elements in all lists. To change that, go into "Tools->Options->Workspace->Templates" and set the value to something bigger, I usually set mine to 600+, when not testing, and I set it to a lower value when testing, because some templates can freeze the whole thing when they're very complex! And to see the actual values, and edit them easily, you should right click on the grey bar with the name of the template, and check the "Data" entry in the popup menu. Stretch the sidebar as needed. And then, when its possible to edit data, you can just double click on the data, and you'll be able to edit the value and input it as either hex or decimal, or when its an enum field, you'll even get a nice dropdown menu for picking an option ! Just be sure to uncheck "Design Mode" in the "Template" menu, because it will freeze. The structure of the whole thing in design mode is a little too much to handle for the tree display. I wrote it all directly in the XML, so I didn't use design mode much. https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/Tutorials/UsingHexEditTemplate/RecommendedSettings.png Both of those template are pretty complex, and might freeze Hex Edit for a minute, so give it some time. Mainly the waza_p.bin.xml one, because its doing a lot of processing, and I'm not good at optimizing those things The SIR0 one is nearly instant on my computer though, on a file with a good amount of pointers offsets to parse. Of course, those are just research tools, and I'm still working on ppmd_statsutil and etc..

So I learned some more stuff on the monster.md, m_level.bin, and waza_p.bin files! waza_p.bin contains the list of level-up moves, HM/TM moves, and egg moves that each pokemon can learn! Each group of 3 pointers points to the 3 lists for a single pokemon. The lists are encoded integers, and should be decoded in a similar fashion to the SIR0 encoded ptr offset list, without adding each subsequent results to each others, just like the stats growth entries in m_level.bin! The level-up move list is basically just the move id, and then the level at which its learned. The HM/TMs list and egg moves list are just a list of move ids. Here are the values for Bulbasaur decoded by hand: Level-Up list: 81 1A 01 => 0x9A, 1 => Tackle lvl 1 81 59 03 => 0xD9, 3 => Growl lvl 3 82 49 07 => 0x149, 7 => Leech Seed lvl 7 81 36 09 => 0xB6, 9 => Vine Whip lvl 9 81 47 0D => 0xC7, 13 => Poison Powder lvl 13 81 67 0D => 0xE7, 13 => Sleep Powder lvl 13 81 4B 0F => 0xCB, 15 => Take Down lvl 15 81 7B 13 => 0xFB, 19 => Razor Leaf lvl 19 0A 15 => 0x0A, 21 => Sweet Scent lvl 21 81 14 19 => 0x94, 25 => Growth lvl 25 81 0C 1B => 0x8C, 27 => Double-Edge lvl 27 83 78 1F => 0x1F8, 31 => Worry Seed lvl 31 5E 21 => 0x5E, 33 => Synthesis lvl 33 83 66 25 => 0x1E6, 37 => Seed Bomb lvl 37 00 HM/TMs list: 0x12 => 18 => Cut 0x18 => 24 => Swagger 0x1D => 29 => Rock Smash 0x30 => 48 => Return 0x32 => 50 => Strength 0x38 => 56 => Double Team 0x43 => 67 => Facade 0x4C => 76 => Giga-Drain 0x68 => 104 => Endure 0x81 0x17 => 151 => Solar Beam 0x81 0x23 => 163 => Bullet Seed 0x81 0x35 => 181 => Swords Dance 0x81 0x45 => 197 => Toxic 0x81 0x5F => 223 => Sunny Day 0x81 0x63 => 227 => Sleep Talk 0x81 0x68 => 232 => Rest 0x82 0x07 => 263 => Secret Power 0x82 0x11 => 273 => Flash 0x82 0x18 => 280 => Sludge Bomb 0x82 0x2D => 301 => Protect 0x82 0x30 => 304 => Substitute 0x82 0x44 => 324 => Hidden Power 0x82 0x46 => 326 => Attract 0x82 0x48 => 328 => Frustration 0x82 0x68 => 360 => Wide Slash 0x83 0x0A => 394 => Vacuum-Cut 0x83 0x3B => 443 => Energy Ball 0x83 0x4A => 458 => Grass Knot 0x83 0x57 => 471 => Natural Gift 0x84 0x16 => 534 => Captivate 0x00 egg moves list: 0x0B => 11 => Charm 0x54 => 84 => Grass Whistle 0x77 => 119 => Nature Power 0x81 0x06 => 134 => Safeguard 0x81 0x57 => 215 => Amnesia 0x81 0x69 => 233 => Ingrain 0x81 0x6D => 237 => Curse 0x81 0x7D => 253 => Petal Dance 0x82 0x03 => 259 => Light Screen 0x82 0x29 => 297 => Magical Leaf 0x82 0x5C => 348 => Skull Bash 0x84 0x19 => 537 => Leaf Storm 0x00 (I used the "item_p_Sky.xlsx" excel datasheet (the keyvalues sheet) from over there to get the move ids : http://apointlessplace.net/wms/research/ ) Here are my latest notes on the format: https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/waza_p.txt The other part of the file contains data for every single moves in the game! They're all ordered in the same way as in the excel sheet. Move #1 is Iron Tail for example. The typing of the move is not contained in there however.. Replacing everything with random junk changed the way the move worked, the effects, the amount of PPs, the accuracy, and etc. But not the typing.. Also, I'm able to parse and write back perfectly the waza_p.bin and monster.md files now with my code! m_level.bin isn't perfect, but only because my PX compression algorithm is a little less efficient on that kind of data. The file is read fine in-game however. They probably have some kind of optimization going on depending on the type of content. I usually get better compression on images, by on data that doesn't repeat itself a lot I get worse results than the original files. Still its like 5% bigger, so not such a big issue.. I still don't know what the waza_p2.bin file is for exactly... waza_p2.bin is only in EoS, and has a lot of duplicate entries with waza_p.bin, from looking at a binary comparison. I don't know yet how the game decides which one to use, or if its even using both! Also, changing all entries in "monster.md" to 0 or 1 randomly will cause this : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/randomvals_monstermd4.avi https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/randomvals_monstermd5.avi Notice how for some reasons some pokes that don't evolve normally, like chatot are the evolved form of bulbasaur, instead of being a bulbasaur.. Also notice how the AI refuses to move around, I'm not exactly sure why.. And the base experience gain seems to be stored somewhere in monster.md, since I would only get 1 exp from fainting starly. The game seems fairly robust with random values (still nowhere near as robust as Pokemon : Gale of Darkness! That game was unkillable ) Now, the big issue I'm left with, is that I have no ideas how the game associates the data in monster.md, with the rest spread around several other files.. They all have a different amount of entries.. MoveSets : 553 StatsGrowth : 571 MonsterMD : 1,155 Kaomado : 1,155 And it appears that, "monster.md" doesn't contain info on what entry in each files to use.. kaomado.kao and monster.md are however perfectly aligned to eachothers, and the portraits are most likely assigned by matching entry indexes from both files! But, it doesn't appear to be the case for the others. This is very, very problematic, because as long as I can't figure this out, I'm stuck. I can't output or label properly the data I'd convert from those files, if I can't link them together! Any input or ideas on how to find those links would be really much appreciated !! I'm guessing there is a possibility its a table inside the overlays, or the main binary.. And while I'm working on ppmd_statsutil, here are 2 little utilities for ROM researchers. If you have no ideas what they're for, you probably don't need those. They're just more convenient than having to decode all those numbers by hand : SIR0PtrOffsetsDecoder : Use this to decode a SIR0 pointer offset list, which can help tremendously in finding out more about what a sir0 file contains! Just copy/paste the SIR0's ptr offset list only, and put it into its own separate file, and drag-and-drop that onto the .bat file, it will output the result in "decoded.txt". The program itself only output the result to the console, so use the batch script, or pipe the output! IntegerListDecoder : Use this one to decode a list of encoded integers. Things like the stats growths in "m_level.bin", the move lists in "waza_p.bin". You can just put all the lists one after the other in a new file, and drag-and-drop that onto the .bat file and all the decoded lists will be outputed to "decoded.txt" ! Thanks! It might come in handy later! That "faction" ally/enemy byte stands out as pretty interesting to me! Its an odd way of doing things.

I figured out a big part of the format of "/BALANCE/monster.md" ! Its very straightforward. I was able to deduct a lot of it just by cross referencing and comparing pokemon with similarities together! monster.md ----------- This file contains the base data for every pokemon in the game. NOTES: - The first entry is a dummy entry. - The file contains 1,155 entries. Offset: Length: Type: Description: ------- ------- ------- ----------------------------------- 0x0 4 char[4] Magic number "MD\0\0" { 0x4D, 0x44, 0x0, 0x0 } 0x4 4 uint32 NbPokemonEntries PokemonData (NbPokemonEntries-1) This contains the data for all the pokemon in the game! [ PokemonEntry 68 Entry for a single pokemon. [ 0x00 NatDexNb#1 2 uint16 National pokedex number. (Can be null in some entries) 0x02 Category 2 uint16 The index of the pokemon's category string. "Seed Pokemon" for example.(Can be null) 0x04 NatDexNb#2 2 uint16 The national pokedex number, again? 0x06 Unk#1 2 uint16 unknown, seems to be 0x1 for most pokemon ! In the later entries, it contains a number that increases. 0x08 PreEvoDexNb 2 uint16 The national pokdex number for this pokemon's pre-evo. Is 0 if no pre-evo! 0x0A EvolutionMethod? 2 uint16 The evolution method required to evolve to this pokemon. Null if unused. (See evolution methods below) 0x0C EvoParam#1 2 uint16 The first parameter for the evolution method. Null if unused. 0x0E EvoParam#2 2 uint16 The second parameter for the evolution method. Null if unused. 0x10 SpriteIndex 2 uint16 The index of the pokemon's sprite. Its the index inside the three pokemon sprite files inside the "/MONSTER/" directory! 0x12 Unk#4 1 uint8 Unknown 0x13 Unk#5 1 uint8 Unknown 0x14 PrimaryType 1 uint8 The pokemon's primary type. (see pokemon types below) 0x15 SecondaryType 1 uint8 The pokemon's secomdary type. 0x16 Unk#8 1 uint8 Unknown 0x17 IQGroup 1 uint8 The pokemon's IQ group (see IQ group below) 0x18 PrimaryAbility? 1 uint8 The pokemon's primary ability's ID. (see abilities below) 0x19 SecondaryAbility? 1 uint8 The pokemon's secondary ability's ID. (can be null) 0x1A Unk#11 2 uint16 Unknown 0x1C Unk#12 2 uint16 Unknown 0x1E Unk#13 2 uint16 Unknown, often is the same value as Unk#14 ! 0x20 BaseHp? 2 uint16 The hp the pokemon has at lvl 1? 0x22 Unk#14 2 uint16 Unknown, seems to often be the same value as Unk#13 ! 0x24 BaseAtk 1 uint8 The attack stat of the pokemon at lvl 1. 0x25 BaseSpAtk 1 uint8 The special attack stat of the pokemon at lvl 1. 0x26 BaseDef 1 uint8 The defense stat of the pokemon at lvl 1. 0x27 BaseSpDef 1 uint8 The base special defense of the pokemon at lvl 1. 0x28 Unk#15 2 uint16 Unknown, seems to be 0x99 on several pre-evos. 0x2A Unk#16 2 uint16 Unknown 0x2C Unk#17 1 uint8 Unknown 0x2D Unk#18 1 uint8 Unknown 0x2E Unk#19 1 uint8 Unknown 0x2F Unk#20 1 uint8 Unknown 0x30 Unk#21 2 uint16 Possibly max level for the pokemon ? 0x32 Unk#22 2 uint16 Unknown. 0x34 Unk#23 2 uint16 Seems to be part of an array together with the 3 next values 0x36 Unk#24 2 uint16 0x38 Unk#25 2 uint16 0x3A Unk#26 2 uint16 End of array. 0x3C Unk#27 2 uint16 Unknown 0x3E Unk#28 2 uint16 Unknown 0x40 Unk#29 2 uint16 Unknown 0x42 Unk#30 2 uint16 Unknown ] ... ] ------- ------- ------- ----------------------------------- =========================== Evolution Requirements =========================== Here are the possible values for the evolution requirement, and how the value of the parameter is used / split : *NOT 100% sure* Value Method Parameter(s) ( 4 bytes available ) ----- ------ ------------------------------------------------- 0x01 Level(+Item) (uint16)Level required to evolve,(uint16)Item ID 0x02 IQ(+Item) (uint16)IQ Req, (uint16)Additional Item ID? 0x03 Item(s) (uint16)Item ID ? (uint16)Additional Item ID? 0x05 Link Cable (uint16)0x00, (uint16)0x01 Known Item IDs --------------- 0x8B - Up-Grade 0x8C - King's Rock 0x8D - Thunderstone 0x8E - Deep Sea Scale 0x8F - Deep Sea Tooth 0x91 - Moonstone 0x92 - Fire Stone 0x93 - Water Stone 0x94 - Metal Coat 0x96 - Dragon Scale 0x98 - Dubious Disc 0x9A - Reaper Cloth 0x9B - Razor Fang 0x9C - Razor Claw 0x9D - Electirizer 0x9E - Magmarizer 0xA3 - Coronet Rock 0xA4 - Mossy Rock 0xA5 - Icy Rock Known Other Item IDs ? ------------------ 0x01 - Link Cable ? 0x04 - 0x05 - Sun Ribbon 0x06 - Lunar Ribbon 0x07 - Beauty Scarf ### NOTE TO SELF : 3 evo-related uint16 for the following evos IQ Togetic 0x02 0x96 0x00 Crobat 0x02 0x96 0x00 Pikachu 0x02 0xC8 0x00 IQ + Item Espeon 0x02 0x64 0x05 Umbreon 0x02 0x64 0x06 Link Cable Gengar 0x05 0x00 0x01 Item + Link Cable Slowing 0x03 0x8C 0x01 Steelix 0x03 0x94 0x01 Kingdra 0x03 0x96 0x01 Electivire 0x03 0x9D 0x01 Magmortar 0x03 0x9E 0x01 Porygon2 0x03 0x8B 0x01 Porygon-Z 0x03 0x98 0x01 Dusknoir 0x03 0x9A 0x01 Gorebyss 0x03 0x8E 0x01 Huntail 0x03 0x8F 0x01 ### ======================= Pokemon Types ======================= Here are the possible values for the pokemon type bytes : Value Type ----- ---------- 0x00 -NONE- 0x01 Normal 0x02 Fire 0x03 Water 0x04 Grass 0x05 Electric 0x06 Ice 0x07 Fighting 0x08 Poison 0x09 Ground 0x0A Flying 0x0B Psychic 0x0C Bug 0x0D Rock 0x0E Ghost 0x0F Dragon 0x10 Dark 0x11 Steel ============== IQ Group ============== Here are the possible values for a pokemon's IQ group: Value IQ Group ----- --------- 0x00 A 0x01 B 0x02 C 0x03 D 0x04 E 0x05 F 0x06 G 0x07 H 0x08 - (Not sure why those two are skipped ) 0x09 - 0x0A I 0x0B J ========================= Pokemon Abilities ========================= Here are the possible abilities: Value Ability ----- ------------ 0x06 Intimidate 0x10 Torrent 0x13 Shell Armor 0x24 Water Absorb 0x2B Run Away 0x46 Blaze 0x47 Pickup 0x4C Shield Dust 0x59 Gluttony 0x63 Adaptability 0x6D Quick Feet 0x1A Overgrow (Latest version of my personal notes : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/monster.md.txt ) I'm also working on a utility to organize and convert all the stats data spread over several files inside the /BALANCE/ directory into XML, along with possibly the other data directories introduced in EoS. I'm going for a one file per pokemon and per category approach this time, since its much less complex than the sprites! Of course, the utility will be able to put everything back together into the appropriate files from the XML! (For the "mappa_*.bin" and "mappa_g*.bin" files I'll have to do something differently probably, as those are tightly linked to every individual dungeons.) EDIT: Seems like I forgot that truepikachu already had figured out a big chunk of "monster.md" and the stats growth.. http://projectpokemon.org/forums/showthread.php?40199-Pokemon-Mystery-Dungeon-2-Psy_commando-s-Tools-and-research-notes&p=192560&viewfull=1#post192560 Anyways, I guess at least I validated some of those things... Though I wasted a lot of time doing all that from scratch..

Alright, I figured out the exact order of the stats: Offset Length Type Description ------ ------ ---- ----------- 0x0 4 int32 The total amount of experience required to reach that level. The value is absolute, not relative to the last level! 0x4 2 uint16 HP growth for this level. 0x6 1 uint8 Attack growth for this level. 0x7 1 uint8 Special attack growth for this level. 0x8 1 uint8 Defense growth for this level. 0x9 1 uint8 Special defense growth for this level. 0xA 2 uint16 Null. ------ ------ ---- ----------- Total: 12 bytes Its basically Attack, SpAttack, Defense, SpDefense. I had to set to 0 the stats growth for the previous levels to get a better reading. And at the same time I noticed that the base stats for bulbasaur are : Atk 3 Def 2 SpA 4 SpD 3 This will probably help finding where base stats are stored.

Oh, alright. I thought you were saying an existing editor was able to edit m_level.bin. And, if you're saying that they're in the opposite order in the savegame, its also possible they're in that order in the stat growth file. I had to guess the value over several level-up, because of that dumb mistake I did yesterday, so even though I put different values in each slots Its possible the reading was inaccurate... But like I said, after I take a little break, I'll do this properly. Unless you want to try it yourself ? I put up the updated tools you'd need to make this much much easier to edit than having to remove and put back the SIR0 container over the compressed pkdpx! I also wrote 3 handy little batch scripts to batch decompress and compress the content of a folder passed as argument: (just replace the exe names with however they're named on your side ^^ Batch compress to "sir0pkdpx" the content of a folder, and delete the input file: @echo off for %%f in (%1\*.*) do ( ppmd_dopx_xp.exe -z "%%f" "%%f.sir0pkdpx" del "%%f" ) Batch decompress only "sir0pkdpx" files inside a folder, and delete the input file: @echo off for %%f in (%1\*.sir0pkdpx) do ( ppmd_unpx_xp.exe "%%f" del %%f ) And here's another one to build the m_level.bin file with the forced fist file offset: @Echo off ppmd_packfileutil_xp.exe -a 0x1200 %1 All of these work with drag and drop, which is really convenient! Howver, it might just be easier to just decompress, edit and recompress a single file I recommend using bulbasaur, if you want to try, because we're sure he's always the very first pokemon!

It would be cool indeed ! One thing I noticed is that, one of the section in the waza_p.bin file has a number of entries very close to the pokemon count, so maybe that's actually what it is. Here's what I got on the file's layout this far: Offset Length Type Description: ------- ------ ---- ------------ 0x0 4 char[4] SIR0 magic number 0x4 4 uint32 Pointer to WazaHeader 0x8 4 uint32 Pointer to SIR0 encoded ptr list 0xC 4 - Null DataBlockA - Unknown [ ... ] PaddingBytes 0-3 - 0xAA padding bytes to align the next block on 4 bytes. DataBlockB - Seems to contain pokemon data. Has 559 entries of 26 bytes each! [ Entry 26 - An entry. [ ] ... ] PaddingBytes 0-15 - 0xAA padding bytes to align the next section on 16 bytes. PtrTable - A table of pointers to entries within DataBlockA. [ Pointer 4 uint32 A pointer to an entry within DatablockA. ... ] PaddingBytes 0-15 - 0xAA padding bytes to align the next section on 16 bytes. WazaHeader 8 - The header the SIR0 refers to. [ PtrDataBlockB 4 uint32 A pointer to the beginning of DataBlockB. PtrPtrTable 4 uint32 A pointer to the beginning of the PtrTable. ] PaddingBytes 0-15 - 0xAA padding bytes to align the next section on 16 bytes. SIR0EcodedPtrOffsets - The encoded ptr list for the SIR0 container. [ ... ] https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/waza_p.txt Actually I tested changing values in the game, and it worked as expected, but I lost my evening yesterday, because I had a brainfart while trying to figure out which value is which. I forgot that I had a cheat to change the exp for the partner pokemon, which was bulbasaur. And I forgot it was on. So I thought I was somehow corrupting the values because bulbasaur had always a negative exp count towards the next level. So I wasted a lot of time packing and repacking everything a hundred times, trying to figure out what I was doing wrong.. So, I'm still not too eager to test this yet again. For now But I'm pretty sure this ordering is the correct one.. And what's JEditor ?

Alright, since I'm really stuck when it comes to sprite research, I began looking at other formats. And I found the level-up data for all pokemons. Its all stored in "/BALANCE/m_level.bin". The file is a pack file that can be unpacked using ppmd_packfileutil, and then the individual files are sir0 wrapped pkdpx compressed files. I have updated dopx and unpx to handle those without having to manually remove the SIR0 container with an hex editor. I'll post the new binaries later today. I'll also update the wiki, but there is still a little doubt in my mind with the order the stats are stored.. /BALANCE/m_level.bin --------------------- m_level.bin is a pack file, with each of its sub-files compressed into PKDPX files, and then wrapped into SIR0 containers. The first sub-file index seems deliberately forced to 0x1200, given there is an extra unneeded row of 0xFF padding before the first one. Its purpose is to indicate the total amount of experience required to reach a level, and indicate the stat increase for each levels up until level 100, for every pokemon in the game! The structure of its sub-files, once decompressed, and after having been removed from its SIR0 container, is exactly the same for each of them. - 571 sub-files. 1 for each pokemons. And possibly a few that are used for special purpose, given there are less pokemon than that in gen 4? - Exactly 100, 12 bytes entries, one after the other. - The entry for level 1 is entirely null, given there is no level 0, and no exp requirements or stat growth needed. Base stats must be stored somewhere else! The first three entries are for Bulbasaur, Ivysaur and Venusaur. The 3 are identical though. But if someone wanted it'd be possible to change each's stats growth individually! The format of a single entry goes as follow: Offset Length Type Description ------ ------ ---- ----------- 0x0 4 int32 The total amount of experience needed to reach that level. 0x4 2 uint16 HP growth for this level. 0x6 1 uint8 Attack growth for this level. 0x7 1 uint8 Defense growth for this level. 0x8 1 uint8 Special attack growth for this level. 0x9 1 uint8 Special defense growth for this level. 0xA 2 uint16 Null. ------ ------ ---- ----------- Total: 12 bytes See my notes : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/m_level.txt Also, I took a quick look at "/BALANCE/waza_p.bin". And looking at a japanese dictionary, "waza" or "わざ" or 技, means technique/art. It most likely refers to either moves or TMs. So I'm going to take a look at that too. EDIT: Here are the new utilities binaries that supports the sir0pkdpx and sir0at4px formats properly!: ppmd_packfileutil 0.53: https://github.com/PsyCommando/ppmdu/releases/download/ppmd_packfileutil_0.53/ppmd_packfileutil_0_53.zip ppmd_dopx 0.31 : https://github.com/PsyCommando/ppmdu/releases/download/ppmd_dopx_0.31/ppmd_dopx_0_31.zip ppmd_unpx 0.41 : https://github.com/PsyCommando/ppmdu/releases/download/ppmd_unpx_0.41/ppmd_unpx_0_41.zip Will update front page shortly.

Alright, I made a new alpha release. Hopefully it got much more robust now ! I fixed a ton of annoying little things. And now, you don't have to update the frame.xml file when you change the resolution of an image. As long as the image resolution is one of the valid ones, the utility will automatically fix all references to that image. And it can be turned off using the -noresfix option! Here's the download link: https://github.com/PsyCommando/ppmdu/releases/download/ppmd_GfxCrunch_0.13a/ppmd_gfxcrunch_0_13.zip * Disabled raw image inport/export for sprites, as it causes issues and is generally not worth it. The program will default to PNG if the raw image format is specified at the command line! * Changed the format of the XML being outputed. So the output of the earlier version cannot be read by this version correctly! * Fixed a bug with character sprites(type 1) that had no particle offset. A null pointer would be written to the WAN header, but only type 0 sprites can have that pointer null! * Fixed SIR0 encoded pointer list for sprites that had null pointers in their header. The null pointers would not be properly skipped when building the list! * Added support for unpacking PKDPX compressed sprites directly. * Fixed logging so that its turned off by default! * Log files are now named after the input filename/directory name! * Added automatic updating of the sprite's meta-frames resolution, when the meta-frame refers to an image in the "imgs" folder, and the resolution of the image differs from the one in the XML data. * Added command-line option "noresfix" to disable the above feature. * Fixed printing of the readme triggering an exception. * Modified the sprite pack file building code so that sprite directories in the input folder do not need to begin with a number! The files are instead read in alphanumerical order. * And a lot of little things.. I've been holding back on this one for a while to make sure it works and that it isn't too rough around the edges. So I hope there won't be too many issues with it !

Wow, that's so neat ! I wanted to write one, but I never really got around to Well, actually, I had the base layout done and etc in C#, but then I completely messed up my project file because somehow a component is referring to itself and crashing the editor?? The debug output refuses to tell me which component or what line.. Anyways, honestly, I tend to prefer messing with the actual data than making GUIs (If I had made a dll with my code it would have been much more fun making UIs IMO) Also, I recently changed the XML structure a bit for the meta-frames, so for the next release it will be a little different. Here's an example of the new output: frames.xml <FrameGroup> <!--0--> <Frame> <ImageIndex>0</ImageIndex> <Unk0>0x0</Unk0> <Offset> <X>492</X> <Y>244</Y> </Offset> <Unk1>0xc</Unk1> <Unk15>0x0</Unk15> <Resolution> <Width>32</Width> <Height>32</Height> </Resolution> <VFlip>0</VFlip> <HFlip>0</HFlip> <Mosaic>0</Mosaic> <XOffsetBit6>0</XOffsetBit6> <XOffsetBit7>0</XOffsetBit7> <YOffsetBit3>0</YOffsetBit3> <YOffsetBit5>0</YOffsetBit5> <YOffsetBit6>0</YOffsetBit6> </Frame> </FrameGroup> (Also, the "name" attribute that some element have will be filled eventually, once I properly identify all the animation for each characters types! There are 2 main kinds so far. Regular characters have 13 animation groups, and extended characters (Hero/Partner) have 35. And then some specific pokes have extra animations too for cutscenes.) However, I probably won't change it again afterwards. Well, I mean, besides the tags that begin with "unk". And possibly the meta-frame offset values. Because, according to what I gathered(turtwig's sprite), the actual perfect center is around 248, 490. So I could basically subtract that from the value to give something relative to the center of the sprite "layer". But, I'm still not sure why the number is that big in the first place, so to be really careful I just write it as-is. I also feel that the naming for meta-frame and meta-frames group is a bit unintuitive and I'd like to change that to something more accurate. For example, a meta-frame group is basically the actual meta-frame the game refers to, not a group of meta-frames. And meta-frames are really a construct of possibly several images.. If you have any suggestions on that it would be appreciated! I'm not a native english speaker too, so that doesn't help Also, thanks for pointing out the typo! For some obscure reasons I always mix up the way the "th"/"ht" at the end of length and height are written ^^; And a -1 index doesn't necessarily implies that the image doesn't exist or can't be displayed. If you look at diglett for example, he leaves a little trail of dirt when he moves around. And it seems that, by using -1, the game instead of allocating another slot in memory to load that dirt image again, just refers to the instance of that image that's already loaded. If you want to have a better idea of what I mean, look at these videos : This is a modified bulbasaur sprite, where the frames of all the running animations are -1. But the idle animations and etc, are all untouched. You'll notice that the game just takes whatever is loaded in memory at the time and tries to display it according to the parameters of the -1 meta-frame. Its especially obvious when I move from one map to the other, moving into the mess hall and out, or climbing the ladder, etc.. : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/temp/gfxutil_repack_15_28415.avi This one is another modified bulbasaur sprite, but this time I copied twice the single meta-frame in several meta-frame groups, offset them a little, and changed the image index for those extra meta-frames to -1 ( keeping unk15 set to 0) : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/temp/gfxutil_repack_57_1074.avi And thanks for the compliment, its really appreciated

Alright, so I managed to get a little more data on meta-frames with -1 as their image index. Those files in the "/GROUND/" folder have orphaned images that aren't referred to by meta-frames ( in some all meta-frames have a -1 image index. Which begs the question, what is the image index for actually ?): "d28p34a3.wan" "d30p33a4.wan" "d51p11a1.wan" "d52p32a3.wan" "d52p32a9.wan" "d53p41a1.wan" "d57p43a2.wan" "h02p99c4.wan" "h02p99c5.wan" "p14p01a1.wan" "s01p01a1.wan" "s08p01a1.wan" "s13p01a1.wan" "s13p03a1.wan" "s13p03a2.wan" "s18p01a3.wan" "s18p01a4.wan" "s20p01a1.wan" "s20p01a2.wan" "v01p05b1.wan" "v01p05b2.wan" "v01p05b3.wan" "v04p03a1.wan" "v05p03a1.wan" "v05p03a2.wan" "v10p03c1.wan" "v19p06a1.wan" "v23p04a1.wan" "v24p04a2.wan" "v37p02a1.wan" And these files are pokemon sprites that got several -1 image indexes in their meta-frames : 0055_diglett 0056_diglett 0057_dugtrio 0058_dugtrio 0102_haunter 0119_koffing 0158_moltres 0277_houndoom 0400_mesprit 0408_magnezone 0423_dusknoir 0424_dusknoir 0425_dusknoir 0436_darkrai 0441_grovyle 0459_beautifly-f 0460_beautifly-m 0485_ninjask 0497_sableye 0502_aggron 0561_tropius 0587_latios 0589_groudon 0591_jirachi Looking at those and looking a debug output from my utility, I was able to figure out that, in every meta-frames groups* where at least one meta-frame has an image index of -1, the value of the "unk15" byte for that frame, or for preceeding and following frames, is non-zero! Or at least, it seems to be non-zero more often when there is a frame with an image index of -1 around! At first, I believed that, the value in unk15, only when the image index was -1, would essentially contain the index of the image the meta-frame refer to. And while it worked for a few sprites, its definitely not the case for all of them! And now, I've found sprites that have a unk15 value that wildly exceeds the amount of images stored in the sprite... For example a sprite with 60 images had unk15 values that would go up to 104! *Meta-frames groups represent a complete frame to be used as an animated frame. They usually contain a single meta-frame, but it can also contain several assembling them into a more complex resulting image. Meta-frames groups are pointed to by a table, which points to the area right after the SIR0 header, where the meta-frames are stored. The last meta-frame of a group has a bitflag to indicate that its the last frame in a group. All of that and more is detailed in my notes though. See my personal notes for details: https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/SIR0_Sprite.txt I'm really in need of a second opinion on this. I'm afraid I'm missing something possibly very obvious. That's what held me back last time, until TruePikachu popped up and basically figured most of the missing blocks at the time. If anyone has question on the format and wants to help researching, feel free to ask !

Oh. Its alright. You also reminded me I had a couple more screenshots I wanted to get on the frontpage

I'm not sure what you mean by that. For one, PMD2 doesn't uses 3d models. (at least I'm guessing that's what you meant)

Alright, so now that character sprites appears to be pretty much 100% supported, I'm trying to work out the subtleties of the WAN fomat. Namely, what to do with meta-frames that have -1 as their "image index". The program just crashes when encountering those for now. Because I don't know of an intelligent way of handling them using the knowledge I have of the format. Meta-frames with an image index of -1 seems to tell the game to simply re-use whatever is in memory at the time of displaying said meta-frame. But, that poses a big issue, because many images in several sprites are essentially "orphaned". Which means that, we don't know in what resolution they are, or when/how/if they should be displayed, because no meta-frame is obviously referring to them.. Throw in the various boolean and values stored in the wan header that we know nothing about and you get a pretty big amount of possibility to deal with.. I'm still a little unsure how to determine how it all works.. If anyone has any input on this, it would be appreciated. If I can get all the sprites in the GROUND folder and the rest of the game files supported, I'll be able to move on to the next graphic format. I'd probably aim for the WTE files first, as I got most of those figured out. Then probably the dungeons/map tiles format. I also found a bug in gfxutil 0.12, when the pointer to the Particle Offsets table is null, its still being added to the SIR0 ptr list, and it shouldn't. It shouldn't cause issues with most character sprites however. Mainly other sprites. But it will be fixed in the next release. I might also change a couple of the XML tags I'm using to make things clearer. Because, the concept of meta-frames and meta-frames group is kinda vague and confusing. I'm not sure what else to name those though Additionally, if anyone would like to work on a GUI project for this, let me know. Because I think we could work out something that would make sprite editing, and animation editing fairly trivial !

YES!! I freaking fixed that sprite corruption bug !!! Screw you chingling, you can't get corrupted anymore !! I also swapped out POCO XML for pugixml, which resulted in a good speedup ! But it still wasn't fast enough, so I fixed my thread pool code, and now it runs on 2 threads on machines with more than 2 cores. And you can force it to use any amount of threads you want, with the "-th" commandline option, regardless of how many core you have! The corruption came from the way I was encoding the SIR0 ptr list, when a pointer offset ended up on a value with an entire 0 byte at the end, the zero byte was discarded, thanks to some wrong assumptions I had. And after doing a full binary compare, I managed to find out about that. I added another xml file to the exported sprite which contains what I believe is the z index of each individual images in the exported "imgs" folder. Only images that have a non-zero z index are listed in there to save needless work and space. I also finally managed an absolutely bit-perfect re-pack of the m_ground.bin file, from xml ! Which is great ! Here's the latest version: https://github.com/PsyCommando/ppmdu/releases/download/ppmd_GfxCrunch_0.12a/ppmd_gfxcrunch_0_12_alpha.zip

Alright, I updated the front page with the latest GUI Frontends + utility package for both packfileutil and kaoutil. I ran a lot of testing, and I hope there won't be bugs. They both run on the .NET 3.5 framework. I also added some screenshot of both frontends as well. I'm also still looking for input on gfxutil in order to continue development! I really need help to figure out what's going on with it. The download link for it is on the front page for those interested. Its a pure console utility for now though.

After seeing people tricking that Japanese smd2mid utility by renaming files as bgm0001.smd and etc.. I got the idea to try swapping music in PMD2 with other games that also uses the Procyon Sound Driver. I tried with "Fushigi no Dungeon - Fuurai no Shiren DS 2 - Sabaku no Majou", which is basically from chunsoft's main series Mystery Dungeon games, Shiren the wanderer and etc.. I just swaped the main menu's theme in Explorers of Sky, with a track from Shiren2, both the smd and the corresponding swd, and it worked ! https://www.dropbox.com/s/ay31gkxtrzfny4m/mod_music_shiren2_07_7431.avi?dl=0 It seems that in Shiren2 the samples are mainly stored in the accompanying swd file, and apparently there isn't a main sound bank like bgm.swd in PMD2. Hopefully, this will help finding some more clues on how the format works!

Yeah, I'd gladly drop Windows XP support, specially since I have no mean to do testing on it. But it seems a lot of people are still stuck on it. And, I didn't have to drop any features, or make any real compromises to make the utility compatible with XP. As long as that's the case, I really don't mind. It beats having people PM-ing me because the program won't run And, I haven't had to deal with threading at all yet, but I'll keep that in mind!

Alright, so thanks to the help of other members on the forum, I was able to finally pick a decent development platform for making little GUI frontends for my utilities. Rejoice my console illiterates friends! There's the first release, for ppmd_packfileutil : https://github.com/PsyCommando/ppmd_packfileutil_gui/releases/download/v1.0/ppmd_packfileutil_gui_1_0.zip I'm planning on making one for ppmd_kaoutil, and for ppmd_gfxcrunch. Nothing very fancy either, but hopefully, people will stop having issues with the working directory changing when drag-and-dropping! Besides, evandixon got something really nice this far working with ppmd_kaoutil, so its probably best to use SkyEditor to deal with portraits changing anyways ! Also, I haven't seen the graphical glitch again since I'm using dslazy instead of dsbuff to rebuild the rom. I guess I'll try to get some more people to try it then ! Here's the alpha for GfxCrunch : https://github.com/PsyCommando/ppmdu/releases/download/ppmd_GfxCrunch_0.1a/ppmd_gfxcrunch_0_1_alpha.zip I'll also update the front page. This is a console only application. Drag-and-drop won't work properly with this one, because of technical limitations. So if you can't use console, stay away from this one until the GUI is released! Please tell me about any errors you get. Especially any graphical glitches in the game!

So, it seems I can still do C#. (Probably that I'm not using the best practices though, but I'm not doing anything fancy. And besides my command line app is already handling most of the validation) I was able to get the thing working in under a day! The app is a big 19kb big, which is a far cry from the 50+ mb with Qt! If you guys are curious : https://github.com/PsyCommando/ppmdu_gui_frontends/releases/download/v1.1/ppmd_packfileutil_gui_1_1.zip I'm fairly happy with the workflow and result! And I was able to compile it for the .NET 3.5 framework, which also works with Windows XP SP2 ! It might not be very portable, but at least that's going to help take care of a bunch of issues my end users have been having! Thanks a lot for the help everyone ! Well thanks for the advice, but I'm not doing anything very fancy. Basically 2 textbox and a few buttons to browse for files and run the utility! I'm just working on some kind of poor man's GUI for people scared of consoles basically. I'm not getting into any kind of extensive GUI programs until I'm done with reversing a good chunk of the game first. And even then, I'd rather let others make GUI tools that just call my command line apps instead to be really honest I just went with windows form to stay as compatible as possible. Honestly, portability was my first concern, but I don't really want to deal with the headaches that comes with it. I want to spend more time writing code, and reversing the game, than dealing with all that stuff.. And besides, in my case, its not like the main utility isn't portable. But if there was something not too complicated, that wouldn't require including 50mb of dlls with each releases, and that wouldn't need much installed on the end user's machine, I'd totally go for that! But thanks for all the info ! That's good to know if I get a little too attached to .NET. Hopefully, I'm not planning anything very big for a while! And if I do something big, I'll probably go with Qt then. Its less annoying to join 50 mb of dlls and etc, with something big, than with a 50kb bare bone utility!

But it must have made a huge exe file, no ? Hmm.. That might just be the easiest solution at this point.. I guess I could give it a try, again. Hopefully, my C# is not too rusty...

So I took a look, and I need 9 dll to get my Qt program to work. And that's the simplest program with a gui that's possible to make.. If I could get rid of unicode support, I'd lose 3. And I think compiling with visual studio's compiler instead of mingw would remove 2. But then I'd probably have to have a separate Qt base to compile for WinXP...

You need only 2 dlls ? That's weird, the windows deployment page seemed like it would be a lot more : http://doc.qt.io/qt-5/windows-deployment.html Weird.. I guess I could take another deeper look. I was planning on just calling the command line utility via the gui application, basically just a bunch of very simple front-ends. Though, you can call c++ dlls from a lot of other languages and even scripted languages. But, my code right now isn't suitable for a dll.. So, just calling the console app through a gui frontend sounds like the easiest way to go. And it seems he's using Qt for his user interface. There doesn't seem to be any C# in the projects you linked or at least, I can't see any. And about the issues with VS and C++11. Its not VS itself, but more C++ .net, or anything that adds new keywords to the C++ language. I can't find the article anymore, thanks to google and duckduckgo not being helpful.. But it seems some of the headers from the c++11 STL contains symbols that clashes with C++ .net. Or some library used by some project types with VS.. Also, the MSVC has less support for C++ 11/14 than GCC or Clang basically. And there are some really annoying bugs you have to deal with. Like on VS2012, you couldn't use std::future with a function type that had a void return type, even though you really should. So, you always have to look up what feature of c++11 VS supports or not. >_<

(I'm not sure if this goes here, but I haven't found any other places that would be a better fit) So I've been mainly making console utilities, but now I'm realizing a lot of people seems intimidated by the console screen. And I've stumbled on some really annoying limitations with console applications too... So I'm thinking writing little frontends for my utilities might solve that issue.. So I've been looking and trying various toolkits for a while, Ultimate ++, Qt, WxWidgets, etc.. But, I've had massive problems with all of them. Ultimate ++ just bypasses the c++ stdlib, which is not cool at all.. And there is no community for it. And The IDE is hard to understand.. Qt is decent, and I'm familiar with it from having worked a lot with it. But, the amount of redistributables are ridiculous for a less than 1,000 lines GUI application... And compiling a program to run Qt statically didn't work for me this far, thanks to really out of date documentation on the subject. Not to mention the licensing BS that comes with static linking. (I still have no freaking ideas why they needed to come up with a legal BS nuance between shared libraries and static libraries, when it really changes nothing in the end..) WxWidget never worked for me. I tried to compile it several times using mingw32, and even the pre-built binaries, but the default codeblocks projects for wx widgets just won't launch, and all I get from searching is dumb ideas like putting dummy symbols in the code.. That's not a very good sign IMO.. And, then there are the built-in visual studio gui toolkits, but, using their gui framework apparently suppresses some C++11 features..(I'm not talking about MFC) So, I'm a little lost as to what to try next.. Does anyone have any suggestions ? I'm open about using another language, as long as its preferably portable and not too complicated. I've considered Java, but these days its pretty hard to make people install java.. My main concern is mainly making a single binary application. With maybe 1-2 accompanying dlls max..

Hehe. That's nice, but stats don't matter much in PMD2. That's pretty much what's so great about it! I literally crushed everything in the game playing as a poochyena, with my partner being eevee, which also has run away as ability. Though, arguably a lot of enemies are ghost types or psychic types, which made things maybe a little easier. There's this particularly thick ghost type that really got the beating of his life, thanks to a violent seed, and repeated crunching I didn't know he had THAT much HPs Also the Quick Feet ability is so damn good in this game ! Fall asleep ? no problems, quick feet makes you wake up earlier! You get confused ? You get to have more attack turns before the enemy to have more chances of hitting them ! And so on Plus each quick feet boost stacks up until each boost expires ! (And run away makes you impossible to keep you from moving around with things like Mean Look !) But the big lesson to get from that game is that, gummies + IQ skills are your friends ! I think that what would make it really hard is playing through the game with Unown and Smeargle Or even magikarp. Simply because they have horrible movesets. Smeargle can get any move using sketch, but early on, you'll want to have something to attack others with besides the A button attack.. So you'll end up wasting Sketches on a few meh moves along the way for sure. And Unown only ever learn hidden power! Magikarp only learns tackle and splash! There might even be some worse pokes than that however