tool Pokemon Mystery Dungeon 2 - Psy_commando's Tools and research notes

[psy_commando]

New build!

https://app.box.com/s/06l5f78lv51stv34pddc89hxhj5n6sxd

I found out that one of the tables I found in the binaries was actually just as long as the list of routines in unionall.ssb.. And after looking at it, the names matched pretty much what was in the routines. So I set it up so that commands refering to unionall routines (most things like JumpCommon and CallCommon) have their parameter replaced by the name of said unionall routine!

It should make things a bit more readable.

Also, I added support for recognizing script file xml. So, you can now drag and drop decompiled script xml(for single decompiled scripts only for now) onto the executable(it needs to be in the same folder for some really dumb reasons..), and it'll compile it with the correct region and etc..

Also, I might have spotted a possible issue with strings. I had to default to utf-8 because of issues when setting it to the actual encoding, and some editors might scramble some characters after its recompiled.. I'll look into finding a definitive solution for this, possibly converting everything to and from utf-8 if need be, instead of just putting the data as-is and expecting the editor not to try to re-encode everything. But for now, \x is your friend! For example, the character for "é" can be replaced by "\xE9". And when the text is parsed, the utility will turn it into the correct character. PMD2 also has its own character escape sequence syntax you can use if you want. Its the ~. Just put an hexadecimal value after, and the corresponding character should be displayed instead. "~E9".

Be sure to let me know of any possible improvements or bugs with the thing. And also, let me know if you think you figured out what a command is for. Most of the script is pretty much unknown or uncertain.

[psy_commando]

I found a bug with how the signed values are handled. The script engine uses 15 bits signed integers, and I figure that the way I'm handling them might have caused the partner in the intro to run off-screen instead of up in front of the hero.. It probably happens in other cutscenes too. I'll try to inverstigate.

I've also been working on the data format, and having some fun with it

[video=youtube;YdzQzueCUkA]

I made a lot of progress in the last few days! We should have a real release in a week or so, provided everything goes well!

[psy_commando]

Alright, I made a new build with script data support!

I also achieved bit perfect recompilation for script files! That means, I can decompile all scripts and recompile them, and the resulting files will be bit per bit identical to the original!

It was pretty much the case originally, besides a single value in the header that was mystifying me and the others.

After gathering some data, I noticed that original files that had the same nb of strings had the same value for "unk1".

After gathering some more, I realized that, the value of "unk1" was actually nb of strings * 1.5, rounded up! Its probably just some dummy value, but now the output is 100% identical to what their compiler does! Its extremely useful when trying to find bugs though, since it helps trim down the number of binary differences. In fact, it allowed me to spot and fix 4 bugs in 5 minutes

But anyways. About the script data! I'm still not 100% sure what are the meaning behind all values in those though. So I just labeled some of these with "unk" and I'm hoping having an easier way to visualize the data will help us understand what they all do!

I figured the actor/"lives" id, object id, the event's common routine id, however.

One thing to note about objects is that, their names aren't unique. They're just a sprite filename. So to try to make it a bit more readable to humans, I put the numeric id followed by an underscore and followed by the sprite name for the object as the objectid attribute.

The parser only reads the number, and doesn't care about the name after the underscore. So just something to keep in mind!

The syntax has changed again, and it will probably change yet again before the actual release. So just a heads up!

Here's the new build:

https://app.box.com/s/mdce0jiaip5v2k0hde7ks1j1dpwitpwr

Once again, if you can, try messing around with the values and let me know what happens! That would be extremely helpful!

[psy_commando]

Made yet another preview build:

https://app.box.com/s/4tkji56cs89u27spghg362d7s18uv2yo

Its mainly bugfixes and a slightly simplified syntax. Routines now have their xml node reflect their type, and the type attribute is omitted, except for aliases(Though I could change that too).

The extra value each routine had that was 0 for common and standard routines is omitted completely with those. Signed values are also properly exported and parsed, hopefully. So no more random "65535" in the middle of nowhere..

Just a reminder but any suggestions/questions are welcome!

[psy_commando]

So, I found out way later that there was a bug with the last preview release where the routine table wasn't rebuilt properly.

And so I made a new preview build

https://app.box.com/s/gxrv5dib47svzowk49aac9lxt3h85o49

Hopefully, this is the last one before I get this thing out.

[Mariohuge]

I'm impressed with all the progress being made on the script editor!

I'd like some help with something else, however.

While I was using statsutil 0.21 and notepad++ to edit the game strings, I suddenly noticed that every single line after 1506 has a black box with the letters LF inside it. I only noticed this while I was editing the personality quiz, but even after ctrl+z-ing all of my edits the boxes were still there. What do these boxes mean, and will their inclusion affect the game text in any way?

[psy_commando]

LF is linefeed. It basically indicates a new line. Did you turn on the option in notepad++ to display all characters? Because each unique string of dialog is separated by a line feed from the next. Its just usually not displayed by the editor, unless you toggle this on.

It shouldn't affect anything, unless the LF box appears before the "\0" of a given string.

[Mariohuge]

Oh, ok then! I was just confused because the first thousand lines didn't have it at all. But if it doesn't alter anything I don't think I need to worry!

By the way, how would one go about editing the chosen pokemon in the personality test? I was able to do it with skyeditor, but I'm trying to do it with the statsutil because when I tried it with skyeditor it left some garbage text around the generic dialogue (kecleon would say "Welcome to the Kecleon Shop! ?o"). And on that subject, how would one go about adding new pokemon and/or new moves over the Dummy slots? It's all still a bit confusing for me, as so far the only things I've "added" per se are dialogue lines.

Again, thanks for all the work you've been putting into decompiling the game!

[evandixon]

when I tried it with skyeditor it left some garbage text around the generic dialogue (kecleon would say "Welcome to the Kecleon Shop! ?o").

What version of Sky Editor did you use? If it is not 4.0.1.0 Beta 2, please try this one:

https://github.com/evandixon/Sky-Editor/wiki/Beta-2 (Note that the solution files will likely not be compatible with future versions)

To avoid cluttering Psy's amazing thread with my tools, further discussion about Sky Editor should go here: https://projectpokemon.org/forums/showthread.php?47811-Sky-Editor-ROM-Hack-Projects

[psy_commando]

Oh, ok then! I was just confused because the first thousand lines didn't have it at all. But if it doesn't alter anything I don't think I need to worry!

 

I tried ripping it on my end and there weren't anything like that in notepad++. Is it possible you changed the encoding, or did you change the line ending automatically to something else? Sometimes some text editors convert/change all line endings on save. I didn't notice it with notepad++ though.

By the way, how would one go about editing the chosen pokemon in the personality test? I was able to do it with skyeditor, but I'm trying to do it with the statsutil because when I tried it with skyeditor it left some garbage text around the generic dialogue (kecleon would say "Welcome to the Kecleon Shop! ?o").

 

Its not yet implemented in statsutil.

I've been a bit careful about implementing it, because I'm planning on introducing a few asm hacks to load a good chunk of the game data via files, so that a lot more things can be modified without having to figure out how to add things to the list in the exectutables. SIR0 are the perfect vector for that, and the game make use of those a lot already.

Its a bit tricky though.

Its not very hard to add simple direct editing of the starters however. I could definitely add something to do it to the current wip release, but I wonder if its worth it?

And on that subject, how would one go about adding new pokemon and/or new moves over the Dummy slots? It's all still a bit confusing for me, as so far the only things I've "added" per se are dialogue lines.

 

Well, when you extract the pokemon data, notice the ones numbered 537 to 551. Those are dummy entries that can be used for pokemon with 2 genders. All the ones named "reserve" are single gender only, and usually meant for NPCs.

Basically, just edit those files to your liking. Then, you might want to make those spawn in dungeons, but I still haven't implemented dungeon editing. Its scheduled for the current WIP of statsutil though. You can set one of those as starter and play as them though.

Right now its probably better to just edit pokemon entries rather than adding though.

For moves its pretty much the same thing. Except moves 543 to the end seems to be unused and available for tweaking. But we don't know much about moves, and its highly probable that their effects are at least partially hard-coded.

Basically, you'll probably have to do trial and error and see what works and what doesn't because nobody is really sure. There has been little interest in testing anything over the years so we don't know all that much.

Again, thanks for all the work you've been putting into decompiling the game!

 

No problems!

[psy_commando]

While I'm working on adding the last touches to statsutil for the next release, I wrote a bit of an explanation/summary on how the script engine works in general.

Or at least, that's how I understand it right now:

About the Script Engine:

---

Each sub-directories in the /SCRIPT directory correspond to a scene/map/tileset. Except /SCRIPT/COMMON.

The COMMON folder houses the master script, unionall.ssb. It contains repetitive functions that are called often

throughout the game from various scripts, and also executes the game's levels and scenes in order.

Level/Scene Directory Naming:

----------------

Each folder is named depending on the kind of level it is.

 

• "D" usually refers to a level/scene right before a dungeon. Or a dungeon hub/entrance.
  
• "V" usually refers to a full screen visual. Such the ones with the large background sceneries.
  
• "T" usually refers to towns, like T01P01 is Treasure town.
  
• "G" usually refers to the guild, and its interior.
  
• "S" are a bit all over the place. They seem to often refer to side-episodes, but some of the main game events also happen in those.
  
• "H" is Sharpedo Bluff. "Home"?
  
• "P" are usually places that don't fit in the previous categories. They're mainly for the scenes that happens on the road to somewhere this far.
  

 

The second set of digits + letters refers to the sub-level.

So for example, treasure town's market, T01P02A, has a higher value in its second set of digits than the town square, T01P01A.

The very last letter seems to indicate variations of a level. So D01P11A, the beach during day, has a variation D01P11B, the beach at dusk!

Now, back to the content of the sub-directories. The script data file defines the behavior and purpose of its accompanying *.ssb script file(s).

Script Data Naming:

----------------

Script files follow a similar naming rule as to the levels/scenes directory names. However, there are a lot more possible letters. The first number right after the first letter is usually the chapter, then the next set of digits is usually the "step"/scene# within that chapter, but not always.

 

• "m" : Usually a main plot scene.
  
• "um" : Usually a "station" for a main plot scene. ("Station" explained in the *.sss file section below)
  
• "s" : Usually, the same kind of things as "m". The differences are unclear.
  
• "us" : Usually, the same kind of things as "um". The differences are unclear.
  
• "n" : Usually a side story scene.
  
• "un" : Usually a side story "station". ("Station" explained in the *.sss file section below)
  
• "c" : Unknown.
  
• "t" : Apparently title screen scenes.
  

 

Of course, since "sss" and "ssa" data files can be named anything for the most part, there are a lot of instances where the naming diverges from those rules. enter.sse files are always named enter.

enter files

----------------

The enter.sse + enter##.ssb pair defines the content and logic of the playable level.

enter.sse instantiate actors and objects at the specified position, and associate action script to them. It also

regroups them by layers, so that several actors/objects can be displayed or hidden simply by hiding or showing a layer.

The enter00.ssb file is always run when entering the level.

The subsequent numbered enter##.ssb files are action scripts, and their number correspond to the ID in the action script id attribute

that an object or actor may have in the enter.sse file.

*.sss files

----------------

The *.sss + *##.ssb files are what they refer to as "Station" by the devs. They're like entire or partial enter.sse + enter##.ssb pair in a way.

They're merged with the current state of the level pretty often.

They're loaded depending on context, aka the current chapter flag, to populate a playable map with actors, objects and etc.

Often, they're picked and loaded by the enter00.ssb script when the level is first entered. But, its not always the case.

*.ssa files

----------------

The *.ssa files are referred to as "Acting" files by the devs. They're mostly used for scripted dialog/cutscenes.

They always have at most a single *.ssb script file. The entity data they contain seems to override any enter.sss's.

However, sometimes, "Acting" sequences are used for playable/interactive scenes too. They can also load "Station".

Several debug scripts are "Acting" sequences for example.

*.lsd files

----------------

There is always at most a single *.lsd file per directory, and its name is the name of the level/scene in lower case.

The *.lsd files seems to list only "Acting" sequences present in the directory. Their exact purpose isn't very clear.

Its possible that one of the functions in the script refers to indices in the directory's *.lsd file to load local

"Acting" sequences.

*.ssb files

----------------

SSB files are plain script data.

Script instructions are grouped into routines/functions. There are 5 types of routines:

[TABLE=class: grid, width: 800, align: left]

[TR]

[TD]Common Routines(type id 9)[/TD]

[TD]Found only within the unionall.ssb file. Those are basically a fusion of all 4 types of routine below.

They can be either of those, depending on the Instruction used to invoke them.[/TD]

[/TR]

[TR]

[TD]Actor Function(type id 3),

Object Function(type id 4),

Performer Function(type id 5)[/TD]

[TD]They're function that are run by an entity specified in the static parameter of the routine, in the SSB's routine table.

Upon the main function of the script being executed, the entities referred by each of the routines will execute the instructions within them.

Another thing to note is that, "entity attributes" instructions, or any instruction that needs to have a specific entity picked as target through a preceding "entity accessors",

such as "lives", "performer" or "objet", does not need to have a target specified within one of these routines! The target will always be the routine's target entity.[/TD]

[/TR]

[TR]

[TD]Function(type id 1)[/TD]

[TD]For a lack of better term, function denote what is both the main and first routine of a script file.(Except in unionall.ssb) Its this routine that's run first usually.

It never takes any parameters, and its the most common form of routine.[/TD]

[/TR]

[/TABLE]

Instructions:

----------------

The instructions in the script for pmd2 are very basic. They're simple codes with a fixed or variable number of parameters.

Each code executes a corresponding function within the game's program at runtime, with the parameters specified.

The method for controlling the code flow is simple jumps and conditional jumps.

The conditional structures are typical switch + case statements which come in different flavors. Most of the case

statements are conditional jumps, while some are more specialized and do not actually trigger a jump.

Some instructions have something close to return values. Its possible to place case statements after

those instructions, and conditionally jump based on the return value.

Its possible that entity accessors actually work this way.

They tend to fall into those categories:

[table=width: 500, class: grid, align: left]

[tr]

[td]Conditionals [/td]

[td]Those include switch statements, and branch statements. [/td]

[/tr]

[tr]

[td]Entity accessors[/td]

[td]Those are either of the 3 : "lives", "object", "performer". They return an entity on which the next appropriate instruction may be executed upon. [/td]

[/tr]

[tr]

[td]Attribute[/td]

[td]They're functions that get or set attributes for an entity picked by an entity accessor. [/td]

[/tr]

[tr]

[td]Method[/td]

[td]They're functions that act upon an entity picked by and entity accessor. [/td]

[/tr]

[tr]

[td]Single Instruction[/td]

[td]Basically the typical instruction. [/td]

[/tr]

[tr]

[td]Closing Statement[/td]

[td]Those mark the end of a function/routine. So far, "End", "Return", and possibly "Null" seems to be used to terminate a routine. "Hold" was also used in a similar way sometimes, but its unclear what it does, since its sometimes used in the middle of a script, and the script would simply continue past the "Hold".[/td]

[/tr]

[/table]

More categories could be made however. But a lot of instructions aren't yet fully understood.

[psy_commando]

I forgot to post it here, but I made yet another preview build. This time, a lot of things are better labeled and formated!

https://app.box.com/s/8wuv2ncderm4e97171wqhdbm8z7won2w

I've been hitting some issues with random dungeons data handling, and my motivation has been pretty low these days, so its been pushing back the release of the next version of statsutil.

Meanwhile, I've also found a really nice assembler for the NDS for romhackers, "ARMIPS assembler v0.7d by Kingcom" and I've been experimenting with it. Its great, it saves so much hassle

I'm working on trying to load at runtime the level list, and I've hooked about 11 references to the level list so far, both in overlay_0011 and the arm9 binary. I'll have to look into the remaining overlay files, and I need to fix my custom made "level_list.bin" sir0, since apparently I've doing something dumb while putting it together..

Hopefully it'll work on the first few tries.. ^^;

EDIT:

Also, for those interested in testing the tool, I shared a little "workbench" for extracting the rom, exporting the scripts, and then rebuilding the scripts and rom. Its a simple bunch of batch scripts that calls the appropriate tool automatically and does everything for you, so you don't have to type stuff in the console all the time.(It should come with the latest preview build of statsutil)

https://app.box.com/s/v3ri7qqisu2rf4db0qfvm7rhx8imxnf7

Basically, just put the rom file in that directory after you've extracted it, drag and drop it onto the first batch scipt, it'll export the rom's file into the "rom" directory.

Then double click on the "decompile everything" batch script, it'll export the script, the text, stats, etc..

Do your edits.

Then if you've just edited the scripts click on the "3b" batch script, so its faster. It'll build a new rom with your changes, named literally "rom.nds".

Then just run "rom.nds" in you favorite emulator.

[psy_commando]

Alright! So my first attempt at loading the level data from a SIR0 file finally worked! Kind of..

[video=youtube;fSgLQYMQl0k]

After struggling with dumb mistakes while assembling my modifications, and involuntarily ROP-ing the game, I managed to make it work without crashing!

But there's some strange issues tied to Z depth and position markers, and camera panning info missing or incorrect somehow? I'm assuming it might be my rip of the level table that might be bad in some way..

But this proves that adding levels to the game isn't something far-fetched at all! At least, so far

Once I work out the issue, I'll try with the NPC table too, so we can add extra NPCs!

And I'll probably try to dump the personality quizz data and make it so you can add as many starters and questions as you want

Then, I think we could probably try to figure out a way to make the game more extensible with some extensive ASM modifications!

EDIT:

Its not the table. Still not too sure what the problem is..

I'll try partially loading the file each times it needs to look up for a name. It'll be slow as hell, but if its an issue with memory management, it'll show.

Edited August 19, 2016 by psy_commando

[psy_commando]

Ok, so I decided to release Statsutil 0.23.

https://github.com/PsyCommando/ppmdu/releases/tag/ppmd_statsutil_0.23a

I didn't include randomly generated dungeon data editing, because I want to be sure what everything does first to avoid having to rewrite over and over, and mess around each times..

I need to investigate. But my motivation is pretty much down to 0 in the last few days. So I assumed it might be better to just release this with just the script editing stuff and stop possibly keeping people waiting over it.

Even though, I'm pretty sure literally everyone waiting on this got thoroughly scared by the format the game scripts are in by now..

I can't really simplify things any more this soon though, since I don't know what can be automated and what can't yet. It'll become clearer the more we find out about the script engine.. I'm pretty sure LSD tables can be automated, but some of the original tables have quirks to them that can't really be reproduced. And, the 3 main file types for a level sse, sss, and ssa could possibly also be simplified into something cleaner and simpler, but it'll require a lot more investigation along with feedback and report from users..

[psy_commando]

Also, I wrote a crappy little tutorial, to try to get people to try it, since I noticed most people seems really intimidated by this.

All you need is in this zip : https://app.box.com/s/v3ri7qqisu2rf4db0qfvm7rhx8imxnf7

Well, besides the rom and emulator of course.

Here's the tutorial in images. I wanted to write a Gist on github instead, but adding images to that is a pain, so there you go:

http://imgur.com/a/bBnmQ

If anyone has any questions about it, feel free to post in here or PM me!

Edited August 25, 2016 by psy_commando

[psy_commando]

Found my first bug after talking someone into using my tool! Because I haven't heard much since I began releasing builds.

For some reasons not all script files are properly handled on windows 10. The program won't pop any warnings, but it'll only export a small number out of the 383 script files it should produce. Some people reported 52, and on my laptop only 29 came out.. It doesn't appear to be tied to threads, since setting it to a single thread didn't change anything.

Since I only have windows 10 on my laptop, its the kind of things I couldn't readily verify myself. But I'll try running more tests on windows 10.

It probably has to do with the version of POCO I'm using. Since I use their directory iterator class. Upgrading to the latest poco might take a while...

Because they transitioned to c++ 11 so they changed their API a bit. Which means I'll have to change things as well if they work completely differently..

[psy_commando]

Ok, I fixed the problem. It was actually my multi-threaded (horrible) task handler that caused the problem. I replaced it with calls to std async, and I seem to be getting the exact same performances! So hopefully this is completely fixed now..

I updated the front page.

Here's the link to the build on github : https://github.com/PsyCommando/ppmdu/releases/tag/ppmd_statsutil_0.23.1a

EDIT:

Also updated the "modding setup" I made for the tutorial, and etc.. It also comes with the latest version of kaoutil now! (Which I completely forgot to release since last November..)

https://app.box.com/s/v3ri7qqisu2rf4db0qfvm7rhx8imxnf7

EDIT2:

I made a release for kaoutil 0.42 since it was long overdue!

https://github.com/PsyCommando/ppmdu/releases/tag/ppmd_kaoutil_0.42

[psy_commando]

Released a new update to statsutil. Mainly to fix script compilation issues, and actually display errors when there are some, and not just silently ignore files with errors.

I accidentally broke error reporting when I changed the way the parallel script compilation worked.

Here's the link :

https://github.com/PsyCommando/ppmdu/releases/tag/ppmd_statsutil_0.23.2a

EDIT:

Also, I forgot to mention, but the tool now generates a compiler_report.txt file which will contain the result of the compilation, with a list of all the errors and warnings about the scripts. Its still pretty basic, glitchy and not very mature yet, but, its leagues better than before

Edited September 12, 2016 by psy_commando

[Tabun_ne]

As a needed step towards finishing my standalone game, I've come to the challenge of having to obtain the graphical effects used in Pokemon moves. Traditionally I've done this by ripping them through emulator tile view, but that involves going through every single move manually. Does there exist the means to extract these graphics from an unpacked rom? I tried running the gfx cruncher on /EFFECT/effect.bin, but the resultant files seem to be in an unspecified format.

And on the topic of ripping sprites from a live game... Regarding dungeon tiles, would it help your research if I provided a guide of which texture mapped to which type of dungeon tile, in for instance Beach Cave? With the tasks in my own project, it's something I could do along the way.

Ex: There are 8 tiles adjacent to any given tile, giving us 2^8=256 possibilities of adjacent-tile configurations. http://i.imgur.com/laKdF4F.png

However, (at least for walls) diagonally adjacent tiles are not counted when their cardinally adjacent tiles are missing, leaving us with 47 different possibilities that are visually distinct. http://i.imgur.com/l00XuHK.png

Each of these possibilities has at least one dungeon tile mapped to it (Note, image does not have every combination filled). http://i.imgur.com/DSrSbz1.png

[psy_commando]

Hey! Did you figure out the weirdness with the pokemon sprite -1 frames?

Anyways, I haven't really looked into effects this far. I do know that the effect.bin file is a "pack file" though. Right now there is no obvious way to rip effects.

You could use something like tilesGGD on the individual files and try to use part of the file as image data and another as palette and try to see if you get the content to display. That's what I usually do initially with most graphic formats I have to tackle. Sometimes it gives perfect results out of the box with some messing around.

And that's funny, gfx crunch seems to just exports the content of the effect.bin pack file like that because it doesn't know the file format, and it apparently labels the result as a monster.md file ^^;

gfx crunch is getting a full rewrite soon though.. Its so glitchy and shoddy..

From a quick glance, the format of at least some of the sub-files in the effect.bin pack file seems fairly well structured. And they're SIR0, so its easy to find out what are pointers and what aren't.

It should be pretty straightforward to reverse in theory.

But yeah, I got a pretty big backlog of stuff to work on, and I'm pretty much the only one actively working on reversing PMD2 formats lately afaik. And I've been pretty much either low on motivation, or dealing with health issues or complicated stuff in the last few days. So, what would really help a lot speed things up would be getting a hand with reversing some of those formats.

I've been working on the map tiles lately, and I made some good progress mainly by reading the game code that loads the maps. But its going to take a while to reverse, and I can't really work on too many of these at the same time. And well since nobody has been helping with figuring the remaining commands in the script engine, I've been trying/intending to do it all myself too..

I might take a deeper look at the effect file format this week, but no promises. I can already tell there's a palette and image slices in there. There's a table with pointers pointing to what looks like 4bpp image data chunks, and another one pointing to a palette for one. And a lot of other stuff, probably animation related. It looks a bit similar to the wan format for the sprites, but a lot less complicated..

And on the topic of ripping sprites from a live game... Regarding dungeon tiles, would it help your research if I provided a guide of which texture mapped to which type of dungeon tile, in for instance Beach Cave? With the tasks in my own project, it's something I could do along the way.

Ex: There are 8 tiles adjacent to any given tile, giving us 2^8=256 possibilities of adjacent-tile configurations. http://i.imgur.com/laKdF4F.png

However, (at least for walls) diagonally adjacent tiles are not counted when their cardinally adjacent tiles are missing, leaving us with 47 different possibilities that are visually distinct. http://i.imgur.com/l00XuHK.png

Each of these possibilities has at least one dungeon tile mapped to it (Note, image does not have every combination filled). http://i.imgur.com/DSrSbz1.png

 

Thanks for the info!

I'm not there yet though. I'm still unsure if dungeon tiles are stored in the same way as regular map tiles. I'll need to be at least able to display the content of the files with probably a test tool before knowing for sure.

[Tabun_ne]

I ran effects.bin through the ppmd_packfileutil this time, and now things start making much more sense. I'm not very well-versed with rom hacking (I've only just now downloaded HxD and tilesGDD to work with), but I think I'll try to research on this effects.bin format as well, probably cross-referencing with the existing WAN format due to how much sense it makes that they would be similar. I don't have much else to do in the immediate future...

Edited September 18, 2016 by Tabun_ne

[Tabun_ne]

http://pastebin.com/JtUTvrR3

These are the results I got when I opened up effect_0117_0x100610.sir0, extracted from the effect.bin and followed the WAN reference of pointers. It was the smallest file on the list, so I went through the whole thing manually. All of the pointer data appears to be consistent and the format seems to fit perfectly, with a few things to note:

ln174: The sprite type reads out as 2. According to your notes, you've never found files with that number before...

ln13 and 27: Meta-frames read out for some wonky results. -1 frames the only meta-frames in the list. Resolution flags of all 1's, and demanding both HFlip and VFlip. It's odd... perhaps the format here is different?

ln139: There is no particle offsets table. It might be because this particular sprite is so tiny, but I could see this being the case for all effects.

I tried viewing the pixel strips in tilesGGD, but couldn't get anything sensible to show (although again, it may be because it's a tiny file that may not hold meaningful information): http://i.imgur.com/r8RB1xz.png

I wonder if decoding it like a WAN file would just solve 90% of this format's mysteries. I did attempt to just run the gfx cruncher on the little file, as both a .sir0 and a .wan extension, but apparently it doesn't recognize the parameters.

[psy_commando]

http://pastebin.com/JtUTvrR3

These are the results I got when I opened up effect_0117_0x100610.sir0, extracted from the effect.bin and followed the WAN reference of pointers. It was the smallest file on the list, so I went through the whole thing manually. All of the pointer data appears to be consistent and the format seems to fit perfectly, with a few things to note:

ln174: The sprite type reads out as 2. According to your notes, you've never found files with that number before...

ln13 and 27: Meta-frames read out for some wonky results. -1 frames the only meta-frames in the list. Resolution flags of all 1's, and demanding both HFlip and VFlip. It's odd... perhaps the format here is different?

ln139: There is no particle offsets table. It might be because this particular sprite is so tiny, but I could see this being the case for all effects.

I tried viewing the pixel strips in tilesGGD, but couldn't get anything sensible to show (although again, it may be because it's a tiny file that may not hold meaningful information): http://i.imgur.com/r8RB1xz.png

I wonder if decoding it like a WAN file would just solve 90% of this format's mysteries. I did attempt to just run the gfx cruncher on the little file, as both a .sir0 and a .wan extension, but apparently it doesn't recognize the parameters.

 

That definitely looks a lot more similar now! I didn't even realize it was that close..

Yeah, I hadn't encountered type 2 this far. And its possible they're a different format. Or maybe there's something in the meta-frames themselves or some other part of the file, indicating how to parse them?

And particle offsets wouldn't make much sense, considering those are the effects that get offset afaik.

I think if I do a few modifications to gfx crunch it should be able to rip those.

And the tool doesn't recognize a file by extension, but it looks at its structure and can guess if its a sprite or not. But I didn't enable handling type 2 and type 3 sprites because I had not tested those. And I know the the tool tends to crash with non-pokemon sprites.

EDIT:

I've begun reversing the code parsing the sprites.

It looks extremely complicated..

At least, it makes me feel better about how complicated and ugly my code in gfx crunch is..

Edited September 20, 2016 by psy_commando

[psy_commando]

Small Update:

I found out that the tilesets are compressed. But figuring out how to decompress them is going alright. The hard part is going to be to do the opposite.. Its a bit similar to the nybble patterns used in the PX compression algorithm. The code is kind of a mess, so I'll have to make sense of it now that I was able to transcribe it exactly how it is in the binaries.

Still no progress on effect sprites. There's something I'm missing here.. And idk if I mentioned it already, but some effect sprites have literally no image data header. Which just leaves me scratching my head..

[Tabun_ne]

I had some time today and ran through a list of all the sir0 files found in the effect.bin, and what WAN-like filetypes they are, assuming they're WAN at all.

http://pastebin.com/UrAGG3TU

I also force-converted the files with the image type of 2 through the gfxcruncher, just curious of the results

https://www.dropbox.com/s/yi904m0da78m4s3/forceChar.zip?dl=0

Even though the palette and image sizes are garbage, the converted product actually survived the process! There's definitely some familiar effects in there that have the same silhouettes as what I manually ripped before... I'm considering comparing them for palette data. Perhaps there's a correlation there...