psy_commando Posted October 17, 2014 Author Posted October 17, 2014 I think those are from Gates to Infinity (3DS), not Explorers (NDS). However, for the entire list of "code from rom", I'm calling BS on that for now, unless more context is given. Oh, I got that part. Its just that I was wondering if there wasn't a link somewhere with what we were talking about that I'd have missed. Its interesting to see they actually store 3d models inside SIR0 files though And, I don't know how you can say its bs, when I don't even know what those codes are for.. Are they file ids or hashes or AR codes or something like that ? And then, what do they do? I haven't actually made any edits to the ROM, only read data from it. I know there are a huge number of pieces of research possible by modifying the source material (i.e. compiled binary) to see what effects it has on the output, but I've only really done things like that when I'm trying to make the software behave differently in a specific way. I don't know why I never actually do it. Case in point, Dwarf Fortress (which is very open to modding) has all the creature and object data stored in plain text files, called raws. However, incorrect modding results in what is called "duped raws" or "duplicated raws", where the game goes crazy because something very weird happened with the resources, and I _mean_ _crazy_ (even by DF standards; you can get things like embarking as elephants with picks made of spit, on dirt made of flesh and stone which is actually bacon). I really want to experiment with duped raws at some point, but never get around to it. You should definitely try. Though, better extract the files first, and then pack them back into a rom, because of the nitrofs's FAT. EoS is pretty straightforward when you edit things. It usually black screen or crashes when you screw something up.
TruePikachu Posted October 17, 2014 Posted October 17, 2014 (edited) And, I don't know how you can say its bs, when I don't even know what those codes are for.. Are they file ids or hashes or AR codes or something like that ? And then, what do they do? Check PMs EDIT: psy_commando is confirming good faith. What exactly is that data? I had believed that to be a troll attempt because it doesn't look like proper data; it is groups of 8 alphanumeric characters, but NOT groups of 8 hex characters. Notably, it is sorted. Is it possibly a list of file IDs, stored in the ROM as a char[8]? Can't be AR codes, those (at least on NDS) are groups of pairs of DWORDs, unless Datel is trying to take the route the old Game Genie took with encrypted codes... Hmmm...I'm going to go check something on Bulbapedia... EDIT: Bulbapedia doesn't have a list of PkMn in GtI. Serebii did, 148-151 or something, so there goes that theory, unless alt forms are present and I only counted Kyurem's. Edited October 18, 2014 by TruePikachu
TruePikachu Posted October 21, 2014 Posted October 21, 2014 (edited) Just as a heads-up, I'm officially dropping support for pmd2mid in favor of pmDJ, "Pokémon Mystery DJ", a collection of tools which will pretty much do the same thing, as well as convert in the other direction (eventually) I have the codebase, right now, as a clone of the useful things from pmd2mid; project does compile properly right now. Codebase is now hosted at Github, https://github.com/TruePikachu/pmDJ, so check there for further updates to my tools. EDIT: Same is going for my notes, now at https://github.com/TruePikachu/pmDJ/wiki Edited October 21, 2014 by TruePikachu
Andibad Posted October 21, 2014 Posted October 21, 2014 Sorry, but I'm not sure I understand what you're trying to say. ^^;What are those codes ? wonder mail code for us version.
psy_commando Posted October 21, 2014 Author Posted October 21, 2014 Update: I made another commit of my code, this time with working code for the unpacker/packer. I also improved it a lot, its even faster now! For some reasons though, packing is faster than unpacking.. 200ms to pack vs 450ms and up to unpack. I don't know why I'm so obsessed with performances ^^; I also fixed a couple of dumb things I did, like in a few loops I was using a "for( auto curentry : m_structure )" kind of loop, and I forgot to put a "&" between auto and the variable name.. The result was a mess of unneeded copies which I didn't even notice until now ^^; Just as a heads-up, I'm officially dropping support for pmd2mid in favor of pmDJ, "Pokémon Mystery DJ", a collection of tools which will pretty much do the same thing, as well as convert in the other direction (eventually)I have the codebase, right now, as a clone of the useful things from pmd2mid; project does compile properly right now. Codebase is now hosted at Github, https://github.com/TruePikachu/pmDJ, so check there for further updates to my tools. EDIT: Same is going for my notes, now at https://github.com/TruePikachu/pmDJ/wiki Sounds good! wonder mail code for us version. That explains everything Honestly, I never even used wonder mail before.. I didn't even attempt a friend rescue So I wasn't too sure what those were. What do those do in particular though ?
TruePikachu Posted October 21, 2014 Posted October 21, 2014 Boy Wonder Mail has gotten shorter, in that case...then again, I know that a lot of the PMD2 ones are checksums. I should get a packer/unpacker written at some point... (yes, I have a copy of the source of that one tool, and the GameFAQs threads bookmarked).
kr3nshaw Posted October 25, 2014 Posted October 25, 2014 I've uploaded the code for swd2dls to https://drive.google.com/folderview?id=0B8n9Q6oziVxuRVBNOTJ6djNMNEU&usp=sharing. All that it does at the moment is read the wavi chunk for offsets that it uses to extract samples from the pcmd chunk. There are two unknown 32-bit integers that I'm sure have something to do with the length and the loop point of the samples, but I haven't quite worked that part out yet...
psy_commando Posted October 25, 2014 Author Posted October 25, 2014 I've uploaded the code for swd2dls to https://drive.google.com/folderview?id=0B8n9Q6oziVxuRVBNOTJ6djNMNEU&usp=sharing. All that it does at the moment is read the wavi chunk for offsets that it uses to extract samples from the pcmd chunk. There are two unknown 32-bit integers that I'm sure have something to do with the length and the loop point of the samples, but I haven't quite worked that part out yet... Sounds good! You might want to take a look at the Wav format. Given it can contain loop points info as well, and would give you a good idea of how its commonly done, and what data needs to be stored! On my side, I've been writing a sprite analyser tool. It might just help figure out the format! At least I hope.. ^^;
TruePikachu Posted October 27, 2014 Posted October 27, 2014 I came up with a theory with the internal filenames last night... ...need to find my DS Lite, ARDSi, and EoT cart... (I know that there is a _lot_ of debug stuff in EoT/EoD, maybe filenames?) ...I wonder if dsLinux can get read access to the cart inserted into the ARDSi...simplifies things if I can grep from the system itself... EDIT: nope, grep of B_DUN_BANNIN_01 results in only matching the bgm0056 SMD/SWD files
evandixon Posted October 28, 2014 Posted October 28, 2014 Speculating, I'd say that the internal names are left over from development. Especially seeing that most of them are only a ~14 character substring of what would logically be the file name. It could be that upon compiling the ROM, the music names were replaced with numbers.
TruePikachu Posted October 28, 2014 Posted October 28, 2014 Speculating, I'd say that the internal names are left over from development. Especially seeing that most of them are only a ~14 character substring of what would logically be the file name. It could be that upon compiling the ROM, the music names were replaced with numbers. Which is exactly why I checked the EoT ROM, since EoT/EoD has a whole lot of debugging stuff that was removed from EoS.
psy_commando Posted October 28, 2014 Author Posted October 28, 2014 I'd say its just an unwanted/unneeded feature of the toolset they've used. There are really no need for meaningful filenames in there, given its the final copy that they ship to production. Also, I've been discovering a few interesting things about sprites.. I still don't have a clue of where to find the actual resolution, but I think I might be close to figuring it out. For those interested, I made myself a little program to analyze sprite files in depth, and fill a report on them : It only works on uncompressed sprites right now, and is still kinda glitchy, but the reports are really useful ! I'm just too slow at drawing parallels based on the data I found in those.. So if anyone wanna run it on a bunch of sprites and try to figure out what is what, that could help ! There are no readme, but the program can print its own detailed readme, if ran without arguments. And the source code was commited to the codeplex page for the library. ppmd_spranalyser_alpha.zip
psy_commando Posted October 31, 2014 Author Posted October 31, 2014 (edited) Ok, so after looking at a ton of sprites, I think I might have found a few correlations here and there between the format of the sprite and the value of the 9 boolean values spread around the 3 parts of the sprite header. But, I think that, I might need the opinion of others to definitely find out what it is exactly that I've found.. So lets compare 3 sprite files. First a regular character sprite, then a sprite for some UI element, then another sprite for some UI element but with some difference. Character sprite "/MONSTER/m_ground.bin/file_0001": Sprite Sub-Header : ------------------------ Offset img data.........: 0x00004004 Offset unk data.........: 0x00003FEC Unknown value0..........: 0x0001 Unknown value1..........: 0x0000 Sprite Info Block : ------------------------ Offset table E..........: 0x00003968 Offset table F..........: 0x00003A68 Offset table G..........: 0x00003F0C NbEntries table G.......: 13 Unknown.................: 4 Unknown #1..............: 0x0000 Unknown #2..............: 0x0000 Unknown #3..............: 0x0000 Unknown #4..............: 0x0000 Sprite Frame Data Block : ------------------------ Offset frame ptr table..: 0x00003F74 Offset palette end......: 0x00003958 Unknown #1..............: 0x0000 Unknown #2..............: 0x0000 Unknown #3..............: 0x0001 Nb frames...............: 30 The UI sprite ("/FONT/cursor.wan"): Sprite Sub-Header : ------------------------ Offset img data.........: 0x00000958 Offset unk data.........: 0x00000940 Unknown value0..........: 0x0000 Unknown value1..........: 0x0000 Sprite Info Block : ------------------------ Offset table E..........: 0x00000908 Offset table F..........: 0x00000000 Offset table G..........: 0x00000928 NbEntries table G.......: 1 Unknown.................: 8 Unknown #1..............: 0x0000 Unknown #2..............: 0x0000 Unknown #3..............: 0x0001 Unknown #4..............: 0x0000 Sprite Frame Data Block : ------------------------ Offset frame ptr table..: 0x00000930 Offset palette end......: 0x000008F8 Unknown #1..............: 0x0001 Unknown #2..............: 0x0001 Unknown #3..............: 0x0001 Nb frames...............: 4 And an alternate version version of it ("/FONT/cursor16.wan"): Sprite Sub-Header : ------------------------ Offset img data.........: 0x00000398 Offset unk data.........: 0x00000380 Unknown value0..........: 0x0000 Unknown value1..........: 0x0000 Sprite Info Block : ------------------------ Offset table E..........: 0x00000348 Offset table F..........: 0x00000000 Offset table G..........: 0x00000368 NbEntries table G.......: 1 Unknown.................: 4 Unknown #1..............: 0x0000 Unknown #2..............: 0x0000 Unknown #3..............: 0x0000 Unknown #4..............: 0x0000 Sprite Frame Data Block : ------------------------ Offset frame ptr table..: 0x00000370 Offset palette end......: 0x00000338 Unknown #1..............: 0x0001 Unknown #2..............: 0x0000 Unknown #3..............: 0x0001 Nb frames...............: 4 Just compare the values that aren't pointers/offset or 32bits integers. Its a pretty odd way of indicating the format of the file no ? I mean, in those 20 bytes they could have fitted a lot more details about the sprite data.. Which kinda makes me doubt a little of what I found here.. Could these be only coincidences ? Or are those really indicating details about the format ? EDIT: Here are a few more that differ: "/GROUND/d01p11b2.wan" Sprite Sub-Header : ------------------------ Offset img data.........: 0x00005D4C Offset unk data.........: 0x00005D34 Unknown value0..........: 0x0000 Unknown value1..........: 0x0000 Sprite Info Block : ------------------------ Offset table E..........: 0x00005854 Offset table F..........: 0x00000000 Offset table G..........: 0x00005CE0 NbEntries table G.......: 1 Unknown.................: 6 Unknown #1..............: 0x0000 Unknown #2..............: 0x0000 Unknown #3..............: 0x0000 Unknown #4..............: 0x0000 Sprite Frame Data Block : ------------------------ Offset frame ptr table..: 0x00005CE8 Offset palette end......: 0x00005844 Unknown #1..............: 0x0000 Unknown #2..............: 0x0000 Unknown #3..............: 0x0001 Nb frames...............: 19 "/GROUND/d79p41a.wan" Sprite Sub-Header : ------------------------ Offset img data.........: 0x00002050 Offset unk data.........: 0x00002038 Unknown value0..........: 0x0000 Unknown value1..........: 0x0000 Sprite Info Block : ------------------------ Offset table E..........: 0x00001F8C Offset table F..........: 0x00000000 Offset table G..........: 0x00001FCC NbEntries table G.......: 1 Unknown.................: 15 Unknown #1..............: 0x0000 Unknown #2..............: 0x0000 Unknown #3..............: 0x0000 Unknown #4..............: 0x0000 Sprite Frame Data Block : ------------------------ Offset frame ptr table..: 0x00001FD4 Offset palette end......: 0x00001F7C Unknown #1..............: 0x0000 Unknown #2..............: 0x0000 Unknown #3..............: 0x0003 Nb frames...............: 25 "/GROUND/as001.wan" Sprite Sub-Header : ------------------------ Offset img data.........: 0x00000A48 Offset unk data.........: 0x00000A30 Unknown value0..........: 0x0000 Unknown value1..........: 0x0000 Sprite Info Block : ------------------------ Offset table E..........: 0x00000A10 Offset table F..........: 0x00000000 Offset table G..........: 0x00000A18 NbEntries table G.......: 1 Unknown.................: 16 Unknown #1..............: 0x0000 Unknown #2..............: 0x0000 Unknown #3..............: 0x0000 Unknown #4..............: 0x0000 Sprite Frame Data Block : ------------------------ Offset frame ptr table..: 0x00000A20 Offset palette end......: 0x00000A00 Unknown #1..............: 0x0000 Unknown #2..............: 0x0000 Unknown #3..............: 0x0004 Nb frames...............: 4 There are a lot more unusual values here ! Edited October 31, 2014 by psy_commando put the wrong report for the first sprite..
TruePikachu Posted November 3, 2014 Posted November 3, 2014 (edited) I'm taking research requests right now, if you want, I can see about checking all the sprites and trying to convert them to something simple like .bmp . I've been really bored recently, and keep dying in NetHack... Alternatively, if someone has another good suggestion for me to try to research, if I deem it "interesting" (something I would have use for), I'll work on it. Are the backgrounds, etc. researched enough to extract? EDIT: I should see about getting `ndsfs` set up on my VM... EDIT: I can't pick up any sort of magic number off my m_ground.bin: $ hexdump -C m_ground.bin | head 00000000 00 00 00 00 57 02 00 00 00 13 00 00 f0 b3 00 00 |....W...........| 00000010 f0 c6 00 00 30 41 00 00 20 08 01 00 10 3a 00 00 |....0A.. ....:..| 00000020 30 42 01 00 10 3a 00 00 40 7c 01 00 c0 ad 00 00 |0B...:..@|......| 00000030 00 2a 02 00 a0 44 00 00 a0 6e 02 00 30 3d 00 00 |.*...D...n..0=..| 00000040 d0 ab 02 00 e0 a8 00 00 b0 54 03 00 d0 3d 00 00 |.........T...=..| 00000050 80 92 03 00 d0 41 00 00 50 d4 03 00 80 31 00 00 |.....A..P....1..| 00000060 d0 05 04 00 a0 31 00 00 70 37 04 00 c0 37 00 00 |.....1..p7...7..| 00000070 30 6f 04 00 90 2b 00 00 c0 9a 04 00 70 22 00 00 |0o...+......p"..| 00000080 30 bd 04 00 10 38 00 00 40 f5 04 00 00 2b 00 00 |0....8..@....+..| 00000090 40 20 05 00 30 2f 00 00 70 4f 05 00 c0 34 00 00 |@ ..0/..pO...4..| EDIT: Figured that out, wrote my own unpacker, working on understanding SIR0 Edited November 4, 2014 by TruePikachu
psy_commando Posted November 4, 2014 Author Posted November 4, 2014 You didn't have to do all that though ^^; Pack files are well known by now. I've had the extractor and re-packer tool coded and on the front page for months now. SIR0 also has been figured out for a long while, its all in the common format notes on the first page of the thread. You might want to check those out, and go from there. The sprite analyser I posted a few post ago is all based on that knowledge. EDIT: And, we've already got a tool to extract sprites to indexed pngs.(I had bmp support, but its just redundant at this point given png does exactly the same thing and has good compatibility) But the thing is, right now we have to guess the resolution of each frames in the sprite, given they can all have different dimensions. 16x32, 32x32, 8x16, name it.. And they can be laid out on the width or on the height.. So there must be something in the sprite file that we didn't see that indicates the proper image resolution.. Maybe something indicating the amount of tiles on the width and height, or simply indicating the resolution in pixels ? Plus, we also found out that some sprite can be in another format than 4bpp... And so far the only obvious differences I spotted using my tool were a couple of 16bits integer. If we can figure resolution out, and then the way animations are stored, we'll be pretty much able to rebuild a sprite from scratch !
TruePikachu Posted November 4, 2014 Posted November 4, 2014 Yeah, I'm writing my own parser so I understand the format, and so that I can more easily try to find the pattern. Do not question my methods
psy_commando Posted November 4, 2014 Author Posted November 4, 2014 Well, that's all good. But I'm just saying that it might be better not to burn yourself out on things like that before getting to the core of the problem! I mean, I'm probably stuck in a way of thinking that keeps me to see the very obvious solution to the issue basically. And maybe that anyone else looking 5 minutes at it could find out what I missed ^^; Its not the first time.. Remember those signed values I mistook for something else in the smd files?
TruePikachu Posted November 4, 2014 Posted November 4, 2014 Well I might have found the pattern, actually, looking only at the Bulbasaur SIR0. Working on coding it up to check, but it looks like the SIR0 includes the actual animations as well (of which the frame sizes would be encoded inside). EDIT: DBS_ENTRY+0x0 is a frame ID, DBS_ENTRY+0x4 & 0xC0 looks to determine the width of the sprite. Further research is required.
psy_commando Posted November 4, 2014 Author Posted November 4, 2014 Yep, they most likely contain animation data. There are 4 places I know of that could actually contain those though. But DBS is a pretty good candidate. Datablock F was pretty high on my list, because in the UI sprites and anything that wasn't a pokemon, the entire block isn't even there at all.. But the data is too short and random to be a sequence of frames... And you might not want to call those SIR0, because SIR0 is a generic container. Like, it can contain PKDPX files and literally anything you want. Most loose files that have the SIR0 header and contain sprites, in the FONT and GROUND folders, have a .wan file extension. Some have another extension though. I'd speculate that they possibly might use 2 nybbles to indicate the amount of vertical and horizontal tiles, given at 40x40 we're not yet past the nybble's max value in terms of tiles. And if you want to take a look, here's a detailed report on Bulbasaur's sprite: https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/file_0000_0x1300.log It kinda structures/organizes the data a little. Maybe it will help.
TruePikachu Posted November 4, 2014 Posted November 4, 2014 https://github.com/TruePikachu/unSIR0/blob/master/025M.dump Or, if you want to see size selection, https://github.com/TruePikachu/unSIR0/blob/master/001.dump (grep /width=2/)
psy_commando Posted November 4, 2014 Author Posted November 4, 2014 Oh, so you cross-referenced DBG,DBH,DBI with DBS ? That looks pretty solid to me! Nicely done! I can't believe I didn't try to do that.. Even though I'm hardly surprised I didn't given my track record ^^; You should try it on "FONT/cursor.wan" and "FONT/cursor16.wan", and then on a sprite file with a lot of randomly sized frames, like vaporeon's, its file number 146 or something really close in "MONSTER/m_ground.bin" (eevee is 145).
TruePikachu Posted November 4, 2014 Posted November 4, 2014 Looks like Vappy is done strangely: 42 realFrameID=26 val0..4=-39,1,-13,64,3072 SpriteFrame: (width=4): ................................ ................................ ................................ ................................ ................................ ........................1....... .......................181...... .......................1861..... 43 realFrameID=28 val0..4=-9,65,-13,-128,3072 SpriteFrame: (width=4): ........1189AA981............... ........19AAAA981............... ........199A9AA891.............. ........19998AA981.............. ........19988AAA891............. .......189818AAA891............. .......1991189AA98A1....11...1.. ......11981.199AA8991...191.191. ......1981..1AAAA98A1...1A911A1. .......111.19AAA8A8991..19A19A1. ...........189A91A98A1...198A91. ............119981A8991...19A1.. .............11111A98A1...1981.. ..................1A8A1..1891... ..................1A88A11891.... ..................18A8899981.... Care to confirm file frame numbers 26 and 28, data-wise? I wouldn't expect the sprite data to be chopped in this odd way, how I'm reading it. I get similar oddness with Torterra, 0x0130=304 EDIT: Neither of those font .WANs give me any sprite data from animation, but four animations are detected for them.
psy_commando Posted November 4, 2014 Author Posted November 4, 2014 Yeah, I specified those particular cases because they completely threw me off ! The cursor16.wan has a different image format, or maybe its cursor.wan.. I mix them up all the time.. What do you mean data wise ? You mean like the report from my tool running on that sprite ? I'm also pm-ing you the images I've extracted from that. AFAIK the frames are really "chopped" that way.. EDIT: Here's what I got for Vaporeon: https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/temp/file_0146_0x2d2e70.log And here's what I got for cursor.wan and cursor16.wan : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/temp/cursor.log https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/temp/cursor16.log ( Just a word of caution: never trust the "average" values in those reports, there's something freaky going on with that.. Probably some implicit signed/unsigned conversion messing the result up.. ) Also, I'll have to sign off for a few hours, so it will take some time before I can reply.
TruePikachu Posted November 5, 2014 Posted November 5, 2014 Wheee! Rewrote my code to make it easier to maintain. Now the textual dumps of sprites will use ANSI coloring (based on the high bits of each component)...and I got animation playback implemented! Vaporeon does have something different with its SIR0, compared to e.g. Eevee; spriteDump and playAnim will both fail with loading Vaporeon ("RESEARCH THIS!" is the error message).
psy_commando Posted November 5, 2014 Author Posted November 5, 2014 Its not just vaporeon, all the eons and several other pokes, along with a very large proportion of things in /GROUND/ also have these little differences.
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now