psy_commando

I.. do.. think.. we're.. getting.. there.. I still have a couple of bug, like notes getting out of sync at the end on some tracks, the issues with the GM track 10 being forced to be a drum track, bgm0026 is pretty much the only track that most players I tried couldn't play, and I'm ignoring a lot of events here and there, and I haven't setup a conversion table yet to convert the PMD2 preset number to General Midi.. But its beginning to sound pretty decent ! EDIT: It also works with Shiren 2 ! What do you mean by "These strings can return a name and species when using c_name and c_kind." ? I mean c_name and c_kind are supposed to return the name and specie of the speaker in a dialog, but what's the link with those strings in all cap ? That sounds interesting.

I've made a lot more progress with the smdl and swdl format! I can parse entire songs now, and I can export midis. Except that I still have a bug to deal with, thanks to the midi lib I'm using.. I'm still looking for a C++ SF2 lib to export a soundfont, and not have to write everything by myself ! I wasn't very lucky finding any.. I also took a look at how the timings for the notes, and the pauses work. I was able to calculate the duration in ticks of each types of prefix delays. And the default PPQN value for smd files is stored in the "song" chunk, its set to 48 by default, the same default as MIDI files. **Disclaimer: Note durations names may make no sense..** The following values under the column "Engine ticks 48 PPQN" were all taken directly from the game, by running 2 tracks, with 2 repeating notes simultaneously, one with prefixed delays and the other with silences with a set duration in ticks instead. Note events in track 1 had all the same pitch, and played note 0. Note events in track 2 had all the same pitch and played note 5. |======|==================================|========================|======================| | Code | Duration | Midi Ticks at 960 PPQN | Engine ticks 48 PPQN | |======|==================================|========================|======================| | NA | Whole note | 3840 | 192 | *Whole note included as reference only ! |------|----------------------------------|------------------------|----------------------| | 0x80 | half note | 1920 | 96 | | 0x81 | dotted quarter note | 1440 | 72 | | 0x82 | 2/3 of a half note | 1240 | 64 | | 0x83 | quarter note | 960 | 48 | | 0x84 | dotted 8th note | 720 | 36 | | 0x85 | 2/3 of a quarter note | 600 | 32 | | 0x86 | 8th note | 480 | 24 | | 0x87 | dotted 16th note | 360 | 18 | | 0x88 | 2/3 of a 8th note | 320 | 16 | | 0x89 | 16th note | 240 | 12 | | 0x8A | dotted 32nd note | 180 | 9 | | 0x8B | 2/3 of a 16th note | 160 | 8 | | 0x8C | 32nd note | 120 | 6 | | 0x8D | dotted 64th note | 90 | 4 | | 0x8E | 2/3 of a 32nd note | 80 | 3 | | 0x8F | 64th note | 60 | 2 | |======|==================================|========================|======================| I can't remember if truepikachu came to the same conclusions. But at least we're sure of their timing now! Every time values for all events also uses PPQN ticks to specify it duration. Also, Here's where the updated song chunk header structure: ------------------ SONG: ------------------ Chunk holding data that applies to the entire song. NOTE: Looking at sample SMDL files from 3 different games, all from Chunsoft, the only thing that changes is the nb of tracks and nb of channels. So its hard to determine what the rest is for or in what format it is.. Offset: Length: Type: Description: ------- ------- ------- ----------------------------------- 0x00 label 4 char[4] Song chunk marker "song" {0x73,0x6F,0x6E,0x67} 0x04 unk1 4 uint32 So far 0x0000 0001 (Little endian) 0x08 unk2 4 uint32 So far 0x10FF 0000 (Little endian) 0x0C unk3 4 uint32 So far 0xB0FF FFFF (Little endian) 0x10 unk4 2 uint16 So far 0x0100 (Little endian) 0x12 ppqn 2 uint16 Pulse Per Quarter Note. Usually 0x30. (See how PPQN works with MIDI) 0x14 unk5 2 uint16 So far 0x01FF (Little endian) 0x16 nbtrks 1 uint8 Number of track chunks. 0x17 nbchans 1 uint8 Possibly number of midi/output tracks. 0x18 unk6 4 uint32 So far 0x0000 000F (Little endian) 0x1C unk7 4 uint32 So far 0xFFFF FFFF 0x20 unk8 4 uint32 So far 0x0000 0040 (Little endian) 0x24 unk9 4 uint32 So far 0x0040 4000 (Little endian) 0x28 unk10 2 uint16 So far 0x0002 (Little endian) 0x2A unk11 2 uint16 So far 0x0008 (Little endian) 0x2C unk12 4 uint32 So far 0x00FF FFFF (Little endian) 0x30 unkpad 16 - 0xFF Padding, unknown reasons.. ------- ------- ------- ----------------------------------- Total: 64 (0x40) There is still a very long way to go though. But at least, most of those values are the same between every single PMD2 tracks. Here the link to my full notes btw : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/PMD2_MusicAndSoundFormats.txt On the side of map tiles, I keep chickening out on having to go through trial and error to figure out the format.. If I had a good understanding of the scripts this might be easier though, seeing as each maps are held together by scripts.. But I don't know much about them.. ^^; And well, I really could use feedback on ppmd_gfxcrunch, and on the pre-release ppmd_statsutil. I'm looking to improve those before setting things in stone for the next few releases, and I need to know if the utilities are suitable for what they were made for basically ^^; I've also been working on designing a automated and convenient way of sharing PMD2 mods. Possibly other romhacks as well. Most other romhacks use a single differential patch for the whole rom, but that's problematic, because it doesn't support different revision and languages of the game, or anything that modifies the CRC of the ROM. And you cannot combine romhacks together, even if they could really be combined (think of a sprite pack and another hack that changes only, say, pokemon names for example ). So, I've considered making a set of 2 utilities. One for easily and automatically assembling a mod package, and the other would be a redistributable utility that would handle extracting and patching selectively the ROM. A mod package is just a zip file, that would contain: - The patch installer - ndstool (might just merge the source code into the installer) - xdelta (might just merge the source code into the installer) - A xml file with the details for patching the ROM, how to handle other languages, what CRCs are supported, etc.. - A directory containing a replica of the ROM's file structure, filled with differential binary patches of original ROM files(for "legality" reasons), and full copies of any new files added. The directory with the files would contain sub-folders, and would be named according to the game ID string they contain files for. So that means there would be a single package for a romhack that was localized in multiple languages for instance! The nice thing, is that people wouldn't need to download patching tools, and do things themselves if they're total noobs and etc.. The patcher would unpack the rom, patch/add the files, rebuild the rom, and that's it. Conflict handling could be done via CRC checks, so the application could ask the user what files they want to keep. The concept could be pushed further even! Any opinions ? Nice ! Don't worry about being late. I haven't been as productive as I wanted to as well.. Is 0x9D9AC near all the other game strings, or is it within the ARM binaries/overlays ? And all the tags I have in my notes are straight from the game's dialog, so they're most likely used. Some of them are context sensitive, and rely on some variables being set. I'd assume that you can fill those variables or call some functions from the script to set those. It might explain why some crash the game, considering strings are referred to via pointers, and dereferencing a null pointer is an instant segfault.

Like evandixon said, you'll probably want to try his sky editor. He's using the utility I made in the background, but he made a more intuitive user interface for it. It would probably be easier for you since you've mentioned you're just starting out.

Character sprites are already exportable. And re-importable too, but some things still need to be figured out to import a few particular sprites properly. Move animations are just regular sprites re-used and animated differently, and some particle effects are layered over them. The cutscenes and some backdrops are stored as a single big image. Some animated backdrops are stored as several images in the same file. I've been adding support for that, but there are still some major things I want to figure out before updating gfxutil. Everything else is stored as tiles. And I've been trying to figure out the tiles' format, but I'm a bit stumped for now. So I went to work on the game's audio instead, waiting to get some kind of big realization.

That's pretty much the idea.. But there are still formats that we haven't figured out yet. And with the tools you could also just make a romhack of the game, instead of writing one from scratch.

Hey ! I made a lot of progress with the sounds files ! I finally realized that the ADPCM samples are actually really encoded using the standard IMA ADPCM format.. >_< I was just handling the samples in the wrong order.. To celebrate, I'll put this little extremely barebone utility that can dump both pcm samples and adpcm samples from SWD files from most games that uses them that I know of Just pass it a swd file that has samples in it, as parameter or via drag and drop. It will dump the wav in the same folder. (its not really meant to be practical) I also tried to write the loop points in the wav file, but I'm not sure how they work yet.. Anyways, here it is : https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/pswdldump_0_0.zip I'll write a real utility eventually.

So, I've been messing around with the SWD files, and I've figure out a lot more about the format. I can extract individual samples, and etc. And I updated my notes : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/PMD2_MusicAndSoundFormats.txt Though, I still haven't found the proper ADPCM implementation they're using. Its possible its the official Nintendo implementation, since I noticed calls to the SWI ADPCM decoder. But I don't know how it works exactly. Using the standard IMA ADPCM tables and implementation yielded pretty quiet and deformed samples.. Not all samples are ADPCM, some are PCM 16 bits. Its indicated in the sample table by a flag value. Also, the swd files that accompany each bgm tracks refers to samples stored in the main bgm.swd bank via their location in a pointer array. Its hard to explain, but it might be clearer upon reading my notes. Tags are pretty interesting. Have you tried each of those tags to see what they do ? I tried out a bunch of them in the past and figured what some of them did : https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/text__str.txt (the LS tag is really interesting) And you can even use "ppmd_statsutil" to convert the text from the .str file into a .txt file and edit the text and then use it again to convert it back into a .str file.

That could help. But yeah, each games handles the logos differently, depending on the image formats it uses.

I talked with someone earlier this week who found it in Pokemon and Colosseum by looking in memory for references to the index of the strings for the type names in the text_e file. I haven't had the time yet to try it on my own, but it sounds like a good way of finding it.

Well, usually 4 bits per pixel images use a palettes. 4 bits is enough to count from 0 to 15, so that allows 16 colors. While the approach you're suggesting would be 2*2*2 colors, which is 8 possible colors. So by using the same space, you have less possible colors. That doesn't seem like it would make much sense, since the space saved is minimal, and the image is compressed anyways. And, I doubt the gamecube interpolate pixels like that. I don't really see why they'd do that vs displaying the actual image at the screen's resolution in a single pass, there is no benefits. I guess the best way to know for sure would be to dump the frame buffer and or back buffer when the image is drawn, and look at how the raw pixels are organized. That would be a good lead. Or maybe just taking a screenshot, but then you'd have to deal with whatever file format its saved in.

That's a pretty nice find ! Btw, have you ever stumbled on the type effectiveness table, or anything of the sort by any chances ?

I doubt it works that way. If it was only 2 colors, it would be 1 bit per pixel. But the image has gradated colors. So its not just 2 colors, but several. I'm guessing there's probably something we've missed in there. The alternating patterns would create faint vertical blue strips on the resulting image. But it actually doesn't.. So either some bytes are ignored, or either they got a particular purpose

I tried taking a look at those, but they're a little odd. Looking at the empty pixels, there seems to be a repeating pattern every 10 bytes.. And I can't get the image to display properly in any formats either. At 4-8-16 bpp there are still gaps between valid pixels that have an odd coloration.. It might be possible to figure it out by editing some pixels and running the game to see what happens, but that's pretty tedious with this game.

I just realized something For some reason, I just completely derped and read bits as byte on the user interface.. even though I've used that thing for nearly a million times before ^^; And I got it to show up: 16 bits per pixels, RGB, little endian, tiled 4x4. And the start offset is 0x80.

Nice! And I've taken a look, but I can't seem to figure it out for sure. The format of the pixels differs from though, so its possible there aren't any palette involved. From looking at the repeating patterns, I'd guess they probably encoded pixels on 2 bytes. Possibly 5 bits per color, and 15 bits per pixels. Its not all that common, but it kinda looks like that. Especially since the highest bit of every groups of 2 bytes is always set to 1 in colosseum's pokemon portraits. And in 15bpp images, that bit is either ignored or possibly used as 1 bit alpha. Now, the problem is that I don't know any viewers that handle 15bpp, so you'd probably have to write some code to convert the pixels to a 16bpp/32bpp bitmap or png, and see if it works. Its probably just something as simple as this: uint16_t rawpixel = 0x8C84; //Example value uint8_t r = ( pixel & 0x7C00 ) >> 10; uint8_t g = ( pixel & 0x3E0 ) >> 5; uint8_t b = pixel & 0x1F; Or maybe the opposite order.

Well, the main concern is the header and data structure. Tileggd ignores the header, but the game needs it to parse the image properly. That's why its not all that much of a good indication to look at it in Tileggd.

Looks pretty good ! And yeah, images are pretty easy to work with most of the time. As long as they're not encoded! And have you tested it in-game ?

Alright. And great idea ! Nice ! I suspected it might have had something to do with transparency, but I couldn't get Tileggd to display it properly.

No problems ! Scripts ? That's neat ! That would imply some good modding potential ! I'll try to take a look at those too. Its just a tile viewer. You'll need to write your own code to do this, since its apparently some format developed in-house. Well.. I'm saying that, but its possible its actually the standard Nintendo texture format used on the gamecube ? I'm not too familiar with the hardware and official formats used with the gamecube. It looks like a pretty simple format though. The hard part might be figuring how to rebuild a valid header from scratch though..

HxD is great ! Though if you're craving for something with a lot of features you might want to give this one a try ! : http://www.hexedit.com/ Its free and its full of awesome features, like auto-parsing based on XML templates ! I use it all the time ! One of the best thing is being able to toggle the display mode to "rainbow" so it colors bytes based on their values ! It makes spotting patterns a breeze ! And the integrated "calculator" is incredibly handy! (You can click on @cursor and the calculator will read and apply endianess on the highlighted value! And you can click go to jump at that address or just make calculations with the value, or even convert to decimal ! ) The learning curve is a little steep at first, but the tips of the day and the help are very helpful most of the time ! (And the source code is available on codeplex too !) And, speaking of all this, I actually made a thread around here a year ago, where I posted a cheat engine based "trainer". Its a lua script that when opened in cheat engine allows you to automatically find the player's party and etc.. And you can view most pokemon's stats with it too. -Gale-of-Darkness-Psy_commando-s-utility-and-research-notes'>http://projectpokemon.org/forums/showthread.php?40367-Pokemon--Gale-of-Darkness-Psy_commando-s-utility-and-research-notes ( Please don't mind my replies in that thread too much, I was in a really bad mood at the time ^^; ) And speaking of that, cheat engine works wonder for finding things in games run on Dolphin ! Its internal debugger, and memory search feature are quite handy. That how I managed to find most of the stuff in my notes, along with Dolphin's own debugger ! I've got pretty much the same background then Well, except I didn't begin with modding smash. And what I meant by symbols is just obfuscated functions and variable names, or maybe even comments.( well I'm not sure if comments could be really called symbols though ) Sometimes those are left in the code, and can be really helpful! And yeah, common sense helps a lot ! Though, since its a compiler that wrote that assembly code from C or some other language, you can get some pretty illogical things going on And that's a really clever way to cross-reference the type table ! I didn't even consider them referring to the string in there ! ^^; I'll give that a try in mystery dungeon ! And thanks ! There's something you gotta keep in mind with Future Sight though. Its a psychic move, but in the main series game until Gen 5 it would hit any pokemon even dark types neutrally! And it wouldn't get STAB, and etc.. http://bulbapedia.bulbagarden.net/wiki/Future_Sight_%28move%29 So it might affect the results of your tests. In retrospect, I guess I should have mentioned I was using files from PXD ^^; However, there is still a little thing that bothers me with the PXD textures. If you look closely, you'll notice that there are some artifacts on some of the pictures. Like one of Poochyena's ears is garbled, and Bulbasaur has that weird blue blob appearing near its ears.. So I'm guessing there might be a little bit more to this.. Maybe there is a subtlety about how colors are indexed ? The best thing would be finding the exact pixel in the raw data that appears garbled and look at its value. I'll also try to take a look at other image formats from PXD when I get the chance. Unless they're all the same that is. If you have anything in particular you'd like me to take a look at feel free to mention it !

What kind of garbage? Did you set the palette offset to the offset indicated in the header ? Have you set the palette's format to 2 bytes per color ? Did you change the tiling of the image to 8x4 ? Did you set the image mode to tiled ? Did you use the arrow keys to resize the image correctly ? Yeah, shadow moves are handled oddly. Back when I was editing moves and etc directly in memory, I noticed shadow moves wouldn't work on non-shadow pokemon. So there is most likely something bypassing regular typing checks. Since all ??? moves are different about how they calculate damages, its possible it has a special role. And, its pretty rare to find someone that can handle ASM ! Are you using IDA or just a regular disassembler / hex editor ? Does Pokemon still have symbols left in its binaries ? Also, just by curiosity, how did you proceed to find the type effectiveness table ? I've been struggling for a while to find it for Pokemon Mystery Dungeon, and I'd assume my method isn't very good ! Also, I don't know if it can be of any use to you, but I found the old notes I wrote on how the game works in memory at runtime. It contains mainly memory locations and data structures, along with observations on how the game deals with several things : https://www.dropbox.com/s/qx8zah411t6dbvt/PPDX_MemoryDataStructureInfo.txt?dl=0

It comes with spending nights hitting your head on your desk trying to figure out how the pixels are arranged XP But yeah, I think what was the most problematic with these images was figuring that the tiles were not square.. That's such a weird idea its no wonder it was hard to figure out I'd be careful with using the ??? type as fairy type though. There's no telling what the game does with that type, well besides being used for Curse. And, you can't specify whether its indexed or not. Its assumed automatically for 8 bits images and lower. Which is kind of a weird way of doing it.. Especially since 16 bits indexed images do exist. But, I haven't found any other raw image viewer that works just as well.. And searching on the web for another one is hard, thanks to the RAW image format used in some digital cameras...

I took a look, and so far, it appears the images in the poke_face.sys file uses indexed colors. For example, here's Poochyena, "face286.fdat", and Bulbasaur, "face001.fdat". They're both 8 bits per pixels, tiled (8x4), indexed. Poochyena's is ~48x88, and Bulbasaur's is ~88x128. The offset of the palette is the value usually at offset 0x48 in the header. It appears to be a 512 colors palette, or maybe two 256 colors palettes, possibly one for regular and one for the shiny portrait I'd guess. Its 2 bytes per color, RGB, and apparently stored in little endian.

Its hard to tell without seeing the actual raw data. What are the names of those files in the pictures ? Just so I can take a look on my side ? Have you tried shifting the image in tileggd by a few bytes ? (look in the image menu for "skip size" and change them all to "byte") Since you're using a multiple bytes per pixel image format to display those, its possible bytes from each pixels aren't properly aligned, thanks to a header or non image data put before the image. Its also possible some of those images are using compression, or use a color palette. The pikachu image doesn't look like its compressed though.

I didn't even remember there was a map viewer in the debug menu ^^; But thanks for mentioning it ! This should make things easier, especially if EoS shares the exact same format! And honestly, we figured a good chunk of the wan format(Except those damn orphaned frames..), which is probably the most complicated format in the game, so it shouldn't be too hard to deal with those in comparison. Selective corruption is your friend And from looking at the files, it seems they come in groups of three. 1 bpl, 1 bpc, 1 bma. But it also seems some have a bpa file instead of a bpc. Maybe that the 'a' is for animated ? I also noticed that sometimes the header of the bpc/bpa files gets longer..