TruePikachu

I came up with a theory with the internal filenames last night... ...need to find my DS Lite, ARDSi, and EoT cart... (I know that there is a _lot_ of debug stuff in EoT/EoD, maybe filenames?) ...I wonder if dsLinux can get read access to the cart inserted into the ARDSi...simplifies things if I can grep from the system itself... EDIT: nope, grep of B_DUN_BANNIN_01 results in only matching the bgm0056 SMD/SWD files

Boy Wonder Mail has gotten shorter, in that case...then again, I know that a lot of the PMD2 ones are checksums. I should get a packer/unpacker written at some point... (yes, I have a copy of the source of that one tool, and the GameFAQs threads bookmarked).

Just as a heads-up, I'm officially dropping support for pmd2mid in favor of pmDJ, "Pokémon Mystery DJ", a collection of tools which will pretty much do the same thing, as well as convert in the other direction (eventually) I have the codebase, right now, as a clone of the useful things from pmd2mid; project does compile properly right now. Codebase is now hosted at Github, https://github.com/TruePikachu/pmDJ, so check there for further updates to my tools. EDIT: Same is going for my notes, now at https://github.com/TruePikachu/pmDJ/wiki

Check PMs EDIT: psy_commando is confirming good faith. What exactly is that data? I had believed that to be a troll attempt because it doesn't look like proper data; it is groups of 8 alphanumeric characters, but NOT groups of 8 hex characters. Notably, it is sorted. Is it possibly a list of file IDs, stored in the ROM as a char[8]? Can't be AR codes, those (at least on NDS) are groups of pairs of DWORDs, unless Datel is trying to take the route the old Game Genie took with encrypted codes... Hmmm...I'm going to go check something on Bulbapedia... EDIT: Bulbapedia doesn't have a list of PkMn in GtI. Serebii did, 148-151 or something, so there goes that theory, unless alt forms are present and I only counted Kyurem's.

I think those are from Gates to Infinity (3DS), not Explorers (NDS). However, for the entire list of "code from rom", I'm calling BS on that for now, unless more context is given. Yeah, I always typo that extension. I haven't actually made any edits to the ROM, only read data from it. I know there are a huge number of pieces of research possible by modifying the source material (i.e. compiled binary) to see what effects it has on the output, but I've only really done things like that when I'm trying to make the software behave differently in a specific way.

Generally, internal names are useful for making sure that the correct files are actually in the correct locations. I'm not entirely sure that the script system has access to the names; I tried a few greps on the ROM... cDusto:/~ chris:$ md5sum pmd-eos.nds ef6e0ec5e1b482aa77c0b2f623128ef9 pmd-eos.nds cDusto:/~ chris:$ grep -boia B_ENV_BEACH_01 pmd-eos.nds | perl -e 'while(my$p=<STDIN>){chomp$p;my($o,$t)=split/:/,$p;printf("0x%08X\n",$o)if$o;}' # Offsets of "B_ENV_BEACH_01", ignore the 0x00000000 0x04987E05 0x04989D69 0x04B1B894 0x00000000 cDusto:/~ chris:$ grep -boia B_ENV_BEACH_01\\. pmd-eos.nds | perl -e 'while(my$p=<STDIN>){chomp$p;my($o,$t)=split/:/,$p;printf("0x%08X\n",$o)if$o;}' # Offsets of "B_ENV_BEACH_01.", ignore the 0x00000000 0x04987E05 0x04989D69 0x04B1B894 0x00000000 cDusto:/~ chris:$ grep -boia B_ENV_BEACH_01\\.SMD pmd-eos.nds | perl -e 'while(my$p=<STDIN>){chomp$p;my($o,$t)=split/:/,$p;printf("0x%08X\n",$o)if$o;}' # Offsets of "B_ENV_BEACH_01.SMD" cDusto:/~ chris:$ grep -boia B._.E.N.V._.B.E.A.C.H._.0.1.\\..S.M.D pmd-eos.nds | perl -e 'while(my$p=<STDIN>){chomp$p;my($o,$t)=split/:/,$p;printf("0x%08X\n",$o)if$o;}' # Offsets of "B_ENV_BEACH_01.SMD", 2-byte chars cDusto:/~ chris:$ (that inline Perl script just drops grep(1)'s repetition of the match text, and converts the offset to hex) You can continue work on the .swf format, or go back to sprite stuff. I'm pretty much not doing research that requires ROM hacking.

hehehe... I think I've disassembled all I need to do so from that program. Of note: * SET_VOLUME and SET_XPRESS are confirmed to be correct for their current settings (Medley uses SET_VOLUME incorrectly) * New command: SET_MODU, which adjusts the MIDI modulation * All other old commands I've documented are CONFIRMED * No other new MIDI commands * "Proper" way to handle SET_TUNE (gonna port a bit of code over to my project) * Can parse the drum_table.txt and prog_table.txt files; of note, the entries to use are determined by...the UNK_AA-related field! It selects the instrument group to use, it doesn't give priorities. I have the line syntax at the end of my notes file. Time to do some coding! EDIT: I seem to be wrong about the UNK_AA field EDIT: False alarm, value of 0 means defaults EDIT: Just pushed an updated version of the converter to the server. smd2mid (my tool) now supports looping, and will automatically remap instruments if it has those two .txt files in the working directory (which I have in the package). It also has SET_TUNE (now SET_BEND) corrected, SET_MODU support, and UNK_AA display in smdDump. EDIT: As a quick side project, I dumped the in-game music titles for the Sky Jukebox, alongside the filenames: I hope that the full internal filenames are somewhere in the ROM, but it doesn't look like they are I might see about getting the other files identified, but I don't know how plausable it will be (especially since EoS lacks the debug EoT/EoD had). I could try making a hack, though, to change the songs in the Jukebox... Here is the dumper code, if anyone has any interest in it.

IDA, only one I've found that works well for me. There is a free version availible, but it lacks pseudocode and platforms other than x86. C89 doesn't support Unicode proper, where C99 supports it to an extent (at least under gcc, I forget if it was a gcc extension that adds the \u escape). It does look like a C++ compiler was used, given that there are functions with names such as "?strncnt@@YAHPBDH@Z_0" (being "int __cdecl strncnt(char const *,int)"). main() doesn't seem to use anything C++ specific, instead using printf() and scanf() (the latter for the debug mode prompt). It doesn't look like this program will be useful for converting anything outside the BGM: songID = 0; while ( 1 ) { sprintf(smdFilename, "%s/bgm%04d.smd", ParamInDir, songID); sprintf(midFilename, "%s/bgm%04d.mid", ParamOutDir, songID); if ( smd2mid(smdFilename, midFilename) ) break; printf("==============================================\n"); ++songID; if ( songID > 201 ) return EXIT_SUCCESS; }

No source code...guess its time to pull out the disassembler > (I can easily handle x86) EDIT: Looks like C and not C++: if ( argc == 1 ) { printf("SMD to MIDI File Converter V%.2f\n", 0.1); printf("Usage : smd2mid.exe [options] <InDir> <OutDir>\n"); printf("Options:\n"); printf("\t -np \t Not use 'prog_table.txt'\n"); printf("\t -nd \t Not use 'drum_table.txt'\n"); printf("\t -l <count> \t Loop count\n"); printf("\t -d \t Debug mode\n"); printf("\n"); result = EXIT_SUCCESS; } That makes things even easier, since I don't have to mess with classes. EDIT: gah, might need to localize strings: printf("âtâ@âCâïé¬èJé»é_é¦é");

Well, I got a huge list of instrument guesses, and apparently bgm0050 has two drum track programs, 0x7E and 0x7F. Anyway, this grouping isn't very useful without the samples <_<. But I have code to guess groups written up. Going to mess around with the outputted MIDI sequences, see how nice I can get them to be, might help with IDing unknown commands.

Yeah, I had some spare time, Sentry Duty music has drums off of MIDI 16. I changed my local copy to instead identify a drum track by sample ID 0x7F. Speaking of which... I did some digging into the .swd files, and might have figured out the mapping for instruments -> samples. Each SMD instrument has a number of samples, which are selected by the associated SWD; this selects the sample to use by the MIDI key number (!) and possibly velocity, mapping into what I'm guessing is a sample in bgm.swd. So now I have a 3,492 line file which lists all the sample ID numbers (0..948), as well as what instruments in what tracks use them; this is at http://cdusto.selfip.com/pmdMus/instSampMap.txt. Now I just need to figure out the sample clusters (the clusters of samples which are for the same instrument), and make a very big .map file to map sample ID numbers to MIDI instruments. EDIT: And I can't come up with any good way to do the mapping. There is probably more stuff in the .swd which could be used to determine instruments from each other, possibly even sample grouping in bgm.swd, but I'm not properly equipped to do a full analyses of the format right now. I'm going to toss my current code up on my server, and see if I can get some sort of globally-accessable svn (compared to the SVN over SSH setup I have right now). Probably going to do a rewrite of the tools, as they are right now, so that they are more helpful both in finding out what still needs research, and in documenting the format gracefully (like consistantly using the command name enums rather than using the name in some places and the opcode in others).

Just a quick update before I head to college, got the 0xD7 SET_TUNE command documented and implemented, and started research on the UNK_A9 and UNK_AA commands (the ones which have values in the header), or rather I researched those values in the header. With a quick Perl script, I concluded that they do not relate to the playback directly, but might be priority levels. This is supported by the fact that, it appears, no two files have the same values for both the A9 and AA fields (+0x0F and +0x0E) in the header. I think that the A9/0x0F field is an enum which identifies the category of piece (things like enviornmentals versus plot music), while the AA/0x0E field gives a priority or identifier or something within the specific category. It is interesting to note that all the pieces except bgm0077 "B_EVENT_TIMEWHE" have a "large" (0x70<n<0x80) A9/0x0F field, while bgm0077 has 0x01 there. If it is true that these correspond to priorities or something, then the existance of UNK_A9 and UNK_AA (the commands) reveals that there is the capability to dynamically override other pieces, at least in theory (I haven't checked to see if they are used anywhere other than the start of tracks). EDIT: Ninjas! Another piece of supporting evidence for my assignments are that the channel volume always gets set, first thing. I'll check dumps of Northern Desert when I'm able to. Yeah, drum assignments would be handled in the .swd files, because as far as the .smd is concerned, they are just another instrument. However, they are always assigned to output ID 15 (MIDI 16), so it is possible the synth engine itself does special handling with it for some things. RAPIDEDIT: I see what you mean, getting the two swapped right now when I have time

My friend, who is pretty familiar with MIDI, confirms 0xE0 SET_XPRESS and 0xE3 SET_VOLUME, and that seems to follow what you quoted. Every track, I believe, starts with SET_VOLUME 107 (this seems to hold for _every_ track, including tempo), and e.g. track 3 (MIDI 3) for the Medley uses SET_XPRESS as if it is for MIDI expression. I said to do the limiting because MIDI restricts those values to 0..127, and if the format is indeed based around MIDI, greater values could have unexpected results. I have not found anything exceed the range, as well. I haven't done much analyses into the drums, aside from aligning them properly to MIDI. I'm not sure, but the drums might be handled differently from chromatic notes. I know they are, to an extent, in MIDI. I'm working on getting the bulk of everything figured out. EDIT: Work on the SWD prgi chunk; just after the header, there is a list of word offsets (like wavi) which are indexed by instrument ID. That is what would link sample numbers in the SMDs to actual instruments, if anything. First field is a uint16_t matching sample number Final update for the night: * Added WAIT_AGAIN, WAIT_ADD, and SET_PAN commands * Made sure to extend the MIDI track so a NOTEOFF isn't put after the end of the track. This seems to have fixed all file load problems, from my testing. * Added another error condition to smdRead.cpp, which will only trigger when trying to dump enviornmentals * Added another program, checkSmds, which takes a list of SMD files as parameters, and sees if smdRead can read all of them without errors. Right now, only enviornmentals in BGM fail - all music tracks parse with no "100% unknown" commands (read: commands we don't know how many arguments there are for).

Just as a heads-up, you probably should assume that the param for 0xE0 and 0xE3 both range 0..127. But...there are two MIDI params which affect the volume. You ofc have controller 7 MAIN VOLUME, but you also have controller 11 EXPRESSION. I'd bet that 0xE0 might be it, while 0xE3 is volume. Generally, with NDS, a tracker is used, which works with SSEQ, which is packaged with a number of other files in the .sdat, which is what most other games use. PMD doesn't, and that's the main issue. Gonna quickly test this before officially updating the toolset EDIT: Seems legit, got that implemented. Toolset has been updated, same link as before: * Now supports .map files, which remap instruments. A line is matched with the format "0x%02X\t%i"; the map I used for http://cdusto.selfip.com/bgm0003.mid (sample) is: 0x1A 38 0x07 8 0x08 8 0x15 26 0x01 98 0x0E 6 0x4B 45 0x0F 15 0x02 75 0x61 0 * Timing for 0x8D..0x8F has been reduced by one tick; this fixes the desync issues * Documentation updated for SET_XPRESS command * Timing added to smdRead.cpp and smdDump: debian-VM:/~/projects/pmd2mid chris:$ ./smdDump bgm0002.smd | head -n 20 Name: B_SYS_MENU.SMD Track count: 12 Track 0: tID=0, oID=1 Events: (5 count) 0 1:1.00 0xE3 0x71 Volume: 113 1 1:1.00 0xA4 0x68 Tempo: 104 2 1:1.00 0x99 -- LOOP -- 3 1:1.00 0x93 0x00 0x09 Wait: 0x0900 ( 2304/192 ) 4 13:1.00 0x98 -- END -- (end events for tID=0, LOOP=12:0.0) Track 1: tID=1, oID=1 Events: (131 count) 0 1:1.00 0xE3 0x71 Volume: 113 1 1:1.00 0xA9 0x79 UNK_A9: param1=121 2 1:1.00 0xAA 0x41 UNK_AA: param1=65 3 1:1.00 0xE0 0x5A Expression: 90 4 1:1.00 0xE8 0x40 UNK_E8: param1=64 5 1:1.00 0xE3 0x71 Volume: 113 6 1:1.00 0xAC 0x4B Sample: 75 * Dumped MIDI files have META marker events indicating UNK_xx instructions; this is in addition to the old "LOOP" markers I had in there Known bugs: * I forgot to update the NOTES file, so ignore that it says that .map support isn't implemented I now believe they started with, and converted from, MIDI: * SET_VOLUME and SET_XPRESS * 48 ticks per clock (pretty common) * Container is NOT a .sdat * Drum set matches MIDI ordering and notes (oh, I got that working) ** Drum set tends to be on output 15 (MIDI channel 16) (the MIDI dumper will automatically swap Ch10 with Ch16, so drums are correct for GM)

Right now, I'm not very concerned with the .SWDs. However, a tool that can make them into .SF2s would be useful for checking progress on SMD research. I can't suggest any particular libraries for NDS formats, as I don't have experience with them myself. I'd ask my friend, who has marginally more experience, but they're offline right now. Right now, from my perspective (which is only concerned with reading), 0xE0 is the channel volume. It is not set in stone, and just represents a guess based on how it is never nonzero in the files I've tested, like BGM0002 (Title) Track 3: tID=3, oID=3 Events: (93 count) 0 0xE3 0x71 UNK_E3: param1=113 // Volume command? Also not decimal-round 1 0xA9 0x79 UNK_A9: param1=121 2 0xAA 0x41 UNK_AA: param1=65 3 0xE0 0x50 Volume: 80 // 0xE0 command. Note the decimal-round '80' value 4 0xE8 0x40 UNK_E8: param1=64 5 0xE3 0x71 UNK_E3: param1=113 // Repeated volume command? Might contribute proof 6 0xAC 0x06 Sample: 6 7 0xD7 0x00 0x00 UNK_D7: param1=0, param2=0 8 0x99 -- LOOP -- If you actually hacking the sequences experimentially can confirm that it is an echo, then I have to figure out how to implement that in midiFile.cpp. I'll do some tests with it as volume, I guess. By all means, you can update your notes with my findings that you agree with. It is what I did with yours, after all. (post …) Game sequencer routines in general are very messy. Only reason I could do research on GB games were because of the helpful .gbs format, which isolates that code from the rest of the code. I don't believe the songs were written in MIDI, per say, first. Definitly they were written in something like a tracker, of some sort, but not necessairily with MIDI. Then again, there is that 48 ticks per beat, which is pretty common with MIDI, so who knows. Regarding the timing, I'm working under the assumption that the there are 48 ticks per beat. Each DELTA_TIME event (previously prefixed-time parameters) waits for a specific number of ticks, as with the WAIT_1BYTE and WAIT_2BYTE commands. This makes BGM0002 match up extremely nicely with the actual song (loaded in Audacity with a click track acting as a metronome). Regarding the backgrounds, I decided to just load up the game in my emu, with a 100% save from GameFAQs (too lazy to look for my DS Lite + ARDSi to dump my actual savegame), and stitch the tiles together with GIMP. Regarding line 41, I just realised that I typoed. Thanks for pointing that out. It is supposed to be 'fname', not 'name'. If you are talking about the "throw" lines: EFAULT: Thrown if the read pointer for a track goes past the end of the track ENOTSUP: Thrown if an event opcode which isn't recorded in the table at the end of my notes is encountered. The main reason is because it needs someone to tell it how many parameter bytes there are for the instruction, since I don't know a good way to program a guess. Both of those stop reading for the track, since it doesn't know how to proceed in the latter case, and it did something wrong in the former. I pretty much exclusively use gcc for compiling. I have a Debian VM on my laptop (host OS is Win7) which I do my programming inside of. In Linux, build instructions consist of './configure', followed by 'make'.

If you are speaking about the .mid on my server, know that it does not reflect the current state of the program; IIRC it was working on the assumption that a NOTE_PLAY event had 1-2 extra bytes (and it would guess about the existance of the 2nd extra byte from the value), rather than the more-accurate checking the flag nibble for length setting. It might also not be taking into account what might have, at that time, been believed to be echo (actually volume). As I said in my version of the specification, it is all thanks to psy_commando that I managed as much as I had. I have, in the past, reverse engineered sequencer formats for the GameBoy (partially by use of a disassembler), so I have that experience to help, but I don't want to even try (and I'm not sure my software even can) disassembling ARM assemblies, so the work here is pretty much just listen, guess, and check. I actually came across this thread (and, by extension, the notes) while searching for a tool to dump the BGs from Sharpedo Bluff; I was trying to get a 3D model assembled in Blender, but eventually noticed that the three images that I had (beach view, exterior, and interior) don't actually match up...I blame space/time getting screwed up (which also explains the mysterious disappearing dorsal fin...). I think Google picked up on your message at the top of the 3rd page (*.rawimg), so you are partially to blame for progress

Meanwhile, I've been hacking away at decoding .smd files. Got quite a bit of work done, in fact. http://cdusto.selfip.com/pmdMus/smd_format.txt is my textual version of my notes on the format (based off psy_commando's work, but I have some changes and a bit more discovered). Additionally, I wrote up a couple of very-alpha tools in http://cdusto.selfip.com/pmdMus/pmd2mid-trunk.tar.gz - the dumper tool (smdDump) is pretty useful for looking over the files (and one can attempt to transcribe from them), while smd2mid, very much work-in-progress, somewhat manages to make .mid files (albeit with incorrect instruments) from .smd files. There are still some bugs that need to be worked out, partially on the MIDI encoding side of things, partially on the SMD decoding side. I get desynchronization in BGM0003 (Marowak Dojo), and BGM0085 (I Don't Want To Say Goodbye)...well... Anyway, I might pull a friend into helping with figuring out some of the unknown commands (supplying said friend with .ogg and .smd versions of some songs, since they don't play PMD). They already managed to help quite a bit, finding problems in the MIDI files I was generating. EDIT: Since not everyone will probably want to try to compile the tools, http://cdusto.selfip.com/bgm0003.mid should play in Windows Media Player, as proof-of-concept (and an older version of the tools)