psy_commando Posted December 27, 2014 Author Share Posted December 27, 2014 (edited) Alright, I think everything is fixed now. At least, I hope.. I also added support for importing / exporting as 4bpp bitmaps, and as raw 4bpp images with an accompanying RIFF palette ! Here's the updated version : https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/ppmd_kaoutil_0.3.zip EDIT: Also, just leaving this here, but if anyone happens to have any info on the data stored after SIR0 files, it could greatly help me getting the sprite tool done ! I need to make sure the SIR0 are reconstructed properly, and this is the only thing I can't figure out.. Well, not counting where the image resolution are stored at.. Edited December 28, 2014 by psy_commando Link to comment Share on other sites More sharing options...
evandixon Posted December 31, 2014 Share Posted December 31, 2014 Using bitmaps, I've successfully inserted the grin image... except the image was converted to 4bpp using Paint, so it was in grayscale. But the grayscale was shown how I expected it to be. I'm still having trouble encoding the image as 4bpp. FreeImageApi doesn't work with it for some reason, and gimp won't export as a 4bpp bitmap (lowest is a 16 bit bitmap with 16 colors in the pallete). Link to comment Share on other sites More sharing options...
psy_commando Posted December 31, 2014 Author Share Posted December 31, 2014 (edited) Using bitmaps, I've successfully inserted the grin image... except the image was converted to 4bpp using Paint, so it was in grayscale. But the grayscale was shown how I expected it to be. I'm still having trouble encoding the image as 4bpp. FreeImageApi doesn't work with it for some reason, and gimp won't export as a 4bpp bitmap (lowest is a 16 bit bitmap with 16 colors in the pallete). You should try png with gimp again if you haven't. I fixed the issue with png palette smaller than 16. I'll take a look into reading 16 bits bitmaps, since it seems it could help with compatibility. And by the way, happy holidays, and thanks for all the help this far! EDIT: I managed to find a new bug, even though I thought I had it covered.. Images with resolutions not divisible by 8 would crash the program.. I got it fixed in 0.31, and now it expands the image to the closest size divisible by 8! It only does this with PNGs and BMPs. Because, I assume anyone using raw images wouldn't want to have this kind of error handling messing with their stuff. Here's the link : https://github.com/PsyCommando/ppmdu/releases/tag/kaoutil_0.31 Also, I tried converting something that was never meant to be represented with a 16 color palette, and tried importing it as a bmp: I used gimp with it and had no issues at all. I took a few screenshots of what I did, maybe it will help ? (I had to combine them into a single long image, because I'm approaching the limit on imgur.. ): Edited December 31, 2014 by psy_commando Link to comment Share on other sites More sharing options...
evandixon Posted December 31, 2014 Share Posted December 31, 2014 It works! The latest version of kaoutil works nicely! I've attached the latest version of Sky Editor that uses it, and a screenshot showing that my indexed png works. Now it's time for me to find another thing to do. Maybe looking into monster.md that TruePikachu researched. Edit: to get sky editor working, you'll need to create the directory Resources/Plugins/RomEditor/Current, since I fixed a bug with it after uploading Sky Editor 12-31-14.zip Link to comment Share on other sites More sharing options...
psy_commando Posted December 31, 2014 Author Share Posted December 31, 2014 It works! The latest version of kaoutil works nicely! I've attached the latest version of Sky Editor that uses it, and a screenshot showing that my indexed png works. Now it's time for me to find another thing to do. Maybe looking into monster.md that TruePikachu researched. Edit: to get sky editor working, you'll need to create the directory Resources/Plugins/RomEditor/Current, since I fixed a bug with it after uploading Nice ! I was getting worried kaoutil was much more broken than I though ^^; And you seem to have gotten the quiz editing fully working now, with the appropriate text and all ! And yeah, monster.md is probably a good direction. If we can edit that file, it will really help getting some decent rom hacking potential going ! You might also want to look around this thread : http://www.gamefaqs.com/boards/955859-pokemon-mystery-dungeon-explorers-of-sky/51698562?page=23?message=240 There's a lot of good info in there. I can't remember what page was about monster.md though.. On my side of things, once I'm done cleaning up the code, and I'm really close to how clean I want it to be for now, I'll begin focusing on sprites. Idk if TruePikachu ever figured out how to get the image dimensions though, it would have been a great help ! And, there is another issue I still haven't figured out yet. What is at the end of SIR0 files, where the second pointer in the header points ? That weird section filed with mainly repeating values, and there doesn't seem to be any real consistency in the amount and value of these between SIR0 files.. Link to comment Share on other sites More sharing options...
evandixon Posted January 1, 2015 Share Posted January 1, 2015 Nice ! I was getting worried kaoutil was much more broken than I though ^^;And you seem to have gotten the quiz editing fully working now, with the appropriate text and all ! And yeah, monster.md is probably a good direction. If we can edit that file, it will really help getting some decent rom hacking potential going ! You might also want to look around this thread : http://www.gamefaqs.com/boards/955859-pokemon-mystery-dungeon-explorers-of-sky/51698562?page=23?message=240 There's a lot of good info in there. I can't remember what page was about monster.md though.. On my side of things, once I'm done cleaning up the code, and I'm really close to how clean I want it to be for now, I'll begin focusing on sprites. Idk if TruePikachu ever figured out how to get the image dimensions though, it would have been a great help ! And, there is another issue I still haven't figured out yet. What is at the end of SIR0 files, where the second pointer in the header points ? That weird section filed with mainly repeating values, and there doesn't seem to be any real consistency in the amount and value of these between SIR0 files.. I've seen conflicting information for monster.md, so I've lost interest in it. I may revisit it when I have more time. In the mean time, I might make an editor for item_p.bin (as described in this spreadsheet). Quickly looking at the SIR0 format, it seems to be 4 bytes of header, 4 bytes of a start index (maybe), and 4 bytes of the end offset. The next 4 bytes are usually if not always all 0. The data at the end I'd guess it garbage, but after removing it from various BALANCE files (ensuring all removed data was after the apparent end index), this happened when starting a new game (attached). Also I picked up money (the strange icon on the floor) and see it in my inventory, and got the message about how it was the world's currency. I could throw it too. Then the game crashed when using a regular attack. The data at the end appears to be garbage but is apparently significant. Maybe it helps when spacing files in the ROM's file system or something. Link to comment Share on other sites More sharing options...
psy_commando Posted January 1, 2015 Author Share Posted January 1, 2015 I've seen conflicting information for monster.md, so I've lost interest in it. I may revisit it when I have more time. In the mean time, I might make an editor for item_p.bin (as described in this spreadsheet). Quickly looking at the SIR0 format, it seems to be 4 bytes of header, 4 bytes of a start index (maybe), and 4 bytes of the end offset. The next 4 bytes are usually if not always all 0. The data at the end I'd guess it garbage, but after removing it from various BALANCE files (ensuring all removed data was after the apparent end index), this happened when starting a new game (attached). Also I picked up money (the strange icon on the floor) and see it in my inventory, and got the message about how it was the world's currency. I could throw it too. Then the game crashed when using a regular attack. The data at the end appears to be garbage but is apparently significant. Maybe it helps when spacing files in the ROM's file system or something. Yeah, I thought it was garbage for a long while too.. But, after a while I noticed that the 0xAA padding at the end was consistently preceded by a single 0 byte.. And today I tried a few things on the "frame.wte" files in the FONT folder. They're SIR0 wrapped files. The first thing I tried was replacing everything in that section with 0xAA padding bytes. The game wouldn't load at all. Then I replaced every values, except the 0 byte and the 0xAA padding with 0x04, and the game would load, but nearly all the dialog boxes and menu frames were missing.. Then I tried adding one by one the bytes that weren't 0x04, 0x00, or 0xAA, and it gave the same result as above.. I also noticed that files that are "related" together share the exact same value in there. Like, "frame.wte", "frame0.wte", "frame1.wte", "frame2.wte", "frame3.wte", and "frame4.wte" all have these bytes in there: 0x04, 0x04, 0x92, 0x0C, 0x14, 0x00, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA The length of each of those 6 files is 0x960 (2,400) bytes, for reference. I also checked in Explorers of Time, and they only have a single "frame.wte" but it shares the same length of 0x960 (2,400)bytes, and the exact same byte sequence: 0x04, 0x04, 0x92, 0x0C, 0x14, 0x00, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA However, the other files in the FONT folder do not share the same value. Some files are actually smaller than these 6, and have longer strings of values.. For example, look at "pagewait.wan". Its only 0x5D0 (1,488)bytes long, and it has the following byte string at the end: 0x04, 0x04, 0x82, 0x2C, 0x88, 0x18, 0x10, 0x04, 0x04, 0x08, 0x04, 0x08, 0x10, 0x04, 0x0C, 0x04, 0x00, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, 0xAA, By elimination, I guess we can be sure that: - Its not the offset of the file in the rom. - Its not a CRC/MD5 check. I'm pretty stumped right now as to what this is for.. The only guess I have right now would be that it might have to do with the way the SIR0 is loaded in memory. Like maybe some kind of grouping to tell which chunks of data has to be loaded together absolutely ? Given how, SIR0 are transformed into SIRO container after being loaded in the WRAM, and how they're loaded in small chunks, it must have some kind of link with that.. However, there doesn't seem to be any correlation between the size of that field and the length of the file, which doesn't make sense if that's really what its for.. I lost my debugging log where I wrote everything I did when I tried to manually trace how sprites were loaded when I formatted my comp after the issues I had recently, so this is going to be a pain in the ass to re-learn everything about how the game loads sprites.. Link to comment Share on other sites More sharing options...
psy_commando Posted January 2, 2015 Author Share Posted January 2, 2015 (edited) Alright! So I started no$gba and did some debugging, and I figured out what the hell those values at the end of the SIR0 files are! They're a list of offsets to every pointers in the file! This is used to modify the value of the pointers once the SIR0 files is loaded in memory, so that they become relative to the NDS memory and not to the file. Here's how the asm code deals with each values in the string of bytes in frame0.wte: 0x04 0x04 0x92 0x0C 0x14 0x00 0xAA 0xAA 0xAA 0xAA 0xAA 0xAA 0xAA 0xAA 0xAA 0xAA (note that the 0xAA bytes are ignored, as they are padding bytes) //------------ //0x04 //------------ r2 = 0 //r2 == 0 now r4 = readbyte(r1(0x20BFD10)) //r4 == 0x4 now, readbyte( address ) r1(0x20BFD10) = r1 + 1 //r1 == 0x20BFD11 now r2(0) = r2 << 7 //r2 == 0x0 now r5(0) = r4(0x4) & 0x7F //r5 == 0x4 now r2(0) = r2 | r5(0x4) //r2 == 0x4 now r5(0x4) = r4(0x4) & 0x80 //r5 == 0 now if( r5 != 0 ) // false goto 0x201F548 // doesn't jump if( r2(0x4) == 0 ) // false goto 0x201F584 // doesn't jump r3(0x20BF3C0) = r3 + r2(0x4) //r3 == 0x20BF3C4 r2(0x4)= readint(r3(0x20BF3C4)) //r2 == 0x910 now r2(0x910) = r2 + r0(0x20BF3C0) //r2 == 0x20BFCD0 now writeint(r2(0x20BFCD0),r3(0x20BF3C4))// writeint( value, address ) goto 0x201F544 //jumps back to loop begining //------------ //0x04 //------------ r2 = 0 //r2 == 0 now r4 = readbyte(r1(0x20BFD11)) //r4 == 0x4 now, readbyte( address ) r1(0x20BFD11) = r1 + 1 //r1 == 0x20BFD12 now r2(0) = r2 << 7 //r2 == 0x0 now r5(0) = r4(0x4) & 0x7F //r5 == 0x4 now r2(0) = r2 | r5(0x4) //r2 == 0x4 now r5(0x4) = r4(0x4) & 0x80 //r5 == 0 now if( r5 != 0 ) // false goto 0x201F548 // doesn't jump if( r2(0x4) == 0 ) // false goto 0x201F584 // doesn't jump r3(0x20BF3C4) = r3 + r2(0x4) //r3 == 0x20BF3C8 r2(0x4)= readint(r3(0x20BF3C8)) //r2 == 0x950 now r2(0x950) = r2 + r0(0x20BF3C0) //r2 == 0x20BFD10 now writeint(r2(0x20BFD10),r3(0x20BF3C8))// writeint( value, address ) goto 0x201F544 //jumps back to loop begining //------------ //0x92 //------------ r4(0x04) = readbyte(r1(0x20BFD12)) //r4 == 0x92 now, readbyte( address ) r1(0x20BFD12) = r1 + 1 //r1 == 0x20BFD13 now r5() = r4(0x92) & 0x7F //r5 == 0x12 now r2(0) = r2(0) | r5(0x12) //r2 == 0x12 now r5(0x12) = r4(0x92) & 0x80 //r5 == 0x80 now if( r5(0x80) != 0 ) // true goto 0x201F548 // jump to 0x201F548 r4(0x92) = readbyte(r1(0x20BFD13)) //r4 == 0x0C now r1(0x20BFD13) = r1 + 1 //r1 == 0x20BFD14 now r2(0x12) = r2 << 0x7 //r2 == 0x900 now r5(0x80) = r4(0xC) & 0x7F //r5 == 0xC now r2(0x900) = r2 | r5(0xC) //r2 == 0x90C now r5(0xC) = r4(0xC) & 0x80 //r5 == 0x0 now if( r5(0xC) != 0 ) // false goto 0x201F548 // doesn't jump if( r2(0x90C) == 0 ) // false goto 0x201F584 //doesn't jump r3(0x20BF3C8) = r3 + r2(0x90C) //r3 == 0x20BFCD4 now r2(0x90C) = readint( r3(0x20BFCD4) ) //r2 == 0x10 now r2(0x10) = r2 + r0(0x20BF3C0) //r2 == 0x20BF3D0 writeint( r2(0x20BF3D0), r3(0x20BFCD4) ) // writeint( value, address ) goto 0x201F544 // jumps back to loop beginning for next value! //------------ //0x14 //------------ r2 = 0 //r2 == 0 now r4 = readbyte(r1(0x20BFD14)) //r4 == 0x14 now, readbyte( address ) r1(0x20BFD14) = r1 + 1 //r1 == 0x20BFD15 now r2(0) = r2 << 7 //r2 == 0x0 now r5(0) = r4(0x14) & 0x7F //r5 == 0x14 now r2(0x14) = r2 | r5(0x14) //r2 == 0x14 now r5(0x14) = r4(0x14) & 0x80 //r5 == 0 now if( r5 != 0 ) // false goto 0x201F548 // doesn't jump if( r2(0x14) == 0 ) // false goto 0x201F584 // doesn't jump r3(0x20BFCD4) = r3 + r2(0x14) //r3 == 0x20BFCE8 r2(0x14)= readint(r3(0x20BFCE8))//r2 == 0x810 now r2(0x810) = r2 + r0(0x20BF3C0) //r2 == 0x20BFBD0 now writeint(r2(0x20BFBD0),r3(0x20BFCE8))// writeint( value, address ) goto 0x201F544 //jumps back to loop begining //------------ //0x0 //------------ r2 = 0 //r2 == 0 now r4 = readbyte(r1(0x20BFD15)) //r4 == 0 now, readbyte( address ) r1(0x20BFD15) = r1 + 1 //r1 == 0x20BFD16 now r2(0) = r2 << 7 //r2 == 0 now r5(0) = r4(0) & 0x7F //r5 == 0 now r2(0) = r2 | r5(0) //r2 == 0 now r5(0) = r4(0) & 0x80 //r5 == 0 now if( r5 != 0 ) // false goto 0x201F548 // doesn't jump if( r2(0) == 0 ) // true goto 0x201F584 // jumps //Function returns ! We're done. Here's something more direct: This is the list of bytes at the end of the frame0.wte file: 04 04 92 0C 14 00 Thos are processed by the code as such : 4 4 + 4 4 + 4 + ( (0x92 & 0x7F) << 7) | 0xC 4 + 4 + ( (0x92 & 0x7F) << 7) | 0xC + 0x14 (Notice how the 0x92 and 0xC were combined into a single entry. And notice how we add the value of each previous offsets.) Which once computed gives this: 0x4 0x8 0x914 0x928 Those are the offsets of the 4 pointers in the frame0.wte file! Knowing these, the program go to every one of those offsets, reads the value of each pointers, and then respectively overwrite each pointer value with those: value at 0x4 + FileAddressInNDSMemory value at 0x8 + FileAddressInNDSMemory value at 0x914 + FileAddressInNDSMemory value at 0x928 + FileAddressInNDSMemory lets say that our FileAddressInNDSMemory is 0x020BF3C0, well this means the pointers would turn into: offset in file: value in file: new value in memory: 0x4 0x910 0x20BFCD0 0x8 0x950 0x20BFD10 0x914 0x10 0x2B0F3D0 0x928 0x810 0x20BFBD0 Basically the rules for reading those are simple. Zero means its the end of the list. Each offset calculated is added to the previous ones. The highest bit of each byte(1000 0000) is reserved for determining whether we must bit shift our current byte by 7, and then bitwise OR the next byte's value with our current byte, if its 1. Or simply use the value of the byte as-is, if its 0. In both case the value of the highest bit is removed, by using a bitwise AND with 0x7F(0111 1111) and the value of the byte. But, anyhow. I should have some code working for handling that in no time ! EDIT: I made a c++ version of the above. Its untested yet though, and it might not be very practical in real-world scenarios: EDIT2: Fixed code, and added encoding code as well: void EncodeSIR0PtrOffsetList( const std::vector<uint32_t> &listoffsetptrs, std::vector<uint8_t> & out_encoded ) { uint32_t offsetSoFar = 0; //used to add up the sum of all the offsets up to the current one for( const auto & anoffset : listoffsetptrs ) { //Offset all the pointer by 0x10 to compensate for the SIR0 header length uint32_t offsetToEncode = anoffset - offsetSoFar; offsetSoFar = anoffset; //set the value to the latest offset, so we can properly subtract it from the next offset. //Encode every bytes of the 4 bytes integer we have to for( int32_t i = 4; i > 0; --i ) { uint8_t currentbyte = ( offsetToEncode >> (7 * (i - 1)) ) & 0x7Fu; if( currentbyte != 0 ) { //If its the last byte to chain, leave the highest bit to 0 ! if( i == 1 ) out_encoded.push_back( currentbyte ); else out_encoded.push_back( currentbyte | 0x80u ); //Set the highest bit to 1, to signifie that the next byte must be chained } } } //Append the closing 0 out_encoded.push_back(0); } std::vector<uint8_t> EncodeSIR0PtrOffsetList( const std::vector<uint32_t> &listoffsetptrs ) { vector<uint8_t> encodedptroffsets( 2u + listoffsetptrs.size() ); //Worst case scenario allocation encodedptroffsets.resize(0); //preserve alloc, allow push backs EncodeSIR0PtrOffsetList(listoffsetptrs,encodedptroffsets); return std::move(encodedptroffsets); } std::vector<uint32_t> DecodeSIR0PtrOffsetList( const std::vector<uint8_t> &ptroffsetslst ) { vector<uint32_t> decodedptroffsets( ptroffsetslst.size() ); //worst case scenario decodedptroffsets.resize(0); auto itcurbyte = ptroffsetslst.begin(); auto itlastbyte = ptroffsetslst.end(); uint32_t offsetsum = 0; //This is used to sum up all offsets and obtain the offset relative to the file, and not the last offset uint32_t buffer = 0; //temp buffer to assemble longer offsets uint8_t curbyte = *itcurbyte; while( itcurbyte != itlastbyte && (curbyte = *itcurbyte) != 0 ) { buffer |= curbyte & 0x7Fu; if( (0x80u & curbyte) != 0 ) { buffer <<= 7u; } else { offsetsum += buffer; decodedptroffsets.push_back(offsetsum); buffer = 0; } ++itcurbyte; } return std::move(decodedptroffsets); } EDIT3: I found a flaw in the encoder code. I'll fix it sometime this week. Basically 32 bits integers that have a single byte that's completely null in-between 2 non-null bytes will not get encoded properly. Take 0x130001 for example. It should give 0xCC 0x80 0x01, but it won't get encoded that way, and would probably come out as 0xCC 0x01... EDIT4: Fixed encoding code ! void EncodeSIR0PtrOffsetList( const std::vector<uint32_t> &listoffsetptrs, std::vector<uint8_t> & out_encoded ) { uint32_t offsetSoFar = 0; //used to add up the sum of all the offsets up to the current one for( const auto & anoffset : listoffsetptrs ) { uint32_t offsetToEncode = anoffset - offsetSoFar; bool hasHigherNonZero = false; //This tells the loop whether it needs to encode null bytes, if at least one higher byte was non-zero offsetSoFar = anoffset; //set the value to the latest offset, so we can properly subtract it from the next offset. //Encode every bytes of the 4 bytes integer we have to for( int32_t i = 4; i > 0; --i ) { uint8_t currentbyte = ( offsetToEncode >> (7 * (i - 1)) ) & 0x7Fu; if( i == 1 ) //the lowest byte to encode is special { //If its the last byte to append, leave the highest bit to 0 ! if( currentbyte != 0 ) out_encoded.push_back( currentbyte ); //If the last byte to append is null, we don't need to append anything // as the automatic bitshift of the last byte will take care of that if a higher byte was non-zero. // In any other cases, null pointers are not to be messed with. } else if( currentbyte != 0 || hasHigherNonZero ) //if any bytes but the lowest one! If not null OR if we have encoded a higher non-null byte before! { //Set the highest bit to 1, to signify that the next byte must be appended out_encoded.push_back( currentbyte | 0x80u ); hasHigherNonZero = true; } } } //Append the closing 0 out_encoded.push_back(0); } Edited January 7, 2015 by psy_commando Link to comment Share on other sites More sharing options...
psy_commando Posted January 10, 2015 Author Share Posted January 10, 2015 (edited) Alright, I figured out a lot with how the "WAN" sprite format works! Basically what we knew: You have the wan header, which has 3, maybe 4, values in it. The first is the pointer to the animation info, the second is the pointer to the image data info, the third is a 16bits boolean that indicates whether the sprite's animation are 8-way animations/character animations, or anything else like a prop or menu item. The animation data has a table of pointers to meta-frames, a pointer to a section filled with a long list of signed 16bits integers, a pointer to a big table that points to other smaller tables containing animation sequences details, which each entries points to another meta-frame over the meta-frame! But here are a couple of neat finds I got: Meta-Frames: The meta-frames are all stored right after the SIR0 header, in what I was previously referring to as Datablock S. Each entry for a single meta-frame is 10 bytes long, and is built like this: Meta-Frame 10 bytes [ Index 2 little Index in the frame table, containing the pointer to the actual image data. unkint 2 little Unknown purpose. YOffset+Flags1 2 little (10 lowest bits)Frame offset on Y axis. (6 highest bits)Flags*(more details below under DataBlock S section) XOffset+Flags2 2 little (9 lowest bits)Frame offset on X axis. (7 highest bits)Flags*(more details below under DataBlock S section) val5 2 little Unknown. (0000 0000 0000 1111) sets the tile order/ID. (0000 0000 1111 0000) sets the palette ID to use for the sprite at runtime. (1111 1111 0000 0000) changes the tile ID as well, but its still unclear how it works.. ] The flags I could figure out, are as follow: **YOffset flags** : bitflags used to tell the game how to read and display the frame. 6 bits are reserved for that purpose. bits 1-2 (1100 0000 0000 0000) indicate the frame's resolution. 01 = Modify resolution obtained through XOffset bits 1-2. More below. 10 = Modify resolution obtained through XOffset bits 1-2. More below. bit 3 (0010 0000 0000 0000) Unknown bit 4 (0001 0000 0000 0000) 0 = Normal 1 = Activate Mosaic mode. bits 5-6 Unknown **XOffset flags** : bitflags used to tell the game how to read and display the frame. 7 bits are reserved for that purpose. bits 1-2 (1100 0000 0000 0000) indicate the frame's resolution. 00 = 8x8 01 = 16x16 10 = 32x32 11 = 64x64 bit 3 (0010 0000 0000 0000) indicates whether the image should be flipped vertically. 0 = Normal 1 = Flipped vertically bit 4 (0001 0000 0000 0000) indicates whether the image should be flipped horizontally. 0 = Normal 1 = Flipped horizontally bits 5-7 Unknown Values of bits 1-2 for the XOffset and YOffset can be combined to get non-square resolution: YOffset: XOffset: Resolution: 00 00 8 x 8 00 01 16 x 16 00 10 32 x 32 00 11 64 x 64 01 00 8 x 16 10 00 16 x 8 01 01 32 x 8 10 01 8 x 32 01 10 32 x 16 10 10 16 x 32 01 11 64 x 32 10 11 32 x 64 Changing bits 5-6 ( 0000 1100 0000 0000 ) of the XOffset to 0 or 1 seems to make the palette used to display the sprite change to either the correct palette(palette 6 or 5 at runtime it seems), OR whatever palette 0 is at runtime.. Still not sure what it does exactly. But it involves the palette used to display the sprite at runtime, because it makes it change in the OAM in the NDS's memory. Animation Meta-Frames: As we already knew, thanks to TruePikachu's findings, those are basically, another reference to a frame, with extra info. It represents a single frame in an animation sequence. However, I was able to confirm and/or figure out what every fields for each animation frames is for, by modifying those in the files directly and looking at the results: Entry 12 bytes A single frame of an animation sequence. [ FrameDur 2 little The duration the frame is displayed. Higher is slower. AnimFrame# 2 little The index (in the meta-frame pointer table at Offset E) of the meta-frame to display. SprOffX 2 little (Signed) Offset of the frame relative to the center of the sprite, on the X axis. SprOffY 2 little (Signed) Offset of the frame relative to the center of the sprite, on the Y axis. ShadowOffX 2 little (Signed) Offset of the shadow relative to the center of the sprite, on the X axis. ShadowOffY 2 little (Signed) Offset of the shadow relative to the center of the sprite, on the Y axis. ] Its hard to tell in what measurement unit the frame duration is, maybe ticks, or milliseconds. I can't think of an easy way to properly calculate that.. But anyhow. Not only does the character sprite has another set of offset values for its sprite, it also has one to set where the shadow blob appears. So if you want, you could make the shadow appear in some arbitrary position that's not under the character sprite for that specific frame.. There's a little more. But I'm still looking into what some values do. Most of the others have no immediately obvious effects.. I'll also, work on the WAN page on the wiki soon. Once I get some good progress on the sprite tool. Also, @evandixon : I finally tried out skyeditor, and you really did a great job with the quiz editor and pokemon face picture tool ! Though, personally I think that adding a button to insert new images for pokemon missing one might be nice. I'm not sure if you were already going that way, but just trying to give some feedback. EDIT: Found out a little more, namely how to set non-squared resolution for the sprites ! Here are my updated notes on the subject: https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/SIR0_Sprite.txt I'll eventually put that on the wiki. EDIT2: Great ! Now I know enough about sprites to rebuild one from scratch ! I updated my notes with the new details! https://dl.dropboxusercontent.com/u/13343993/my_pmd_research_files/FileFormats/SIR0_Sprite.txt Edited January 12, 2015 by psy_commando Link to comment Share on other sites More sharing options...
psy_commando Posted January 16, 2015 Author Share Posted January 16, 2015 Alright, I got most of the new sprite reading code working now ! I also decided that my sprite extractor will export images along with xml files containing the data contained in the sprite file. It will also use the same layout for importing sprites back into a WAN file. Here's the current layout I'm going for, for each sprite files that was extracted: SpriteName/ |- animations.xml |- frames.xml |- offsets.xml |- spriteinfo.xml |- palette.pal |- imgs/ ||- <individual images as png, bmp, or raw> I could have placed all the xml in a single file, but it would have been kinda chaotic, and overcrowded.. The palette will be optional on import, as most image formats already contain it. I'll get into more details once I actually have a release candidate ! I picked xml, because its supported by nearly every script languages ever, and most compiled languages have access to xml libraries. I've used Poco's XML library, since I already use another part of their lib in my code. Its not great, convenient, or pretty, but it does the job.. Hopefully, thanks to xml anyone can write tools to handle the data in there, especially considering the amount of data can be quite huge. It would have been impossible to have summed up sprites as simply images in a folder, or to use simple text files to represent references between the various elements of the sprite format! But xml does a pretty good job at it, and its still human readable and editable for those dedicated souls out there ^^; Also, thanks to trying out the tool on various sprite files, I found out that the "Meta-frames pointer table" (used to be Offset E), is in fact pointing to a list of meta-frames, not always a single meta-frame. You need to assemble the images pointed at by the meta-frames in that list to build the final on-screen image, using the offsets values in the meta-frame to position image chunks properly. Like if you take rayquaza, each of his frames are made up of 4 images with varied resolutions that are assembled together. That's quite a big breakthrough ! I'll need to rewrite part of the code now though.. Link to comment Share on other sites More sharing options...
psy_commando Posted January 20, 2015 Author Share Posted January 20, 2015 I got an early build working. It supports only exporting wan sprites, and its still pretty buggy. It will work on pokemon sprites and any other sprites that ends in ".wan". However, the pokemon sprites are compressed when they're unpacked from the "m_attack.bin" and "monster.bin" so you have to decompress them first with another tool. This will be integrated in the program eventually though. I was a little bored, and got creative with the progress display It works just like all my other utilities, you can just drag and drop a file onto it, or use the console. (Though, make sure the path where the file you're drag and dropping from is doesn't contain white spaces, or it won't work, thanks to a limitation in the way the OS passes the arguments.. You can just put the exe in the same folder as the sprite file you want to extract though, that will fix the issue. ) https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/tmp/ppmd_gfxcrunch_pre-alpha.zip I'm planning on writing all my graphics handling code into GfxCrunch.exe in the future, so there's no need for any other tool for handling sprites and potraits and etc! Link to comment Share on other sites More sharing options...
evandixon Posted January 21, 2015 Share Posted January 21, 2015 Good work! Sorry I haven't been more active lately. I've been working on other projects, and real life happened (it's been a while since that happened), but still checking the forums for progress on this stuff. I hope to work more on this stuff in the near future, including adding an "Add" button to the portraits tab in Sky Editor, and polishing it up more in general. Link to comment Share on other sites More sharing options...
psy_commando Posted January 21, 2015 Author Share Posted January 21, 2015 Good work! Sorry I haven't been more active lately. I've been working on other projects, and real life happened (it's been a while since that happened), but still checking the forums for progress on this stuff. I hope to work more on this stuff in the near future, including adding an "Add" button to the portraits tab in Sky Editor, and polishing it up more in general. Thanks ! And that's all good. I just have a lot of free time these days, so maybe its just me being a little too active on this ! Link to comment Share on other sites More sharing options...
psy_commando Posted January 24, 2015 Author Share Posted January 24, 2015 So, I managed to export bulbasaur's sprite to XML + images, then re-import it into the game, and it works !! UI sprites will need some more work however. I'm still stumped as to how they work exactly.. And there aren't that many to run tests on.. Since this is much more complex than any other utilities I've made before, I'll need people to help find bugs I think.. So many things can go wrong ! I'll work on making the utility releasable, and try to clean up a little. I'm going to do a much more thorough cleaning once we figure out how the whole format works, to avoid having to redo that everytime! Also, I've found out that, animation groups can contain references to the same sequence several times. So, I've had to rewrite my code a little to account for that. I also modified the notes a little. Link to comment Share on other sites More sharing options...
psy_commando Posted January 29, 2015 Author Share Posted January 29, 2015 (edited) Anyways, I made some progress with the tool. You now can unpack and repack all the sprites contained in the 3 files containing all the pokemon sprites "/MONSTER/monster.bin", "/MONSTER/m_attack.bin", "/MONSTER/m_ground.bin". It recognize them by the name of the folder you're repacking. However you can force it with a command line option too. Here's a no guaranty early release : https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/tmp/ppmd_gfxcrunch_prealpha-2.zip Its untested right now. All I know is that every sprite files I've thrown at it works, including the 3 packed sprite files. And sprite unpacking and re-packing is kinda slow, but I can't do much about it. It seems its the XML library I use that's relatively slow. I found, and fixed, a huge bug with my tiled image handler, where the height and width of an image were inverted. I can't believe I never found out about it before... And its just a single line, that caused all that messing around.. ugh.. Also, I found a bug with it I have absolutely no clues how to fix : [ATTACH=CONFIG]12050[/ATTACH] For some reason, chimbling is the only pokemon I've seen this far that has graphic corruption. And only its first frame of animation pointing towards the bottom does this! I'm kinda stumped.. EDIT: lol I actually spelt Chingling, chimbling Edited January 30, 2015 by psy_commando Link to comment Share on other sites More sharing options...
evandixon Posted January 29, 2015 Share Posted January 29, 2015 If your xml library is slow, you can try to find a JSON library. It's a bit less wordy so it may be quicker. Link to comment Share on other sites More sharing options...
psy_commando Posted January 30, 2015 Author Share Posted January 30, 2015 (edited) Hmm.. Thanks for the suggestion. That would probably help! But for now I'll stick with XML, because I already wrote the parsers and etc, and I don't want to burn myself out on that, given that tiny glitch with the sprites I pointed earlier is bumming me out a lot right now ^^; I'll see about possibly using JSON later. But, I think it also has to do with how POCO isn't really a specialized XML lib. And how I'm not even sure whether I'm using SAX or DOM exactly right now.. Also, someone brought to my attention that, all my utility don't work on Windows XP. And I only found out today that, microsoft implemented in VS2012 and up something that prevents code compiled the regular way to run in Windows XP. So you have to use a different build target. So I'll be updating all the utilities to include 2 executable, one for Windows XP and one for Windows 7 and up. I might just change compiler soon, because I'm a little annoyed with the handling of things with the MSVC.. And I've also been trying out making little frontends for some of my utilities by using the Ultimate++ framework: http://www.ultimatepp.org/ ( the ide they kinda force on you is pretty meh though.. Its a farcry from Qt's ) EDIT: Idk if I'll stick with Ultimate ++, its getting a little overly complicated to deal with... EDIT2: Updates are live. Edited January 31, 2015 by psy_commando Link to comment Share on other sites More sharing options...
psy_commando Posted February 3, 2015 Author Share Posted February 3, 2015 (edited) For some reasons I repacked the rom using Dslazy, which is also just a ndstools frontend, and the glitch didn't show up this time... Not sure what's going on here.. EDIT: It might be the trimming on the rom that does that ? Anyways, I also found out that the drag-and-drop approach all my utilities favor is flawed if you rely on resources files to be in your working directory.. Oh joy... Then I looked at Qt for user interfaces. I've been using it for a while on several other projects, but then I remembered you need to distribute about 40,000 dlls. I hope there's some kind of workaround.. Edited February 4, 2015 by psy_commando Link to comment Share on other sites More sharing options...
psy_commando Posted February 6, 2015 Author Share Posted February 6, 2015 Alright, so thanks to the help of other members on the forum, I was able to finally pick a decent development platform for making little GUI frontends for my utilities. Rejoice my console illiterates friends! There's the first release, for ppmd_packfileutil : https://github.com/PsyCommando/ppmd_packfileutil_gui/releases/download/v1.0/ppmd_packfileutil_gui_1_0.zip I'm planning on making one for ppmd_kaoutil, and for ppmd_gfxcrunch. Nothing very fancy either, but hopefully, people will stop having issues with the working directory changing when drag-and-dropping! Besides, evandixon got something really nice this far working with ppmd_kaoutil, so its probably best to use SkyEditor to deal with portraits changing anyways ! Also, I haven't seen the graphical glitch again since I'm using dslazy instead of dsbuff to rebuild the rom. I guess I'll try to get some more people to try it then ! Here's the alpha for GfxCrunch : https://github.com/PsyCommando/ppmdu/releases/download/ppmd_GfxCrunch_0.1a/ppmd_gfxcrunch_0_1_alpha.zip I'll also update the front page. This is a console only application. Drag-and-drop won't work properly with this one, because of technical limitations. So if you can't use console, stay away from this one until the GUI is released! Please tell me about any errors you get. Especially any graphical glitches in the game! Link to comment Share on other sites More sharing options...
psy_commando Posted February 6, 2015 Author Share Posted February 6, 2015 After seeing people tricking that Japanese smd2mid utility by renaming files as bgm0001.smd and etc.. I got the idea to try swapping music in PMD2 with other games that also uses the Procyon Sound Driver. I tried with "Fushigi no Dungeon - Fuurai no Shiren DS 2 - Sabaku no Majou", which is basically from chunsoft's main series Mystery Dungeon games, Shiren the wanderer and etc.. I just swaped the main menu's theme in Explorers of Sky, with a track from Shiren2, both the smd and the corresponding swd, and it worked ! https://www.dropbox.com/s/ay31gkxtrzfny4m/mod_music_shiren2_07_7431.avi?dl=0 It seems that in Shiren2 the samples are mainly stored in the accompanying swd file, and apparently there isn't a main sound bank like bgm.swd in PMD2. Hopefully, this will help finding some more clues on how the format works! Link to comment Share on other sites More sharing options...
psy_commando Posted February 10, 2015 Author Share Posted February 10, 2015 Alright, I updated the front page with the latest GUI Frontends + utility package for both packfileutil and kaoutil. I ran a lot of testing, and I hope there won't be bugs. They both run on the .NET 3.5 framework. I also added some screenshot of both frontends as well. I'm also still looking for input on gfxutil in order to continue development! I really need help to figure out what's going on with it. The download link for it is on the front page for those interested. Its a pure console utility for now though. Link to comment Share on other sites More sharing options...
psy_commando Posted February 16, 2015 Author Share Posted February 16, 2015 YES!! I freaking fixed that sprite corruption bug !!! Screw you chingling, you can't get corrupted anymore !! I also swapped out POCO XML for pugixml, which resulted in a good speedup ! But it still wasn't fast enough, so I fixed my thread pool code, and now it runs on 2 threads on machines with more than 2 cores. And you can force it to use any amount of threads you want, with the "-th" commandline option, regardless of how many core you have! The corruption came from the way I was encoding the SIR0 ptr list, when a pointer offset ended up on a value with an entire 0 byte at the end, the zero byte was discarded, thanks to some wrong assumptions I had. And after doing a full binary compare, I managed to find out about that. I added another xml file to the exported sprite which contains what I believe is the z index of each individual images in the exported "imgs" folder. Only images that have a non-zero z index are listed in there to save needless work and space. I also finally managed an absolutely bit-perfect re-pack of the m_ground.bin file, from xml ! Which is great ! Here's the latest version: https://github.com/PsyCommando/ppmdu/releases/download/ppmd_GfxCrunch_0.12a/ppmd_gfxcrunch_0_12_alpha.zip Link to comment Share on other sites More sharing options...
psy_commando Posted February 22, 2015 Author Share Posted February 22, 2015 Alright, so now that character sprites appears to be pretty much 100% supported, I'm trying to work out the subtleties of the WAN fomat. Namely, what to do with meta-frames that have -1 as their "image index". The program just crashes when encountering those for now. Because I don't know of an intelligent way of handling them using the knowledge I have of the format. Meta-frames with an image index of -1 seems to tell the game to simply re-use whatever is in memory at the time of displaying said meta-frame. But, that poses a big issue, because many images in several sprites are essentially "orphaned". Which means that, we don't know in what resolution they are, or when/how/if they should be displayed, because no meta-frame is obviously referring to them.. Throw in the various boolean and values stored in the wan header that we know nothing about and you get a pretty big amount of possibility to deal with.. I'm still a little unsure how to determine how it all works.. If anyone has any input on this, it would be appreciated. If I can get all the sprites in the GROUND folder and the rest of the game files supported, I'll be able to move on to the next graphic format. I'd probably aim for the WTE files first, as I got most of those figured out. Then probably the dungeons/map tiles format. I also found a bug in gfxutil 0.12, when the pointer to the Particle Offsets table is null, its still being added to the SIR0 ptr list, and it shouldn't. It shouldn't cause issues with most character sprites however. Mainly other sprites. But it will be fixed in the next release. I might also change a couple of the XML tags I'm using to make things clearer. Because, the concept of meta-frames and meta-frames group is kinda vague and confusing. I'm not sure what else to name those though Additionally, if anyone would like to work on a GUI project for this, let me know. Because I think we could work out something that would make sprite editing, and animation editing fairly trivial ! Link to comment Share on other sites More sharing options...
sylveonlove Posted February 22, 2015 Share Posted February 22, 2015 can i add my own additions onto models? Link to comment Share on other sites More sharing options...
psy_commando Posted February 22, 2015 Author Share Posted February 22, 2015 can i add my own additions onto models? I'm not sure what you mean by that. For one, PMD2 doesn't uses 3d models. (at least I'm guessing that's what you meant) Link to comment Share on other sites More sharing options...
Recommended Posts
Create an account or sign in to comment
You need to be a member in order to leave a comment
Create an account
Sign up for a new account in our community. It's easy!
Register a new accountSign in
Already have an account? Sign in here.
Sign In Now