xd Pokemon XG: NeXt Gen

[StarsMmd]

Yes, it took me a lot of time recompressing and inserting it everytime... what are you using to automate it ?

I wrote some code in swift. Swift is still mac only though so it's not been very portable so far. It's nothing too complicated though. I found some c code for lzss compression which I just use directly for compression. The .fsys format is pretty straight forward so putting the compressed files back in is simple. You just look up the offset to place it at. Then I use the .toc file from the ISO to look up the offset of the .fsys file in the game and put it back in.

[Made_s]

Yeah, the compression code is what I don't have, I can only write a bit of Java.

[StarsMmd]

Yeah, the compression code is what I don't have, I can only write a bit of Java.

I think this is the code I used lzss.c. I was able to use it directly in swift but don't know if you know a way to use c code in java. The lzss algorithm is really well documented though and the fsys files use the typical, standard, usual compression. Maybe you can find a library somewhere online or attempt to translate it since c and java have really similar syntax.

[Made_s]

So far (since yesterday) I found out that most of the textures contained in model files are compressed CMPR files with only a few uncompressed C8 files ( somewhat documented here Image_Formats ).

There is a program that can convert .png images to CMPR here which you can use with the commands listed here .

Since the files don't contain any header data for textures (that I know of) I'm trying to use a different solution and will try to write a program which can search for the texture in the file by using a CMPR version of the original texture you can get through Dolphin which you'll have to provide. It replaces that section with the custom CMPR file you provide. The searching is a bit more complicated than just searching for the exact data though, because the program to convert to CMPR doesn't give the exact same data, because there is not *one* algorithm for compressing to CMPR.

Even though the colors are changed, most pixel indexes keep the same patterns, even when values are replaced with other ones, so if you know about pattern searches with ubstituted patterns that could help me, if not I'll continue to try to come up with one myself , I have somewhat of a lead of how I could write that.

Example comparison between CMPR images ( one raw data from model file , the other converted image from Dolphin):

The last 4 bytes of each 8 byte section are the pixel indexes with 2 bits per pixel selecting one of 4 colors generated by the first 4 bytes, each 32 bytes are one block and so on (see documentation)(excluding the first 64 bytes of the file which is a header exclusive to the converted file which I added to the Raw file for conversion to .png to compare and see whether these are even matching files):

Link

There may also be a third image format in the files like RGB565,RGB5A3 or RGBA32 (or something else since I don't know whether it even fits any of those ) since there are large areas with uncompressed texture data that are very large but result in a small image ( example is tha back texture of Salamence I showed earler , which I thought was a shader, but was just a really small texture), which doesn't fit C 8 and CMPR.

I wouldn't know how to edit the C8 files though, since those are color palette formats and I didn't see any palettes in the file before the data chunks.

The search script would first search for CMPR sections with the same length as the Dolphin texture and then compute a value of how similar the patterns of the sections are and then take the one with the highest value.

There may be another way of doing this, finding all the textures and numbering them , extracting them, then reinserting them by running the numbering script again and inserting it at a specific number. (This is probably the easiest way )

None of the textures in the files have Magic Bytes so even the second option would need some kind of patterns search.

(Sorry for how all-over-the-place this is) ( Also, does the Music Tool work for you ? )

Edited July 18, 2016 by Made_s

[StarsMmd]

So far (since yesterday) I found out that most of the textures contained in model files are compressed CMPR files with only a few uncompressed C8 files ( somewhat documented here Image_Formats ).

There is a program that can convert .png images to CMPR here which you can use with the commands listed here .

Since the files don't contain any header data for textures (that I know of) I'm trying to use a different solution and will try to write a program which can search for the texture in the file by using a CMPR version of the original texture you can get through Dolphin which you'll have to provide. It replaces that section with the custom CMPR file you provide. The searching is a bit more complicated than just searching for the exact data though, because the program to convert to CMPR doesn't give the exact same data, because there is not *one* algorithm for compressing to CMPR.

Even though the colors are changed, most pixel indexes keep the same patterns, even when values are replaced with other ones, so if you know about pattern searches with ubstituted patterns that could help me, if not I'll continue to try to come up with one myself , I have somewhat of a lead of how I could write that.

Example comparison between CMPR images ( one raw data from model file , the other converted image from Dolphin):

The last 4 bytes of each 8 byte section are the pixel indexes with 2 bits per pixel selecting one of 4 colors generated by the first 4 bytes, each 32 bytes are one block and so on (see documentation)(excluding the first 64 bytes of the file which is a header exclusive to the converted file which I added to the Raw file for conversion to .png to compare and see whether these are even matching files):

Link

There may also be a third image format in the files like RGB565,RGB5A3 or RGBA32 (or something else since I don't know whether it even fits any of those ) since there are large areas with uncompressed texture data that are very large but result in a small image ( example is tha back texture of Salamence I showed earler , which I thought was a shader, but was just a really small texture), which doesn't fit C 8 and CMPR.

I wouldn't know how to edit the C8 files though, since those are color palette formats and I didn't see any palettes in the file before the data chunks.

The search script would first search for CMPR sections with the same length as the Dolphin texture and then compute a value of how similar the patterns of the sections are and then take the one with the highest value.

There may be another way of doing this, finding all the textures and numbering them , extracting them, then reinserting them by running the numbering script again and inserting it at a specific number. (This is probably the easiest way )

None of the textures in the files have Magic Bytes so even the second option would need some kind of patterns search.

(Sorry for how all-over-the-place this is) ( Also, does the Music Tool work for you ? )

Thanks for all the information. I'll look into too and see if I can spot anything. Ideally we'd be able to find some kind of header or table somewhere. Once I'm able to recognise the cmpr data it should be possible. I'll have to take some time to get up to speed but will get back to you.

I ran the program just to have a look and it all looks fine but I haven't actually used it yet just because I've been busy. Will potentially give it a go tonight though.

[Made_s]

The CMPR data is really easy to recognize, they mostly look like the examples I gave you.

[StarsMmd]

The CMPR data is really easy to recognize, they mostly look like the examples I gave you.

Sweet, I'm going to fire up my laptops now and take a look.

[Made_s]

I just found some info on the palettes and header if you do manage to find them Link

[StarsMmd]

I just found some info on the palettes and header if you do manage to find them Link

Alright, here's what I've found so far. So there a few small differences between the model files depending on whether they are in battle models or overworld models, etc. The one's you're referring to, where the coordinate data starts at 0xE80 are the pkx ones I believe. These are the in battle models. The model data starts at 0xE60 in these files (with the header). In files with the overworld models (usually found in the .fsys files for maps) the model data's header starts at 0x00. The header is 0x20 bytes long.

The first 4 bytes in the header are the file size of the model including the header. The next 4 bytes are a pointer to a list of pointers. This pointer's offset is relative to the end of the header (i.e. from 0xE80 in pkx or 0x20 in overworld models). Each of the pointers in that list is also relative to the end of the header. The next 4 bytes is the number of pointers there are.

From what I've seen, the first pointer in the list points to a list of texture headers. The first 4 bytes being the offset of the header followed by what I presume to be the width and height? Each header appears to be 0x74 bytes long. I don't think there are any other reliable pointers to each of the individual headers so we may have to just add 0x74 bytes to the offset each time. I also haven't found a value to determine the number of texture files yet. Also the headers aren't necessarily in the same order that the actual data is in.

[Made_s]

Thanks, with this I can probably write something that extracts the textures into a format that can be converted by the szs tool.

I already wrote something very similar to this for the sound effect extraction/insertion.

If there are only CMPR and C8 textures, the type can easily be determined by dividing the length of the file ( which is just the difference between offsets of following pointers ( the file I'm using right now definitely has all pointers in order, if that isn't the case in every file I'll have to write something that sorts them)) by the pixel count, since CMPR has 4 bits per pixel and C8 1 byte per pixel.

Do you know whether the texture data is all in one block? If not so then getting the length might be a bit more complicated.

About finding a stop for the texture table, the end contains two of the 0x2C long sections that are between the texture poitners, so we could just let the program check whether the an entry starts with the start of one of those sections.

Edited July 19, 2016 by Made_s

[StarsMmd]

That's good to hear and I'm really happy I was able to help ??

I'm pretty sure the headers should store the dimensions of the texture and maybe even whether or not it's c8 or cmpr

[Made_s]

These sections seperating the texture pointers contain their own offset which is pretty dumb unless a pointer points to that value for some reason instead of just pointing directly to the offset.

[StarsMmd]

These sections seperating the texture pointers contain their own offset which is pretty dumb unless a pointer points to that value for some reason instead of just pointing directly to the offset.

Yeah I noticed that as well. Have no idea why it's there. I hope we can figure out what the rest of the data is. It will probably make the extraction and insertion process a lot easier.

[Made_s]

Images (I forgot to color the type in the selected area pink)

The lime green part in picture 1 points to the black part in Picture 2 and the blue value in Picture 2 points to the yellow

part ( which is just the start of the data after that 0x20 long line).

I think this is the Palette data for C8 (and possibly other Palette formats) since only the texture with the format ( the pink sections in Picture 1) of 0x09 ( which is for C8) have this pointer and the CMPR (CMPR doesn't use external palettes ) ones (the one with 0x0E for type) have 0x00000000 at that part.

Inserting Palette based formats might be a problem since the palette size probably varies with the picture you want to insert, which would require a whole lot of pointer editing at multiple places.

I wasn't able to make a C8 file out of C8 data yet, mostly because there seem to be issues with the header I haven't figured out yet (the header in the file I'm making not the ones in the data). If I get this to work I should be able to extract those and CMPR files.

Also do you know of any Hex Editor that can highlicht areas of code ? Like the style options in Notepad++. It would making these notes a lot easier and prettier.

Edited July 19, 2016 by Made_s

[StarsMmd]

Images (I forgot to color the type in the selected area pink)

The lime green part in picture 1 points to the black part in Picture 2 and the blue value in Picture 2 points to the yellow

part ( which is just the start of the data after that 0x20 long line).

I think this is the Palette data for C8 (and possibly other Palette formats) since only the texture with the format ( the pink sections in Picture 1) of 0x09 ( which is for C8) have this pointer and the CMPR (CMPR doesn't use external palettes ) ones (the one with 0x0E for type) have 0x00000000 at that part.

Inserting Palette based formats might be a problem since the palette size probably varies with the picture you want to insert, which would require a whole lot of pointer editing at multiple places.

I wasn't able to make a C8 file out of C8 data yet, mostly because there seem to be issues with the header I haven't figured out yet (the header in the file I'm making not the ones in the data). If I get this to work I should be able to extract those and CMPR files.

Also do you know of any Hex Editor that can highlicht areas of code ? Like the style options in Notepad++. It would making these notes a lot easier and prettier.

It looks like you've made a lot of progress. I think it would be better to just force the image to use the same number of colours as (or fewer than) the original image. When I was replacing other texture files from the game with pngs I'd edited, if I accidentally used more than the 200byte palette from the original then I'd display an error. Also, I would import the new image back into the original file, keeping the header from the original. It might be easier to do it that way for now.

I don't use any particularly exciting hex editors unfortunately. I actually annotated very similar images to that on some of my posts somewhere on this forum .

[Made_s]

How did you import the textures before ?

Also the header I'm talking about is the one I need for the program to cnvert it to png so you can edit it.

[StarsMmd]

How did you import the textures before ?

Also the header I'm talking about is the one I need for the program to cnvert it to png so you can edit it.

I edited textures that were in a different format. Not the textures from models though. It's things like the title logo adding a fairy type icon.

I see. The header the program expects would be different from the ones in the model files. Do you know what they should look like?

[Made_s]

What format were those images in ?

Yes, I mostly figured it out by watching how different file's headers differ in combination with the gamecube file

codumentation. The problem is it still throw errors occasionally and I'm trying to find out why.

[StarsMmd]

What format were those images in ?

Yes, I mostly figured it out by watching how different file's headers differ in combination with the gamecube file

codumentation. The problem is it still throw errors occasionally and I'm trying to find out why.

From what I heard it's pretty standard. They have a header which starts with the width and height in pixels, then the bits per pixel. They were typically 8 or 16 bit indexes into a palette at the end of the file which was 0x100/0x200 bytes long I think. I did see a lot of other files with different formats though. Oh also the pixels were divided into 8x8 or 8x4 blocks.

[Made_s]

Sounds like C4 and C8, could you give me the name/location of one of those files ?

[StarsMmd]

Sounds like C4 and C8, could you give me the name/location of one of those files ?

umm, for example in fight_common.fsys there is "uv_icn_type_big_00". I think there are a lot of them that start with "uv". The data is really easy to recognise by eye and you can find it in a lot of other places. common.fsys and field_common.fsys have some. nintendo_logo.fsys has one too but I was never quite able to decode that one. You find a lot of them in the .fsys files for menus too. Also all the files in poke_body, poke_face and poke_dance are textures for the box sprites, mugshots and purify chamber animated sprites respectively.

[Made_s]

The nintendo logo looks like CMPR to me. ¯\_(ツ)_/¯

So how exactly did you edit those uv files ? did you use something to convert them ?

[StarsMmd]

The nintendo logo looks like CMPR to me. ¯\_(ツ)_/¯

So how exactly did you edit those uv files ? did you use something to convert them ?

yeah there's this program called Tiledggd that is good for displaying these kinds of files. You feed it the starting offset of the image data (so after the header which is usually 0x80 in these). You set the image width and height, the start offset for the palette if there is one, the bits per pixel, the rgb order, whether or not there is an alpha channel, whether the data is tiled and if so the width and height of each block. For now I've just been getting this information by inspecting the hex with a hex editor. Once all the settings are done correctly you will be able to see the image in the program and there's an option to export it as png and I think you can also export to another format but I can't remember for sure.

Tiledggd doesn't have any support for converting backwards from a png back to the original format so I had to write that code myself.

[Made_s]

So if you already have that, waht should the program be able to do? I haven't written anything yet because it's been a bit unclear what you want it to do.

[StarsMmd]

So if you already have that, waht should the program be able to do? I haven't written anything yet because it's been a bit unclear what you want it to do.

It would be useful to have a program that can extract the textures from models and to replace them directly in the model files with an edited png. I was wondering how feasible it would be to just dump one large image from the models that contains all the textures in one. It could be easier to extract and reimport that way.