DP sprite extraction gone wrong

[waffleoRai]

Hey!

I'm working on some code to dump sprites of interest from pokemon ROMs (my goal is to add it to my own tools - I do not need a tool that does ROM editing or reading and I do not need an existing sprite dump) and since I already had various Nitro format parsing code from my file explorer I thought I would start with Diamond/Pearl (specifically, my copy of Diamond). 

I've had no issue with the majority of NCGR and NCLR files, can read NARC files just fine, LZ decompression seems to work just fine. However, there's a ton of totally normal looking NCGRs (from hex view) that when dumped to png seem to produce nothing but garbage static.

I tried reading them linear, I tried reading them tiled, I tried all kinds of wacky combinations of tile sizes and image dimensions. Garbage. I thought I might have a really weird bug in my program, so I decided to try viewing them with Tinke.... and got the exact same thing!

 

On one hand, good to know I am not completely crazy, but on the other hand this would imply that either Diamond is full of garbage image data or there's some other trick you need to do to read the image data that isn't indicated in the NCGR file itself. Are there higher level sprite/tile maps (NCER etc) that map pixel-by-pixel rather than by tile? Why have a NCGR at all then?

The files I'm looking at are in poketool/pokegra/pokegra.narc.

Generally, I am having a lot of difficulty finding clear documentation on the location of the various pokemon sprites for DPPt (eg. battle sprites) and potential quirks like this with their formatting. Maybe it's just something that's known in the community so nobody feels the need to write it up? Maybe it's just hard to look up (always get buried with existing spritesheets when try to search)?

I dunno, all I can think of now is to just troll through random bin files for tables or run game in emulators and look at RAM/VRAM dumps. Maybe dig through the code of an open source pokemon gen IV specific sprite extractor/injector tool? I just feel so stupid running around in circles over this problem that seems obvious and has been solved for 15+ years...
 

Anyway, any help would be appreciated - even redirecting me to an old post asking the same question! 

Thanks!
-B

[waffleoRai]

UPDATE

Got it! I think.

For the sake of anyone who might come across this in the future and doesn't already know, here's the trick. For some baffling reason, GF decided to basically encrypt the pokemon battle sprites (like someone who really wanted to rip them wouldn't just dump from an emulator VRAM, but okay). I really should have been able to tell from looking that the data were scrambled/encrypted, but I didn't really know why they would bother encrypting DS assets stored in a standard format...? Weird.

The encryption method appears to be the same as the one used for DP save files. Only the actual pixel data in the RAHC block is encrypted, which is why the rest of the NCGR looks normal. The decryption seed for the PRNG is the last word in the encrypted data. The algorithm goes word by word backwards from there until it gets to the start of the data. Words are read Little-Endian. I seemed to get the proper results by xor-ing with the lower 16 bits of the PRNG output, not the upper like the linked doc suggests.

Had to pop PokeDsPic DP into ILSpy to get that answer. So thanks to the dev of PokeDsPic! None of the other DS tools I tried seemed to be able to read these sprites.

No idea if this exact algorithm is used for Pt, HGSS, or Gen V since I haven't checked yet. They are definitely obscured though - least I got scrambled garbage when I tried to look at battle sprites in these other games.