Why are pkm files stored decrypted?

[Zaz]

I'm currently trying to figure out Gen 3 game saves, and it is going quite well, however when reading discussions on this forum I find one thing confusing.

While discussing PKM files, the consensus seem to be that storing the data in decrypted form is much better than the alternative. I'd like to argue that the opposite is true.

The main argument for the decrypted storage is, as far as I can tell, that parsing the files is faster.

My counter to that is: Is that really relevant?

My computer can parse around 50000 encrypted gen 3 files in a second, so unless gen 6 adds a considerable load, aren't we good performance-wise?

The argument I wish to present _against_ decrypted storage is that the file format feels very unintuitive from an implementation standpoint.

Basically, someone has to define exactly what 'decrypted' means. While the answer to questions like "Which is the correct subsection order?" or "Big endian or little endian?" may feel intuitive, someone that is not-official has to make that decision.

While the encrypted format is pre-defined as valid when the game can parse the file, the decrypted files are valid when unofficial software considers them valid.

This just seems strange to me. I would guess that the praxis grew out of the fact that one very popular piece of software started doing it and all others just conformed to that behavior.

I figure every piece of software that handles pkm files already has parts that take the raw byte array (or stream) from the file save and creates some sort of internal storage representation to use internally. All software can also export the internal version back to the encrypted form. So why add another conversion to the _other_ external form?

I haven't been coding save files for very long, so I expect that there are many things I don't know here, so feel free to educate me. : D

And yes, I only made this topic because exporting encrypted storage files is much easier for me right now. Anyone curious about why can ask, but I guess most people wont be. ^^

[evandixon]

If pkm files must be decrypted to be read by programs, encrypting them when saving them as pkms is an extra step. Especially when the only thing pkm files are used by are editing programs.

If you want to store them encrypted, the typical extention is .bin.

The real question is why have them stored encrypted?

[theSLAYER]

I agree with Evandixon entirely.

To add on, I believe that GF had the encryption implemented in Pokemon to prevent data extraction and manipulation,

however hackers triumphantly rendered it useless by cracking the encryption.

Since the encryption is cracked, there isn't any point to save in encrypted form, as being able to extract the data, kind of defeats the purpose of having it encrypted.

[Zaz]

Evandixon, one thing I think you're skipping over without mentioning is my assertion that there isn't one single version of the 'decrypted' data. (But I will admit that it seems that everyone but me seem to use the same variant.)

My argument is basically 'one encrypted version, many decrypted versions - hierarchically, the encrypted one is the only arbiter of truth'.

Also, bin files - binary files - would be a fitting extension to both these kinds of files, since they both contain binary data. But that extension also carries no association with pokemon, so maybe .epkm or something would be better? ^^

Now the reason to store them encrypted I'd think be self evident. If i wish to copy one pokemon from one save file to another, I'd just copy the correct 80 bytes from one file, and write them to the other. No calculation, encrypting, decrypting, moving bytes around needed at all.

Also it seems unneccesary to decrypt the entire file if maybe I'm just interested in some specific part of the file. Say I wish to just list which unique monster types are stored in a save. I don't actually need to calculate any sub sections for that, so 'decrypting' the data would be wasted effort

And I admit that it is an extra step, but it is a step that every application that parses save files already has code for.

theSLAYER, I like your argument, it seems almost philosofical in a 'stickin it to the man' kind of way. : )

You basically mean that the developers of the game have one version of the data, the 'truth' so to speak, and then add encryption and obfuscation to hide it from us. And we are smart enough to see through their deception and display the truth for all to see?

Anyway, thanks for replying you guys, I really appreciate it. : )

[evandixon]

The real reason they're stored decrypted is because Pokesav saved them decrypted, and everyone else did in their tools for the sake of consistency and compatibility. If you as a developer want to make your own tool that saves encrypted Pokémon as .epkm files, you can do that; however, users will be a bit annoyed if it doesn't support the standard .pkm everyone's used to.

If you want to list unique monster types, you don't have any standard to follow. Users don't care whether or not the program internally decrypts the Pokémon. All they care about is that it supports their saves/pokemon and it works.

[Zaz]

Yeah, I know.. (still makes me a sad panda ^^ )

Do you know if the exact specs for a third gen pkm file are published somewhere? I guess its just XOR the stuff that should be xor'ed, and store the subsections in the order {species, moves, EVs, misc} since that is the order every example seems to give?

Also, I just realized I lied about the type example thing. It is too found in the subsections, so that was a bad example. >_<

[theSLAYER]

Evandixon, one thing I think you're skipping over without mentioning is my assertion that there isn't one single version of the 'decrypted' data. (But I will admit that it seems that everyone but me seem to use the same variant.)

...

You basically mean that the developers of the game have one version of the data, the 'truth' so to speak, and then add encryption and obfuscation to hide it from us. And we are smart enough to see through their deception and display the truth for all to see?

Yup precisely!

Also, I'm quite sure that everyone uses the same decrypted format,

cause I don't think you will get a reliable result if you decrypted it using another method.

(I once attempted decryption for Gen V, before I published my program.

During the beta tests, messed up the rng Seed and block shuffling on two different accounts.

that led to me never decrypting it, hence no program was able to read my decrypted files.)

[Afaik]

Think of it as the master key (our sharp eyes detecting the deception) to that one lock (the one truth).

Using any other key (any other argument) will not decrypt the file correctly (instead corrupts the truth).

Do you know if the exact specs for a third gen pkm file are published somewhere?

I believe this is what you're looking for

Gen III Pkm Data Structure

Gen III Save Structure

For Comparison:

Gen IV Pkm Data Structure

As a note, .pkm structure changed in every generation.

[Zaz]

I have used those bulbapedia articles extensively to parse the save files, but the focus of the articles are where to find the data in the encrypted file, not how to combine the result into a pkm file. It is only through subtext that the sub section order is understood to be what it is.

Anyway, I know no one asked, but I will say it anyway. :b

I simply load the encrypted data into a buffer and lazy load the encryption/decryption work until it is actually needed. It made for a very sleek and intuitive solution that was really easy to extend to cover the entire save file. At least it seemed like a cleaner solution compared to most other tools i've checked out, but then I couldn't find any sources for most any tools? Don't you guys share code in any way? I know codemonkey is on github and evandixion has code on codeplex, which is great, but those is the only one i have found? The other tools I had to reflect into to figure out how they worked. ^^

[theSLAYER]

I have used those bulbapedia articles extensively to parse the save files, but the focus of the articles are where to find the data in the encrypted file, not how to combine the result into a pkm file. It is only through subtext that the sub section order is understood to be what it is.

Anyway, I know no one asked, but I will say it anyway. :b

I simply load the encrypted data into a buffer and lazy load the encryption/decryption work until it is actually needed. It made for a very sleek and intuitive solution that was really easy to extend to cover the entire save file. At least it seemed like a cleaner solution compared to most other tools i've checked out, but then I couldn't find any sources for most any tools? Don't you guys share code in any way? I know codemonkey is on github and evandixion has code on codeplex, which is great, but those is the only one i have found? The other tools I had to reflect into to figure out how they worked. ^^

To my understanding, you're doing Gen 3 .pkm files right?

I'm sorry that I can't help you there; to my recollection, Pokemon Save Editing only kicked in full blow at the introduction of Gen IV.

Back then there was Pokesav, then came in Jiggly's PPSE, some hints of Pokestock, and eventually Pokegen.

Finally around the time of B2W2 release, appeared PikaEdit.

It's only around the introduction of HGSS, then I noticed the surfacing of Gen III editing tools, Enciclopedia (spanish I think?) and Pokestock's "PokeGBA" or "Pokebox" w/ Gen III functionability.

Gen III lasted between 2002 (JP RS) to 2005 (US E), with DP first released in Japan in 2006.

It came close to decade (actual figure is 8 years +) for the most stable version (IMO) of Advanced Generation Editing to come to light,

released only in the third quarter of this year: KazoWAR's A-save editor.

Anything Gen III related before that, to me, was all RAM editing.

My point to the long, long history lesson, is that I haven't seen any source codes for Gen III,

Probably because some programs and sources might have been lost in time (File hosting sites died, data corruption etc),

and it have might been deemed obsolete by some, especially due to "deemed complexity" of the nature of the file structure;

the PID-IV relations, and whatnot.

(also including the fact that Gen IV and V data can be edited easily with existing programs.

only problem for newbs is they might fail to create gen III data on pokegen,

due to palpark trash bytes)

:'(

[Zaz]

Thanks for the info, I really appreciate that you guys take the time to explain these things. : )

[codemonkey85]

It is true - the whole reason we use decrypted PKM files is because COM (creator of Pokesav) did it. He didn't do many things correctly, but we learned some of his bad habits all the same.

At least it seemed like a cleaner solution compared to most other tools i've checked out

Really? Cleaner than just defining the binary layout in a multi-tier structure then creating a pointer at the memory location of the save file? I find my way so easy to work with in code!

struct pokemon_obj : pkmunencryptblock,pkmblocka,pkmblockb,pkmblockc,pkmblockd {};
//...
struct box_obj { // size: 0x1000
public:
std::array<pokemon_obj,30> pokemon; //
uint16 : 16;
uint16 checksum;
byte buf[0x0c];
box_obj()
{
	memset(this,0,sizeof(box_obj));
}
};
//...
struct bw2savblock_obj { //
//...
std::array<box_obj,24> boxes; // size: 0x18000
//...
};

Then I get to do nifty stuff like:

std::cout << "The Attack IV of the Pokemon stored in Box 1, slot 1 is " << sav->cur.boxes[0].pokemon[0].ivs->attack << "\n";