Help Wanted! - Pokémon DS Save Structures

[codemonkey85]

Hello all.

As you may know, the Project Pokémon Wiki has a (rather extensive) documentation on the Diamond and Pearl save file structure. However, there is quite a bit of information missing from this page. Additionally, there is no documentation up regarding the Platinum save file structure.

It would be of great benefit to save hacking developers (Sabresite and myself for example), as well as the wiki and site as a whole, if we could get these structures mapped out. And of course, anything that helps Sabresite helps PokéMod.

If you have a means for extracting your Diamond / Pearl / Platinum save file(s), and a hex editor (like HxD), you can help map the save file structure by simply making a copy of your save file, making a change, saving, and comparing the previous save with the current save. (By the way, one of our members, evandixon, has created a toolkit to be used for save file mapping.)

Additionally, I am going to create a small tool within the next few days with the express purpose of fixing the save file checksums in Diamond, Pearl, and Platinum. With this tool and a hex editor, you can edit the save file manually, and then load the save to see what changed.

EDIT: And here it is.

Things this thread has accomplished so far:

• Trainer card signature structure for DP and Platinum (credit goes to Jiggy-Ninja!)
  
• Pokédex data structure for DP and Platinum (thanks again to Jiggy-Ninja!)
  
• Location of Running Shoes obtained flag for DP (credit goes to evandixon!).
  
• Value for the direction you are facing for Platinum (credit goes to Relyte!), for DP - credit goes to me!
  
• The Pokétch Pokémon History and Trainer History apps for DP
  

There is a long list of things to look for specifically, and here are a few in no particular order (admins and mods feel free to edit this list):

• Flags for which trainers have been battled
  
• Flags for which events have been completed (i.e. received items from NPCs, received eggs from NPCs, used medicine on Psyducks, and so on).
  
• How shiny one's badges are (from polishing them).
  
• The Feebas tile coordinates (or however they're stored).
  
• The Pokétch current color (changed with the Color Changer app).
  
• How many steps are on the Pokétch Step Counter app.
  
• How the Pokétch Dot Artist art is stored.
  
• The coordinates of the markings on the Pokétch Marking Map app.
  
• How entries in the Pal Pad are stored.
  
• How entries in the Journal are stored.
  
• The location for the answers chosen for various NPC questions regarding favorite Pokémon etc.
  
• Which trendy phrases have been unlocked.
  
• Which extra PC wallpapers have been unlocked.
  
• Which location has been set by the globe in the GTS / GT building.
  
• Which locations have been unlocked on the globe in the GTS / GT building (by trading with people from those locations).
  
• Which phrases have been chosen for when a battle is won, lost, etc. in the Battle Tower.
  
• Which Battle Frontier symbols have been collected.
  
• How battle videos are stored.
  
• How underground bag and PC items are stored.
  
• How Secret Base location is stored.
  
• How information on Secret Base decoration configuration is stored (which pieces and where they are located).
  
• How number of Secret Base rocks cleared is stored.
  
• How Pal Park last migration date and last migration game is stored.
  
• How the game can tell if the DS clock has been changed.
  
• Which Key Item (if any) is registered to the Select button.
  

I will be constantly adding to and editing this post for more details.

If anyone is interested in helping out or has any questions, post here, PM me, meet me on IRC, or check out our social group. If we get organized, we can have this thing done in no time, and that will ensure that we are ready to crack HeartGold and SoulSilver when the time comes.

DateTimeBytes.zip

Journal.zip

Edited June 13, 2009 by codemonkey85
Added tool to convert between date and bytes stored in save file.

[Neo]

Yeah id love to help but could you explain to me what i would need to do and what programs are needed? Like a tutorial.

[Endless Eden]

So all thats needed is a hex editor and 2 SAV files(lets say diamond)and you compare the differences in the hex of the 2 say one has a certain item the other doesnt a certain line of hex whould be different is that what it involves?

[codemonkey85]

Pretty much exactly what you just said, telos.

NeoDraven, I mention and link to what programs you need in the first post.

EDIT: Here is an example of how to find something in the save file.

Let's say you are looking for where the Trainer ID is stored. You know your Trainer ID is 12345. You would simply use HxD to open your save file, and search for an integer of 12345. You might find that number in a couple areas; if that is the case, try it with a different save file with a different TID. It's really just process of elimination.

As far as comparing two files is concerned, you could save one file right before you battle a trainer and back it up, and then save another copy after you beat that trainer. Comparing the files in HxD will yield all of the changes, and you may be able to locate the flag for "this trainer defeated". That's a little tougher though, but that's essentially what needs to be done.

[Endless Eden]

Pretty much exactly what you just said, telos.

NeoDraven, I mention and link to what programs you need in the first post.

so the hxd hex editor is the one your using?I looked at hex at one stage and it kind of baffled me a bit.If i were to help whould i have to understand what all the numbers correspond to or whould i just report diffrences in the lines of hex to you(after a certain event,item obtained etc.)

[codemonkey85]

Well, we're trying to figure out what the numbers correspond to. That's why it's best to look for values you already know. See the example I edited into my previous post.

[Neo]

Oh i get it now thanks.

Ill start working on it soon.

[Endless Eden]

ahh so what you mean is if you know the line of hex that corresponds to a certain item,ID etc.that when your looking for diffrences in hex you should find the line you know and navigate from there? this is probably a stupid question but flags?what are flags about?

[codemonkey85]

if you know the line of hex that corresponds to a certain item,ID etc.

That's not what I meant. As in the example I posted, I simply meant you should start with a value you know, like your Trainer ID. You know what your Trainer ID is, so you search the file's hex for that number, and that's how you find where in the file that information is stored. Again, that is the goal here... to see how the information in the game is organized in the save file, so that we know where to look for it when we go to read it or edit it.

this is probably a stupid question but flags?what are flags about?

By "flag" I mean a value indicating the status of something. For instance, whether or not you have defeated a certain trainer. I have no idea what the structure is, but I imagine it will be a value for "defeated" and a value for "undefeated", maybe as simple as either 1 or 0.

[Endless Eden]

i was looking at the wiki and is the structure like it is laid out there?like,you have save index number coming after ID is that how it whould be in the hex editor your save index after ID.I understand the whole blocks thing that makes sense but..I will have to try this myself i will get back to you when i understand things better maybe then i can help.

[Jiggy-Ninja]

• The Pokédex data structure - Pokédex obtained or not, current mode (Sinnoh or National), National unlocked, forme viewer unlocked, languages unlocked, flags for seen, caught, genders / formes recorded, first genders / formes seen for each, and languages.
  
• The direction your character is facing (up, down, left, or right)... I know it seems silly but I'm going for a complete map here.
  
• The signature on the back of your Trainer Card (and the structure that information is stored in).
  

These sound fun, and the middle should be pretty easy.

I have a Japanese Platinum, but since the save structures are identical it won't matter much.

Comparing the files in HxD will yield all of the changes, and you may be able to locate the flag for "this trainer defeated". That's a little tougher though, but that's essentially what needs to be done.

HxD has a compare function?

It's not really tougher to locate flags. The process is exactly the same, it just might take a little longer.

The only think difficult about flags is that multiple flags can be stored in one byte, like how Ribbons are stored on a Pokemon.

[codemonkey85]

i was looking at the wiki and is the structure like it is laid out there?like,you have save index number coming after ID is that how it whould be in the hex editor your save index after ID.

What you are looking at specifically is a block footer, which each save block has. Everything that we know about the save files is up on that page, and that is how the information we know of is laid out. It would stand to reason that you would find the information in the hex editor in the same manner and order.

Try it yourself. Load up a Pokémon DS save file, run a search for the PID of a Pokémon you own (and know the PID of), and see where you find it.

HxD has a compare function?

Sure does. Analysis --> File Compare. Then you can hit F3 I think to iterate through the differences.

It's not really tougher to locate flags. The process is exactly the same, it just might take a little longer.

The only think difficult about flags is that multiple flags can be stored in one byte, like how Ribbons are stored on a Pokemon.

Bitfields are a bit tricky, but there's also the fact that you have to remember to save twice between saves so the small block is in the same position in both files. Otherwise it might be tough to make accurate comparisons. Plus, other things will be changing as you save, and we haven't identified them all.

Also, thanks everyone for volunteering. Like I said, this helps the whole site, so it's pretty awesome that we have a group effort going here!

[Neo]

Your welcome Its pretty easy to understand really. I did the Trainer ID Thing and it worked.

[Jiggy-Ninja]

I solved the trainer card signature thingy. Surprisingly, it's a little more complicated than I thought.

First, the offsets.

D/P: 1,536 bytes, starting at 0x00005904

Plat: 1,536 bytes, starting at 0x00005BA8

The signature area is 192 x 64 pixels, for a grand total of 12,288 pixels. This area is divided up into 8x8 groups of pixels. Every 8 bytes of the signature's memory block represents one of these boxes, the first being in the upper left, in the order left-to-right, top-to-bottom (just like reading English).

Within each of those 8 byte structures, each byte represents a row within the pixel group, the least significant byte being the topmost row. Each bit within each byte represents a pixel (1 = Black, 0 = White) in that row, with the least significant bit representing the leftmost pixel.

So, in order to make an 8x8 group with a one pixel black border, you would use the values: FF 81 81 81 81 81 81 FF. 2 dimensionally represented in binary, this gives you:

11111111

10000001

10000001

10000001

10000001

10000001

10000001

11111111

For a 2 pixel wide stripe down the left side, you use: 03 03 03 03 03 03 03 03

11000000

11000000

11000000

11000000

11000000

11000000

11000000

11000000

It's important to remember that the DS uses whichever Endian (I think it's Little) puts the bits in the reverse order that we read them, with the least significant bits to the left.

192 of these groups are assembled to form the image that makes your signature.

Next, Pokedex! This should be fun!

[codemonkey85]

A few things:

1. That is some seriously impressive work. Thank you, and kudos to you! I think I will use this information to work up a quick signature viewer (and maybe eventually editor) app just for fun. Sabresite will be happy to see this, as he wanted to add it to PokéMod as well.

2. If it's all right with you, I'd like to put this on the wiki. I don't have time to format it and stuff now, but if it's not done by the time I get home tonight, I'll put it up (with your permission).

3. FYI, everything in the DPPt save files is, as far as I know, written in little endian like you suggest. However, call me crazy, but it looks like your example of 03 (11000000) is in big endian format.

4. I am certainly going to be late for the BBQ I'm "getting ready" for. ;p

[ZMaster]

...

Next, Pokedex! This should be fun!

Ooo, so that means you'll crack down on the parts of the Pokédex like the language extensions?

[Jiggy-Ninja]

A few things:

1. That is some seriously impressive work. Thank you, and kudos to you! I think I will use this information to work up a quick signature viewer (and maybe eventually editor) app just for fun. Sabresite will be happy to see this, as he wanted to add it to PokéMod as well.

2. If it's all right with you, I'd like to put this on the wiki. I don't have time to format it and stuff now, but if it's not done by the time I get home tonight, I'll put it up (with your permission).

3. FYI, everything in the DPPt save files is, as far as I know, written in little endian like you suggest. However, call me crazy, but it looks like your example of 03 (11000000) is in big endian format.

4. I am certainly going to be late for the BBQ I'm "getting ready" for. ;p

1) Thank you. Wasn't really that hard. Using my Pearl, I dumped saves with both an all black and blank sig, and used HxD to find the differences. The gigantic block of Fs in the Black sig save that was all 0s in the Blank sig save was a dead giveaway for the location. Then, I made a number of test case sigs, which included just the corners, and black bars across the sides and top. Analyzing those test cases let me get the way they were stored. Then, I found the Plat location and did a quick check to make sure it was stored the same way. Only took maybe an hour, 2 at the most, including the time to type up the post.

2) This is a topic made to recruit people to map out the save structure to improve the wiki. Posting stuff here that you didn't want on the wiki would seem quite odd to me. :confused:

3) My example is correct, and it is little endian. Big endian is the way our numbers are normally written, with the biggest numbers to the left. The number 4523 has 4 thousands, 5 hundreds, 2 tens, and 3 ones. The equivalent number in little endian would be 3254.

03, when converted to binary and little endian, is written as 11000000.

You're crazy.

4) Sounds like fun.

---------- Post added at 11:12 PM ---------- Previous post was at 11:06 PM ----------

Ooo, so that means you'll crack down on the parts of the Pokédex like the language extensions?

I shall do my best. First though, I need to know what you're talking about. I hardly ever use my Pokedex, so I'm not very familiar with all it's functions.

If you could detail all of the Pokedex's functions outside of the familiar "Seen it, caught it, got the data" thing, that'd be a big help.

I'll probably start with a clean game, and see what changes as I see and capture Pokemon in the SinnohDex.

This promises to be a fun challenge.

[codemonkey85]

I do realize that this thread is meant for the Wiki to be updated, so I guess I was just being overcautious in asking your permission. ;p

Hmm, I don't know how I confused the endianness there. I guess I confused the order of the bytes with the order of the bits, since every time I read data from the save file, the binary data is in big endian like it's supposed to be. Anyway.

Here is a breakdown of the functions for the Pokédex:

• Obviously there is Seen and Caught.
  
• There is also a slight difference in sprites for male and female Pokémon, and as you know many Pokémon have alternate "formes" (with the "e" for whatever reason) as well. The first Pokémon of a particular species that you see is recorded in the 'Dex as its default forme. In other words, if you see a female Pikachu before you see a male Pikachu, the female forme (with the tail split thing) will be the default Pikachu sprite in your 'Dex. To view alternate formes (aside from the first one you saw), you must have unlocked the forme feature in the 'Dex (an NPC from Rowan's lab does it for you at some point in the game).
  
• Every Pokémon in the Platinum version has foreign Pokédex flavor text stored in the game, which is viewable if you have acquired a Pokémon of that language in your game. As an example, if someone trades you a Japanese Pikachu, you will have the English and Japanese 'Dex entries for Pikachu unlocked in your game. If you get a Spanish Charmander, you can read Charmander's Spanish 'Dex entry. So on and so forth. However, you must unlock this feature before you can access the foreign entries... an NPC unlocks this for your 'Dex sometime after beating the E4. (You can trade for his German Magikarp afterward too.) In Diamond and Pearl, this feature was limited to only a handful of Pokémon, like Magikarp, Pikachu, and some others.
  
• As you know, there is a Sinnoh Pokédex and a National Pokédex. I think I found the byte that tells you which ones have been unlocked in D/P, but I never did get to Platinum. I also never got around to looking for the byte that tells you what mode the 'Dex is currently in (as you can switch them at will).
  

So basically we will need to figure out which Pokémon have been seen, which have been caught, which Pokémon have which formes as their defaults, which formes have been unlocked, which languages have been unlocked, which 'Dex modes have been unlocked, and which 'Dex mode is currently in use.

I think that's it.

[Jiggy-Ninja]

I do realize that this thread is meant for the Wiki to be updated, so I guess I was just being overcautious in asking your permission. ;p

Hmm, I don't know how I confused the endianness there. I guess I confused the order of the bytes with the order of the bits, since every time I read data from the save file, the binary data is in big endian like it's supposed to be. Anyway.

You're forgiven.

[*]Obviously there is Seen and Caught.

Quite obvious, yes.

[*]There is also a slight difference in sprites for male and female Pokémon, and as you know many Pokémon have alternate "formes" (with the "e" for whatever reason) as well. The first Pokémon of a particular species that you see is recorded in the 'Dex as its default forme. In other words, if you see a female Pikachu before you see a male Pikachu, the female forme (with the tail split thing) will be the default Pikachu sprite in your 'Dex. To view alternate formes (aside from the first one you saw), you must have unlocked the forme feature in the 'Dex (an NPC from Rowan's lab does it for you at some point in the game).

Probably someone that spoke British English translated that part, since I think that's the British spelling of "form". Not 100% sure though.

Is the form part usable right after you get the Pokedex, or is there something that needs to be done to unlock it? Also, is every alternate form available to view from the beginning, or do you need to unlock those too?

[*]Every Pokémon in the Platinum version has foreign Pokédex flavor text stored in the game, which is viewable if you have acquired a Pokémon of that language in your game. As an example, if someone trades you a Japanese Pikachu, you will have the English and Japanese 'Dex entries for Pikachu unlocked in your game. If you get a Spanish Charmander, you can read Charmander's Spanish 'Dex entry. So on and so forth. However, you must unlock this feature before you can access the foreign entries... an NPC unlocks this for your 'Dex sometime after beating the E4. (You can trade for his German Magikarp afterward too.) In Diamond and Pearl, this feature was limited to only a handful of Pokémon, like Magikarp, Pikachu, and some others.

Cool. So each language dex entry for each Pokemon is unlocked as you get it?

And which dex entry is unlocked depends on the 0x17 Original Language value, right?

Where is the kid that unlocks this feature?

[*]As you know, there is a Sinnoh Pokédex and a National Pokédex. I think I found the byte that tells you which ones have been unlocked in D/P, but I never did get to Platinum. I also never got around to looking for the byte that tells you what mode the 'Dex is currently in (as you can switch them at will).

Yup, know this too.

So basically we will need to figure out which Pokémon have been seen, which have been caught, which Pokémon have which formes as their defaults, which formes have been unlocked, which languages have been unlocked, which 'Dex modes have been unlocked, and which 'Dex mode is currently in use.

I think that's it.

That's it is a bit of an understatement, but now I've got a good starting point.

[codemonkey85]

Here is a map of Sinnoh, for reference.

Is the form part usable right after you get the Pokedex, or is there something that needs to be done to unlock it? Also, is every alternate form available to view from the beginning, or do you need to unlock those too?

-The forme part is usable after it is unlocked by one of Rowan's aides. I believe this happens in the building right between Route 218 and Canalave City.

-Only the alternate formes that you have personally witnessed in-game are able to be viewed. And it is important to note again that the first forme of a Pokémon that is seen becomes the default sprite in the 'Dex.

Cool. So each language dex entry for each Pokemon is unlocked as you get it?

Yes. It's actually a lot of fun to collect them, really.

And which dex entry is unlocked depends on the 0x17 Original Language value, right?

I believe so, yes. Those values are also consistent with what appears to be the original language of the trainer itself, located in the save file structure.

Where is the kid that unlocks this feature?

Reffering to the foreign entry viewer:

He is on Route 226, in a house on a little island. You are unfortunately unable to visit him until after you get the National Pokédex.

[Jiggy-Ninja]

so the hxd hex editor is the one your using?I looked at hex at one stage and it kind of baffled me a bit.If i were to help whould i have to understand what all the numbers correspond to or whould i just report diffrences in the lines of hex to you(after a certain event,item obtained etc.)

While it is best to know what you're doing, just reporting notable differences and similarities for other people to interpret is helpful too.

If you do that, be sure to upload the actual save files too.

---------- Post added at 07:20 PM ---------- Previous post was at 06:26 PM ----------

After searching through 3 save files, I am fairly confident that the Sinnoh and National Dex flags for Platinum are at 0x1642 and 0x1643, respectively.

Now comes the fun part. deciphering the huge bitfield that stores the data for our beloved ぜんこくずかん.

...I just noticed, I think I used a Pokedex complete code on my Japanese Platinum (which I totally forgot about), since I have all the forms and languages unlocked for all the dex entries.

That means that the flags for the Forme and Language extensions are probably at 0x1640 and 0x1641, since they're both 00h on the two new saves I made and 01h on the regular Plat save. I don't know which is which yet, but I'll figure it out.

In my Jap Plat save, there's a big block of data 493 bytes long (familiar number, anyone) of nothing but 3Fh. It starts at 0x1451. I suspect that each byte is a bitfield that represents which parts of a Pokemon's Dex entry is unlocked.

This is progressing much faster than I thought.

---------- Post added at 07:33 PM ---------- Previous post was at 07:20 PM ----------

Eh, this is odd.

For the new saves I started, the only dex entries in it are seeing Starly, seeing Turtwig, and owning Piplup.

At the place I predicted Piplup's dex flags to be, there's a 0x02. But at the places for Turtwig and Starly, there's nothing at all.

Bit 1 seems to be the "Captured" flag, in any case.

Most peculiar.

[Poryhack]

Just wanted to chip in and say great job; this thread is shaping up to be very useful.

I think I'd actually be capable of this stuff but just don't have the time. =(

[codemonkey85]

For the new saves I started, the only dex entries in it are seeing Starly, seeing Turtwig, and owning Piplup.

At the place I predicted Piplup's dex flags to be, there's a 0x02. But at the places for Turtwig and Starly, there's nothing at all.

Are you sure you were looking at the correct save block? This could be the backup block, assuming you saved right after you got Piplup and right before you saw Starly and Turtwig.

Other than that, I dunno.

I think I'd actually be capable of this stuff but just don't have the time. =(

Exactly the reason I'm recruiting grunts! I mean, uh, great job everyone.

EDIT: Here is that tool for fixing checksums that I mentioned at the beginning of this thread.

[Jiggy-Ninja]

Are you sure you were looking at the correct save block? This could be the backup block, assuming you saved right after you got Piplup and right before you saw Starly and Turtwig.

Other than that, I dunno.

Exactly the reason I'm recruiting grunts! I mean, uh, great job everyone.

EDIT: Here is that tool for fixing checksums that I mentioned at the beginning of this thread.

I make sure to save at least twice (sometimes 3 or 4 times just because) before examining the save.

Plus, it's impossible to save after getting Piplup and before battling Turtwig, you go straight into the battle.

I also saved just before entering Sandgem, after I saw the Starly.

[codemonkey85]

Yeah, I figured you knew what you were doing. Just thought I'd ask the obvious question (I'm good at that I guess?).

Plus, it's impossible to save after getting Piplup and before battling Turtwig, you go straight into the battle.

Oh right.

I guess the next thing I would try is capturing Starly and seeing if bit 1 in what you expect to be Starly's flag (0x15DD I guess) changes like it's supposed to. But that is pretty odd.