Help with sorting out to find duplicates

[GuardianOfExile]

Hey there everyone. I recently lost my Sun and Ultra Sun saves on my 2DS in an attempt to get homebrew on there with a bunch of my Pokemon and didn't think to back up the SD card because of how close I was to finishing. Long story short, my SD card got corrupted and when Windows 10 asked to "repair" it, it actually just wiped the entire drive and didn't warm me. 

 

So I found a way to get my old pokemon back through old backup saves on my other SD card (which I have backed up since in the incident) and to my avail, I found the Pokemon on a variety of saves. Now, these Pokemon are the culmination of almost 15 years since I was 5 and couldn't bear to think that I just lost them like that. Thousands of pokemon over the years and I wouldn't even know where to start on which ones I lost. Not even PokeBank could hold them all. 

 

My issue though is that while I edited them or whenever this happened, some of them got duplicated many times. Is there a program that helps me find duplicates and I can see a wide overview of my Pokemon? I thought of using Porybox but it hasn't been updated in two years, there is no way to download an entire box, and its incredibly sluggish with loading Pokemon. PkHeX is super helpful but only has like 31 boxes which is no where near enough for me, unless it can be increased or something like that than I will just use that to help me search through everything. 

 

Thanks to whoever can help me out! I very much appreciate it!

Edited October 9, 2019 by GuardianOfExile

[theSLAYER]

5 hours ago, GuardianOfExile said:

Hey there everyone. I recently lost my Sun and Ultra Sun saves on my 2DS in an attempt to get homebrew on there with a bunch of my Pokemon and didn't think to back up the SD card because of how close I was to finishing. Long story short, my SD card got corrupted and when Windows 10 asked to "repair" it, it actually just wiped the entire drive and didn't warm me. 

 

So I found a way to get my old pokemon back through old backup saves on my other SD card (which I have backed up since in the incident) and to my avail, I found the Pokemon on a variety of saves. Now, these Pokemon are the culmination of almost 15 years since I was 5 and couldn't bear to think that I just lost them like that. Thousands of pokemon over the years and I wouldn't even know where to start on which ones I lost. Not even PokeBank could hold them all. 

 

My issue though is that while I edited them or whenever this happened, some of them got duplicated many times. Is there a program that helps me find duplicates and I can see a wide overview of my Pokemon? I thought of using Porybox but it hasn't been updated in two years, there is no way to download an entire box, and its incredibly sluggish with loading Pokemon. PkHeX is super helpful but only has like 31 boxes which is no where near enough for me, unless it can be increased or something like that than I will just use that to help me search through everything. 

 

Thanks to whoever can help me out! I very much appreciate it!

if you use PKHeX to dump the entire box into a folder on your PC, you should be able to compare if there's duplicates, based on the checksum of the file. (it's in the file name).

If I'm only mistaken, this would only work for exact copies tho. (down to the level, exp, moves etc)

[GuardianOfExile]

What do u mean by checksum? Is that some PKHeX has in it?

[theSLAYER]

19 minutes ago, GuardianOfExile said:

What do u mean by checksum? Is that some PKHeX has in it?

The string of characters that follow.
 

Looking at the string may give you an idea of how similar the entries (within the same species) are.
For example, these 2 Rattata are clones of each other, the only difference being one has a third move Bite.

As you can see, most of the trailing digits are the same.


This method may be laborious, but it's the only one I can think of right now.

Additionally:

Also, if the exact copies already exist in the folder you're trying to dump to, PKHeX wouldn't export them.
This would help to reduce the amount you need to check, as PKHeX don't export out exact copies/duplicates when they already exist.

edit: When I say dump boxes, I'm referring to this function in PKHeX:

Also, remember when you're doing the "string comparisons", the files have to belong to the same generation.
(compare PK5 to PK5, compare PK7 to PK7 etc)

[Guest]

Once you're done doing Dump Boxes on every save file, you can go to Tools --> Data --> PKM Database, then in that window, at the top, go to Tools --> Create Data Report.

You can hide unnecessary columns by selecting a cell in that column and right-clicking Hide Column, and you can easily go down the list and compare PIDs and such to see which Pokémon are dupes. It might take a while to go through the entire list if you have so many Pokémon, but you only need to do it once, and it's probably worth it.

Once you're done... Make a backup! And keep it in a safe place, so you never have to do something like this again.

[theSLAYER]

4 minutes ago, Ammako said:

Once you're done doing Dump Boxes on every save file, you can go to Tools --> Data --> PKM Database, then in that window, at the top, go to Tools --> Create Data Report.

You can hide unnecessary columns by selecting a cell in that column and right-clicking Hide Column, and you can easily go down the list and compare PIDs and such to see which Pokémon are dupes. It might take a while to go through the entire list if you have so many Pokémon, but you only need to do it once, and it's probably worth it.

Once you're done... Make a backup! And keep it in a safe place, so you never have to do something like this again.

This is definitely better, ^ recommended!

[GuardianOfExile]

Oh geez! Thank you both for your help! I'll try this after my class tomorrow and will let you know how it goes!

[GuardianOfExile]

So I was able to narrow down all my Pokemon to around 1800 and are in several locations and backed up. Now it is a matter of getting all Ultra Moon Pokemon back into Ultra moon and respectively with Moon. And I'll create the save files for Sun and Ultra Sun onto my 2DS and export it onto my SD card so I can import the remaining Pokemon and all the events I have missed. Thanks again for you help! 

[Kaphotics]

Keep in mind PKHeX's database allows you to delete all clones based on a hash of the pkm's details. Helps clear up things if there's EXP differences, EV, etc.