GTS: website research

[Vlad]

Post additions (pinned replies):

[Apr 24th, 2009] AngelSl (pcap log of GTS communication): post#12

[Mar 1st, 2010] LordLandon (logged GTS communication in more close details): post#39

[Mar 2st, 2010] LordLandon (information on actually trading with a fake GTS server): post#41

[Mar 2st, 2010] LordLandon (python script to act as GTS server): post#43

[Mar 22nd, 2010] LordLandon (python script, IP fix): post#100

[Mar 23rd, 2010] magical (clarification how data prefix is encoded on website): post#104

[Mar 23rd, 2010] イーブイ (started Wiki article, check next page after clicking this link for more test results): post#105

[Apr 8th, 2010] AngelSl (dnsserver.py modification hint -to support CNAMEs and not only IPs): post#193

[Apr 10th, 2010] Vlad (port check tool in AutoIT3): post#224

[Apr 11th, 2010] Vlad (dns and http server package): post#129

Note about LordLandon's Python application:

You get "An error occurred while attempting to communicate (00000) Please turn off the power, restart this DS system, and try again.

If this message keeps appearing, please contact Nintendo Consumer Service." when the .pkm file is a storage file. It must be a party file in order to work properly (issue is that storage files are 136bytes while party once are 236bytes).

Original post text:

Odd that no one is curious, but I am so here I go!

Basically the game talks to Nintendo and the global trade center, but I wonder if we can get some of that data from external tools. I am pretty sure the official GTC flash site also reads from here, anyway I found out the URL addresses by hex searching in the rom file (platinum), here are the results:

http://gamestats2.gs.nintendowifi.net/pokemondpds/common/setProfile.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/worldexchange/post.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/worldexchange/post_finish.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/worldexchange/get.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/worldexchange/result.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/worldexchange/delete.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/worldexchange/return.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/worldexchange/search.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/worldexchange/exchange.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/worldexchange/exchange_finish.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/worldexchange/info.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/battletower/roomnum.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/battletower/download.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/battletower/upload.asp
http://gamestats2.gs.nintendowifi.net/pokemondpds/battletower/info.asp

Obviously one is the battletower and other is the trade. Now it seems to operate on GET and POST, since I found a lot of HTTP headers. They also set a User-Agent that I think is called "GameSpyHTTP" (but might be a variable name or refrence, as I also saw "User-Agent: HTTP_X_GAMECD" a line or two above.

Anyway it is possible to get information of trades and pokemons available using external tools, thus you may create a "alert me when X is available" tool if you like, or filter out those "I want a Mew for a Mew" people that are just annoying.

Any thoughts or information you have learned?

Edited April 24, 2010 by Vlad
Renamed GTC to GTS (typo), added post "stickies" and that's it.

[Sabresite]

Yeah you are correct on your information. We have done research into this already and currently are figuring out the encryption. However it seems very difficult without the ability to debug the game as it is running.

Good luck on your research though

On and regarding making an external tool. If you use that website too much from a specific IP, especially a static one associated with a website server, I would take the assumption that they may figure it out

[Vlad]

I just share what I come across as I see almost none other results on Google of what others have concluded -I just try to contribute.

Another thing is that it seems that it operates on arrays, at least the game. That might mean it might send or receive a array with data at some point.

Another thing is that you can (on each URL) access by sending a GET "pid" with a value of maximum 2147483647 (decimal, maximum 32-bit integer value). It returns a random string that looks like a session id as it changes each time you request the page. It is always 32 chars and contains letters and characters.

Additionally you have a GET "hash" that can be a random string containing numbers and letters. It's lenght is not specified as you may go up and up as much as you like, only you will get "error: token expired" all the time. To call "hash" you need to specify a "pid", and since pid alone returns a random string that looks like a hash.

If you do a GET pid with a random decimal (let's say "1"), you get a session key (let's call it that). Now if you add the GET hash and add the key you got from the earlier call, you get "error: incorrect hash". Now if you do a new request but only change the GET pid value (keeping GET hash the same) you get "error: token expired" so that must mean it expects a valid reply based on the session key you got.

I assume the next step would be allowing you to enter if you "enter the right password" based on the key you get, this is something the game does and this way they keep "us" out from their servers.

Also having user-agent set to "GameSpyHTTP" does not seem to do anything, the results remain the same.

I guess the game generates a pid based on time, perhaps ds id but I would think that the pid would be something it gets by requesting the server too, otherwise you would have to blindly try random numbers between 1-2147483647 until the DS gets a key that is not in use, I mean I guess the most logical way the game communicates with the server is like this:

1. Request a PID from the server, get a PID or a "maxusers" error

2. Use PID to request a session key

3. Use PID and send in decoded session key

At this point, the PID is flagged valid and you are allowed to browse and request more data, so the PID will be registered as "in use" and no one else can use it as it will not be send out by the server to new connecting clients. When there is a timeout, the session data is cleared and the PID is now publicly available for new connecting clients.

If this is how it really works, this means there is a maximum connection of 2 147 483 647 (2 billion 147 million 483 thousand 647 users) witch is plenty imo... Anyway, I assume you are also stuck on this part, or there is something I have missed.

I have also tried to use a network sniffer tool to monitor the connection between me using the flash based GTC by forcing it to only show HTTP based requests and connections, but I didn't really find much of useful info, didn't find the right data at all.

I will attempt by looking for connections done to the domains holding the urls posted above and see if there is any results when I access the flash site.

*Edit*

Alternatively, you may harvest GTC data from a mirror, thus utilizing the Pokémon GTS website. It is flash based but accesses data from the server, and it is conveniently in plain text format, hehe.

Example, http://www.pokemon-gts.net/data/pokemon/1.txt would bring up information on Bulbasaur. I have not decoded what is what, but there are 2 lines on top that are headers most likely, and the rest of the data is just information of what is happening.

Headers consist of 10 columns, each value is separated by comma (,).

The body is made of 14 columns but no comma separations this time.

The site also uses

http://www.pokemon-gts.net/data/entrusted.txt - available for trade (trading away)

http://www.pokemon-gts.net/data/requested.txt - requested by others (offers)

http://www.pokemon-gts.net/data/exchanged.txt - exchanged, more like a "log" file but I can't make out if it just yet, perhaps it counts how many that have been traded over I guess

I am looking into the flash files of the site, since the main flash loads sub-flashes it would be possible to get more information where to find trainer information, level, e.g. by disassembling the files.

*Edit*

Trying that, the sub flash files returns only images and sounds. The core files might contain scripts, but they all seem encrypted so that's lame. ;P Anyway I'll see what I figure out by looking on the text files, hopefully something useful.

[AngelSL]

Hi. I have access to a softAP which my DS uses to connect to Nintendo Wifi (see my other thread in the same section). Nintendo Wifi club uses SSL (the one on the PC bottom floor), so I would'nt be surprised if this uses SSL too (I saw SSL handshakes when I was logging GTS, btw)

If you happen to need my help, just throw me a PM.

[Vlad]

Thanks AngelSl, it's nice of you.

Right now I just made a small GTS site, nothing special (works fine in Firefox but not IE6) that let's you basically access the pokemon data on the gts site without having to load all the flash and all, and shows who is trading X for X (same pokemon) and who wants a pokemon that is banned from GTS (mew, celebi, darkraid, e.g.) so it is easier to see the valid results.

Only issue is that the moves they know are ID numbers apparently but the values do not correspond to my database, like 33 for me is tackle while on the file it can be 210 or what ever. Also locations where the pokemon come from seem to get the values from a table of some sort, takes ages to figure out all the valid ids from 1-250 to get the location names lol.

[Kaarosu]

Pretty cool stuff going on here keep up the good work.

By the way.

http://www.pokemon-gts.net/data/exchanged.txt - exchanged, more like a "log" file but I can't make out if it just yet, perhaps it counts how many that have been traded over I guess

That seems to be a list of actual pokemon traded by their National Dex #

[Vlad]

I noticed that the data on the site is basically logs 1 day behind, thus what is available right now does not seem to be there. For that I think I really need to use the other domain address, think it's time to contact AngelSl and see what he got to say on the matter.

Anyway the exchanged.txt is a log of the most traded pokemon. Left side is the national id and the right side is the "rank". requested.txt and entrusted.txt work the same way, only it shows the top wanted and deposited.

The /pokemon/#.txt files contain 2 header lines and 100 entries showing what were traded for what, level, move abilities, e.g. but it is all a log from yesterday.

[AngelSL]

I haven't had time to log GTS yet (the last time I didn't save the pcap file), but I'll do it today after my homework's done.

[Vlad]

I don't think pokemon-gts.net contains up-to-date data. gamestats2.gs.nintendowifi.net on the other hand does. Checked data on the gts site, was 1 day old.

Btw, while checking my router for connection info when I connected to Nintendo with my NDS, I got a interesting IP that leads me to a router login (haha): 92.21.129.247 - not so secure is it? Having it exposed like that.

[bear831]

The information is always a few hours off and I am not so sure how they cycle through there updates like what times.

[damio]

Right now I just made a small GTS site, nothing special (works fine in Firefox but not IE6) that let's you basically access the pokemon data on the gts site without having to load all the flash and all, and shows who is trading X for X (same pokemon) and who wants a pokemon that is banned from GTS (mew, celebi, darkraid, e.g.) so it is easier to see the valid results.

It works with Opera too (10.0 alpha)

I think this would've helped with the Daisuke pokemon, now all we need is a Time Machine, and we are good to go.

[AngelSL]

I've attached a pcap file of my packet logging.

Here's what I did:

> Login

> Search for a Bulbasaur: Either gender, Any level

> Search for a Darkrai: Either gender, Any level

> Logout

And I incorrectly said it was SSL. Sorry.

GTS Login SearchBulbasaurEitherAny SearchDarkraiEitherAny.zip

[Vlad]

Thanks AngelSl! I'll take a look into the file and see what I can do.

*Edit*

It looks like it communicates using SSLv3, at least requests a certificate each time you get on the GTS (the certificate has an expiration date in 2015) that the NDS reads and gets it's public key, then using that to establish a connection with the server.

I am trying to make my PHP work with OpenSSL but something bugs -argh. If someone that develop would like to take a peak in the certificate and attempt to open a connection would the GTS -it would be lovely.

The extracted certificate from the logged packet:

nintendo.zip

Edited April 24, 2009 by Vlad

[Sabresite]

Certificates and crytography is not my strong suite :\

[Vlad]

SSL (inc. certificates) is not my strong side either. Need time to learn about how SSL operates and if I can use PHP to fetch data. Never looked into it before now so I am kinda blank on experience too.

Anyway it's nice that we do have some progress at least. I'll drop off messages as I make new discoveries on this matter in case someone is interested.

[codemonkey85]

Only issue is that the moves they know are ID numbers apparently but the values do not correspond to my database, like 33 for me is tackle while on the file it can be 210 or what ever. Also locations where the pokemon come from seem to get the values from a table of some sort, takes ages to figure out all the valid ids from 1-250 to get the location names lol.

I'm rather curious about this. Are you saying that sometimes the number for Tackle is 210, and other times it is other numbers? Perhaps there is an encryption going on here. But why encrypt move and location numbers and not Pokédex numbers?

In any case, excellent job! This is cooler than I ever expected. If it weren't for the possibility of Nintendo eventually blocking people out, there could be a lot of uses for this in hunting events (as previously mentioned), or even just casual trades.

[Vlad]

Just to make it clear so there is no confusion, pokemon-gts is a log that display trades 1 day ago. The "live" data is yet to be extracted as there is some SSL going on, it's not the easiest thing to do.

Anyway about the moves, it's basically that the custom database index (id) of moves is not matching the official. Simply because our databases are made by us so it's all custom. Or because the website just have another index order, who knows.

It does not seem encrypted, as Tackle is #91. On my database it's #33. That means I need to like log the new ID that correspond to the right abilities but that's a lot of work. :<

[derrick]

so some one can use this to see what pokemon are on the gts on there computer aand its up to date?

[Vlad]

That's the plan, derrick. To simulate being the NDS and login from the computer then check what you want to check. Thus not having to depend on logging on the GTS with the NDS all the time...

Especially if you plan to mass-check many at once and it's much easier to be able to filter out results like people that trade X away for X (same pokemon) or trading away for legendaries (very few want to give away legendaries for normal pokemon). BUT first, someone must manage to login properly. I myself have read some and played with some Java code but yet to succeed in authenticating properly.

[codemonkey85]

Some questions / ideas:

1. If you figure out how to mimic the DS as far as logging on to the GTS is concerned, I wonder how much access you could have to the GTS database? In other words, maybe it is possible to access the Pokémon data itself, instead of the data tables the site uses? Assuming there's a way to avoid alerting Nintendo, it would be pretty neat if you could import / export .PKM files to and from the GTS using a PC. Maybe the GTS could even be used as a medium between your DS game and a PC?
   
2. What kind of doors could mimicking the DS and logging into Nintendo's servers open? Could we then access trade and battle records? Battles stored on the Global Terminal using the Battle Recorder? PC box images that were saved on the GT? The Wi-Fi Square?
   
3. Also, how about combining knowledge of how to mimic the DS and knowledge of the DS Wi-Fi protocol? Connect your DS to your PC by the PC mimicking another DS or a Wii, and use your PC application to monitor / control communication between the DS and Nintendo's servers?
   

This is pretty exciting stuff, I think.

[derrick]

but what about nintendo i mean wouldent some lawsuit be filed or some thing? if they found out? also if yall need some one to test this i could possabley help in some way

[Vlad]

1. If you figure out how to mimic the DS as far as logging on to the GTS is concerned, I wonder how much access you could have to the GTS database? In other words, maybe it is possible to access the Pokémon data itself, instead of the data tables the site uses? Assuming there's a way to avoid alerting Nintendo, it would be pretty neat if you could import / export .PKM files to and from the GTS using a PC. Maybe the GTS could even be used as a medium between your DS game and a PC?

I assume the access is limited to what you can do with the NDS. Except on the NDS you for example are limited to maximum 8 trainers at once when you search while tapping in would let us read all of the available. Exporting or importing, well you would indeed be able to upload pokemon data but I am not sure if they use the same structure as pkm files, I assume they hold a little more info so the server knows who to tell when the one that uploaded the pokemon logs in on GTS. If you notice its then you see that it has been traded and get the other pokemon.

1. What kind of doors could mimicking the DS and logging into Nintendo's servers open? Could we then access trade and battle records? Battles stored on the Global Terminal using the Battle Recorder? PC box images that were saved on the GT? The Wi-Fi Square?

You can access the data you can access on the NDS, since the server handles request trough the ASP scripts -thus they are pre-programed what kind of data they will return us.

1. Also, how about combining knowledge of how to mimic the DS and knowledge of the DS Wi-Fi protocol? Connect your DS to your PC by the PC mimicking another DS or a Wii, and use your PC application to monitor / control communication between the DS and Nintendo's servers?

I read the packet and I see that the server gives the NDS a certificate and based on that the client authenticates with the server. Now I've been trying to make a Java program (simple) connect and do just this but I have yet to succeed.

If we get this to work then it will also work with any other NDS game that has a Wifi mode with some internet activity going on. They are all based on the same principle. Now I am no pro at this, I just play with code I find trough documentation and google and see if I can make something work.

[derrick]

this would be so awsome!!!

i hope it works i tryed the link to your interntet site and it just said gateway time out

[coolness]

is this all not illegal?

my laptop is telling that

i can install that certificate

what will happen if i do that?

[Guest]

Man, this is so interesting!

If we manage to import the Pokémon's file, and create a converter to convert it into a .pkm file, we could basically take any Pokémon from the GTS for free.

But about lawsuits...