===================================================
==  Baz the Poochyena's PMD:EoS/T/D - StatsUtil  ==
===================================================
Version	   : 0.23.2 (WIP)
Released   : 2015/03/29
Written by : psy_commando (psycommando@gmail.com)

The entire sourcecode for all my tools that are part of this suite is available here:
https://github.com/PsyCommando/ppmdu/

----------------------------------------------------------------------------------------------------
Changelog:
----------------------------------------------------------------------------------------------------
- 0.1 : Initial release!
- 0.11(2015/03/30):
    * Fixed specifying a single argument, with no options, not running Export ALL properly!
    * Fixed specifying a directory for import, with no options, not running Import All properly!
    * Removed slow progress indicator for text_*.str file parsing and writing! Exec time cut from 
      ~6s to >1s !
-0.2(2015/09/09):
    * Fixed item import/export for Explorers of Sky!
    * Removed support for Explorers of Time/Darkness for now. The format differs too much!
-0.21(2015/09/10):
    * Fixed game string import.
-0.22(2016/06/18):
    * Fixed bad SIR0 padding.
-0.23(2016/08/22):
    <-!!!!!-> Remade entirely how parameters are processed at the command line, because of the new features requiring extra details. <-!!!!->
    * Added XML configuration file system, to handle most game versions specific data.
    * Added ability to XML conf file to refer to append external configuration files by using a <External filename="" /> node.
    * Added Script decompiler + recompiler. Everything is human readable XML.
    * Added game detection. The ROM root directory specified at the commmand line is analysed to accurately determine the version.
    * Improved PMD2 characterset support. Whenever extended multi-byte characters are used in any other locale than
       Japanese, they're turned into an escaped sequence of characters. That means all multi-bytes characters beginning with 
       a 0x81 to 0x84 byte. 0x81 0xF4 for example is used for the little music note in other locale than Japanese. While 
       in the Japanese locale,  most hiragana and katakana characters are multi-byte combination of 0x81 to 0x84 followed
       by another byte. That's how Shift-JIS encoding works.
    * The game strings parser now supports hexadecimal byte escape sequences: \xhh, in addition to the 
        already supported \n, \0.
    * The script strings parser supports the same escape characters!
    * The configuration files are now loaded relatively to the executable's directory instead of the current working directory.
    * References to entities, levels, face portraits, common routines, and more are replaced with their debug name and other info
      stored and editable in the "pmd2scriptdata.xml" file! 
      This should make everything more readable, and customizable/maintainable no matter if there are ASM hacks applied to the game or not. 
    * Added support for importing/exporting script data along with scripts.
    * Added support for dumping the level list to a SIR0 file. Its pretty much completely useless to anyone for now however.
-0.23.1(2016/08/24):
    * Fixed issue on windows 10 where a lot of the scripts wouldn't export or import properly. The multithreading class was at fault.
-0.23.2(2016/09/09):
    * Fixed the logs and console not displaying errors and silently skipping a file when there was an error in it. This was a side effect
      of changing the method of handling multi-threading, since I forgot I was not checking for exceptions..
    * Added output of a compiler report file with all the issues encountered during script compilation.
    * Improved a lot the error output. It now shows the actual line the error is on, and an approximate positions of the error on the line.

----------------------------------------------------------------------------------------------------
License info:
----------------------------------------------------------------------------------------------------
My code and tools are CC0. 
Free to re-use in any ways you may want to!
No crappyrights, all wrongs reversed! :3

This is not including any libraries I'm using that have their own licenses of course!!
The license for each of these libraries is indicated in their respective header files,
and they're all isolated in the "libraries" subfolder !

----------------------------------------------------------------------------------------------------
Special Thanks:
----------------------------------------------------------------------------------------------------
-> Many, many thanks to Zhorken for reversing most of the "PX" compression format! - https://github.com/Zhorken
-> A big thanks to Nerketur for his help !
-> A big thanks to TruePikachu for his help !
-> A big thanks to Evandixon for his help !
-> Thanks to the people from GBATemp and Project Pokemon Forums !
-> Thanks to MNChino for helping with some of my tools !
-> ..and anyone else I may have forgotten about ! ^^;

----------------------------------------------------------------------------------------------------
About:
----------------------------------------------------------------------------------------------------

This tool can export various data from the Pokemon Mystery Dungeon : Explorers of Sky/Time/Darkness 
games(Only tested with Sky for now).

Right now it supports:
- Game text (Some dialog, menus, item names, move names, pokemon names, etc..)
- Move data (Base power, type, name, descritpion, etc.. )
- Pokemon data (Name, category, type, stats growth, level-up moves, hm/tm moves, egg moves, etc..)
- Game scripts and script data.(This include all scripted dialogs, interactions, cutscenes, game logic, 
                                 and basically anything happening out of dungeon)

------------------------------
--Random Uninteresting Fact--:
------------------------------
The weird name is inspired from my EoS playthrough, where I cheated to get a Poochyena as starter. 
He turned out really good after ingesting so many gummies and pretty much carried the team 
most of the game! XD
And, I just love that pokemon, so yeah, that's pretty much it..

----------------------------------------------------------------------------------------------------
How to use:
----------------------------------------------------------------------------------------------------

** IMPORTANT **
    - Keep all XML files with the executable, or if you have to move them, use the proper command line option.

The easiest way to work with the tool is to open a command prompt, and running an import/export all command:

Example line for exporting all:
    ppmd_statsutil.exe -e -romroot "EoSRomRoot" "ExplorersOfSky_Out"

Example line for importing all:
    ppmd_statsutil.exe -i -romroot "EoSRomRoot" "ExplorersOfSky_Out"


The -i tells the tool to import, while -e tells it to export. 

The -romroot followed by the directory containing the extracted rom's files sets the directory 
the tool will work with for either exporting from, or importing into. 
Its expected to contain:
    * A "data" directory containing the game's extracted filesystem. 
    * A "arm9.bin" file. 
    * A "overlay" directory containing the overlay_####.bin files for the game.

The one and only argument, "ExplorersOfSky_Out", is the path to export the data to, or to import from.
Its expected to have a certain directory structure on import (Directories may be entirely ommited, when they don't contain any data):
    - Any game text data must be under the "game_strings" directory, and contain text files named after their target language.
    - Any item data xml files must be under the "item_data" directory and contain the xml for all items.
    - Any move data xml files must be under the "move_data" directory and contain the xml for all moves.
    - Any pokemon data xml files must be under the "pokemon_data" directory and contain the xml for all pokemon/entities.
    - The script sub-directories must be under the "scripts" directory.


Exporting:
----------

* Exporting Pokemon data to XML:
    Type this at the command line (without the first set of quotes): 
        "ppmd_statsutil.exe -e -pokemon -romroot "path/To/PMD2/ROM/Root/Directory" "Path/To/Output/Directory""
        Replace the paths with the appropriate paths! You can omit the output path, 
        and the xml data will be exported to a default directory.

* Exporting move data to XML:
    Type this at the command line (without the first set of quotes): 
        "ppmd_statsutil.exe -e -moves -romroot "path/To/PMD2/ROM/Root/Directory" "Path/To/Output/Directory""
        Replace the paths with the appropriate paths! You can omit the output path, 
        and the xml data will be exported to a default directory.

* Exporting item data to XML:
    Type this at the command line (without the first set of quotes): 
        "ppmd_statsutil.exe -e -items -romroot "path/To/PMD2/ROM/Root/Directory" "Path/To/Output/Directory""
        Replace the paths with the appropriate paths! You can omit the output path, 
        and the xml data will be exported to a default directory.

* Exporting game text to text file:
    Type this at the command line (without the first set of quotes): 
        "ppmd_statsutil.exe -e -text -romroot "path/To/PMD2/ROM/Root/Directory" "Path/To/Output/File/text.txt""
        Replace the paths with the appropriate paths! And replace the filenames as desired. 
        You can omit the output path, and the text data will be exported to a text file with the default name.

* Exporting game scripts to XML:
    Type this at the command line (without the first set of quotes): 
        "ppmd_statsutil.exe -e -scripts -romroot "path/To/PMD2/ROM/Root/Directory" "Path/To/Output/Directory""
        Replace the paths with the appropriate paths! You can omit the output path, 
        and the xml data will be exported to a default directory.


Importing:
----------

* Importing Pokemon data from XML back into the game:
    Enter this at the commandline (without the first set of quotes):
        "ppmd_statsutil.exe -i -pokemon -romroot "path/To/PMD2/ROM/Root/Directory" "path/To/Extracted/Data/Directory/To/Import""
        Replace the paths with the appropriate paths!

* Importing moves data from XML back into the game:
    Enter this at the commandline (without the first set of quotes):
        "ppmd_statsutil.exe -i -moves -romroot "path/To/PMD2/ROM/Root/Directory" "path/To/Extracted/Data/Directory/To/Import""
        Replace the paths with the appropriate paths!

* Importing items data from XML back into the game:
    Enter this at the commandline (without the first set of quotes):
        "ppmd_statsutil.exe -i -items -romroot "path/To/PMD2/ROM/Root/Directory" "path/To/Extracted/Data/Directory/To/Import""
        Replace the paths with the appropriate paths!

* Importing game text from a text file back into the game:
    Enter this at the commandline (without the first set of quotes):
        "ppmd_statsutil.exe -i -text -romroot "path/To/PMD2/ROM/Root/Directory" "path/To/Extracted/Data/Directory/To/Import""
        Replace the paths with the appropriate paths!

* Importing Scripts from XML back into the game:
    Enter this at the commandline (without the first set of quotes):
        "ppmd_statsutil.exe -i -scripts -romroot "path/To/PMD2/ROM/Root/Directory" "path/To/Extracted/Data/Directory/To/Import""
        Replace the paths with the appropriate paths!

More Details:
-------------

* Pokemon Data:
    The Pokemon data is outputed to several xml files. One for every Pokemon. Its name contains the
    index of its primary gender in the game data, followed by the Pokemon's name in the game strings
    at the moment of export.
    Most Pokemon files contains 2 "gender entities" those are basically how the game deals with
    pokemon having more than one gender. Even mono-gendered species, or genderless ones have a 
    "secondary gender entity". 
    Only special Pokemon don't have 2, and those appears to be mainly dummies used during cutscenes.

    If the data was exported from Explorers of Sky, the Pokemon will also have 2 moveset tags,
    one for each of its entries in the game files. For some reasons EoS has duplicate entries
    for all Pokemon movesets, some are slightly different.

* Move Data:
    The move data is exported in a similar way to Pokemon data. If the game it was exported from
    is Explorers of Sky, a duplicate entry of the move's data will be present. That's because the
    EoS game has a copy of all move data, and its unclear if its even used, but better 
    output it with the rest I guess.

* Item Data:
    The item data is exported just like the last two types of data. Some items may have exlusive data.
    But only past a certain item index are items considered exclusive items.

* Text Data:
    The text data is exported in the simplest method possible, as a raw text file!
    Each line is a single string in the game. All special characters are escaped with a backslash!
    So, end of lines are "\n", end of string are "\0" and other unprintable characters are "\ddd" where
    'd' is a decimal digit.

    Be sure not to use your enter key ever in that text file, or you'll create new text 
    entries for each new lines you make! And since the game looks for those entries by 
    the order they appear in, you'd only screw up all the text for the entire game!

    Also, keep in mind that, if you export other game data, that allow editing a few strings,
    like Pokemon data, importing a the text data afterwards might overwrite the changes you've
    made, as the text data file that is exported also contains those strings! It does contains every
    strings used in the entire game after all !

* Scripts:
    Scripts are fairly simple overall. Each sub-directory is a level/map. 
    "enter" script(s) + data are for anything interactable on a playable map.
    "Acting" script + data are for cutscenes mainly.
    "Station" script(s) + data are for things added to a map depending on the progress in the game for the most part.

    Its probably better to look at the thread or the wiki/notes for details on those, as I don't have a complete understanding of them
    yet myself. And things are likely to change a lot.

To edit those, a reference sheet might come in handy, and you can pick one over there: 
http://apointlessplace.net/wms (look for a item_p.xlsx excel sheet, and go to the keyvalue tab)
Those numbers next to all those things are the same values that end up in the XML data!
You can also go on the Project Pokemon Wiki and look at the monster.md and the waza_p.bin page!.
http://projectpokemon.org/wiki/Pokemon_Mystery_Dungeon_Explorers_of_Sky

----------------------------------------------------------------------------------------------------
Readme:
----------------------------------------------------------------------------------------------------
ppmd_statsutil.exe (option) input path (output path) 
  ->option          : An option from the list below..(optional)
  ->input path      : Path to the file to export, or the directory to assemble.
  ->output path(opt): Output path. The result of the operation will be placed, and named according to this path!


Options:
  -i          : Specifying this will force import!
  -e          : Specifying this will force export!
  -pokemon    : Specifying this will import or export only Pokemon data!
  -moves      : Specifying this will import or export only move data!
  -items      : Specifying this will import or export only items data!
  -text       : Specifying this will import or export only the game strings specified!
  -scripts    : Specifying this will import or export only the game scripts!
  -locale     : Force the utility to use the following locale string when importing/exporting the string file! Using this option will bypass using the gamelang.xml file to figure out the game's language when exporting game strings!
  -scriptdebug: When specified, all debug related commands in the scripts are tweaked so they're enabled by default!
  -cfg        : Set a non-default path to the pmd2data.xml file.
  -xmlesc     : If present, this makes the program use XML/HTML escape sequences, instead of C ones. AKA &#x0A; instead of \n for example. This is mainly interesting for dealing with a XML parser that requires standard XML escape chars!
  -romroot    : Specify the root of the extracted rom directory to work with! The directory must contain both a "data" and "overlay" directory, and at least a "arm9.bin" file! The "data" directory must contain the rom's files and directories!The "overlay" directory must contain the rom's many "overlay_00xx.bin" files!
  -th         : Used to set the maximum number of threads to use for various tasks during execution.
  -log        : Turn on logging to file.


Examples:
ppmd_statsutil.exe -i -e -pokemon -moves -items -text -scripts -locale "C" -scriptdebug -cfg "path/to/pmd2/config/data/file" -xmlesc -romroot "path/to/extracted/rom/root/directory" -th 2 -log "c:/pmd_romdata/data.bin" "c:/pmd_romdata/data" 

Description:
To export game data to XML, you have to append "-e" to the
commandline, followed with the option corresponding to what to export.
You can import data from XML into a PMD2 game's data, by
specifying "-i" at the commandline, followed with the
option corresponding to what to import.

----------------------------------------------------------------------------------------------------
Documentation :
----------------------------------------------------------------------------------------------------
See my thread here for links to my notes and etc: 
http://projectpokemon.org/forums/showthread.php?40199-Pokemon-Mystery-Dungeon-2-Psy_commando-s-Tools-and-research-notes
