LGPE romfs data Reverse Engineering

[Kaphotics]

Starting a topic related to reverse engineering the romfs, specific for the switch pkm titles.

Reverse engineering ROM filesystem data for the Let's Go! Pikachu/Eevee games is easier than the previous games for the 3DS, as the games lack usage of NARC/GARC packed files in favor of having mostly unpacked contents. The romfs is obtained by decrypting the ROM and unpacking with tools such as hactool.

The Nintendo Switch patching system uses delta patching and full-file replacement with a virtual romfs. The 3DS patching method only had a virtual romfs file used full file replacement. Smaller patches are now possible, with best results if most files aren't packed together/compressed. The Nintendo Switch uses LZ4 as its primary (de)compression method as it is insanely fast relative to other methods.

A fair amount of content is kept between games; certain data structures are reused without change, as there's no need to reinvent the wheel. This is advantageous for reverse engineering, as we can compare known past game data to the current games. By comparing old and new, the updated data structures can be documented quickly and noted for changes.

One of the common problems in game development is translating human readable data into concise blobs of data for the game to use. Programs like pk3DS, pkNX, and various parsers are created to visualize the binaries the game uses, but during the game design phase, serialization from words -> data is much more difficult. To alleviate this issue, it appears that GameFreak has employed FlatBuffers as a way to handle irregular data structures like Wild Encounter data, Pokémart data, and others. They likely serialize (with a custom parser) from an excel spreadsheet into json, then compile the json with a flatbuffer schema to get the compiled binary and source code to read the compiled data. Reversing FlatBuffers is much more difficult than concatenated structures due to vtables and other annoyances, but it is possible with enough dedication 

The NDS and 3DS games used .narc and .garc as their main way to package files; this is no longer the case for LGPE. All files are unpacked, although they still use .sarc, mini, and other various sdk containers to bundle together some files (e.g. GUI assets). To replace the blz/lz11 compressed files, they now use lz4 in addition to 'zipping' into a .gfpak archive.

LGPE fortunately include filenames in the 12,000+ files that you'll get with unpacking the romfs. The game retrieves files starting with the root folder ("bin"), followed by the subfolder path (with "/" directory separators). A fair amount of indirect / non-hardcoding is present for loading game assets, which is accomplished by some data structures storing a 64bit hash [hashing a file's path starting with "bin" or only the filename]. This hash is computed via a 64-bit FNV hash one character at a time. The game keeps various hashtables (file magic "AHTB"), which stores the hash and the string that was hashed. This allows for easy retrieval of the value's index within an array (e.g. to set an event flag based off the name of the flag, rather than the absolute index).

Feel free to discuss your findings about the game romfs here!

[Kaphotics]

pkNX has been set up as a class library for easy reuse in other projects. Inside the codebase is re-implementation of various game data structures and file locations.

• Structures
• Game File Locations

Note: these urls are from when this reply is posted. For the most up-to-date reference, please browse for the files using the current commits.

pkNX also includes an example of flatbuffer file conversion using a flatbuffer schema (.fbs), from *.bin -> *.json, and *.json -> bin. Use the provided fbs and bat files, which use the flatc.exe to convert the data. Keep them in the same folder as the stuff you're converting