research Idea: A New Rom Hack Distribution Format

[evandixon]

For a while I thought to myself, there should be a ROM hack format that patches individual files in the ROM, instead of the ROM itself. Meanwhile, psy_commando had a similar idea (I'm making a new thread so that one will be exclusively for PMD2).

The current discussion for those who haven't read it:

I've also been working on designing a automated and convenient way of sharing PMD2 mods. Possibly other romhacks as well.

Most other romhacks use a single differential patch for the whole rom, but that's problematic, because it doesn't support different revision and languages of the game, or anything that modifies the CRC of the ROM. And you cannot combine romhacks together, even if they could really be combined (think of a sprite pack and another hack that changes only, say, pokemon names for example ).

So, I've considered making a set of 2 utilities. One for easily and automatically assembling a mod package, and the other would be a redistributable utility that would handle extracting and patching selectively the ROM.

A mod package is just a zip file, that would contain:

- The patch installer

- ndstool (might just merge the source code into the installer)

- xdelta (might just merge the source code into the installer)

- A xml file with the details for patching the ROM, how to handle other languages, what CRCs are supported, etc..

- A directory containing a replica of the ROM's file structure, filled with differential binary patches of original ROM files(for "legality" reasons), and full copies of any new files added.

The directory with the files would contain sub-folders, and would be named according to the game ID string they contain files for. So that means there would be a single package for a romhack that was localized in multiple languages for instance!

The nice thing, is that people wouldn't need to download patching tools, and do things themselves if they're total noobs and etc.. The patcher would unpack the rom, patch/add the files, rebuild the rom, and that's it.

Conflict handling could be done via CRC checks, so the application could ask the user what files they want to keep.

The concept could be pushed further even! Any opinions ?

I've actually had a similar idea, although I think we should take it a step further and think of this like Minecraft Forge mod packs. Each mod would be a zip file with patches like you've described, but would be contained in the parent "mod pack" zip (and maybe renamed to .mod). For example, a hypothetical mod pack could contain two mods: New Starters and New Backgrounds. New Starters would modify overlay13 to change the personality test results and text_en.str to change the corresponding message ("And a calm type like you... will be a ____"), while New Backgrounds replaces some pictures in /BACK.

Each mod could contain multiple file types: some being binary patches, others being full copies of files, while others can patch certain parts of a file, like different mods changing different parts of the text_en.str. Maybe something like text_en.str.lines containing "line X=Some text; line2=Some other text".

When I was thinking about it, I was going to have Sky Editor know how to patch ROMs, but having a patcher inside the mod pack is a good idea. If ndstool and xdelta aren't merged into the installer, the could be tucked away in a directory called "Utilities", while a program called "Patcher" sits in the root of the mod pack, waiting for someone to open it.

I've actually had a similar idea, although I think we should take it a step further and think of this like Minecraft Forge mod packs. Each mod would be a zip file with patches like you've described, but would be contained in the parent "mod pack" zip (and maybe renamed to .mod). For example, a hypothetical mod pack could contain two mods: New Starters and New Backgrounds. New Starters would modify overlay13 to change the personality test results and text_en.str to change the corresponding message ("And a calm type like you... will be a ____"), while New Backgrounds replaces some pictures in /BACK.

I haven't had the chance to take a look at minecraft forge mod packs.

But yeah that could work. The utility could parse the XML data of each sub-mods and install them itself one after the other. Though, I don't know what advantage it would have over putting them all together into a modpack. Considering modpacks are meant to be distributed together anyways.

Each mod could contain multiple file types: some being binary patches, others being full copies of files, while others can patch certain parts of a file, like different mods changing different parts of the text_en.str. Maybe something like text_en.str.lines containing "line X=Some text; line2=Some other text".

That sounds good.

But, one possible problem with patching only some part of the file is that, its done assuming the file was in a certain default state. If some extra lines were added by another mod, its hard to know if the lines another mod would change would be the right ones.

However, most of the tools I've made extract things in a format that could be rebuilt with many changes properly and re-distributed.. It might be possible to basically rebuild the ROM piece by piece doing it that way ?

Though, it might be a little extreme.. Especially since we don't know if it will catch on. But, who knows, with the new PMD game coming out, it could possibly be applied to it as well ?

(I'm planning on reversing that game as soon as I can get a decrypted ROM of it ! I sold my old 3DS and well, it was already updated past the firmware that could decrypt games easily.. I've actually taken a look at gates to infinity and etrian mystery dungeon and it seems pretty familiar. They even use compiled lua scripts for nearly everything ! The 3d model format looks pretty straightforward as well. But they're using a compression format that I've seen in other chunsoft games, but never found out what it was.. Probably another LZ compression variant. )

When I was thinking about it, I was going to have Sky Editor know how to patch ROMs, but having a patcher inside the mod pack is a good idea. If ndstool and xdelta aren't merged into the installer, the could be tucked away in a directory called "Utilities", while a program called "Patcher" sits in the root of the mod pack, waiting for someone to open it.

Now that you mention it. It might be better not to put too much relatively unrelated features into sky editors, if I could say so. Because SkyEditor seems to be heading towards being more of a powerful dev tool.

And while devs would feel at home with it, the average users might freak out if they're offered too many options. And well, it would be easier to debug two smaller programs and codebases than a single larger one. Anyways, that's just my opinion, feel free to disregard ^^;

But, I digress. I also considered making a patching program, but its way easier to have the patcher distributed in with the rest.. And people wouldn't have to go looking for the right version of the program and etc..

Then again, some people might see it as a security risk.. And they'd be right, really.. given anyone could put whatever they want instead of that utility..

But, the mods could also be applied using an external utility. There would be the light-weight redistributable patcher utility, and the full-size one would combine the patch maker and a patcher together.

So that people that don't want to trust the utility in a modpack can just use the full-size stand-alone program instead. The XML data is really what would contain all the important mod specific details.

And I grabbed the source code for xdelta and ndstools. It shouldn't be too hard to work with and combine them, as long as the dependencies can be dealt with.

EDIT: Scratch that, I took a deeper look and they're both true C-style references salads I think it would be much easier to just include them in the same package as separate executable..

I've been working on upgrading Sky Editor to make making ROM hacks like these easier, and before I actually write code for making patches, I think it's time that we ROM hack tool developers discussed and made a standard.

Here's my proposition: a "mod pack" zip contains one or more smaller mods and the patching tools. A "mod" zip contains all the data needed for 1 patch. Having various smaller mods allows them to be reused in other packs, and to let users pick which mods to apply to their ROM. For example, maybe in Rutile Ruby or Star Sapphire, what if I hypothetically wanted the increased XP curve, but didn't want any changes to items. I could then apply the "XP Curve" mod, but not the "Item Balance" mod.

Here's a more technical proposition:

File structure of the Mod zip (probably called .ndsmod or .3dsmod):

/Mod.json

• Contains information about the mod like Name, Author, Target ROM, etc.
  
• Lists IO operations that need to be done (maybe I want to swap /Data/Sound/BGM/bgm0045.smd and /Data/Sound/BGM/bgm0003.smd or delete the entire /TOP directory)
  
• Can optionally list other mods that must be applied before or after this one. (For mods of a mod; patching files that are added by other mods).
  

(Optional) /Patcher.exe

• Optional patcher in case weird file types are included, or to allow for randomizers.
  
• Usage: "patcher.exe [directory]", where directory is the path of the ROM directory (the one containing arm9.bin, arm7.bin, and the Data and Overlay directories).
  

/Files/

• Contains certain kinds of patch files in the same directory structure as ndstool extracts them.
  
• Ex. /Files/Data/BACK/expback.bgp.xdelta is an xdelta patch for the file at /Data/BACK/expback.bgp
  
• Ex. /Files/Data/FONT/kaomado.kao.kaopatch is a special file the patching utility knows how to handle that adds more images inside the file at /Data/FONT/kaomado.kao. The advantage over a binary difference is that multiple mods can independently add images without interfering with each other.
  
• Ex. /Files/Data/Back/MyBackground.bgp.file is a brand new file to be added at /Data/Back/MyBackground.bgp
  
• Any directories present anywhere in the folder hierarchy in /Files will be added in the appropriate place.

File structure of the Mod Pack zip:

/Pack.json

• Contains information about the mod pack like Name, Author, Target ROM, etc.
  

/Patcher.exe and/or /Patcher.bat and/or other comparable OS specific patcher file. Should probably be required to support command line arguments (patcher.exe [inputFile] [outputFile]).

/Mods/ - contains one or more mod zips.

/Tools/ - contains any tools or executables needed by the patcher. Exact name doesn't matter as long as end users don't need to worry about its contents.

/Tools/FileFormats.json

• Maps executables (or comparable files like .py) that know how to handle individual file patches to the proper extension, like the .kaopatch example above
  
• Ex. Could have an entry for .kaopatch that points to "/Tools/RomEditor.exe -kaopatch -i {Input} -o {Output}", where {Input} and {Output} are replaced with input and output paths.
  
• Ex. Could have an entry for .kaopatch that points to "/Tools/KaoPatch.exe" where usage "kaopatch.exe [inputFile] [outputFile]" is implied
  

This format is designed for NDS Roms, but should also work with 3DS Roms (as long as the patcher is included).

Formats of the .json files can be finalized later.

This format should be as legal as xdelta patches unless I've overlooked something. Feel to post questions/comments/concerns about any aspect of it, especially if you're a ROM hacking tool developer or a ROM hack developer.

[Edit 8/2] Updated details to match my current development.

Edited August 3, 2015 by evandixon