tool Pokemon Mystery Dungeon 2 - Psy_commando's Tools and research notes

[Zerea]

First off just wanted to say thank you for all the effort you put into this game. The work you and others have put in to deeply understand and reverse-engineer it is actually incredible.

However I'm running into a couple problem using statsutils. I follow the directions to the letter (and have done so multiple times over with fresh starts), however upon loading into the newly recompiled ROM, I notice several item entries are half-replaced by a Plain Seed. Their descriptions still read the item in question, and the game sorts them in the same position as before, but they have the use properties of a seed. So far I've noticed this with several different TMs, Cleanse Orbs, Link Boxes, and maybe a few others. Perhaps related to this or different entirely, but nearly every seed item dropped in a dungeon is instead replaced by a 2 Poke ,, item(?) that goes into my Treasure Bag and either disappears or is added to my wallet upon sorting. I think only Blinker seeds were "unaffected" by this in the fact that I either picked up the 2 poke item or a blinker seed. All other items seemed to function as normal.

I thought perhaps changes I was making to the XMLs were screwing it up, even though they were only move/pokemon changes. So I did a fresh decompile and immediately recompiled without changing anything at all and still have these issues. The compiler report shows no errors whatsoever. I've tried using completely new games to test, as well as a different ROM download to see if those were the causes, and still no help.

Sorry to bug you with this rather silly problem, but I couldn't find anybody else having this issue so I wanted to ask the person who made the tool. Thanks for any help.

[psy_commando]

I'm glad all this work is useful to some people! ^^

I didn't notice that problem before. Thanks for mentioning this!

Do you know if there are some specific items that always end up corrupted like that? And are you using Explorers of Sky?

And a workaround could be to use the command line parameter to only export things unrelated to items.

Edited December 13, 2017 by psy_commando

[Zerea]

Thanks for quick reply!

Yes this is Explorers of Sky(US). This list is absolutely NOT exhaustive, as my storage is not "complete". In order of appearance when sorted in Storage:

Brick Break
Fire Blast
Hidden Power
Sludge Bomb
Toxic
Wide Slash
Cleanse Orb
Foe-Pause Orb
Mobile Orb
Link Box

Everything up to Brick Break and after Link Box functions 100% fine, with only those above being broken and the ones in between functioning fine (that I own) Based on this trend I'd imagine several more TMs/HMs are broken as well as a couple more Orbs that I don't currently have in storage. I could just use SkyEditor to add in all the TMs/Orbs I don't have to fully flesh out the list if that would be helpful. I also tried looking at said items in SkyEditor and they just show as their "proper" form, as the Hidden Power TM or as a Link Box. But when I add one to my inventory or storage it displays in game as a Plain Seed with the associated description, but still functions as a Plain Seed in that it can be eaten.

As for the seeds in dungeons, looking at them with SkyEditor shows them as "Pok+" and sorting the treasure bag removes these 'items' and places it into the wallet correctly. Any seed I take into the dungeon myself retains full functionality, works properly, can get put on the ground and picked up again with no issues. It's just any new seeds that spawn in the dungeon are broken and drop as 2 Poke instead. On a possibly related note I think normal Poke drops may have entirely disappeared from the possible spawned loot items, or that perhaps they're just being replaced with these "seeds"?

And thanks for the tip on just manually using the cmd instead of relying on the .bat to do it all. I'll be sure to give that a try when I have time and report back with progress.

EDIT: Alright so it took me a ridiculous amount of time to figure out how to manually use statsutil and it has led me on quite the amateur journey tonight. I played around with so many different syntax possibilities and read and reread the .txt file over and over trying to figure it out. I then finally get past the hurdle of figuring out the proper syntax without throwing an invalid first argument (which by the way was caused by not specifying an output directory, even though the .txt says it would just default a path). Only to find out that statsutil is NOT the data extractor from the .NDS as I thought it would do, as it is instead just the decompiler and requires an already extracted set of data files and .bins to turn into XML. I then started to try the Import command, but once again I realized all this did was update the .bin files and had no way to turn this all back into the playable .NDS. At that point I realized I could just forget trying to manually use the CMD and instead just use the first few .bats you gave in your pmd2_modding_setup to extract everything into XML cleanly, make my changes to pokemon/move data, nuke the entire item_data folder and compile everything using the .bat. Lo and behold, EVERYTHING WORKS. Seeds(Poke?) in dungeons work perfectly fine, all the broken TMs and Orbs are back to original form and all edits made to move_data and pokemon_data still hold.

So something goes haywire when either decompiling or recompiling item_data and some items don't fully make it back to all their proper spots. This has no impact on me as I have no desire to edit anything item related, and it seems there's only a rather limited amount of things you can do with them anyway (at least with this set of tools). However for the sake of completeness and bug fixing it might be worth looking into at one point.

Sorry for the huge wall of text but my inexperience caused me to go down quite the rabbit hole tonight and thought I'd share so other amateur users finding this post might find it helpful/amusing. I'm sure I was probably missing some core functionality of statsutil or just being really bad at figuring out what I needed to type to get what I wanted out of it.

Edited December 13, 2017 by Zerea
More Info

[psy_commando]

Thanks for the update! I suspected the item handling was somehow broken, but this confirms it!

And yeah, I built my tools to work with the extracted file system from a NDS rom, in order to make it more compatible with other tools. Since editing the .nds file directly would cause a lot of issues on the compatibility side of things. But I'm planning on adapting statsutil's functionalities into a full GUI application eventually. Just like I've been doing with the sprite editor. And eventually make all those tools work together with a map editor. So hopefully this will be a lot less hassle in the future!

I'll try to get the bugs fixed with item handling in the meantime. If you have any other suggestions/bugs, please let me know! ^^

Edited December 14, 2017 by psy_commando

[Zerea]

Wow I can't wait til the day I see all these tools working in one complete application. I've seen the progress you made with the sprite editor GUI so imagining all three at once in the quality of the application like you've got for sprite editor will be out of this world.

Thanks for the insight and for leading me to a workaround. I'll be sure to let you know if I notice any secret bugs lurking in a corner but the ROMs have been running beautifully so far. Keep up the great work!

[psy_commando]

So after testing I can't seem to get the items to corrupt.. Can you tell me more about what you did @Zerea ?

Did you use any tools on the rom afterwards? And what version of statsutil are you using?

 

Also I updated audioutil to fix a bug, and add multi-sf2 export on games with no main bank.

https://github.com/PsyCommando/ppmdu/releases/tag/ppmd_audioutil_0.37

[Zerea]

I'll try to break it down step by step as accurately as I can. This is with a NA Version of Explorers of Sky running on Desmume 0.9.11. I put additional notes in parenthesis to try and keep the step-by-step clean. I've tried this with two completely separate sources and same result. I have Administrator rights on my computer but I also ran each program as Administrator just to be on the safe side.

1. Downloaded pmd2_modding_setup from this link. Right-clicked on the .zip and went to Properties and clicked Unblock before extracting in case Windows was stopping some core process.

2. Dragged a copy of the Explorers of Sky ROM into the directory and renamed to rom.nds

3. Dragged the rom.nds onto 1a_ExtractRom_DRAG_ROM_ON_ME.bat (also tried 1b_ExtractRomNamed_rom.nds), and waited for the cmd prompt to complete and populate the rom directory and then close.

4. Double Clicked 2a_DecompileEverything and waited for that to finish to see both portrait and romstats directories created.

5. Made any changes I wanted to the appropriate pokemon_data/move_data XMLs (which ultimately didn't matter because those directories aren't the problem.)

6. Double clicked 3a_CompileEverythingAndMakeRom, waited until it completely finished, overwrote the original rom.nds, and closed. Check the compiler_report.txt and see 323/323 succeeded and none failed. (Out of curiosity I also tried removing the original ROM from the directory and having it create a fresh rom.nds, same result)

7. Opened up the brand new rom.nds in Desmume 0.9.11 (optionally renaming to its default name so I could quickly reuse a save state to test problems instead of having to sit through the entire beginning. I have also tried this with an exported .sav file made with an original untouched ROM. I've also done this by starting a brand new game and the very first item I run into in Beach Cave after the intro is a 2 Poke item. Editing in a broken TM via SkyEditor or SkyJEditor also produces the same broken TM.)

8. Encountered said Plain Seed bugs on the aforementioned items, what it looks like in game below. Kangaskhan acknowledges it as a Plain Seed, it can be given to my team to hold, but not use in town like a proper TM. And can be eaten in a dungeon to produce the text "But it does nothing..." just as a normal Plain Seed does. However it is in the same exact spot in storage that the original item had and as you can see still maintains some descriptive text.

Spoiler

9. As for the second problem regarding the 2 Poke drops, after more research and testing in various dungeons I've found that it's actually not just replacing seeds, it's entirely scrambling the item pool. Normal dungeons just have otherwise normal drops except for the completely intermittent and random Plain Seed/2Poke item as seen below replacing some other item that was supposed to go there. I have also yet to run into a single normal Poke loot pile, I only ever see other normal drops (such as apples, berries, sticks, normal seeds, gummis, and the random Plain Seed/Poke replacements.

Spoiler

When I tried this out in SOS dungeons however things got completely messed up. Rescue dungeons that I 100% knew the layout of from beginning to end in terms of item drops and locations on each floor were now entirely different items altogether. The locations were the exact same, floor layouts the exact same, but the item in that spot was entirely different. Most were replaced with the 2 Poke item, others replaced with relatively benign items like gravelrocks or other seeds. Proteins turned into 2 poke items, poke drops were entirely replaced with gravelrocks, 2 poke items, or functioning seeds, Sitrus berries became geo pebbles ... Here you can see the left two images showing a perfectly functioning ROM I made by recompiling without item_data showing two Poke Piles of that exact quantity given every run (also does this same thing on an unmodified ROM). On the right two images are what those turned into on a recompiled ROM including item_data and they've transformed into a Violent Seed and a 2 Poke/Plain. Every single item in this rescue has been morphed into something else, nothing is its original item.

Spoiler

As a final thing to try, I tried doing the process over again from scratch with an already modded/rebuilt ROM (I always did it from a fresh ROM since that seemed to be safest option) and i found out something that may be the cause of the issue. I notice on extracting and decompiling a fresh ROM there are exactly 1,400 entries in the item_data folder that ends on 1399__M_D1____.xml. When I did this process with my properly functioning ROM I also came to the exact result of 1,400 entires. However when I extracted and decompiled a broken ROM, I discovered item_data had LOST an entry and went down to 1,399 total. The missing file is in fact 1399__M_D1____.xml which upon opening up has an almost identical long description of Used TM, which is very peculiar since every single other M_D1 entry only has the long description of [M:D1]$$$. I can't help but feel these are correlated in some manner.

If you'd like to try that exact rescue dungeon the SOS code is:

Spoiler

NN#QQ5 8=3S%P +@HY#4
NNJ&6H N+=MWN RJ74#=  
PC0H8P 2YS9XX N2%2C6

Hope this was enough information for you to sort it out, cause I can't think of anything more to add!

Edited December 14, 2017 by Zerea

[psy_commando]

Ok, first, never use save states if you're modding the game. Save states are just literally memory dumps of the nds's ram into a file, and since the files inside the rom have changed and aren't at the same place anymore, it will end up being really, really messed up. Regular in-game saves are fine though. Also, its probably not a good idea to post the entire roms here(I downloaded it already,  so you probably should remove the link). And, also you really shouldn't need to run any of my tools as administrator. In general, don't run anything as administrator if you can.

But yeah, it seems like you have the latest version of statsutil, so that's good. I'll try to run some more experiments! Thanks for the details!

Edited December 14, 2017 by psy_commando

[Zerea]

Thanks for the advice. I've yet to experience any strange behavior on my working ROM with rather extensive save state usage, but I also regularly use normal saving as well. I wanted to be thorough in how I was testing the compiled ROM so that's why I did various methods of starting/loading the game from as many angles as I could think of to see if at least one of them worked.

And thanks for the tip on removing my link, I didn't even think when I was including my base ROM what I was actually posting onto the forums, I just wanted to include it in case that there was something fundamentally wrong with it when I went to extract the contents, oops!

Glad I could help. I'm not sure what else I could possibly do to identify the problem but if you think of something please let me know.

[Zerea]

How difficult is it to go through the game's .bin files manually? Is there something available I could use or would a tool have to be written?

Also does anybody have a download to the items_p_Sky.xlsx sheet? I keep seeing it mentioned in technical documentation and the website hosting it no longer exists. Thanks evandixon!

Edit - So taking my time reading through this entire thread I can see it's not a simple task of analyzing the .bin and I'm definitely way over my head. Was hoping I could do some poking around and see what results I end up with but I think I lack the basic understanding required to do so manually.

Edited December 18, 2017 by Zerea

[evandixon]

7 hours ago, Zerea said:

How difficult is it to go through the game's .bin files manually? Is there something available I could use or would a tool have to be written?

Also does anybody have a download to the items_p_Sky.xlsx sheet? I keep seeing it mentioned in technical documentation and the website hosting it no longer exists

I uploaded my copy to the tech doc.

[Zerea]

Thanks a ton! It was killing me constantly seeing it referenced and not being able to find a DL.

[Zerea]

Decided to try poking around the bin files with a hex editor and using the spreadsheet and technical documentation I haven't found it too difficult to navigate. I've managed to find a workaround for the item_data problem I'm having up above by simply doing small edits to the files directly and bypassing the compiler rebuilding item_data for me, which only strengthens my suspicions that something goes haywire when it attempts to write over item_p.

By digging through item_s_p I've also learned a lot about how the game handles Exclusive items, but I'm also still stumped by quite a lot. My main problem is I can't figure out how the game knows that a particular string of 4-bytes within item_s_p associates with one unique item that I'm guessing is defined in item_p? The data structure within the s_p data table doesn't appear to contain any unique ties to a particular item entry, yet somehow the game knows exactly what item those 4-bytes need to go to. I found this out by changing the 3rd and 4th bytes of Ledyba's item to another pokemon and seeing the effects of the item transfer over to the new pokemon, instead of it producing the effects of that equivalent item onto Ledyba. Changing the exclusive item_ids within the pokemon_data seemed to do absolutely nothing in terms of producing effects as only changing the 3rd and 4th bytes or changing the BasePokemonIndex were successful. My only conclusion from this is that the exact location within the table is what correlates to the item in question?

Maybe one of you experts can shed some light on this for me. Essentially I'm trying to figure out if I were to hypothetically create duplicate pokemon/item/exclusive item entries using the reserve spots at the end of the table in each file, how would I create a valid association between everything? After hours pouring over the documentation and the excel sheet I still have no clue where the item_s_p/item_p/pokemon chain comes from and I'm kind of just going around in circles.

Edited December 18, 2017 by Zerea

[psy_commando]

Well, like most data stored in SIR0 files, its all meant to be loaded as an array in memory. I haven't yet looked into how the indice for special items is obtained though.

But technically, if the item array size isn't hardcoded, you should be able to just refer to the new item ids in the dungeon and shop data spawn lists. But I have yet to make a tool for that, because a bunch of those are hardcoded in the game executable. I've managed to mod the game's binary in the past to load a variable size map list with some asm edits, so I figure I should be able to mod the game to put those lists into files as well. But I'm working on other stuff for now.

[Zerea]

Well thank you for finally clearing that up and confirming my suspicions. I can now rest easy knowing the solution is pretty much out of my capabilities cause messing around with the binaries is going to be way more difficult than just downloading a hex editor and using other people's great documentation. I'll leave it to the experts when they get around to it.

In the meantime I've diverted my attention towards playing with the Unks inside waza trying to find anything that sticks. Right now I'm trying to give a thorough testing to Unk12 and in that regard I found this snippet of info on Ultimate Pokemon Center. As a caveat, the author did specify this was for Explorers of Time/Darkness and it's possible Explorers of Sky utilizes something different.

Critical Hits

Attacks can’t be a critical hit if the target has the Lucky Chant status, it has Battle Armor or Shell Armor, or the skill Critical Dodger is enabled for the target.

    A variable X is set to the move’s initial critical hit rate. X is then halved if the attacker is male or gender-unknown.
    If the attacker has the Focus Energy status, X is set to 999.
    If the attacker is holding Scope Lens, or the attacker’s Sharpshooter is enabled, X rises by 15.
    If the attacker has Super Luck, X rises by 15.
    If the target is holding Patsy Band, X rises by 15.
    If the attack is "super effective" and the attacker’s Type-Advantage Master is enabled, X is set to 40.
    If a random integer from 0 through 99 is less than X and Teary Cape doesn’t affect the target, the attack is a critical hit and the Damage is doubled if the attacker has Sniper, or multiplied by 1.5 otherwise.

I was curious has to how this info was obtained, but the author's linked github shows numerous different programming projects/tools/utilities and I do recall his name popping up in that huge GameFAQs thread so I don't think it's a stretch to assume some amount of effort and data mining was done. I wanted to see how accurate this was, in particular the line

X is then halved if the attacker is male or gender-unknown.

which just sounds absolutely ridiculous to me that female has two times higher base critical chance than males or unknowns so I set out to test and here were my rather quick results. Forgive the appearance, excel formatting is NOT my strong suit, also SS =  The IQ Skill Sharpshooter. No other crit modifiers were included.

Early conclusions to draw:

• Unk12 is without a doubt directly related to the critical chance of an ability.
• Either there is another component to the base critical chance of each ability, or the entire formula is wrong. Unk12 = ~60 with absolutely no other modifiers is 100% Crit
• X/2 if Male or Unknown appears to be completely false, as in fact the data favored males, but much too low sample size to say anything more.
• Sharpshooter being X+15 seems to be fairly accurate.

That's all I have for tonight, I'd definitely love to refine the sample sizes and include gender unknown data as well, but I don't want to go overboard because my goal isn't to perfectly fine tune the crit formula, it's only to find out Unk12's role and how it relates to criticals.

Also, psy_commando I hope my posts asking for help or posting my findings aren't derailing this thread's original purpose as your research repository. If they're disruptive to your work I'll be happy to remove them.

Edited December 19, 2017 by Zerea

[psy_commando]

Thanks for the data! I was pretty sure it had to do with critical chance, but I wasn't sure if it was a ratio or something else!

And you're not derailing anything, this thread was meant to share research results and etc, so you're welcome to post any findings you wanna share! ^^

 

[Zerea]

Spent most of today cataloguing waza_p_bin into a new sheet on the item_p_sky spreadsheet. Got inspired by looking at the format of the item_p and also wanted an easy way to compare values across all the moves. Here's the updated document if anybody wishes to peruse the move_data in Excel format https://drive.google.com/open?id=1FpIibBFuRCq8fiXPQsIbDo1G0qATMHzd. Feel free to do as you wish with the sheet, it's mostly for my own testing and cataloguing purposes.

From that work today I've more or less completely figured out Unk4&Unk5/Bitfield 1 except for a few wonky exceptions. Bitfield 1 determines which entities in a dungeon a move interacts with, or how said move can interact with these entities. Here's the complete list of all values of the bitfield, and the most accurate description I can come up with as well as examples of each.

0	Melee Range (Spark, Shadow Claw, Fury Cutter, Disable, Encore...)
2	Facing Adjacent Pokemon (Psych Up, Substitute, Thrash?)
16	3-Front Facing Tiles (Wide Slash)
32	1-Tile Radius (Dark Pulse, Thunderbolt, Stun Spore...)
48	All Enemies in Room (Discharge, Spacial Rend, Heat Wave...)
49	All Allies in Room (Agility, Aromatherapy, Refresh, Accupressure...)
50	Everything in Room (Defog, Baton Pass)
53	Everything in Room but User (Earthquake, Magnitude)
54	All Allies but User (Helping Hand)
64	2-Tile Length (Quick Attack, Vacuum Wave, Bullet Punch...)
80	Movement Altering (Stops or changes how movement occurs, such as through Tactics or Cowering)
82	Line Attack (Signal Beam, Dragonbreath, Hydro Pump...)
96	All Enemies on Floor (Perish Song)
97	All Allies on Floor (Morning Sun)
98	Everything on Floor (Trick Room)
115	Self/Weather (Double Team, Calm Mind, Reflect, Rain Dance, Sandstorm...)
116	2-Turn Attack (Sky Attack, Dig, Fly, Bounce, Charge...)
128	Cuts Corners (Sand-Attack, Smokescreen, Water Gun, Poison Sting...)
133	Facing Pokemon & Cuts Corners (Guard Swap, Power Swap, Gastro Acid)
144	2-Tile Length (Ice Shard?)
255	Facing Pokemon (Only for Me First)
561	All Allies in room HP Recovery (Softboiled)
609	All Allies on floor HP Recovery (Moonlight, Milk Drink)
627	User HP Recovery (Synthesis, Heal Order, Roost)
1334	All Allies in Room but User Full HP/Status Recovery (Healing Wish, Lunar Dance)

Nearly all of them are very straightforward and intuitive, however there are some notable out-of-place moves that clearly belong with another group which can only make me assume that the answer lies within Bitfield 2 or somewhere else. Thrash is its own case among the three others in group 2 and honestly should be in its own group. Why Ice Shard is specifically in its own group all alone when it has identical function to group 64 is beyond me. The final four groups having to do with recovery are rather inconsistent and confusing, and this is where you can really see that there's far more than determines a move's function beyond this Bitfield. Recover, even though it is self-healing is all the way down in group 115, when it could clearly be placed in 627. Healing Wish and Lunar Dance in particular have me completely bamboozled as they are literal copies of each other besides Base PP and Move ID, and yet Lunar Dance has the extra effect of restoring PP as well.

This once again reminds me of my problem I had with exclusive items, as how can the game differentiate the effects of these two moves when every single variable that is contained in waza_p is identical (besides PP/move ID)? Can't help but now wonder if there's also some hardcoded stuff relating to the effects of moves lying around in the binaries as well. Perhaps tomorrow when I have more energy I'll change Healing Wish's move ID to Lunar Dance's and see if that doesn't lead somewhere. I'd like to do this for many combinations to see if all of these groups are fully transitive. I know you can make group 48 easily hit the whole floor by shifting them to group 96, but could you do the same for group 0?

Edited December 20, 2017 by Zerea
Fixed erroneous values on spreadsheet

[psy_commando]

Well, my assumption had been that those were bitfield,  aka the game would check for the value of a specific bit to toggle something on or off, but it could also be an enumeration.. I used to modify those bitfield in memory while running the game, and it seemed to me that unk#4-5 values both contributed to the resulting effect of the move. Like unk5 seemed to affect ranged move particularities..

I took some notes from your analysis, but I think it might need some testing to see how it works in practice?

[Zerea]

Ah okay so that's what it means by bitfield, to be honest I just saw it as a naming mechanism and had no associated description of what it meant in my head. If that's the case then I'm inclined to agree with your initial assumption of it being a bitfield.Here's my further research for today::

Some moves behave perfectly fine by only changing Bitfield 1. Melee moves can be made to act like 1 tile radius, 2 Range moves, line moves, room wide, floor wide, allies only, everything, and generally speaking the reverse is true for moves starting in those groups as well. Room Wides can be made to act like melee only and so on. With only one exception so far, every attack move combination I've tried has always turned into Unk 4's associated group and seemingly ignores any Unk 5 value. A 2-Length/Line Attack combination is always a 2-Length attack, a line attack/floor wide is always a line attack, etc..

However a couple I've tested so far do not cleanly translate by only editing Bitfield 1 (or even Bitfield2).

• 2-Turn charging attacks - Changing only one or both bitfields to 116 simply results in self damage and nothing more, which means the other Unks clearly affect these moves (or possibly more). I've yet to do the reverse of changing the two-turn into something else.
• Changing a room wide to a 64/82 (1-Tile Radius/Line Attack) has so far been the only example of the game not utilizing Unk 4 as the core identity of a move. Two very noticeable things have happened.
  1. The game believes this move is now actually a line attack instead of a 1-Tile Radius, as any direction not facing the target results in a failed move. Bitfield 2 has superceded 1 in priority.
  2. The accuracy calculations have become beyond mangled by some unknown variable. Setting accuracy to 100/125/150/200 and even its absolute maximum of 255 still results in a <5% hit chance. This is not the move failing to register a valid target by saying "The move failed.", this is the game repeatedly returning, "The move missed!" This is interesting to note because 125 Accuracy moves are supposed to be denoted by the game as guaranteed to hit, such as Aerial Ace and Swift. So clearly another Unk works as a backup/addition to the Accuracy. On this topic I've noticed in damage calculation GameFAQs posts they refer to a value known as Accuracy 2 but not into great detail and only that it's used in niche cases.

So I did the next logical step and started looking at nearby Unk values and noticed Unk7 to generally have default values that very closely mimic typical Accuracy ranges, such as 88, 95, and 100. However as I was doing this I also thought I caught an interesting trend where Unk6 has something to do with chance of secondary effects on moves activating and got distracted. Turns out this doesn't appear to be the case, as nearly every move has a non-zero Unk6 value and many completely unrelated moves share identical values. It appeared to increase the frequency of these side effects at least somewhat, maybe up to ~20-30% at maximum value if I had to estimate but it's hard to say. It's also somehow tied in with accuracy as low values of Unk6 result in nearly no hit chance. Finally, I noticed a very bizarre pattern in that every attack that was "Bite-based", as in it was designed around the concept of biting with teeth (Bite, Crunch, Hyper Fang, Thunder Fang etc..), all had completely identical Unk6 values. considering how erratic and nonsensical Unk6's values seem to be, it's hard to just write this off as mere coincidence. However I also don't see how associating each "bite" is productive in any manner. To my knowledge there's absolutely no game mechanic that is exclusive to bites only. I've been meaning to investigate other "attack families" such as perhaps the punches or kicks, but...

  ..yet again I was distracted by an even more interesting trend i noticed, one that actually has concreted into something coherent. Unk 18 is at least a secondary function for how a move operates, and further specifies the usage of a move. What's more to this, its neighbor Unk17 acts as an on/off switch (Is this a bitfield?) for the value set by Unk18 . Specifically, Unk17's value determines whether or not the next byte is read as a damage causing attack, or an attack that is not supposed to damage. Here's a table with all of my findings thus far, the values along the left hand side are the possible values of Unk18:

1	Orb Affecting User/Environment				1	Enemy Damage Dealing 3-Front Facing Tiles
2	1-Tile Radius Enemy Non Damaging			2	Enemy Damage Dealing 1-Tile Radius
3	Cuts Corners Enemy Non Damaging				3	Enemy Damage Dealing Cuts Corners
4	Team Summon/HP Recovery					9	Enemy Damage Dealing Room Wide
5	All Allies on floor HP Recovery				10	Counterattack During Sleep
6	Everything on Floor					11	Enemy Damage Dealing Melee Range
7	All Enemies on Floor					12	Enemy Damage Dealing Three Random Directions
10	Variable/Random Move					13	Enemy Damage Dealing 2-Tile Length
11	Melee Range Non Damaging				14	Enemy Damage Dealing Line Attack
14	Line Related Debuff					15	Enemy Damage Dealing Room Wide
15	All Enemies in Room Non Damaging			18	Everything in Room but User Damage Dealing
16	All Allies in Room Buff/Heal				19	Duplicate entries..?
17	Entire Room Non Damaging				20	2-Turn Charging Status Changing Damage Dealing Attack
19	User Buff/Static HP Recovery				26	1-Tile Radius Everything/Walls Damage Dealing Attack
20	Two-Turn Non Damaging Buff				27	2-Tile Radius Everything/Walls Damage Dealing Attack
21	Dynamic User HP Recovery				28	2-Turn Charging Damage Dealing Attack
22	All Allies but User Buff		
23	User, Restores Used TMs		
24	Changes Floor/Weather/Ground Tile Status		
25	Breaks down Walls		
29	Floor wide Floor alteration		
30	Non Damaging Move Using Facing Pokemon		
31	Non Damaging Move Using Facing Pokemon & Cuts Corners		

Formatting this was not enjoyable so I split the example moves down below, my deepest apologies for the eyesore.

Spoiler

IF    Unk17= 0                                                                                        IF    Unk17 = 1
1  Trawl Orb, See-Trap, Trapbuster                                                        1    Wide Slash
2  Stun Spore, Lovely Kiss                                                                      2    Dark Pulse, Thunderbolt
3  Leech Seed, Metal Sound, Hypnosis                                                  3     Fire Blast, Dragon Rage
4  Beat Up, Morning Sun(?)                                                                    9     Vacuum Cut(?)
5  Moonlight, Milk Drink                                                                          10    Snore
6  Trick Room                                                                                         11    Metal Claw, Tackle, Sky Uppercut
7  Perish Song                                                                                       12    Thrash
10  Metronome, Follow Me, Sleep Talk                                                   13    Extremespeed, Quick Attack, Bullet Punch
11  Confuse Ray, Disable, Foresight                                                      14    Flamethrower, Dragonbreath, Signal Beam
14  String Shot, Blowback(the effect from Roar)                                    15    Heat Wave, Spacial Rend, Discharge, Blizzard
15  Defog, Dark Void, Heal Block, Growl                                                18    Earthquake, Magnitude
16  Agility, Heal Bell, Safeguard                                                             19    Duplicate entries..? (Avalanche, Reversal)
17  Baton Pass, Haze, Teeter Dance                                                     20    Fly, Bounce, Dive, Dig, Shadow Force)
19  Dragon Dance, Recover, Avalanche(?), Endure                              26    Selfdestruct
20  Charge                                                                                             27    Explosion
21  Synthesis, Swallow                                                                           28    Sky Attack, Solarbeam
22  Helping Hand   
23  Recycle  
24  Rain Dance, Spikes, Stealth Rock, Hail   
25  Rock Smash          
29  Drought Orb, Fill-in Orb, One-Room Orb     
30  Psych Up, Psycho Shift, Substitute         
31  Guard Swap, Heart Swap, Gastro Acid

Discovering this has clarified a ton of confusion I had regarding move identity and it feels like a lot more moves have "proper homes" now, but there still remains a few notable outliers that have yet to be resolved

1. The Lunar Dance/Healing Wish issue hasn't been resolved and I still don't see any conceivable way the game differentiates the two abilities. Even changing the Move ID itself resulted in nothing. I have a modded waza_p binary with two literally identical skills from Byte 1 to Byte 26, they're even skills that are near perfect copies of each other in terms of effects to begin with, and yet in game they produce different effects. :shrug: Only thing I have left to try is to swap their bytes and see if that works.
2. Morning Sun appears to be completely lost in every single one of his groups. It was out of place in Bitfield 1/2 and remains out of place in Unk 18. It is a carbon copy of Moonlight, as it's weather-variable HP recovery for every team member on the floor, yet sits in Group 4 where only Beat Up and the Orb equivalent Rollcall Orb sits. To be honest Milk Drink is the odd one out, as Morning Sun and Moonlight are exact copies of each other in terms variable floor wide wide healing whereas Milk Drink is only static floor wide healing. But there's other floor wide heals in the game and no designation for "Floor Wide Static HP Recovery" so he'd be left somewhere weird too, they just decided to have Morning Sun be the one left out I suppose. I'm not sure if I'll ever find anything to fully resolve this.
3. For some inexplicable reason Bide, Avalanche, and Revenge were all coded as two attacks rolled into one. The first attack applies the "charging by absorbing damage" status effect, the second attack does the release. Each of Bide's attacks is associated with a proper group,... yet Avalanche and Revenge both have their damage dealing attack parts sorted off into their own group when they do the exact same thing as Bide! I got nothing...
4. The move entries for Orb/Seed effects are an absolute trainwreck, as well as the Unk17/18 values for them having little to no coherency between themselves, let alone the rest of the moves. The naming for some of them gives absolutely no indication of its relevant use-item or effect source.
   • Famish - It's not Hunger Seed
   • Observer - Not Luminous Orb
   • Possess - Literally have no clue
   • Searchlight - Not Luminous Orb either
   • Siesta - ???????????????????????

          All the others were actually properly named and so I could easily find an item-effect association, even if they ultimately had an Unk18 value that   made zero-sense. However the ones above i simply removed from my folder and compiled without to see if anything had changed. I tried out a couple "Maybe these?" Orbs/Seeds to see if any not associated effects were broken, and with the exception of Warp Orb, Mug Orb, Quick Orb/All-Mach Orb, all others remained fully functional. That's right, a Hunger Seed apparently knew exactly how to cause Famine with no actual move entry for Famine to begin with. Mug Orb stopped stealing an item and did nothing but hit my pokemon for a significant chunk of damage . And Quick Orb/All-Mach had the rather amusing effect of attempting to work on enemies instead of myself and my team. The animation and text appeared for the enemy pokemon's movement speed increase, but no actual increase was made. Warp Orb just flat out stopped working.

And that's about it for tonight. I'll probably look more into Unk6/7 tomorrow or just see if solving Unk 17/18 has opened up any new patterns to spot. I don't really want to spend a lot of time investigating orbs because they're very not enjoyable to experiment with. Instead of being useless when I horribly mangle what should be their core data files, 90% of them work perfectly fine. That's like ripping the engine out of a car and it somehow keeps running, but one of the windows no longer rolls down and you have to occasionally slap your dashboard to get your radio to work. How am I supposed to solve that? =/

I've also updated my spreadsheet here with the Unk 17/18 Data if anybody wishes to deal with the large amount of data in a more manipulable format.

Edited December 23, 2017 by Zerea
Crossed out mis-information. Unk17 is related to Taunt and Unk 18 is of unknown application. See below posts for more info.

[Zerea]

Removing 403_Itemize*(turns out removing any file below Itemize in the structure also does this.) led to this very amusing result. Haven't laughed this hard in a long time.

Apparently this is also doable by using it on an enemy with Mirror Move active. I absolutely love that the developers created a text string for this exact scenario.

Edited December 23, 2017 by Zerea

[Zerea]

Sorry for these huge posts but I've been on quite the journey of many discoveries lately, hope they're not too much of a pain to read through!

I came to quite a fun discovery tonight by messing around yet again with the mysterious move_data for the inconsistent Orbs and Seeds and found that seemingly regardless of which data file I deleted, the large majority of the orbs reacted the same way each time. Either they appeared to have no ill effects, had some small quirk to them now, or just flat out broke entirely and did nothing. I also noticed that deleting any file also messed with the way certain moves were displayed in game on both the Move List menu as well as the Message Log inside a dungeon, and also modified their behavior. Why were my original moves replaced with Fire Fang and only hit Melee range? Why was Mist now dropping a Thunderbolt on my head? Why was Earthquake now healing every single thing in the entire room but me..? More importantly.. why wasn't every move broken in this manner? Brick Break would work perfectly fine, Outrage, Cross Chop all fully functional. Ideas started coming together and several quick lookups resulted in these findings:

Moves that had no problems, to name a few.

Cross Chop

Outrage

Sharpen

Aromatherapy

 

Moves that were broken.

Thunderbolt - Is now named Mist and drops a Thunderbolt on me

Recover - Is now named Earthquake and heals everything in room besides me

Agility - Does nothing at all

Discharge - Is now named Fire Fang and only hits one enemy in Melee range

Dark Pulse - Is now named Ominous Wind and hits entire room

Shockwave - Is now named Quick Attack and has a 2-Tile Range now

Notice any correlation? No? How about if I put a little more information down and re-order it?

Spoiler

15 - Aromatherapy, All Allies in Room
55 - Sharpen, Self/User Move
90 - Cross Chop, Melee
91 - Outrage, Melee
95 - Agility, Does nothing
117 - Recover
118 - Earthquake, Damages everyone but User
129 - Thunderbolt
130 - Mist, Self/User Move
189 - Shockwave
190 - Quick Attack, 2-Tile Length Attack
436 - Dark Pulse
437 - Ominous Wind, Room Wide Attack
521 - Discharge
522 - Fire Fang, One Enemy in Melee Range

Bingo. The broken moves were copying the name and attributes of their next-door neighbor but still keeping their original animations and skill effects. Suddenly Discharge had the range of Fire Fang, Thunderbolt was using the target parameters of Mist (Self), and hilariously, Recover was using the targets of Earthquake and healing the entire room! But why were all these moves borrowing their neighbor's attributes and not Outrage, or Aromatherapy? And why did Agility just do nothing at all and waste my turn? Well if you noticed another pattern you might have already noticed that every skill under Agility started to become a copycat, and every skill above it seed to function perfectly fine. And that's because the file I deleted to cause these results was in fact 095_Agility! Deleting Agility had removed all 26 Bytes of data stored in waza_p and shifted every single move's bytes upwards 26 spots to fill in the gap. To use an analogy,  imagine if a bunch of people were standing in line to get into a building and one person had suddenly lost all of their clothes and was completely naked. The person right behind them feels horrible and so offers their clothes to the person in front, but now suddenly that person is naked as well.. and the process continues all the way to the end of the line.

This revelation was quite an amazing feeling for me as someone who's not been doing ROM exploring/editing for more than a week and I now know quite a lot more of how a move functions inside the game, even if I don't fully understand all the specific details quite yet.

I'm gonna attempt to try and explain what I think the main process is and I apologize in advance if I use words incorrectly or butcher their meaning, as understanding ROM hacking, coding, programming, is still a very new concept to me. If I put a (?) next to a word/phrase it's because I'm not sure if I've used it correctly.

1. The core identity of each move belongs to an array(?) inside waza_p. This includes the actual "meat" of what a move does: Does it do damage? Does it buff a stat, then which stat, by how much? Does it debuff a stat, which stat, how much? etc.., it includes what animation to play, what sound effect to play, and perhaps a couple others I can't think of at the moment. I picture this as the "framework" of a move, it's the core structure of the house(move) and ultimately sets the boundaries of what you can and can't do. There's also a pointer(?) to at least some scripts related to each move, particularly ones tied to TMs.
2. 26 Bytes are placed on around and top of each move's "framework" and this is what fills in the building. Things like the number of each building (Move ID), how many and what type of people its meant to house (Bitfield 1/Unk 17/Unk18), what color the house is (Move type).. you get the idea.

So the identity of a move is written into this array(?) and loaded in a very specific order(?) and the game is basically saying, "Give me all the secondary characteristics of the move Iron Tail, I already know that Iron Tail is first in line and I need to play this animation and this sound with it along with this effect..., but I need to know how much damage it does, what type it is, who it will be used on ..." To carry on the above analogy of people lining up to get into a building, it's kind of like a bouncer verifying what each guest is wearing into the building so he knows where to direct them. He already knows each spot in line is specifically numbered and named to a guest and has a rough idea of the shape and appearance of each guest, he just isn't sure if they're going to be wearing a red or blue hat,  big or small glasses, have long/short hair, blue/green eyes, and so on. No matter what combination of clothes or modifications you try to pretend to be another person in line and fool the bouncer, the bouncer sees through it all and says "Wow you're wearing the same exact outfit as Iron Tail! But you have the exact height and weight of Fire Blast as well as standing in spot 157 and not spot 001. so I know you're actually Fire Blast"

Silly analogy aside, what does all of this mean? Well it tells us what is easy to change and what isn't easy to change with statsutil or just changing values with a hex editor.

What you CAN change with statsutil/modifying Hex values:

• Base Power
• Move Typing
• Range of Attack (melee only, in a line, whole room, whole floor..)
• Target(s) of Attack (Enemies only, allies only, user only, all three..)
• Accuracy
• Base PP
• Critical Chance
• And a couple more as of yet undefined variables, these might just be further specifiers for the above categories however.

What you CANNOT change with statsutil/modifying Hex values:

• Whether this move does damage, no damage, or heals
• Whether this move increases or decreases a stat and/or has a secondary effect
• How much the move increases/decreases a stat
• The chance of the move increasing/decreasing a stat or activating its secondary effect
• A move's effectiveness versus another type
• The animation and graphics associated with a move
• The sound effects that play when a move is used
• And perhaps a couple others that are evading me at the moment or I haven't thought of.

However there is one thing that I believe should be possible. If you fill in the deleted file's space with blank zeros, it should maintain the integrity of the remaining move entries and prevent this "shifting of the line". It will of course mean that move has absolutely no data associated with it and will remain functionless, but the rest of the moves should still function properly as the integrity of the line has not been broken.

I hope I didn't utterly mangle that explanation or miss the mark completely with my line/bouncer analogy. Some of this information might come as dead obvious to you more experienced modders out there or you strongly suspected it already. Storing data in arrays is extremely common and this is definitely not the only one in the game.This was just an "Ah Ha!" moment for me as someone brand new to ROM exploring/modifying with zero game design or programming expertise and I wanted to share my excitement.

I'm sad that it won't be as easy as I would've liked to swap moves around or modify their core attributes, but at least now I figured out why Healing Wish and Lunar Dance can have identical 26 bytes and still produce differing effects!

Perhaps I should take a break and just play the game for awhile now. I don't want to overwhelm you with these huge posts back-to-back all the time, especially if they're distracting you from your current project.

Edited December 22, 2017 by Zerea

[Frostbyte0x70]

So I have been following this thread for a while and I have finally decided to register here and post something.
First of all, thanks to Zerea for the move data spreadsheet, I've been trying to figure out the meaning of the unknown values for some time, and the data has been really useful to me.
Now I will share my findings:

Bitfield 2 Controls the way enemies use moves. It works the same way as bitfield 1: If a move has a value of 48, enemies will use it if there's a party member in the same room.
You can try and give a value of 48 to some move like Tackle, enemies will try to hit you from far away (and obviously fail)
For some reason certain moves have different values for bitfield 1 and 2 when they should be the same, for example Discharge has Bitfield 1 = 48 and Bitfield 2 = 64. Although it is a move which hits all enemies in the room, other pokemon will only try to attack you with it if you are up to 2 tiles away from them.

Enemies (and I guess allies too) use the Unk6 value of a move as some sort of "weight". The larger this number is, the greater is the chance of a pokemon using it. A large value here means enemies will use the move constantly, and setting this to 0 causes pokemon not to use that specific move at all.

Unk7 is that other accuracy value you were talking about. If the move misses due to this second accuracy, a different message appears (I think it's 'the move missed X' instead of 'the move failed to impact X')

Unk10 is the number of times a move hits. 0 means it hits a random amount of times.

Unk13 = 1 means the move is affected by Magic Coat

Unk14 = 1 means the move is affected by Snatch

If a move has an Unk15 value of 1, it can't be used under the muzzled status (caused by a silence orb). That's why biting moves all have Unk15 = 1

Unk17 = 1 means the move can be used while taunted

Unk19 controls the message that comes up when the move is used. A value of 0 means default message, 1 would be the next string in the game ('Oh no! (Pokemon) chose (move)!') 2 = Regular attack message, and so on

Orb-effect moves are easy to test, I just make a quicksave in a dungeon, edit the save file and add those moves to the team leader or the enemies:

Famish: Decerases by 10 points the belly of all enemies in the room. Enemies/party members will act the same way as if they had eaten a hunger seed once they get affected 10 times by this move.
Observer: Foe-Seal orb.
Possess: Probably has something to do with the 'Possess Orb' from Mystery Dungeon 1. Using the move does nothing.
Searchlight: Has the same effect as the luminous orb.
Siesta: Same effect as the slumber orb.

 

I hope that helped. Sorry if I made any grammar mistakes, English is not my first language.

 

PS: @psy_commando, the sprite editor crashes for me every time I try to import/export sprite palletes (no matter if I save/load it as .txt or .pal). I don't have a github account to open an issue so I post it here.

Edited December 22, 2017 by End45
Added Unk13, Unk14 & Unk17

[Zerea]

Never actually expected that spreadsheet to be useful, glad somebody did something good with it!

And wow these are some incredible finds. To be honest many of these were on my to-do list and I had already gotten started on them but you beat me to the punch!

I think I perfectly understand BItfield 2 now thanks to your info, and it perfectly ties in to something I read on a guide relating to controlling your team members.

"It's important to know that an ally may not use certain
moves even when you want that move to be used. Like Blizzard
when an enemy is in sight or Defense Curl whenever you want.
This is because the ally has a separate range they think the
move has when it's actually something else.

Defense Curl, even though it affects only the user, is only
used by an ally if there's an enemy already standing next
to this ally. The ally thinks Defense Curl has a range of
one tile away, and waits for an enemy to come before it will
ever use the move."

It now makes a lot sense looking at some these moves and relating them how I experience them being used by my allies and enemy pokemon. Now it makes sense why it felt like Smeargle was being really inconsistent with when he decided to Helping Hand but used Haze immediately. This was their way of stopping useless move usage like enemies spamming buffs when you weren't anywhere near them. For all intents and purposes your team member thinks Defense Curl is a Melee Range Attack that just happens to only hit itself for 0 damage and buff its Defense by 1 stage, which is really funny to think about! Too bad I didn't think back to this guide or I would've caught this connection to Bitfield 2 sooner, but good thing you were doing your own research anyway!

That's neat that the supposed secondary accuracy was Unk7, I figured it probably was but I got distracted by Unk 17 and 18 before I could put together tests. I'm still not sure how they interact together though other than they are probably multiplied together in some capacity to get your true accuracy. You can test this by setting either value to 255 and the other to 0 and the move will never hit. Which makes me wonder if Unk7 is actually Accuracy 1 and Unk8 is Accuracy 2.

Unk15 however is amazing to me. I thought I noticed the "bite" trend in Unk6 values but it turns out those were all just sheer coincidence, but there actually was a value for this after all! This was also on my to-do list because looking through a list of damage flags on another GameFAQs I saw one for being muzzled so this was something I wanted to explore in the near future. I'm surprised at how deep they went with what is affected by the muzzled status, as it is quite literally ANY move that utilizes the mouth in ANY way. Drinking, Biting, Making a noise with your mouth are all blocked. In fact the only out of place one seems to be Absorb, which the developers must think is done via.. biting..? But yet Mega/Giga Drain aren't...

And I can't believe I didn't fully look at the movelist within the save editor for Orbs! I never even THOUGHT they would be included and that I could manually give a pokemon them mid quicksave. I kept trying to use different orbs and finding out which ones got screwed up! But I'm glad to see all the ones stumping me are the ones that ended up having absolutely STUPID assocations. In what world does Siesta = Sleeping?!?! Observer = Petrifying the whole room?? Famish isn't used anywhere in the game and Possess flat out is leftover data from PMD1!

As I was writing this post I thought back to those other damage flags on that guide and a lightbulb exploded in my head and I quickly figured out Unks #11,13,14, and 16!

Unk11 - Move is affected/not affected by Taunt actually Unk 17 instead

Unk13 - Move is affected/not affected by Magic Coat

Unk14 - Move is affected/not affected by Snatch

Unk16 - Move can affect/not affect Frozen targets  kind of but not really...

Unk18 - No longer makes any sense without Unk 17, appears to do absolutely nothing to change a move's grouping.

 

Edited December 22, 2017 by Zerea
Got too ahead of myself..

[Frostbyte0x70]

The one that controls taunted status is Unk17, I've tested it.

And i'm not sure about Unk16, it totally looks like it determines if a move can be used on a frozen target but I have also tested it and it's not that, you can't use tri attack on a frozen pokemon. Also Sacred fire is the only damage-dealing fire type move that has Unk16 = 1, and it affects frozen targets anyway.

Edit: Unk11 is the max amount of times you can power up a move using Ginseng

Edited December 22, 2017 by End45

[Zerea]

Hmm now this has really spiced things up. Unk17 is definitely Taunt, it just seemed to correlate so well with 18 I thought they were tied together. It just so happened that the abilities affected by Taunt also did the same effect I came up with..

And now I'm not even sure the purpose of Unk18 at all. My targeting options never change without changing Bitfield 1, NPC's without changing Bitfield 2.. but this numbering is far too coincidental to be nothing. I now genuinely have no clue now what these numbers do.

Unk 16 is just as confusing. Changing the values now do absolutely nothing even though nearly every single move with Unk16=0 will thaw out EXCEPT FOR ONE, and every single move with Unk17=1 with do nothing EXCEPT FOR ONE. I thought maybe those two were just mistakes, but swapping their values, or the values of any other skill DOES NOTHING.

I've also been unable to replicate your mention of how Unk7 produces a different message when a move fails to hit. Changing a move to 0/100 and 100/0 Unk7/8 values always gave me the same message on a miss. I never once saw "The move failed to impact." A move either returns "The move failed", when no valid targets are in range, or "The move missed" when there is a valid target in range but it has failed the accuracy check. Both cases always returned "The move missed".

5 steps forward 3 steps back... *sigh*

Edit - Why didn't I see that on Unk11 The 99s were a dead giveaway. I'm jealous of your eyes, mine must not be functioning properly.

But now the remaining ones are the really confusing ones.

Unk9 - 544 entries with a value of 75 that have every possible combination of other attributes between them. 1 entry of 50 and 2 entries of 25 that have absolutely nothing in common or a clue as to why they aren't also 75 or what a value of 75 even means. Apparently there's something in game that Swords Dance, Sandstorm, and Icicle Spear do that no other move in the game does and they're all related to one another.

Unk16 - Apparently does nothing even though 99% of offensive fire attacks that thaw freeze, as well as every status curing move, have the same value of 0, and 99% of every other move has the same value of 1 and doesn't affect frozen targets... and reversing their values doesn't change this. Is this really a hardcoded property.? If so then what on earth do every other Fire offensive move in the game (and all status curers) do that Sacred Fire doesn't also do? I'm completely stumped.

Unk18 - Has SOMETHING to do with a move's identity/function. Nearly every move with a similar function as denoted in Bitfield 1 also shares the same Unk18 value, unique skill effects are left to their own group, but switching a move to another value produces absolutely no discernible result.

Edited December 23, 2017 by Zerea