Did you copy the entire RomFS directory from your ROM dump? You should only be moving over files that you edited with pk3DS. You can always find GARCs in Options->About GARCs. Depending on the game, it'll output every GARC that pk3DS can load/edit. Example of Omega Ruby: https://i.imgur.com/YFJ4dIR.png
So if, for example, I only randomized Wild Encounters, Trainer Pokemon, and Static Encounters in Moon, I would move the following to my SD Card:
/a/0/8/3
/a/1/0/4
/a/1/0/5
/a/1/0/6
/a/1/5/5
The rest of the GARCs wouldn't be moved to my SD Card because they were not modified and there serves no purpose of moving them.
As for CROs, there are only a few notable ones:
- DllBattle.cro (Gen 6; Type Chart)
- DllField.cro (Gen 6; Static Encounters, Gift Pokemon)
- DllPoke3Select.cro (Gen 6; Starter Models)
- Shop.cro (Gen 7; Poke Mart data)
If you're only moving files that were edited, the game should experience no slowdown whatsoever. I've made several randomizers and never came across the issues you mentioned because I only moved what was edited, the rest is loaded by the game at default. Luma LayeredFS only checks for edited files on the SD Card, and if said file doesn't exist, it loads from the game on its own -- hence the extra, unedited files are unnecessary