The protocol has been dubbed Ni-Fi, it's basically standard wifi with some proprietary tweaks, and it's the same as that used in DS-DS local communications. There's some documentation here, but it doesn't seem as if its very complete and interest seems to have dried up long ago. As far as I know Wii homebrew has never focused on the DS communication aspects, so I don't know what kind of groundwork would have to be laid before one could start to write homebrew that utilizes DS communication.