Okay... a little bit of research answered part of my own question and then my head exploded...
If I log into the Bunny Shell, at least in Arm Mode, I can indeed issue the command QUACK WINDOWS r and my run dialog box pops up on the host...
However, I might have over complicated my original idea. If I just need OS detection and if payload.txt is really just a shell script, as the bang line at the top of the some of the examples implies, couldn't I add a loop there to wait for a file and act on it? So in theory...
Main payload.txt starts, executes a.vbs, begins to loop while waiting.
a.vbs determines the OS and writes osversion.txt back to the same directory as itself
payload.txt now sees osversion.txt and continues with the appropriate actions.
This doesn't help with adjusting ATTACKMODE on the fly but it would help.
As far as switching the attack mode I tried this... An "empty" payload with just SERIAL and HID modes, then from inside a shell over the serial I issued another "ATTACKMODE HID STORAGE SERIAL" to add the storage option (presuming I need to keep my existing options). This locked up my session and things were not happy... Similarly "ATTACKMODE STORAGE" also failed. Fundamentally I'm assuming that ATTACKMODE is a one time only thing. It would be really nice if it could be reinvoked to change or add the chosen modes. Say, start with Storage and HID, determine the lay of the land and then remove Storage (once osversion.txt is written) and add RNDIS_Ethernet. This would allow us to use, but then hide, the mounted drive from the view of the user.