I'm not exactly sure i understand your question...
The program would be run like any normal payload,with the difference that it decrypts itself into memory at some point...which would be pretty much the only point it would be detectable...
The AV wouldn't be able to detect it as a known 'virus',except of course,if it is a bad encryption,or it somehow behaves stupidly...
Well,if we look at my post here,you see the same conclusions from sablefoxx ,though he stated it clearer...the program will be catched in memory or in the decrypter embedded in the file,though there are workarounds for both...BTW Sc0rpi0 i would NOT recommend creating a rootkit though,since it makes the targeted system VERY unstable and mis configures it pretty bad too,that is,if you aren't an expert on what you are doing...
¨Whatever you choose,good luck :-)