Need help with RTMPE protocol handling on Myspace.com

[Laska]

Hey guys, it all started yesterday when I thought it would be no big deal to dl some mp3s from myspace.

woah, I was wrong. It seems myspace.com switched to an 'RTMPE' protocol for their mp3 streams.

You can use an url catcher or this premade script (http://foxylion.xail.net/myspacedump/) to capture the direct rtmp link

let's parse http://www.myspace.com/feverray for a test. the first track 'When I grow up' would be

rtmpe://akafms-music02.myspacecdn.com/ondemand/mp3:137/std_3f3e6f2962504f8daebeba6df6b833df

now I tried with a tool called rtmpdump 1.6 (sources & binaries here), console output says:

rtmpdump.exe -r rtmpe://akafms-music02.myspacecdn.com/ondemand/mp3:137/std_3f3e6f2962504f8daebeba6df6b833df -o "When I Grow Up.flv"

DEBUG: Parsing...

DEBUG: Parsed protocol: 3

DEBUG: Parsed host : akafms-music02.myspacecdn.com

DEBUG: Parsed app : ondemand

DEBUG: Parsed playpath: mp3:137/std_3f3e6f2962504f8daebeba6df6b833df

DEBUG: Setting buffer time to: 36000000ms

Connecting ...

DEBUG: Protocol : RTMPE

DEBUG: Hostname : akafms-music02.myspacecdn.com

DEBUG: Port : 1935

DEBUG: Playpath : mp3:137/std_3f3e6f2962504f8daebeba6df6b833df

DEBUG: tcUrl : rtmpe://akafms-music02.myspacecdn.com:1935/ondemand

DEBUG: app : ondemand

DEBUG: flashVer : LNX 9,0,124,0

DEBUG: live : no

DEBUG: timeout : 300 sec

DEBUG: Connect, ... connected, handshaking

DEBUG: HandShake: Client type: 06

DEBUG: HandShake: DH pubkey position: 1215

DEBUG: HandShake: Client digest offset: 410

DEBUG: HandShake: Initial client digest:

BE 6E A7 30 2C F8 D9 74 B3 D2 10 94 56 12 39 9B 79 62 AA 24 EF 91 21 78 6E 53 C8

88 2D FB 0F 4C

DEBUG: HandShake: Type Answer : 06

DEBUG: HandShake: Server Uptime : 387834394

DEBUG: HandShake: FMS Version : 3.5.1.1

DEBUG: HandShake: Server DH public key offset: 90

DEBUG: HandShake: Secret key:

C5 F3 E5 7A 7C 00 9F 9C BA CA 11 9A 89 67 FD 06 19 B2 D0 E1 1C 2E C4 01 BD 2E 58

F7 51 DD 33 F8 E9 B8 CE F5 D2 80 7B D1 C2 54 42 44 34 18 74 3D DE D8 4A C5 DC 7

4 75 F3 10 BF 21 AB 35 3E BF 56 D0 55 72 C5 26 80 F1 7D 18 99 2C 71 6A 0E 78 06

76 81 41 A7 E1 B2 47 B7 1E 86 54 64 D1 B8 E6 AF 71 28 03 53 9D ED 5A 83 EF D7 2D

2B FE 38 5A 50 7D D5 D6 12 CF EC 75 72 5D 26 4F D5 00 CE 2D BC

DEBUG: RC4 Out Key:

54 39 6F C4 B1 2E 4F E0 A0 81 AB 1F 73 E0 82 0E

DEBUG: RC4 In Key:

76 7D B7 A2 A7 66 C6 5B 85 33 A1 8F A2 33 F6 B7

DEBUG: HandShake: Client signature digest position: 410

DEBUG: HandShake: Digest key:

46 80 B3 67 87 5C 0B 9D 77 45 A4 51 D8 E7 53 55 B5 8F 7D E0 B5 67 62 09 F6 AD 08

00 83 96 A1 31

DEBUG: HandShake: Signature calculated:

54 F6 29 26 ED 0E 17 13 76 88 5D 57 0E 0F 4D CC E0 74 41 EB 68 1F 65 63 BD 3B 9C

CC F0 0D 13 DF

DEBUG: HandShake: Server sent signature:

54 F6 29 26 ED 0E 17 13 76 88 5D 57 0E 0F 4D CC E0 74 41 EB 68 1F 65 63 BD 3B 9C

CC F0 0D 13 DF

DEBUG: HandShake: Genuine Adobe Flash Media Server

DEBUG: HandShake: Calculated digest key from secure key and server digest:

C8 0F 93 6B 1F BA D6 23 1F C3 29 91 EA B3 B7 4E 43 07 55 B2 0B AE 24 77 E4 9F EA

6A 41 D1 B8 EE

DEBUG: HandShake: Client signature calculated:

37 15 50 92 28 C6 80 B1 22 D0 01 78 A2 09 40 1B 15 43 A1 FB 24 C8 B5 DF 5E D1 EA

77 54 3C 0C D0

DEBUG: HandShake: Handshaking finished....

DEBUG: Connect, handshaked

Connected...

Starting download at 0.000 KB

DEBUG: GetNextMediaPacket, received: server BW

DEBUG: GetNextMediaPacket, received: client BW

DEBUG: HandlePing, received ping. type: 0, len: 6

DEBUG: GetNextMediaPacket, unknown packet type received: 0xf0

ERROR: ReadN, RTMP recv error 10060

ERROR: ReadPacket, failed to read RTMP packet header. type: 5

Closing connection... done!

Well, I know next to nothing about network protocols (I'm more into openGL gfx stuff ;)),

so I don't quite get what's going on here. I see rtmpdump tries to calculate some kind of

fake signature to establish a handshake. But where are these DH and secret keys taken for

the calc from? is there more than one RTMPE format? Has anybody any good links to some

documentation about RTMPE/RTMPTE?

Do I also need some kind of session ID token before the dl starts?

Do I need something more from the SWF player which normally hands over the .mp3 link (like some

logic check)

Can I just override the first unrecognized packet and capture a length of say 4 mb incoming

data (if I don't know about how to calculate the filesize) or does 0xf0 mean the connection

was terminated?

Thanks for any input and help on this!

Laska

update: for those interested, I finally found some documentation here

http://lkcl.net/rtmp/RTMPE.txt

[digip]

rtmpe data is damn near impossible to save, but there are ripping utilities out there that claim to be able to do it. I can grab stuff from normal flash players, but anything using rtmpe always gave me issues and playback was distorted if ever able to get anything at all. Most of the time it was just garbage data, I think in part because its not regular data, but often has DRM in it. Regular rtmp is as far as I know plain data streams, while rtmpe is the DRM stuff(don't quote me on that though). There is a rtmp ripper, http://linuxcentre.net/getiplayer/download/ but probably won't work for anything that has DRM in it.

[Laska]

Hey digip, thanks for the link! Seems the authors of this tool were forced

to remove rmpte support in their successor flvstreamer. bummer :/

I have a feeling I might just miss some token or SWF player check (like location

or size of the SWF) as the handshake with the calculations 'seems' to work already

in the old version.

but maybe rtmpdump does not recognize all command bytes yet, or Adobe changed

something.

I'm still trying to understand the source and all possible -options of this tool

to rule out I just forgot some parameters in the command line ;)

At least I found some (maybe incomplete) documentation and linked it in the first post.

[digip]

Do you stay on the page you are ripping from the entire time its pulling in the file, or once it starts downloading do you leave the page. I think some of these use XML for file check sequencing(kind of like a CRC in a packet) and may periodically update those keys and such. They may even check to see if the requester is still there with a keepalive or something and will timeout the download if they can't verify referrer information.

[operat0r_001]

http://arrggggggggg_pirates_arebad.org/tor...acOSX_Windows_v

wonder if this works .. I dont know of any rtmpe feeds

I use real media catcher and uslsnooper to do these types of things

http://rmccurdy.com/upload1/Replay%20Media%20Catcher.zip

uslsnooper : free

[Brian Sierakowski]

Don't know exactly what you're trying to accomplish, but have you tried the low tech way?

What you can do is use an audio recording suite (audacity, or sound recorder even), set your input to "stereo mix", hit record on the program, then hit play on myspace.

You can grab any audio that comes through your speakers this way, making it handy for when I was looking for some sound effects not to long ago.

Let me know if you think this suits your need and you'd like a bit better guidance.

[digip]

Don't know exactly what you're trying to accomplish, but have you tried the low tech way?

What you can do is use an audio recording suite (audacity, or sound recorder even), set your input to "stereo mix", hit record on the program, then hit play on myspace.

You can grab any audio that comes through your speakers this way, making it handy for when I was looking for some sound effects not to long ago.

Let me know if you think this suits your need and you'd like a bit better guidance.

I used to do that with cool edit back in the win95 days and still do once in a while with soundforge today, but lets say you want to download some video as well, not just the audio, a camtasia screen cap just won't have the same quality as the original files. Audio downloads are simple, and I still use the method you speak of with soundforge(which is how I got the last danger mouse album) but it would be nice to have the files just downloaded from the sites directly.

Myspace used to be an easy target since they used flash as a mitm to where the actual mp3 was stored, but they changed their service to now use streaming rtmp files, which is much harder to rip from, short of resorting to tricks of using the old soundcard to grab "what you hear" while its playing.

[Zimmer]

Yay I can help... I have been working on RTMPE implmentation for python (to download video to watch offline (NO PIRACY))...

All data from links below... no data from me is guranteed to be correct or wrong etc...

First off RTMPE is encrypted using RC4, RTMPDUMP is able to read rtmpe streams, but my success has been nill. There use to be a app called Replay Media Catcher but due to pressure from Adobe they have droped RTMPE support.

Ok here is goes...

in RTMPE you send a request for a connection... you send a data including a Flash Player Key (Static) and Diffie Hellman Public Key. It then uses you Private Key (I believe) and the servers Public Key to estalish a secret key that is then used as the RC4 key that is used for encryption.

Another thing adobe uses is SWF verification which is it takes a hash and uses it to provide extra 'security'.

Technical Specs...

http://lkcl.net/rtmp/RTMPE.txt

http://www.rtmpd.com/export/425/trunk/docs...PEHandshake.pdf

http://lkcl.net/rtmp

Also any help is appreciated if anyone is willing to help me with a python implmetation of this. :)

EDIT

Hulu goes even farther by using a swf file to be run on an encrypted string that is used for the rtmpe url (so requires a program to run swf files (Adobe, gnash etc)) (XBMC has had problems with this (see hulu plugin chat in forums (both dev thread a old thread that is now locked)).

Edit again... for anyone interested here is the python code (really ruff!!!!!!!)

DH.py

#Random Stuff not implmented VALUES IN __init__ are not cryptographically good!!!!!!!!!!!!!!
import os
class DHM(object):#DHM, Diffie Hellman Merkle
    def __init__(self):
        self.g = 2
        self.prv_key = int(os.urandom(16)) #128 bit prv_key... uses /dev/random (http://en.wikipedia.org/wiki//dev/random) or on Microsoft Windows is uses CryptGenRandom (http://en.wikipedia.org/wiki/CryptGenRandom)
        self.pub_key = int()
        self.s = int()#Secret Key between Alice and Bob
        DH_Numbers = [9583845938493831,#These are no cryptographically strong! yes I use them for my DH keys... 
        8489394839487203,#because this is a client.. 
        2394928504049491,#I DON'T care if the servers video can be easily decrypted
        1114959492103583
        ]
        self.p = DH_Numbers[random.randint(0,3)]
    def gen_public_key(self):
        self.pub_key = self.g**self.prv_key%self.p
        return self.pub_key
    def gen_shared_key(self):
        #Determines Shared Key
        self.s = self.A**self.b % self.p
        return self.s
    def SetP(self, p):
        self.p = p
    def SetG(self, g):
        self.g = g
    def SetPrv(self, prv_key):
        self.prv_key = prv_key
    def Pub_Prv(self):
        return self.pub_key, self.prv_key
    def get_p(self, bit_length):
        self.primes[bit_length]
def DH_Shared(PRV_KEY, PUB_KEY)
    DH_S = DHM()
    DH_S.A = PUB_KEY
    DH_S.b = PRV_KEY
    DH_S.gen_shared_key()
def DHKeyGenerate(bit):
    if not bit == 128:
        print 'ONLY TAKES 128 bit now!!'
    DH__ = DHM()
    DH__.gen_public_key()
    DH__.Pub_Prv()

RTMPE.py

import sockets
import struct
import time
import hmac
import hashlib
import DH


def send_data(data):
    pass
    #set up later, used to send data
def big_endian(integer):
    bytes = struct.pack('>i', integer)
    return bytes

RTMP_SIG_SIZE = 1536

SHA256DL = 32 #" SHA 256-byte Digest Length "

RandomCrud ='\xf0\xee\xc2\x4a\x80\x68\xbe\xe8\x2e\x00\xd0\xd1\x02\x9e\x7e\x57\x6e\xec\x5d\x2d\x29\x80\x6f\xab\x93\xb8\xe6\x36\xcf\xeb\x31\xae'

SWFVerifySig = '\x1\x1'

GenuineFMSConst = "Genuine Adobe Flash Media Server 001" #" 36 bytes long "

GenuineFPConst  = "Genuine Adobe Flash Player 001" #" 30 bytes long "

GenuineFMSConstCrud = GenuineFMSConst + RandomCrud

GenuineFPConstCrud = GenuineFPConst + RandomCrud

clientsig = []

def GetServerDHOffset(bytes):
    offset = bytes[0] + bytes[1] + bytes[2] + bytes[3]
    offset = modulo(offset,632)
    offset = offset + 8
    return offset
    
def GetServerGenuineConstDigestOffset(bytes):
    offset = byte[0] + byte[1] + byte[2] + byte[3]
    offset = modulo(offset,728)
    offset = offset + 776
    return offset

def GetClientDHOffset(bytes):
    offset = byte[0] + byte[1] + byte[2] + byte[3]
    offset = modulo(offset,632)
    offset = offset + 772
    return offset

def GetClientGenuineConstDigestOffset(bytes):
    offset = byte[0] + byte[1] + byte[2] + byte[3]
    offset = modulo(offset,728)
    offset = offset + 12
    return offset
def HMACsha256(msg_, key_):
    hmac_sha256 = hmac.new(key = key, msg = msg, digestmod = hashlib.sha256)
    return hmac_sha256.digest() #or should I use the hexdigest()???
dhpkl = GetClientDHoffset(clientsig[1532:1536])
DHPrivateKeyC, DHPublicKeyC = DHKeyGenerate(128) #" 128-bit "/
clientsig[dhpkl:dhpkl+128] = DHPublicKeyC
cdl = GetClientGenuineConstDigestOffset(clientsig[8:12])
msg = clientsig[0:cdl] + clientsig[cdl+SHA256DL:RTMP_SIG_SIZE]#Had -1 but no need sense python is diffrent indices then the reverse engineering
clientsig[cdl:cdl+SHA256DL] = HMACsha256(msg, GenuineFPConst)
send(command + clientsig)
recv(command + serversig)
def MSSFMT1():
    sdl = GetClientGenuineConstDigestOffset(serversig[8:12])
    msg = serversig[0:sdl] + serversig[sdl+SHA256DL:RTMP_SIG_SIZE]
    Compare(serversig[sdl:sdl+SHA256DL-1], HMACsha256(msg, GenuineFMSConst))
    dhpkl = GetClientDHoffset(serversig[1532:1536])
    DHPublicKeyS = serversig[dhpkl:dhpkl+128]
def MSSFMT2():
    sdl = GetServerGenuineConstDigestOffset(serversig[772:776])
    msg = serversig[0:sdl] + serversig[sdl+SHA256DL:RTMP_SIG_SIZE]
    Compare(serversig[sdl:sdl+SHA256DL], HMACsha256(msg, GenuineFMSConst))
    dhpkl = GetServerDHoffset(serversig[768:772])
    DHPublicKeyS = serversig[dhpkl:dhpkl+128]
def ENCRYPTKEY():
    DHSharedSecret = DH_Shared(DHPrivateKeyC, DHPublicKeyS)

RTMPE_SOCKETS.py

import socket
from time import time
from time import sleep
import sys
'''
NOT WORKING
class com(object):
    def __init__(self, host):
        self.host = host
        self.port = 1935
        self.address = (self.host , self.port)
        self.sock = None
    def send(self, command):
        self.sock.send(command)
    def conn(self):
        self.sock = socket(AF_INET, SOCK_STREAM)
        self.sock.connect(self.address)
    def recv(self):
        data = self.__sock.recv(BUFSIZE)
        return data
'''
#url format rtmp://
url = 'cp21927.edgefcs.net'
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((url, 1935))