New/Better/Easier method: Fiddler2 Proxy Plugin

[ZigZagJoe]

Ummmm ... [don't you like sentences that start with that word???] For lack of any better reason, I was using WinRaR wrong (I'm 51 so excuse me for being used to WinZip lol]. Didn't notice it extracted the entire BIN directory into the Scripts folder. Moved things where they should be and all seems good.

I turned on TURBO - to anyone who may be reading this, don't turn on TURBO with Fiddler already running. All kinds of errors happen! When I did it properly (closed and reopened both Fiddler and IE), things just hummed along. I did notice 2 things - first, in the Log, every once in a while I'd see the likes of this (see red area):

[log]

Second, now in Fiddler, is it ok that now all the audio-sjl-a- ... lines show in the Caching column no-cache, no-store, must-revalidate, ..., and the file size is always 44,405 (altho the MP4s do seem to be fine and named correctly and play OK).

I think I need a bigger HD! And once again, thanks ZZJ!

no problem!

yes, they will all be 44kb in size. that is how turbo mode works - yoink the good mp4 that was downloaded, save it, then replace the data that the browser (and pandora) gets with a blank mp4 file that is only 5 seconds long. thus, it plays a song every 5 seconds. the caching stuff is just fiddler playing it safe.

also, HUZZAH! that log finally told me what I wanted to know about an error that popped up from time to time. i've been trying to hunt it down forever; turns out that was part of the code which checked to make sure that data length was valid. but, it had a stupid mistake in the line that would report if something was wrong *doh*. It's nothing harmful, however. I found Pandora's servers from time to time hiccup (it could be caused by fiddler, i don't know) and give an error page, hence the need for code to catch it so we don't save 1kb files. i feel kinda stupid the bug was really harmless this entire time, though.

seeing as it is playing fine now, i'm guessing your net framework had a problem reading the resource stream. I'll have to check and make sure that it is not a .net 3 specific thing, as that would be bad for general use (I don't think 3.0 is widely deployed yet)

[Hawaii Guy]

seeing as it is playing fine now, i'm guessing your net framework had a problem reading the resource stream. I'll have to check and make sure that it is not a .net 3 specific thing, as that would be bad for general use (I don't think 3.0 is widely deployed yet)

FYI I am running Vista Business and .NET 3.5 SP1, IE8 and just recently Firefox.

I did run into a hiccup/error with Fiddler (sorry I rushed and forgot to save stuff but will next time). It had something to do with a problem with scripting on the page, and it went back to saving .DAT files, lots of em. Is this the bug you were talking about?

[ZigZagJoe]

FYI I am running Vista Business and .NET 3.5 SP1, IE8 and just recently Firefox.

I did run into a hiccup/error with Fiddler (sorry I rushed and forgot to save stuff but will next time). It had something to do with a problem with scripting on the page, and it went back to saving .DAT files, lots of em. Is this the bug you were talking about?

the error in the page was probably caused by it hacking out some of the ads, probably. shouldn't cause it to save dat files, however.

[Hawaii Guy]

the error in the page was probably caused by it hacking out some of the ads, probably. shouldn't cause it to save dat files, however.

Just some questions for ya ...

What is the "Got audio url" x 4 below? I also see in the http -a-1.pandora and -1-2.pandora but all is good so I'm just curious what goes on.

Running mp4tags.exe "C:\Pandora Music\\The Click - Girl With A Mind.mp4" -A "Fading Yellow: A Smorgasbord Of U.S. Pop-Sike 45's, Vol. 6 (1966-70)" -a "The Click" -s "Girl With A Mind" -g "Rock " -w "SID: 'Knock Three Times Radio'" -c "Ripped by PandoraSaver [Z]"

Inspecting XML doc: http://www.pandora.com/radio/x...1828&...mp;arg6=aacplus

Sniffed playlist answer

Got audio url: http://audio-sjl-a-1.pandora.com/access/...DiqtTBek%2F5orHV0ZA9XWcc7walnR6NUVJ

Got audio url: http://audio-sjl-a-2.pandora.com/access/...u7d1%2FGm019GanvgydVYdmfmExtF3oHE2g

Got audio url: http://audio-sjl-a-1.pandora.com/access/.....Fx2wjxGQR0iShnS

Got audio url: http://audio-sjl-a-1.pandora.com/access/.....ZSeFriF5lsZMhrW

Inspecting URL: http://audio-sjl-a-1.pandora.com/access/.....%2BM1FI7p32Y%3D

Matched audio URL: http://audio-sjl-a-1.pandora.com/access/.....%2BM1FI7p32Y%3D

Cached albumArt: http://images-sjl-4.pandora.com/images/pub...1_130W_130H.jpg

When it finds a song, it seems that you are still having mp4tags.exe execute to (re)tag the file. Is this true and necessary? I ask because I rename my files [down the road] from .MP4 to .M4A, so if you sniff the file and the file name matches, will it still try to rewrite my .M4A file? AND - if you remember - I edit the tags and image with information when the song originally was released. E.g. "Friends" from Elton John's Greatest Hits album changes to his "Friends" album with the cover art of that release. If mp4tags.exe is rewriting the file, it will overwrite my customized edits.

Do you notice the double "\\" in my examples? I'm guessing it's because the PandoraSaver Settings Folder Structure puts in a "\" even if left blank. But all works fine with files going to the right places.

Got song: Olivia Newton-John - "Who Are You Now?" on 'Come On Over'

File already exists! (\Olivia Newton-John - Who Are You Now.mp4)

Tagging mp4 file.

Running mp4tags.exe "C:\Pandora Music\\Olivia Newton-John - Who Are You Now.mp4" -A "Come On Over" -a "Olivia Newton-John" -s "Who Are You Now?" -g "Rock " -w "SID: 'Knock Three Times Radio'" -c "Ripped by PandoraSaver [Z]" -P "C:\Pandora Music\\album.png"

I saw this error come up. I didn't notice any commercial at the time. I have seen a "commercial" pop up like a song in the Pandora browser scrolling songs window (if you follow my convoluted explanation lol).

Inspecting URL: http://audio-sjl-a-2.pandora.com/access/.....FkLgYp7zA%3D%3D

Matched audio URL: http://audio-sjl-a-2.pandora.com/access/.....FkLgYp7zA%3D%3D

Error while handling URL: http://audio-sjl-a-2.pandora.com/access/.....FkLgYp7zA%3D%3D

Len = 347

infos:

System.Collections.Generic.KeyNotFoundException

The given key was not present in the dictionary.

at System.ThrowHelper.ThrowKeyNotFoundException()

at System.Collections.Generic.Dictionary`2.get_Item(TKey key)

at Saver.AutoTamperResponseBefore(Session oSession) in C:\Documents and Settings\zigzagjoe\Desktop\PandoraSaver\new\src\Saver.cs:line 623

And, would it be difficult for you to make the field options in the mp4tags.exe program options for us to either blank out or put in our own static data?

Running mp4tags.exe "C:\Pandora Music\\Love - Alone Again Or.mp4" -A "The Forever Changes Concert" -a "Love" -s "Alone Again Or" -g "Rock " -w "SID: 'Knock Three Times Radio'" -c "Ripped by PandoraSaver [Z]" -P "C:\Pandora Music\\album.png"

Thanks.

[ZigZagJoe]

Just some questions for ya ...

What is the "Got audio url" x 4 below? I also see in the http -a-1.pandora and -1-2.pandora but all is good so I'm just curious what goes on.

Running mp4tags.exe "C:\Pandora Music\\The Click - Girl With A Mind.mp4" -A "Fading Yellow: A Smorgasbord Of U.S. Pop-Sike 45's, Vol. 6 (1966-70)" -a "The Click" -s "Girl With A Mind" -g "Rock " -w "SID: 'Knock Three Times Radio'" -c "Ripped by PandoraSaver [Z]"

Inspecting XML doc: http://www.pandora.com/radio/x...1828&...mp;arg6=aacplus

Sniffed playlist answer

Got audio url: http://audio-sjl-a-1.pandora.com/access/...DiqtTBek%2F5orHV0ZA9XWcc7walnR6NUVJ

Got audio url: http://audio-sjl-a-2.pandora.com/access/...u7d1%2FGm019GanvgydVYdmfmExtF3oHE2g

Got audio url: http://audio-sjl-a-1.pandora.com/access/.....Fx2wjxGQR0iShnS

Got audio url: http://audio-sjl-a-1.pandora.com/access/.....ZSeFriF5lsZMhrW

Inspecting URL: http://audio-sjl-a-1.pandora.com/access/.....%2BM1FI7p32Y%3D

Matched audio URL: http://audio-sjl-a-1.pandora.com/access/.....%2BM1FI7p32Y%3D

Cached albumArt: http://images-sjl-4.pandora.com/images/pub...1_130W_130H.jpg

When it finds a song, it seems that you are still having mp4tags.exe execute to (re)tag the file. Is this true and necessary? I ask because I rename my files [down the road] from .MP4 to .M4A, so if you sniff the file and the file name matches, will it still try to rewrite my .M4A file? AND - if you remember - I edit the tags and image with information when the song originally was released. E.g. "Friends" from Elton John's Greatest Hits album changes to his "Friends" album with the cover art of that release. If mp4tags.exe is rewriting the file, it will overwrite my customized edits.

Do you notice the double "\\" in my examples? I'm guessing it's because the PandoraSaver Settings Folder Structure puts in a "\" even if left blank. But all works fine with files going to the right places.

Got song: Olivia Newton-John - "Who Are You Now?" on 'Come On Over'

File already exists! (\Olivia Newton-John - Who Are You Now.mp4)

Tagging mp4 file.

Running mp4tags.exe "C:\Pandora Music\\Olivia Newton-John - Who Are You Now.mp4" -A "Come On Over" -a "Olivia Newton-John" -s "Who Are You Now?" -g "Rock " -w "SID: 'Knock Three Times Radio'" -c "Ripped by PandoraSaver [Z]" -P "C:\Pandora Music\\album.png"

I saw this error come up. I didn't notice any commercial at the time. I have seen a "commercial" pop up like a song in the Pandora browser scrolling songs window (if you follow my convoluted explanation lol).

Inspecting URL: http://audio-sjl-a-2.pandora.com/access/.....FkLgYp7zA%3D%3D

Matched audio URL: http://audio-sjl-a-2.pandora.com/access/.....FkLgYp7zA%3D%3D

Error while handling URL: http://audio-sjl-a-2.pandora.com/access/.....FkLgYp7zA%3D%3D

Len = 347

infos:

System.Collections.Generic.KeyNotFoundException

The given key was not present in the dictionary.

at System.ThrowHelper.ThrowKeyNotFoundException()

at System.Collections.Generic.Dictionary`2.get_Item(TKey key)

at Saver.AutoTamperResponseBefore(Session oSession) in C:\Documents and Settings\zigzagjoe\Desktop\PandoraSaver\new\src\Saver.cs:line 623

And, would it be difficult for you to make the field options in the mp4tags.exe program options for us to either blank out or put in our own static data?

Running mp4tags.exe "C:\Pandora Music\\Love - Alone Again Or.mp4" -A "The Forever Changes Concert" -a "Love" -s "Alone Again Or" -g "Rock " -w "SID: 'Knock Three Times Radio'" -c "Ripped by PandoraSaver [Z]" -P "C:\Pandora Music\\album.png"

Thanks.

"got audio url: http://...." means it sniffed a playlist response which contained those URLs (and associated artist etc data with them), so when pandora requests a song with that url it knows what that song's info is. i have it echo them for log purposes so I can try to guess what went wrong if something errors.

the keynotfound exception is the one i mentioned above - a small error in the code which reports if a file request is too short. you may safely ignore it.

the double \ is because it always adds a trailing slash. it's legal by windows standards and should cause no problems.

mp4tags only tags when a file is downloaded, as downloaded files have *no* tags at all. it will not tag again(overwrite) unless you have the option "always update file tags" set in advanced settings checked.

unless you have "strict dup checking" in advanced settings checked, it effectively ignores song extensions. if you find it is ripping the same files already present lower the dup tolerance under advanced settings to 90. (from default of 100) so, if you have a song named artist - song.m4a it will see that and not re-download the song.

(see the visual description at http://i141.photobucket.com/albums/r49/zig.../PS_help_v1.gif )

also, what do you mean by static data? (like what fields). i've contemplated having it look up songs on gracenote to better fill in fields, but ultimately decided it was not necessary.

[ZigZagJoe]

In the interim, I have set up a privoxy for those who use Tor. Download http://zzj.itf-inc.com/Privoxy.PS.zip , extract it somewhere, and then run "Setup.RUNONCE.vbs" (which is in the zip). it will shutdown fiddler if it is running, set IE settings to use privoxy, set fiddler to chain with privoxy, and finally launch privoxy and fiddler.

Afterwards, you can use "Run.PandoraSaver.vbs" to run privoxy and fiddler2, or "Uninstall.vbs" to reset IE's proxy settings. This script assumes that you have fiddler installed to C:/Program Files/Fiddler, if not, it will prompt you to manually run it.

Use the browserPAC I mentioned before in your browser:

Create a file named "BrowserPAC.js" somewhere, and paste the following into it:

// script which IE or firefox can use to decide how to route a certain URL
// this can also be used to avoid pandora's US-only restriction

function FindProxyForURL(url, host) {

 if (shExpMatch(host, "*pandora.com*")) // url contains pandora.com
        return "PROXY 127.0.0.1:8888";    
            
 return "DIRECT"; // no proxy
}

For firefox, go to Tools Menu>Options->Advanced->Network->Settings and set it to use "Automatic proxy configuration URL". in the box provided, enter "file:///C:/BrowserPAC.js" (or wherever you a put it; using forward slashes instead of backslashes). This will have Firefox ONLY use fiddler for pandora and for nothing else.

For IE, on Fiddler, go to Tools->WinINET Options then, on the "Internet Properties" box that came up, go to Connections->LAN Settings, disable "Use a proxy server for your LAN" and check "Use automatic configuration script" in which you put "file:///C:/BrowserPAC.js" [

you can also use http://zzj.itf-inc.com/BrowserPAC.js as the URL, as opposed to creating a file on your computer.

What this package essentially does is use privoxy as a "translator" to allow fiddler to access SOCKS proxies (which is what Tor uses). Assumes Tor is using port 9050, if not, modify the line "forward-socks4a / 127.0.0.1:9050 ." in config.txt to reflect which port it uses.

so, the end result is:

Ye Olde Interwebz <=> TOR <=> Privoxy <=> Fiddler <= [PandoraSaver] => Your Browser, running Pandora

I also uploaded a fixed version of PS with the filesize-reporting-error corrected. it doesn't effect anything, so you do not need to download it if you already have it already. link is same as always, http://zzj.itf-inc.com/PandoraSaver1005-ZZJ.zip

[boem]

Thanks a lot ZZJ. I can now rip the songs with the co-working of TOR + Privoxy + Fiddler + PandoraSaver. But only ripping not listening because of the error "It's taking longer than expected..." The ripping keeps going on, despite that. I'm still experimenting what the correct launching sequence of the different apps is. Also, after a while this error pops up (see attached file). This happened a couple of times.

I'm sure things will get fixed over time but I'm already very happy. Thanks again ZZJ.

[ZigZagJoe]

Thanks a lot ZZJ. I can now rip the songs with the co-working of TOR + Privoxy + Fiddler + PandoraSaver. But only ripping not listening because of the error "It's taking longer than expected..." The ripping keeps going on, despite that. I'm still experimenting what the correct launching sequence of the different apps is. Also, after a while this error pops up (see attached file). This happened a couple of times.

I'm sure things will get fixed over time but I'm already very happy. Thanks again ZZJ.

No problem. By the way, what browser are you using? I think someone else has reported this problem before (ripping, but not playing) with IE.

As for launch sequence, I would first launch Tor, then Privoxy, then Fiddler, and finally the browser. Just playing it safe (it really shouldn't make much of a difference what order they are launched in, just that everything has been launched when you try to look at a page).

I'm guessing the error is related to one of the proxies along the line loosing the rest of the request, thus ending up with a partial XML file. It's not a crippling error, just means that it will be unable to read the next 4 songs (probably). Might change the dialog box into a log message...

Standalone Update:

Still chuggin away. The tunnel stuff is being a right pain in the ass. (Pandora uses SSL to download the flash applet and authenticate, thus I must support tunneling for it.)

Misc:

It's an option to have this look up on Gracenote, I registered for that a long time ago and finally noticed that they sent me a message. If it allows a way to get large CD images, i'll probably implement it. Haven't had time to look at what exactly their API provides yet (besides the obvious song info). Though, i am not exactly clear what the non-commercial license entails - I'll have to make sure everything is fine with redistributing it and such.

[boem]

I'm using Avant which is virtually IE7. Would it be something to do with the port setting? I see in the setup.runonce that port 8887 is specified for proxy. But when fiddler starts, the port in LAN/Proxy settings chnages to 8888.

[Hawaii Guy]

also, what do you mean by static data? (like what fields). i've contemplated having it look up songs on gracenote to better fill in fields, but ultimately decided it was not necessary.

I was referring to the line right below my question, which is your mp4tags.exe executed line, specifically referring to the -w and -c fields....

And, would it be difficult for you to make the field options in the mp4tags.exe program options for us to either blank out or put in our own static data?

Running mp4tags.exe "C:\Pandora Music\\Love - Alone Again Or.mp4" -A "The Forever Changes Concert" -a "Love" -s "Alone Again Or" -g "Rock " -w "SID: 'Knock Three Times Radio'" -c "Ripped by PandoraSaver [Z]" -P "C:\Pandora Music\\album.png"

And an FYI - I was going through some of the ripped music from the turbo session I did the other day and noticed that there were a couple of files that were mistagged (file name, -a, -s and -P were for a totally different artist). But I didn't care based on the number of songs I got vs. 2 incorrect.

[ZigZagJoe]

I'm using Avant which is virtually IE7. Would it be something to do with the port setting? I see in the setup.runonce that port 8887 is specified for proxy. But when fiddler starts, the port in LAN/Proxy settings chnages to 8888.

nah, unrelated to port settings. the port settings getting changed is how this chains with fiddler - it writes to use privoxy, and then when fiddler launches, it reads those settings, "copies them" and then changes IE's settings to use fiddler first instead. fiddler will then use privoxy for any requests. It's a bit convoluted, but it does work.

You might try firefox and see if that plays while ripping. (just set it to either use the browserPAC or 127.0.0.1:8888)

I was referring to the line right below my question, which is your mp4tags.exe executed line, specifically referring to the -w and -c fields....

And, would it be difficult for you to make the field options in the mp4tags.exe program options for us to either blank out or put in our own static data?

Running mp4tags.exe "C:\Pandora Music\\Love - Alone Again Or.mp4" -A "The Forever Changes Concert" -a "Love" -s "Alone Again Or" -g "Rock " -w "SID: 'Knock Three Times Radio'" -c "Ripped by PandoraSaver [Z]" -P "C:\Pandora Music\\album.png"

And an FYI - I was going through some of the ripped music from the turbo session I did the other day and noticed that there were a couple of files that were mistagged (file name, -a, -s and -P were for a totally different artist). But I didn't care based on the number of songs I got vs. 2 incorrect.

I suppose i could make the station-in-composer field togglable, the comment would be a bit harder however (there isn't room in the config screen for it). Could make it a setting in the registry, i suppose.

Somehow the proxy-host has started working perfectly overnight. I didn't touch anything, i swear! It scares me.

Everyone: Please make any posts in http://hak5.org/forums/index.php?showtopic=12047 as this thread has gotten a bit too long

Also, the standalone version has been released! (no fiddler or privoxy required anymore) See thread.