1125 replies to this topic

[ZigZagJoe]

Saver2 - http://ridetheclown.com/wp/saver2/

DO NOT REPLY TO THIS THREAD - I WILL NOT SEE IT! INSTEAD, GET ON IRC BY CLICKING THE "LIVE HELP" BUTTON ON SPROXY, by launching IRCHelp.exe in your install folder, OR by joining #saver2 ON irc.coldfront.net !

You can also get on irc by going to this web page.

Edited by ZigZagJoe, 05 May 2012 - 11:47 AM.

[cryptk]

I just installed the new Beta and I am trying it out with Firefox Portable, Pandora One on High Quality and Turbo Mode... So far it seems to be working great.

Your download link for Firefox Portable wasn't working for me so i had to download it on my own and configure it myself but other than that the setup went very smooth and even with that hiccup I was able to get everything configured with nothing more than the instructions in this post.

If I get any errors or problems I will be sure to post about it with an error log. Is there anything in particular that you want me to test out since it seems that I am one of the first using the new code?

[ZigZagJoe]

I just installed the new Beta and I am trying it out with Firefox Portable, Pandora One on High Quality and Turbo Mode... So far it seems to be working great.

Your download link for Firefox Portable wasn't working for me so i had to download it on my own and configure it myself but other than that the setup went very smooth and even with that hiccup I was able to get everything configured with nothing more than the instructions in this post.

If I get any errors or problems I will be sure to post about it with an error log. Is there anything in particular that you want me to test out since it seems that I am one of the first using the new code?

One of the first.

Another note - I'm working on an improved turbo mode (and general plugin handling). This method prevents pandora from getting fidgety as it responds with a blank file immediately, of either 5, 10, 15, 20, 30, 60 second length, picked based on how long it thinks it will take to finish downloading the current file.

Also, i should note that the old pandorasaver is incompatible with the new sproxy, and same deal for saver2 - i've had to make changes to the plugin interface so it will fail if you try to load the new with the old or vice versa.

[vrwarp]

Written in C#

Maybe this can be ported to Mono so it can run on linux =)?

[ZigZagJoe]

Maybe this can be ported to Mono so it can run on linux =)?

Yeah, i mean to try it sometime. Re-encoding wouldn't work without wine (Or maybe replaced with linux binaries?) but the rest is fairly standard C# code, nothing too exotic. (that i can recall)

[cryptk]

been using it for a while now with the setup that I mentioned in my earlier post, so far no problems! It is grabbing the mp3 (high quality) grabbing the artwork, sorting the music how I told it to, making the playlist files... I would go so far as to say that aside from having to take an ever so slightly more complicated route to some of the settings it is exactly like the older version... which is a good thing since the old version was great!



Good work man, good work

[ZigZagJoe]

Okay, i've updated the link above with a new (still beta) version.

This is a mostly pandora update:

Pandora
Greatly improved turbo mode (!!)
Configuration dialog
Some bugfixes

Saver2
Fixed newmusic not having ext
Plugin infrastructure additions

Sproxy
Various internal bugfixes


Turbo changes are basically it's much more kind to pandora's servers now, you only need to run one pandora at a time in turbo (moreover more than one will prevent it from working properly) + it should not fail due to a song taking too long to download.

Other changes: Low bandwidth mode is obsolete and has been removed. All songs will now start playing immediately, and so on. (in effect, it's always in low bandwidth mode)

Known bugs: if you exit pandora while a song download is in progress, but pandora has not changed to the new song, it will never be saved. (song never displayed -> pandora never displays aart -> song never ripped as aart is the trigger when present)

Notes about how the changed turbo mode works: Instead of sproxy download all of a file and then let the plugins have a whack at it before sending it off to pandora (or whomever requested it), it asks the plugins first if it should handle it specially. it is up to the plugins to mark no streaming (so that something can be inserted into the source, like the pandora main page auto refresh), or to cache the song so that plugins can look at it at all (by default, files are not cached to save RAM, as sproxy does not have an inspection interface there is no point). It's also possible for a plugin to respond to a request immediately and either have sproxy not download what the server is sending or download it and send to plugins only. (how turbo mode is done)

%source% tag added for saver2 file naming/dir organization stuff.

a note for people making their own pandora saving dealie: this will save ALL xmlrpcs to the temp(in sproxy dir) folder! End up with stuff like PANDORA_ADDFEEDBACK_i6ul5Gb00Zk7bjl0.XML, PANDORA_GETFRAGMENT_YDaWDZ404gSJ81mu.XML, PANDORA_GETSTATIONS_p5UnMb9P3x5X1RFa.XML, etc.

Also, i noted the introduction of a canListen method - perhaps this is the way they disable listening over 40 hours or something? I'll dissassemble the flash sometime and check. Could always rewrite it to be 1.

I'm looking into making an Imeem saver as i like it when i'm just lookin for a certain song... its metadata is actually very similar to deezer's (very, very similar). But the thing i am stuck on at the moment is that i need to determine how to relate a file to metadata.

Grooveshark, slacker are on the to-do list but grooveshark requires me to check out the metadata again. slacker is ready to implement once i confirm nothing has changed.

[cryptk]

I am running the new version now, everything seems to be working great as always! I will let you know if I run into any problems

[ZigZagJoe]

I am running the new version now, everything seems to be working great as always! I will let you know if I run into any problems

re-download it if you are using turbo - there was a flaw relating to random name generation and turbo mode which caused songs to not be downloaded (due to using the same temp name for all the songs sniffed at once - yay psuedorandom random generator).

I've also added an entry on the status page for it, you now get to know known stations count, how many songs are known, avg download speed, turbo mode status, adn some other bits. Most importantly, it shows pending songs and their statuses such as

CODE

waiting for aart       Daft Punk - "Aerodynamic" on 'Discovery' (ID S87105)
downloading now       Marilyn Manson - "Sweet Dreams (Are Made Of This)" on 'Smells Like Children' (ID S262798)
waiting to download       Future Prophecies - "Concrete Combat" on 'Moving Shadow 05.2 - Live Mix By Calyx' (ID S370446)

More importantly, I've added a throttle function to sproxy. go into the settings, and enter the desired download rate in KB/s (up to your connections max ofc). Do note that throttling assumes an equal bandwidth distribution for all active requests, so the end download speed available to one connection is throttleTarget/numActiveConnections. So with 2 active connections, and a 300kb/s target, each gets about 150kb/s.

Why does this matter? it will cause turbo to mess up for 3 songs or so if throttling is active and you are making other requests via sproxy besides pandora, as it thinks your download speed is much higher. (it uses average of the last 3 songs to compute turbo wait time)

I've made motions towards getting the selective downloading implemented (status page RPC and form, + trayicon).

Also, instead of integrating the DatFile, which is used as a "simple" resource storage mechanism, more elegant than using a zip and some outside library, i've made it into a lib. To use it, reference the dll.

Methods are relatively simple->

static bool DatFile.Verify(string filename): verify the md5 hash of the payload
static void DatFile.Create(IList files, string destfilename, bool useCompression): create a new dat file

non-static methods as follows

Constructor(): create a new datfile object. does nothing important

bool Compressed: true if using gzip compression, false otherwise
int Count: number of entries in dat file read

byte[] GetData(DatEntry entry): decompressing an entry if neccisary, it returns a copy of the data for it
byte[] GetData(string entry): same as above, except uses an entry name instead of entry struct

void Read(string filename): read a datfile from disk. verifies md5 before reading

this[int index]: returns datentry for index
this[string key]: returns datentry for key (entry name)

compression is gzip. in the case of the turbo data, which was 14 mp3 and m4a files (blank), ~500kb uncompressed, it reduced it to 14kb. entries are uncompressed on demand rather than as a whole for sake of performance. in addition, when an entry is uncompressed it remains uncompressed for performance reasons as well. zipping the lot gets it to about 8kb, as the gzip entries don't share dictonaries. Nor is there a way to have c# gzip stream support multiple files in one gzip entry - it's all or nothing.

simple GUI to make dats: http://zzj.itf-inc.c...ds/DatMaker.exe

name of entrys is the filename, minus any slashes (thus C:\Documents and Settings\SProxy.exe becomes SProxy.exe"). Case sensitive.

plugin "interface" is relatively solid now, i'll post about it later. (interface being how saver2 talks to its plugins, not a true c# interface)

[cryptk]

done


Is there a way to make it check if the pending song is already downloaded before it downloads and saves it and if so have pandora move along as fast as possible to get to a yet to be downloaded song?


or is this what you are talking about when you say that you are working towards selective song downloading...

[cryptk]

also, another idea. The status page is awesome except for the fact that you have to refresh it constantly for it to be of any real use. Is there a way that you could make it display in real time? Or maybe even jsut give an option to auto-refresh it every XX seconds?

[ZigZagJoe]

also, another idea. The status page is awesome except for the fact that you have to refresh it constantly for it to be of any real use. Is there a way that you could make it display in real time? Or maybe even jsut give an option to auto-refresh it every XX seconds?

could do yeah

if i were really clever i'd use AJAX but that'd be something of a pain to do as each plugin would need to maintain their own ajax parts

by selective song downloading, i meant having all songs that were sniffed (and not blacklisted, when that is implemented) being stored instead of immediatly ripped and at the user's discretion they are saved or discarded. conversly using the blacklist as a whitelist in this mode means songs that match blacklist patterns are automaticly saved (add a favorite artist or something and go wild)

however, yes, i plan to move duplicate detection to the sniffed stage and if a song has duplicate discard the rest of the server's response and use a 5 second blank file.

Edit: Implemented shunning and uploaded same link. Songs that already exist and can't be upgraded (if enabled) are not downloaded if in turbo mode, and the next song is played in 5 seconds. (no less to be nice to pandora servers)

[cryptk]

very nice man.


The other thing I noticed.

Usually when I stop using Saver, I close the web browser (FFox Portable) then wait for SProxy to finish downloading the current song, then close SProxy.

I saw where if I close the browser while it was downloading a song it would finish downloading the song and then hang saying "waiting for album art" or something to that effect and it would obviously never get it.

Perhaps you could implement a nice close button where SProxy would finish downloading all of the pending songs, download all of their album art and then close nice and clean (not waiting on anything).

Also a small (although cumulative) speed boost would be a check to see if the required album art already exists and if so skip downloading it.

If there are 10 songs it downloads from an album there is no need to download the album art 10 times, only once is needed.

[cryptk]

could do yeah

if i were really clever i'd use AJAX but that'd be something of a pain to do as each plugin would need to maintain their own ajax parts

by selective song downloading, i meant having all songs that were sniffed (and not blacklisted, when that is implemented) being stored instead of immediatly ripped and at the user's discretion they are saved or discarded. conversly using the blacklist as a whitelist in this mode means songs that match blacklist patterns are automaticly saved (add a favorite artist or something and go wild)

however, yes, i plan to move duplicate detection to the sniffed stage and if a song has duplicate discard the rest of the server's response and use a 5 second blank file.

Edit: Implemented shunning and uploaded same link. Songs that already exist and can't be upgraded (if enabled) are not downloaded if in turbo mode, and the next song is played in 5 seconds. (no less to be nice to pandora servers)

I wouldn't think each plugin would need it's own ajax parts. Have Sproxy have the ajax stuff and then each plugin just feeds SProxy what it needs as the contents but I don't know how much you are tied into the development of SProxy...

Even just having it auto-refresh every 20 seconds or so would be great.

[cryptk]

and shameless plug for me.... it is my 25th birthday woot!

[cryptk]

think I may have found a bug....


If I double-click on a line in the SPoxy window it throws an exception and dies. I did it twice to confirm it and I have the logs from both times... Here are the relavant log entries...


FIRST TIME:

CODE

7/12/2009 2:33:56 AM    #182    GET http://audio-sjl-t2-1.pandora.com/access/?version=4&lid=33458331&token=y2d3u%2BAg8OSG1WT1kD2UeYSvWeJZqRcGGFYxZu0BhL1q8d424D2Ieba%2Fq2JGiDkpHTt4q8g95Hp9luXbN52SPtrpgSuOZiXuNSCZGZayBtG8XSbWpZhmIW%2B69qtvEpf46iEegNg%2FcUOq%2Fd6kJW79uhpGAkqfsIK0CiipVqi94JjtnAJHjTYAGuNXIhLdXy2ueMSixhh32LiZtg1wFd6gpqqEE4Y
S%2BO%2BaSQCgdSSxpgIVWrZfa3vZeS6XDwsTd%2Bjk4cqzX3W36lib7sWHz26Po9zjcxlErbH0pdNz%2BSSR%2FFw3kbGbnNUh4Txj1K5HVNNr
7/12/2009 2:33:56 AM    Something has went horribly wrong in SProxy.

Exception Information:
'StartIndex cannot be less than zero.
Parameter name: startIndex'
   at System.String.InternalSubStringWithChecks(Int32 startIndex, Int32 length, Boolean fAlwaysCopy)
   at Proxy.MainForm.listDisplay_DoubleClick(Object sender, EventArgs e) in C:\Documents and Settings\zigzagjoe\Desktop\PS\Saver2\SProxy\MainForm.cs:line 146
   at System.Windows.Forms.Control.OnDoubleClick(EventArgs e)
   at System.Windows.Forms.ListView.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Caused by: mscorlib

7/12/2009 2:33:56 AM    Pandora: sniffed The Veronicas - "Everything I'm Not" on 'The Secret Life Of...' (ID S414226)
7/12/2009 2:33:56 AM    S414226: Shunning song, already exists!
7/12/2009 2:33:57 AM    S58599: Finished downloading
7/12/2009 2:33:58 AM    S58599: Song AART ready, submitting
7/12/2009 2:33:58 AM    Pandora: sniffed White Zombie - "More Human Than Human" on 'Astro-Creep: 2000' (ID S58599)
7/12/2009 2:33:58 AM    Saver2: Song ready: White Zombie - "More Human Than Human" on 'Astro-Creep: 2000' (ID S58599)
7/12/2009 2:33:58 AM    Saver2: Beginning processing of song White Zombie - "More Human Than Human" on 'Astro-Creep: 2000' (ID S58599)
7/12/2009 2:33:58 AM    #183    GET http://images-sjl-1.pandora.com/images/public/amz/8/2/3/1/093624991328_130W_130H.jpg?lid=33458331
7/12/2009 2:33:59 AM    Beginning stop: 'Critical Error'
7/12/2009 2:33:59 AM    SProxy stopping: ran for 00:06:41.9140000
7/12/2009 2:33:59 AM    184 requests serviced, 1 active connections.
7/12/2009 2:33:59 AM    174.3KB sent, 140.6MB received
7/12/2009 2:33:59 AM    Cleaning up pending connections (1)
7/12/2009 2:33:59 AM    Notifying #183
7/12/2009 2:33:59 AM    SProxy stopped.
7/12/2009 2:33:59 AM    Stopped in: 103 ms
7/12/2009 2:33:59 AM    FILE SAVED, Source: Pandora; MP3 format, 6453079 bytes
7/12/2009 2:34:00 AM    Saver2: Tagging aac/mp3 file.
7/12/2009 2:34:00 AM    Saver2: Saved AART to 'F:\Music\White Zombie\Astro-Creep 2000\Astro-Creep 2000.png'
7/12/2009 2:34:00 AM    Saver2: Tagging completed
7/12/2009 2:34:00 AM    Saver2: Processing completed.
7/12/2009 2:34:00 AM    Pandora: AART Reaper stopped
7/12/2009 2:34:01 AM    Saver2: Worker stopped.

SECOND TIME:

CODE

7/12/2009 2:35:03 AM    Something has went horribly wrong in SProxy.

Exception Information:
'StartIndex cannot be less than zero.
Parameter name: startIndex'
   at System.String.InternalSubStringWithChecks(Int32 startIndex, Int32 length, Boolean fAlwaysCopy)
   at Proxy.MainForm.listDisplay_DoubleClick(Object sender, EventArgs e) in C:\Documents and Settings\zigzagjoe\Desktop\PS\Saver2\SProxy\MainForm.cs:line 146
   at System.Windows.Forms.Control.OnDoubleClick(EventArgs e)
   at System.Windows.Forms.ListView.WndProc(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)
   at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

Caused by: mscorlib

7/12/2009 2:35:04 AM    #17    POST http://www.pandora.com/radio/xmlrpc/v24?rid=6249214P&method=sync
7/12/2009 2:35:04 AM    #18    GET http://autocomplete.pandora.com/crossdomain.xml
7/12/2009 2:35:04 AM    #17 POST 208 bytes to www.pandora.com
7/12/2009 2:35:04 AM    Beginning stop: 'Critical Error'
7/12/2009 2:35:04 AM    SProxy stopping: ran for 00:00:08.1080000
7/12/2009 2:35:04 AM    19 requests serviced, 2 active connections.
7/12/2009 2:35:04 AM    12.7KB sent, 1.4MB received
7/12/2009 2:35:04 AM    Cleaning up pending connections (2)
7/12/2009 2:35:04 AM    Notifying #17
7/12/2009 2:35:04 AM    Notifying #18
7/12/2009 2:35:04 AM    Pandora: Inspecting XMLRPC 'sync': http://www.pandora.com/radio/xmlrpc/v24?rid=6249214P&method=sync
7/12/2009 2:35:04 AM    SProxy stopped.
7/12/2009 2:35:04 AM    Stopped in: 158 ms
7/12/2009 2:35:05 AM    Pandora: AART Reaper stopped
7/12/2009 2:35:05 AM    Saver2: Worker stopped.

EDIT: I don't know if this is the cause or not, but this line jumps out

at Proxy.MainForm.listDisplay_DoubleClick(Object sender, EventArgs e) in C:\Documents and Settings\zigzagjoe\Desktop\PS\Saver2\SProxy\MainForm.cs:line 146

It looks like it is maybe looking for a file that is on YOUR desktop rather than where it is actually located at (within the now compiled executable).


I could be wrong though, I am not a programmer, just a web developer, lol.

[ZigZagJoe]

very nice man.


The other thing I noticed.

Usually when I stop using Saver, I close the web browser (FFox Portable) then wait for SProxy to finish downloading the current song, then close SProxy.

I saw where if I close the browser while it was downloading a song it would finish downloading the song and then hang saying "waiting for album art" or something to that effect and it would obviously never get it.

Perhaps you could implement a nice close button where SProxy would finish downloading all of the pending songs, download all of their album art and then close nice and clean (not waiting on anything).

Also a small (although cumulative) speed boost would be a check to see if the required album art already exists and if so skip downloading it.

If there are 10 songs it downloads from an album there is no need to download the album art 10 times, only once is needed.

this is true, didn't think about caching the album art. though, i should note that pandora doesn't playu many songs from the same album (most is 5 ish i've found)

album art reaping is already implemented - it defaults to 30 seconds (pandora tends to start downloading about 5% of song length before the next song comes, so it won't download it needlessly). if no AART in that time, the song is submitted and saver2 will download it.

i do plan on adding a confirmation dialog when songs are still pending when sproxy is closing, a plugin can't stop the closing but it can delay it.

Also, i wrote SProxy as a replacement host for pandorasaver, which used fiddler2. It doesn't have exactly the same featureset as fiddler, namely no built inspecting and stuff like that, but for plugin handling/web request handling it's superior IMO. (fiddler always caches the response before sending, leads to horrible memory use and also causes stuff to time out). It's not quite as complete insofar as HTTP protocol support goes either, It's currently at 1.0 while fiddler is at 1.1. I got kinda stuck implementing chunked transfer encoding and decided meh. SProxy is fine as a global proxy now, so that's good enough for me. (you can plug it in as a proxy for everything in firefox and nothing dies is what i mean)

So yeah, if something needs changing, i can fix it. And that error in particular was related to copying a URL by double-clicking. I changed something a while ago and never fixed that. Woops.

That line just tells me where in the source it died on my computer, it has nothing to do with the executable. in this case, it was a bit to cut off the front of the text (get rid of GET and flags). Except i removed flags, so it was looking for something that was not there.

AJAX: Yeah, true, i could have a thing which checks for all marked cells, but the problem is establishing ownership - SProxy won't know what "songsripped" is nor who does know it; however there is no nice way to signify that in the source. Also, each plugin that wanted to make use of that would have to have a response set for each variable. So it'd get somewhat tedious. I'll probably just go for auto-refresh as a hidden setting. (edit under advanced settings).

Anyways, updated with the song aart caching and that sproxy bugfix. I contemplated having it respond with saved data in pandora, but it's still made the request already so might as well ready the 8kb or so it's sent. Also, album art is saved in png rather than the jpg that pandora needs, so that would have to be converted. But, if saver2 has to download the aart and it's already present it will used the saved one (if that feature is enabled)

Random statistics:
Saver2 contains 2657 lines of source
Pandora contains 850 lines of source
SProxy contains 5005 lines of source
Deezer contains 276 lines of source
DatLib contains 266 lines of source

[cryptk]

yep, the double-click error is gone!


good work man... If I think of anything else or find any other weird bugs, I will let you know.


now if only there was a way around having pandora say "It is taking longer than expected to find your next song"...

[ZigZagJoe]

yep, the double-click error is gone!


good work man... If I think of anything else or find any other weird bugs, I will let you know.


now if only there was a way around having pandora say "It is taking longer than expected to find your next song"...

that would be the definition of "weird bug" ... when does this happen? slow connection?

[cryptk]

very fast connection. It is a thing with pandora, not with your software. I think because of their licensing you can only listen to a song so many times within a certain timespan and my pandora just runs out of music to play since I just leave your program running 24/7 on turbo mode, lol.


I don't think it is a problem on your end though.