(Pandora) Saver2

[ZigZagJoe]

Thank you so much. My problem seems fixed so far!

I've mentioned this sometime before, but have you ever considered adding ripping support for Spotify? It's pretty amazing. If you think that you would be interested in I can get you an account, let me know.

If you can get an account i will take a look at it.

Edit: Took a look at it. Uses https for normal communication and kills itself when a debugger is detected. There's no sane way to save music from it, really. Seems to do a checksum of its executable as well.

[bulletp2000]

I strongly suggest you upgrade to the new versions because you're using a version of SP which is over half a year old... some odd bugs/iffy implementations such as what you are running into have been fixed - that being one of the biggest. I'm honestly at a bit of a loss at why you are using such an odd setup...

Because I have custom code that needs to be invoked. And I don’t understand enough of the new architecture to move my code (if it even can be done). Is the source to all of the new stuff published?

Here’s what I understand so far . . .

---------------------------------------

With the PSaver1 code, the executable SProxy.exe was launched and then it:

1) (somehow) created the Deezer, Grooveshark, and Pandora menus

2) (somehow) loaded the PandoraSaver.dll

I wrote (somehow), b/c I’m a little fuzzy on the details. Anyway, thereafter the Firefox client would load Pandora.com’s flash plug-in which re-routed calls (via proxy) to PandoraSaver code which took care of:

a) Writing the music files

B) transcoding if necessary

c) blacklists (custom code)

d) rating songs positive/negative based on mp3 tag (custom code)

Note that I use all of a,b,c, and d. So far, so good, except that my code base is getting old . . .

---------------------------------------

With the new Saver2 code, the executable SProxy.exe still launches the whole thing:

1) (somehow) creates the Saver2 menu

2) (somehow) loads the Saver2 plug-in

Now where do the following activities occur?

Downloading the music

Writing the music files

Transcoding if necessary

blacklists (custom code)

rating songs (custom code)

In other words, how much of the code that I have in PandorSaver would need to move? And where would it move to? I don't understand enough of the boundary between SProxy and the plug-in to know where my code now lives.

I know that you re-architected the thing to allow for better plug-ins, but I didn’t see any development documentation on the saver2 web site.

My apologies if this has already been FAQd.

[ZigZagJoe]

http://hak5.org/forums/index.php?showtopic...st&p=151300 has some of this.

also, i'm on irc right now, if you want to ask questions there: http://zzj.itf-inc.com/s2/irc/ or irc.coldfront.net and join the #saver2 channel.

there is also dev stuff at http://zzj.itf-inc.com/s2/S2dev.zip

SProxy checks any dlls in the same folder as it for classes implementing the SPlugin interface (should be ISPlugin, i guess?). Any dlls which do have the ShouldLoad() method called, which is where a plugin should check if its dependencies are available (say invoke dlls and such).

After all plugins are loaded, they then have the Start() method called. If an exception is thrown or a nonzero code is returned, Unload() is called.

When Start() is called for saver2, first it sets up the basic menu items. It then verifies the presence of mplayer.exe, mp4tags.exe, and lame.exe.

It then sends a notify to ALL sproxy plugins (the PluginMgr.NotifyAll method) with the arg NOTIFY_SAVER2_PLUGIN_INIT [1000] and object of this (the saver class instance). any plugin which returns 1000 is added to a list by PluginMgr.NotifyAll and is considered to be a Saver plugin meaning it will get various notifications as to the state of songs.

If saver plugins were found, the worker thread is started and then the plugins are notified with NOTIFY_SAVER2_PLUGIN_START and null.

In the SaverPlugin class, which all plugins should extend, if AddMenuLink or AddConfLink are set to true, menuitems are added for those two (a menuitem which launches the SaverSvcUrl and a menuitem which calls DoConfig(), respectively).

SetupMenu is then called, if AddMenuLink || AddConfLink the menuitem it is passed with is our submenu (named {SaverName}) otherwise it is called with the Saver2 menuitem. If a sub menu was created, it is then added to Saver2's menu.

Finally, OnStarted() is called in which you should do any setup of your own.

Once all this is done, Saver2 sorts all the acquired menu items alphabetically, grouping menus with subitems first.

Saver2 then launches the autorun command (if set). If something goes horribly wrong, savers are notified with NOTIFY_SAVER2_CRIT_ERROR, which calls Unload and causes the plugin to no longer look at any responses from SProxy.

Saver2 is now started and sproxy has finished loading.

A request is made through the proxy. All SPlugins have their TamperBefore(Proxy.Session s, ref Proxy.ProxyHandlingMode mode) method called. If a plugin implements SaverPlugin, it will need to set mode.DoPostTamper = true for the SniffMusic method to be called. TamperBefore has the headers the server sent back available in the session object but not the response body.

SniffMusic is called when a request which a plugin has set DoPostTamper is called and the plugin in question has been started and is enabled (has __(name)_Enabled set to true). In here, a plugin should inspect the contents of the session and pull out metadata or songdata or what not. When it finds song data, it should save said song data to disk. Easier to explain this in code:

SongInfo s = NewSong() // method of SaverPlugin, generates a new song belonging to this class

// you really should create the songinfo objects when the metadata is sniffed, though and fields like artist title etc need to be populated

session.SaveResponseBody(s.TempFile);

Parent.AddSong(s); // Parent is the Saver plugin's instance, set by the SaverPlugin base class when notified.

The plugin's involvement ends here. It found a song, and passed both the data of said song (mp3, etc) along with its metadata to saver2.

Some plugin bits:

// THIS MUST BE PRESENT!

protected override Proxy.CallResult GetEnabledInfo() {

return new Proxy.CallResult(Settings.GetBool("Enabled", true) ? 1 : 0, Settings.GetMangledName("Enabled"));

}

.ctor () {

Settings.Default("Enabled", true); // adds key to config.

SaverName = "Example";

SaverVersion = "1.0";

SaverSvcUrl = "http://example.com";

}

I recommend you take a look at SimplePluginExample.cs

AddSong, in saver2 then does some sanity checks and then calls Prepare() which figures out paths and what not, if there is a duplicate, etc. if a song already exists and will not be replacing another song, savers are notified with NOTIFY_SAVER2_SONG_SAVED and the song, the song is then disposed (temp file deleted along with album art). The song disappears into the abyss, as it were.

Otherwise the song goes through a blacklist stub which does nothing. If auto rip is on or the song is marked for immediate download (ImmediateDownload) which is the case when a song which is not done downloading in pandora but is rated good it then goes on to ProcessSong.

If not, the song is then added to the pending songs list, a tray notification displayed if enabled, and savers notified with NOTIFY_SAVER2_SONG_READY and the song.

In ProcessSong: NotifySavers(NOTIFY_SAVER2_SONG_PROC, song);

the dir in which the song will go is created, the duplicate deleted (if this is the case) and references removed.

If the song will be encoded, it it sent to the encodermanager which will either begin encoding it immediately (if a processor is available) otherwise FinishSong is called. Songs already in the encoder's format will not be reencoded.

FinishSong moves Song.TempFile to Song.FileName (the formatted name), writes it to the newmusic playlist, tags it, and then NotifySavers(NOTIFY_SAVER2_SONG_SAVED, song)

shit that was a lot of text.

if in wait for approval mode, ApproveSong() must be called to trigger saving. You can either do this by calling the method directly, or opening a url which contains "?saver/catch&song=[sONG ID]".

Dismissal of a song causes it to be deleted as well (the temp file).

Hopefully this will get you started somewhat. Sounds a lot worse than it is really.

[bulletp2000]

Did you change something in the Pandora Client (either Jan 22 or Jan 24 release) that now ties it to a specify proxy server?

The client was working great for me using the old SProxy.exe. With the newest client, it will not work anymore.

First I get "failed to establish connection". Then when I changed the listening ports for both version of SProxy to be the same port (I know - bad idea) - I get "failed to execute XMLRPC"

Also, how does the client specify which port it is listening on?

I know that you intend to bundle your Pandora Client and Saver2 together, but there really is no reason that they should be coupled. Theoretically, someone could use Fiddler behind the client as the proxy, or go to the internet directly.

[ZigZagJoe]

Plugins: Downloading the music and metadata

Saver2: Writing the music files

Saver2: Transcoding if necessary

Technically saver2, but it's not implemented: Blacklisting (custom code)

N/A: rating songs (custom code)

Saver2 has no concept of rating songs, and the pandora plugin can read feedback but not send it. this would require an authtoken + postdata: the url alone will just generate an exception on pandora's end. Pandora will approve songs based on feedback if you have enabled it. (hmm, just realized there is no facility to dismiss songs based on negative feedback).

One could implement a pseudo blacklist by making generous use of the approve and dismiss methods with the song's ID. It's currently not implemented because 1. noone has requested it so far and 2. i'm not happy with the implementation in the original pandorasaver. Couldn't deal with other organization and naming formats, for one.

[ZigZagJoe]

Did you change something in the Pandora Client (either Jan 22 or Jan 24 release) that now ties it to a specify proxy server?

The client was working great for me using the old SProxy.exe. With the newest client, it will not work anymore.

First I get "failed to establish connection". Then when I changed the listening ports for both version of SProxy to be the same port (I know - bad idea) - I get "failed to execute XMLRPC"

Also, how does the client specify which port it is listening on?

I know that you intend to bundle your Pandora Client and Saver2 together, but there really is no reason that they should be coupled. Theoretically, someone could use Fiddler behind the client as the proxy, or go to the internet directly.

Actually, I'm leaning towards the contrary: separating the client and saver2/sproxy. It's mostly a matter of convenience that they are together for now.

Also, changing the listening port without restarting sproxy will not do anything and two servers can not listen on the same port.

The client does not listen on ports. It's a client, it knows how to play music from pandora, and that's about it. It has no concept of saving music (other than sending an approval to saver2 if possible).

It should be able to go to the internet directly.. testing it here, it is working fine. Make sure you have your proxy settings correct.

[bulletp2000]

The client does not listen on ports. It's a client, it knows how to play music from pandora, and that's about it. It has no concept of saving music (other than sending an approval to saver2 if possible).

You were right, I had to re-start SProxy.exe. But that still doesn't explain why I had to reconfigure the port from 8888 to 8887.

Most application that send requests to the internet establishe a connection based on some configuration criteria. The browsers also allow you to use a proxy (which is nice). Firefox has its own settings, IE has its settings, Chrome, etc. This is how you can use one proxy for IE, and another for Firefox.

If your Pandora Player doesn't have its own communication configuration settings, it must be picking them up by default from one of the other browser configs. IE, perhaps?

[ZigZagJoe]

You were right, I had to re-start SProxy.exe. But that still doesn't explain why I had to reconfigure the port from 8888 to 8887.

Most application that send requests to the internet establishe a connection based on some configuration criteria. The browsers also allow you to use a proxy (which is nice). Firefox has its own settings, IE has its settings, Chrome, etc. This is how you can use one proxy for IE, and another for Firefox.

If your Pandora Player doesn't have its own communication configuration settings, it must be picking them up by default from one of the other browser configs. IE, perhaps?

Yes, it picks them up from IE. They can be manually configured but when not in sproxy they are not, so the base wininet settings are used. Fiddler may not be causing wininet to update proxy settings, or i know for a fact the old version of sproxy don't when set IE proxies are set , so until you relaunch the browser or go into the internet settings they will be pointing at a port that has no server on it.

(settings could be changed by editing the various httpwebrequests to force a certain one on or off - this happens when running under sproxy. there is no external way to do so, however.)

[bulletp2000]

They can be manually configured (by editing the various httpwebrequests to force a certain one on or off - this happens when running under sproxy) but when not in sproxy they are not.

I'm sorry. I didn't understand what you meant by this sentence.

[ZigZagJoe]

I'm sorry. I didn't understand what you meant by this sentence.

In the source, I could force HttpWebRequests to use a certain proxy or not. It doesn't really have any bearing on what you asked, though.

@ USSRNUKE, sorry, i was AFK. If you are using the browserPAC method, you must add a section for youtube in it. Should be fairly obvious how to do it, though; just hit reload in proxy settings (in firefox) when you have done. Also, don't check the second option in youtube settings unless you want to get music only.

[thaeds]

So now my pandora works great over network share. It doesn't stop downloading, but it still sometimes doesn't rename *.mp3.tmp files to .mp3, even though it puts them into the correct folder. As far as I can tell it's exactly right once I manually rename it. Almost always has cover art, has artist and album tags, ect. Except it seems as if the *.tmp.mp3 songs may have 3-10 seconds of blank sound on the end of them... I can't tell for sure, as some songs are meant to have blank sound on the end, but it seems like every .mp3.tmp does, while other songs in the same album do not.

So I'm just writing here to check if it's okay for me to just rename *.mp3.tmp files to *.mp3, and if there is any reason why they may have some empty space on the end. Many thanks!

[ZigZagJoe]

So now my pandora works great over network share. It doesn't stop downloading, but it still sometimes doesn't rename *.mp3.tmp files to .mp3, even though it puts them into the correct folder. As far as I can tell it's exactly right once I manually rename it. Almost always has cover art, has artist and album tags, ect. Except it seems as if the *.tmp.mp3 songs may have 3-10 seconds of blank sound on the end of them... I can't tell for sure, as some songs are meant to have blank sound on the end, but it seems like every .mp3.tmp does, while other songs in the same album do not.

So I'm just writing here to check if it's okay for me to just rename *.mp3.tmp files to *.mp3, and if there is any reason why they may have some empty space on the end. Many thanks!

Any empty space is likely there from pandora unless it is actual empty space like zeros when seen in a hex editor which would be a bit odd. Yeah, it's fine to rename them back to mp3s - the tmp files are the finished product with the new tag + the mp3 content of the original which are then renamed to the final extension.

I'm guessing there is some weird lag related to the remotely mounted share: the Move command isn't throwing an exception but is getting eaten somewhere along the line.

[thaeds]

Any empty space is likely there from pandora unless it is actual empty space like zeros when seen in a hex editor which would be a bit odd. Yeah, it's fine to rename them back to mp3s - the tmp files are the finished product with the new tag + the mp3 content of the original which are then renamed to the final extension.

I'm guessing there is some weird lag related to the remotely mounted share: the Move command isn't throwing an exception but is getting eaten somewhere along the line.

Very cool, I wrote a quick and dirty macro to rename them, so no complaints here. Thanks!

[XDS]

If the irc client is standalone and using java could you set something up to connect to freenode with a different UID and ident ?

[ZigZagJoe]

If the irc client is standalone and using java could you set something up to connect to freenode with a different UID and ident ?

be possible yeah

[XDS]

be possible yeah

Freenode is logistically blocking mibbit thats why i ask.

Maybe a cloan is in order. :)

[ZigZagJoe]

Freenode is logistically blocking mibbit thats why i ask.

Maybe a cloan is in order. :)

http://www.pjirc.com/main.php is the applet that is being used for it + some javascript for the entry field

[XDS]

SDK for CAD coming in 2 days! I made the payment 2 hours ago. :)))))))

:P

[XDS]

you have PM :)

Please let me know as soon as you can when you receive it.

[ZigZagJoe]

you have PM :)

Please let me know as soon as you can when you receive it.

Right, I've been taking a look at it. Fairly simple, just need to eat a few messages (prev), so i went and translated the various values into declarations in my source. Then, looking again at the register bit.... it needs either a title (but our title changes! nope...) or a class name (fuuuuuck). The later is bad because all windows in C# have a weirdshit classname and there is bluntly put no way to make it even semi distinct (all SProxy windows/controls share the same class name - thus, it's useless).

i'm poking at some sort of C++/CLR faggotry which will create a hidden window and then call a callback into C# where the various messages are handled (idea being, give this hidden window a proper name/class). But other than this it's disable changing of title text (i'd really rather not, if it even works) or nothing.

[XDS]

Right, I've been taking a look at it. Fairly simple, just need to eat a few messages (prev), so i went and translated the various values into declarations in my source. Then, looking again at the register bit.... it needs either a title (but our title changes! nope...) or a class name (fuuuuuck). The later is bad because all windows in C# have a weirdshit classname and there is bluntly put no way to make it even semi distinct (all SProxy windows/controls share the same class name - thus, it's useless).

i'm poking at some sort of C++/CLR faggotry which will create a hidden window and then call a callback into C# where the various messages are handled (idea being, give this hidden window a proper name/class). But other than this it's disable changing of title text (i'd really rather not, if it even works) or nothing.

Now it looks like there are more pipes than we need/want with it. I'm beginning to think it might be more trouble than its worth. It's got .NET and VB roots but alas.

Some things I noticed about it too....

You have to change a song in order for the values to print. So either the first song won't print if the player is started first. Or the user would have to wait for the next song to start.

It also doesn't have Last.FM scrobbling like i thought :(

I was however able to find a Last.FM scrobbler that was just put together for grooveshark.

But it doesn't have live listening support.

UGH.

The sauce, its weak.

:(

ps: I'm started to get really aggravated with all this cold weather too. wtf.

Anyways....I'll talk to carlos and see what he can brew up for us all.

[ZigZagJoe]

Now it looks like there are more pipes than we need/want with it. I'm beginning to think it might be more trouble than its worth. It's got .NET and VB roots but alas.

The sauce, its week.

:(

ps: I'm started to get really aggravated with all this cold weather too. wtf.

Anyways....I'll talk to carlos and see what he can brew up for us all.

It's not (really) CAD's fault - it's .NET's. The API that CAD uses is about what i expected, sending messages via SendMessage(). Not terribly hard to implement (and truthfully, I'd rather it than the alternatives). But since it wants to be able to send messages back, you need to either provide a non-changing window title (loosing station name and paused state from the client) or a window class name.

However, .NET is fail in that it has what's as i understand it a bug in the window creation code, which as such disables the creation of somewhat different window class names instead getting a nice exception. And trying to register a global name would be seriously bad practice on top of i have no idea how to do it.

So, idea was, have a C++ program (utilizing the CLR) in which I _do_ know how to create a window create it, hide it, but give it a proper class name then have an eventhandler which will call back into C# (and the client) where the client can do whatever it likes, when a window message is recieved. Rather convoluted, but it should work.

Problem i'm facing now is you can only define a static wndproc (ugh) when doing a register, so i need a way to get back to the eventhandler which was registered in the class. The method i used in c++, keep a pointer to the object in the window userdata won't work here, looks like. Or write it all statically.

I found a thing that does the same thing in C#, but the problem remains: getting back to the original. Might have to do some douchebaggery with a static list or summat.

[XDS]

It's not (really) CAD's fault - it's .NET's. The API that CAD uses is about what i expected, sending messages via SendMessage(). Not terribly hard to implement (and truthfully, I'd rather it than the alternatives). But since it wants to be able to send messages back, you need to either provide a non-changing window title (loosing station name and paused state from the client) or a window class name.

However, .NET is fail in that it has what's as i understand it a bug in the window creation code, which as such disables the creation of somewhat different window class names instead getting a nice exception. And trying to register a global name would be seriously bad practice on top of i have no idea how to do it.

So, idea was, have a C++ program (utilizing the CLR) in which I _do_ know how to create a window create it, hide it, but give it a proper class name then have an eventhandler which will call back into C# (and the client) where the client can do whatever it likes, when a window message is recieved. Rather convoluted, but it should work.

Problem i'm facing now is you can only define a static wndproc (ugh) when doing a register, so i need a way to get back to the eventhandler which was registered in the class. The method i used in c++, keep a pointer to the object in the window userdata won't work here, looks like. Or write it all statically.

...See where you are going with this. (Completely took for granted the station names and event handlers)

It's not easy digesting new code for me, you seem to have a killer instinct for this. o.O

You got one over one me :)

From what I can tell you want an extra process to handle the callbacks, now that might mean re-integrating controls back to the main client, and I know thats something you don't want to do.

Is that going to be a dormant process once the post messages are received and responded to ?

I don't mind the extra pipes as long as there is no water constantly flowing threw them.

Anyways, I Suppose it doesn't really matter much since it would be very bleeding edge.

.NET is waaaaaay over my head, so you got me.

I can just barely understand VB and a touch of C.

I'll have to look at the SDK some more and do some reading up on how .NET works. As well as doing some more digesting of CAD's overall coding structure. =/ We'll see what happens.

Thanks by the way!

~X

[ZigZagJoe]

Is that going to be a dormant process once the post messages are received and responded to ?

I don't mind the extra pipes as long as there is no water constantly flowing threw them.

Anyways, I Suppose it doesn't really matter much since it would be very bleeding edge.

.NET is waaaaaay over my head, so you got me.

I can just barely understand VB and a touch of C.

I'll have to look at the code some more and do some reading up on how .NET works.

and thanks by the way!

it's not a new process - it's just a hidden window belonging to the same process.

[XDS]

it's not a new process - it's just a hidden window belonging to the same process.

Ah , sorry for the misunderstanding there. , Well i'm glad you finally where able to get S2 talking with CAD. :)

I'm still studying up on all of this.... schtuff! < sigh >

I'm a quick study but even with 10 cups of coffee I don't know if I will be able to catch up with you :)

Sunday is my birthday, so i won't be around to much. See you guys all monday and good luck!