Getting Pandora Jar working (again)

[tab126]

Okay - sorry for the posting yet another thread on this, but I sank way too much time into getting this tool running today. I want to save you guys the trouble.

At the time of writing (5th December, 2006), the most recent version of PJ I see on this forum is 7.9 Alpha. I am a developer, so downloaded it to try/use (nothing else had worked).

A lot of people are trying to use PJ in Firefox. After cracking open the source, I seriously doubt that will work. The cache processor for Firefox/Mozilla is a stub returning an empty ArrayList. Since the cache processor seems be the channel through which the pandora XML is found/returned, PJ has no idea where to find the information. The IE cache processor is functional, though, and works for me. There are a few issues w/ getting the right song when I say "grab," but I haven't played with it enough to debug. Actually, as I'm writing this, the WRONG songs are coming in. That's weird...

I tried hijacking the code from the IE processor and using it in the FF/Mozilla processor, but that didn't work. I'll have to actually read the code to figure out what's going on.

Which brings me to a small beef w/ the developer. Great product, but why did you take out apache logging? It is really difficult to get out useful debugging messages. For instance: I want to dump out the HTTP request and XML file.. but doing it to the screen clutters. I love the log4j method of class-level settings for verbosity.

Also, the front-end seems to be a bit out of sync w/ the backend. None of the artist/track/album info ever appears. It just has the normal "unable to locate..." I'm not good enough in javascript to fix that... and I really don't care as long as the mp3's are tagged properly. Just wanted to make the note for people that aren't looking on the back-end.

There seems to be a fair amount of interest in this, though. I've got a little time to kill in the next few days. I'll probably at least get it working for me.

[tab126]

After a little more poking around (pushing file info into the CacheEntry class in the IE cacheProcessor), some of the problems seemed to be caused by the cacheprocessor returning old files. Commenting out (my) line 181 (yours may be a little different - i don't remember how much I've modified the file) of CacheProcessorController seems to help a little. The line in question was a break; statement in an if w/i an while loop. Actually, it is probably line 180 for everyone else.

But again, the behavior of what PJ gets and when is weird. To get around that, I had to thread the downloads. I can now get 4 songs (I assume the last 4 through pandora?). That matches some of the comments in the other threads about 4 songs per xml. But when I change station, do I get another XML file? I still seem to be missing 1 or 2 files here and there.

Also, there seems to be a bit of an issue pressing the "Grab this track" button repeatedly. I'll bet that it has to do w/ PJ being busy downloading while I'm pushing the button additional times. I'm on a really slow connection... and it seems to be cleared up in my threaded version. Is that a bug that somebody should look at?

[CaveMan]

wow... detailed

[falconfox]

sounds good tab126. you seem to have found the problem that i am having, which is it doesn't work in firefox.

are you planning on making this work in IE, or will you help out getting it to work with firefox?

[tab126]

It does work in IE. Firefox is the one that's goofed. I was planning on focusing more on getting exactly what file I wanted (because all of this crap about downloading 4 files and PJ apparently not knowing which is playing seems... fishy).

I don't really know that much about the caching that firefox uses. That said, it is pretty clear that the author doesn't either (not to his fault at all). Suffices to say the word, "clusterfuck" is intermingled with what seems to be a rant in German ... and Ich spreche kein Deutsch. However, since there seems to be demand for firefox, I wouldn't be opposed to trying to hack together a brute-force-esque workaround to try and pull the XML files from the cache. That seems to be what the IE processor is doing anyway. The silver lining in all of this is that I don't see anywhere that the version of flash being used makes a different. Aside from the backend precluding anything from working, I've tried 3 different flash versions and they all look the same to me. This seems very much more of a javascript/java issue. And I'm not exactly sold on the usefulness of the javascript piece.

[cooper]

all of this crap about downloading 4 files and PJ apparently not knowing which is playing seems... fishy

I would've sworn I pointed that out. See the Readme files.

Basically, the Flash applet is supposed to tell the javascript in the browser when which track is done playing. I can't get that bit to work anymore, which greatly limits the functionality I can put into the program. So it just downloads all 4 tracks in the XML and leaves it to you to push out stuff you don't want.

I don't really know that much about the caching that firefox uses. That said, it is pretty clear that the author doesn't either. Suffices to say the word, "clusterfuck" is intermingled with what seems to be a rant in German ... and Ich spreche kein Deutsch.

It's probably Dutch. :)

Mozilla decides, based on the height of the moon, wether the fetched XML should be placed as a separate file in its cache directory, or be embedded in the _CACHE_00X_ files. If you're lucky enough to have a version that places the XML in a separate file, you can use the same CacheProcessor as IE and Opera use.

The _CACHE_MAP_ file and some metadata in _CACHE_00X_ files should tell you where which cache entries are. Problem is that I can't get any sensible data out of it, and when I thought I was getting closer that same block of code that worked... somewhat on Seamonkey gave ludicrous results on FireFox. But by all means have another stab at it.

Stuff to look at:

http://www.securityfocus.com/infocus/1832

http://lxr.mozilla.org/seamonkey/source/netwerk/cache/src/

For that last link, particularly the nsDiskCacheMap.h and nsDiskCacheEntry.h are interesting, but keep in mind that they're already at version 1.10 with their cache format. I don't know which versions of which programs use which version specifically, or what the changes have been over time.

However, since there seems to be demand for firefox, I wouldn't be opposed to trying to hack together a brute-force-esque workaround to try and pull the XML files from the cache. That seems to be what the IE processor is doing anyway.

IE and Opera CacheProcessor (should) pull the youngest file from the browser cache that contains the Pandora XML response which in turn contains the next 4 tracks to play.

The silver lining in all of this is that I don't see anywhere that the version of flash being used makes a different. Aside from the backend precluding anything from working, I've tried 3 different flash versions and they all look the same to me. This seems very much more of a javascript/java issue. And I'm not exactly sold on the usefulness of the javascript piece.

As was written in the Readme.txt this version was to support all browsers that maintain a browser cache and can get the Pandora applet going on any OS that has a 1.4 and up JVM.

The Readme.dev file should contain some thoughts on where the program stands.

I tried to upload the sources into the Hak.5 Subversion repository yesterday, but it seems to have disappeared. The guy running it hasn't yet gotten back to me.

As for the java script: exactly. This is a case of the previous version biting me in the ass. I SUCK at JavaScript, and have no intention to pick it up. I tried to keep the Javascript as much identical to the previous version as possible in the hopes that some of the UI stuff could be reused, but I'm not getting the callbacks from the applet anymore. If I did, I would know which track from the XML file was being played, when a new track finished playing, etc so I would know when would be a good time to grab, and all those other nifty things like getting some stats placed in Last.FM that people seem to really want.

I took out Apache logging as I felt it was excess fluff the user wouldn't be particularly interesting in, and any dev could put back at his own leasure in under 3 seconds.

I've never changed channel on Pandora, so I don't know what the program would do. I think I read in the threads of one of the other Pandora grabbers that this does indeed result in a new XML file.

I never tried re-calling the grab url while a download was in progress. As the server isn't single-threaded I would expect it to go out and look for a new XML, but since the last search yielded the correct XML the minimum time a new file has to have will result in no new Pandora XML being found.

[tab126]

I would've sworn I pointed that out. See the Readme files.

You probably did (and reading them, you seem to)... but figuring the whole thing out from code is MUCH more entertaining than reading. I'm notoriously bad for not reading doc too carefully. I blame the business world where doc is at least 3 months behind the code.

It's probably Dutch.

I haven't made it over to the Netherlands, yet, so I haven't learned the basics of Dutch from my buddies. It is on my to-do list, though.

But back to the topic ... supporting both methods of caching in FF seems doable, given time. That being said, I'm not sure that I want to be the one to throw time at it where there are more noble causes.

I SUCK at JavaScript, and have no intention to pick it up.

Unfortunately, we're in the same boat here. I, too, am a backend guy. However, javascript has bitten me enough times so that I'm not opposed to messing with it. I just wish there were printlines. I think this is the noble cause towards which time should be devoted. Too bad I don't have a clue how the scripts are looking up the information in the first place.

Apache logging - point taken. It just sucks that there aren't debugging logs in place already (I am REALLY lazy).

Changing channels in pandora does download a new XML. This is how I've lost a song or two.

I never tried re-calling the grab url while a download was in progress. As the server isn't single-threaded I would expect it to go out and look for a new XML, but since the last search yielded the correct XML the minimum time a new file has to have will result in no new Pandora XML being found.

This raises a "thread-safety" flag with me. In looking at the code, I don't remember seeing sync blocks anywhere. If multiple server threads are pulling from the same cache db, what's to keep things from breaking?

Anyway, this seems very similar to what is being written about pandora unleashed. The main reason I wanted this program is so I could hit the button and download the 1 song. Grabbing everything is great, but less is more, to me. I should spend some time on the front-end. But I'll do it later... I have to run off to immigration before they deport me.

[cooper]

supporting both methods of caching in FF seems doable, given time. That being said, I'm not sure that I want to be the one to throw time at it where there are more noble causes.

Unless someone can get the Flash applet callback thing working again, the prospect of this particular program is kinda poor.

If it does get fixed, things are really looking up though.

I, too, am a backend guy. However, javascript has bitten me enough times so that I'm not opposed to messing with it. I just wish there were printlines. I think this is the noble cause towards which time should be devoted. Too bad I don't have a clue how the scripts are looking up the information in the first place.

The way it works in PJ 7.4 is that the Flash applet should look in the JavaScript environment (wherever that may be) for an object named Pandora that contains a fireEvent method. If it finds this, it will call this method with 2 parameters, the first being a string that will allow you to identify the object that is the second parameter.

When I run PJ 7.9a in FF or Seamonkey that has Venkman (the Mozilla JavaScript debugger) going, as soon as the Pandora Flash applet loads, the javascript that contains the Pandora object disappears. It gets unloaded or something. No idea why that's happening. I tried moving the object declaration to a different javascript file, and sure enough that other file got unloaded (or more specifically, it disappeared from Venkman's view). I don't know how they do it, but that's what's happening.

This raises a "thread-safety" flag with me. In looking at the code, I don't remember seeing sync blocks anywhere. If multiple server threads are pulling from the same cache db, what's to keep things from breaking?

I didn't bother myself with this that much. I'm assuming this is being used in a somewhat friendly environment, and the worst breakage that you'll conceivably incur is that an MP3 file gets downloaded more than once. You'll have to be fairly quick to open this particular hole though. Basically in the time it takes to scan the directory for new entries, as every rescan called after that will only include the files that have been added since the last time it was called. And if a file to be downloaded already exists when the download is started, it will skip the file, so again, while the hole is present, it's small, and given the expected use of this program, not particularly worrying.

Anyway, this seems very similar to what is being written about pandora unleashed. The main reason I wanted this program is so I could hit the button and download the 1 song. Grabbing everything is great, but less is more, to me. I should spend some time on the front-end. But I'll do it later... I have to run off to immigration before they deport me.

That was kinda the goal, but since the front-end isn't telling the backend which song was just played (since the Flash applet refuses to divulge this information) I'm stuck. The idea was to check for a new Pandora XML when each song finished playing, and to keep the content of the last 2 XML files that got parsed on file. When a grab is called, the artist name and track name is provided in that second object passed to the fireEvent method of the Pandora JavaScript object, which we could then look up in the list, and download the associated MP3 URL for.

[tab126]

Okay, so I hijacked the player page and stored it in the webroot to start serving the pages/scripts out of there. I figured that it would give me a lot more control over how stuff was working.

I also put in a few of my own javascript functions that would attempt to a) create an instance of a pandora object (as defined in the public API, not the scripts included w/ PJ) and b) register my custom function as an event handler for the callback.

So none of that worked. Well, I've got the object, and I think I'm registered... but the callback never comes. But my object stays alive! I'm not sure if I've got the name wrong or what. I'm hoping somebody that's better with javascript could tell me how to figure out when the shockwave calls a defined or undefined javascript function. Any takers on that one?

But I took a break... I wrote another requestProcessor to handle generic http requests so it could act as a proxy and unleashed it upon www.haoting.com. From that site, you can stream .wma's of a lot of music that you actually select. I didn't quite finish, but it was a fun detour. But if you want to check the site out, make sure you have the chinese character set installed (to make the gibberish pretty).

Anyway - that was just an update on things I've tried. Hopefully somebody else will get an idea/piggy-back.

[lgchicken]

could someone make a firefox plugin which downloads the cache of the mp3 file, then renames it and tags it up with the artist and song name?

a sure way of getting the right cache file (im using a cache checker ff plugin) is to open the latest cache file with 'token' in it's name.

That forwards to a extensionless file which is really an mp3.

[tab126]

That's a good idea. I'd do it if I knew the FF extension framework. If anyone knows it -- I'll help. Just IM me. or reply to this topic.

[tab126]

Alright -- a few more hours thrown at this, and I finally got the whole system working (for me). There were a number of issues and changes that I made. Some may be needed, some may not be. But here's the deal:

First I pulled down http://www.pandora.com/?cmd=mini and stored it as player.html in the webroot folder of my PJ distro. I had to re-do all of the links to make them work properly (including the one in index.html to make it refer to the new page). Note that you have to restart your PJ server when adding any file under the webroot. Took me a hour+ to figure that one out.

In that file, I changed tuner.js to be served locally (so that i could debug it). This seemed to keep the errors i was having from happening. That is the part that I don't really think is needed. Here is the part that I think is:

Instead of bashing the existing Pandora API, I wrote a function to register a callback. The whole file is basically the public pandora API (to make sure it is defined) plus:

function doSongCallback(eventData) {

    //eventData is an array with

    //songName, artistName, songURL, and artURL

    alert('name: '+eventData.songName );    

    pandoraSongPlayed(eventData.songName, eventData.artistName);    

}



var registered = 0;

function tabRegister() {



    if(registered != 0) return;



    if (typeof(Pandora) != "undefined") {

        doSongCallback('blah','blah');

        //Pandora.eventHandlers['SongPlayed']=doSongCallback;

        //Pandora.setEventHandler('SongPlayed',function(eventName, arg) {alert(eventName);};);

        Pandora.setEventHandler('SongPlayed',doSongCallback);

        alert('Callbacks registered');

        registered = 1;

    }

}

To get it to work, I modified the first line of downloadMP3() to call tabRegister(). The effect is that you have to hit the "grab" button once to register everything, then you can happily rip tunes. I could probably make it a bit slicker with an onload() function or something, but i really don't see the point.

On the backend, I made a change to GrabMP3ProcessRequestHandler.processPandoraTrackData() to make it select only the song you want to download (as opposed to the four found). The data should come in the request as "artist" and "title" .. see above code segment for the info delivered by the callback.

And the major issue on the backend (which I suspect is the reason some people are failing to download w/o an errors) was character encoding. Request.java's urlDecode method should use "UTF-8" as the character set, not "UTF".

Recompile/build and you should be set.

Now, for the issues:

1) Doesn't work in Firefox. The callbacks never actually call Pandora object's eventfire function. If you do what I said about tuner.js, you could hijack the onTunerEvent method to do that, but I don't think that function has enough data to do much. There is a lot to figure out there.

2) This version seems to pull from the web rather than cache. While this wouldn't be an issue on most connections, my connect is really slow.

3) None of the track info comes through on the PJ page. As far as I can see, it is because the processors aren't set up to handle those URL's.. it might be because I downloaded the page containing the pandora player, though. Personally, I don't care much to see an artist bio. I'm looking at other stuff 99% of the time.

Hope that helps.

[lgchicken]

can you upload your altered version?

[tab126]

If somebody wants to put up some space, sure. It is kind of an uber-develepor version, though. Massive logging, a few popups (or maybe i took those out). How it works is NOT how the normal of PJ works. But yeah - tell me where to put it.