another way to time shift ...(proxy timeshift see top pg2)

[JohnnyChimpo]

i use linux and pandor's jar doesn't seem to work under linux (or under XP for me at least)....

reviewed lots of stuff - this is what I have working (in very cobbled together fashion):

• use a local proxy to save all req/resp to/from pandora to files (similarly named pairs). Currently I am using webscarab.

• run pandora basically normally -- but thru the proxy

• gather all the pair files and look for the v8 files that contain xml -- They will have GETs with a URL string that contains "radio/xmlrpc/v8"

• for each v8 request find its response - which is a hunk of XML

• for each <struct /> in the xml locate the NV pair "audioURL". Its value will be URL for a get for a MP3 file

• locate the req/resp file pair that contains the value of "audioURL". When you find this, you have all the attributes of the song in the XML <struct /> and the response file containing the MP3.

-

• extract song/trac attributes from <struct />, tag and tuck the mp3 file away....

I have a java hack that does about 1/2 of this and seems to work...

would like to hear feed back or whatever???

incidentally - for me - pandora's jar works mostly accept for the inability to get the song/artist in the javascript (which means it doesn't work much at all)....However, gLastSongPlayed in one of the JS files does return some string - which I think is the value "matchingSeed" in the <struct /> mentioned above.....if this is so (someone care to find out??) -- then pandora's jar could maybe patched up...

[cooper]

incidentally - for me - pandora's jar works mostly accept for the inability to get the song/artist in the javascript (which means it doesn't work much at all)....However, gLastSongPlayed in one of the JS files does return some string - which I think is the value "matchingSeed" in the <struct /> mentioned above.....if this is so (someone care to find out??) -- then pandora's jar could maybe patched up...

...which is what I'm doing. Time is the main issue.

I've rewritten the server so that you can run it using java 1.4 and up, that it uses the XML file from the temp file folder from either Opera, IE or the Mozilla family and uses it to grab the MP3, name it, tag it and store it. It will work on any OS that uses one of those browsers too, so Mac, Linux, *BSD, Solaris etc. users will be able to use this one aswell.

I still need to add the downloading of the MP3 someplace configurable and the tagging of the file, both of which can be gleaned from the last release. Once that is in (today) I'm going to do a pre-release so you people can tell me how it works. I'll add Last.FM registration, the fetching of the band's BIO and the grabbing of the album cover from Pandora later.

[homebrew Hacks]

Last.FM has released yesterday a new Flash-Player for their radio-streams. The mp3´s and a playlist xml-file are stored in the same folder as for pandora. Can you implement a feature to save and tag the mp3 from last.fm streams too?

[Zoltan]

great! but I wonder why the playlist-file does not get extended when all the files in it have been played...

Man it would be so great if we could just know what the format of the string for the md5-hash is... You could just type in a Song and let it download from last.fm or pandora! ;-P But brutforcing would take some months or years I guess! ;-)

And of course it would be illegal...

[cooper]

And of course it would be illegal...

Only if you get caught. :twisted:

[cooper]

I still need to add the downloading of the MP3 someplace configurable and the tagging of the file, both of which can be gleaned from the last release. Once that is in (today) I'm going to do a pre-release so you people can tell me how it works.

Well, I can grab, you can go completely nuts with the destination filename and I can tag it once it's in with artist, album, track, tracknumber and release year.

Only problem is that currently in IE I need to manually call the grab url. Seems like the Javascript isn't working out so well in there. I'll investigate that tomorrow and hopefully release then.

Due to this not working right I didn't get to test the Moz browsers so no idea how well those work. More on that tomorrow aswell.

I'm currently looking at just shy of 1.5 MB to download for the zipped program. You need a few more kb to get the sources and stuff. But I'll get into that in more detail at release time.

[JohnnyChimpo]

use jdk 1.5x

seems ok - did notice some stuff that doesn't look quite rite - but it is a start...

use like:

                        String v8 = <contains the v8 xml>;

            PandoraTrackBunch ptb = new PandoraTrackBunch(v8);

            Iterator<Map> iter = ptb.iterator();

            while(iter.hasNext())

            {

                Map m = iter.next();

                String trackName = (String)m.get("songTitle");

                String artistName = (String)m.get("artistSummary");

                System.out.println("song--artist:  " + trackName + "--" + artistName); 

            }

fyi: each map has an nv pair matching the names and values in XML; per struct...

parser code

-----------------------------

package pir;

import java.util.*;

import java.io.*;

import org.w3c.dom.*;

import javax.xml.parsers.*;

import org.xml.sax.*;



public class PandoraTrackBunch

{



    private ArrayList<Map> listOfTrackMaps = new ArrayList<Map>(5);

    

    public PandoraTrackBunch(String xml)

        throws SAXException, ParserConfigurationException, IOException

    {

        Document dom = null; 

        DocumentBuilder parser = null;

        DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();

        dbf.setValidating(false);

        parser = dbf.newDocumentBuilder();

        dom = parser.parse(new InputSource(new StringReader(xml)));

        

        Node structNode = null; 

        NodeList structList = dom.getElementsByTagName("struct");

        int len = structList.getLength();

        for(int ii = 0;  ii < len; ii++)

        {

            structNode = structList.item(ii);

            listOfTrackMaps.add(makePandoraMapFromNode(structNode));

        }        

    }

    

    private static Map makePandoraMapFromNode(Node sNode)

        throws SAXException

    {

        String name = null;

        String value = null;

        String s = null;

        Node mem = null;

        Node n_or_v = null;

        NodeList nvpair = null;

        int nvlen; 

        Map theMap = new HashMap( 22 ); 

        

        NodeList members = sNode.getChildNodes();

        int len = members.getLength();

        for(int ii = 0;  ii < len; ii++)

        {

            //

            //each node here should be a 'member' node...

            //put in safety check???

            //

            mem = members.item(ii);

            //System.out.println("what is this nods name:  " + mem.getNodeName());

            name = null; 

            value = null;

            

            nvpair = mem.getChildNodes();

            nvlen = nvpair.getLength();

            for(int jj = 0;jj < nvlen; jj++)

            {

                n_or_v = nvpair.item(jj);

                s = n_or_v.getNodeName();

                if(s.equals("name"))

                {

                    name = n_or_v.getFirstChild().getNodeValue();

                }

                else if(s.equals("value"))

                {

                    Node vNode = n_or_v.getFirstChild();

                    if(null != vNode)

                        value = vNode.getNodeValue();

                }

                else

                throw new SAXException("unexpected node named [" + s + "]"); 

            }

            if( name == null ) 

                throw new SAXException("while parseing a struct member the name was null");

            if( value == null )

                value = ""; 

                

            theMap.put(name,value);

        }

        return(theMap);

    }

    

    public int size()

    {

        return(listOfTrackMaps.size());

    }

    

    public Iterator<Map> iterator()

    {

        

        Iterator<Map> callerIter = new Iterator<Map>()

        {

            private Iterator iter = listOfTrackMaps.iterator();

            

            public boolean hasNext()

            {

                return(iter.hasNext());

            }

            

            public Map next()

            {

                return((Map)iter.next());

            }

            

            public void remove()

            {

                throw new UnsupportedOperationException("remove not supported");

            }

        }; 

        return(callerIter);

    }

    

    

}

[cooper]

I'm using JDom and XPath to parse that XML data.

Takes about 20 lines of code to put everything together.

[cooper]

Damn. Sometimes that Lizard can be a pretty rotten program to deal with...

Mozilla's browsers decide, based on the direction of the wind it seems, wether they will store the Pandora XML in a file in the Cache directory, or embed it in those _CACHE_00X_ files. To make matters worse, they somehow manage to not update the timestamp that says when the file was last modified, even when it was.

End result is that I can find the XML in the file, but I can't get my hands on the timestamp, which is typically stored in a DIFFERENT file, without completely parsing the cache files.

SO, I've begun writing a Mozilla cachefile parser.

In the mean time I still haven't managed to get the Javascript callbacks to work. Like, _at all_. Not in Moz, not in IE, nada. Venkman isn't helping much either.

I could put up a pre-alpha that will be solid, but will effectively be locked to cruise control, and you won't see much of it doing its thing in the browser window. Probably no sense in doing that...

Guess from this side it's still primarily patience that's required.

[JohnnyChimpo]

that is where a local proxy can be helpful...

by using one, then the v8/xml files (and mp3's too) are always saved to the same place and following a consistent fashion...

not that you'd do this - but a proxy would work for IE too...or safari, opera, galeon, k-whatever browser....

basically it takes the browser out of the equation...

[cooper]

I understand and appreciate that, but at work I'm already behind a proxy, and I don't want to have to run this program when I'm just browsing.

I still feel Pandora's Jar is the cleanest solution to the problem. It just takes a little work. Well, maybe a lot of work... ;)

[pandoralover]

that is where a local proxy can be helpful...

by using one, then the v8/xml files (and mp3's too) are always saved to the same place and following a consistent fashion...

not that you'd do this - but a proxy would work for IE too...or safari, opera, galeon, k-whatever browser....

basically it takes the browser out of the equation...

So how have you used webscarab to make this work? I noticed you can script it so I was thinking a simple scarab script could do most of the grunt work...

[JohnnyChimpo]

well I hadn't thought about a webscarab plugin - relatively noob to webscarab...

All I have done is run webscarab in default config - and then open FF up to pandora and let it rip for an hour or so...

I then looked at the the <webscarab.tmp>/conversations directory and noticed all the data/files and noticed how it matches up requests to responses.

From there it is just scraping thru request files for the V8XML parsing that up, locate the mp3 response file (based on XML), strip the webscarab header from mp3 response, tag it and tuck away...

I have this process hacked up - to the point where I can locate the MP3 file and have the XML meta data about the mp3. Tonite I hope to get the header strip and tagging done...

[JohnnyChimpo]

got a really early cut of this idea ....got no place to upload it to for sharing...does this forum have place.

(I wish to avoid some those free file hosting services.....)

[cooper]

Sent you a PM.

[JohnnyChimpo]

WebScarab/Fiddler + Pandora time shifter (latest ver - scroll down for directions):

update: http://www.wirschell.nl/pandora/pan-v5.jar

V5

---------------

-major code refactoring

-fixed few small bugz with fiddler support

V4 (never rreleased)

-----

-fixed a problem with punct marks in tracks & artists for cddb queries.

-added basic support for archived fiddler data (still abit shaky - but functional)

-updated this post for additional fiddler directions

V3

------

-Added CDDB queries similar to pandora.jar (FYI: some of the stuff done to cddb queries could be useful in pandora's jar, too).

-Fixed a bug or two, or three, or 4

-Fixed so that decent description of command line args show up.

V2

--------

-ignored non-getFragment V8 xml request (i.e. gets rid of some harmless error messages).

-tested captured output from WebScarab running under linux - works fine (-W may go away)

-fixed a couple of exception cases that didn't report good messages

V0

------

-Initial release

Prerequisites/setup

--------------------------------------------

• ANT

JDK 1.5x

Webscarab (http://www.owasp.org/index.php/Category:OW...bScarab_Project) or Fiddler (http://www.fiddlertool.com/fiddler/)

Some browser that can play pandora. only tested with flash 7 & 8 under firefox (v2) & IE6 with FP9

Basic knowledge of how to run WebScarab or Fiddler

Rudimentary knowledge of running java programs and using ant.

unzip/unjar the latest pan.jar into some area.

Run ANT to generate 'classes' directory

Here is a how to

--------------------------------------------

• :arrow: run webscarab or fiddler

:arrow: open your browser and configure it to use webscarab as a proxy. If you are using fiddler and IE (i think fiddler only works with IE) & you have installed fiddler already - there should be a button on the IE tool bar to start 'fiddling' (that is activate fiddler and set IE to proxy thru it). There should also be a menu option to start fiddling.

:arrow: goto Pandora and listen away. I suggest not doing anything else with this browser (I haven't tested how well this hack ignores irrelevant data from webscarab or fiddler). Run another browser that is not using webscarab/fiddler as a proxy and surf with that one.

:arrow: when you are done listening, shutdown your browser and webscarab. If you are using fiddler, highlight/select all the conversations in the list on the left hand side of fiddler screen and use the Save -> Eveything to Zip (basic instructions here: http://www.fiddlertool.com/Fiddler/help/ui.asp).

:arrow: make sure you have compiled pan. A simple 'ant' in the base pan directory should build everything.

:arrow: make sure your classpath is set up to include ./classes and all the jars in ./lib.

:arrow: run the program (java pir.pan) w/o arguments - it should display basic information about usage.

:arrow: If you are using fiddler, unzip the zip file you made (see above), into some temp directory. It should contain a file name _index.htm and sub-directory called raw (with req/resp files).

:arrow: If you are running WebScarab locate the webscarabXXXX.tmp directory where it puts its conversations. This directory will be stored in your $TMP/$TEMP directory or where temp stuff is stored on the WebScarab host. WebScarab creates a new tmp directory per session, make sure you are looking at the correct one.

:arrow: run the 'pan' program like (for WebScarab):

java pir.pan  -D ./conversations/  -O output

:arrow: run the 'pan' program like (for Fiddler)

java pir.pan -F  -D fiddler1/  -O out_fid -Q

While it is running - it should print very simple messages as it process MP3 files. In the above example, it assumes that your pwd is a directory that contains the 'conversations' directory from webscarab. After the program runs, there would be a directory ./output that contains your goodies.

My test environment

---------------------------------------------

Web scarab and my browser (FF2.0) were running on XP, After listening to Pandora for 1/2 day, I shut things down and tarred up the web scarab temp directory and moved it to linux (FC5) and ran the java program there. All the development was done on the linux/fc5

For fiddler tests, I ran on the same XPPro box, but with IE and FP9. After a basic feasibility test under IE/FP9; I actually listened to Pandora via Pandora's box ( http://www.cfdan.com/posts/Wrapper_Applica...n_Task_Tray.cfm ) for a whole day. During this time, I would periodically make a zip bundle described above, and then delete the conversations listed in the sessions window (delete key works fine). Thinking back, this may not be wise as some parts of the Pandora v8 'fragment' may span across zip bundles. However, my first/test bundle of about 5 tracks works fine...YMMV

lemme know what you think...(oh there is lots more to do and fix - but usage feed back would be helpfull)

[JohnnyChimpo]

FWIW -- i have the desire, when working on windows, to use an app like pandora's box or similar widget...These require IE.

Well, WebScarab + IE + pandora has some issue - that I haven't nailed yet. But I found fiddler + IE 6 + Flahs8r22 + pandora -- does a very similar thing as FF+flahs8+webscarab+pandora. That is fiddler will save off all the request/response pairs to files in a very similar fashion as Webscarab. Some adjustments to 'pan' could probably make proxy timeshifting of the combo of IE+f8+fiddler+pandora work. By extension, this should (maybe should) also work for IE based pandora tools like pandora's box (my goal in this exercise is to get 'proxyshift' of pandora working via a windows tool like pandora's box to work so I can use FF as my browser to do stuff with and PB/IE to listen to tunes -- maybe even with fp9 - assuming http req/resp goes thru the proxy - even if fp9 doesn't save like the fp8 does. )

http://www.fiddlertool.com/fiddler/

http://www.cfdan.com/posts/Wrapper_Applica...n_Task_Tray.cfm

Couple of questions for anyone following this thread:

At work, my XPPro desk top insists on upgrading my Flash to 9 when running under IE (after I have donwgraded to fp8). My test home 'family' PC is an XP home PC - it is running flash8 - and been that way forever. Yet it doesn't complain about upgrading to FP9 when I go to pandora. Why?

Anyone got any ideas on why IE/Pandora sometimes chokes using a local proxy like WebScarab - or using WebScarab on a 2nd machine (tried that too, and Pandora still refused to load)?

[cooper]

At work, my XPPro desk top insists on upgrading my Flash to 9 when running under IE (after I have donwgraded to fp8). My test home 'family' PC is an XP home PC - it is running flash8 - and been that way forever. Yet it doesn't complain about upgrading to FP9 when I go to pandora. Why?

My guess is your work setup runs a logon script that brings everything up to a certain version. It notices your Flash is older than expected, so it automagically upgrades it.

[JohnnyChimpo]

could be - I will check that... doesn't have any problem with FP8r24 in FF

Also, I can go to some other flash pages and IE doesn't bark at me about upgrades...for example http://www.adobe.com/cfusion/knowledgebase...cfm?id=tn_15507 doesn't complain..

then again - the object tag on that page only calls for 6,0,0,0.....Pandora's wants 7,x,x,x but that still doesn't explain the persistent nag to move to 9...

[JohnnyChimpo]

It doesn't matter (for me at least)....

So even though my work XPPro insists on flash9 under XP - I let it install/run FP9 for pandora.....Using fiddler (just like I use WebScarab)....I could capture req/resp to/from pandora....

The save as format for Fiddler is very similar to WebScarab's ....Should be an easy hack to add 'fiddler' support to Pan....

AND...once your IE is configured to use fiddler...Pandora's box works via fiddler too (no real surprises there....)

So it is (well should be easy) possible to get IE+FP9+Pandora+Fiddler+Pan to proxy 'timeshift' ....

(Still haven't figured out why my IE insists on FP9...Not too important at this point...).

[JohnnyChimpo]

If anybody is following this, I just updated the main post about 'pan' the hack that digests local proxy output to timeshift pandora content.

For those keeping score, with pan I have been able to do 'proxied' timeshifts on:

Linux/FC5 + FF2 and FP7.0 r63 + WebScarab

WinXPPro + FF2 and FP8r2x + WebScarab

WinXPPro + IE6 and FP9.?? + Fiddler

I am pretty sure (but not tested yet) that:

Linux/FC5 + FF2 and FP9x (there is a beta out) +Webscarab

XP + FF2 + FP9x + WebScarab

would also work.

[wolf08]

I'm following your instructions, but I think that I'm missing a file related to pan(-v3).jar

When I try and build it with ant, I get this error:

Buildfile: Build.xml does not exist

Build failed

Do I need to download this file from somewhere, or create it?

THanks, Wolf08

[JohnnyChimpo]

Grab the v5 update. But with v3 (and all of them) - the contents of the jar should be complete - nothing else is needed (well except for the prereq.).

You should unjar the contents of pan-vX.jar into some directory, and run ant from 'inside' that directory. FYI: jar and zip are mostly the same. So many unzip programs will correctly unzip the jar, too.

The contents of the jar should be

build.xml

lib/

src/

After running ant, a new directory should appear, It will be called 'classes'. Classes will contain the compiled version of pan.