Jump to content

downloading a site


Recommended Posts

i know firefox will download a webpage as a complete webpage. is there any addon to firefox or a program i could use that would download like that but would donwload the entire website. like following all links that stay inside the website.

or even better a program like flashget that would let you pick what pates and links you want to download from.

in case anyone is wondering why. my music studio class was recently changed to guitar lessons. it will most likely be changed back within 2 -3 days. so i want to download everything from the class i can before its gone.

Link to comment
Share on other sites

Save this to a bat script:


echo "Choose a site to mirror."
SET /P website="[example: www.google.com] : "
::::::  Pick a Web Browser Agent if you wish.
::Opera/9.24 (Windows NT 5.1; U; en)
::Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
::IE6 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)

wget -erobots=off -m -r -l 1000 –p -E –k -P mirrorShit/ "%website%" --referer="%website%" --user-agent="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" "%website%" -do Debug.txt

This is something I wrote for wget so I can mirror a website if need be. It won't get you any server side scripting, but whatever is web facing, it will grab the whole site, so long as it can follow links on each page through the site. Warning though, this could run hours if a site is large enough. Like, don't try to mirror google.com or something with large forums. Change the user agent to yoru needs. Some sites actually block Google bot, so if it doesn't find anything, change it to something like IE or Opera, FF, etc.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Create New...