Looking for a program that can gather entire URL Directory

[VilleValoV]

Looking for a program that can find full branches of a submitted URL. i.e

if i typed

Zoo.com

the desired result would be

Zoo.com/shops

zoo.com/shops/shirts

zoo.com/shops/shirts/girls

zoo.com/animals

zoo.com/animals/bats

zoo.com/animals/bats/naturalhabitat

zoo.com/blah/blah

zoo.com/blah/blah/blah

And so on.

Does a program like this exist?

[digip]

Looking for a program that can find full branches of a submitted URL. i.e

if i typed

Zoo.com

the desired result would be

Zoo.com/shops

zoo.com/shops/shirts

zoo.com/shops/shirts/girls

zoo.com/animals

zoo.com/animals/bats

zoo.com/animals/bats/naturalhabitat

zoo.com/blah/blah

zoo.com/blah/blah/blah

And so on.

Does a program like this exist?

If they are public and indexed by a search engine, such as google, then you can try "site:zoo.com" as a google search to see what it finds.

ex: http://www.google.com/search?hl=en&q=s...G=Google+Search

You could also try a thrid party app to follow links in a domain and return all found links from each page within the site, excluding links out of the base domain.

[VilleValoV]

You could also try a thrid party app to follow links in a domain and return all found links from each page within the site, excluding links out of the base domain.

that's what im looking for, something automated that can be used along with another program to simplify a certain task.

[digip]

that's what im looking for, something automated that can be used along with another program to simplify a certain task.

You could try writing a wget script to follow all relative links of a domain and output it to a text file.

[AdvenSpirit]

Well, I am not sure this program will do what you say what you want. But I have used it before, and it does copy entire websites and a few directories connected to said website.

I say take a looks and see.

HTTrack Website Copier

[digip]

If you have wget installed and in your path, save this in a bat script called spider.bat and run it. It will do the rest for you.

:123
@echo OFF
cls
echo "Choose a site to spider links."
SET /P website="[example: www.google.com] : "

wget -erobots=off --accept="html,htm,php,asp,aspx,jsp,chm,shtml" -rH --spider -P SpiderDownloadShit/ -D%website% --user-agent="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" %website% -do SpiderDebug.txt 

echo Links found in %website%. SpiderDownloadShit/ is to be ignored.  > Spiderlinks.txt

find "saved" SpiderDebug.txt >> Spiderlinks.txt

del SpiderDebug.txt

rmdir SpiderDownloadShit /s /q

::pause
goto:123

Edit: Cleaned it up and fixed it so you dont have any folders or files to delete afterwards!! Now it only spiders the site for available links.

[Rab]

Problem is, you don't know which files and folders exist, so trying all possible filenames is gonna be a fail effort. AFAIK there isn't a 'get all the porn off this site' tool :(

[digip]

Problem is, you don't know which files and folders exist, so trying all possible filenames is gonna be a fail effort. AFAIK there isn't a 'get all the porn off this site' tool :(

This is only going to give you a quick list of links it can find from within the page you feed it, it doesn't know if hidden directories or ones with no links to follow. Without root or remote access to the machine itself, you won't be able to list all the directories under the site.

[SomeoneE1se]

it would be simple to create a http scanner that brute forces the url or uses a dictionary file.

[digip]

it would be simple to create a http scanner that brute forces the url or uses a dictionary file.

I think I made that in another post somewhere, but yeah, it can be fed a list to go through. It was in a thread someone asked about a url brute forcer or something to that effect. I think I posted the code to do it too, just have to find the thread.

[Sparda]

http://www.fukung.net/v/119/how-to-catch-script-kiddies.gif

[nicatronTg]

http://www.fukung.net/v/119/how-to-catch-script-kiddies.gif

Lol, an actual every ip adress in the world program... now we need one for IPv6.

[Sparda]

Lol, an actual every ip adress in the world program... now we need one for IPv6.

I felt it was relevent to this thread.

[manuel]

http://home.snafu.de/tilman/xenulink.html

[DingleBerries]

doesnt Goolag have a feature that does this

[nicatronTg]

I felt it was relevent to this thread.

It is because it is very true. The fact that just about every script kiddie would download it is a given.

Here is a start:

1.1.1.1
1.1.1.2
1.1.1.3
1.1.1.4
1.1.1.5
1.1.1.6
1.1.1.7
1.1.1.8
1.1.1.9
1.1.2.0

[SomeoneE1se]

It is because it is very true. The fact that just about every script kiddie would download it is a given.

Here is a start:

1.1.1.1
1.1.1.2
1.1.1.3
1.1.1.4
1.1.1.5
1.1.1.6
1.1.1.7
1.1.1.8
1.1.1.9
1.1.2.0

you do know that's not how it'd go?

[digip]

you do know that's not how it'd go?

You had to ask? Or was that a rhetorical question :)

Anyway, @nicatronTg, if you look at the thread, he is not trying to scan every ip address(which what you posted wouldn't work anyway). He wants to find directiories under a site, ex: somesite.com where it would search for

somesite.com/somevariable

somesite.com/somevariableetc

somesite.com/somevariableetcetc

somesite.com/etc/etc/etc

In doing so, when you generate enough misses, the webadmin(If he is not sleeping on the job) will see his error logs filling up with crap and ban your IP address anyway, so it really is of no use to use this method to enumerate a site.

BUT, if you have fingered your target system well and know certain flaws, common folder structures, etc, you can make separate scripts that can go out and check the server for specific directories, or even flaws in things like PHP, mysql, etc., just by navigating to the site or adding commands, scripts, and things to the URL. Just use common sense when testing this out. Preferably on your own site or network, we'll call it Pentesting. Do it to another site, and well, now your just asking for trouble.

[uber_tom]

Lol, an actual every ip adress in the world program... now we need one for IPv6.

lol, like anyone cares about or fully understands IPv6. (honestly its easier to remember mac addresses).

shouldn't it filter out network addresses like:

10.x.x.x

172.16.x.x

169.168.x.x

and emphasize 127.0.0.1.

not that I'm bored or anything.

[Rab]

well, this would be easy if you had the password... and tbh thats prolly easier.

[digip]

Automated software can be configured to just ban you after too many attempts. Here is some stuff that works against people tyring to brute force SSH, but I am sure it can be configured site wide as well: http://isc.sans.org/diary.html?storyid=4408

[m1d63t]

A program that you might like is BlackWidow by Softbyte Labs. If I read your post right then it might help you out.

http://softbytelabs.com/us/downloads.html

[TrinitronX]

You can look at the robots.txt file on the site for a list of hidden areas and follow those links too.