[Deep Web Crawler Building 101]

On 6/6/2018 at 7:46 AM, i8igmac said:

7958661109946400884391936

7,958,661,109,946,400,884,391,936

Does this look like a kabillion?

katrillion?

36**16

36 characters. a-z + 0-9

26+10=36

A onion link looks Looks typically 16 characters long

So 36 to the 16th power

36**16=7958661109946400884391936
 

Kazillion dns queries. impossible.

there are already onion search engines. You can pull millions of working onions.

 

Makes me think about scanning these onions for sqli/lfi injections...

 

Anyways if you had 500 botnet GIG devices.  you might accomplish this. I wish I had government access to perform this kind of scanning.

 

I diverse a government job after this post.

Wait, so is it actually possible to create a web crawler that would work on the Tor network websites?! That's an incredible idea! I never thought it could be possible. I mean, it's a prolonged network for browsing, and that's why I didn't know it was possible. I'm also pretty new to web-crawling, but the basics are something that I have known since I was a student. I think a template for your project could be https://proxycrawl.com/scraping-api-avoid-captchas-blocks. They have some of the best web crawling features on the entire web. Besides that, they also tried to develop a service that would work on the Tor network.

[Deep Web Crawler Building 101]

hmm

[Deep Web Crawler Building 101]

I completely agree with the author