Jump to content

Scrape IP range


Recommended Posts

Desired end state: Scan and scrape a range of IP addresses, looking for a router model within an an HTML page, and output the IP address, if found. Note: Multiple versions of the router are open and the xpaths to the hardware versions are different...

Background: I've been goofing around with nmap, scanning a few subnets around my IP address (Comcast) and have noticed that 5-10 Belkin F5D model routers per class B subnet are open on port 80 and about half of them have no admin password established. I want to do a more extensive scan but want to limit the IPs returned to the F5D routers, not every device open on port 80 so I don't have to go to each page and manually identify them. From what I understand python doesn't work well with nmap and I am not sure a good way to approach this and I was struggling to get something like BeautifulSoup to successfully scrape a config page.

Any ideas?

Edited by whitehatvet
Link to comment
Share on other sites

Not mine but touche I guess? or at least I understand what you are saying....though it's trivial to find an open router's config page that is password protected. It's not inviting the hacking of that router, it's using it as a reference in order to parse HTML data from. If I posted one of the open IP addresses, I'd be more apt to agree with you...

That being said maybe a 'better' way to show this is to show you an xpath to one of the routers, written two different ways.

Link to comment
Share on other sites

That's what I'm trying to do...I'm not looking for someone to hand me a turn key solution...

Though I'm new to coding, I have already written code, though admittedly easy and probably inefficient, that identifies IPs with port 80 open and exports the results to a text file and script (for automatically opening them in firefox) so I can browse through them. That is obviously is time intensive so I tried to use BeatifulSoup and similar methods to scrape the Belkin page, as I said above, but none work. There is no title to the page (just the IP), links are not being detected, and I have been unable to specify the hardware field to specifically parse that data.

I understand that you and others can write code to do it; I'm just looking for advice on how you would go about it since the methods I have tried failed...

Link to comment
Share on other sites

Well, here's what I would do. I would execute nmap through my Java application (look up Java Process object and reading from the InputStream). From the output of nmap, I would determine if an IP has port 80 open. If so, I would make an HTTP Request for that IP and store the result into a string. Then, I would parse the string for the information I need. Finally, display it on the interface.

Link to comment
Share on other sites

Thanks for the feedback. I'm only in my 2nd semester of Java and didn't know you could use it to execute nmap or do an HTTP request like that. I appreciate it and I'll start playing around with it tonight. Sounds like it should be fairly easy...

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Recently Browsing   0 members

    • No registered users viewing this page.
  • Create New...