Shodan API | search and actually get more than 100 results?

[vailixi]

So lately I've been checking out shodan.io and finding some interesting stuff. What I noticed is by default the CLI won't display more than 100 results at a times. What I got the from the dev was that it displays 100 results per page but you could loop through the pages and display as many results as you want to but you have to tell it where to end so you don't burn up all of your credits.

I'm really not sure how to loop through the pages and display more than 100 results. Anybody have any experience with this?

[achillean]

If your sole intent is to download results and analyze them I would recommend checking out the CLI. It was a "download" command with a "limit" option to let you specify how many results you want to download. For example:

shodan download --limit 1000 mongodb port:27017

The above would download 1000 results for the search query port:27017 and save the results in a file called mongodb.json.gz. I recommend always saving the results of a search query so you can keep it for historical purposes and do whatever analysis you want without needing to redo the search request (i.e. it's faster and doesn't use API credits to analyze a file).

With regards to the API, you can iterate over the search results using the page parameter of the search() method. I.e. you start with page=1 and then call api.search(query, page=page) in a loop to go through however many results you want. You can also check out the code for the search_cursor() method which basically does what I just described:

https://github.com/achillean/shodan-python/blob/master/shodan/client.py#L309

I hope that answers your question!

[vailixi]

dd@dd:~$ shodan download --limit 1000 vsftpd "vsftpd 2.3.4"
Search query:			vsftpd 2.3.4
Total number of results:	2102
Query credits left:		0
Output file:			vsftpd.json.gz
  [####################################]  100%
Notice: fewer results were saved than requested
Saved 100 results into file vsftpd.json.gz

Gets to 10% and quits at 100 results every time whether the --limit 1000 flag is used or not. ":)"

Thanks for your time achillean. I'm sure you're a busy guy.

Edited December 31, 2015 by vailixi