[[C++] Best method to organize massive word lists?]

2 hours ago, digip said:

I might be missing the point, but this should not be a difficult thing to the point that you would want to do this with a C program(other than maybe for the mental floss exercise and trying to reinvent the wheel with some complex routines). Depending on how much you need to modify things, just do it from the command line with utilities already designed for this, where the heavy lifting of programming has been done for you. Tools like cat and sort, handle this kind of thing and what they were designed for.

If you just need to merge and sort, then a one liner command would be easiest under Linux.

On linux:

cat file1 file2 file3 file4 | sort -u > list.txt

On windows, put your files in same directory then run the following 2 commands (windows will need a few more steps, but still easy-peasy):

:: Open a cmd prompt, then open powershell:
C:\ > powershell [hit enter]
:: Run each of the following commands
dir C:\path\to\files\* -include *.txt -rec | gc | out-file C:\unique\path\to\results\result.txt 
gc C:\unique\path\to\results\result.txt | Sort-Object | Get-Unique | out-file C:\path\to\sorted\wordlist.txt

On windows, you can't chain the entire command up above to pipe the first into the second; while you won't get an error, it's a sharing violation of the file in use, so the second command needs to be input by itself after the first command completes or the wordlist at the end will be empty. Add this in a powershell script and it's just a matter of putting the files in the path to find them, then run it(could do similar with a bash script). You can add a cleanup routine to this to remove unwanted files as well. Just be sure not to put the result file in the same directory as where you scan for files to merge, or you will fill up your HDD in a loop of a file that grows forever! 

I like the idea of using the command line because you might be on a system where you actually want to cat some output and merge then sort the output for whatever reason, maybe even for ex-filtration needs, vs needing to have a compiled C program do this, requires being able to move the binary to those systems, and/or compiled for that specific OS, something not needed if you know your environment per OS. I assume MAC OSX would be very similar to the Linux/*nix equivalence of commands and utilities as well.

If you need anything more complicated than this, then I'd say do the database thing as suggested above so you can tag words in different groups and sort dynamically into different list categories, but to merge a sorted, unique list, keep it simple with the command line.

Huh I never knew linux had built in command line tools for this express purpose.

Well I suppose that makes the program a little useless in terms of practical use with the tools already available. It was a good brain stretch thinking about how I would have to handle the data and do this all myself in C++. I was at work the whole time thinking how I would disassemble the files to more manageable sized "chunks" and having separate methods to de-dup and organize the output depending on user choices.

Sounds like I really was trying to reinvent the wheel lol. Still, there is always pride to be had in writing all the code yourself and going through the process :P

[[C++] Best method to organize massive word lists?]

3 hours ago, Sebkinne said:

I'd probably use something like external merge-sort for this, but a database like @noncenz suggested should be a good choice for this.

I just looked up external merge sort and I think that might be perfect. I database sounds interesting but I want this to be a simple tool I can redistribute and keep it simple for users!

[[C++] Best method to organize massive word lists?]

It does not have to be I would just prefer it be in A programming language. I also thought about python but I knew more C++.

 

Although either one I would see myself having the same issues outlined in the original post.

[[C++] Best method to organize massive word lists?]

Hey there!

I am trying to write my own application for merging and organizing wordlists. The problem I am having is thinking of the best way to filter through all the candidates to remove duplicates, organize them, etc.

Im going through it and I have to be able to open pretty big word lists already in existence (anywhere from 1gb to 5gb PER FILE). I also am not sure how I would compare say candidate number 555740 against all the previous options without going back through each possible option for comparison. I thought maybe organize the results I have into separate temp .txt files that are alphabetical but I could see the same result with huge lists.

Any advice on where I should go with these problems?

[NMap Host Scan VS Network Scan Question]

So this is definitely me not knowing nearly enough about networking but when i scan a live host on my lab network with Nmap (Using Metasploitable as the OS as a VM) i get a list of services and the ports they run on as you would expect.

However if i perform a scan using my ISP provided IPV6 IP address i simply get the port 111 and rpcbind.

So all im trying to confirm is, i wont actually be able to get a list of services and ports for a particular host until im in the network, correct? Scanning the ISP IP is basically just scanning my router isn't it?