Merge pw lists

[Sizzlik]

hi..i would like to know if anybody knows a nice tool or easy way to merge password lists. In my case..i got the rockyou.txt wich contains 14mio passwords and i got a master list with 19mio passwords.

Now i want to join the both together, ignoring duplicates, adding new ones...and of course in future test other lists against my list to check for new passwords.

When i google it, i only find tools and answers that append the contents of files to the end of the main file...but a pw list with tons of duplicates.

So..does anybody know a tool? Or a way to reach that goal? I thought about finding a way in excel..but im really bad at excel..i can make a invoice..thats it

[Ddanraks]

4 hours ago, Sizzlik said:

 

Sort or uniq? Man pahes might help.

[aethernaut]

I doubt that Excel has enough lines to manage merging those lists.

It's a LONG time since I've done this but if each of your lists is sorted and unique I believe that you can use mli2 from Hashcat Utils

I believe that the stages would be something like this.

cat list1.txt | sort | uniq > list1_su.txt
cat list2.txt | sort | uniq > list2_su.txt
./mli2.bin list1_su.txt list2_su.txt > list_full.txt

Check first using smaller test lists containing just a few words!

[Cap_Sig]

There are several ways to do this...

One of the simplest is using sort

sort -u pwd1.txt pwd2.txt -o master.txt

Other methods posted should work as well.  There are also hashcat utils to aid in this.  Another option you might see is dymerge but personally it does not work for me due to its limit on list size. 

[Sizzlik]

Perfect..thanks for the help!

Now another question crossed my mind..what size of a PW list is reasonable? I mean..there are wordlists out with over a billion passwords (1gb file) wich is totally stupid on a BB..so how big is close to "too big"? I think the list now gets close to 200mb..maybe someone did the math on how long the BB takes to process xxx passwords on a brute force?
(And isnt it called dictionary attack anymore when you use a pw list? In my mind the bruteforce attack was to try EVERY possible combination..if you have a Tianhe-2 supercomputer.)

[aethernaut]

Yes, using a password list is a dictionary or wordlist attack. Brute-forcing is progressively trying every possible combination of letters, numbers, and symbols a password, of possibly unknown length, may contain.

4 hours ago, Sizzlik said:

how long the BB takes to process xxx passwords

How long depends on what you are doing, and if BB means Bash Bunny it is likely to be a long time and it certainly won't be hash cracking.

Assuming you wish to throw a user/password combo at something, like BruteBunny does, you can see how long it would take by making a small list to test it with that contains the password** so that you can check it actually works.

4 hours ago, Sizzlik said:

..what size of a PW list is reasonable?

Remember with payloads like BruteBunny if you use a list of account names with 9 entries, the BB will have to process the entire wordlist 9 times. So if you tried that just with your Rockyou list (14M passwords?) that's up to 126M logon attempts. If it checks at one per second (24/7) that's up to 4 years and there is no guarantee the list actually contains the password. If you find it's faster than one per second, divide accordingly.

**You will know the password because it will be your equipment you are testing this on.

[Sizzlik]

On 2/5/2020 at 2:32 AM, aethernaut said:

Yes, using a password list is a dictionary or wordlist attack. Brute-forcing is progressively trying every possible combination of letters, numbers, and symbols a password, of possibly unknown length, may contain.

How long depends on what you are doing, and if BB means Bash Bunny it is likely to be a long time and it certainly won't be hash cracking.

Assuming you wish to throw a user/password combo at something, like BruteBunny does, you can see how long it would take by making a small list to test it with that contains the password** so that you can check it actually works.

Remember with payloads like BruteBunny if you use a list of account names with 9 entries, the BB will have to process the entire wordlist 9 times. So if you tried that just with your Rockyou list (14M passwords?) that's up to 126M logon attempts. If it checks at one per second (24/7) that's up to 4 years and there is no guarantee the list actually contains the password. If you find it's faster than one per second, divide accordingly.

**You will know the password because it will be your equipment you are testing this on.

that helps me alot. thanks! yes..i ment BashBunny with BB..i didnt knew it loads the wordlist again every attempt. I thought it opens it and rattles it down 

[aethernaut]

11 hours ago, Sizzlik said:

I thought it opens it and rattles it down

It may well be much quicker which is why I suggested testing on a smaller list just so you can get an idea as to how many years it would take 🙂. However with a large wordlist I doubt it will be that quick.

Of course if someone ever used this in the real world  they could come up against something like Fail2Ban which would bring the whole exercise to a sudden and grinding halt!