-
-
Notifications
You must be signed in to change notification settings - Fork 90
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
confusing default --line-max-size
#10
Comments
Hello ! What But anyway, i suppose you used For example, there is Also, empty lines are automatically removed by duplicut. These aditional behaviors exist because duplicut is mean to aggregate password wordlists, without losing the order, and without having duplicates. And in a passwords wordlist context, i rarely want to keep lines longer than 14 chars, as they might be a garbage line, a too long password to deserve to be guessed, of a parsing error from the tool that generated this line. Empty line are also deleted for being obviously useless in a wordlist of passwords. Anyway, if you want to test duplicut, i recommend you to check at these files from my test suite:
|
Anyway, if i answered your doubts, and if the issue is resolved, feel free to close it. Othersiwe, i'll be happy to debug with you ! |
Adding
|
Another interesting interesting 'user warning' would be to inform user if no |
@freeroute , can you please confirm me if the problem was due to |
I just tested duplicat with a 2.5 GB dictionary.
file dictionary_private.dic : data
time sort -u dictionary_private.dic >dict_sort_uniq.txt
real 5m40,168s
user 13m9,512s
sys 0m7,682s
time duplicut dictionary_private.dic -o dict_dedupe.txt
real 0m47,435s
user 0m32,963s
duplicut is much faster than the "sort -u " command.
but the result not same. counting the lines of new worldlists.
wc -l dict_*
171193011 dict_dedupe.txt
205241662 dict_sort_uniq.txt
number of lines of the original file:
wc -l dictionary_private.dic
206282806 dictionary_private.dic
What can cause this discrepancy?
The text was updated successfully, but these errors were encountered: