doc/CHANGES

The following changes have been made between John 1.9.0 and 1.9.1-dev:

* Let "--node" numbers range be any multiple of "--fork" process count.
* Fixed "--status" to work right on sessions that use "--fork" with "--node".
* Added two external modes: Filter_NoRepeats skips candidate passwords that
contain the same character more than once, and Filter_Repeats is its inverse.

The following changes have been made between John 1.8.0 and 1.9.0:

* Increased the interleaving for bcrypt on x86-64 from 2x to 3x for a major
speedup on CPUs without SMT.  Unfortunately, this sometimes results in a minor
performance regression when running multiple threads on CPUs with SMT.
* Recognize the $2b$ bcrypt prefix.
* In the generic crypt(3) format, detect descrypt with valid vs. invalid salts
as separate id's for our heuristics on supported hash types.
* Introduced a number of optimizations for faster handling of large password
hash files, including loading, cracking, and "--show".  Some of these use more
memory than before, yet in a more efficient manner.
* Benchmark using all-different candidate passwords of length 7 by default.
* Dropped undocumented special handling of "Mc" in 'c' and 'C' rule commands.
* Dropped undocumented limitation of the 'M' and 'Q' rule commands where they
would sometimes memorize/check only up to the current hash type's length limit
yet this optimization wouldn't necessarily be transparent (e.g., if a later
command would extract a substring from above the hash type's length limit and
bring it to within the limit).
* Implemented special-case handling of repeated rule commands '$', '^', '[',
']', '{', and '}', as well as faster handling of the 'D' command.
* When built with "--fork" support, disallow session names with all-digit
suffixes since these clash with those produced by "--fork".
* Forward SIGTERM to --fork'ed children.
* Set stdout to line buffered (rather than potentially fully buffered), except
for "--stdout", "--show", and auxiliary programs such as "unshadow".
* On Windows, restore normal processing of Ctrl-C in case our parent (such as
Johnny the GUI) had disabled it.
* Added linux-x86*-avx512 and linux-x86*-avx2 make targets, which use
respectively AVX-512 and AVX2 for bitslice DES.
* Added linux-mic make target for Intel MIC (first generation Xeon Phi, aka
Knights Corner), which uses its 512-bit SIMD intrinsics for bitslice DES.
(For second generation Xeon Phi, aka Knights Landing, use linux-x86-64-avx512.)
* Added linux-arm64le, linux-arm32le-neon, and linux-arm32le make targets.
(The first two of these make use of ASIMD or NEON for bitslice DES.)
* Added linux-sparc64 make target.
* Made a minor optimization to MMX and SSE2 assembly code for LM hash.
* Dropped Ultrix and SCO support.
* Don't probe for alternate config file names (like john.ini when on Unix).
* "DokuWiki" external mode sample has been added to the default john.conf.
* Fixed operator precedence in the external mode compiler to be the same as C.
* Fixed an out of bounds write bug in the external mode virtual machine.
* Fixed a bug introduced in version 1.7.4 in the wordlist rules engine, where
some sequences of rule commands could overflow a word buffer.
* Fixed a bug where unaligned access SSE/AVX instructions would unnecessarily
be generated by GCC 4.6+ in the bitslice DES code in non-OpenMP builds.
* Fixed a bug where "Warning: no OpenMP support for this hash type" could be
printed in "--stdout" mode.
* Made assorted other bugfixes, portability and documentation enhancements.

The following changes have been made between John 1.7.9.8 and 1.8.0:

* Revised the incremental mode to let the current character counts grow for
each character position independently, with the aim to improve efficiency in
terms of successful guesses per candidate passwords tested.
* Revised the pre-defined incremental modes, as well as external mode filters
that are used to generate .chr files.
* Added makechr, a script to (re-)generate .chr files.
* Enhanced the status reporting to include four distinct speed metrics (g/s,
p/s, c/s, and C/s).
* Added the "--fork=N" and "--node=MIN[-MAX]/TOTAL" options for trivial
parallel and distributed processing.
* In the external mode compiler, treat character literals as unsigned.
* Renamed many of the formats.
* Updated the documentation.
* Relaxed the license for many source files to cut-down BSD.
* Relaxed the license for John the Ripper as a whole from GPLv2 (exact version)
to GPLv2 or newer with optional OpenSSL and unRAR exceptions.
* Assorted other changes have been made.

The following changes have been made between John 1.7.9 and 1.7.9.8:

* Enhanced the support for DES-based tripcodes by making use of the bitslice
DES implementation and supporting OpenMP parallelization.
* Implemented bitmaps for fast initial comparison of computed hashes against
those loaded for cracking.  This is applied before hash table lookups, and it
allows for the use of smaller hash tables (thereby saving memory) while
achieving the same or greater speed that larger hash tables previously did.
The speed increase is due to improved locality of reference (where only the
smaller bitmap is accessed all the time, whereas the larger hash table behind
it is only accessed for a percentage of comparisons and additionally it is
smaller than it would otherwise need to be).
* Tuned the bitmap and hash table sizes and thresholds based on testing on
saltless hashes on a Core 2'ish CPU.
* When cracking LM hashes, don't store the ASCII encodings of the hashes in
memory, but instead reconstruct them from the binary hashes for writing into
john.pot when a password gets cracked.
* With 32-bit x86 builds and at least MMX enabled, the "two hashes at a time"
code for bcrypt is now enabled for GCC 4.2 and newer.  This change is made
based on benchmark results for different builds made with different versions of
GCC on CPUs ranging from Pentium 3 to Core i7.  Unfortunately, there's a known
performance regression with this change on Atom.  Previously, this code was
only enabled for x86-64 and/or OpenMP-enabled builds.
* The formats interface has been enhanced to better support GPU implementations
(in jumbo), as well as fast hashes on multi-CPU systems (not yet made use of).
* Assorted minor corrections to Cygwin builds were made.
* Fixed a bug in the Keyboard external mode (uninitialized variables on
"--restore" or when minlength is greater than 1).
* Enhanced the generic crypt(3) format to handle possible NULL returns from
crypt() and crypt_r().
* Updated the FAQ.

The following changes have been made between John 1.7.8 and 1.7.9:

* Added optional parallelization of the MD5-based crypt(3) code with OpenMP.
* Added optional parallelization of the bitslice DES code with OpenMP.
* Replaced the bitslice DES key setup algorithm with a faster one, which
significantly improves performance at LM hashes, as well as at DES-based
crypt(3) hashes when there's just one salt (or very few salts).
* Optimized the DES S-box x86-64 (16-register SSE2) assembly code.
* Added support for 10-character DES-based tripcodes (not optimized yet).
* Added support for the "$2y$" prefix of bcrypt hashes.
* Added two more hash table sizes (16M and 128M entries) for faster processing
of very large numbers of hashes per salt (over 1M).
* Added two pre-defined external mode variables: "abort" and "status", which
let an external mode request the current cracking session to be aborted or the
status line to be displayed, respectively.
* Made some minor optimizations to external mode function calls and virtual
machine implementation.
* The "--make-charset" option now uses floating-point rather than 64-bit
integer operations, which allows for larger CHARSET_* settings in params.h.
* Added runtime detection of Intel AVX and AMD XOP instruction set extensions,
with optional fallback to an alternate program binary.
* In OpenMP-enabled builds, added support for fallback to a non-OpenMP build
when the requested thread count is 1.
* Added relbench, a Perl script to compare two "john --test" benchmark runs,
such as for different machines, "make" targets, C compilers, optimization
options, or/and versions of John the Ripper.
* Additional public lists of "top N passwords" have been merged into the
bundled common passwords list, and some insufficiently common passwords were
removed from the list.
* Many minor enhancements and a few bug fixes were made.

The following changes have been made between John 1.7.7 and 1.7.8:

* The bitslice DES S-box expressions have been replaced with those generated
by Roman Rusakov specifically for John the Ripper.  The corresponding assembly
code for x86 with MMX, SSE2, and for x86-64 with SSE2 has been re-generated.
For other CPUs and for AVX/XOP, C compilers do a reasonably good job of
generating the code from the supplied C source files (with intrinsics where
relevant).  The S-box expressions that we were using before had a 21% larger
gate count, so theoretically this could provide a 21% speedup.  In practice,
though, a 12% to 14% speedup at DES-based crypt(3) hashes is typical.
This effort has been sponsored by Rapid7: https://www.rapid7.com
* Corrected support for bcrypt (OpenBSD Blowfish) hashes of passwords
containing non-ASCII characters (that is, characters with the 8th bit set).
Added support for such hashes produced by crypt_blowfish up to 1.0.4, which
contained a sign extension bug (inherited from older versions of John).
The old buggy behavior may be enabled per-hash, using the "$2x$" prefix.
* The external mode virtual machine's performance has been improved through
additional multi-op instructions matching common instruction sequences
(assign-pop and some triple- and quad-push VM instructions were added).
* A few minor bug fixes and enhancements were made.

The following changes have been made between John 1.7.6.1 and 1.7.7:

* Added Intel AVX and AMD XOP instruction sets support for bitslice DES
(with C compiler intrinsics).  New make targets: linux-x86-64-avx,
linux-x86-64-xop, linux-x86-avx, and linux-x86-xop (these require recent
versions of GCC and GNU binutils).
* A "dummy" "format" is now supported (plaintext passwords encoded in
hexadecimal and prefixed with "$dummy$") - for faster testing and tuning of
custom wordlists, rule sets, .chr files, and external modes on already known or
artificial passwords, as well as for testing of future and modified versions of
John itself.
* Apache "$apr1$" MD5-based password hashes are now supported along with the
FreeBSD-style MD5-based crypt(3) hashes that were supported previously.  Hashes
of both of these types may be loaded for cracking simultaneously.
* The "--salts" option threshold is now applied before removal of previously
cracked hashes for consistent behavior with interrupted and continued sessions.
* The "Idle = Y" setting (which is the default) is now ignored for
OpenMP-enabled hash types when the actual number of threads is greater than 1.
(Unfortunately, it did not work right at least with GNU libgomp on Linux.)
* When a cracking session terminates or is interrupted, John will now warn the
user if the cracked passwords printed to the terminal while cracking are
potentially incomplete.  It will advise the user to use the "--show" option to
see the complete set of cracked passwords with proper post-processing.
* When loading hashes specified on a line on their own (feature introduced in
1.7.6), the loader will now ignore leading and trailing whitespace.
* Unless a hash type is forced from the command line, the loader will now print
warnings about additional hash types seen in the input files (beyond the hash
type autodetected initially).
* For use primarily by the jumbo patch (and later by future enhancements to the
official versions as well), the loader now includes logic to warn the user of
ambiguous hash encodings (e.g. LM vs. NTLM vs. raw-MD5, all of which may be
represented as 32 hexadecimal characters) and of excessive partial hash
collisions, which it works around (these are typically caused by an incomplete
implementation of a new hash type).
* The "unique" and "unshadow" programs have been made significantly faster.
* "DateTime", "Repeats", "Subsets", "AtLeast1-Simple", "AtLeast1-Generic", and
"Policy" external mode samples have been added to the default john.conf.
* The self-tests have been enhanced to detect more kinds of program bugs.
* A few minor bug fixes and enhancements were made.

The following changes have been made between John 1.7.6 and 1.7.6.1:

* Corrected a logic error introduced in JtR 1.7.4.2: in "single crack" mode,
we need a salt's key buffer even when we have no words corresponding to that
salt's hashes to base candidate passwords on.  We need this buffer to hold
other salts' successful guesses for testing against this salt's hashes.

The following changes have been made between John 1.7.5.1 and 1.7.6:

* Generic crypt(3) support (enabled with "--format=crypt") has been added for
auditing password hash types supported by the system but not yet supported by
John's own optimized cryptographic routines (such as SHA-crypt and SunMD5).
* Optional parallelization of the above has been implemented by means of OpenMP
along with glibc's crypt_r(3) or Solaris' MT-safe crypt(3C).
* Optional parallelization of John's own optimized code for the OpenBSD-style
Blowfish-based crypt(3) (bcrypt) hashes with OpenMP has been added.
* A more suitable version of 32-bit x86 assembly code for Blowfish is now
chosen on Core i7 and similar CPUs (when they happen to run a 32-bit build).
* More optimal DES S-box expressions for PowerPC with AltiVec (making use of
the conditional select operation) contributed by Dumplinger Boy (Dango-Chu)
have been integrated.
* The bitslice DES C source code has been reworked to allow for the use of
arbitrary SIMD intrinsics, which was previously only implemented for AltiVec
as a special case.
* Support for SSE2 and MMX intrinsics with bitslice DES (as an alternative to
the supplied assembly code) has been added (currently only enabled for SSE2 on
x86-64 when compiling with GCC 4.4+).
* Support for mixed-type longer virtual vectors (such as SSE2+MMX, SSE2+ALU,
AltiVec+ALU, and other combinations) with bitslice DES has been added (not
enabled by default yet, primarily intended for easy benchmarks on future CPUs,
with future compiler versions, with even more SIMD instruction sets, and with
different DES S-box expressions that might be available in the future).
* The obsolete 32-bit SPARC assembly implementation of DES has been dropped.
* The loader will now detect password hashes specified on a line on their own,
not only as part of an /etc/passwd or PWDUMP format file.
* When run in "--stdin" mode and reading candidate passwords from a terminal
(to be typed by the user), John will no longer mess with the terminal settings.
* John will now restore terminal settings not only on normal termination or
interrupt, but also when forcibly interrupted with two Ctrl-C keypresses.

The following changes have been made between John 1.7.5 and 1.7.5.1:

* A new numeric variable has been added to the word mangling rules engine:
"p" for position of the character last found with the "/" or "%" commands.

The following changes have been made between John 1.7.4.2 and 1.7.5:

* Support for the use of "--format" along with "--show" or "--make-charset" has
been added.
* The choice of .rec and .log filenames for custom session names has been made
more intuitive.
* Support for "\r" (character lists with repeats) and "\p0" (reference to the
immediately preceding character list/range) has been added to the word mangling
rules preprocessor.
* The undefined and undocumented behavior of some subtle word mangling rules
preprocessor constructs has been changed to arguably be more sensible.
* Some bugs were fixed, most notably JtR crashing on no password hashes loaded
(bug introduced in 1.7.4.2).

The following changes have been made between John 1.7.4 and 1.7.4.2:

* Major performance improvements for processing of very large password files
or sets of files, especially with salt-less or same-salt hashes, achieved
primarily through introduction of two additional hash table sizes (64K and 1M
entries), changes to the loader, and smarter processing of successful guesses
(to accommodate getting thousands of hashes successfully cracked per second).
* Many default buffer and hash table sizes have been increased and thresholds
for the use of hash tables lowered, meaning that John will now tend to use
more memory to achieve better speed (unless it is told not to with the
"--save-memory" option).
* Some previously missed common website passwords found on public lists of
"top N passwords" have been added to the bundled common passwords list.
* Some bugs introduced in 1.7.4 and affecting wordlist mode's elimination of
consecutive duplicate candidate passwords have been fixed.

The following changes have been made between John 1.7.3.4 and 1.7.4:

* Support for back-references and "parallel" ranges has been added to the
word mangling rules preprocessor.
* The notion of numeric variables (to be used for character positions
and substring lengths along with numeric constants supported previously)
has been introduced into the rules engine.  Two pre-defined variables
("l" for initial or updated word's length and "m" for initial or
memorized word's last character position) and 11 user-defined variables
("a" through "k") have been added.  Additionally, there's a new numeric
constant: "z" for "infinite" position or length.
* New rule commands have been added: "A" (append, insert, or prefix with a
string), "X" (extract a substring from memory and insert), "v" (subtract
and assign to a numeric variable).
* New rule reject flags have been added: ":" (no-op, for use along with the
"parallel" ranges feature of the preprocessor) and "p" (reject unless word
pair commands are allowed, for sharing of the same ruleset between "single
crack" and wordlist modes).
* Processing of word mangling rules has been made significantly faster in
multiple ways (caching of the current length, less copying of data, code
and data placement changes for better branch prediction and L1 cache usage,
compiler-friendly use of local variables, code micro-optimizations,
removal of no-op rule commands in an initial pass).
* The default rulesets for "single crack" and wordlist modes have been
revised to make use of the new features, for speed, to produce fewer
duplicates, and to attempt additional kinds of candidate passwords (such
as for years 2010 through 2019 with "year-based" rules).
* The idle priority emulation code has been optimized for lower overhead when
there appears to be no other demand for CPU time.
* The default for the Idle setting has been changed from N to Y.

The following changes have been made between John 1.7.3.1 and 1.7.3.4:

* "make check" has been implemented (for Unix-like systems only).
* The "--test" option will now take an optional argument - the duration of each
benchmark in seconds.
* Section .note.GNU-stack has been added to all assembly files to avoid the
stack area unnecessarily being made executable on Linux systems that use this
mechanism.
* Some very minor bugs that did not affect normal operation have been fixed.
* Some unimportant compiler warnings have been fixed, a source code comment has
been made more verbose and more complete.

The following changes have been made between John 1.7.3 and 1.7.3.1:

* Corrected the x86 assembly files for building on Mac OS X.
* Merged in some generic changes from JtR Pro.

The following changes have been made between John 1.7.2 and 1.7.3:

* Two Blowfish-based crypt(3) hashes may now be computed in parallel for much
better performance on modern multi-issue CPUs with a sufficient number of
registers (e.g., x86-64).
* Bitslice DES assembly code for x86-64 has been converted to use
instruction pointer relative addressing (needed for Mac OS X support).
* New make targets: macosx-universal, macosx-x86-64, solaris-x86-64-cc,
solaris-x86-64-gcc, solaris-x86-sse2-cc, solaris-x86-sse2-gcc,
solaris-x86-mmx-cc, solaris-x86-mmx-gcc, solaris-x86-any-cc, linux-ia64;
other changes to the Makefile.
* Minor bug fixes.
* "DumbForce" and "KnownForce" external mode samples have been added to the
default john.conf.

The following changes have been made between John 1.7.1 and 1.7.2:

* Bitslice DES assembly code for x86-64 making use of the 64-bit mode
extended SSE2 with 16 XMM registers has been added for better performance
at DES-based crypt(3) hashes with x86-64 builds on AMD processors.
* New make target for FreeBSD/x86-64.

The following changes have been made between John 1.7.0.2 and 1.7.1:

* Bitslice DES code for x86 with SSE2 has been added for better performance
at DES-based crypt(3) hashes on Pentium 4 and SSE2-capable AMD processors.
* Assorted high-level changes have been applied to improve performance
on current x86-64 processors.
* New make target for NetBSD/SPARC64.
* Minor source code cleanups.

The following changes have been made between John 1.7 and 1.7.0.2:

* Minor bug and portability fixes.
* Better handling of certain uncommon scenarios and improper uses of John.
* Bonus: "Keyboard" cracker included in the default john.conf (john.ini)
that will try sequences of adjacent keys on a keyboard as passwords.

The following changes have been made between John 1.6 and 1.7:

* Bitslice DES code for x86 with MMX: more than twice faster than older
non-bitslice MMX code.
* Bitsliced the LM hash code as well: now several times faster.
* Significant improvements to the generic bitslice DES code: +20% on RISC.
* PowerPC G4+ AltiVec support (Mac OS X and Linux): effective 128-bitness
for bitslice DES, resulting in huge speedups.
* First attempt at generic vectorization support for bitslice DES.
* Two MD5 hashes at a time for extra ILP on RISC: up to +80% on Alpha EV5+.
* Generic Blowfish x86 assembly code in addition to the original Pentium
version: +15% on the Pentium Pro family (up to and including Pentium III),
+20% on AMD K6 (Pentium 4 and newer AMD CPUs are more happy running the
original Pentium code for Blowfish).
* Verbose logging of events to the global or a session-specific log file.
* Better idle priority emulation with POSIX.1b (POSIX.4) scheduling calls.
* System-wide installation support for *BSD ports and Linux distributions.
* AIX, DU/Tru64 C2, HP-UX tcb files support in unshadow.
* New make targets for Linux/x86-64, Linux/PowerPC, FreeBSD/Alpha,
OpenBSD/x86-64, OpenBSD/Alpha, OpenBSD/SPARC, OpenBSD/SPARC64,
OpenBSD/PowerPC, OpenBSD/PA-RISC, OpenBSD/VAX, NetBSD/VAX, Solaris/SPARC64,
Mac OS X (PowerPC and x86), SCO, BeOS.
* Bug and portability fixes, and new bugs.
* Bonus: "Strip" cracker included in the default john.conf (john.ini).