If you are not familiar with fuzzing, this motivation document provides a quick 1-page introduction.
fzgo
is a prototype of golang/go#19109 "cmd/go: make fuzzing a first class citizen, like tests or benchmarks".
fzgo
supports some conveniences like fuzzing rich signatures and auto-generation of fuzzing functions.
The basic approach is that fzgo
integrates dvyukov/go-fuzz
into go test
, with the heavy lifting being done by go-fuzz
, go-fuzz-build
, and the go
tool. The focus
is on step 1 of a tentative list of "Draft goals for a prototype" outlined in this
comment on #19109:
Step 1. Prototype proposed CLI, including interaction with existing 'go test'.
fzgo
supports the -fuzz
flag and several other related flags proposed in the March 2017
#19109 proposal document. fzgo
also supports typical go
commands
such as fzgo build
, fzgo test
, or fzgo env
(which are implemented by wrapping the go
tool).
Any and all feedback is welcome!
- Rich signatures like
FuzzRegexp(re string, input []byte, posix bool)
are supported, as well as the classicFuzz(data []byte) int
form used bygo-fuzz
. - The corpus is automatically used as deterministic input to unit tests when running a normal
go test
. - Individual corpus files can be unit tested via
fzgo test -fuzz=. -run=TestCorpus/<name>
. go-fuzz
requires a two step process.fzgo
eliminates the separate manual preparation step.fzgo
automatically caches instrumented binaries inGOPATH/pkg/fuzz
and re-uses them if possible.- The fuzzing corpus defaults to
GOPATH/pkg/fuzz/corpus
. - The
-fuzzdir=/some/path
flag allows the corpus to be stored elsewhere (e.g., a separate corpus repo);-fuzzdir=testdata
stores the corpus under<pkgpath>/testdata/fuzz/fuzzname
(hence typically in VCS with the code under test). fuzz
andgofuzz
build tags are allowed but not required.- An optional genfuzzfuncs utility can automatically create fuzzing functions for all of the public functions and methods in a package of interest. This makes it quicker and easier to start fuzzing.
Usage: fzgo test [build/test flags] [packages] [build/test flags]
Examples:
fzgo test # normal 'go test' of current package, plus run any corpus as unit tests
fzgo test -fuzz=. # fuzz the current package with a function starting with 'Fuzz'
fzgo test -fuzz=FuzzFoo # fuzz the current package with a function matching 'FuzzFoo'
fzgo test ./... -fuzz=FuzzFoo # fuzz a package in ./... with a function matching 'FuzzFoo'
fzgo test sample/pkg -fuzz=FuzzFoo # fuzz 'sample/pkg' with a function matching 'FuzzFoo'
Rich signatures like Fuzz(re string, input []byte, posix bool)` are supported, as well Fuzz(data []byte) int.
Fuzz functions must start with 'Fuzz'.
The following flags work with 'fzgo test -fuzz':
-fuzz regexp
fuzz at most one function matching regexp
-fuzzdir dir
store fuzz artifacts in dir (default pkgpath/testdata/fuzz)
-fuzztime d
fuzz for duration d (default unlimited)
-parallel n
start n fuzzing operations (default GOMAXPROCS)
-timeout d
fail an individual call to a fuzz function after duration d (default 10s, minimum 1s)
-c
compile the instrumented code but do not run it
-v
verbose: print additional output
$ go get -u github.com/thepudds/fzgo/...
$ go get -u github.com/dvyukov/go-fuzz/...
Note: if you already have an older dvyukov/go-fuzz
, you might need to first delete it from GOPATH as described in
the dvyukov/go-fuzz repo.
The go-fuzz
source code must be in your GOPATH, and the go-fuzz
and go-fuzz-build
binaries must be
in your path environment variable.
Note: Module-mode is not supported (#15), but you can fuzz a module with fzgo
as long as the code under test is in GOPATH and you set GO111MODULE=off
env variable.
This is a simple prototype. Don't expect great things. ;-)
That said, there is reasonable test coverage and fzgo
is hopefully beta quality. Automatically generating fuzz functions is implemented in a separate genfuzzfuncs utility that is more alpha quality.
Testing is primarily done with the nice internal testscripts
package used by the core Go team to test the go
tool
and extracted at rogpeppe/go-internal/testscript.
The primary changes between the current fzgo prototype vs. the March 2017 proposal document:
- fzgo supports rich signatures.
- The corpus location does not default to
<pkgpath>/testdata/fuzz
, but instead follows the approach outlined here and more precisely described in PR #7. - Initially, fzgo disallowed multiple fuzz functions to match (per the March 2017 proposal), but as an experiment fzgo now allows multiple fuzz functions to match in order to support something like 'go test -fuzz=. ./...' when there are multiple fuzz functions across multiple packages. Fuzzing happens in round-robin manner if multiple fuzz functions match.
- The proposal document suggested
GOPATH/pkg/GOOS_GOARCH_fuzz/
for a cache, but the prototype instead usesGOPATH/pkg/fuzz/GOOS_GOARCH/
. - The initial proposal document suggested generating new mutation-based inputs during
go test
when-fuzz
was not specified. In order to keepgo test
deterministic,fzgo
does not do that, but now does use the corpus as a deterministic set of inputs duringgo test
when-fuzz
is not specified. Also, the proposal document suggested-fuzzinput
as a way of specifying a file from the corpus to execute as a unit test.fzgo
instead uses the normal-run
argument togo test
. For example,fzgo test -run=TestCorpus/4fa128cf066f2a31 some/pkg
runs the any file in thesome/pkg
corpus with a filename matching4fa128cf066f2a31
. - Some of the commentators at #19109 suggested
-fuzztime duration
as a way of controlling when to stop fuzzing. The proposal document does not include-fuzztime
andgo-fuzz
does not support it, but it seems useful in general and-fuzztime
is in the prototype (and it proved useful while testing the prototype). This might be removed later. - For experimentation,
FZGOFLAGSBUILD
andFZGOFLAGSFUZZ
environmental variables can optionally contain a space-separated list of arguments to pass togo-fuzz-build
andgo-fuzz
, respectively.
fuzz.F
ortesting.F
signature for fuzzing function.- Allowing fuzzing functions to reside in
*_test.go
files. - Anything to do with deeper integration with the compiler for more robust instrumentation. This prototype is not focused on that area.
- Any of a much larger set of preexisting build flags like
-ldflags
,-coverprofile
. - Areas covered in the March 2017 proposal document, outside of the direct user-facing behavior that this prototype focuses on. That said, the majority of user-facing behavior mentioned in the proposal document is either implemented in the prototype or explicitly mentioned in this list as not implemented.
The argument parsing in 'go test' is bespoke, and the argument parsing in fzgo
is an approximation of that.
That might be OK for an early prototype. The right thing to do might be to extract
src/cmd/go/internal/test/testflag.go,
which includes this comment:
// The flag handling part of go test is large and distracting.
// We can't use the flag package because some of the flags from
// our command line are for us, and some are for 6.out, and
// some are for both.