Benchmany runs Go benchmarks across many git commits.
benchmany run [-C git-dir] [-n iterations] <commit or range>... benchmany plot [-C git-dir] <commit or range>... benchmany list [-C git-dir] <commit or range>...
"benchmany run" runs the benchmarks in the current directory <iterations> times for each commit in <commit or range> and writes the benchmark results to log.<commit hash>. Benchmarks may be Go testing framework benchmarks or benchmarks from golang.org/x/benchmarks.
<commit or range>... can be either a list of individual commits or a revision range. For the spelling of a revision range, see "SPECIFYING RANGES" in gitrevisions(7). For exact details, see the --no-walk option to git-rev-list(1).
Benchmany will check out each revision in git-dir. The current directory may or may not be in the same git repository as git-dir. If git-dir refers to a Go installation, benchmany will run make.bash at each revision; otherwise, it assumes go test can rebuild the necessary dependencies. Benchmany also supports using gover (https://godoc.org/github.com/aclements/go-misc/gover) to save and reuse Go build trees. This is useful for saving time across multiple benchmark runs and for benchmarks that depend on the Go tree itself (such as compiler benchmarks).
Benchmany supports multiple ways of prioritizing the order in which individual iterations are run. By default, it runs in "sequential" mode: it runs the first iteration of all benchmarks, then the second, and so forth. It also supports a "spread" mode designed to quickly get coverage for large sets of revisions. This mode randomizes the order to run iterations in, but biases this order toward covering an evenly distributed set of revisions early and finishing all of the iterations of the revisions it has started on before moving on to new revisions. This way, if benchmany is interrupted, the revisions benchmarked cover the space more-or-less evenly. Finally, it supports a "metric" mode, which zeroes in on changes in a benchmark metric by selecting the commit half way between the pair of commits with the biggest difference in the metric. This is like "git bisect", but for performance.
Benchmany is safe to interrupt. If it is restarted, it will parse the benchmark log files to recover its state.
"benchmany plot" reads the benchmark logs generated by "benchmany run" for the given <commit or range> and plots the results of benchmarks over time.
Plot supports various output formats. By default, it outputs a text table that can be used as input to most plotting programs. The -html option produces a Google Charts page that can be used to interactively view benchmark results.
Plot supports a -filter flag that reduces noise in benchmark results using an adaptive Kolmogorov-Zurbenko (KZA) low-pass filter. This filter retains sharp changes in performance, so you can see exactly when significant changes happened, while smoothing out transient noise.
Plot supports normalizing results to a baseline using the -baseline flag. This is useful when looking for relative changes, or when plotting many different benchmarks that may have wildly different expected results.
"benchmany list" prints a list of log file names generated by "benchmany run" for the given <commit or range>. By default, logs are listed from oldest to newest revision. This is useful as input to comparison tools like http://rsc.io/benchstat.