Skip to content

Commit

Permalink
feat(bench): add suite & formatters, update benchmark()
Browse files Browse the repository at this point in the history 
- add `suite()` benchmark runner
- update `BenchmarkOpts` & `benchmark()`
  - add `size` option to configure calls per iteration
  - add `format` option to configure formatter
- add `BenchmarkFormatter` interface
- add `FORMAT_DEFAULT`, `FORMAT_CSV` & `FORMAT_MD` formatters
  • Loading branch information
@postspectacular
postspectacular committed Mar 12, 2021
1 parent 0329b7b commit 5ea02bd
Show file tree
Hide file tree
Showing 10 changed files with 353 additions and 26 deletions.
61 changes: 58 additions & 3 deletions packages/bench/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,17 +11,20 @@ This project is part of the

- [About](#about)
- [Status](#status)
- [Related packages](#related-packages)
- [Installation](#installation)
- [Dependencies](#dependencies)
- [Usage examples](#usage-examples)
- [API](#api)
- [Benchmarking with statistics](#benchmarking-with-statistics)
- [Benchmark suites](#benchmark-suites)
- [Output formatting](#output-formatting)
- [Authors](#authors)
- [License](#license)

## About

Benchmarking utilities w/ optional statistics.
Benchmarking utilities w/ various statistics & formatters (CSV, Markdown etc.).

Though no public API change (only additions), since v2.0.0 this library
internally attempts to use high-res ES
Expand All @@ -37,6 +40,11 @@ still only sourced via `Date.now()`.

[Search or submit any issues for this package](https://github.com/thi-ng/umbrella/issues?q=%5Bbench%5D+in%3Atitle)

### Related packages

- [@thi.ng/csv](https://github.com/thi-ng/umbrella/tree/develop/packages/csv) - Customizable, transducer-based CSV parser/object mapper and transformer
- [@thi.ng/hiccup-markdown](https://github.com/thi-ng/umbrella/tree/develop/packages/hiccup-markdown) - Markdown parser & serializer from/to Hiccup format

## Installation

```bash
Expand All @@ -51,11 +59,11 @@ yarn add @thi.ng/bench
<script src="https://unpkg.com/@thi.ng/bench/lib/index.umd.js" crossorigin></script>
```

Package sizes (gzipped, pre-treeshake): ESM: 688 bytes / CJS: 750 bytes / UMD: 833 bytes
Package sizes (gzipped, pre-treeshake): ESM: 1.37 KB / CJS: 1.47 KB / UMD: 1.47 KB

## Dependencies

None
- [@thi.ng/api](https://github.com/thi-ng/umbrella/tree/develop/packages/api)

## Usage examples

Expand Down Expand Up @@ -116,6 +124,9 @@ See
[api.ts](https://github.com/thi-ng/umbrella/tree/develop/packages/bench/src/api.ts)
for configuration options.

Also see the [formatting](#output-formatting) section below for other output
options. This example uses the default format...

```ts
benchmark(() => fib(40), { title: "fib", iter: 10, warmup: 5 });
// benchmarking: fib
Expand All @@ -128,6 +139,7 @@ benchmark(() => fib(40), { title: "fib", iter: 10, warmup: 5 });

// also returns results:
// {
// title: "fib",
// iter: 10,
// total: 7333.72402,
// mean: 733.372402,
Expand All @@ -140,6 +152,49 @@ benchmark(() => fib(40), { title: "fib", iter: 10, warmup: 5 });
// }
```

### Benchmark suites

Multiple benchmarks can be run sequentially as suite (also returns an array of
all results):

```ts
b.suite(
[
{ title: "fib2(10)", fn: () => fib2(10) },
{ title: "fib2(20)", fn: () => fib2(20) },
{ title: "fib2(30)", fn: () => fib2(30) },
{ title: "fib2(40)", fn: () => fib2(40) },
],
{ iter: 10, size: 100000, warmup: 5, format: b.FORMAT_MD }
)

// | Title| Iter| Size| Total| Mean| Median| Min| Max| Q1| Q3| SD%|
// |------------------------|-------:|-------:|-----------:|-------:|-------:|-------:|-------:|-------:|-------:|-------:|
// | fib2(10)| 10| 100000| 54.34| 5.43| 5.15| 4.40| 8.14| 4.84| 6.67| 20.32|
// | fib2(20)| 10| 100000| 121.24| 12.12| 12.13| 11.73| 12.91| 11.93| 12.35| 2.61|
// | fib2(30)| 10| 100000| 152.98| 15.30| 14.51| 13.93| 20.77| 14.35| 16.35| 12.65|
// | fib2(40)| 10| 100000| 164.79| 16.48| 15.60| 15.01| 19.27| 15.42| 18.80| 9.34|
```

Same table as actual Markdown:

| Title| Iter| Size| Total| Mean| Median| Min| Max| Q1| Q3| SD%|
|------------------------|-------:|-------:|-----------:|-------:|-------:|-------:|-------:|-------:|-------:|-------:|
| fib2(10)| 10| 100000| 54.34| 5.43| 5.15| 4.40| 8.14| 4.84| 6.67| 20.32|
| fib2(20)| 10| 100000| 121.24| 12.12| 12.13| 11.73| 12.91| 11.93| 12.35| 2.61|
| fib2(30)| 10| 100000| 152.98| 15.30| 14.51| 13.93| 20.77| 14.35| 16.35| 12.65|
| fib2(40)| 10| 100000| 164.79| 16.48| 15.60| 15.01| 19.27| 15.42| 18.80| 9.34|

### Output formatting

The following output formatters are available. Custom formatters can be easily
defined (see source for examples). Formatters are configured via the `format`
option given to `benchmark()` or `suite()`.

- `FORMAT_DEFAULT` - default plain text formatting
- `FORMAT_CSV` - Comma-separated values (w/ column header)
- `FORMAT_MD` - Markdown table format

## Authors

Karsten Schmidt
Expand Down
13 changes: 12 additions & 1 deletion packages/bench/package.json
View file
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"name": "@thi.ng/bench",
"version": "2.0.31",
"description": "Benchmarking utilities w/ optional statistics",
"description": "Benchmarking utilities w/ various statistics & formatters (CSV, Markdown etc.)",
"module": "./index.js",
"main": "./lib/index.js",
"umd:main": "./lib/index.umd.js",
Expand Down Expand Up @@ -48,6 +48,9 @@
"typedoc": "^0.20.28",
"typescript": "^4.2.2"
},
"dependencies": {
"@thi.ng/api": "^7.1.3"
},
"files": [
"*.js",
"*.d.ts",
Expand All @@ -56,11 +59,15 @@
"keywords": [
"benchmark",
"bigint",
"csv",
"execution",
"format",
"functional",
"hrtime",
"markdown",
"measure",
"statistics",
"table",
"timing",
"typescript"
],
Expand All @@ -73,6 +80,10 @@
},
"sideEffects": false,
"thi.ng": {
"related": [
"csv",
"hiccup-markdown"
],
"year": 2018
}
}
79 changes: 77 additions & 2 deletions packages/bench/src/api.ts
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import type { Fn, Fn0, Fn2 } from "@thi.ng/api";

export type TimingResult<T> = [T, number];

export interface BenchmarkOpts {
Expand All @@ -11,25 +13,47 @@ export interface BenchmarkOpts {
* @defaultValue 1000
*/
iter: number;
/**
* Number of calls per iteration, i.e. total number of iterations will be
* `iter * size`.
*
* @defaultValue 1
*/
size: number;
/**
* Number of warmup iterations (not included in results).
*
* @defaultValue 10
*/
warmup: number;
/**
* If true, writes progress & results to console.
* Result formatter
*
* @defaultValue FORMAT_DEFAULT
*/
format: BenchmarkFormatter;
/**
* If false, all output will be supressed.
*
* @defaultValue true
*/
print: boolean;
output: boolean;
}

export type OptsWithoutTitle = Omit<BenchmarkOpts, "title">;

export interface BenchmarkSuiteOpts extends OptsWithoutTitle {}

export interface BenchmarkResult {
title: string;
/**
* Number of iterations
*/
iter: number;
/**
* Number of calls per iteration
*/
size: number;
/**
* Total execution time for all runs (in ms)
*/
Expand Down Expand Up @@ -65,3 +89,54 @@ export interface BenchmarkResult {
*/
sd: number;
}

export interface BenchmarkFormatter {
/**
* Called once before the benchmark suite runs any benchmarks.
*/
prefix: Fn0<string>;
/**
* Called once for each given benchmark in the suite. Receives benchmark
* options.
*/
start: Fn<BenchmarkOpts, string>;
/**
* Called once per benchmark, just after warmup. Receives warmup time taken
* (in milliseconds) and benchmark opts.
*/
warmup: Fn2<number, BenchmarkOpts, string>;
/**
* Called once per benchmark with collected result.
*/
result: Fn<BenchmarkResult, string>;
/**
* Called once after all benchmarks have run. Receives array of all results.
*/
total: Fn<BenchmarkResult[], string>;
/**
* Called at the very end of the benchmark suite. Useful if a format
* requires any form of final suffix.
*/
suffix: Fn0<string>;
}

export interface Benchmark {
/**
* Benchmark title
*/
title: string;
/**
* Benchmark function. Will be called `size` times per `iter`ation (see
* {@link BenchmarkOpts}).
*/
fn: Fn0<void>;
/**
* Optional & partial benchmark specific option overrides (merged with opts
* given to suite)
*/
opts?: Partial<OptsWithoutTitle>;
}

export const FLOAT = (x: number) => x.toFixed(2);

export const EMPTY = () => "";
51 changes: 31 additions & 20 deletions packages/bench/src/benchmark.ts
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,28 @@
import type { BenchmarkOpts, BenchmarkResult } from "./api";
import { benchResult } from "./bench";
import { timedResult } from "./timed";
import { FORMAT_DEFAULT } from "./format/default";

export const DEFAULT_OPTS: BenchmarkOpts = {
title: "benchmark",
iter: 1e3,
size: 1,
warmup: 10,
output: true,
format: FORMAT_DEFAULT,
};

export const benchmark = (
fn: () => void,
opts?: Partial<BenchmarkOpts>
): BenchmarkResult => {
opts = { title: "", iter: 1e3, warmup: 10, print: true, ...opts };
const { iter, warmup, print } = opts;
print && console.log(`benchmarking: ${opts.title}`);
const t = benchResult(fn, warmup)[1];
print && console.log(`\twarmup... ${t.toFixed(2)}ms (${warmup} runs)`);
print && console.log("\texecuting...");
const _opts = <BenchmarkOpts>{ ...DEFAULT_OPTS, ...opts };
const { iter, size, warmup, output, format } = _opts;
output && outputString(format!.start(_opts));
const t = benchResult(fn, warmup * size)[1];
output && outputString(format!.warmup(t, _opts));
const samples: number[] = [];
for (let i = iter!; --i >= 0; ) {
samples.push(timedResult(fn)[1]);
samples.push(benchResult(fn, size)[1]);
}
samples.sort((a, b) => a - b);
const total = samples.reduce((acc, x) => acc + x, 0);
Expand All @@ -30,18 +38,10 @@ export const benchmark = (
) /
mean) *
100;
if (print) {
console.log(`\ttotal: ${total.toFixed(2)}ms, runs: ${iter}`);
console.log(
`\tmean: ${mean.toFixed(2)}ms, median: ${median.toFixed(
2
)}ms, range: [${min.toFixed(2)}..${max.toFixed(2)}]`
);
console.log(`\tq1: ${q1.toFixed(2)}ms, q3: ${q3.toFixed(2)}ms`);
console.log(`\tsd: ${sd.toFixed(2)}%`);
}
return {
iter: iter!,
const res: BenchmarkResult = {
title: _opts.title,
iter,
size,
total,
mean,
median,
Expand All @@ -51,4 +51,15 @@ export const benchmark = (
q3,
sd,
};
output && outputString(format!.result(res));
return res;
};

/**
* Only outputs non-empty strings to console.
*
* @param str
*
* @internal
*/
export const outputString = (str: string) => str !== "" && console.log(str);
22 changes: 22 additions & 0 deletions packages/bench/src/format/csv.ts
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
import { BenchmarkFormatter, EMPTY, FLOAT } from "../api";

export const FORMAT_CSV: BenchmarkFormatter = {
prefix: () => `Title,Iterations,Size,Total,Mean,Median,Min,Max,Q1,Q3,SD%`,
start: EMPTY,
warmup: EMPTY,
result: (res) =>
`"${res.title}",${res.iter},${res.size},${[
res.total,
res.mean,
res.median,
res.min,
res.max,
res.q1,
res.q3,
res.sd,
]
.map(FLOAT)
.join(",")}`,
total: EMPTY,
suffix: EMPTY,
};
18 changes: 18 additions & 0 deletions packages/bench/src/format/default.ts
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
import { BenchmarkFormatter, EMPTY, FLOAT } from "../api";

export const FORMAT_DEFAULT: BenchmarkFormatter = {
prefix: EMPTY,
start: ({ title }) => `benchmarking: ${title}`,
warmup: (t, { warmup }) => `\twarmup... ${FLOAT(t)}ms (${warmup} runs)`,
result: ({ iter, size, total, mean, median, min, max, q1, q3, sd }) =>
// prettier-ignore
`\ttotal: ${FLOAT(total)}ms, runs: ${iter} (@ ${size} calls/iter)
\tmean: ${FLOAT(mean)}ms, median: ${FLOAT(median)}ms, range: [${FLOAT(min)}..${FLOAT(max)}]
\tq1: ${FLOAT(q1)}ms, q3: ${FLOAT(q3)}ms
\tsd: ${FLOAT(sd)}%`,
total: (res) => {
const fastest = res.slice().sort((a, b) => a.mean - b.mean)[0];
return `Fastest: "${fastest.title}"`;
},
suffix: () => `---`,
};
Loading

0 comments on commit 5ea02bd

Please sign in to comment.