You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 86c700e
Browse filesBrowse the repository at this point in the historyBrowse files
Copy file name to clipboardExpand all lines: docs/config/benchmark.md
+8-1Lines changed: 8 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -46,11 +46,18 @@ When defined, Vitest will run all matched files with `import.meta.vitest` inside
46
46
47
47
Include the `samples` array of per-iteration timings on every benchmark result. Disabled by default to reduce memory usage; enable when a custom reporter or API consumer needs the raw samples.
48
48
49
+
## benchmark.provider
50
+
51
+
-**Type:**`string`
52
+
-**Default:**`undefined` (uses the built-in provider)
53
+
54
+
The benchmark provider that executes registered benchmarks and returns their results. Set this to a module path whose default export implements `BenchmarkProvider`. Relative paths are resolved from the project root.
55
+
56
+
See the [Custom Benchmark Provider](/guide/advanced/benchmark-provider) guide for setup instructions and the provider API.
49
57
50
58
## benchmark.suppressExportGetterWarnings
51
59
52
60
-**Type:**`boolean`
53
61
-**Default:**`false`
54
62
55
63
Suppress the warning printed when a benchmark accesses module export getters too many times. Vitest tracks getter access during benchmark runs because Vite's module runner wraps every export in a getter, and excessive access can dominate the measurement (see [Module Runner Overhead](/guide/benchmarking#module-runner-overhead)). Enable this when you've intentionally accepted the overhead, or when the warning is noisy for benchmarks where the getter cost is negligible.
This is an advanced, experimental API. If you only need to run benchmarks with Vitest's built-in provider, read the [Benchmarking](/guide/benchmarking) guide instead.
5
+
:::
6
+
7
+
Vitest uses a benchmark provider to execute the functions registered with `bench` and convert their measurements into results that Vitest can report. The built-in provider uses [Tinybench](https://github.com/tinylibs/tinybench), but you can replace it to use another benchmarking engine or execution strategy.
8
+
9
+
## Setup
10
+
11
+
Set [`benchmark.provider`](/config/benchmark#benchmark-provider) to the path of your provider module. Relative paths are resolved from the project root.
12
+
13
+
```ts [vitest.config.ts]
14
+
import { defineConfig } from'vitest/config'
15
+
16
+
exportdefaultdefineConfig({
17
+
test: {
18
+
benchmark: {
19
+
provider: './benchmark-provider.ts',
20
+
},
21
+
},
22
+
})
23
+
```
24
+
25
+
The module must has a default export with an object that implements `BenchmarkProvider`. This example wraps Tinybench to demonstrate how registrations and results flow through a provider. If you use Tinybench in your provider, add it as a direct dependency of your project.
for (const { name, fn, fnOpts } ofregistrations) {
40
+
bench.add(name, fn, fnOpts)
41
+
}
42
+
43
+
awaitbench.run()
44
+
45
+
returnbench.tasks.map((task) => {
46
+
const result =task.result
47
+
48
+
if (result.state==='errored') {
49
+
throwresult.error
50
+
}
51
+
if (result.state!=='completed') {
52
+
thrownewError(`Benchmark "${task.name}" ended in the "${result.state}" state`)
53
+
}
54
+
55
+
return {
56
+
...result,
57
+
name: task.name,
58
+
}
59
+
})
60
+
},
61
+
} satisfiesBenchmarkProvider
62
+
63
+
exportdefaultprovider
64
+
```
65
+
66
+
## Provider API
67
+
68
+
Vitest calls `provider.run(group)` when a registration's `.run()` method is called, or once for all runnable registrations passed to `bench.compare()`. The `group` contains:
69
+
70
+
-`test`: the test that registered the benchmarks. `test.context.signal` is aborted when the run is cancelled.
71
+
-`config`: the resolved benchmark configuration for the current project.
72
+
-`registrations`: runnable benchmarks in registration order. Every registration contains `name`, `fn`, and optional `fnOpts` for lifecycle hooks, cancellation, async behavior, and sample retention.
73
+
-`options`: benchmark run options passed to `.run()` or `bench.compare()`, if any.
74
+
75
+
The provider is responsible for honoring the run and registration options and for running every benchmark function and its `beforeAll`, `beforeEach`, `afterEach`, and `afterAll` hooks according to the benchmarking engine's lifecycle. If execution fails, throw the error to fail the test.
76
+
77
+
`run` must resolve to one `BenchResult` for every runnable registration. Results are matched to registrations by `name` and are the source for `.run()` return values, comparison tables, reporters, and saved benchmark results. A custom engine must convert its measurements into the Tinybench-compatible `BenchResult` shape exported by `vitest`.
78
+
79
+
Registrations created by `bench.from()` are loaded by Vitest and are not passed to the provider.
80
+
81
+
## Provider Lifetime
82
+
83
+
Vitest imports the provider module on first use and caches its default export for the lifetime of the worker. The API does not have separate setup or teardown hooks; keep worker-scoped state on the provider object when needed.
Copy file name to clipboardExpand all lines: docs/guide/benchmarking.md
+7-5Lines changed: 7 additions & 5 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ title: Benchmarking | Guide
4
4
5
5
# Benchmarking
6
6
7
-
Vitest lets you write benchmarks alongside your tests using the `bench` fixture from the [test context](/guide/test-context). Benchmarks are powered by [Tinybench](https://github.com/tinylibs/tinybench) and are defined inside regular `test()` calls, giving you access to the full power of Vitest's test runner: retries, lifecycle hooks, filtering, and assertions.
7
+
Vitest lets you write benchmarks alongside your tests using the `bench` fixture from the [test context](/guide/test-context). The built-in benchmark provider is powered by [Tinybench](https://github.com/tinylibs/tinybench), and benchmarks are defined inside regular `test()` calls, giving you access to the full power of Vitest's test runner: retries, lifecycle hooks, filtering, and assertions.
The `bench()` function registers a benchmark without executing it. Calling `.run()` runs the benchmark and returns the result. After the test completes, Vitest prints a single-row version of the [comparison table](#comparing-benchmarks) (ops/sec, mean time, percentiles, etc.), so you get the same output for a one-off benchmark as you do for `bench.compare()`.
24
24
25
25
::: warning
26
-
The `bench` fixture is only available in files matched by [`benchmark.include`](/config/#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`). Using `{ bench }` inside a regular test file will throw an error.
26
+
The `bench` fixture is only available in files matched by [`benchmark.include`](/config/benchmark#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`). Using `{ bench }` inside a regular test file will throw an error.
27
27
28
28
Whether a file participates in the benchmark run is decided by the filename, not by whether the test uses the `bench` fixture. Renaming `parser.test.ts` to `parser.bench.ts` (or adjusting `benchmark.include`) is what moves it into the benchmark project.
29
29
:::
30
30
31
31
## Running Benchmarks
32
32
33
-
Benchmark files are matched by [`benchmark.include`](/config/#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`) and run in their own project, separate from your regular tests. There are three ways to run them, depending on whether you want to skip them, run them alongside tests, or run them on their own.
33
+
Benchmark files are matched by [`benchmark.include`](/config/benchmark#benchmark-include) (default: `**/*.{bench,benchmark}.?(c|m)[jt]s?(x)`) and run in their own project, separate from your regular tests. There are three ways to run them, depending on whether you want to skip them, run them alongside tests, or run them on their own.
34
34
35
35
### `vitest` (default)
36
36
37
-
Without [`benchmark.enabled`](/config/#benchmark-enabled), the `vitest` command only runs regular tests. Benchmark files are ignored entirely. This is the default and the right choice for day-to-day development, since benchmarks are slow and noisy and shouldn't run on every save.
37
+
Without [`benchmark.enabled`](/config/benchmark#benchmark-enabled), the `vitest` command only runs regular tests. Benchmark files are ignored entirely. This is the default and the right choice for day-to-day development, since benchmarks are slow and noisy and shouldn't run on every save.
38
38
39
39
### `vitest` with `benchmark.enabled`
40
40
@@ -72,6 +72,8 @@ vitest bench parser
72
72
vitest bench -t JSON
73
73
```
74
74
75
+
To execute benchmarks with another benchmarking engine or execution strategy, see the [Custom Benchmark Provider](/guide/advanced/benchmark-provider) guide.
76
+
75
77
## Comparing Benchmarks
76
78
77
79
Use `bench.compare()` to compare multiple benchmarks against each other:
@@ -330,7 +332,7 @@ Use the same template in `bench.from()` so each project reads its own artifact.
330
332
331
333
Benchmarks are inherently flaky: CPU load, thermal throttling, GC pressure, and background processes all affect results. Vitest takes several steps to minimize this noise:
332
334
333
-
-**Separate project**: Benchmark files are grouped into their own project based on the [`benchmark.include`](/config/#benchmark-include) pattern. The `bench` fixture is only exposed in files matched by that pattern. Using it inside a regular test file will throw an error.
335
+
-**Separate project**: Benchmark files are grouped into their own project based on the [`benchmark.include`](/config/benchmark#benchmark-include) pattern. The `bench` fixture is only exposed in files matched by that pattern. Using it inside a regular test file will throw an error.
334
336
-**No concurrency**: Tests within a benchmark file always run sequentially. Benchmark files themselves also run one at a time, never in parallel. This prevents benchmarks from interfering with each other.
0 commit comments