Configuration
Bencher can be customized by adding .bencher/config.yaml
to the root of the repository.
Suppressing errors on regression
By default, bencher’s check will pass if a change contains a regression but benchmarks are
otherwise successful (the threshold for what’s considered a regression can be customized using
rules, as described below). To cause the check to fail on a regression,
suppress_failure_on_regression
can be set to false.
# Defaults to true.
suppress_failure_on_regression: false
Rules
Rules can be used to configure the minimum threshold % for what’s considered an improvement and a what’s considered a regression.
Rule Types
Rules can either be applied globally or per-directory.
Global Rules
Global rules are placed under the global
tag, and affect files in the entire repository.
global:
reg_min: 10
imp_min: -10
Per-Directory Rules
Rules can also be applied to files whose directory matches a regular expression.
per_dir:
worker/dir.+:
reg_min: 10
imp_min: -15
doc/tail.*:
reg_min: 5
imp_min: -5
Rule Fields
reg_min
Required Field
The minimum (exclusive) threshold % for a change to be considered a regression. Any benchmark with a delta greater than reg_min
will be classified as a regression. For example:
global:
reg_min: 10 # A benchmark should increase by more than 10% to be consdered a regression.
imp_min: -10
imp_min
Required Field
The minimum (exclusive) threshold % for a change to be considered an improvement. Any benchmark with a delta less than imp_min
will be classified as an improvement. For example:
global:
imp_min: -10 # A benchmark should decrease by more than 10% to be considered an improvement.
reg_min: 10
benchmarks
Allows for the configuration of thresholds for benchmarks whose name matches a regular expression. reg_min
and imp_min
can be set for each benchmark.
global:
benchmarks:
SendRecv/msgsize:
reg_min: 55
imp_min: -20
GoParse3339InLocation:
reg_min: 5
imp_min: -4
dimensions
Allows for the configuration of thresholds on a per-dimension basis. reg_min
and imp_min
can be set for each dimension.
global:
dimensions:
alloc/op:
reg_min: 5
imp_min: -2.47
lost:
reg_min: 16.78
imp_min: -25.17
foo:
reg_min: 20
imp_min: -12
Resolving req_min and imp_min
req_min
and imp_min
are determined from a benchmark through the following process (Ties are currently broken arbitrarily, but eventually, whatever occurs first in the config file will win a tie):
* if there is no configuration:
* default `reg_min` and `imp_min` to 5.
* else if a `per_dir` rule matches the benchmark's directory:
* if any sub-rule in `benchmarks` matches the benchmark's name:
* use the matching `benchmarks` sub-rule.
* else if `dimensions` sub-rule matches the benchmark's dimension:
* use the matching `dimensions` sub-rule.
* else use the matching `per_dir` rule.
* else if a `benchmarks` rule in `global` matches the benchmark's name:
* use the matching `benchmarks` rule.
* else if a `dimensions` rule in `global` matches the benchmark's dimension:
* use the matching `dimensions` rule.
* else:
* use the `global` rule.
Dependencies
If your benchmarks require external dependencies (such as a database), bencher provides a config section that lets you configure docker images to be spun up before benchmarks.
dependencies:
services:
redis:
# Specify the name of the docker image.
image: redis
# List of container's ports which will be available to the benchmarks at 127.0.0.1:$PORT
ports:
- 6379
# Volume mounts can reference a named volume (defined later) or a path relative to the root of the repository.
volumes:
# Volume-mounting a file in the repo.
- ./dev/redis.conf:/etc/redis/redis.conf
# Mounting a named volume.
- redis_data:/data
postgresql:
image: postgres:12
ports:
- 5432
environment:
POSTGRES_PASSWORD: supersecure
POSTGRES_USER: groot
POSTGRES_DB: mydb
volumes:
- postgres_12_data:/data/pgdata-12
# List of named volumes to create.
volumes:
- redis_data
- postgres_12_data
Environment Variables
Environment variables can be specified in the top-level environment
field to have them passed to benchmarks. The variables GOPATH
, HOME
, and PATH
are currently reserved and will be ignored if specified.
environment:
# It takes a little while for the database to start up on bencher, so wait a bit longer for the DB to start.
DB_STARTUP_TIMEOUT: 60s
# TODO: change this before launching our product.
STORE_PASSWORDS_IN_PLAINTEXT: true
# It has numbers so it's secure!
ADMIN_PASSWORD: 0r1jt3ch
# Why is this even a thing??
BECOME_SKYNET_AND_TAKE_OVER_THE_WORLD: false
Example Configuration
suppress_failure_on_regression: true
global:
reg_min: 10.9
imp_min: -25.5
benchmarks:
dimensions:
alloc/op:
reg_min: 5
imp_min: -2.47
lost:
reg_min: 16.78
imp_min: -25.17
foo:
reg_min: 20
imp_min: -12
per_dir:
worker/dir.+:
reg_min: 10
imp_min: 15
doc/tail.+:
reg_min: 70
imp_min: -58
benchmarks:
SendRecv/msgsize:
reg_min: 55
imp_min: -20
GoParse3339InLocation:
reg_min: 5
imp_min: -4
dependencies:
services:
redis:
image: redis
ports:
- 6379
volumes:
- ./dev/redis.conf:/etc/redis/redis.conf
- redis_data:/data
postgresql:
image: postgres:12
ports:
- 5432
environment:
POSTGRES_PASSWORD: supersecure
POSTGRES_USER: groot
POSTGRES_DB: mydb
volumes:
- postgres_12_data:/data/pgdata-12
volumes:
- redis_data
- postgres_12_data
environment:
DB_STARTUP_TIMEOUT: 60s