☰
Bencher is available on the Github marketplace!
Bencher is available on the Github marketplace!
Navigation :

Configuration

Bencher can be customized by adding .bencher/config.yaml to the root of the repository.

Suppressing errors on regression

By default, bencher’s check will pass if a change contains a regression but benchmarks are otherwise successful (the threshold for what’s considered a regression can be customized using rules, as described below). To cause the check to fail on a regression, suppress_failure_on_regression can be set to false.

# Defaults to true.
suppress_failure_on_regression: false

Rules

Rules can be used to configure the minimum threshold % for what’s considered an improvement and a what’s considered a regression.

Rule Types

Rules can either be applied globally or per-directory.

Global Rules

Global rules are placed under the global tag, and affect files in the entire repository.

global:
  reg_min: 10
  imp_min: -10
Per-Directory Rules

Rules can also be applied to files whose directory matches a regular expression.

per_dir:
  worker/dir.+:
    reg_min: 10
    imp_min: -15
  doc/tail.*:
    reg_min: 5
    imp_min: -5

Rule Fields

reg_min

Required Field

The minimum (exclusive) threshold % for a change to be considered a regression. Any benchmark with a delta greater than reg_min will be classified as a regression. For example:

global:
  reg_min: 10 # A benchmark should increase by more than 10% to be consdered a regression.
  imp_min: -10
imp_min

Required Field

The minimum (exclusive) threshold % for a change to be considered an improvement. Any benchmark with a delta less than imp_min will be classified as an improvement. For example:

global:
  imp_min: -10 # A benchmark should decrease by more than 10% to be considered an improvement.
  reg_min: 10
benchmarks

Allows for the configuration of thresholds for benchmarks whose name matches a regular expression. reg_min and imp_min can be set for each benchmark.

global:
  benchmarks:
    SendRecv/msgsize:
      reg_min: 55
      imp_min: -20
    GoParse3339InLocation:
      reg_min: 5
      imp_min: -4
dimensions

Allows for the configuration of thresholds on a per-dimension basis. reg_min and imp_min can be set for each dimension.

global:
  dimensions:
    alloc/op:
      reg_min: 5
      imp_min: -2.47

    lost:
      reg_min: 16.78
      imp_min: -25.17

    foo:
      reg_min: 20
      imp_min: -12

Resolving req_min and imp_min

req_min and imp_min are determined from a benchmark through the following process (Ties are currently broken arbitrarily, but eventually, whatever occurs first in the config file will win a tie):

* if there is no configuration:
  * default `reg_min` and `imp_min` to 5.
* else if a `per_dir` rule matches the benchmark's directory:
  * if any sub-rule in `benchmarks` matches the benchmark's name:
    * use the matching `benchmarks` sub-rule.
  * else if `dimensions` sub-rule matches the benchmark's dimension:
    * use the matching `dimensions` sub-rule.
  * else use the matching `per_dir` rule.
* else if a `benchmarks` rule in `global` matches the benchmark's name:
  * use the matching `benchmarks` rule.
* else if a `dimensions` rule in `global` matches the benchmark's dimension:
  * use the matching `dimensions` rule.
* else:
  * use the `global` rule.

Example Configuration

suppress_failure_on_regression: true
global:
  reg_min: 10.9
  imp_min: -25.5
  benchmarks:
  dimensions:
     alloc/op:
       reg_min: 5
       imp_min: -2.47

     lost:
       reg_min: 16.78
       imp_min: -25.17

     foo:
       reg_min: 20
       imp_min: -12

per_dir:
  worker/dir.+:
    reg_min: 10
    imp_min: 15

  doc/tail.+:
    reg_min: 70
    imp_min: -58
    benchmarks:
      SendRecv/msgsize:
        reg_min: 55
        imp_min: -20

      GoParse3339InLocation:
        reg_min: 5
        imp_min: -4