☰
Bencher is available on the Github marketplace!
Bencher is available on the Github marketplace!
Navigation :

Configuration

Bencher can be customized by adding .bencher/config.yaml to the root of the repository.

Suppressing errors on regression

By default, bencher’s check will pass if a change contains a regression but benchmarks are otherwise successful (the threshold for what’s considered a regression can be customized using rules, as described below). To cause the check to fail on a regression, suppress_failure_on_regression can be set to false.

# Defaults to true.
suppress_failure_on_regression: false

Rules

Rules can be used to configure the minimum threshold % for what’s considered an improvement and a what’s considered a regression.

Rule Types

Rules can either be applied globally or per-directory.

Global Rules

Global rules are placed under the global tag, and affect files in the entire repository.

global:
  reg_min: 10
  imp_min: -10
Per-Directory Rules

Rules can also be applied to files whose directory matches a regular expression.

per_dir:
  worker/dir.+:
    reg_min: 10
    imp_min: -15
  doc/tail.*:
    reg_min: 5
    imp_min: -5

Rule Fields

reg_min

Required Field

The minimum (exclusive) threshold % for a change to be considered a regression. Any benchmark with a delta greater than reg_min will be classified as a regression. For example:

global:
  reg_min: 10 # A benchmark should increase by more than 10% to be consdered a regression.
  imp_min: -10
imp_min

Required Field

The minimum (exclusive) threshold % for a change to be considered an improvement. Any benchmark with a delta less than imp_min will be classified as an improvement. For example:

global:
  imp_min: -10 # A benchmark should decrease by more than 10% to be considered an improvement.
  reg_min: 10
benchmarks

Allows for the configuration of thresholds for benchmarks whose name matches a regular expression. reg_min and imp_min can be set for each benchmark.

global:
  benchmarks:
    SendRecv/msgsize:
      reg_min: 55
      imp_min: -20
    GoParse3339InLocation:
      reg_min: 5
      imp_min: -4
dimensions

Allows for the configuration of thresholds on a per-dimension basis. reg_min and imp_min can be set for each dimension.

global:
  dimensions:
    alloc/op:
      reg_min: 5
      imp_min: -2.47

    lost:
      reg_min: 16.78
      imp_min: -25.17

    foo:
      reg_min: 20
      imp_min: -12

Resolving req_min and imp_min

req_min and imp_min are determined from a benchmark through the following process (Ties are currently broken arbitrarily, but eventually, whatever occurs first in the config file will win a tie):

* if there is no configuration:
  * default `reg_min` and `imp_min` to 5.
* else if a `per_dir` rule matches the benchmark's directory:
  * if any sub-rule in `benchmarks` matches the benchmark's name:
    * use the matching `benchmarks` sub-rule.
  * else if `dimensions` sub-rule matches the benchmark's dimension:
    * use the matching `dimensions` sub-rule.
  * else use the matching `per_dir` rule.
* else if a `benchmarks` rule in `global` matches the benchmark's name:
  * use the matching `benchmarks` rule.
* else if a `dimensions` rule in `global` matches the benchmark's dimension:
  * use the matching `dimensions` rule.
* else:
  * use the `global` rule.

Dependencies

If your benchmarks require external dependencies (such as a database), bencher provides a config section that lets you configure docker images to be spun up before benchmarks.

dependencies:
  services:
    redis:
      # Specify the name of the docker image.
      image: redis
      # List of container's ports which will be available to the benchmarks at 127.0.0.1:$PORT
      ports:
        - 6379
      # Volume mounts can reference a named volume (defined later) or a path relative to the root of the repository.
      volumes:
        # Volume-mounting a file in the repo.
        - ./dev/redis.conf:/etc/redis/redis.conf
        # Mounting a named volume.
        - redis_data:/data
    postgresql:
      image: postgres:12
      ports:
        - 5432
      environment:
        POSTGRES_PASSWORD: supersecure
        POSTGRES_USER: groot
        POSTGRES_DB: mydb
      volumes:
        - postgres_12_data:/data/pgdata-12
  # List of named volumes to create.
  volumes:
    - redis_data
    - postgres_12_data

Environment Variables

Environment variables can be specified in the top-level environment field to have them passed to benchmarks. The variables GOPATH, HOME, and PATH are currently reserved and will be ignored if specified.

environment:
  # It takes a little while for the database to start up on bencher, so wait a bit longer for the DB to start.
  DB_STARTUP_TIMEOUT: 60s
  # TODO: change this before launching our product.
  STORE_PASSWORDS_IN_PLAINTEXT: true
  # It has numbers so it's secure!
  ADMIN_PASSWORD: 0r1jt3ch
  # Why is this even a thing??
  BECOME_SKYNET_AND_TAKE_OVER_THE_WORLD: false

Example Configuration

suppress_failure_on_regression: true
global:
  reg_min: 10.9
  imp_min: -25.5
  benchmarks:
  dimensions:
     alloc/op:
       reg_min: 5
       imp_min: -2.47

     lost:
       reg_min: 16.78
       imp_min: -25.17

     foo:
       reg_min: 20
       imp_min: -12

per_dir:
  worker/dir.+:
    reg_min: 10
    imp_min: 15

  doc/tail.+:
    reg_min: 70
    imp_min: -58
    benchmarks:
      SendRecv/msgsize:
        reg_min: 55
        imp_min: -20

      GoParse3339InLocation:
        reg_min: 5
        imp_min: -4
dependencies:
  services:
    redis:
      image: redis
      ports:
      - 6379
      volumes:
        - ./dev/redis.conf:/etc/redis/redis.conf
        - redis_data:/data
    postgresql:
      image: postgres:12
      ports:
        - 5432
      environment:
        POSTGRES_PASSWORD: supersecure
        POSTGRES_USER: groot
        POSTGRES_DB: mydb
      volumes:
        - postgres_12_data:/data/pgdata-12
  volumes:
    - redis_data
    - postgres_12_data
environment:
  DB_STARTUP_TIMEOUT: 60s