The goal of bench is to benchmark code, tracking execution time, memory allocations and garbage collections.

Installation

You can install the development version from GitHub with:

Features

bench::mark() is used to benchmark one or a series of expressions, we feel it has a number of advantages over alternatives.

  • Always uses the highest precision APIs available for each operating system (often nanoseconds).
  • Tracks memory allocations for each expression.
  • Tracks the number and type of R garbage collections per expression iteration.
  • Verifies equality of expression results by default, to avoid accidentally benchmarking inequivalent code.
  • Has bench::press(), which allows you to easily perform and combine benchmarks across a large grid of values.
  • Uses adaptive stopping by default, running each expression for a set amount of time rather than for a specific number of iterations.
  • Expressions are run in batches and summary statistics are calculated after filtering out iterations with garbage collections. This allows you to isolate the performance and effects of garbage collection on running time (for more details see Neal 2014).

The times and memory usage are returned as custom objects which have human readable formatting for display (e.g. 104ns) and comparisons (e.g. x$mem_alloc > "10MB").

There is also full support for plotting with ggplot2 including custom scales and formatting.

Usage

bench::mark()

Benchmarks can be run with bench::mark(), which takes one or more expressions to benchmark against each other.

bench::mark() will throw an error if the results are not equivalent, so you don’t accidentally benchmark inequivalent code.

Results are easy to interpret, with human readable units.

By default the summary uses absolute measures, however relative results can be obtained by using relative = TRUE in your call to bench::mark() or calling summary(relative = TRUE) on the results.

bench::press()

bench::press() is used to run benchmarks against a grid of parameters. Provide setup and benchmarking code as a single unnamed argument then define sets of values as named arguments. The full combination of values will be expanded and the benchmarks are then pressed together in the result. This allows you to benchmark a set of expressions across a wide variety of input sizes, perform replications and other useful tasks.

set.seed(42)

create_df <- function(rows, cols) {
  as.data.frame(setNames(
    replicate(cols, runif(rows, 1, 1000), simplify = FALSE),
    rep_len(c("x", letters), cols)))
}

results <- bench::press(
  rows = c(10000, 100000),
  cols = c(10, 100),
  {
    dat <- create_df(rows, cols)
    bench::mark(
      min_iterations = 100,
      bracket = dat[dat$x > 500, ],
      which = dat[which(dat$x > 500), ],
      subset = subset(dat, x > 500)
    )
  }
)
#> Running with:
#>     rows  cols
#> 1  10000    10
#> 2 100000    10
#> 3  10000   100
#> 4 100000   100
results
#> # A tibble: 12 x 12
#>    expression   rows  cols      min     mean   median      max `itr/sec` mem_alloc  n_gc n_itr total_time
#>    <chr>       <dbl> <dbl> <bch:tm> <bch:tm> <bch:tm> <bch:tm>     <dbl> <bch:byt> <dbl> <int>   <bch:tm>
#>  1 bracket     10000    10 777.93µs   1.06ms 945.44µs   2.05ms    939.      1.17MB    19   206   219.34ms
#>  2 which       10000    10 428.34µs 561.91µs 501.98µs   1.52ms   1780.    827.04KB    34   617    346.7ms
#>  3 subset      10000    10 862.13µs    1.1ms 979.71µs   2.24ms    908.      1.28MB    31   298   328.04ms
#>  4 bracket    100000    10  14.02ms  15.79ms  15.79ms  18.75ms     63.3    11.54MB    46    54   852.51ms
#>  5 which      100000    10   9.11ms  11.22ms  11.04ms  14.59ms     89.1     7.91MB    28    72   808.14ms
#>  6 subset     100000    10  14.92ms   16.8ms  16.51ms  19.87ms     59.5    12.68MB    56    44   739.24ms
#>  7 bracket     10000   100   6.77ms   8.55ms   8.34ms  11.69ms    117.      9.71MB    30    70   598.76ms
#>  8 which       10000   100   2.71ms   3.81ms   3.76ms   7.05ms    262.      5.91MB    18    88   335.59ms
#>  9 subset      10000   100   6.56ms   8.33ms   8.16ms  12.38ms    120.      9.84MB    33    67   557.92ms
#> 10 bracket    100000   100   99.4ms  107.1ms  109.8ms  112.1ms      9.34   97.47MB    98     3    321.3ms
#> 11 which      100000   100  52.18ms  58.46ms  59.46ms  65.01ms     17.1    59.51MB    59    41       2.4s
#> 12 subset     100000   100 103.06ms 110.99ms 108.92ms 121.19ms      9.01   98.62MB    86    17      1.89s

Plotting

ggplot2::autoplot() can be used to generate an informative default plot. This plot is colored by gc level (0, 1, or 2) and faceted by parameters (if any). By default it generates a beeswarm plot, however you can also specify other plot types (jitter, ridge, boxplot, violin). See ?autoplot.bench_mark for full details.

You can also produce fully custom plots by un-nesting the results and working with the data directly.

system_time()

bench also includes system_time(), a higher precision alternative to system.time().