Skip to content

Benchmark a series of functions

Source: R/mark.R
mark.Rd

Benchmark a list of quoted expressions. Each expression will always run at least twice, once to measure the memory allocation and store results and one or more times to measure timing.

Usage

mark(
  ...,
  min_time = 0.5,
  iterations = NULL,
  min_iterations = 1,
  max_iterations = 10000,
  check = TRUE,
  memory = capabilities("profmem"),
  filter_gc = TRUE,
  relative = FALSE,
  time_unit = NULL,
  exprs = NULL,
  env = parent.frame()
)

Arguments

...

Expressions to benchmark, if named the expression column will be the name, otherwise it will be the deparsed expression.

min_time

The minimum number of seconds to run each expression, set to Inf to always run max_iterations times instead.

iterations

If not NULL, the default, run each expression for exactly this number of iterations. This overrides both min_iterations and max_iterations.

min_iterations

Each expression will be evaluated a minimum of min_iterations times.

max_iterations

Each expression will be evaluated a maximum of max_iterations times.

check

Check if results are consistent. If TRUE, checking is done with all.equal(), if FALSE checking is disabled and results are not stored. If check is a function that function will be called with each pair of results to determine consistency.

memory

If TRUE (the default when R is compiled with memory profiling), track memory allocations using utils::Rprofmem(). If FALSE disable memory tracking.

filter_gc

If TRUE remove iterations that contained at least one garbage collection before summarizing. If TRUE but an expression had a garbage collection in every iteration, filtering is disabled, with a warning.

relative

If TRUE all summaries are computed relative to the minimum execution time rather than absolute time.

time_unit

If NULL the times are reported in a human readable fashion depending on each value. If one of 'ns', 'us', 'ms', 's', 'm', 'h', 'd', 'w' the time units are instead expressed as nanoseconds, microseconds, milliseconds, seconds, hours, minutes, days or weeks respectively.

exprs

A list of quoted expressions. If supplied overrides expressions defined in ....

env

The environment which to evaluate the expressions

Value

A tibble with the additional summary columns. The following summary columns are computed

  • expression - bench_expr The deparsed expression that was evaluated (or its name if one was provided).

  • min - bench_time The minimum execution time.

  • median - bench_time The sample median of execution time.

  • itr/sec - double The estimated number of executions performed per second.

  • mem_alloc - bench_bytes Total amount of memory allocated by R while running the expression. Memory allocated outside the R heap, e.g. by malloc() or new directly is not tracked, take care to avoid misinterpreting the results if running code that may do this.

  • gc/sec - double The number of garbage collections per second.

  • n_itr - integer Total number of iterations after filtering garbage collections (if filter_gc == TRUE).

  • n_gc - double Total number of garbage collections performed over all iterations. This is a psudo-measure of the pressure on the garbage collector, if it varies greatly between to alternatives generally the one with fewer collections will cause fewer allocation in real usage.

  • total_time - bench_time The total time to perform the benchmarks.

  • result - list A list column of the object(s) returned by the evaluated expression(s).

  • memory - list A list column with results from Rprofmem().

  • time - list A list column of bench_time vectors for each evaluated expression.

  • gc - list A list column with tibbles containing the level of garbage collection (0-2, columns) for each iteration (rows).

See also

press() to run benchmarks across a grid of parameters.

Examples

dat <- data.frame(x = runif(100, 1, 1000), y=runif(10, 1, 1000))
mark(
  min_time = .1,

  dat[dat$x > 500, ],
  dat[which(dat$x > 500), ],
  subset(dat, x > 500))
#> # A tibble: 3 × 13
#>   expression           min median `itr/sec` mem_alloc `gc/sec` n_itr  n_gc
#>   <bch:expr>        <bch:> <bch:>     <dbl> <bch:byt>    <dbl> <int> <dbl>
#> 1 dat[dat$x > 500,… 26.2µs 28.3µs    34378.    4.15KB     11.7  2942     1
#> 2 dat[which(dat$x … 27.6µs 29.3µs    32800.    2.77KB     27.8  2362     2
#> 3 subset(dat, x > … 46.5µs 49.4µs    18712.    5.46KB     26.4  1415     2
#> # ℹ 5 more variables: total_time <bch:tm>, result <list>, memory <list>,
#> #   time <list>, gc <list>