Quickstart¶
To illustrate neurtu usage, will will benchmark array sorting in numpy. First, we will generator of cases,
import numpy as np
import neurtu
def cases()
rng = np.random.RandomState(42)
for N in [1000, 10000, 100000]:
X = rng.rand(N)
tags = {'N' : N}
yield neurtu.delayed(X, tags=tags).sort()
that yields a sequence of delayed calculations, each tagged with the parameters defining individual runs.
We can evaluate the run time with,
>>> df = neurtu.timeit(cases())
>>> print(df)
wall_time
N
1000 0.000014
10000 0.000134
100000 0.001474
which will internally use timeit
module with a sufficient number of evaluation to work around the timer precision
limitations (similarly to IPython’s %timeit
). It will also display a progress bar for long running benchmarks,
and return the results as a pandas.DataFrame
(if pandas is installed).
By default, all evaluations are run with repeat=1
. If more statistical confidence is required, this value can
be increased,
>>> neurtu.timeit(cases(), repeat=3)
wall_time
mean max std
N
1000 0.000012 0.000014 0.000002
10000 0.000116 0.000149 0.000029
100000 0.001323 0.001714 0.000339
In this case we will get a frame with a
pandas.MultiIndex for
columns, where the first level represents the metric name (wall_time
) and the second – the aggregation method.
By default neurtu.timeit
is called with aggregate=['mean', 'max', 'std']
methods, as supported
by the pandas aggregation API. To disable,
aggregation and obtains timings for individual runs, use aggregate=False
.
See neurtu.timeit documentation for more details.
To evaluate the peak memory usage, one can use the neurtu.memit
function with the same API,
>>> neurtu.memit(cases(), repeat=3)
peak_memory
mean max std
N
10000 0.0 0.0 0.0
100000 0.0 0.0 0.0
1000000 0.0 0.0 0.0
More generally neurtu.Benchmark
supports a wide number of evaluation metrics,
>>> bench = neurtu.Benchmark(wall_time=True, cpu_time=True, peak_memory=True)
>>> bench(cases())
cpu_time peak_memory wall_time
N
10000 0.000100 0.0 0.000142
100000 0.001149 0.0 0.001680
1000000 0.013677 0.0 0.018347
including [psutil process metrics](https://psutil.readthedocs.io/en/latest/#psutil.Process).
For more information see the Examples.