Tools to benchmark Python solutions on runtime performance and visualize. Based on timeit
, it primarily aims to functionally simulate the timeit behaviour and hence the name! This facilitates benchmarking on multiple datasets and solutions.
Latest PyPI stable release :
pip install benchit
Pull latest development release on GitHub and install in the current directory :
pip install -e git+https://github.com/droyed/benchit.git@master#egg=benchit
Consider a setup to compare NumPy ufuncs - sum, prod, max on arrays varying in their sizes. To keep it simple, let's consider 1D
arrays. Thus, we would have :
>>> import numpy as np
>>> funcs = [np.sum,np.prod,np.max]
>>> inputs = [np.random.rand(i) for i in 10**np.arange(5)]
>>> import benchit
>>> t = benchit.timings(funcs, inputs)
It's a dataframe-like object and as such we can plot it. It automatically adds in specs into the title area to convey all of available benchmarking info :
>>> t.plot(logy=True, logx=True)
Let's consider a setup where functions accept more than one argument. Let's take the case of computing euclidean distances between two 2D
arrays. We will feed in arrays with varying number of rows and 3 columns to represent data in 3D
Cartesian coordinate system and benchmark two commonly used functions in Python.
>>> from sklearn.metrics.pairwise import pairwise_distances
>>> from scipy.spatial.distance import cdist
>>> fns = [cdist, pairwise_distances]
>>> import numpy as np
>>> in_ = {n:[np.random.rand(n,3), np.random.rand(n,3)] for n in [10,100,500,1000,4000]}
>>> t = benchit.timings(fns, in_, multivar=True, input_name='Array-length')
>>> t.plot(logx=True)
We will extend the previous example to make the second argument a variable too and study the trend as we vary the number of columns, resulting in subplots.
# Get benchmarking object (dataframe-like) and plot results
>>> R = np.random.rand
>>> in_ = {(n,W):[R(n,W), R(n,W)] for n in [10, 100, 500, 1000] for W in [3, 20, 50, 100]}
>>> t = benchit.timings(fns, in_, multivar=True, input_name=['nrows', 'ncols'])
>>> t.plot(logx=True, sp_ncols=2, sp_argID=0, sp_sharey='g')
For plotting, we are using number of rows as the x-axis base.
Use sp_argID=1
to switch-over to use number of cols as the x-axis base instead.
Let's manufacture a simple forward-filling scheme based on indices of True values in a boolean-array :
# Functions
def repeat(b):
idx = np.flatnonzero(np.r_[b,True])
return np.repeat(idx[:-1], np.diff(idx))
def maxaccum(b):
return np.maximum.accumulate(np.where(b,np.arange(len(b)), 0))
in_ = {(n,sf): np.random.rand(n)<(100-sf)/100. for n in [100,1000,10000,100000,1000000] for sf in [20, 40, 60, 80, 90, 95]}
t = benchit.timings([repeat, maxaccum], in_, input_name=['Array-length','Sparseness %'])
t.plot(logx=True, sp_ncols=2, save='singlegrp_id0_ffillmask_timings.png')
1. Plotting on notebooks?
Use benchit.setparams(environ='notebook')
before plotting. Check out sample notebook run.
2. Get a quick glance into the benchmarking trend before the actual one
Use benchit.setparams(rep=1)
before plotting. Then, use benchit.setparams()
for a proper benchmarking.
3. Get a quicker glance into plot layout and vague benchmarking trend before the actual one
Use benchit.setparams(timeout=1e-5, rep=1)
before plotting. Then, use benchit.setparams()
for a proper benchmarking.
4. Working with multi-variable datasets to study trend w.r.t. each argument?
Use nested loops to set-up input datasets as shown earlier. More information is available in documentation.
As a general rule, it's advisable to work on Python 3.6
or newer for better plotting experience.