criterion performance measurements
overview
want to understand this report?
20 x 40/penner
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.1928879180479846e-8 | 3.197123685932725e-6 | 2.1480472696942967e-8 |
| Standard deviation | 1.4808432309528002e-8 | 6.993867387604297e-8 | 3.194962407851257e-8 |
Outlying measurements have moderate (0.245180150332159%) effect on estimated standard deviation.
20 x 40/joshcc4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.7187668987810755e-7 | 4.855574278362733e-5 | 2.9586144187255e-7 |
| Standard deviation | 1.3089423227555922e-7 | 9.356224768654558e-7 | 1.561302266902007e-7 |
Outlying measurements have moderate (0.15102196450088992%) effect on estimated standard deviation.
20 x 40/igastako
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.907401097546603e-6 | 5.060549112735208e-4 | 3.6655791520200966e-6 |
| Standard deviation | 2.174185358495688e-6 | 1.2472573306271282e-5 | 2.820418479423391e-6 |
Outlying measurements have moderate (0.15816074464453808%) effect on estimated standard deviation.
20 x 40/halogen
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.385759869645004e-6 | 2.060840893452662e-4 | 1.3821164747252043e-6 |
| Standard deviation | 8.437219460133669e-7 | 4.76383654308324e-6 | 9.256802846565373e-7 |
Outlying measurements have moderate (0.16754288501285988%) effect on estimated standard deviation.
20 x 40/kuribas
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.455507385107581e-8 | 7.053840826625331e-6 | 1.088566886890371e-7 |
| Standard deviation | 6.574347352217565e-8 | 3.1431417537936966e-7 | 1.6004404410335327e-7 |
Outlying measurements have severe (0.5579844475822898%) effect on estimated standard deviation.
20 x 40/kuribas 2.0
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.3914585734710856e-8 | 4.875464279266973e-6 | 4.5608012161807735e-8 |
| Standard deviation | 2.4529691198230736e-8 | 1.517670041200023e-7 | 3.7406080062378825e-8 |
Outlying measurements have moderate (0.3896725377082858%) effect on estimated standard deviation.
20 x 100/penner
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.730150938745991e-7 | 1.1920047968525723e-5 | 1.6088835326973003e-7 |
| Standard deviation | 8.159534061559223e-8 | 5.446805835125509e-7 | 1.2384257891095594e-7 |
Outlying measurements have severe (0.5536208307702241%) effect on estimated standard deviation.
20 x 100/joshcc4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.625797926438596e-7 | 1.6476635386225127e-4 | 2.8247113268578344e-7 |
| Standard deviation | 1.5377776884583868e-7 | 8.803006284824875e-7 | 2.0866386020696286e-7 |
Outlying measurements have no (9.258450519696416e-3%) effect on estimated standard deviation.
20 x 100/igastako
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.425766605938163e-5 | 8.316741757156606e-3 | 9.271497827991089e-5 |
| Standard deviation | 5.308274598228197e-5 | 2.602286273427096e-4 | 9.952942692333805e-5 |
Outlying measurements have moderate (0.11410662389040739%) effect on estimated standard deviation.
20 x 100/kuribas
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.2423925541321412e-7 | 2.557657625979289e-5 | 2.1172163312484677e-7 |
| Standard deviation | 1.555441949864325e-7 | 7.377496098363851e-7 | 3.308840903215775e-7 |
Outlying measurements have moderate (0.3061363751721711%) effect on estimated standard deviation.
20 x 100/kuribas 2.0
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.347353723584538e-7 | 1.9520268892671485e-5 | 1.883011782143724e-7 |
| Standard deviation | 1.2171773434832546e-7 | 7.242058602904247e-7 | 1.9340857068391514e-7 |
Outlying measurements have moderate (0.431567833855429%) effect on estimated standard deviation.
4 x 10000/penner
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.051241368230145e-5 | 6.06598067434806e-3 | 3.781836325444411e-5 |
| Standard deviation | 2.141410637305599e-5 | 1.1305690811949904e-4 | 2.1821529191313688e-5 |
Outlying measurements have slight (2.562326869806056e-2%) effect on estimated standard deviation.
4 x 10000/joshcc4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.668690446978178e-5 | 4.608408184900912e-3 | 2.5871341398822634e-5 |
| Standard deviation | 1.647936274123908e-5 | 7.971989828927279e-5 | 2.452780927570502e-5 |
Outlying measurements have slight (2.3242630385487607e-2%) effect on estimated standard deviation.
4 x 10000/kuribas
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.9724350896743615e-5 | 9.39385924107498e-3 | 6.371869831667092e-5 |
| Standard deviation | 4.0528282233419824e-5 | 1.3226406744133517e-4 | 5.030327301300109e-5 |
Outlying measurements have slight (3.1217481789802e-2%) effect on estimated standard deviation.
4 x 10000/kuribas 2.0
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.223716510989717e-5 | 9.893215449400704e-3 | 6.0332987179063935e-5 |
| Standard deviation | 4.600985251646268e-5 | 1.221957740239556e-4 | 8.570378368209458e-5 |
Outlying measurements have slight (3.222222222222222e-2%) effect on estimated standard deviation.
20 x 10000/penner
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.264480818500123e-5 | 1.0349741716482409e-2 | 4.728749487574631e-5 |
| Standard deviation | 2.566646320386393e-5 | 1.2147160940400713e-4 | 4.270599951623904e-5 |
Outlying measurements have slight (3.222222222222222e-2%) effect on estimated standard deviation.
20 x 10000/joshcc4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.3962391611172945e-5 | 1.783952842492059e-2 | 1.6797848157702744e-5 |
| Standard deviation | 1.2675536423941395e-5 | 4.856601849380017e-5 | 2.683938923629385e-5 |
Outlying measurements have slight (4.158790170132318e-2%) effect on estimated standard deviation.
20 x 10000/kuribas
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.6307880045814474e-4 | 1.3041679211185668e-2 | 9.720097860722691e-5 |
| Standard deviation | 1.0728316338939076e-4 | 3.056566911802538e-4 | 1.1608324641337338e-4 |
Outlying measurements have slight (3.6982248520709936e-2%) effect on estimated standard deviation.
20 x 10000/kuribas 2.0
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.576449735782325e-5 | 1.3152594920643764e-2 | 1.628738220971447e-4 |
| Standard deviation | 1.0384799140581943e-4 | 3.092502974956146e-4 | 1.4741255700412759e-4 |
Outlying measurements have slight (3.6982248520710026e-2%) effect on estimated standard deviation.
200 x 10000/penner
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.0928912774969763e-4 | 2.3704536768392303e-2 | 2.386280581559569e-4 |
| Standard deviation | 1.1541435968142613e-4 | 5.137625090759203e-4 | 1.7413908191021084e-4 |
Outlying measurements have slight (4.75e-2%) effect on estimated standard deviation.
200 x 10000/joshcc4
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 7.539665119378203e-4 | 9.815158424193325e-2 | 1.0674786200248199e-3 |
| Standard deviation | 4.789664552516319e-4 | 1.365167808208838e-3 | 5.872948616096065e-4 |
Outlying measurements have slight (9.876543209876541e-2%) effect on estimated standard deviation.
200 x 10000/kuribas
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.4794920732135244e-4 | 1.681762412808774e-2 | 1.4331871009753278e-4 |
| Standard deviation | 6.506223008510007e-5 | 3.46754367076809e-4 | 9.028541186269294e-5 |
Outlying measurements have slight (4.158790170132311e-2%) effect on estimated standard deviation.
200 x 10000/kuribas 2.0
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.706902535331921e-4 | 1.6286790048535792e-2 | 2.4160042284246744e-4 |
| Standard deviation | 1.6018481206177943e-4 | 4.949427382891409e-4 | 3.201258731722022e-4 |
Outlying measurements have slight (7.843771700467297e-2%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.