ROC AUC calculator with a confidence interval
Paste your scores and labels and you get four things back. The AUC, a bootstrap interval around it, a count of the tied scores that fed into it, and the precision recall AUC on the same rows. The AUC comes from the rank based Mann Whitney U statistic rather than from measuring a plotted curve, and every result says which method produced it.
A point estimate on its own cannot tell you whether the gap between two models is real. So the interval, the tie count and the paired comparison are why this page exists. Everything runs in your browser.
The calculator
Rows are score label, one case per line, split on a space or a comma. Labels can be 1 and 0, or pos and neg. Both boxes start with an illustrative sample so you can see the output before you paste. Replace it with your own data.
One score per line, the same cases in the same order as model A, which supplies the labels. Leave it empty to skip the comparison.
Cross check against the trapezoid
| Method | AUC |
|---|---|
| Rank based U statistic | 0.8194 |
| Trapezoid over the ROC points (trapezoidal rule) | 0.8194 |
| Distance between the two | 0.0000 |
Ties in your scores
| Quantity | Value |
|---|---|
| Score values shared by more than one row | 1 |
| Rows sitting on a shared score | 2 |
| Positive and negative pairs at the same score | 1 |
| Share of the AUC coming from half credit ties | 0.0139 |
| AUC if every tied pair fell your way | 0.8333 |
| AUC if every tied pair fell against you | 0.8056 |
Tied values take the average rank, the default in scipy.stats.rankdata, so a tied positive and negative pair scores half a point. The last two rows are the bounds those half points sit between, derived from the tie count.
Precision recall on the same rows
| Quantity | Value |
|---|---|
| Average precision, step sum (scikit-learn definition) | 0.8218 |
| Precision recall baseline, positives over all rows (Saito and Rehmsmeier) | 0.5000 |
Same rows, rarer positives
| Negatives multiplied by | Negative rows | ROC AUC | Average precision | PR baseline |
|---|---|---|---|---|
| 1 | 6 | 0.8194 | 0.8218 | 0.5000 |
| 2 | 12 | 0.8194 | 0.7204 | 0.3333 |
| 5 | 30 | 0.8194 | 0.5731 | 0.1667 |
| 10 | 60 | 0.8194 | 0.4805 | 0.0909 |
Each row copies every negative case again, which simulates a rarer positive class on your own data (Saito and Rehmsmeier). Simulation, not new measurement.
Model A against model B
| Quantity | Value |
|---|---|
| AUC model A | 0.8194 |
| AUC model B | 0.6944 |
| Difference, A minus B | 0.1250 |
| 95 percent interval on the difference | -0.0139 to 0.3333 |
The interval on the difference covers zero, so this sample does not separate the two models. Both models are scored on the same resampled cases in every draw, which keeps the pairing (DeLong and colleagues on correlated ROC curves).
How the number is computed
Ranking is the whole job. Google's crash course defines the area under the ROC curve as the probability that a model, given a randomly chosen positive and a randomly chosen negative, ranks the positive one higher (Google, ROC and AUC). So you can reach it by counting pairs instead of measuring a shape.
Sort the scores, hand out ranks, add up the ranks belonging to the positive rows and call that sum R1. The U statistic then strips out the positives' own internal ordering. SciPy writes the relation as R1 equals U1 plus n1 times (n1 plus 1) divided by two (SciPy, mannwhitneyu), so U1 is R1 minus that term. The rank sum is not the U statistic, and the subtraction is what separates them.
ranks r_i tied scores share the average rank R1 sum of the ranks of the positive rows U1 = R1 - n1 * (n1 + 1) / 2 AUC = U1 / (n1 * n0)
Two anchors help you read whatever comes back. Google's crash course puts a model that does as well as random guessing at one half, describing that as a fifty percent probability of ranking a random pair correctly (Google), and puts a perfect classifier at an area under the curve of one (Google).
What the interval tells you
One AUC is a point estimate from one sample, and resampling shows its spread. The function scipy.stats.bootstrap computes a two-sided bootstrap confidence interval of a statistic (SciPy, bootstrap) and supports the percentile, basic and bias-corrected and accelerated methods (SciPy). The interval above is the percentile one, drawn separately from your positive rows and your negative rows so the class counts stay put.
SciPy describes n_resamples as the number of resamples performed to form the bootstrap distribution of the statistic (SciPy), and the resamples box is that same knob. Turn it up for a steadier interval, down for speed. SciPy cites Efron and Tibshirani, An Introduction to the Bootstrap, Chapman and Hall/CRC, 1993, as its reference for bootstrap intervals (SciPy). Treat the result as an estimate. Resampling reuses the rows you already have.
Ties, and why they move the number
Scores landing on the same value have to be broken somehow, and the choice changes the answer. The default in scipy.stats.rankdata assigns each tied value the average of the ranks that would have been assigned to all the tied values (SciPy, rankdata), which is what runs here, so a tied positive and negative pair contributes half a point. scipy.stats.rankdata also offers min, max, dense and ordinal (SciPy), and swapping the rule moves the U statistic. The tie panel reports how many pairs are involved and the two bounds those half points sit between.
Ties reach the p value as well. In scipy.stats.mannwhitneyu the exact method computes the exact p-value and no correction is made for ties (SciPy). When ties are present and either sample is small, roughly fewer than ten observations, SciPy suggests you consider passing a permutation method (SciPy).
Comparing two models on the same cases
Two models scored on the same rows are not two independent samples. DeLong, DeLong and Clarke-Pearson made that point, that when two or more empirical ROC curves are constructed from tests performed on the same individuals, statistical analysis of the differences must take into account the correlated nature of the data (PubMed record). Their approach uses the theory of generalized U-statistics to generate an estimated covariance matrix (PubMed record), published in Biometrics in September 1988 (PubMed record).
The comparison panel keeps that pairing in a simpler form. Each bootstrap draw picks one set of cases and scores both models on it, so a difference is never taken across two unrelated draws, and the reported range is the percentile interval of those differences. Read the interval, not the sign of the gap. When it stays clear of zero you have shown a difference on this data at the level you chose. When it covers zero, you have not.
A gap in AUC can also hide a difference in shape. Saito and Rehmsmeier note that ROC AUC can be inaccurate for fair comparisons when two ROC curves cross each other (PLOS ONE), so a narrow gap is worth plotting before you act on it. The companion ROC and PR curve threshold calculator draws both curves and picks a cut point.
Why AUC ignores class balance and precision recall does not
Change the mix of positives and negatives and the AUC stays where it was. Saito and Rehmsmeier report that ROC plots are unchanged between balanced and imbalanced datasets (PLOS ONE), while precision recall plots change between balanced and imbalanced datasets (PLOS ONE). The rarer positives table shows that on your own rows. Each row copies every negative case again, and the ROC AUC column holds still while average precision slides. Saito and Rehmsmeier give the baseline of a precision recall curve as the ratio of positives, P divided by P plus N (PLOS ONE), so the same average precision means something different at a different prevalence.
Average precision here follows the scikit-learn definition, the sum over thresholds of (R_n minus R_n-1) times P_n, where P_n and R_n are the precision and recall at the nth threshold (scikit-learn). scikit-learn warns that linear interpolation of points on the precision recall curve gives an overly optimistic measure of classifier performance, and that this linear interpolation is what is applied when computing area with the trapezoidal rule (scikit-learn). So the step sum drives the precision recall number and trapezoids are kept for the ROC cross check only.
Precision is the proportion of all of a model's positive classifications that are actually positive, TP over TP plus FP (Google), and recall, also called the true positive rate, is TP over all actual positives, TP over TP plus FN (Google). Read both when your classes are lopsided. On a heavily imbalanced dataset where one class appears about one percent of the time, a model that predicts negative every time would score ninety nine percent on accuracy despite being useless (Google).
Check it against SciPy and scikit-learn
The tool computes the area a second way as a check. scikit-learn's sklearn.metrics.auc computes the area under a curve using the trapezoidal rule (scikit-learn), and in scikit-learn's roc_curve output element i of the false positive rate array is the false positive rate of predictions with score greater than or equal to thresholds[i] (scikit-learn). Sweeping your thresholds from high to low builds that sequence. Both routes print side by side above, so you can see where they land on your rows rather than trusting that the tie handling left them alone.
You can also push the same rows through the libraries. scikit-learn's roc_auc_score computes the area under the receiver operating characteristic curve from prediction scores (scikit-learn).
import numpy as np
from scipy.stats import mannwhitneyu, bootstrap
from sklearn.metrics import roc_auc_score
y = np.array([1, 1, 0, 1, 0, 0]) # your labels, 1 is positive
s = np.array([.92, .88, .81, .78, .55, .22]) # your scores
pos, neg = s[y == 1], s[y == 0]
def auc(p, n):
return mannwhitneyu(p, n, alternative='two-sided').statistic / (p.size * n.size)
ci = bootstrap((pos, neg), auc, vectorized=False, paired=False,
n_resamples=2000, method='percentile',
confidence_level=0.95).confidence_interval
print(auc(pos, neg), roc_auc_score(y, s), ci.low, ci.high)What this page does not do
- Binary labels only. scikit-learn notes that multiclass ROC AUC currently handles only the macro and weighted averages (scikit-learn), and neither is computed here.
- No partial AUC. With max_fpr set, scikit-learn's roc_auc_score returns the standardized partial AUC over the range from zero to max_fpr (scikit-learn).
- No curve plot and no threshold picking. The companion ROC and PR curve threshold calculator handles both.
- No opinion on which metric you should report. Google's crash course states that AUC and ROC work well for comparing models when the dataset is roughly balanced between classes (Google), which is a reason to read the precision recall number beside the AUC when yours is not.
Where these numbers come from
Every figure on this page is computed in your browser from the rows you paste. No dataset is uploaded, nothing is stored, and the page keeps working with the network off. The AUC uses the rank based U statistic in the form SciPy documents (SciPy), ties take average ranks following the default in scipy.stats.rankdata (SciPy), and the interval is a percentile bootstrap, the interval type scipy.stats.bootstrap documents (SciPy). Resampling uses a seed you can see and change, so the same rows with the same seed return the same interval every time.
Formulas are printed next to the source that states them. The sample in the boxes is illustrative input rather than measured data, and the rarer positives table is a simulation built by copying your own negative rows.
Before you rely on a number
- Estimates use standard published formulas. Real results vary with your data, your settings, and your runtime.
- This tool is for planning and teaching. Check a result against your own measurement before you rely on it.
- All computation runs client side. No data leaves your browser.
Sources
- sklearn.metrics.roc_auc_score, scikit-learn developers
- Metrics and scoring, scikit-learn developers
- sklearn.metrics.roc_curve, scikit-learn developers
- sklearn.metrics.auc, scikit-learn developers
- scipy.stats.mannwhitneyu, SciPy developers
- scipy.stats.rankdata, SciPy developers
- scipy.stats.bootstrap, SciPy developers
- ROC and AUC, Google Machine Learning Crash Course
- Accuracy, precision and recall, Google Machine Learning Crash Course
- Saito and Rehmsmeier, PLOS ONE
- DeLong, DeLong and Clarke-Pearson, PubMed record, US National Library of Medicine
About the author
Written by Michael Lip. Michael Lip builds open-source ML tools and developer utilities at zovo.one. ml0x is part of the Zovo Tools network, a collection of free, privacy-first tools for developers and data scientists. zovo.one
ml0x publishes free machine learning calculators and explainers. Every number on this page is computed in your browser from the inputs you enter. Nothing is sent to a server. Written by Michael Lip, part of the Zovo Tools network.