Lesson 10 · Scikit-learn deep dive
scikit-learn Classification Metrics: Precision, Recall, F1, ROC AUC
accuracy, confusion_matrix, precision, recall, F1, classification_report, ROC AUC and precision-recall curves in scikit-learn, plus a reusable evaluator.
- CourseScikit-learn deep dive
- Lesson10 of 18
- Video15 min
- FormatJupyter notebook · 10 code cells
What you'll learn
Data
No separate download needed — the notebook creates or downloads everything it uses.
📓 Full notebook
Download .ipynbScikit-learn Deep-Dive, Video 10: Classification Metrics#
- Video ten of the eighteen-part series: judging a classifier honestly, beyond plain accuracy.
- Precision, recall, F1, confusion_matrix, classification_report, and ROC-AUC.
- Let's get into it.
Part 1: accuracy_score - Why It Can Mislead on Imbalanced Data#
import numpy as np
from sklearn.metrics import accuracy_score
y_true_imbalanced = np.array([0]*95 + [1]*5)
y_pred_always_zero = np.zeros(100, dtype=int)
print(accuracy_score(y_true_imbalanced, y_pred_always_zero))
Part 2: confusion_matrix - TP, FP, TN, FN#
from sklearn.metrics import confusion_matrix
cm = confusion_matrix(y_true_imbalanced, y_pred_always_zero)
print(cm)
tn, fp, fn, tp = cm.ravel()
print('TN:', tn, 'FP:', fp, 'FN:', fn, 'TP:', tp)
Part 3: precision_score and recall_score - the Tradeoff#
from sklearn.metrics import precision_score, recall_score
y_true_demo = np.array([1, 0, 1, 1, 0, 1, 0, 0])
y_pred_demo = np.array([1, 0, 0, 1, 1, 1, 0, 0])
print(round(precision_score(y_true_demo, y_pred_demo), 3))
print(round(recall_score(y_true_demo, y_pred_demo), 3))
Part 4: f1_score - Balancing Precision and Recall#
from sklearn.metrics import f1_score
print(round(f1_score(y_true_demo, y_pred_demo), 3))
high_precision_low_recall = np.array([0, 0, 0, 1, 0, 0, 0, 0])
print(round(precision_score(y_true_demo, high_precision_low_recall), 3))
print(round(recall_score(y_true_demo, high_precision_low_recall), 3))
print(round(f1_score(y_true_demo, high_precision_low_recall), 3))
Part 5: classification_report - All Metrics at Once#
from sklearn.metrics import classification_report
print(classification_report(y_true_demo, y_pred_demo, target_names=['negative', 'positive']))
Part 6: predict_proba and decision_function#
from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
Xc, yc = load_breast_cancer(return_X_y=True)
Xc_train, Xc_test, yc_train, yc_test = train_test_split(Xc, yc, test_size=0.25, random_state=42, stratify=yc)
clf = LogisticRegression(max_iter=5000).fit(Xc_train, yc_train)
probabilities = clf.predict_proba(Xc_test)
print(probabilities[:3].round(3))
Part 7: roc_curve and roc_auc_score#
from sklearn.metrics import roc_curve, roc_auc_score
positive_probs = probabilities[:, 1]
fpr, tpr, thresholds = roc_curve(yc_test, positive_probs)
auc = roc_auc_score(yc_test, positive_probs)
print(len(thresholds))
print(round(auc, 3))
Part 8: precision_recall_curve#
from sklearn.metrics import precision_recall_curve, average_precision_score
precisions, recalls, pr_thresholds = precision_recall_curve(yc_test, positive_probs)
avg_precision = average_precision_score(yc_test, positive_probs)
print(len(pr_thresholds))
print(round(avg_precision, 3))
Part 9: Multiclass Metrics - the average Parameter#
from sklearn.datasets import load_wine
Xw, yw = load_wine(return_X_y=True)
Xw_train, Xw_test, yw_train, yw_test = train_test_split(Xw, yw, test_size=0.25, random_state=42, stratify=yw)
wine_clf = LogisticRegression(max_iter=5000).fit(Xw_train, yw_train)
wine_preds = wine_clf.predict(Xw_test)
print(round(f1_score(yw_test, wine_preds, average='macro'), 3))
print(round(f1_score(yw_test, wine_preds, average='weighted'), 3))
print(round(f1_score(yw_test, wine_preds, average='micro'), 3))
Part 10: A Real Pattern - a Reusable evaluate_classifier Function#
def evaluate_classifier(y_true, y_pred):
return {
'accuracy': round(accuracy_score(y_true, y_pred), 3),
'precision_macro': round(precision_score(y_true, y_pred, average='macro'), 3),
'recall_macro': round(recall_score(y_true, y_pred, average='macro'), 3),
'f1_macro': round(f1_score(y_true, y_pred, average='macro'), 3)
}
print(evaluate_classifier(yw_test, wine_preds))
Wrap-Up: What You Learned#
- accuracy_score can mislead badly on imbalanced data; a model predicting only the majority class can still score high.
- confusion_matrix breaks predictions into true positives, false positives, true negatives, and false negatives.
- precision measures how trustworthy a positive prediction is; recall measures how many actual positives were caught.
- f1_score is the harmonic mean of precision and recall, penalizing lopsided performance.
- classification_report prints precision, recall, F1, and support for every class at once.
- predict_proba and decision_function give scores instead of hard labels, needed to build ROC and PR curves.
- roc_curve and roc_auc_score trace and summarize true-positive rate versus false-positive rate across thresholds.
- precision_recall_curve is often more informative than ROC on heavily imbalanced data.
- macro, weighted, and micro averaging control how multiclass precision, recall, and F1 combine across classes.
- That wraps up classification metrics. Next up: Regression Metrics - MSE, RMSE, MAE, and R-squared.
🧠 Test yourself
Pick an answer to check it. Your best score is saved in this browser.
The lesson builds this example:
y_true_imbalanced = np.array([0]*95 + [1]*5) y_pred_always_zero = np.zeros(100, dtype=int) print(accuracy_score(y_true_imbalanced, y_pred_always_zero))
What is printed, and what does it teach?
95 of 100 predictions are right even though every positive is missed. That is why accuracy can mislead on imbalanced data.
Precision and recall differ in what way?
Precision is TP / (TP + FP). Recall is TP / (TP + FN).
Using y_true_demo = [1,0,1,1,0,1,0,0] and y_pred_demo = [1,0,0,1,1,1,0,0], what are precision_score and recall_score?
There are 3 true positives, 1 false positive and 1 false negative. That gives 3/4 for both precision and recall.
For
high_precision_low_recall = [0,0,0,1,0,0,0,0]against the same y_true_demo, why is f1_score about 0.4, much lower than its precision of 1.0?The model is right about its one positive call but finds just 1 of 4 positives. The harmonic mean of 1.0 and 0.25 is 0.4, which punishes lopsided results.
In
f1_score(yw_test, wine_preds, average='macro')for the three-class wine data, what does average='macro' mean?Macro treats every class equally. 'weighted' weights by class support, and 'micro' pools all counts, so it equals accuracy for single-label problems.
💻 Practice exercises
Try each one in the notebook or in your own editor before opening the solution.
Recreate the always-predict-zero model on 95 negatives and 5 positives. Print its accuracy, its confusion matrix, and its recall for the positive class.
💡 Hint
Use zero_division=0 in precision_score to avoid warnings.
✅ Show solution
import matplotlib matplotlib.use('Agg') import numpy as np from sklearn.metrics import accuracy_score, confusion_matrix, recall_score, precision_score y_true = np.array([0] * 95 + [1] * 5) y_pred = np.zeros(100, dtype=int) print('accuracy:', accuracy_score(y_true, y_pred)) cm = confusion_matrix(y_true, y_pred) print(cm) tn, fp, fn, tp = cm.ravel() print('TN', tn, 'FP', fp, 'FN', fn, 'TP', tp) print('recall:', recall_score(y_true, y_pred)) print('precision:', precision_score(y_true, y_pred, zero_division=0))
Compute precision, recall and F1 by hand from the confusion matrix counts for y_true_demo and y_pred_demo, then check they match scikit-learn's functions.
💡 Hint
Unpack tn, fp, fn, tp with confusion_matrix(...).ravel().
✅ Show solution
import matplotlib matplotlib.use('Agg') import numpy as np from sklearn.metrics import confusion_matrix, precision_score, recall_score, f1_score y_true = np.array([1, 0, 1, 1, 0, 1, 0, 0]) y_pred = np.array([1, 0, 0, 1, 1, 1, 0, 0]) tn, fp, fn, tp = confusion_matrix(y_true, y_pred).ravel() precision = tp / (tp + fp) recall = tp / (tp + fn) f1 = 2 * precision * recall / (precision + recall) print('by hand:', round(precision, 3), round(recall, 3), round(f1, 3)) print('sklearn:', round(precision_score(y_true, y_pred), 3), round(recall_score(y_true, y_pred), 3), round(f1_score(y_true, y_pred), 3)) assert np.isclose(precision, precision_score(y_true, y_pred)) assert np.isclose(recall, recall_score(y_true, y_pred)) assert np.isclose(f1, f1_score(y_true, y_pred))
Train a LogisticRegression on the breast cancer data, then print ROC-AUC and average precision from predict_proba. Also find the threshold that gives recall of at least 0.95 with the highest precision, using precision_recall_curve.
💡 Hint
Use positive_probs = clf.predict_proba(X_test)[:, 1] and then filter where recalls[:-1] >= 0.95.
✅ Show solution
import matplotlib matplotlib.use('Agg') import numpy as np from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.metrics import roc_auc_score, average_precision_score, precision_recall_curve X, y = load_breast_cancer(return_X_y=True) X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.25, random_state=42, stratify=y) clf = LogisticRegression(max_iter=5000).fit(X_train, y_train) probs = clf.predict_proba(X_test)[:, 1] print('ROC-AUC:', round(roc_auc_score(y_test, probs), 3)) print('average precision:', round(average_precision_score(y_test, probs), 3)) precisions, recalls, thresholds = precision_recall_curve(y_test, probs) ok = np.where(recalls[:-1] >= 0.95)[0] best = ok[np.argmax(precisions[:-1][ok])] print('threshold:', round(float(thresholds[best]), 3), 'precision:', round(float(precisions[best]), 3), 'recall:', round(float(recalls[best]), 3))
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



