Lesson 4 · ML algorithms deep dive
Decision Trees: How They Really Work | ML Algorithms #4
An algorithm that explains its own real reasoning. Real mushroom characteristics, labeled genuinely edible or genuinely poisonous. Let's get into it. Open a…
- CourseML algorithms deep dive
- Lesson4 of 12
- Video17 min
- FormatJupyter notebook · 12 code cells
- Data1 dataset
What you'll learn
Datasets used in this lesson
Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.
- mushroom_dataset.csv preview78.3 KB
📓 Full notebook
Download .ipynbML Algorithms Deep-Dive, Video 4: Decision Trees#
- Video four of the 12-part series: an algorithm that explains its own real reasoning.
- Real mushroom characteristics, labeled genuinely edible or genuinely poisonous.
- Let's get into it.
Before You Start#
- Open a new Jupyter Notebook in VS Code and select your Python interpreter as the kernel.
- You'll need pandas, NumPy, Matplotlib, and scikit-learn.
- Place mushroom_dataset.csv in the same folder as this notebook.
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from sklearn.tree import DecisionTreeClassifier, plot_tree, export_text
from sklearn.model_selection import train_test_split, cross_val_score, StratifiedKFold
mushrooms = pd.read_csv('mushroom_dataset.csv')
feature_cols = [c for c in mushrooms.columns if c not in ('actual', 'predicted')]
print(f'Real mushrooms in this dataset: {len(mushrooms)}')
print(f'Real features per mushroom: {len(feature_cols)}')
print(mushrooms['actual'].value_counts())
Part 1: The Core Idea#
X = mushrooms[feature_cols].values
y = mushrooms['actual'].values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42, stratify=y)
print(f'Real training mushrooms: {len(X_train)}')
print(f'Real test mushrooms: {len(X_test)}')
Part 2: How a Split Gets Chosen#
def gini_impurity(labels):
_, counts = np.unique(labels, return_counts=True)
proportions = counts / counts.sum()
return 1 - np.sum(proportions ** 2)
print(f'Real Gini impurity of the full real training set: {gini_impurity(y_train):.4f}')
edible_only = y_train[y_train == 'e']
print(f'Real Gini impurity of an all-edible real group: {gini_impurity(edible_only):.4f}')
Part 3: Fitting an Unconstrained Real Tree#
full_tree = DecisionTreeClassifier(random_state=42).fit(X_train, y_train)
print(f'Real training accuracy: {full_tree.score(X_train, y_train):.4f}')
print(f'Real test accuracy: {full_tree.score(X_test, y_test):.4f}')
print(f'Real tree depth: {full_tree.get_depth()}')
print(f'Real number of leaves: {full_tree.get_n_leaves()}')
A real, honest surprise: both scores land at a perfect 1.0, and the unconstrained tree stopped naturally at just 8 real levels and 20 real leaves, not the sprawling, memorized structure a name like 'unconstrained' might suggest. This real dataset genuinely is that cleanly separable by a handful of physical mushroom traits; that's a property of the data, not something to expect from every real dataset.
Part 4: Controlling Complexity with Max Depth#
for depth in [1, 2, 3, 4, 5, None]:
limited_tree = DecisionTreeClassifier(max_depth=depth, random_state=42).fit(X_train, y_train)
train_acc = limited_tree.score(X_train, y_train)
test_acc = limited_tree.score(X_test, y_test)
print(f'max_depth={depth}: real train acc={train_acc:.4f}, real test acc={test_acc:.4f}')
Part 5: When Overfitting Actually Shows Up#
X_small, _, y_small, _ = train_test_split(X_train, y_train, train_size=60, random_state=42, stratify=y_train)
for depth in [1, 2, 3, None]:
small_tree = DecisionTreeClassifier(max_depth=depth, random_state=42).fit(X_small, y_small)
train_acc = small_tree.score(X_small, y_small)
held_out_acc = small_tree.score(X_test, y_test)
print(f'60-sample max_depth={depth}: real train acc={train_acc:.4f}, real held-out acc={held_out_acc:.4f}')
Even here, with a real training set shrunk to 60 mushrooms, deeper trees didn't overfit further, because the real underlying pattern is genuinely simple: a few real physical traits do almost all the work. A real dataset with messier, more overlapping classes would show a real accuracy gap opening up between train and held-out scores as depth grows; that gap is the real signal to watch for and prune against, whether or not it appears here.
Part 6: Reading the Tree#
shallow_tree = DecisionTreeClassifier(max_depth=3, random_state=42).fit(X_train, y_train)
print(export_text(shallow_tree, feature_names=feature_cols))
plt.figure(figsize=(16, 8))
plot_tree(shallow_tree, feature_names=feature_cols, class_names=shallow_tree.classes_, filled=True, fontsize=8)
plt.title('Real Depth-3 Decision Tree')
plt.show()
Part 7: Feature Importance#
depth5_tree = DecisionTreeClassifier(max_depth=5, random_state=42).fit(X_train, y_train)
importances = pd.Series(depth5_tree.feature_importances_, index=feature_cols).sort_values(ascending=False)
print(importances.head(6))
Part 8: Cross-Validating the Depth Choice#
skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
for depth in [1, 2, 3, 4, 5, 8, None]:
scores = cross_val_score(DecisionTreeClassifier(max_depth=depth, random_state=42), X_train, y_train, cv=skf, scoring='accuracy')
print(f'max_depth={depth}: real mean CV accuracy={scores.mean():.4f}')
Unlike video one's regularization sweep, where more complexity eventually hurt, this real search never found a real downside to growing further. That's not a contradiction, it's a reminder that the honest answer, 'how much complexity is too much,' genuinely depends on the real dataset in front of you, and has to be checked, not assumed either way.
Part 9: Strengths, Weaknesses, and When to Use It#
- Strength: fully real, human-readable decision logic, no black box.
- Strength: handles real categorical features natively, no scaling or encoding tricks required.
- Weakness: a single tree can be unstable, small real changes in training data can produce a different real tree structure.
- Weakness: prone to real overfitting on messier, noisier, or smaller real datasets than this one.
- Use it when explaining the real 'why' behind a prediction matters as much as the prediction itself.
Part 10: Saving Your Work#
plt.figure(figsize=(18, 9))
plot_tree(depth5_tree, feature_names=feature_cols, class_names=depth5_tree.classes_, filled=True, fontsize=7)
plt.title('Real Depth-5 Decision Tree (Final Model)')
plt.savefig('decision_tree_final.png', dpi=150)
plt.show()
import joblib
joblib.dump(depth5_tree, 'decision_tree_model.joblib')
reloaded_tree = joblib.load('decision_tree_model.joblib')
print(f'Real original model test accuracy: {depth5_tree.score(X_test, y_test):.4f}')
print(f'Real reloaded model test accuracy: {reloaded_tree.score(X_test, y_test):.4f}')
Wrap-Up: What You Learned#
- Gini impurity, computed from scratch, and how it decides every real split.
- An unconstrained real tree that turned out not to overfit, a genuine property of this particular real data.
- A real overfitting gap that finally appeared once the training set was deliberately shrunk.
- Reading a real tree's actual decision logic directly, in text and as a diagram.
- Ranking real features by importance, and cross-validating the depth choice honestly.
- Video five moves to random forests, which fix a single tree's real instability by growing many of them. Subscribe so it lands automatically see you there.
🧠 Test yourself
Pick an answer to check it. Your best score is saved in this browser.
What does Gini impurity measure?
Gini = 1 - sum(p_i^2); a group of one class has impurity 0.
What is the Gini impurity of a group with 50% 'e' and 50% 'p'?
1 - (0.5^2 + 0.5^2) = 0.5.
The unconstrained tree scored 1.0 on both train and test in the lesson. What was the explanation?
The tree stopped at depth 8 with only 20 leaves; the data is simply easy to separate.
Which tool prints a tree's rules as readable text?
export_text shows the if/else logic of each split.
Which is a known weakness of a single decision tree?
Single trees have high variance, which motivates random forests.
💻 Practice exercises
Try each one in the notebook or in your own editor before opening the solution.
Write
gini(labels)for a list of labels and test it on ['a','a','b','b'] and ['a','a','a'].💡 Hint
Gini = 1 - sum of squared proportions.
✅ Show solution
import numpy as np def gini(labels): _, counts = np.unique(labels, return_counts=True) p = counts / counts.sum() return 1 - np.sum(p ** 2) print(gini(['a','a','b','b']), gini(['a','a','a']))
Fit
DecisionTreeClassifier(max_depth=2)on Iris and print the text rules withexport_text.💡 Hint
Pass feature_names=load_iris().feature_names.
✅ Show solution
from sklearn.datasets import load_iris from sklearn.tree import DecisionTreeClassifier, export_text iris = load_iris() tree = DecisionTreeClassifier(max_depth=2, random_state=42).fit(iris.data, iris.target) print(export_text(tree, feature_names=list(iris.feature_names)))
On the breast cancer data (75/25 split, seed 42), print train and test accuracy for max_depth in [1,2,3,5,None] and then the top 3 feature importances for max_depth=3.
💡 Hint
Build a pd.Series of feature_importances_ indexed by feature names.
✅ Show solution
import pandas as pd from sklearn.datasets import load_breast_cancer from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier d = load_breast_cancer() Xtr, Xte, ytr, yte = train_test_split(d.data, d.target, test_size=0.25, random_state=42, stratify=d.target) for depth in [1, 2, 3, 5, None]: t = DecisionTreeClassifier(max_depth=depth, random_state=42).fit(Xtr, ytr) print(depth, round(t.score(Xtr, ytr), 3), round(t.score(Xte, yte), 3)) t3 = DecisionTreeClassifier(max_depth=3, random_state=42).fit(Xtr, ytr) imp = pd.Series(t3.feature_importances_, index=d.feature_names).sort_values(ascending=False) print(imp.head(3))
Found this useful?
All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.



