Lesson 1 · ML algorithms deep dive

Linear Regression from Scratch in Python | ML Algorithms #1

The first video of a 12-part series: one real algorithm per video, covered in full depth. Real classic car data: fuel economy, weight, horsepower, and…

⬇ Download notebook ⬇ Notebook + data (.zip)Open in Colab ↗

CourseML algorithms deep dive
Lesson1 of 12
Video25 min
FormatJupyter notebook · 17 code cells
Data1 dataset

scikit-learn statsmodels pandas NumPy Matplotlib SciPy

What you'll learn

The Problem Linear Regression Solves
The Math Behind the Line
Multiple Regression
Multicollinearity
Checking the Assumptions
Why a Single Train/Test Split Isn't Enough Here
Regularization: Ridge and Lasso
Evaluation Metrics, Recapped

Datasets used in this lesson

Save these next to the notebook. In Google Colab, upload them with the 📁 icon on the left first.

mtcars.csv1.7 KB

📓 Full notebook

Download .ipynb

ML Algorithms Deep-Dive, Video 1: Linear Regression#

The first video of a 12-part series: one real algorithm per video, covered in full depth.
Real classic car data: fuel economy, weight, horsepower, and displacement.
Let's get into it.

Before You Start#

Open a new Jupyter Notebook in VS Code and select your Python interpreter as the kernel.
You'll need pandas, NumPy, Matplotlib, SciPy, scikit-learn, and statsmodels.
Place mtcars.csv in the same folder as this notebook.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy import stats
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import KFold, cross_val_score
from statsmodels.stats.outliers_influence import variance_inflation_factor
from statsmodels.stats.stattools import durbin_watson
import statsmodels.api as sm

cars = pd.read_csv('mtcars.csv')
print(f'Real cars in this dataset: {len(cars)}')
print(cars[['model', 'mpg', 'wt', 'hp', 'disp']].head())

Real cars in this dataset: 32
               model   mpg     wt   hp   disp
0          Mazda RX4  21.0  2.620  110  160.0
1      Mazda RX4 Wag  21.0  2.875  110  160.0
2         Datsun 710  22.8  2.320   93  108.0
3     Hornet 4 Drive  21.4  3.215  110  258.0
4  Hornet Sportabout  18.7  3.440  175  360.0

Part 1: The Problem Linear Regression Solves#

plt.figure(figsize=(8, 5))
plt.scatter(cars['wt'], cars['mpg'], color='steelblue')
plt.title('Real Fuel Economy vs. Real Car Weight')
plt.xlabel('Weight (1000 lbs)')
plt.ylabel('Miles per Gallon')
plt.show()

No description has been provided for this image

print(f"Real correlation between weight and mpg: {cars['wt'].corr(cars['mpg']):.4f}")

Real correlation between weight and mpg: -0.8677

Part 2: The Math Behind the Line#

X_simple = cars[['wt']].values
y = cars['mpg'].values
simple_model = LinearRegression().fit(X_simple, y)
print(f'Real intercept: {simple_model.intercept_:.4f}')
print(f'Real slope: {simple_model.coef_[0]:.4f}')
print(f'Real R-squared: {simple_model.score(X_simple, y):.4f}')

Real intercept: 37.2851
Real slope: -5.3445
Real R-squared: 0.7528

plt.figure(figsize=(8, 5))
plt.scatter(cars['wt'], cars['mpg'], color='steelblue', label='Real cars')
x_line = np.linspace(cars['wt'].min(), cars['wt'].max(), 50)
y_line = simple_model.predict(x_line.reshape(-1, 1))
plt.plot(x_line, y_line, color='darkred', label='Fitted line')
plt.title('Real Fuel Economy vs. Weight, with the Fitted Regression Line')
plt.xlabel('Weight (1000 lbs)')
plt.ylabel('Miles per Gallon')
plt.legend()
plt.show()

Part 3: Multiple Regression#

X_multi = cars[['wt', 'hp', 'disp']].values
multi_model = LinearRegression().fit(X_multi, y)
print(f'Real intercept: {multi_model.intercept_:.4f}')
for name, coef in zip(['wt', 'hp', 'disp'], multi_model.coef_):
    print(f'Real coefficient for {name}: {coef:.4f}')
print(f'Real R-squared: {multi_model.score(X_multi, y):.4f}')

Real intercept: 37.1055
Real coefficient for wt: -3.8009
Real coefficient for hp: -0.0312
Real coefficient for disp: -0.0009
Real R-squared: 0.8268

Holding horsepower and displacement fixed, each additional real 1000 pounds still costs roughly 3.8 real miles per gallon, the dominant real effect. Horsepower's own real coefficient is small and displacement's is nearly zero, even though displacement alone correlated with mpg at negative 0.85. That mismatch is worth investigating before trusting these real numbers.

Part 4: Multicollinearity#

print(cars[['wt', 'hp', 'disp']].corr())

            wt        hp      disp
wt    1.000000  0.658748  0.887980
hp    0.658748  1.000000  0.790949
disp  0.887980  0.790949  1.000000

X_with_const = sm.add_constant(cars[['wt', 'hp', 'disp']])
vif_values = [variance_inflation_factor(X_with_const.values, i) for i in range(X_with_const.shape[1])]
for name, vif in zip(X_with_const.columns, vif_values):
    print(f'Real VIF for {name}: {vif:.2f}')

Real VIF for const: 20.47
Real VIF for wt: 4.84
Real VIF for hp: 2.74
Real VIF for disp: 7.32

This is the real explanation: displacement doesn't lack real predictive power on its own, its real information is largely redundant with weight once weight is already in the model, so the fitting process can't cleanly separate their individual real contributions. That's exactly why part three's coefficients shouldn't be read as 'displacement doesn't matter.'

Part 5: Checking the Assumptions#

fitted_values = multi_model.predict(X_multi)
residuals = y - fitted_values
plt.figure(figsize=(8, 5))
plt.scatter(fitted_values, residuals, color='steelblue')
plt.axhline(0, color='darkred', linestyle='--')
plt.title('Real Residuals vs. Real Fitted Values')
plt.xlabel('Fitted mpg')
plt.ylabel('Residual')
plt.show()

bp_stat, bp_p, _, _ = sm.stats.diagnostic.het_breuschpagan(residuals, X_with_const)
print(f'Real Breusch-Pagan statistic: {bp_stat:.4f}, p-value: {bp_p:.4f}')
jb_stat, jb_p = stats.jarque_bera(residuals)
print(f'Real Jarque-Bera statistic: {jb_stat:.4f}, p-value: {jb_p:.4f}')
sh_stat, sh_p = stats.shapiro(residuals)
print(f'Real Shapiro-Wilk statistic: {sh_stat:.4f}, p-value: {sh_p:.4f}')
dw_stat = durbin_watson(residuals)
print(f'Real Durbin-Watson statistic: {dw_stat:.4f}')

Real Breusch-Pagan statistic: 0.9459, p-value: 0.8143
Real Jarque-Bera statistic: 4.0376, p-value: 0.1328
Real Shapiro-Wilk statistic: 0.9273, p-value: 0.0330
Real Durbin-Watson statistic: 1.3673

Two real normality tests disagreeing at a small sample size isn't a bug, it's an honest, common real outcome: Shapiro-Wilk has more power to detect small real deviations, Jarque-Bera less so. With only thirty-two real cars, don't treat either verdict as final; look at the real residual plot itself, which shows nothing dramatically non-normal.

Part 6: Why a Single Train/Test Split Isn't Enough Here#

kfold = KFold(n_splits=5, shuffle=True, random_state=42)
cv_r2 = cross_val_score(LinearRegression(), X_multi, y, cv=kfold, scoring='r2')
cv_mae = -cross_val_score(LinearRegression(), X_multi, y, cv=kfold, scoring='neg_mean_absolute_error')
print(f'Real per-fold R-squared: {[round(v, 4) for v in cv_r2]}')
print(f'Real mean CV R-squared: {cv_r2.mean():.4f}')
print(f'Real per-fold MAE: {[round(v, 4) for v in cv_mae]}')
print(f'Real mean CV MAE: {cv_mae.mean():.4f}')

Real per-fold R-squared: [np.float64(0.7888), np.float64(0.4365), np.float64(0.8736), np.float64(0.7548), np.float64(0.8402)]
Real mean CV R-squared: 0.7388
Real per-fold MAE: [np.float64(2.2469), np.float64(2.1832), np.float64(1.1376), np.float64(2.8113), np.float64(2.3757)]
Real mean CV MAE: 2.1509

The real spread across folds tells its own story: one real fold scores an R-squared as low as 0.44, another as high as 0.87, purely from which six or seven real cars happened to land in the held-out group. The full-data R-squared of 0.83 from part three was optimistic; the honest real out-of-sample estimate is closer to 0.74, with real uncertainty around it that a single number can't convey.

Part 7: Regularization: Ridge and Lasso#

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X_multi)
ols_scaled = LinearRegression().fit(X_scaled, y)
print(f'Real standardized OLS coefficients: {[round(c, 4) for c in ols_scaled.coef_]}')

Real standardized OLS coefficients: [np.float64(-3.6604), np.float64(-2.1025), np.float64(-0.1143)]

for alpha in [0.1, 1.0, 10.0]:
    ridge = Ridge(alpha=alpha).fit(X_scaled, y)
    lasso = Lasso(alpha=alpha).fit(X_scaled, y)
    print(f'alpha={alpha}: Ridge {[round(c, 4) for c in ridge.coef_]}, Lasso {[round(c, 4) for c in lasso.coef_]}')

alpha=0.1: Ridge [np.float64(-3.6049), np.float64(-2.0799), np.float64(-0.181)], Lasso [np.float64(-3.5953), np.float64(-2.0398), np.float64(-0.1218)]
alpha=1.0: Ridge [np.float64(-3.2287), np.float64(-1.9316), np.float64(-0.6138)], Lasso [np.float64(-3.0006), np.float64(-1.4679), np.float64(-0.2022)]
alpha=10.0: Ridge [np.float64(-2.1813), np.float64(-1.561), np.float64(-1.4142)], Lasso [np.float64(-0.0), np.float64(-0.0), np.float64(-0.0)]

for alpha in [0.1, 1.0, 10.0]:
    ridge_cv = cross_val_score(Ridge(alpha=alpha), X_scaled, y, cv=kfold, scoring='r2').mean()
    lasso_cv = cross_val_score(Lasso(alpha=alpha), X_scaled, y, cv=kfold, scoring='r2').mean()
    print(f'alpha={alpha}: Real CV R-squared Ridge={ridge_cv:.4f}, Lasso={lasso_cv:.4f}')
ols_cv_scaled = cross_val_score(LinearRegression(), X_scaled, y, cv=kfold, scoring='r2').mean()
print(f'Real CV R-squared, plain OLS: {ols_cv_scaled:.4f}')

alpha=0.1: Real CV R-squared Ridge=0.7408, Lasso=0.7456
alpha=1.0: Real CV R-squared Ridge=0.7511, Lasso=0.7497
alpha=10.0: Real CV R-squared Ridge=0.7663, Lasso=-0.0749
Real CV R-squared, plain OLS: 0.7388

A real, honest result: at alpha equals 10, Ridge's cross-validated R-squared actually rises to about 0.77, mildly beating plain OLS's 0.74, because shrinking the collinear coefficients reduces real overfitting to noise. Lasso at that same strength collapses to negative territory, worse than just predicting the real average mpg, because it zeroed out every real coefficient entirely. Regularization isn't automatically better, and more of it isn't automatically better either; it has to be tuned and checked against a real held-out score, not assumed.

Part 8: Evaluation Metrics, Recapped#

n_obs = len(y)
n_predictors = X_multi.shape[1]
r_squared = multi_model.score(X_multi, y)
adjusted_r_squared = 1 - (1 - r_squared) * (n_obs - 1) / (n_obs - n_predictors - 1)
print(f'Real R-squared: {r_squared:.4f}')
print(f'Real adjusted R-squared: {adjusted_r_squared:.4f}')

Real R-squared: 0.8268
Real adjusted R-squared: 0.8083

Mean absolute error is the real average size of a miss, in real miles per gallon, regardless of direction. Root mean squared error is similar but squares errors first, so a few real large misses pull it up more than MAE. R-squared and adjusted R-squared describe real explanatory power; MAE and RMSE describe real prediction error in the target's own real units, which is usually the more useful number to report to a non-technical real audience.

Part 9: Strengths, Weaknesses, and When to Use It#

Strength: every coefficient has a direct, real, interpretable meaning.
Strength: fast to fit, no hyperparameter search required for the plain version.
Weakness: assumes a real linear relationship; real curved patterns need transformed features or a different algorithm entirely.
Weakness: sensitive to real multicollinearity and to real outliers, as parts four and five showed directly.
Use it when interpretability matters as much as accuracy, and when the real relationship is plausibly close to linear.

Part 10: Saving Your Work#

final_model = Ridge(alpha=10).fit(X_scaled, y)
predicted_mpg = final_model.predict(X_scaled)
plt.figure(figsize=(8, 5))
plt.scatter(y, predicted_mpg, color='steelblue')
plt.plot([y.min(), y.max()], [y.min(), y.max()], color='darkred', linestyle='--')
plt.title('Real Actual vs. Predicted MPG (Final Ridge Model)')
plt.xlabel('Actual MPG')
plt.ylabel('Predicted MPG')
plt.savefig('linear_regression_final_fit.png', dpi=150)
plt.show()

import joblib
joblib.dump(final_model, 'linear_regression_model.joblib')
reloaded_model = joblib.load('linear_regression_model.joblib')
print(f'Real original model R-squared: {final_model.score(X_scaled, y):.4f}')
print(f'Real reloaded model R-squared: {reloaded_model.score(X_scaled, y):.4f}')

Real original model R-squared: 0.8070
Real reloaded model R-squared: 0.8070

Wrap-Up: What You Learned#

The real math behind a fitted regression line, and how it extends to multiple real predictors.
Diagnosing real multicollinearity with correlation and VIF, and why it distorts coefficients rather than predictions.
Testing every real OLS assumption directly: linearity, constant variance, normality, and independence.
Why a real small dataset needs cross-validation, not a single lucky or unlucky train/test split.
Ridge and Lasso regularization, including a real case where mild Ridge helps and heavy Lasso actively hurts.
Video two moves to classification: logistic regression, on a real, larger medical dataset. Subscribe so it lands automatically see you there.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.

☕ Buy me a coffee Other ways to help