Mathew K Analytics

Lesson 51 · Pandas Full Course

Scaling and Normalizing Data in Pandas and scikit-learn (Do It Right) | Pandas Course #51

In this lesson you'll be able to: Explain why features on different scales mislead a model Min-max scale and standardise columns with plain pandas Use…

📓 Full notebook

Download .ipynb

Pandas 51: Scaling and Normalizing Data#

In this lesson you'll be able to:

  • Explain why features on different scales mislead a model
  • Min-max scale and standardise columns with plain pandas
  • Use scikit-learn's StandardScaler, MinMaxScaler and RobustScaler
  • Tame outliers and skew with robust scaling and a log transform
  • Fit a scaler on training data only, and prove scaling helps Data: Palmer penguins, plus California housing for outliers

Warm-up: encoding, from lesson 50#

Lesson 50 turned text categories into numbers. Same idea, penguins.

import pandas as pd
pens = pd.read_csv("data/pandas/penguins.csv")
print(pens.shape)
pens.head()
(344, 7)
species island bill_length_mm bill_depth_mm flipper_length_mm body_mass_g sex
0 Adelie Torgersen 39.1 18.7 181.0 3750.0 MALE
1 Adelie Torgersen 39.5 17.4 186.0 3800.0 FEMALE
2 Adelie Torgersen 40.3 18.0 195.0 3250.0 FEMALE
3 Adelie Torgersen NaN NaN NaN NaN NaN
4 Adelie Torgersen 36.7 19.3 193.0 3450.0 FEMALE
dummies = pd.get_dummies(pens["island"], dtype=int)
dummies.sum()
Biscoe       168
Dream        124
Torgersen     52
dtype: int64

Why scale? Big numbers shout loudest#

Problem: columns measured in different units can't be compared fairly.

num = ["bill_length_mm", "bill_depth_mm",
       "flipper_length_mm", "body_mass_g"]
pens = pens.dropna(subset=num)
pens[num].agg(["min", "max", "mean"]).round(1)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
min 32.1 13.1 172.0 2700.0
max 59.6 21.5 231.0 6300.0
mean 43.9 17.2 200.9 4201.8
a = pens.loc[0, num]
b = pens.loc[1, num]
gap = (a - b).abs()
gap
bill_length_mm        0.4
bill_depth_mm         1.3
flipper_length_mm     5.0
body_mass_g          50.0
dtype: float64

Quick check: which column decides the distance between these two birds?

sq = gap ** 2
print(round(sq.sum() ** 0.5, 1))
(sq / sq.sum()).round(3)
50.3
bill_length_mm       0.000
bill_depth_mm        0.001
flipper_length_mm    0.010
body_mass_g          0.989
dtype: float64

Scaling puts every column on a similar range, so each gets a fair vote.

Min-max scaling: squeeze every column into 0 to 1#

Formula: (x - min) / (max - min). Smallest becomes 0, largest becomes 1.

mass = pens["body_mass_g"]
mass_mm = (mass - mass.min()) / (mass.max() - mass.min())
print(mass.head(3).tolist())
print(mass_mm.head(3).round(3).tolist())
[3750.0, 3800.0, 3250.0]
[0.292, 0.306, 0.153]
feats = pens[num]
mm = (feats - feats.min()) / (feats.max() - feats.min())
mm.agg(["min", "max"])
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
min 0.0 0.0 0.0 0.0
max 1.0 1.0 1.0 1.0
mm.head().round(3)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
0 0.255 0.667 0.153 0.292
1 0.269 0.512 0.237 0.306
2 0.298 0.583 0.390 0.153
4 0.167 0.738 0.356 0.208
5 0.262 0.893 0.305 0.264

Common mistake: scaling the whole table, text columns included

(pens - pens.min()) / (pens.max() - pens.min())
---------------------------------------------------------------------------
ArrowNotImplementedError                  Traceback (most recent call last)
File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1097, in ArrowExtensionArray._evaluate_op_method(self, other, op, arrow_funcs)
   1096 try:
-> 1097     result = pc_func(self._pa_array, other)
   1098 except pa.ArrowNotImplementedError as err:

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\compute.py:255, in _make_generic_wrapper.<locals>.wrapper(memory_pool, *args)
    254     return Expression._call(func_name, list(args))
--> 255 return func.call(args, None, memory_pool)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\_compute.pyx:405, in pyarrow._compute.Function.call()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\error.pxi:155, in pyarrow.lib.pyarrow_internal_check_status()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pyarrow\error.pxi:92, in pyarrow.lib.check_status()

ArrowNotImplementedError: Function 'subtract_checked' has no kernel matching input types (large_string, large_string)

The above exception was the direct cause of the following exception:

TypeError                                 Traceback (most recent call last)
Cell In[10], line 1
----> 1 (pens - pens.min()) / (pens.max() - pens.min())

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\common.py:85, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
     82     other = sanitize_array(other, None)
     83     other = ensure_wrapped_if_datetimelike(other)
---> 85 return method(self, other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:198, in OpsMixin.__sub__(self, other)
    196 @unpack_zerodim_and_defer("__sub__")
    197 def __sub__(self, other):
--> 198     return self._arith_method(other, operator.sub)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:9153, in DataFrame._arith_method(self, other, op)
   9150 self, other = self._align_for_op(other, axis, flex=True, level=None)
   9152 with np.errstate(all="ignore"):
-> 9153     new_data = self._dispatch_frame_op(other, op, axis=axis)
   9154 return self._construct_result(new_data, other=other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\frame.py:9196, in DataFrame._dispatch_frame_op(self, right, func, axis)
   9190     assert self.columns.equals(right.columns)
   9191     # TODO: The previous assertion `assert right._indexed_same(self)`
   9192     #  fails in cases with empty columns reached via
   9193     #  _frame_arith_method_with_reindex
   9194 
   9195     # TODO operate_blockwise expects a manager of the same type
-> 9196     bm = self._mgr.operate_blockwise(
   9197         right._mgr,
   9198         array_op,
   9199     )
   9200     return self._constructor_from_mgr(bm, axes=bm.axes)
   9202 elif isinstance(right, Series) and axis == 1:
   9203     # axis=1 means we want to operate row-by-row

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\managers.py:1704, in BlockManager.operate_blockwise(self, other, array_op)
   1700 def operate_blockwise(self, other: BlockManager, array_op) -> BlockManager:
   1701     """
   1702     Apply array_op blockwise with another (aligned) BlockManager.
   1703     """
-> 1704     return operate_blockwise(self, other, array_op)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\internals\ops.py:65, in operate_blockwise(left, right, array_op)
     63 res_blks: list[Block] = []
     64 for lvals, rvals, locs, left_ea, right_ea, rblk in _iter_block_pairs(left, right):
---> 65     res_values = array_op(lvals, rvals)
     66     if (
     67         left_ea
     68         and not right_ea
     69         and hasattr(res_values, "reshape")
     70         and not is_1d_only_ea_dtype(res_values.dtype)
     71     ):
     72         res_values = res_values.reshape(1, -1)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\array_ops.py:279, in arithmetic_op(left, right, op)
    270 right = ensure_wrapped_if_datetimelike(right)
    272 if (
    273     should_extension_dispatch(left, right)
    274     or isinstance(right, (Timedelta, BaseOffset, Timestamp))
   (...)
    277     # Timedelta/Timestamp and other custom scalars are included in the check
    278     # because numexpr will fail on it, see GH#31457
--> 279     res_values = op(left, right)
    280 else:
    281     # TODO we should handle EAs consistently and move this check before the if/else
    282     # (https://github.com/pandas-dev/pandas/issues/41165)
    283     # error: Argument 2 to "_bool_arith_check" has incompatible type
    284     # "Union[ExtensionArray, ndarray[Any, Any]]"; expected "ndarray[Any, Any]"
    285     _bool_arith_check(op, left, right)  # type: ignore[arg-type]

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\ops\common.py:85, in _unpack_zerodim_and_defer.<locals>.new_method(self, other)
     82     other = sanitize_array(other, None)
     83     other = ensure_wrapped_if_datetimelike(other)
---> 85 return method(self, other)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arraylike.py:198, in OpsMixin.__sub__(self, other)
    196 @unpack_zerodim_and_defer("__sub__")
    197 def __sub__(self, other):
--> 198     return self._arith_method(other, operator.sub)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1158, in ArrowExtensionArray._arith_method(self, other, op)
   1154 if pa.types.is_string(self._pa_array.type) or pa.types.is_large_string(
   1155     self._pa_array.type
   1156 ):
   1157     try:
-> 1158         result = self._evaluate_op_method(other, op, ARROW_ARITHMETIC_FUNCS)
   1159     except (pa.ArrowInvalid, pa.ArrowTypeError):
   1160         result = self._str_arith_method_object_fallback(other, op)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\pandas\core\arrays\arrow\array.py:1099, in ArrowExtensionArray._evaluate_op_method(self, other, op, arrow_funcs)
   1097     result = pc_func(self._pa_array, other)
   1098 except pa.ArrowNotImplementedError as err:
-> 1099     raise TypeError(self._op_method_error_message(other_original, op)) from err
   1100 return self._from_pyarrow_array(result)

TypeError: operation 'sub' not supported for dtype 'str' with dtype 'object'
nums = pens.select_dtypes("number")
print(nums.columns.tolist())
((nums - nums.min()) / (nums.max() - nums.min())).max()
['bill_length_mm', 'bill_depth_mm', 'flipper_length_mm', 'body_mass_g']
bill_length_mm       1.0
bill_depth_mm        1.0
flipper_length_mm    1.0
body_mass_g          1.0
dtype: float64

Your turn#

Min-max scale flipper length. Which penguin scores exactly 1?

fl = pens["flipper_length_mm"]
fl_mm = (fl - fl.min()) / (fl.max() - fl.min())
pens.loc[fl_mm == 1, ["species", "island", "flipper_length_mm"]]
species island flipper_length_mm
283 Gentoo Biscoe 231.0

Standardisation: how many standard deviations from average?#

Formula: (x - mean) / std. The result is called a z-score.

z = (feats - feats.mean()) / feats.std()
z.head().round(2)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
0 -0.88 0.78 -1.42 -0.56
1 -0.81 0.13 -1.06 -0.50
2 -0.66 0.43 -0.42 -1.19
4 -1.32 1.09 -0.56 -0.94
5 -0.85 1.75 -0.78 -0.69
z.agg(["mean", "std"]).round(3)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
mean 0.0 -0.0 -0.0 0.0
std 1.0 1.0 1.0 1.0

Quick check: how far above average is the heaviest penguin?

heavy = z["body_mass_g"].idxmax()
print(pens.loc[heavy, "species"], pens.loc[heavy, "body_mass_g"])
print(round(z.loc[heavy, "body_mass_g"], 2))
Gentoo 6300.0
2.62
unusual = (z.abs() > 2).sum()
unusual
bill_length_mm       6
bill_depth_mm        5
flipper_length_mm    9
body_mass_g          9
dtype: int64
zgap = (z.loc[0] - z.loc[1]) ** 2
(zgap / zgap.sum()).round(3)
bill_length_mm       0.009
bill_depth_mm        0.762
flipper_length_mm    0.222
body_mass_g          0.007
dtype: float64
ax = z.plot(kind="box", figsize=(8, 3.5), rot=0)
ax.set_title("Penguin measurements as z-scores")
ax.set_xlabel("Measurement")
ax.set_ylabel("Standard deviations from mean")
No description has been provided for this image

Min-max: a fixed 0 to 1 range. Z-scores: centred on 0, spread of 1.

The scikit-learn scalers#

Problem: remember the scaling rule and apply it to new data later.

from sklearn.preprocessing import StandardScaler
std = StandardScaler().set_output(transform="pandas")
z_sk = std.fit_transform(feats)
z_sk.head().round(2)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
0 -0.88 0.79 -1.42 -0.56
1 -0.81 0.13 -1.06 -0.50
2 -0.66 0.43 -0.42 -1.19
4 -1.32 1.09 -0.56 -0.94
5 -0.85 1.75 -0.78 -0.69
print(std.mean_.round(1))
print(std.scale_.round(1))
[  43.9   17.2  200.9 4201.8]
[  5.5   2.   14.  800.8]

Quick check: the pandas and scikit-learn z-scores differ slightly. Why?

print(round(feats["body_mass_g"].std(), 2))
print(round(feats["body_mass_g"].std(ddof=0), 2))
801.95
800.78

Common mistake: passing a single column with single brackets

std.fit_transform(pens["body_mass_g"])
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[22], line 1
----> 1 std.fit_transform(pens["body_mass_g"])

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py:327, in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    325 @wraps(f)
    326 def wrapped(self, X, *args, **kwargs):
--> 327     data_to_wrap = f(self, X, *args, **kwargs)
    328     if isinstance(data_to_wrap, tuple):
    329         # only wrap the first output for cross decomposition
    330         return_tuple = (
    331             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    332             *data_to_wrap[1:],
    333         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:969, in TransformerMixin.fit_transform(self, X, y, **fit_params)
    954         warnings.warn(
    955             (
    956                 f"This object ({self.__class__.__name__}) has a `transform`"
   (...)
    964             UserWarning,
    965         )
    967 if y is None:
    968     # fit method of arity 1 (unsupervised transformation)
--> 969     return self.fit(X, **fit_params).transform(X)
    970 else:
    971     # fit method of arity 2 (supervised transformation)
    972     return self.fit(X, y, **fit_params).transform(X)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py:928, in StandardScaler.fit(self, X, y, sample_weight)
    926 # Reset internal state before fitting
    927 self._reset()
--> 928 return self.partial_fit(X, y, sample_weight)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\base.py:1403, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1393     estimator._validate_params()
   1395 with (
   1396     config_context(
   1397         skip_parameter_validation=(
   (...)
   1401     callback_management_context(estimator),
   1402 ):
-> 1403     return fit_method(estimator, *args, **kwargs)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py:965, in StandardScaler.partial_fit(self, X, y, sample_weight)
    963 xp, _, X_device = get_namespace_and_device(X)
    964 first_call = not hasattr(self, "n_samples_seen_")
--> 965 X = validate_data(
    966     self,
    967     X,
    968     accept_sparse=("csr", "csc"),
    969     dtype=supported_float_dtypes(xp, X_device),
    970     ensure_all_finite="allow-nan",
    971     reset=first_call,
    972 )
    973 n_features = X.shape[1]
    975 callback_ctx = self._init_callback_context()

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:3041, in validate_data(_estimator, X, y, reset, validate_separately, skip_check_array, **check_params)
   3039         out = X, y
   3040 elif not no_val_X and no_val_y:
-> 3041     out = check_array(X, input_name="X", **check_params)
   3042 elif no_val_X and not no_val_y:
   3043     out = _check_y(y, **check_params)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1076, in check_array(array, accept_sparse, accept_large_sparse, dtype, order, copy, force_writeable, ensure_all_finite, ensure_non_negative, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, estimator, input_name)
   1069         else:
   1070             msg = (
   1071                 f"Expected 2D array, got 1D array instead:\narray={array}.\n"
   1072                 "Reshape your data either using array.reshape(-1, 1) if "
   1073                 "your data has a single feature or array.reshape(1, -1) "
   1074                 "if it contains a single sample."
   1075             )
-> 1076         raise ValueError(msg)
   1078 if dtype_numeric and hasattr(array.dtype, "kind") and array.dtype.kind in "USV":
   1079     raise ValueError(
   1080         "dtype='numeric' is not compatible with arrays of bytes/strings."
   1081         "Convert your data to numeric values explicitly instead."
   1082     )

ValueError: Expected a 2-dimensional container but got <class 'pandas.Series'> instead. Pass a DataFrame containing a single row (i.e. single sample) or a single column (i.e. single feature) instead.
one = std.fit_transform(pens[["body_mass_g"]])
one.describe().round(2)
body_mass_g
count 342.00
mean 0.00
std 1.00
min -1.88
25% -0.81
50% -0.19
75% 0.68
max 2.62
from sklearn.preprocessing import MinMaxScaler
mms = MinMaxScaler().set_output(transform="pandas")
mm_sk = mms.fit_transform(feats)
print(round((mm_sk - mm).abs().max().max(), 10))
0.0
wide = MinMaxScaler(feature_range=(-1, 1))
wide.set_output(transform="pandas")
wide.fit_transform(feats).agg(["min", "max"])
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
min -1.0 -1.0 -1.0 -1.0
max 1.0 1.0 1.0 1.0
back = mms.inverse_transform(mm_sk.head(3))
back = pd.DataFrame(back, columns=num)
back.round(1)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
0 39.1 18.7 181.0 3750.0
1 39.5 17.4 186.0 3800.0
2 40.3 18.0 195.0 3250.0

Outliers and skew: robust scaling and logs#

Problem: a few huge values squash everyone else together.

house = pd.read_csv("data/pandas/housing.csv")
pop = house["population"]
pop.describe().round(0)
count    20640.0
mean      1425.0
std       1132.0
min          3.0
25%        787.0
50%       1166.0
75%       1725.0
max      35682.0
Name: population, dtype: float64
pop_mm = (pop - pop.min()) / (pop.max() - pop.min())
print(round(pop_mm.median(), 3))
print(round((pop_mm < 0.1).mean(), 3))
0.033
0.96
from sklearn.preprocessing import RobustScaler
rob = RobustScaler().set_output(transform="pandas")
pop_rob = rob.fit_transform(house[["population"]])
pop_rob.describe().round(2)
population
count 20640.00
mean 0.28
std 1.21
min -1.24
25% -0.40
50% 0.00
75% 0.60
max 36.80
import numpy as np
cols = ["population", "total_rooms", "median_income"]
logged = np.log1p(house[cols])
pd.DataFrame({"before": house[cols].skew(),
              "after": logged.skew()}).round(2)
before after
population 4.94 -1.04
total_rooms 4.15 -1.08
median_income 1.65 0.23
import matplotlib.pyplot as plt
fig, axes = plt.subplots(1, 2, figsize=(9, 3.5))
pop.plot.hist(bins=50, ax=axes[0])
axes[0].set_title("Raw population")
logged["population"].plot.hist(bins=50, ax=axes[1])
axes[1].set_title("log1p(population)")
for ax in axes:
    ax.set_xlabel("Value")
    ax.set_ylabel("Number of blocks")
plt.show()
No description has been provided for this image

Your turn#

Log total_bedrooms, then standardise it. Report the new skew and mean.

beds = np.log1p(house["total_bedrooms"])
beds_z = (beds - beds.mean()) / beds.std()
print(round(beds_z.skew(), 2), round(beds_z.mean(), 2))
print(beds_z.isna().sum())
-0.99 0.0
207

Fit on training data only#

Problem: the scaler must not peek at the data we test on.

from sklearn.model_selection import train_test_split
y = pens["species"]
X_tr, X_te, y_tr, y_te = train_test_split(
    feats, y, test_size=0.25, random_state=42, stratify=y)
print(X_tr.shape, X_te.shape)
(256, 4) (86, 4)

Common mistake: transforming before the scaler has been fitted

fresh = StandardScaler().set_output(transform="pandas")
fresh.transform(X_te)
---------------------------------------------------------------------------
NotFittedError                            Traceback (most recent call last)
Cell In[34], line 2
      1 fresh = StandardScaler().set_output(transform="pandas")
----> 2 fresh.transform(X_te)

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\_set_output.py:327, in _wrap_method_output.<locals>.wrapped(self, X, *args, **kwargs)
    325 @wraps(f)
    326 def wrapped(self, X, *args, **kwargs):
--> 327     data_to_wrap = f(self, X, *args, **kwargs)
    328     if isinstance(data_to_wrap, tuple):
    329         # only wrap the first output for cross decomposition
    330         return_tuple = (
    331             _wrap_data_with_container(method, data_to_wrap[0], X, self),
    332             *data_to_wrap[1:],
    333         )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\preprocessing\_data.py:1108, in StandardScaler.transform(self, X, copy)
   1093 """Perform standardization by centering and scaling.
   1094 
   1095 Parameters
   (...)
   1105     Transformed array.
   1106 """
   1107 xp, _, X_device = get_namespace_and_device(X)
-> 1108 check_is_fitted(self)
   1110 copy = copy if copy is not None else self.copy
   1111 X = validate_data(
   1112     self,
   1113     X,
   (...)
   1119     ensure_all_finite="allow-nan",
   1120 )

File ~\AppData\Local\Programs\Python\Python312\Lib\site-packages\sklearn\utils\validation.py:1721, in check_is_fitted(estimator, attributes, msg, all_or_any)
   1718     return
   1720 if not _is_fitted(estimator, attributes, all_or_any):
-> 1721     raise NotFittedError(msg % {"name": type(estimator).__name__})

NotFittedError: This StandardScaler instance is not fitted yet. Call 'fit' with appropriate arguments before using this estimator.
fresh.fit(X_tr)
tr_z = fresh.transform(X_tr)
te_z = fresh.transform(X_te)
pd.DataFrame({"train": tr_z.mean(), "test": te_z.mean()}).round(2)
train test
bill_length_mm 0.0 -0.14
bill_depth_mm -0.0 0.02
flipper_length_mm 0.0 -0.04
body_mass_g 0.0 -0.06

Quick check: the test means aren't 0. Is something wrong?

No: test rows reuse the training mean and spread, like real new data.

Your turn#

Fit a MinMaxScaler on X_tr, transform X_te. Do test values stay in 0 to 1?

mm_fit = MinMaxScaler().set_output(transform="pandas")
mm_fit.fit(X_tr)
mm_fit.transform(X_te).agg(["min", "max"]).round(3)
bill_length_mm bill_depth_mm flipper_length_mm body_mass_g
min -0.038 0.048 0.068 0.042
max 0.796 0.964 0.983 0.917

Mini-project: does scaling help a model tell species apart?#

Plan: k nearest neighbours on raw data, then on scaled data, then compare.

from sklearn.neighbors import KNeighborsClassifier
raw_knn = KNeighborsClassifier(n_neighbors=5)
raw_knn.fit(X_tr, y_tr)
raw_acc = raw_knn.score(X_te, y_te)
print(round(raw_acc, 3))
0.791
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(tr_z, y_tr)
sc_acc = knn.score(te_z, y_te)
print(round(sc_acc, 3))
1.0
wrong = X_te[raw_knn.predict(X_te) != y_te]
print(y_te[wrong.index].value_counts())
species
Chinstrap    13
Gentoo        3
Adelie        2
Name: count, dtype: int64
accs = pd.Series({"Raw": raw_acc, "Standardised": sc_acc})
ax = accs.plot(kind="bar", rot=0, figsize=(6, 3.5))
ax.set_title("KNN accuracy on penguin species")
ax.set_xlabel("Features")
ax.set_ylabel("Test accuracy")
plt.show()
No description has been provided for this image
gain = (sc_acc - raw_acc) * len(y_te)
print(f"Scaling fixed {gain:.0f} of {len(y_te)} test penguins")
Scaling fixed 18 of 86 test penguins

Recap#

  • Distance-based models let big-number columns dominate
  • Min-max maps to 0 to 1; z-scores centre on 0 with spread 1
  • Scale numeric columns only, and pass 2D data to scalers
  • Robust scaling and log1p tame outliers and skew
  • Fit on the training data, then transform both piles

Challenges#

  1. Repeat the mini-project with MinMaxScaler. Same result?
  2. Scale housing's median_income with Standard and Robust scalers; compare max.
  3. Try n_neighbors from 1 to 15 on raw data. Does any k match scaling?

What's next#

Lesson 52: Feature Engineering with Pandas Create new, more informative columns from the ones you already have.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.