Mathew K Analytics

Lesson 5 · R Programming: Zero to Expert

Statistics in R: Hypothesis Tests, Regression & Modelling (Full Course) | R 105

Video 5: from describing data to testing ideas and building models. Datasets: penguins, mtcars, PlantGrowth, ToothGrowth and sleep. R script:…

⬇ Download R script⬇ R notebook (.ipynb)

What you'll learn

Files for this lesson

Save these in one folder, open the script in RStudio and set that folder as the working directory (Session → Set Working Directory → To Source File Location).

📓 Full notebook

Download .ipynb

Statistics and Modelling in R#

Video 5: from describing data to testing ideas and building models.

Datasets: penguins, mtcars, PlantGrowth, ToothGrowth and sleep.

R script: 05_statistics_modelling.R. Download it from this page and open it in RStudio, or run this notebook with an R kernel (for example in Google Colab: Runtime → Change runtime type → R).

Setup and a quick recap#

Load the packages we need for the whole video.

library(tidyverse)

The tidyverse loads its core packages, dplyr, ggplot2, tidyr and friends, and prints its usual list of conflicts, which we can safely ignore.

library(broom)

Nothing is printed, which means broom loaded without problems.

library(palmerpenguins)

peng <- penguins |>
  drop_na(body_mass_g, flipper_length_mm, sex)
nrow(penguins)
nrow(peng)

The original data has 344 penguins, and after dropping rows with missing values we keep 333 of them.

Descriptive statistics#

Centre: mean and median. Spread: sd, var, quantiles and IQR.

mass <- peng$body_mass_g

mean(mass)
median(mass)
sd(mass)
var(mass)

The mean body mass is about 4207 grams and the median is 4050, a bit lower. The standard deviation is about 805 grams, and the variance is roughly 648 thousand, which is in squared grams and hard to picture.

quantile(mass)
quantile(mass, probs = c(0.1, 0.9))
IQR(mass)
range(mass)

The quartiles are 2700, 3550, 4050, 4775 and 6300 grams. Ten percent of penguins weigh under 3300 grams, and ninety percent weigh under 5440. The interquartile range is 1225 grams, the gap between the first and third quartile, and the full range runs from 2700 to 6300.

peng |>
  group_by(species) |>
  summarise(
    n      = n(),
    mean   = mean(body_mass_g),
    median = median(body_mass_g),
    sd     = sd(body_mass_g),
    iqr    = IQR(body_mass_g)
  )

Adelie and Chinstrap penguins both average around 3700 grams, while Gentoo penguins average about 5090 grams. The standard deviations are all between about 380 and 500 grams, so the groups are similarly spread out but sit at different centres.

x <- c(12, 15, 14, 13, 16, 15, 14, 250)
mean(x)
median(x)

The mean jumps to about 43.6, which describes none of the values, while the median stays at a sensible 14.5.

Pause and try: mean, median and sd of flipper length

for each sex of penguin.

peng |>
  group_by(sex) |>
  summarise(
    mean_flip   = mean(flipper_length_mm),
    median_flip = median(flipper_length_mm),
    sd_flip     = sd(flipper_length_mm)
  )

Females average about 197 millimetres and males about 205, so males have flippers roughly seven millimetres longer on average.

Probability distributions#

d = density, p = probability below, q = quantile, r = random.

set.seed(42)
heights <- rnorm(1000, mean = 170, sd = 10)

head(round(heights, 1))
mean(heights)
sd(heights)

The first heights are values like 183.7, 164.4 and 173.6. The sample mean is about 169.7 and the standard deviation about 10, very close to what we asked for.

ggplot(tibble(height = heights), aes(x = height)) +
  geom_histogram(aes(y = after_stat(density)),
                 bins = 30, fill = 'grey80') +
  stat_function(fun = dnorm,
                args = list(mean = 170, sd = 10),
                colour = 'steelblue', linewidth = 1.2) +
  labs(title = '1000 simulated heights',
       x = 'Height (cm)', y = 'Density') +
  theme_minimal(base_size = 14)

The grey histogram of our simulated heights follows the blue bell curve closely, peaking near 170 and fading out by about 140 and 200.

pnorm(180, mean = 170, sd = 10)
1 - pnorm(190, mean = 170, sd = 10)
pnorm(180, 170, 10) - pnorm(160, 170, 10)

About 84 percent of people are shorter than 180 centimetres, only about 2.3 percent are taller than 190, and about 68 percent fall between 160 and 180.

qnorm(0.5, mean = 170, sd = 10)
qnorm(0.95, mean = 170, sd = 10)
qnorm(c(0.025, 0.975))

The median height is exactly 170, the ninety fifth percentile is about 186.4 centimetres, and the standard normal cut points are minus 1.96 and plus 1.96.

dbinom(5, size = 10, prob = 0.5)
dbinom(0:10, size = 10, prob = 0.5) |> round(3)
pbinom(2, size = 10, prob = 0.5)

Exactly five heads out of ten has a probability of about 0.246, the most likely single outcome. The full list is symmetric, from 0.001 for zero heads up to 0.246 in the middle and back down. Getting two heads or fewer has a probability of about 0.055.

set.seed(1)
rbinom(8, size = 10, prob = 0.5)
runif(5, min = 0, max = 1) |> round(3)
sample(1:6, size = 10, replace = TRUE)

rbinom gave head counts like 4, 4, 5 and 7. runif gave five decimals spread between zero and one, and sample rolled ten dice, all whole numbers from one to six.

Pause and try: if IQ is normal with mean 100 and sd 15,

what share of people score above 130? What score is the top 1%?

1 - pnorm(130, mean = 100, sd = 15)
qnorm(0.99, mean = 100, sd = 15)

About 2.3 percent of people score above 130, and you need a score of about 134.9 to be in the top one percent.

Simulation: let the computer flip the coins#

Simulate coins, dice and the central limit theorem.

set.seed(2024)
flips <- sample(c('H', 'T'), size = 100, replace = TRUE)
table(flips)
mean(flips == 'H')

Out of one hundred flips we got 52 heads and 48 tails, so the proportion of heads is 0.52, not exactly one half.

set.seed(2024)
n_flips <- 1:5000
heads <- rbinom(5000, size = 1, prob = 0.5)
running <- tibble(n = n_flips,
                  prop = cumsum(heads) / n_flips)

ggplot(running, aes(n, prop)) +
  geom_line(colour = 'steelblue') +
  geom_hline(yintercept = 0.5, linetype = 'dashed') +
  scale_x_log10() +
  labs(title = 'The law of large numbers',
       x = 'Number of flips (log scale)',
       y = 'Running proportion of heads') +
  theme_minimal(base_size = 14)

On the left, after only a few flips, the line jumps wildly around. As the number of flips grows towards five thousand, it settles down right on the dashed line at one half.

set.seed(7)
rolls <- sample(1:6, size = 6000, replace = TRUE)
table(rolls)
mean(rolls)

two_dice <- sample(1:6, 6000, TRUE) + sample(1:6, 6000, TRUE)
round(prop.table(table(two_dice)), 3)

Each face of a single die appears roughly a thousand times, and the average roll is about 3.5, exactly as theory says. With two dice, seven is the most common total at about 0.17, while two and twelve are rare at under 0.03 each.

The central limit theorem#

Means of samples are roughly normal, even if the data is not.

set.seed(10)
waits <- rexp(5000, rate = 1 / 10)

ggplot(tibble(wait = waits), aes(wait)) +
  geom_histogram(bins = 40, fill = 'tomato') +
  labs(title = 'Raw data: very skewed waiting times',
       x = 'Minutes', y = 'Count') +
  theme_minimal(base_size = 14)

The histogram is extremely skewed: most waits are short, piled up near zero, with a long tail stretching out past fifty minutes.

set.seed(10)
sample_means <- replicate(
  5000,
  mean(rexp(30, rate = 1 / 10))
)

mean(sample_means)
sd(sample_means)
10 / sqrt(30)

The sample means average about 10, the true mean wait. Their standard deviation is about 1.81, very close to the theoretical value of ten divided by the square root of thirty, which is 1.83.

ggplot(tibble(m = sample_means), aes(m)) +
  geom_histogram(aes(y = after_stat(density)),
                 bins = 40, fill = 'grey80') +
  stat_function(fun = dnorm,
                args = list(mean = 10, sd = 10 / sqrt(30)),
                colour = 'steelblue', linewidth = 1.2) +
  labs(title = 'Means of 5000 samples of size 30',
       x = 'Sample mean (minutes)', y = 'Density') +
  theme_minimal(base_size = 14)

And there it is. The means form a lovely bell shape centred on 10, matching the blue normal curve closely, with only a slight lean to the right.

Pause and try: repeat the experiment with samples of size 5.

Is the standard error bigger or smaller? Is it still normal?

set.seed(10)
small_means <- replicate(5000, mean(rexp(5, rate = 1 / 10)))
sd(small_means)
10 / sqrt(5)
c(above_15 = mean(small_means > 15),
  below_5  = mean(small_means < 5))

The standard error is now about 4.49, matching theory's 4.47, so small samples give much less precise means. And about 14 percent of means land above fifteen but only 10 percent below five, so with samples of five the means are still visibly skewed to the right.

Sampling and confidence intervals#

Estimate a population value from a sample, with a margin of error.

adelie <- peng |> filter(species == 'Adelie')
flip <- adelie$flipper_length_mm

n    <- length(flip)
xbar <- mean(flip)
se   <- sd(flip) / sqrt(n)
c(n = n, mean = xbar, se = se)

We have 146 Adelie penguins with a mean flipper length of about 190.1 millimetres, and a standard error of about 0.54.

t_crit <- qt(0.975, df = n - 1)
t_crit

lower <- xbar - t_crit * se
upper <- xbar + t_crit * se
round(c(lower = lower, upper = upper), 2)

The critical t value is about 1.98, just above the 1.96 from the normal. So our 95 percent confidence interval runs from about 189.0 to 191.2 millimetres.

tt <- t.test(flip)
tt$conf.int
tt$estimate

t.test gives the same interval, 189.0 to 191.2, and the same estimate of about 190.1, so our hand calculation was right.

What does 95% confidence really mean?#

Simulate 100 samples from a known population and count hits.

set.seed(99)
ci_sims <- map(1:100, function(i) {
  s <- rnorm(25, mean = 50, sd = 8)
  ci <- t.test(s)$conf.int
  tibble(sim = i, lower = ci[1], upper = ci[2])
}) |>
  list_rbind() |>
  mutate(hit = lower <= 50 & upper >= 50)

mean(ci_sims$hit)

93 of our one hundred intervals captured the true mean of fifty. That's close to 95, and with only one hundred tries, a couple either way is just random variation.

ggplot(ci_sims, aes(y = sim, xmin = lower, xmax = upper,
                    colour = hit)) +
  geom_linerange(linewidth = 0.8) +
  geom_vline(xintercept = 50, linetype = 'dashed') +
  scale_colour_manual(values = c(`TRUE` = 'grey60',
                                 `FALSE` = 'red')) +
  labs(title = '100 confidence intervals, true mean = 50',
       x = 'Value', y = 'Simulation') +
  theme_minimal(base_size = 14)

Each horizontal line is one interval. Most are grey and cross the dashed line at fifty, but a handful of red ones miss it completely, just by bad luck of the sample.

Hypothesis tests and p-values#

H0 = no effect. p-value: how surprising is our data if H0 is true?

set.seed(3)
fair_heads <- rbinom(10000, size = 100, prob = 0.5)

observed <- 61
mean(abs(fair_heads - 50) >= abs(observed - 50))
binom.test(observed, n = 100, p = 0.5)$p.value

In our ten thousand fair experiments, about 3.6 percent were as extreme as sixty one heads. The exact binomial test agrees, with a p-value of about 0.035.

t-tests: comparing means#

One-sample, two-sample (Welch) and paired.

t.test(flip, mu = 192)

The t statistic is about minus 3.5 with 145 degrees of freedom, and the p-value is about 0.0006. The sample mean of 190.1 sits below 192, and the confidence interval from 189.0 to 191.2 does not contain 192. So the data are very unlikely if the true mean were 192.

t.test(flipper_length_mm ~ sex, data = peng)

This is the Welch two-sample t-test. Females average about 197.4 and males about 204.5 millimetres. The t statistic is about minus 4.8, and the p-value is about 2 times ten to the minus six, tiny. The interval for the difference runs from roughly minus 10 to minus 4 millimetres.

t.test(flipper_length_mm ~ species, data = peng)

R stops with an error: the grouping factor must have exactly two levels. We have three species, so a t-test is the wrong tool.

This one is meant to fail, so you can see the error message.

peng |>
  filter(species %in% c('Adelie', 'Chinstrap')) |>
  t.test(flipper_length_mm ~ species, data = _)

Comparing just Adelie and Chinstrap works. Adelie averages about 190.1 and Chinstrap about 195.8 millimetres, with a p-value of about 1.3 times ten to the minus seven.

Paired t-test#

The sleep data: 10 patients, each tried two sleeping drugs.

sleep_wide <- sleep |>
  pivot_wider(names_from = group, values_from = extra,
              names_prefix = 'drug')
sleep_wide

We now have one row per patient, with an ID column and the extra sleep under drug one and drug two side by side.

t.test(sleep_wide$drug2, sleep_wide$drug1, paired = TRUE)

The mean difference is 1.58 extra hours of sleep with drug two, with a confidence interval from about 0.70 to 2.46 hours. The t statistic is about 4.06 and the p-value about 0.003.

t.test(sleep_wide$drug2, sleep_wide$drug1)$p.value

Ignoring the pairing, the p-value jumps to about 0.08, which would not be significant at the usual level.

Checking assumptions#

t-tests assume roughly normal data, or a decent sample size.

ggplot(peng, aes(sample = flipper_length_mm)) +
  stat_qq(alpha = 0.6) +
  stat_qq_line(colour = 'red') +
  facet_wrap(~ sex) +
  labs(title = 'Normal QQ plots of flipper length') +
  theme_minimal(base_size = 14)

In both panels the points follow the red line in the middle but bend away from it at the ends, in a gentle S shape. That's because each sex mixes three species, so the data is not perfectly normal.

Chi-square test of independence#

Are two categorical variables related?

tab <- table(peng$species, peng$island)
tab

Adelie penguins live on all three islands, but Gentoo penguins appear only on Biscoe, and Chinstrap only on Dream. That looks like a very strong relationship.

chi <- chisq.test(tab)
chi
round(chi$expected, 1)

The chi-squared statistic is huge, about 284, on 4 degrees of freedom, and the p-value is below 2.2 times ten to the minus sixteen, effectively zero. The expected counts show what independence would look like, for example about 44 Gentoo on Dream, where we actually saw none.

Pause and try: is penguin sex related to species?

Build the table and run the chi-square test.

sex_tab <- table(peng$species, peng$sex)
sex_tab
chisq.test(sex_tab)$p.value

Each species is split almost exactly in half between females and males, and the p-value is about 0.98.

One-way ANOVA#

PlantGrowth: dried plant weight under a control and two treatments.

PlantGrowth |>
  group_by(group) |>
  summarise(n = n(), mean = mean(weight), sd = sd(weight))

The control plants average about 5.03, treatment one about 4.66 and treatment two about 5.53. Each group has ten plants and a standard deviation between about 0.44 and 0.79.

ggplot(PlantGrowth, aes(group, weight, fill = group)) +
  geom_boxplot(alpha = 0.5, show.legend = FALSE) +
  geom_jitter(width = 0.1, size = 2) +
  labs(title = 'Plant weight by treatment',
       x = NULL, y = 'Dried weight') +
  theme_minimal(base_size = 14)

Treatment two sits highest and treatment one lowest, with the control in between. The boxes overlap a little, so we need a test to judge whether these differences are real.

plant_aov <- aov(weight ~ group, data = PlantGrowth)
summary(plant_aov)

The F statistic is about 4.85 on 2 and 27 degrees of freedom, with a p-value of about 0.016. So there is evidence that at least one group mean differs from the others.

TukeyHSD(plant_aov)

Only one comparison is clearly significant: treatment two versus treatment one, a difference of about 0.87 with an adjusted p-value of about 0.012. Neither treatment differs significantly from the control, with adjusted p-values of about 0.39 and 0.20.

Pause and try: in ToothGrowth, does tooth length (len) differ

between the three vitamin C doses? Use aov and TukeyHSD.

tooth <- ToothGrowth |> mutate(dose = factor(dose))
tooth_aov <- aov(len ~ dose, data = tooth)
summary(tooth_aov)
TukeyHSD(tooth_aov)$dose |> round(4)

The F statistic is about 67 with a p-value around ten to the minus fifteen, a very strong effect of dose. Tukey shows every pair differs: about 9.1 longer from dose 0.5 to 1, 15.5 from 0.5 to 2, and 6.4 from 1 to 2, all with tiny adjusted p-values.

Correlation tests#

How strongly do two numeric variables move together?

cor(mtcars$wt, mtcars$mpg)
cor.test(mtcars$wt, mtcars$mpg)

The correlation between weight and fuel economy is about minus 0.87, a strong negative relationship: heavier cars do fewer miles per gallon. The test gives a p-value of about 1.3 times ten to the minus ten, and the 95 percent interval for the correlation runs from about minus 0.93 to minus 0.74.

cor.test(~ bill_length_mm + bill_depth_mm, data = peng)$estimate

peng |>
  group_by(species) |>
  summarise(r = cor(bill_length_mm, bill_depth_mm))

Across all penguins, the correlation between bill length and depth is about minus 0.23, slightly negative. But within each species it's positive: about 0.39 for Adelie, 0.65 for Chinstrap and 0.65 for Gentoo.

set.seed(123)
fake_p <- replicate(1000, t.test(rnorm(20), rnorm(20))$p.value)
mean(fake_p < 0.05)

4.8 percent of these tests came out significant, almost exactly one in twenty, even though there was no real difference in any of them.

Simple linear regression#

Model mpg as a straight-line function of car weight.

ggplot(mtcars, aes(wt, mpg)) +
  geom_point(size = 2.5) +
  geom_smooth(method = 'lm', formula = y ~ x) +
  labs(title = 'Heavier cars use more fuel',
       x = 'Weight (1000 lbs)', y = 'Miles per gallon') +
  theme_minimal(base_size = 14)

The points fall from top left to bottom right, and the blue line with its grey confidence band captures the trend well. A few cars sit noticeably above or below the line.

fit1 <- lm(mpg ~ wt, data = mtcars)
fit1

The intercept is about 37.29 and the slope for weight is about minus 5.34.

summary(fit1)

The coefficients table shows the weight slope of minus 5.34 with a standard error of about 0.56, a t value of about minus 9.6, and a p-value of about 1.3 times ten to the minus ten. At the bottom, the residual standard error is about 3.05 miles per gallon, and the R-squared is about 0.75.

Tidy model output with broom#

tidy() = coefficients, glance() = one row of fit stats,

augment() = data plus fitted values and residuals.

tidy(fit1, conf.int = TRUE)
glance(fit1) |> select(r.squared, adj.r.squared, sigma, AIC)

tidy returns a tibble with one row per term, including the estimate, standard error, p-value and the confidence limits for weight, minus 6.49 and minus 4.20. So we're fairly confident each extra thousand pounds costs somewhere between about four and six and a half miles per gallon. glance shows the R-squared of 0.753, adjusted R-squared of 0.745, sigma of 3.05 and an AIC of about 166.

fit1_aug <- augment(fit1)
fit1_aug |>
  select(.rownames, mpg, wt, .fitted, .resid) |>
  arrange(desc(abs(.resid))) |>
  head(5)

The biggest misses are the Fiat 128 and Toyota Corolla, which get about 6.9 and 6.4 miles per gallon more than predicted. The Chrysler Imperial also beats its prediction by about 6.

Regression diagnostics#

Check: linearity, constant spread, normal residuals, influence.

ggplot(fit1_aug, aes(.fitted, .resid)) +
  geom_point(size = 2.5) +
  geom_hline(yintercept = 0, linetype = 'dashed') +
  geom_smooth(method = 'loess', formula = y ~ x,
              se = FALSE, colour = 'red') +
  labs(title = 'Residuals vs fitted',
       x = 'Fitted mpg', y = 'Residual') +
  theme_minimal(base_size = 14)

The red curve is U shaped: residuals are positive at low and high fitted values, and negative in the middle. That's a sign the true relationship is slightly curved, not perfectly straight.

fit1_aug |>
  select(.rownames, wt, mpg, .hat, .cooksd) |>
  slice_max(.cooksd, n = 3)

The Chrysler Imperial has the largest Cook's distance, about 0.53, followed by the Toyota Corolla and Fiat 128. None is above one, so no single car is dragging the line around badly.

fit1_quad <- lm(mpg ~ wt + I(wt^2), data = mtcars)
tidy(fit1_quad) |> select(term, estimate, p.value)
glance(fit1_quad)$r.squared

The squared term has a positive estimate of about 1.17 with a p-value of about 0.003, so the curve is real. The R-squared rises from 0.75 to about 0.82.

Predictions with predict()#

Pass new data with the same column names as the model.

new_cars <- tibble(wt = c(2, 3, 4.5))

predict(fit1, newdata = new_cars)
predict(fit1, newdata = new_cars, interval = 'confidence')
predict(fit1, newdata = new_cars, interval = 'prediction')

The predictions are about 26.6, 21.3 and 13.2 miles per gallon for cars of two, three and four and a half thousand pounds. For a three thousand pound car, the confidence interval for the average is about 20.1 to 22.4, but the prediction interval for a single car is much wider, about 14.9 to 27.6.

Pause and try: fit mpg ~ hp. What is the slope, the R-squared,

and the predicted mpg for a 150 horsepower car?

fit_hp <- lm(mpg ~ hp, data = mtcars)
tidy(fit_hp) |> select(term, estimate, p.value)
glance(fit_hp)$r.squared
predict(fit_hp, newdata = tibble(hp = 150))

The slope is about minus 0.068 miles per gallon per horsepower, so every extra hundred horsepower costs about 6.8 miles per gallon. The R-squared is about 0.60, lower than weight's 0.75, and a 150 horsepower car is predicted to do about 19.9 miles per gallon.

Multiple regression#

Many predictors: each slope 'holds the others fixed'.

fit2 <- lm(mpg ~ wt + hp, data = mtcars)
tidy(fit2, conf.int = TRUE) |>
  mutate(across(where(is.numeric), \(x) round(x, 3)))
glance(fit2) |> select(r.squared, adj.r.squared, sigma)

The weight slope is now about minus 3.88 and horsepower about minus 0.032, and both are significant. Weight's p-value rounds to zero, and horsepower's is about 0.001. Together they explain about 83 percent of the variation, with an adjusted R-squared of 0.815 and typical errors of about 2.6 miles per gallon.

Factors in a model#

A factor becomes dummy variables: each level vs a baseline.

tooth_fit <- lm(len ~ supp + dose, data = tooth)
tidy(tooth_fit) |> select(term, estimate, std.error, p.value)
levels(tooth$supp)

The intercept of about 12.46 is the predicted length for orange juice at the 0.5 dose. Switching to vitamin C, VC, lowers length by about 3.7. Dose 1 adds about 9.1 and dose 2 adds about 15.5 compared with dose 0.5. The supplement levels are OJ and VC, so OJ is the baseline.

Interactions#

Does the effect of one predictor depend on another? Use a * b.

tooth |>
  group_by(supp, dose) |>
  summarise(mean_len = mean(len), .groups = 'drop') |>
  ggplot(aes(dose, mean_len, colour = supp, group = supp)) +
  geom_point(size = 3) +
  geom_line(linewidth = 1) +
  labs(title = 'Orange juice helps most at low doses',
       x = 'Dose (mg/day)', y = 'Mean tooth length',
       colour = 'Supplement') +
  theme_minimal(base_size = 14)

The orange juice line sits well above vitamin C at doses 0.5 and 1, but at dose 2 the two lines meet. The lines are not parallel, which is exactly what an interaction looks like.

tooth_int <- lm(len ~ supp * dose, data = tooth)
tidy(tooth_int) |>
  select(term, estimate, p.value) |>
  mutate(across(where(is.numeric), \(x) round(x, 4)))

Now we have six terms. The main supplement effect of minus 5.25 applies at dose 0.5, and the interaction term suppVC with dose 2 is about plus 5.33, with a p-value of about 0.024. So at dose 2, the vitamin C penalty almost completely disappears, matching the plot.

Comparing models#

Nested models: anova() F-test. Any models on the same data: AIC.

anova(tooth_fit, tooth_int)

Adding the interaction reduces the residual sum of squares from about 820 to 712, with an F of about 4.1 and a p-value of about 0.02. So the interaction model fits significantly better.

AIC(tooth_fit, tooth_int)

fit3 <- lm(mpg ~ wt + hp + qsec + am, data = mtcars)
AIC(fit1, fit2, fit3)
anova(fit2, fit3)

For the tooth models, the interaction model has the lower AIC, about 333 versus 337, so AIC agrees with the F-test. For the car models, AIC drops from about 166 to 157 when we add horsepower, and the bigger model with quarter mile time and transmission gives about 154. But the F-test p-value is about 0.07, so those two extra predictors add only weak evidence.

Logistic regression#

For yes/no outcomes: model the log-odds with glm(family = binomial).

peng_sex <- peng |>
  mutate(male = as.integer(sex == 'male')) |>
  select(male, sex, species, body_mass_g, bill_depth_mm)

ggplot(peng_sex, aes(body_mass_g, male, colour = species)) +
  geom_jitter(height = 0.05, alpha = 0.6) +
  geom_smooth(method = 'glm', formula = y ~ x, se = FALSE,
              method.args = list(family = binomial)) +
  labs(title = 'Probability a penguin is male',
       x = 'Body mass (g)', y = 'Male (1) or female (0)') +
  theme_minimal(base_size = 14)

The points sit in two bands, females at zero and males at one. Each species gets an S shaped curve rising from zero to one, and the Gentoo curve sits far to the right, because Gentoo penguins are much heavier overall.

logit1 <- glm(male ~ body_mass_g + bill_depth_mm + species,
              data = peng_sex, family = binomial)
tidy(logit1) |>
  mutate(across(where(is.numeric), \(x) signif(x, 3)))

Body mass has a positive estimate of about 0.0063 per gram, and bill depth about 1.69 per millimetre, both with tiny p-values. So heavier penguins with deeper bills are more likely to be male. The species terms are negative, minus 0.27 for Chinstrap and minus 2.9 for Gentoo, but their p-values of about 0.59 and 0.08 mean they are not clearly different from Adelie once mass and bill depth are known.

exp(coef(logit1)) |> round(3)
exp(coef(logit1)['body_mass_g'] * 100)

The odds ratio for one millimetre of bill depth is about 5.4, so each extra millimetre multiplies the odds of being male by more than five. For body mass, one gram barely matters, an odds ratio of about 1.006, but one hundred grams multiplies the odds by about 1.88.

new_peng <- tibble(
  species       = c('Adelie', 'Adelie', 'Gentoo'),
  body_mass_g   = c(3400, 4200, 5000),
  bill_depth_mm = c(17.5, 19, 15.5)
)
predict(logit1, newdata = new_peng, type = 'response') |>
  round(3)

A light Adelie with a shallow bill has only about a 0.04 probability of being male. A heavy Adelie with a deep bill is almost certainly male at about 0.99, and our Gentoo comes out at about 0.65.

peng_sex <- peng_sex |>
  mutate(prob = predict(logit1, type = 'response'),
         pred = if_else(prob > 0.5, 'male', 'female'))

conf_mat <- table(predicted = peng_sex$pred,
                  actual = peng_sex$sex)
conf_mat
mean(peng_sex$pred == peng_sex$sex)

The confusion matrix shows 147 females and 153 males classified correctly, with only 18 females called male and 15 males called female. Overall accuracy is about 0.90, so ninety percent of penguins are classified correctly.

Train/test split#

Judge a model on data it has never seen.

set.seed(2025)
train_rows <- sample(nrow(peng_sex),
                     size = round(0.7 * nrow(peng_sex)))
train <- peng_sex[train_rows, ]
test  <- peng_sex[-train_rows, ]
c(train = nrow(train), test = nrow(test))

We have 233 penguins for training and 100 held back for testing.

logit_train <- glm(male ~ body_mass_g + bill_depth_mm + species,
                   data = train, family = binomial)

test_prob <- predict(logit_train, newdata = test,
                     type = 'response')
test_pred <- if_else(test_prob > 0.5, 'male', 'female')
table(predicted = test_pred, actual = test$sex)
mean(test_pred == test$sex)

On the hundred unseen penguins, the model gets 91 right, a test accuracy of 0.91. That's in line with the 0.90 we saw before, so the model generalises well. With a different seed it could easily come out a little lower.

Pause and try: in mtcars, model am (1 = manual) from wt.

What is the odds ratio for wt? P(manual) for a 3000 lb car?

logit_am <- glm(am ~ wt, data = mtcars, family = binomial)
exp(coef(logit_am)) |> round(4)
predict(logit_am, newdata = tibble(wt = 3),
        type = 'response')

The odds ratio for weight is about 0.018, so each extra thousand pounds cuts the odds of a manual gearbox by about 98 percent. A three thousand pound car has about a 0.49 probability of being manual, close to a coin flip.

Many models at once: nest + map + broom#

Fit the same model separately for each group.

by_species <- peng |>
  nest(data = -species)
by_species

We get a tibble with just three rows, one per species, and a list column called data holding each species' own tibble: 146 rows for Adelie, 119 for Gentoo and 68 for Chinstrap.

species_models <- by_species |>
  mutate(
    model = map(data, \(d) lm(body_mass_g ~ flipper_length_mm,
                                data = d)),
    coefs = map(model, tidy),
    stats = map(model, glance)
  )

species_models |>
  unnest(coefs) |>
  filter(term == 'flipper_length_mm') |>
  select(species, estimate, std.error, p.value)

Every species shows a clear positive slope. Each extra millimetre of flipper adds about 33 grams for Adelie, 35 for Chinstrap and 54 for Gentoo, all with very small p-values.

species_models |>
  unnest(stats) |>
  select(species, r.squared, sigma, nobs)

Flipper length explains about 22 percent of the mass variation within Adelie, about 41 percent within Chinstrap, and about 50 percent within Gentoo.

Project: what predicts penguin body mass?#

Explore, build models step by step, compare, check, predict,

and write a conclusion.

peng |>
  select(where(is.numeric), -year) |>
  cor() |>
  round(2)

Flipper length has the strongest correlation with body mass, about 0.87, followed by bill length at about 0.59. Bill depth is negatively correlated with mass, about minus 0.47, which smells like Simpson's paradox again.

m1 <- lm(body_mass_g ~ flipper_length_mm, data = peng)
m2 <- lm(body_mass_g ~ flipper_length_mm + species,
         data = peng)
m3 <- lm(body_mass_g ~ flipper_length_mm + species + sex,
         data = peng)
m4 <- lm(body_mass_g ~ flipper_length_mm + bill_length_mm +
           bill_depth_mm + species + sex, data = peng)

Nothing prints, but four model objects appear in the Environment pane.

models <- list(m1 = m1, m2 = m2, m3 = m3, m4 = m4)

map(models, glance) |>
  list_rbind(names_to = 'model') |>
  select(model, r.squared, adj.r.squared, sigma, AIC) |>
  mutate(across(where(is.numeric), \(x) round(x, 3)))

Each step improves the model. Adjusted R-squared climbs from 0.76 for flippers alone to about 0.87 with sex, and a little higher with the bills, while sigma falls from about 393 grams to about 287. AIC agrees, dropping steadily from about 4928 to about 4724, so model four is the best of the four.

anova(m3, m4)

The F-test comparing model three with model four has a p-value of about 4 times ten to the minus five, so the bill measurements add real information, even after flipper length, species and sex.

tidy(m4, conf.int = TRUE) |>
  select(term, estimate, conf.low, conf.high, p.value) |>
  mutate(across(where(is.numeric), \(x) round(x, 3)))

Holding everything else fixed, males are about 390 grams heavier than females, and each millimetre of flipper adds about 16 grams. Gentoo penguins are about 1015 grams heavier than Adelie penguins with the same measurements, and Chinstraps about 251 grams lighter. Bill depth adds about 67 grams per millimetre and bill length about 18. P-values shown as zero are simply smaller than 0.0005.

augment(m4) |>
  ggplot(aes(.fitted, .resid, colour = species)) +
  geom_point(alpha = 0.7) +
  geom_hline(yintercept = 0, linetype = 'dashed') +
  labs(title = 'Model 4: residuals vs fitted',
       x = 'Fitted body mass (g)', y = 'Residual (g)') +
  theme_minimal(base_size = 14)

The residuals form an even cloud around zero for all three species, mostly within about plus or minus 600 grams, with no curve and no funnel shape. The assumptions look reasonable.

set.seed(42)
idx <- sample(nrow(peng), size = round(0.7 * nrow(peng)))
peng_train <- peng[idx, ]
peng_test  <- peng[-idx, ]

rmse <- function(model) {
  pred <- predict(model, newdata = peng_test)
  sqrt(mean((peng_test$body_mass_g - pred)^2))
}

Nothing prints yet. We now have a training set, a test set, and a small rmse function ready to use.

formulas <- list(
  m1 = body_mass_g ~ flipper_length_mm,
  m3 = body_mass_g ~ flipper_length_mm + species + sex,
  m4 = body_mass_g ~ flipper_length_mm + bill_length_mm +
         bill_depth_mm + species + sex
)

map_dbl(formulas, \(f) rmse(lm(f, data = peng_train))) |>
  round(1)

On the unseen test penguins, the flipper only model is off by about 390 grams on average, model three by about 299, and model four by about 296. Adding species and sex is a big win, while the bill measurements help only a little.

new_bird <- tibble(flipper_length_mm = 215,
                   bill_length_mm = 47, bill_depth_mm = 15,
                   species = 'Gentoo', sex = 'male')
predict(m4, newdata = new_bird, interval = 'prediction') |>
  round(0)

Model four predicts about 5237 grams for this male Gentoo, with a 95 percent prediction interval from about 4667 to 5807 grams.

Conclusion#

  1. Flipper length is the strongest single predictor (r = 0.87).

  2. Species and sex matter a lot: males are ~390 g heavier and

    Gentoo ~1000 g heavier than Adelie at the same size.

  3. Model 4 explains ~87% of the variation; test error ~300 g.

Recap#

Describe -> distributions -> simulate -> CIs -> tests -> models.

lm for numbers, glm(binomial) for yes/no, broom to tidy,

diagnostics to check, test data to judge predictions.

Found this useful?

All lessons, notebooks and datasets here are free. If they helped you, a coffee keeps new lessons coming.