MAE, MSE, RMSE, R2 score and cross-validation · CAB107

Regression error metrics compare predictions with true values.

Metric	Definition
MAE	Mean of absolute errors
MSE	Mean of squared errors
RMSE	Square root of MSE, in the units of the target
R-squared	Proportion of target variance explained by the model

RMSE penalises large errors more than MAE because errors are squared before averaging, so one large error counts disproportionately.

Worked example: absolute errors 1, 1, 1, 1, 11. MAE = 15/5 = 3.0. MSE = (1 + 1 + 1 + 1 + 121)/5 = 25, so RMSE = 5.0. RMSE exceeds MAE whenever errors vary.

On held-out data R-squared can be negative: a value of -0.2 means the model fits worse than a baseline that always predicts the mean of the target.

k-fold cross-validation splits the data into k parts, trains on k-1 and validates on the remaining part, rotates, and averages the scores. Its purpose is to estimate how well a model generalises using several different splits.

from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X, y, cv=5, scoring='r2')
print(scores.mean())

Exam trap: R-squared is not a percentage of correct predictions, and cross-validation does not create new data.

Try two questions

Question 1 · Tier 2, Understand

Why does RMSE penalise large errors more than MAE does?

Errors are cubed before averaging, so large errors are magnified much more strongly
RMSE ignores the small errors completely and counts only the largest few errors in the set
Errors are squared before averaging, so one large error counts disproportionately
RMSE takes the median of the errors instead of their mean, so extremes are ignored

Show the answer

C. Squaring magnifies large residuals before the root brings the units back.

Why not A: RMSE squares errors, not cubes them.

Why not B: Small errors still contribute.

Why not D: RMSE averages squares and then takes the root.

Question 2 · Tier 4, Analyse

In 5-fold cross-validation on 100 rows, how many rows are used for training and for validation in each round?

20 for training and 80 for validation
100 for training and 20 for validation
50 for training and 50 for validation
80 for training and 20 for validation

Show the answer

D. Each fold holds 100 / 5 = 20 rows, and the remaining 80 train the model.

Why not A: This swaps the two roles.

Why not B: Training and validation rows cannot overlap or total more than 100.

Why not C: A 50-50 split would be 2-fold, not 5-fold.

Key points

Try two questions