Linear and polynomial regression

Linear regression models a continuous target as a linear function of the inputs, for example y = 3x + 2. The coefficient 3 is the change in y for a one-unit increase in x, and the intercept 2 is the value of y when x is zero. The coefficients are usually found by minimising the sum of squared errors (ordinary least squares).

Polynomial regression adds powers of the input, such as x and x squared, to fit curves. It is still a linear model because it is linear in its coefficients, even though it uses powers of the input.

from sklearn.linear_model import LinearRegression
from sklearn.preprocessing import PolynomialFeatures
X2 = PolynomialFeatures(degree=2).fit_transform(X)
model = LinearRegression().fit(X2, y)

Raising the degree increases flexibility. If it is pushed far beyond what the data supports, the curve fits the training points closely but generalises badly: overfitting, a high-variance problem. A degree-9 polynomial fitted to 10 points can reach zero training error yet a very large test error. Too low a degree gives underfitting, a high-bias problem. Regularisation or a lower degree reduces overfitting.

Exam trap: zero training error is a warning sign, not a success. Judge a model by its error on unseen data.