Previously on Linear Regression…

A linear model is written as

\[ y = \beta_0 + \beta_1 x + e \]

where \(y\) is the outcome, \(x\) is the predictor, \(\beta_0\) is the intercept, and \(\beta_1\) is the slope. The notation \(e\) is the model’s error.

Notation:

• Population Parameters: \(\beta_0\) and \(\beta_1\)

• Sample statistics (point estimates for the parameters): \(b_0\) and \(b_1\)

• Estimated/Predicted outcome: \(\hat{y} = b_0 + b_1 x\)

Residuals are the leftover variation in the data after accounting for the model fit:

Data = Fit + Residual

A Residual of the \(i^{th}\) observation \((x_i,y_i)\) is the difference between the observed (\(y_i\)) and estimated/predicted \(\hat{y}_i\).

\[ e_i = y_i - \hat{y}_i \]

Least Squares Regression

In this lecture, we will learn about:

• Objectively making a linear regression model that best fit the data using ordinary least squares method.

• Least-squares derivation.

• Residual analysis describing the strength of the linear regression fit.

Examples of Best Fit Linear Models on Data

Idealized Examples of Residual Scatter Plots

Terms:

• Homoscedastic residuals \(\longrightarrow\) constant variance.

• Heteroscedastic residuals \(\longrightarrow\) non-constant variance.

Ordinary Least Squares Assumptions

1. The data is randomly sampled independent observations.

2. The distribution of the residuals is normally distributed and the residuals exhibits homoscedasticity.

3. The independent variables are uncorrelated with the error term.

4. The regression model is linear in the coefficients and the error term.

5. The independent variables should not be collinear - or correlated to each other.

Sum of Squared Error (SSE) and the Total Sum of Squares (TSS)

The Sum of Squared Error (SSE) is a metric of left-over variability in the \(y\) values if we know \(x\).

\[ SSE = \sum_{i=1}^n (e_i)^2 \]

The Total Sum of Squares (SST) is a metric measure the variability in the \(y\) values by how far they tend to fall from their mean, \(\bar{y}\).

\[ SST = \sum_{i=1}^n (y_i - \bar{y})^2 \]

where \(\bar{y} = \frac{1}{n} \sum_{i-1}^n y_i\), and \(n\) is the number of observations.

Minimizing the SSE

To find the best linear fit, we minimize the SSE.

\[ SSE = \sum_{i=1}^n (e_i)^2 = \sum_{i=1}^n (y_i - \hat{y_i})^2 \] Plugging-in the linear equation \(\hat{y_i} = b_0 + b_1 x\), we have

\[ SSE = \sum_{i=1}^n (y_i - (b_0 + b_1 x)^2. \] Minimizing the above equation over all possible values of \(b_0\) and \(b_1\) is a calculus problem. Take the derivative of SSE with respect to \(b_1\), set it equal to zero, and solve for \(b_1\).

Long story short,

\[ \begin{aligned} b_1 = & \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} \\ b_0 = & \bar{y} - b_1 \bar{x} \end{aligned} \] where \(\bar{x}\) and \(\bar{y}\) are the mean of \(x\) and \(y\) respectively.

The Estimated Slope and Intercept

We can rewrite the slope as follows \[ \begin{aligned} b_1 & = \frac{\sum_{i=1}^n (x_i - \bar{x})(y_i - \bar{y})}{\sum_{i=1}^n (x_i - \bar{x})^2} \\ & = \frac{\sqrt{\frac{1}{n-1} \sum_{i=1}^n (y_i - \bar{y})^2}}{\sqrt{\frac{1}{n-1} \sum_{i=1}^n (x_i - \bar{x})^2}} \frac{\sum_{i=1}^n{(x_i - \bar{x})(y_i - \bar{y})}}{\sqrt{\sum_{i=1}^n (x_i - \bar{x})^2 \sum_{i=1}^n (y_i - \bar{y})^2}} \\ b_1 & = \frac{s_y}{s_x} r \end{aligned} \] where \(s_y\) and \(s_x\) are the standard deviations of \(x\) and \(y\) respectively, and \(r\) is the pearson correlation coefficient of \(x\) and \(y\).

Least-Squares Example Visualization

Why do we care about the math when we can just use R progamming language?

• Using least-squares regression is typically the most helpful when dealing with data.

• The slope in terms of the standard deviations and correlation helps us interpret it more precisely.

• The math helps us understand more thoroughly of how linear regression works.

• The ratio \(\frac{s_y}{s_x}\) in the least squares slope \(b_1\) tells us the average change in the predicted values of the response variable when the explanatory variable increases by 1 unit.

\[ b_1 = \frac{s_y}{s_x} r \]

Measuring the Strength of the Linear Fit

The coefficient of determination can then be calculated as

\[ R^2 = \frac{SST - SSE}{SST} = 1 - \frac{SSE}{SST} \]

where

\[ SSE = \sum_{i=1}^n (e_i)^2 \hspace{10px} \text{ and } \hspace{10px} SST = \sum_{i=1}^n (y_i - \bar{y})^2. \] The range of \(R^2\) is from 0 to 1. \(R^2\) is the a measure of how well the linear regression fits the data.

Interpretation:

• Value of 0 means 0% of the variance in the data is captured by the model. (bad and not a “good” fit)

• Value of 1 means 100% of the variance in the data is captured by the model. (over-fitting and impacts the ability for the model to generalize about the population)

• Ideally, you want an \(R^2\) value close to 1.

In the case for a linear model with one predictor and one outcome, the relationship between the correlation and the coefficient of determination is \(R^2 = r^2\).

Summary

In this lecture, we talked about the following:

• Evaluating linear models by using residual scatter plots with idealized examples of baised vs unbiased patterns, and heteroscedastic errors vs homoscedastic errors.

• The derivation of the ordinary least squares method to find the best fit linear model into data.

• The coefficient of determination \(R^2\) for evaluating the linear model.

You can try this least squares regression interactive demo to visualize on how it works.

In the next lecture, we will talk about:

• Weaknesses of the linear regression model - involving outliers.

• Linear regression with categorical predictors.

• Linear regression with multiple predictors.