Previously on Linear Regression…

• The Simple Linear Regression with the equation we fit into the data given as \[ \hat{y} = b_0 + b_1 x + e \] where \(y\) is the predicted/estimated outcome, \(x\) is the predictor, \(b_0\) is the intercept, and \(b_1\) is the slope.

• The Residuals where it is the difference between the observed (\(y_i\)) values and estimated/predicted \(\hat{y}_i\) values. \[ e_i = y_i - \hat{y}_i \]

• The Correlation which is a metric describing the strength of the linear association of two numerical variables where it can have values between -1 (perfect negative) and +1 (perfect positive).

Previously on Least Squares…

Least-Squares Example Visualization: Shown here is some data (orange dots) and the best fit linear model (red line) y = 5.37 + 0.62*x . You can try this least-squares regression interactive demo to visualize on how it works.

Linear Regression Contininued…

In this lecture, we will learn about:

• Weaknesses of the linear regression model - involving outliers.

• Linear regression with categorical predictor.

• Linear regression with multiple predictors.

Outliers in Linear Regression

Three plots, each with a least squares line and corresponding residual plot. Each dataset has at least one outlier. Image Source: OpenIntro: IMS Section 7.3.

Outliers in Linear Regression

Types of outliers.

• A point (or a group of points) that stands out from the rest of the data is called an outlier.

• Outliers that fall horizontally away from the center of the cloud of points are called leverage points.

• Outliers that influence on the slope of the line are called influential points.

We must be cautious on removing outliers in our modeling. Sometimes outliers are interesting cases that might be worth investigating and it might even make amodel much better.

Try out this least-squares regression interactive demo to play around with outliers in least squares regression.

Categorical Predictors with Two Levels

Example:

Total auction prices for the video game Mario Kart, divided into used (\(x = 0\)) and new (\(x = 1\)) condition games. The least squares regression line is also shown.

Categorical Predictors with Two Levels

Interpreting the slope and intercept for two-level categorical predictor:

• The estimated intercept is the value of the outcome variable for the first category (i.e., the category corresponding to an indicator value of 0).

• The estimated slope is the average change in the outcome variable between the two categories.

example:

\[\widehat{\texttt{price}} = 42.87 + 10.9 \times \texttt{condnew}\]

The average selling price of a used version of the game is \(42.9\). The slope indicates that, on average, new games sell for about $10.9 more than used games.

Linear Regression Model with Multiple Predictors

Since the beginning of introducing the Simple Linear Regression (SLR) and the least squares regression, we have been using one predictor and one outcome.

A multiple regression model (MLR) is a linear model with many predictors. In general, we write the model as

\[\hat{y} = b_0 + b_1 x_1 + b_2 x_2 + \cdots + b_k x_k\]

when there are \(k\) predictors. We always calculate \(b_i\) using R.

Linear Regression Model with Multiple Predictors

Key points:

• Multicolinearity is a common issue and it complicates model estimation. This means that if two or more independent variables are correlated (or collinear), it can cause problems when fitting the model (becomes very sensitive to small changes) and interpreting the results. Collinear explanatory variables reduces the precision of the estimated coefficients, which weakens the statistical power of your regression model.

• However, MLR can be a good model for a more complicated data set that needs more than one explanatory variable to model and interpret a more complicated system. The interpretation of the coefficients (\(b_0\), \(b_1\), \(\cdots\), \(b_k\)): The mean change in the dependent variable for each 1 unit change in the explanatory variable when you hold all of the other independent variables constant.

• We will revisit the Multiple Linear Regression (MLM) later in the course after we learn about probability, which can help us decide which explanatory variables to keep.

R-Squared vs Adjusted R-Squared

The R-squared is the a measure of how well the simple linear regression fits the data but this is a biased estimate of the amount of variability explained by the model when applied to model with more than one predictor.

\[R^2 = 1 - \frac{SSE}{SST}\]

The adjusted R-squared describes the power of the regression model that contain different numbers of predictors.

This is computed as

\[ \begin{aligned} R_{adj}^{2} &= 1 - \frac{s_{\text{residuals}}^2 / (n-k-1)} {s_{\text{outcome}}^2 / (n-1)} \\ &= 1 - \frac{s_{\text{residuals}}^2}{s_{\text{outcome}}^2} \times \frac{n-1}{n-k-1} \end{aligned} \]

where \(n\) is the number of observations used to fit the model and \(k\) is the number of predictor variables in the model.

The term \(k-1\) is called the degrees of freedom.

R-Squared vs Adjusted R-Squared

Key Points:

• Both \(R^2\) and \(R_{adj}^{2}\) tells you the percentage of variation in the data explained by the model.

• Using either \(R^2\) or \(R_{adj}^{2}\) depends on the number of samples and the number of predictors.

• If you have enough observations and small number of predictors (degrees of freedom), the difference between \(R^2\) and \(R_{adj}^{2}\) are small. If you have a lot of predictors, it is common to use the \(R_{adj}^{2}\).

Additional Thoughts:

• \(R^2\) and \(R_{adj}^{2}\) does not tell you if the models error is either homoscedastic or heteroscedasitic. You can look at the residual scatterplots for checking homoscedasticity or heteroscedasiticity

• \(R^2\) and \(R_{adj}^{2}\) does not tell you if the two or more explanatory variables are collinear. You can use the correlation coefficient to check if two ro more variables are collinear.

Summary

In this lecture, we talked about the following:

• Types of outliers in the data and how it can affect the least squares regression model.

• Categorical Predictors with Two Levels in a simple regression model and its interpretations.

• A short exposure to the Multiple Linear Regression (MLR) model.

• The difference between R-squared and the Adjusted R-squared.

Looking ahead: In the next few weeks, we will talk about using the concept of probability to know if the sample statistics (e.g. mean, slope, intercept, coefficients, etc) are likely not caused by chance for a given statistical significance level.

In the next lecture, we will talk about:

• Introduction to probability, basic probability rules, and basic probability theorems.