# Multiple Regression

Multiple Regression

Chapter Goals

After completing this chapter, you should be able to:

§     Apply multiple regression analysis to business decision-making situations

§     Analyze and interpret the computer output for a multiple regression model

§     Perform a hypothesis test for all regression coefficients or for a subset of coefficients

§     Fit and interpret nonlinear regression models

§     Incorporate qualitative variables into the regression model by using dummy variables

§     Discuss model specification and analyze residuals

The Multiple Regression Model

Multiple Regression Equation

Multiple Regression Equation

Standard Multiple Regression Assumptions

§          The values  xi and the error terms  ?i are independent

§          The error terms are random variables with mean 0 and a constant variance, s2.

(The constant variance property is called

homoscedasticity)

Standard Multiple Regression Assumptions

§          The random error terms, ?i , are not correlated with one another, so that

§          It is not possible to find a set of numbers, c0, c1, . . . , ck, such that

(This is the property of no linear relation for

the Xj’s)

Example:
2 Independent Variables

§    A distributor of frozen desert pies wants to evaluate factors thought to influence demand

§   Dependent variable:       Pie sales (units per week)

§   Independent variables:   Price (in \$)

§    Data are collected for 15 weeks

Pie Sales Example

Sales = b0 + b1 (Price)

Estimating a Multiple Linear
Regression Equation

§     Excel will be used to generate the coefficients and measures of goodness of fit for multiple regression

§     Excel:

§    Tools / Data Analysis… / Regression

§     PHStat:

§    PHStat / Regression / Multiple Regression…

Multiple Regression Output

The Multiple Regression Equation

Coefficient of Determination, R2

§    Reports the proportion of total variation in  y explained by all  x  variables taken together

§    This is the ratio of the explained variability to total sample variability

Coefficient of Determination, R2

Estimation of Error Variance

§     Consider the population regression model

§     The unbiased estimate of the variance of the errors is

where

§     The square root of the variance, se , is called the standard error of the estimate

Standard Error, se

§    R2 never decreases when a new  X  variable is added to the model, even if the new variable is not an important predictor variable

§   This can be a disadvantage when comparing models

§    What is the net effect of adding a new variable?

§   We lose a degree of freedom when a new  X variable is added

§   Did the new  X  variable add enough explanatory power to offset the loss of one degree of freedom?

§     Used to correct for the fact that adding non-relevant independent variables will still reduce the error sum of squares

(where n = sample size, K = number of independent variables)

§    Adjusted R2 provides a better comparison between multiple regression models with different numbers of independent variables

§    Penalize excessive use of unimportant independent variables

§ Smaller than R2

Coefficient of Multiple Correlation

§     The coefficient of multiple correlation is the correlation between the predicted value and the observed value of the dependent variable

§     Is the square root of the multiple coefficient of determination

§     Used as another measure of the strength of the linear relationship between the dependent variable and the independent variables

§     Comparable to the correlation between Y and X in simple regression

Evaluating Individual
Regression Coefficients

§    Use t-tests for individual coefficients

§    Shows if a specific independent variable is conditionally important

§    Hypotheses:

§   H0: ?j = 0 (no linear relationship)

§   H1: ?j ? 0  (linear relationship does exist

between xj and y)

Evaluating Individual
Regression Coefficients

H0: ?j = 0 (no linear relationship)

H1: ?j ? 0  (linear relationship does exist

between xi and y)

Test Statistic:

(df = n – k – 1)

Evaluating Individual
Regression Coefficients

H0: ?j = 0

H1: ?j ¹ 0

Confidence Interval Estimate
for the Slope

Confidence Interval Estimate
for the Slope

Test on All Coefficients

§    F-Test for Overall Significance of the Model

§    Shows if there is a linear relationship between all of the  X  variables considered together and  Y

§    Use F test statistic

§    Hypotheses:

H0: ?1 = ?2 = … = ?k = 0  (no linear relationship)

H1: at least one  ?i ? 0   (at least one independent

variable affects Y)

F-Test for Overall Significance

§    Test statistic:

where F has   k  (numerator) and

(n – K – 1)  (denominator)

degrees of freedom

§    The decision rule is

F-Test for Overall Significance

F-Test for Overall Significance

H0: ?1 = ?2 = 0

H1: ?1 and ?2 not both zero

a = .05

df1= 2      df2 = 12

Tests on a Subset of Regression Coefficients

§     Consider a multiple regression model involving variables  xj and  zj , and the null hypothesis that the  z variable coefficients are all zero:

Tests on a Subset of Regression Coefficients

§     Goal: compare the error sum of squares for the complete model with the error sum of squares for the restricted model

§    First run a regression for the complete model and obtain SSE

§    Next run a restricted regression that excludes the  z  variables (the number of variables excluded is  r)  and obtain the restricted error sum of squares  SSE(r)

§    Compute the  F  statistic and apply the decision rule for a significance level  a

Prediction

§     Given a population regression model

§     then given a new observation of a data point

(x1,n+1, x 2,n+1, . . . , x K,n+1)

the best linear unbiased forecast of  yn+1 is

§      It is risky to forecast for new X values outside the range of the data used to estimate the model coefficients, because we do not have data to support that the linear model extends beyond the observed range.

Using The Equation to Make Predictions

Predictions in PHStat

§     PHStat | regression | multiple regression …

Predictions in PHStat

Input values

Residuals in Multiple Regression

Nonlinear Regression Models

§    The relationship between the dependent variable and an independent variable may not be linear

§    Can review the scatter diagram to check for  non-linear relationships

§ The second independent variable is the square of the first variable

§     where:

?0 = Y intercept

?1 = regression coefficient for linear effect of X on Y

?2 = regression coefficient for quadratic effect on Y

?i = random error in Y for observation i

Linear vs. Nonlinear Fit

§    Compare the linear regression estimate

§    Hypotheses

§                        (The quadratic term does not improve the model)

§                        (The quadratic term improves the model)

Hypotheses

§                        (The quadratic term does not improve the model)

§                        (The quadratic term improves the model)

§     The test statistic is

Compare  R2 from simple regression to

§    If R2 from the quadratic model is larger than R2 from the simple model, then the quadratic model is a better model

§    Purity increases as filter time increases:

§     Simple regression results:

y = -11.283 + 5.985 Time

The Log Transformation

§     Original multiplicative model

§     Transformed multiplicative model

Interpretation of coefficients

For the multiplicative model:

§    When both dependent and independent variables are logged:

§    The coefficient of the independent variable  Xk can be interpreted as

a 1 percent change in  Xk leads to an estimated  bk percentage change in the average value of  Y

§    bk is the elasticity of Y with respect to a change in Xk

Dummy Variables

§    A dummy variable is a categorical independent variable with two levels:

§   yes or no, on or off, male or female

§   recorded as 0 or 1

§    Regression intercepts are different if the variable is significant

§    Assumes equal slopes for other variables

§    If more than two levels, the number of dummy variables needed is (number of levels – 1)

Dummy Variable Example

Dummy Variable Example

Interpreting the
Dummy Variable Coefficient

Interaction Between Explanatory Variables

§    Hypothesizes interaction between pairs of  x variables

§    Response to one  x  variable may vary at different levels of another  x  variable

§    Contains two-way cross product terms

§

Effect of Interaction

§    Given:

§ Without interaction term, effect of X1 on Y  is measured by ?1

§ With interaction term, effect of X1 on Y  is measured by ?1 + ?3 X2

§    Effect changes as X2 changes

Interaction Example

Significance of Interaction Term

§     The coefficient  b3 is an estimate of the difference in the coefficient of  x1 when  x2 = 1  compared to when  x2 = 0

§     The t statistic for  b3 can be used to test the hypothesis

§     If we reject the null hypothesis we conclude that there is a difference in the slope coefficient for the two subgroups

Multiple Regression Assumptions

Assumptions:

§    The errors are normally distributed

§    Errors have a constant variance

§    The model errors are independent

Analysis of Residuals
in Multiple Regression

§    These residual plots are used in multiple regression:

§ Residuals vs. yi

§   Residuals vs. x1i

§   Residuals vs. x2i

§ Residuals vs. time (if time series data)

Chapter Summary

§     Developed the multiple regression model

§     Tested the significance of the multiple regression model

§     Discussed adjusted R2 ( R2 )

§     Tested individual regression coefficients

§     Tested portions of the regression model

§     Used quadratic terms and log transformations in regression models

§     Used dummy variables

§     Evaluated interaction effects

§     Discussed using residual plots to check model assumptions