13.2 Multiple Regression
LEARNING OBJECTIVES
- Develop a multiple regression model.
- Use a multiple regression model to predict values of the dependent variable.
- Interpret the partial regression coefficients.
Previously, we learned about simple linear regression, which models the linear relationship between one independent variable [latex]x[/latex] and one dependent variable [latex]y[/latex]. The equation for the regression line is:
[latex]\begin{eqnarray*} \hat{y} & = & b_0+b_1x \\ \\ \hat{y} & = & \mbox{predicted value of } y \\ x & = & \mbox{value of the independent variable} \\ b_0 & = &y\mbox{-interecept of the line} \\ b_1 & = & \mbox{slope of the line} \end{eqnarray*}[/latex]
Multiple regression is an extension of simple linear regression where there is still only one dependent variable [latex]y[/latex] but two or more dependent variables [latex]x_1,x_2,\ldots,x_k[/latex]. Multiple regression is motivated by scenarios where many independent variables may be simultaneously connected to a dependent variable. For example, the price of product is related to demand for the product, the time of year, and the competition’s price.
The equation for the multiple regression model is:
[latex]\begin{eqnarray*} \hat{y} & = & b_0+b_1x_1+b_2x_2+\cdots+b_kx_k \end{eqnarray*}[/latex]
where [latex]\hat{y}[/latex] is the predicted value of [latex]y[/latex], [latex]x_1,x_2,\ldots,x_k[/latex] are the independent variables, [latex]b_0,b_1,\ldots,b_k[/latex] are the regression coefficients, and [latex]k[/latex] is the number of independent variables.
We will use Excel to generate the values of the regression coefficients. However, unlike simple linear regression where we used individual, built-in functions to find the slope and [latex]y[/latex]-intercept, we use a regression summary table to generate the values of the regression coefficients. As we will see, the regression summary table contains lots of information relating to the multiple regression model. For now, we will use the regression summary table to find the regression coefficients to create the multiple regression model. In later sections, we will learn about some of the other information contained on the regression summary table.
USING EXCEL TO CREATE A REGRESSION SUMMARY TABLE
In order to create a regression summary table, we need to use the Analysis ToolPak. Follow these instructions to add the Analysis ToolPak.
- Enter the data into an Excel worksheet.
- Go to the Data tab and click on Data Analysis. If you do not see Data Analysis in the Data tab, you will need to install the Analysis ToolPak.
- In the Data Analysis window, select Regression and then click OK.
- In the Input Y Range, enter the cell range for the [latex]y[/latex] (dependent variable) data.
- In the Input X Range, enter the cell range for the [latex]x[/latex] (independent variables) data.
- Click on Labels in first row if the you included the column headings in the input range.
- From the Output Options, select the location where you want the output to appear. The default is a new worksheet.
- Click OK. Excel will then generate a regression summary table.
NOTES
- For the Input X Range, the data for the independent variables must all be together. That is, the columns (or rows) containing the data for the independent variables must all be consecutive. If the column (or row) containing data for the dependent variable is in between two columns (or rows) containing independent variables, copy the dependent variable column and paste the dependent variable column at the beginning or end of the columns (or rows) of data. Make sure to delete the original dependent variable column after placing a copy at the beginning or end of the data.
- There are several other options available in Regression input window, such as for confidence intervals or information about residuals. We will not need any of this other information, so leave everything else unchecked.
- This website provides a detailed explanation of the information contained on the regression summary table.
EXAMPLE
The human resources department at a large company wants to develop a model to predict an employee’s job satisfaction from the number of hours of unpaid work per week the employee does, the employee’s age, and the employee’s income. A sample of 25 employees at the company is taken and the data is recorded in the table below. The employee’s income is recorded in $1000s and the job satisfaction score is out of 10, with higher values indicating greater job satisfaction. Develop a multiple regression model to predict the job satisfaction score from the other variables.
| Job Satisfaction | Hours of Unpaid Work per Week | Age | Income ($1000s) |
| 4 | 3 | 23 | 60 |
| 5 | 8 | 32 | 114 |
| 2 | 9 | 28 | 45 |
| 6 | 4 | 60 | 187 |
| 7 | 3 | 62 | 175 |
| 8 | 1 | 43 | 125 |
| 7 | 6 | 60 | 93 |
| 3 | 3 | 37 | 57 |
| 5 | 2 | 24 | 47 |
| 5 | 5 | 64 | 128 |
| 7 | 2 | 28 | 66 |
| 8 | 1 | 66 | 146 |
| 5 | 7 | 35 | 89 |
| 2 | 5 | 37 | 56 |
| 4 | 0 | 59 | 65 |
| 6 | 2 | 32 | 95 |
| 5 | 6 | 76 | 82 |
| 7 | 5 | 25 | 90 |
| 9 | 0 | 55 | 137 |
| 8 | 3 | 34 | 91 |
| 7 | 5 | 54 | 184 |
| 9 | 1 | 57 | 60 |
| 7 | 0 | 68 | 39 |
| 10 | 2 | 66 | 187 |
| 5 | 0 | 50 | 49 |
Solution:
There are three independent variables: hours of unpaid work per week, age, and income ($1000s). Let [latex]x_1[/latex] be the hours of unpaid work per week, let [latex]x_2[/latex] be age, and let [latex]x_3[/latex] be income ($1000s). The regression summary table generated by Excel is shown below:
| SUMMARY OUTPUT | ||||||
| Regression Statistics | ||||||
| Multiple R | 0.711779225 | |||||
| R Square | 0.506629665 | |||||
| Adjusted R Square | 0.436148189 | |||||
| Standard Error | 1.585212784 | |||||
| Observations | 25 | |||||
| ANOVA | ||||||
| df | SS | MS | F | Significance F | ||
| Regression | 3 | 54.189109 | 18.06303633 | 7.18812504 | 0.001683189 | |
| Residual | 21 | 52.770891 | 2.512899571 | |||
| Total | 24 | 106.96 | ||||
| Coefficients | Standard Error | t Stat | P-value | Lower 95% | Upper 95% | |
| Intercept | 4.799258185 | 1.197185164 | 4.008785216 | 0.00063622 | 2.309575344 | 7.288941027 |
| Hours of Unpaid Work per Week | -0.38184722 | 0.130750479 | -2.9204269 | 0.008177146 | -0.65375772 | -0.10993671 |
| Age | 0.004555815 | 0.022855709 | 0.199329423 | 0.843922453 | -0.04297523 | 0.052086864 |
| Income ($1000s) | 0.023250418 | 0.007610353 | 3.055103771 | 0.006012895 | 0.007423823 | 0.039077013 |
The coefficients for the multiple regression model are in the Coefficients column in the bottom part of the table. The value of [latex]b_0[/latex] is in the Intercept row, so [latex]b_0=4.7993[/latex]. The value of [latex]b_1[/latex], the coefficient for [latex]x_1[/latex], is in the Hours of Unpaid Work per Week row, so [latex]b_1=-0.3818[/latex]. The value of [latex]b_2[/latex], the coefficient of [latex]x_2[/latex], is in the Age row, so [latex]b_2=0.0046[/latex]. The value of [latex]b_3[/latex], the coefficient of [latex]x_3[/latex], is in the Income row, so [latex]b_3=0.0233[/latex].
The multiple regression equation is
[latex]\begin{eqnarray*} \hat{y} & = & 4.7993-0.3818x_1+0.0046x_2+0.0233x_3 \\ \\ \hat{y} & = & \mbox{predicted job satisfaction score} \\ x_1 & = & \mbox{hours of unpaid work per week} \\ x_2 & = & \mbox{age} \\ x_3 & = & \mbox{income (\$1000s)}\end{eqnarray*}[/latex]
NOTES
- When writing down the multiple regression equation, remember to define what the variables represent in the context of the question. That is, state what [latex]\hat{y}[/latex] and the independent variables represent in relation to the question.
- A couple of the columns on the right side of the regression summary table generated by Excel where deleted in order to fit the table onto the page. These columns are not necessary for the work we will be doing.
Watch this video: Business Excel Business Analytics #50: Introduction to Multiple Regression, Data Analysis Regression by ExcelIsFun [13:33]
Regression Coefficients
Recall that the slope [latex]b_1[/latex] in the simple linear regression model [latex]\hat{y}=b_0+b_1x[/latex] tells us how the dependent variable [latex]y[/latex] changes for a single unit increase in the independent variable [latex]x[/latex]. In a similar way, each regression coefficient [latex]b_i[/latex] represents the change (increase or decrease) in the dependent variable for a one unit increase in the corresponding independent variable [latex]x_i[/latex], while all the other variables are held constant. When interpreting a regression coefficient, it is important to be specific to the question, using the actual names of the variables and correct units.
EXAMPLE
The human resources department at a large company wants to develop a model to predict an employee’s job satisfaction from the number of hours of unpaid work per week the employee does, the employee’s age, and the employee’s income. A sample of 25 employees at the company is taken and the data is recorded in the table below. The employee’s income is recorded in $1000s and the job satisfaction score is out of 10, with higher values indicating greater job satisfaction.
| Job Satisfaction | Hours of Unpaid Work per Week | Age | Income ($1000s) |
| 4 | 3 | 23 | 60 |
| 5 | 8 | 32 | 114 |
| 2 | 9 | 28 | 45 |
| 6 | 4 | 60 | 187 |
| 7 | 3 | 62 | 175 |
| 8 | 1 | 43 | 125 |
| 7 | 6 | 60 | 93 |
| 3 | 3 | 37 | 57 |
| 5 | 2 | 24 | 47 |
| 5 | 5 | 64 | 128 |
| 7 | 2 | 28 | 66 |
| 8 | 1 | 66 | 146 |
| 5 | 7 | 35 | 89 |
| 2 | 5 | 37 | 56 |
| 4 | 0 | 59 | 65 |
| 6 | 2 | 32 | 95 |
| 5 | 6 | 76 | 82 |
| 7 | 5 | 25 | 90 |
| 9 | 0 | 55 | 137 |
| 8 | 3 | 34 | 91 |
| 7 | 5 | 54 | 184 |
| 9 | 1 | 57 | 60 |
| 7 | 0 | 68 | 39 |
| 10 | 2 | 66 | 187 |
| 5 | 0 | 50 | 49 |
Previously, we found the multiple regression equation to predict the job satisfaction score from the other variables:
[latex]\begin{eqnarray*} \hat{y} & = & 4.7993-0.3818x_1+0.0046x_2+0.0233x_3 \\ \\ \hat{y} & = & \mbox{predicted job satisfaction score} \\ x_1 & = & \mbox{hours of unpaid work per week} \\ x_2 & = & \mbox{age} \\ x_3 & = & \mbox{income (\$1000s)}\end{eqnarray*}[/latex]
- Interpret the regression coefficient for hours of unpaid work per week.
- Interpret the regression coefficient for age.
- Interpret the regression coefficient for income.
Solution:
- [latex]b_1=-0.3818[/latex]. Interpretation: For a one hour increase in the hours of unpaid work per week, the job satisfaction score decreases by 0.3818, while the other variables are held constant.
- [latex]b_2=0.0046[/latex]. Interpretation: For a one year increase in the age of the employee, the job satisfaction score increases by 0.0046, while the other variables are held constant.
- [latex]b_3=0.0233[/latex]. Interpretation: For a $1000 increase in income, the job satisfaction score increases by 0.0233, while the other variables are held constant.
NOTES
- Remember to include “while the other variables are held constant” with the interpretation of each regression coefficient. We can only talk about how the change in one independent variable affects the dependent variable, so the values of the other variables must be kept fixed.
- When writing down the interpretation of each regression coefficient, remember to be specific to the question using the actual names of the independent and dependent variables and appropriate units.
- Each regression coefficient has the same units as the dependent variable.
- Income is measured in $1000s, so a one unit increase in income actually corresponds to a $1000 increase in income.
Making Predictions with a Multiple Regression Model
As with simple linear regression, a multiple regression model can be used to make predictions about the dependent variable from specific values of the independent variables. To make a prediction, substitute the corresponding values of the independent variables into the multiple regression equation and calculate out the value of [latex]\hat{y}[/latex]. Watch out for the units of measurement for each variable when using the multiple regression equation—the units of the values entered into the independent variable [latex]x_i[/latex] in the multiple regression equation must match the units of the independent variable in the sample data.
EXAMPLE
The human resources department at a large company wants to develop a model to predict an employee’s job satisfaction from the number of hours of unpaid work per week the employee does, the employee’s age, and the employee’s income. A sample of 25 employees at the company is taken and the data is recorded in the table below. The employee’s income is recorded in $1000s and the job satisfaction score is out of 10, with higher values indicating greater job satisfaction.
| Job Satisfaction | Hours of Unpaid Work per Week | Age | Income ($1000s) |
| 4 | 3 | 23 | 60 |
| 5 | 8 | 32 | 114 |
| 2 | 9 | 28 | 45 |
| 6 | 4 | 60 | 187 |
| 7 | 3 | 62 | 175 |
| 8 | 1 | 43 | 125 |
| 7 | 6 | 60 | 93 |
| 3 | 3 | 37 | 57 |
| 5 | 2 | 24 | 47 |
| 5 | 5 | 64 | 128 |
| 7 | 2 | 28 | 66 |
| 8 | 1 | 66 | 146 |
| 5 | 7 | 35 | 89 |
| 2 | 5 | 37 | 56 |
| 4 | 0 | 59 | 65 |
| 6 | 2 | 32 | 95 |
| 5 | 6 | 76 | 82 |
| 7 | 5 | 25 | 90 |
| 9 | 0 | 55 | 137 |
| 8 | 3 | 34 | 91 |
| 7 | 5 | 54 | 184 |
| 9 | 1 | 57 | 60 |
| 7 | 0 | 68 | 39 |
| 10 | 2 | 66 | 187 |
| 5 | 0 | 50 | 49 |
Previously, we found the multiple regression equation to predict the job satisfaction score from the other variables:
[latex]\begin{eqnarray*} \hat{y} & = & 4.7993-0.3818x_1+0.0046x_2+0.0233x_3 \\ \\ \hat{y} & = & \mbox{predicted job satisfaction score} \\ x_1 & = & \mbox{hours of unpaid work per week} \\ x_2 & = & \mbox{age} \\ x_3 & = & \mbox{income (\$1000s)}\end{eqnarray*}[/latex]
Predict the job satisfaction score for a 40-year old employee who works two hours of unpaid work per week and has an income of $75,000.
Solution:
The values of the independent variables we need to enter into the multiple regression model are [latex]x_1=2[/latex], [latex]x_2=40[/latex], and [latex]x_3=75[/latex]:
[latex]\begin{eqnarray*}\hat{y} & = & 4.7993-0.3818x_1+0.0046x_2+0.0233x_3 \\ & = & 4.7993-0.3818 \times 2+0.0046 \times 40+0.0233 \times 75 \\ & = & 5.96 \end{eqnarray*}[/latex]
The predicted job satisfaction score for a 40-year old employee who works two hours of unpaid work per week and has an income of $75,000 is 5.96.
NOTES
- In the sample data, income is measured in $1000s. So an income of $75,000 would be recorded as 75 in the sample data. So, we enter 75 for the value of [latex]x_3[/latex]
- To get the most accurate answer, use Excel to calculate out the value of [latex]\hat{y}[/latex], clicking on the corresponding cells containing the values of the coefficients in the regression summary sheet.
Assumptions about the Multiple Regression Model
The multiple regression model given above is the model we create from sample data—a sample is taken from the population and the sample data is used to find the regression coefficients in the model. So the regression coefficients, [latex]b_0, b_1, \ldots, b_k[/latex], are estimates of the corresponding population parameters for the regression coefficients, [latex]\beta_0, \beta_1, \ldots, \beta_k[/latex].
The population model for the multiple regression equation is
[latex]\begin{eqnarray*} y & = & \beta_0+\beta_1x_1+\beta_2x_2+\cdots+\beta_kx_k +\epsilon \end{eqnarray*}[/latex]
where [latex]x_1,x_2,\ldots,x_k[/latex] are the independent variables, [latex]\beta_0,\beta_1,\ldots,\beta_k[/latex] are the population parameters of the regression coefficients, and [latex]\epsilon[/latex] is the error variable. Because the independent variables do not account for all of the variability in the dependent variable [latex]y[/latex], the error variable [latex]\epsilon[/latex] captures the effects of variables other than the independent variables.
We must make certain assumptions about the regression model, in particular about the errors/residuals for the population data, in order to obtain valid conclusions about the multiple regression model. Because we do not have the population data to work with, we cannot verify if these conditions are met. However, much of regression analysis, including testing how well the data fit the model, depends on these assumptions being true.
Assumptions about the multiple regression model include:
- The model is linear.
- The errors/residuals have a normal distribution.
- The mean of the errors/residuals is 0.
- The variance of the errors/residuals is constant.
- The errors/residuals are independent.
Concept Review
In multiple regression, two or more independent variables are used to predict one dependent variable. We can find the values of the regression coefficients for the multiple regression model by generating a regression summary table in Excel. Each regression coefficient represents the change in the dependent variable [latex]y[/latex] for a single unit increase in the corresponding independent variable, while the other variables are held fixed. Certain assumptions about the errors in a multiple regression model are necessary in order to test the validity of the model.
Attribution
“13.1 One-Way ANOVA“ in Introductory Statistics by OpenStax is licensed under a Creative Commons Attribution 4.0 International License.
