13.5 Testing the Significance of the Overall Model

Previously, we learned that the population model for the multiple regression equation is

[latex]\begin{eqnarray*} y & = & \beta_0+\beta_1x_1+\beta_2x_2+\cdots+\beta_kx_k +\epsilon \end{eqnarray*}[/latex]

where [latex]x_1,x_2,\ldots,x_k[/latex] are the independent variables, [latex]\beta_0,\beta_1,\ldots,\beta_k[/latex] are the population parameters of the regression coefficients, and [latex]\epsilon[/latex] is the error variable.  The error variable [latex]\epsilon[/latex] accounts for the variability in the dependent variable that is not captured by the linear relationship between the dependent and independent variables.  The value of [latex]\epsilon[/latex] cannot be determined, but we must make certain assumptions about [latex]\epsilon[/latex] and the errors/residuals in the model in order to conduct a hypothesis test on how well the model fits the data.  These assumptions include:

• The model is linear.
• The errors/residuals have a normal distribution.
• The mean of the errors/residuals is 0.
• The variance of the errors/residuals is constant.
• The errors/residuals are independent.

Because we do not have the population data, we cannot verify that these conditions are met.  We need to assume that the regression model has these properties in order to conduct hypothesis tests on the model.

Testing the Overall Model

We want to test if there is a relationship between the dependent variable and the set of independent variables.  In other words, we want to determine if the regression model is valid or invalid.

• Invalid Model.  There is no relationship between the dependent variable and the set of independent variables.  In this case, all of the regression coefficients [latex]\beta_i[/latex] in the population model are zero.  This is the claim for the null hypothesis in the overall model test:  [latex]H_0: \beta_1=\beta_2=\cdots=\beta_k=0[/latex].
• Valid Model.  There is a relationship between the dependent variable and the set of independent variables.  In this case, at least one of the regression coefficients [latex]\beta_i[/latex] in the population model is not zero.  This is the claim for the alternative hypothesis in the overall model test:  [latex]H_a: \mbox{at least one } \beta_i \neq 0[/latex].

The overall model test procedure compares the means of explained and unexplained variation in the model in order to determine if the explained variation (caused by the relationship between the dependent variable and the set of independent variables) in the model is larger than the unexplained variation (represented by the error variable [latex]\epsilon[/latex]).  If the explained variation is larger than the unexplained variation, then there is a relationship between the dependent variable and the set of independent variables, and the model is valid.  Otherwise, there is no relationship between the dependent variable and the set of independent variables, and the model is invalid.

The logic behind the overall model test is based on two independent estimates of the variance of the errors:

• One estimate of the variance of the errors, [latex]MSR[/latex], is based on the mean amount of explained variation in the dependent variable [latex]y[/latex].
• One estimate of the variance of the errors, [latex]MSE[/latex], is based on the mean amount of unexplained variation in the dependent variable [latex]y[/latex].

The overall model test compares these two estimates of the variance of the errors to determine if there is a relationship between the dependent variable and the set of independent variables.  Because the overall model test involves the comparison of two estimates of variance, an [latex]F[/latex]-distribution is used to conduct the overall model test, where the test statistic is the ratio of the two estimates of the variance of the errors.

The mean square due to regression, [latex]MSR[/latex], is one of the estimates of the variance of the errors.  The [latex]MSR[/latex] is the estimate of the variance of the errors determined by the variance of the predicted [latex]\hat{y}[/latex]-values from the regression model and the mean of the [latex]y[/latex]-values in the sample, [latex]\overline{y}[/latex].  If there is no relationship between the dependent variable and the set of independent variables, then the [latex]MSR[/latex] provides an unbiased estimate of the variance of the errors.  If there is a relationship between the dependent variable and the set of independent variables, then the [latex]MSR[/latex] provides an overestimate of the variance of the errors.

[latex]\begin{eqnarray*} SSR & = & \sum \left(\hat{y}-\overline{y}\right)^2 \\  \\ MSR & =& \frac{SSR}{k} \end{eqnarray*}[/latex]

The mean square due to error, [latex]MSE[/latex], is the other estimate of the variance of the errors.  The [latex]MSE[/latex] is the estimate of the variance of the errors determined by the error [latex](y-\hat{y})[/latex] in using the regression model to predict the values of the dependent variable in the sample.  The [latex]MSE[/latex] always provides an unbiased estimate of the variance of errors, regardless of whether or not there is a relationship between the dependent variable and the set of independent variables.

[latex]\begin{eqnarray*} SSE & = & \sum \left(y-\hat{y}\right)^2\\  \\ MSE & =& \frac{SSE}{n -k-1} \end{eqnarray*}[/latex]

The overall model test depends on the fact that the [latex]MSR[/latex] is influenced by the explained variation in the dependent variable, which results in the [latex]MSR[/latex] being either an unbiased or overestimate of the variance of the errors.  Because the [latex]MSE[/latex] is based on the unexplained variation in the dependent variable, the [latex]MSE[/latex] is not affected by the relationship between the dependent variable and the set of independent variables, and is always an unbiased estimate of the variance of the errors.

The null hypothesis in the overall model test is that there is no relationship between the dependent variable and the set of independent variables.  The alternative hypothesis is that there is a relationship between the dependent variable and the set of independent variables.  The [latex]F[/latex]-score for the overall model test is the ratio of the two estimates of the variance of the errors, [latex]\displaystyle{F=\frac{MSR}{MSE}}[/latex] with [latex]df_1=k[/latex] and [latex]df_2=n-k-1[/latex].  The p-value for the test is the area in the right tail of the [latex]F[/latex]-distribution to the right of the [latex]F[/latex]-score.

Steps to Conduct a Hypothesis Test on the Overall Regression Model

1. Write down the null hypothesis that there is no relationship between the dependent variable and the set of independent variables:

   [latex]\begin{eqnarray*} H_0: &  &  \beta_1=\beta_2=\cdots=\beta_k=0 \\ \\ \end{eqnarray*}[/latex]

2. Write down the alternative hypotheses that there is a relationship between the dependent variable and the set of independent variables:

   [latex]\begin{eqnarray*}  H_a: &  &  \mbox{at least one } \beta_i \mbox{ is not 0} \\ \\ \end{eqnarray*}[/latex]

3. Collect the sample information for the test and identify the significance level [latex]alpha[/latex].
4. The p-value is the area in the right tail of the [latex]F[/latex]-distribution.  The [latex]F[/latex]-score and degrees of freedom are

   [latex]\begin{eqnarray*}F & = & \frac{MSR}{MSE} \\ \\ df_1 & = & k \\ \\  df_2 &  = & n-k-1 \\ \\ \end{eqnarray*}[/latex]

5. Compare the p-value to the significance level and state the outcome of the test:
   • If p-value[latex]\leq \alpha[/latex], reject [latex]H_0[/latex] in favour of [latex]H_a[/latex].
     • The results of the sample data are significant.  There is sufficient evidence to conclude that the null hypothesis [latex]H_0[/latex] is an incorrect belief and that the alternative hypothesis [latex]H_a[/latex] is most likely correct.
   • If p-value[latex]\gt \alpha[/latex], do not reject [latex]H_0[/latex].
     • The results of the sample data are not significant.  There is not sufficient evidence to conclude that the alternative hypothesis [latex]H_a[/latex] may be correct.
6. Write down a concluding sentence specific to the context of the question.

The calculation of the [latex]MSR[/latex], the [latex]MSE[/latex], and the [latex]F[/latex]-score for the overall model test can be time consuming, even with the help of software like Excel.  However, the required [latex]F[/latex]-score and p-value for the test can be found on the regression summary table, which we learned how to generate in Excel in a previous section.

_{Watch this video: Basic Excel Business Analytics #51: Testing Significance of Regression Relationship with p-value by ExcelIsFun [20:44]}

Concept Review

The overall model test determines if there is a relationship between the dependent variable and the set of independent variable.  The test compares two estimates of the variance of the errors ([latex]MSR[/latex] and [latex]MSE[/latex]).  The ratio of these two estimates of the variance of the errors is the [latex]F[/latex]-score from an [latex]F[/latex]-distribution with [latex]df_1=k[/latex] and [latex]df_2=n-k-1[/latex].  The p-value for the test is the area in the right tail of the [latex]F[/latex]-distribution.  The p-value can be found on the regression summary table generated by Excel.

The overall model hypothesis test is a well established process:

1. Write down the null and alternative hypotheses in terms of the regression coefficients.  The null hypothesis is the claim that there is no relationship between the dependent variable and the set of independent variables.  The alternative hypothesis is the claim that there is a relationship between the dependent variable and the set of independent variables.
2. Collect the sample information for the test and identify the significance level.
3. The p-value is the area in the right tail of the [latex]F[/latex]-distribution.  Use the regression summary table generated by Excel to find the p-value.
4. Compare the p-value to the significance level and state the outcome of the test.
5. Write down a concluding sentence specific to the context of the question.