Lecture 7: Hypothesis Test and Confidence Intervals of Linear Regression with a Single Regressor

\[ H_0: \beta_1 = \beta_{1,0} \text{ vs. } H_1: \beta_1 \neq \beta_{1,0} \]

The null hypothesis is that \(\beta_1\) is equal to a specific value \(\beta_{1,0}\), and the alternative hypothesis is the opposite.

The general form of the t-statistic is

The t-statistics for testing \(\beta_1\) is then

The p-value is the probability of observing a value of \(\hat{\beta}_1\) at least as different from \(\beta_{1,0}\) as the estimate actually computed (\(\hat{\beta}^{act}_1\)), assuming that the null hypothesis is correct.

Accordingly, under the null hypothesis, the p-value for testing \(\beta_1\) can be expressed with a probability function as

With a large sample, the t statistic is approximately distributed as a standard normal random variable. Therefore, we can compute \[p\text{-value} = \pr\left(|t| > |t^{act}| \right) = 2 \Phi(-|t^{act}|)\] where \(\Phi(\cdot)\) is the c.d.f. of the standard normal distribution.

The null hypothesis is rejected at the 5% significance level if the \(p\text{-value} < 0.05\) or, equivalently, \(|t^{act}| > 1.96\).

The OLS estimation of the linear regression model of test scores against student-teacher ratios, together with the standard errors of all parameters in the model, can be represented using the following equation,

The heteroskedasticity-robust standard errors are reported in the parentheses, that is, \(SE(\hat{\beta}_0) = 10.4\) and \(SE(\hat{\beta}_1) = 0.52\).

The superintendent's question is whether \(\beta_1\) is significant, for which we can test the null hypothesis against the alternative one as \[ H_0: \beta_1 = 0, H_1: \beta_1 \neq 0 \]

The t-statistics is \[ t = \frac{\hat{\beta}_1}{SE(\hat{\beta}_1)} = \frac{-2.28}{0.52} = -4.38 < -1.96 \]

The p-value associated with \(t^{act} = -4.38\) is approximately 0.00001, which is far less than 0.05.

Based on the t-statistics and the p-value, we can say the null hypothesis is rejected at the 5% significance level. In English, it means that the student-teacher ratios do have a significant effect on test scores.

In some cases, it is appropriate to use a one-sided hypothesis test. For example, the superintendent of the California school districts want to know whether an increase in class sizes has a negative effect on test scores, that is, \(\beta_1 < 0\).

For a one-sided test, the null hypothesis and the one-sided alternative hypothesis are ¹

\[ H_0: \beta_1 = \beta_{1,0}, H_1: \beta_1 < \beta_{1,0} \]

• The t-statistic is the same as in a two-sided test \[ t = \frac{\hat{\beta}_1 - \beta_{1,0}}{SE(\hat{\beta}_1)} \]
• Since we test \(\beta_1 < \beta_{1,0}\), if this is true, the t-statistics should be statistically significantly less than zero.
• The p-value is computed as \(\pr(t < t^{act}) = \varPhi(t^{act})\).
• The null hypothesis is rejected at the 5% significance level when the \(p-\text{value} < 0.05\) or \(t^{act} < -1.645\).
• In the application of test scores, the t-statistics is -4.38, which is less than -1.645 and -2.33 (the critical value for a one-sided test with a 1% significance level). Thus, the null hypothesis is rejected at the 1% level.

The error term \(u_i\) is homoskedastic if the conditional variance of \(u_i\) given \(X_i\) is constant for all \(i = 1, \ldots, n\). Mathematically, it says \(\var(u_i | X_i) = \sigma^2,\, \text{ for } i = 1, \ldots, n\), i.e., the variance of \(u_i\) for all i is a constant and does not depend on \(X_i\).

In contrast, the error term \(u_i\) is heteroskedastic if the conditional variance of \(u_i\) given \(X_i\) changes on \(X_i\) for \(i = 1, \ldots, n\). That is, \(\var(u_i | X_i) = \sigma^2_i,\, \text{ for } i = 1, \ldots, n\).

e.g.. A multiplicative form of heteroskedasticity is \(\var(u_i|X_i) = \sigma^2 f(X_i)\) where \(f(X_i)\) is a function of \(X_i\), for example, \(f(X_i) = X_i\) as a simplest case.

As long as the least squares assumptions holds, whether the error term, \(u_i\), is homoskedastic or heteroskedastic does not affect unbiasedness, consistency, and the asymptotic normal distribution of the OLS estimators.

• The unbiasedness requires that \(E(u_i|X_i) = 0\)
• The consistency requires that \(E(X_i u_i) = 0\), which is true if \(E(u_i|X_i)=0\).
• The asymptotic normal distribution requires additionally that \(\var((X_i-\mu_X)u_i) < \infty\), which still holds as long as Assumption 3 holds, that is, no extreme outliers of \(X_i\).

The existence of heteroskedasticity affects the efficiency of the OLS estimator

• Suppose \(\hat{\beta}_1\) and \(\tilde{\beta}_1\) are both unbiased estimators of \(\beta_1\). Then, \(\hat{\beta}_1\) is said to be more efficient than \(\tilde{\beta}_1\) if \(\var(\hat{\beta}_1) < \var(\tilde{\beta}_1)\).
• When the errors are homoskedastic, the OLS estimators \(\hat{\beta}_0\) and \(\hat{\beta}_1\) are the most efficient among all estimators that are linear in \(Y_1, \ldots, Y_n\) and are unbiased, conditional on \(X_1, \ldots, X_n\).
• See the Gauss-Markov Theorem below.

For \(\mathbf{X} = [X_1, \ldots, X_n]\) ²

Here I use the vector notation to represent all observations of \(X_i\) for \(i=1, \ldots, n\). We will formally introduce the matrix notation for a linear regression model and the OLS estimation in the next lecture.

1. \(E(u_i| \mathbf{X}) = 0\)
2. \(\var(u_i | \mathbf{X}) = \sigma^2_u,\, 0 < \sigma^2_u < \infty\)
3. \(E(u_i u_j | \mathbf{X}) = 0,\, i \neq j\)

Note that the conditional expectations in the Gauss-Markov conditions regard all observations \(\mathbf{X}\), not just one observation, \(X_i\). However, all the Gauss-Markov conditions can be derived from the least squares assumptions plus the homoskedasticity assumption. Specifically,

• Assumptions (1) and (2) imply \(E(u_i | \mathbf{X}) = E(u_i | X_i) = 0\).
• Assumptions (1) and (2) imply \(\var(u_i| \mathbf{X}) = \var(u_i | X_i)\). With the homoskedasticity assumption, \(\var(u_i | X_i) = \sigma^2_u\), Assumption (3) then implies \(0 < \sigma^2_u < \infty\).
• Assumptions (1) and (2) imply that \(E(u_i u_j | \mathbf{X}) = E(u_i u_j | X_i, X_j) = E(u_i|X_i) E(u_j|X_j) = 0\).

The class of linear conditionally unbiased estimators consists of all estimators of \(\beta_1\) that are linear function of \(Y_i, \ldots, Y_n\) and that are unbiased, conditioned on \(X_1, \ldots, X_n\).

For any linear estimator \(\tilde{\beta}_1\), it can be written as

where the weights \(a_i\) for \(i = 1, \ldots, n\) depend on \(X_1, \ldots, X_n\) but not on \(Y_1, \ldots, Y_n\).

\(\tilde{\beta}_1\) is conditionally unbiased means that

By the Gauss-Markov conditions, from Equation (\ref{eq:beta1-tilde}), we can have

For Equation (\ref{eq:e-beta1-tilde}) being satisfied with any \(\beta_0\) and \(\beta_1\), we must have \[ \sum_i a_i = 0 \text{ and } \sum_i a_iX_i = 1 \]

We have known that \(\hat{\beta}_1\) is unbiased both conditionally and unconditionally. Next, we show that it is linear. \[ \hat{\beta}_1 = \frac{\sum_i (X_i - \bar{X})(Y_i - \bar{Y})}{\sum_i (X_i - \bar{X})^2} = \frac{\sum_i (X_i - \bar{X})Y_i}{\sum_i (X_i - \bar{X})^2} = \sum_i \hat{a}_i Y_i \] where the weights are \[ \hat{a}_i = \frac{X_i - \bar{X}}{\sum_i (X_i - \bar{X})^2}, \text{ for } i = 1, \ldots, n \] Since \(\hat{\beta}_1\) is a linear conditionally unbiased estimator, we must have \[ \sum_i \hat{a}_i = 0 \text{ and } \sum_i \hat{a}_i X_i = 1 \] which can be simply verified.

\[H_0: \beta_1 = \beta_{1,0} \text{ vs } H_1: \beta_1 \neq \beta_{1,0}\]

where

the former of which is the homoskedasticity-only standard error of \(\hat{\beta}_1\) and the latter is the standard error of the regression.

When the classical least squares assumptions hold, the t-statistic has the exact distribution of \(t(n-2)\), i.e., the Student's t distribution with \((n-2)\) degrees of freedom.

\[ t = \frac{\hat{\beta}_1 - \beta_{1,0}}{\hat{\sigma}_{\hat{\beta}_1}} \sim t(n-2) \]

What follows is to show the above equation is true when all classical assumptions are true.

The t statistic can be rewritten as

where

\[\sigma^2_{\hat{\beta}_1} = \frac{\sigma^2_u}{\sum_i (X_i - \bar{X})^2} \]

is the homoskedasticity-only variance of \(\hat{\beta}_1\) when the variance of errors \(\sigma^2_u\) is known.

\[ z_{\hat{\beta}_1} =\frac{\hat{\beta}_1 - \beta_{1,0}}{\sigma_{\hat{\beta}_1}} \]

is the z-statistic which has a standard normal distribution, that is, \(z_{\hat{\beta}_1} \sim N(0, 1)\)

\[ W = (n-2)\frac{s^2_u}{\sigma^2_u} = \frac{\sum_i\hat{u}_i^2}{\sigma^2_u} = \sum_i \left(\frac{\hat{u}_i}{\sigma_u}\right)^2 \]

It can be shown that W is the sum of squares of \((n-2)\) independent standard normally distributed variables, which results in a chi-squared distribution with \((n-2)\) degrees of freedom. That is, \(W \sim \chi^2(n-2)\), which is also independent of \(z_{\hat{\beta}_1}\). Therefore, the t-statistic in Equation (\ref{eq:t-stat-b1a}), as the ratio of \(z_{\hat{\beta}_1}\) and \(\sqrt{W/(n-2)}\), is distributed as \(t(n-2)\).