Econ 425T Homework 2

Due Feb 3, 2023 @ 11:59PM

1 Least squares is MLE (10pts)

Show that in the case of linear model with Gaussian errors, maximum likelihood and least squares are the same thing, and $C_p$ and AIC are equivalent.

2 ISL Exercise 6.6.1 (10pts)

3 ISL Exercise 6.6.3 (10pts)

4 ISL Exercise 6.6.4 (10pts)

5 ISL Exercise 6.6.5 (10pts)

6 ISL Exercise 6.6.11 (30pts)

You must follow the typical machine learning paradigm to compare at least 3 methods: least squares, lasso, and ridge. Report final results as

Method	CV RMSE	Test RMSE
LS
Ridge
Lasso
…

7 ISL Exercise 5.4.2 (10pts)

8 ISL Exercise 5.4.9 (20pts)

9 Bonus question (20pts)

Consider a linear regression, fit by least squares to a set of training data $(x_1,y_1),\dots ,(x_N,y_N)$ drawn at random from a population. Let $\hat{\beta }$ be the least squares estimate. Suppose we have some test data $({\tilde{x}}_1,{\tilde{y}}_1),\dots ,({\tilde{x}}_M,{\tilde{y}}_M)$ drawn at random from the same population as the training data. If $R_{\text{train}}(\beta )=\frac{1}{N}\sum _{i=1}^N(y_i-{\beta }^T{x}_i{)}^2$ and $R_{\text{test}}(\beta )=\frac{1}{M}\sum _{i=1}^M({\tilde{y}}_i-{\beta }^T{\tilde{x}}_i{)}^2$. Show that $$\mathrm{E}[R_{\text{train}}(\hat{\beta })]<\mathrm{E}[R_{\text{test}}(\hat{\beta })].$$