Statistical Learning Practice Problems
Question 1
You’re comparing two models: Model A is very simple (like fitting a horizontal line), and Model B is very complex (like a high-degree polynomial). Which model would you expect to have higher bias and which would have higher variance? Explain your reasoning.
Question 2
A data scientist claims “I achieved 99% accuracy on my training data, so my model is excellent!” What would you tell them about why this might not be a reliable measure of model quality?
Question 3
You’re building a model to predict house prices based on square footage, number of bedrooms, and neighborhood. Is this a regression or classification problem?
☐ Regression
☐ Classification
Question 4
TRUE or FALSE: If you add more predictors to a linear regression model, the training error will always decrease (or stay the same).
☐ TRUE
☐ FALSE
Question 5
You fit two different models: a linear model and a k-nearest neighbors model with k=1. Which model is more flexible? Which would you expect to have lower training error?
Question 6
In 4-fold cross-validation with 800 observations, how many observations are used for training in each fold? How many observations are used for validation in each fold?
Question 7
You’re comparing different values of k for k-nearest neighbors using cross-validation. You find that k=1 gives the lowest training error, but k=9 gives the lowest cross-validation error. Which value should you choose for your final model and why?
Question 8
TRUE or FALSE: Leave-One-Out Cross-Validation (LOOCV) has lower bias but higher variance compared to 10-fold cross-validation when estimating test error.
☐ TRUE
☐ FALSE
Question 9
You have 300 observations and want to use 5-fold cross-validation to compare 4 different models. How many total models will you fit during this process?
Question 10
Consider the dataset:
\[ \begin{matrix} \text{Sales} & \text{Advertising} & \text{Price}\\ 100 & 5 & 20\\ 120 & 7 & 18\\ 80 & 3 & 25\\ \end{matrix} \]
If you fit the model: Sales = 80 + 10×Advertising - 2×Price, what is the predicted sales when for the first observation?
Question 11
You fit a regression model and get \(R^2 = 0.75\) on training data. Your colleague fits a different model to the same training data and gets \(R^2 = 0.85\). Can you conclude that your colleague’s model will perform better on new data? Why or why not?
Question 12
Consider fitting a model: salary ~ experience + education to predict salaries. You get β̂_experience = 3.2 (where salary is in thousands and experience is in years). Write a complete interpretation of this coefficient.
Solutions
Try to work out the problems above before looking at the solutions. They are provided here to help you check your work.
Question 1
Answer: Model A (simple) would have higher bias and lower variance. Model B (complex) would have lower bias and higher variance.
Simple models like a horizontal line make strong assumptions about the relationship (that it’s constant), leading to high bias when the true relationship is more complex. However, they’re stable across different training sets (low variance). Complex models can capture intricate patterns (low bias) but are sensitive to small changes in training data, leading to high variance. This is the fundamental bias-variance tradeoff.
Question 2
Answer: Training accuracy is not a reliable measure because it doesn’t tell us how the model will perform on new, unseen data. A 99% training accuracy often indicates overfitting.
The model may have memorized the training data rather than learning generalizable patterns. What matters is performance on test/validation data that the model hasn’t seen during training. High training performance with poor test performance is a classic sign of overfitting.
Question 3
Answer: ☐ Regression
Since we’re predicting house prices (a continuous, quantitative outcome), this is a regression problem. Classification would involve predicting categorical outcomes like “expensive vs. affordable” or “sell vs. don’t sell.”
Question 4
Answer: ☐ TRUE
Adding more predictors to a linear regression model will always decrease the training RSS (and thus training error) or keep it the same. This is because ordinary least squares finds the coefficients that minimize training error, so additional predictors can only help the fit on training data (though they may hurt generalization to test data).
Question 5
Answer: The k-nearest neighbors model with k=1 is more flexible. The k=1 model would have lower training error.
K=1 KNN can create very complex decision boundaries and can perfectly fit any training dataset (training error = 0), making it extremely flexible. Linear models assume a specific functional form and are less flexible. However, this extreme flexibility of k=1 usually leads to overfitting and poor test performance.
Question 6
Answer: Training: 600 observations per fold. Validation: 200 observations per fold.
With 800 observations and 4 folds:
- Each fold uses 800/4 = 200 observations for validation
- Remaining 800 - 200 = 600 observations for training
Question 7
Answer: Choose k=9 for the final model.
Cross-validation error is a better estimate of how the model will perform on new data compared to training error. k=1 having the lowest training error likely indicates overfitting. The fact that k=9 has the lowest CV error suggests it strikes the best balance between bias and variance for this dataset.
Question 8
Answer: ☐ TRUE
LOOCV uses almost all data for training (n-1 observations), giving estimates closer to the true test error (lower bias). However, the training sets in LOOCV are highly similar to each other, making the individual error estimates highly correlated and thus the overall estimate has higher variance compared to k-fold CV.
Question 9
Answer: 20 models total.
5-fold CV means we fit each of the 4 models 5 times (once for each fold). So 4 models × 5 folds = 20 total model fits.
Question 10
Answer: Predicted Sales = 90
Sales = 80 + 10×Advertising - 2×Price Sales = 80 + 10×5 - 2×20 Sales = 80 + 50 - 40 = 90
Question 11
Answer: No, you cannot conclude that your colleague’s model will perform better on new data.
Higher \(R^2\) on training data only indicates better fit to the training data, which may be due to overfitting. The colleague’s model might be more complex and capturing noise rather than true signal. To properly compare models, you need to evaluate their performance using a metric that approximates (or calculates) the test error rather than the training error.
Question 12
Answer: Holding education constant, for each additional year of experience, salary is expected to increase by 3.2 thousand dollars (or $3,200) on average.
This interpretation includes: (1) the ceteris paribus condition (holding other variables constant), (2) the direction and magnitude of the effect, (3) the proper units (thousands of dollars and years), and (4) that this represents an average relationship, not a guarantee for any individual.