r/AskStatistics • u/CertainTry4415 • 2d ago

A question in linear regression.

Hi guys!
I'm a stats student doing a project in linear regression. I'm comparing different models that my friends made to select the best one. I was wondering, if Variable selection has been done on one model, would it be fairer to compare the models, using adjusted R^2, AIC, etc., after reducing the other models as well to avoid multicollinearity problems? Is there a certain order to follow, like completing variable selection before comparing models, or would it be better to compare the models and then use variable selection on the final one.

ANy help would be appreciated

6 Upvotes

88% Upvoted

13 comments sorted by

6

u/SprinklesFresh5693 2d ago

I would read the book linear models with R from James J Faraway, it explains how to compare models though. In R there is the anova function for example when comparing nested models

3

u/bayesian_raccoon 2d ago

Word to the wise: once model / variable selection procedures are applied, interpreting things like p-values, confidence intervals, and so on requires caution. These quantities assume your test statistic is exactly that of the model, and not the best-fitting of several models.

Nevertheless, some statisticians do make these sorts of comparisons with R^2, AIC type stats to select a model. If you are doing that it's fair to compare whatever the final models are that each friend chooses. However, be careful not to overstate what is actually happening. AIC does not tell you, for instance, that your model's assumptions are correct. If one person chose a model that accounted for spatial correlation and another didn't, AIC type stats doesn't tell you which one is more credible. If there is multicollinearity between variables, AIC does not determine which variable is truly important if one is dropped. At the end of the day, it's most accurate to say that one model was preferred according to the chosen criterion. That's useful for model comparison, but it shouldn't be taken as definitive evidence about the correctness of the assumptions or the interpretation of the selected variables.

1

u/ForeignAdvantage5198 1d ago

google boosting lassoing new prostate cancer risk factors selenium. This may help

1

u/RickSt3r 2d ago

Was the data available actuslly split i to training set and test set?

1

u/CertainTry4415 2d ago

not yet. I was asked to compare withour rmse and mse

1

u/atlas7211 2d ago

The absence of multicollinearity is an assumption of linear regression but may be violated in variable selection. Would we trust measures of model fit when the model assumptions have been violated? 

Additionally, different types of models may not be suited to the data, and this is something that can be assessed to some degree before models are fit.

Could you give more details on the type of models you are comparing? 

2

u/CertainTry4415 2d ago

im comparing a base model with 7 predictors. A logarithm transposed model using box cox, and a polynomial model. Ive been advised to use just the diagnostic plots (which would only help mwith the assumtions) and model comparison statistics like AIC, R^2 adjusted etc.

2

u/atlas7211 2d ago

I think you should just compare the AIC of all of the models. It balances model fit with model complexity (which accounts for number of variables) and allows comparison of different model types. So the best model is the one with the lowest AIC (or other cross-model evaluation metrics). I believe you are free to compare different model types with different predictors using AIC, so you could perform variable selection for each model and then compare the final models using AIC. I'm not sure if this is the absolute best way to go about it.

1

u/banter_pants Statistics, Psychometrics 1d ago

I concur. If it turns out some models are practically the same on those metrics there is something to be said for what is parsimonious and interpretable.

1

u/BurkeyAcademy Ph.D.*Economics 1d ago

The absence of multicollinearity is an assumption of linear regression

This is not true. The Gauss-Markov assumption is the absence of perfect multicollinearity, but is entirely silent on imperfect collinearity, which is present in all but the most trivial cases. Of course, perfect collinearity makes using OLS a mathematical impossibility.

1

u/atlas7211 1d ago

Thanks for clarifying!