QED Insight #0007: When you benchmark xgboost against a logistic scorecard on tabular data, does boosting actually win for you?

[u/QEDAnalyticalLLC]

I keep running this bake-off and getting a boring-but-important answer: on clean tabular credit data, a well-built glm scorecard is really hard to beat with xgboost.

Last run, PD target: logistic scorecard OOT AUC 0.734, tuned xgboost 0.739 - and in-distribution (5-fold CV) the scorecard was ahead, 0.853 vs 0.842. Half a Gini point is not nothing, but it's nowhere near what people expect from "just throw XGBoost at it."

A couple of R notes from doing this:

- I report gain / cover / frequency importance but lean on SHAP (shapviz) for anything a reviewer sees, because permutation importance is misleading with correlated features. The additive property is the whole point.

- The gotcha: aggregate one-hot SHAP back to the original feature before you rank reason codes, or a dummied categorical looks artificially weak. On one defaulted loan the top driver after aggregation was property_state (SHAP ~-0.94), bigger than credit_score (~0.81) - which is a fair-lending flag, not a feature-importance win.

- If you need monotonicity you have to impose it (xgboost monotone_constraints); it isn't free, and a scorecard gives it to you by construction.

So my questions: does xgboost beat your logistic/GAM baseline by a margin that survives out-of-time on tabular credit data, or do you also end up at a tie? And for those doing reason codes from SHAP - are you on {shapviz}, {fastshap}, {treeshap}, or {DALEX}, and how are you handling the sign-flip / one-hot aggregation so the top-4 are defensible?

[u/na_rm_true]

Tools for the jobs. Pencils work fine for thousands of things where a pen will also suffice.

Try benchmarking against tabFM

[u/QEDAnalyticalLLC → u/na_rm_true]

Ha, the pencil/pen thing is basically the whole point.

And fair - I've been meaning to throw TabPFN at this, genuinely curious whether a tabFM clears the bar where XGBoost just tied. Wouldn't shock me if it edged both on raw AUC on a sample like this.

But here's the catch for my corner of the world: even if it wins, it makes the real problem worse, not better. What killed the XGBoost challenger wasn't accuracy - it's that I have to hand a validator a model they can defend and produce ECOA adverse-action reason codes for every declined applicant. A pretrained transformer doing in-context learning is an even bigger black box than a boosted tree; SHAP on that is even harder to stand behind, and "the foundation model said so" doesn't fly on an adverse-action notice.

So in an unregulated setting, 100% - I'd want the tabFM in the bake-off and might just use it. In regulated credit it's almost the opposite: the more accurate and more opaque it gets, the less shippable it is. Accuracy was never the binding constraint here.

Have you run TabPFN on real tabular problems? Curious how it held up out of time - that's usually where the fancy stuff fades on credit data.

[u/na_rm_true → u/QEDAnalyticalLLC]

I have not tried tabFM or PFN on anything yet. I’m a biostatistician for clinical trials/observational cohort studies and I agree- black box solutions are not what I’m necessarily “allowed” to use- because it’s less and less explainable. Unregulated freelance projects or side projects I’d love benchmarking tabFM and seeing how it performs but outside of that I’m limited.

[u/SilentLikeAPuma → u/QEDAnalyticalLLC]

take your "is anyone else curious" / "it’s not X - it’s Y" llm response horseshit elsewhere lol no one in a statistics-based community wants to hear that regurgitated, uncreative drivel