Classifying coronary heart disease risk from NHANES survey data (2011-2018), with a full leakage audit and calibration check [P]

Github repo :

https://github.com/YouCele/nhanes-chd-classification

I built a project that predicts self-reported, physician-diagnosed coronary heart disease using four cycles of NHANES data (2011-2012 to 2017-2018), about 21,500 adults after cleaning. It compares logistic regression with random forest and gradient boosting, trained on demographics, blood pressure, body measurements and a lipid panel.

NHANES has a questionnaire section that asks directly about other cardiovascular diagnoses, like stroke, heart attack and angina, along with the CHD outcome itself. If I included those variables, PR-AUC jumped from 0.23 to 0.51. That's mostly the model learning that people who report one cardiovascular diagnosis tend to report others too, not something about real risk factors. So I removed that whole section and wrote up how big the leakage effect was, instead of just quietly dropping the columns.

The class weighted logistic regression, needed because CHD prevalence is only about 4%, gave badly miscalibrated raw probabilities. The mean predicted risk was close to 30% while the real rate was 4%. I fixed this with a sigmoid recalibration fit on the development set, before the test set was touched.

I also picked and froze the decision threshold on the dev set before looking at the test set, so the reported test metrics aren't tuned after the fact.

Final results on the held out test set: ROC-AUC 0.875 and PR-AUC 0.239 for logistic regression, and random forest and gradient boosting came out about the same. Age alone gets 0.83 AUC. Blood pressure, cholesterol and body size explain most of what's left. PPV at the chosen threshold is 0.13, so most positive predictions are wrong, which is expected given how rare CHD is in this data, and I state this directly in the report instead of leaving it out.

Smoking status, diabetes and blood pressure medication use aren't in the current features. NHANES has all three, I just haven't added them yet.

Happy to hear feedback, especially on the calibration step or anything in how I handled the leakage that looks wrong