Machine Learning and Fairness: Adult Census Dataset
Summary: An accurate Machine Learning model is not necessarily fair. Accuracy can hide discriminatory bias if historical data is unbalanced. Using the Adult Census dataset, we analyze fairness metrics (Demographic Parity) on Random Forest and XGBoost, revealing "proxy variables" with SHAP.
Beyond Simple Accuracy
Machine Learning models are deciding who gets a mortgage, who gets hired, and who ends up in prison. Yet, most Data Scientists only look at Accuracy. A 90% accurate model can hide devastating systemic bias against ethnic or gender minorities.
I used the UCI Adult Census Dataset to demonstrate how deep-rooted this problem is.
The Test Pipeline
I pitted three heavyweight algorithms against each other: Logistic Regression, Random Forest, and XGBoost, tasking them with predicting whether an individual earns more than $50,000 a year. But besides the F1-Score, I added the calculation of *Demographic Parity* and *Equal Opportunity*.
Results: What the Model Really "Thinks"
All models achieved very high accuracy. But looking at the fairness metrics, the picture was scary.
| Algorithm | Accuracy | Demographic Parity Gap | Equal Opportunity Gap |
|---|---|---|---|
| Logistic Regression | 82% | High | Moderate |
| Random Forest | 85% | Very High (Discriminatory) | Very High |
| XGBoost | 87% | Extreme | Extreme |
*Table 1: Results and Fairness Metrics*
The more powerful the model (XGBoost), the more it "learns" to ruthlessly exploit historical correlations of inequality present in the data.
SHAP analysis revealed the black box: variables like Gender heavily influenced the final prediction, acting as hidden *proxy variables*. Conclusion? Fairness must be engineered upstream (e.g., re-weighting), it is not optional.
### Insights - [UCI Adult Census Dataset](https://archive.ics.uci.edu/dataset/2/adult) - [SHAP (SHapley Additive exPlanations)](https://shap.readthedocs.io/)