Measuring what formal privacy guarantees actually cost a credit-risk model

A study of privacy-preserving machine learning applied to credit-default prediction, completed as part of my MSc Data Science studies at the University of Basel. The project trains models under two DP strategies — Local DP (noise added on-device) and Central DP-SGD (noise added to gradients via Opacus) — and evaluates the resulting utility-privacy-fairness trade-off using real membership-inference attacks.

Completed as part of my MSc Data Science coursework (Privacy Preserving Methods, Spring 2026) at the University of Basel, applying differential privacy techniques to two real credit-default datasets.

Within the team project, my contribution focused on the DP-SGD training pipeline (via Opacus), the Local DP perturbation pipeline, both membership-inference attacks (threshold and shadow-model), and the fairness analysis across privacy budgets and datasets. The findings below are the project's shared results.

Credit-scoring models are trained on sensitive financial data. Standard training can leak information about individual training examples, and formal privacy guarantees typically come at the cost of predictive performance — a trade-off that needs to be measured, not assumed.

• Train baseline (non-private) and differentially private credit-default classifiers
• Implement DP-SGD training using PyTorch and Opacus, and LDP via Laplace perturbation
• Evaluate models across a range of privacy budgets (ε = 1, 3, 8)
• Run membership-inference attacks (threshold + shadow-model) to empirically assess privacy leakage
• Analyze fairness of predictions across income/credit-limit tiers

A PyTorch training pipeline with an Opacus privacy engine wrapping the optimizer for per-example gradient clipping and noise injection under DP-SGD, a separate Laplace-mechanism pipeline for Local DP, a from-scratch attack pipeline (threshold + shadow-model membership inference, following Shokri et al. 2017 and Carlini et al. 2021), and a fairness-analysis module stratifying results by income/credit-limit tier. An interactive Streamlit dashboard (deployed to Hugging Face Spaces) exposes the trained results and a live LDP playground.

Implemented classifiers trained on two real datasets — Give Me Credit (GMC, ~150k rows, 10 features) and the UCI Default of Credit Card dataset (~30k rows, 23 features) — under standard (non-private) training, Central DP-SGD (via Opacus), and Local DP (Laplace perturbation before training). Evaluated each configuration on AUC-ROC/F1, susceptibility to two independent membership-inference attacks, and fairness metrics across privacy levels.

Tuning DP-SGD hyperparameters (clipping norm, noise multiplier, batch size) to retain usable predictive performance at meaningful privacy budgets was one difficulty. The bigger finding was that LDP's per-feature noise budget doesn't scale: on the 23-feature UCI dataset, sequential composition splits ε across every feature, collapsing utility in a way DP-SGD does not experience.

Swept privacy budgets and clipping thresholds systematically, comparing utility, membership-inference attack success rate, and fairness metrics at each setting to characterize the trade-off empirically rather than relying on default hyperparameters.

On the 10-feature GMC dataset, DP-SGD stayed close to the non-private baseline across all tested budgets (AUC 0.857–0.861 vs. 0.867 baseline for ε=1–8). On the 23-feature UCI dataset, LDP collapsed at ε=3 (AUC 0.358, below random chance) because sequential composition forces each feature to share the budget (ε/23 ≈ 0.13 per feature); DP-SGD was unaffected (AUC 0.71–0.73) since gradient noise doesn't scale with feature count. Membership-inference attacks — including against the non-private baseline — stayed at attack AUC ≈ 0.50 across every configuration, meaning the dataset's natural class skew (6.7% positive in GMC) was already a strong empirical defense before any DP mechanism was applied. Fairness analysis showed privacy noise degrading GMC performance roughly equally across income tiers, with no systematic directional bias.

Differential privacy guarantees are only meaningful when paired with empirical validation — membership-inference attacks provided a concrete signal that, on this data, the class imbalance was doing more empirical privacy work than the DP mechanisms themselves. The LDP-vs-DP-SGD comparison also made the feature-count sensitivity of local noise mechanisms concrete in a way that reading the theory alone does not.

• Extend evaluation to additional attack types beyond membership inference
• Explore per-group privacy budgets to address fairness trade-offs directly
• Test LDP with per-feature budget allocation instead of uniform splitting, to see if it closes the gap with DP-SGD on high-dimensional data