When applied to subscription businesses, one of the most important questions is:
Which users are likely to leave, and why?
In this project, I explored the KKBox dataset (a large Asian music streaming service) to answer that question. The work went beyond just predicting churn. I wanted to understand the drivers of churn and whether interventions, such as enabling auto-renewal, actually lead to increased retention.
The Challenge: Churn in Subscription Businesses
Churn directly impacts revenue. Acquiring a new user is far more expensive than keeping an existing one, so even small improvements in retention can make a big difference.
The KKBox dataset was perfect for this analysis because it combined:
- Demographics (e.g., gender, registration method)
- Transactions (e.g., subscription length, cancellations, auto-renewal)
- User logs (daily song plays, unique songs, completion ratios)
This gave me both predictive signals and behavioral patterns to analyze.
Step 1: Predicting Churn
I started with prediction models to answer: Who is likely to churn?
- Baseline Logistic Regression: Weak (AUC ≈ 0.42). Too simple for such a non-linear problem.
- XGBoost: Strong results (AUC ≈ 0.98, AP ≈ 0.79).
Top predictive features included:
- Auto-renewal (strong negative correlation with churn)
- Last_gap_days (inactivity gaps strongly increased churn)
- Cancel_rate (more cancellations → higher churn risk)
- Engagement features (unique songs per day, completion ratio)
Step 2: Explaining Predictions
To interpret the black-box model, I used SHAP values.
Key insights:
- Auto-renewal was highly predictive of retention.
- Long inactivity gaps sharply increased churn risk.
- High engagement reduced churn risk.
But, correlation ≠ causation. Just because auto-renew predicted retention doesn’t mean flipping auto-renew to “on” would make people stay.
Step 3: Moving to Causal Insights
This is where things got interesting. I applied causal inference methods to simulate what would happen if we actually intervened:
- Propensity score modeling: estimated the probability of a user being auto-renewed given their history.
- Inverse Probability Weighting (IPW): reweighted the dataset to mimic a randomized experiment.
- Mediation analysis: tested whether engagement was mediating the effect of auto-renew on churn.
Findings:
- The Local Average Treatment Effect (ATE) of auto-renewal was negative (≈ -0.26). Forcing auto-renew didn’t reduce churn.
- Mediation analysis showed that engagement, not auto-renew, explained retention. Without active listening, auto-renew had little effect.
Step 4: Lessons Learned
- Prediction isn’t enough – models can flag who’s at risk, but don’t tell you what to do.
- Correlation misleads – auto-renew looks powerful but isn’t a causal driver.
- Engagement is the real lever – features like inactivity gaps and completion ratio actually drive churn.
- Causal methods bridge the gap – IPW and mediation helped simulate interventions to test actionability.
Conclusion
This project moved from prediction → explanation → causation.
- Prediction (XGBoost) gave us accuracy.
- Explainability (SHAP) told us which features mattered.
- Causal inference revealed which levers were real and which were misleading.
For subscription businesses, the insight is clear: don’t just predict churn, understand what actually prevents it.
Check out the full code on GitHub.