Self-Driving Retention Engine for a DTC E-Commerce Brand

A subscription-driven DTC e-commerce brand wanted retention marketing that runs itself and proves its own worth. We built an engine that does both: every night it scores all 552,649 customers, decides who should hear what, sends the emails itself, and measures the causal effect against a permanent randomized control group.

The problem

Retention tools send a lot of email and claim a lot of credit. Attribution by last click cannot distinguish revenue the emails caused from revenue that would have arrived anyway. The brand needed a system whose numbers it could defend.

What we built

• Nightly lifecycle scoring. Models trained on 1.4 million orders of history assign each customer exactly one of 14 lifecycle campaigns with a calibrated probability. The reorder model reached 0.7741 ROC-AUC on walk-forward holdouts; among its top thousand picks, 29 percent went on to purchase, roughly 39x the base rate.
• Its own send program. The engine paces its emails across the day and enforces exactly-one-campaign per customer: 26,805 sends in a ten-day window with zero duplicates.
• A permanent 5 percent holdout. A randomized control group is never emailed, so lift is measured, not asserted. Over the first measured window the engine drove an estimated 27,591 in incremental revenue (95 percent confidence interval roughly 6,348 to 48,835, statistically significant on revenue). We report this as evidence, not proof, and the conversion-rate difference alone was not significant.
• A head-to-head win. On an identical holdout, our reorder model scored 0.6517 AUC where the incumbent retention tool's equivalent scored 0.4645, worse than random.

How it works

Four data sources (store, subscriptions, email, and the incumbent tool) merge into an embedded analytical database of 8.1 million rows. The whole system runs unattended on a small cloud VM, nightly, with health-gated auto-rollback; it completed 43 of its last 45 scheduled runs without intervention. Along the way it surfaced a business insight the brand acted on: 27.3 percent of all units ever shipped had effectively been given away free.

What it proves

Machine learning in production is a plumbing discipline. The models matter, but what made this system trustworthy is the control group, the calibration, and the fact that it runs every night without a human in the loop.