Shopify Retention Engine That Measures Itself: The Artiflow Case Study
The retention engine behind artiflow.studio: three models retrained nightly, one decision per customer, sends paced across the day, and a permanent 5 percent holdout that reports what the emails caused.
- 568,836
- customers scored nightly
- 3
- models retrained every night
- 5%
- permanent holdout
- 0.78
- reorder model auc
Retention tools send a lot of email and claim a lot of credit. Artiflow is the engine we built to do the opposite: decide, for every customer, which one message would help right now, hand that decision to the brand's own email platform, and measure what it caused against customers it deliberately never emails. It has run every night for a subscription-driven DTC brand since July 2026 and is now the product at artiflow.studio.
The problem
A replenishment brand's next repeat customer is already in its order history. But a "days since last order" column cannot tell a buyer who is in rhythm from one who is overdue, a paused subscriber from a churned one, or a product people re-buy at 65 percent from one they re-buy at 6. And no attribution report can say what the emails actually caused, because loyal customers would have bought anyway.
What we built
Every night at 3:30:
- Three models, retrained from scratch. Reorder (0.7835 ROC-AUC on a recent night; 32.5 percent of its top thousand went on to buy, against a 0.84 percent base rate), subscription (8.2 percent in its top thousand against a heuristic's 0.3) and cross-sell. Each is graded walk-forward on data it never saw, against the naive method it replaced, and the result is written to a table every night, including the night it loses.
- One decision per customer. 568,836 customers each get exactly one of 14 lifecycle campaigns with a calibrated probability, one to three products, and a plain sentence saying why. The highest score anywhere in the system is 0.598; nothing reads 99 percent.
- The decision, delivered. Each one lands in the brand's email platform as an event: 16 fields, up to three products, one link that adds all of them to the cart. The brand's own flows render the email.
- Restraint, computed. Of 99,667 candidates one morning, 24,827 were eligible. The rest were unmailable on the email platform, inside the 21 day cooldown, or had just bought something.
- Paced, not blasted. A timer fires every 20 minutes across the day and sends the remaining pool divided by the fires left. Across the whole program the hourly counts stayed flat within about five percent, untuned.
- A permanent 5 percent holdout. Chosen by a hash of the customer id, held back every night, recorded and cooled down like everyone else. Lift is a subtraction, not a story.
What it measured
Head to head on the same customers, graded on orders neither system had seen, Artiflow's reorder model scored 0.65 where the incumbent tool's scored 0.46, worse than a coin flip, with a bootstrapped interval that excludes zero. Its un-emailed picks went on to order about 1.5 times as often as the incumbent's emailed list. On the holdout, the first ten-day read measured about 27,600 in incremental revenue, with a 95 percent interval of roughly 6,300 to 48,800. The instrument keeps reporting every night, and it is built to say so when a campaign is only taking credit for purchases that were coming anyway. The measurement design is written up in measuring marketing lift with a permanent control group.
What it proves
A retention engine earns trust by what it refuses to claim. Artiflow retrains nightly, scores honestly, leaves most people alone, and keeps a control group that can check every number. That is the product: not a promise of lift, but an instrument that measures it.
Artiflow is available as a product.
Every screen shown is demo data: an invented coffee brand rendered through Artiflow's real payload.
Frequently asked questions
- How is the lift measured?
- Five percent of customers, chosen by a stable hash of the customer id, are never emailed by Artiflow. Every report compares the emailed group with them, with a confidence interval, on the brand's own orders. Attribution at 2, 7 and 14 days is shown beside it, labelled as attribution.
- Why does nothing score 99 percent?
- Every campaign score is a calibrated probability of that action within about 90 days, fitted on a leak-free holdout. The highest score anywhere in the system on a recent night was 0.598. Where the data runs out, the score stops rather than extrapolating.
- Does it send the emails?
- No. It sends the decision. Each one lands in the brand's own email platform as an event with the customer, up to three products, one link that adds them all to the cart, and the reason in a sentence. The brand's flows render the email.