Case study ·
CS-04
·
Recurly
Design impact as a time series: task success from 49% to 100%
Recurly’s first repeatable usability benchmark: six revenue tasks, one script, every release. Design quality became a time series, and the practice outlived me.
Cancel subscription
49→100%
Run 1
49%
Run 2
100%
Add subscription
29→91%
Run 1
29%
Run 2
91%
Issue refund
20→87%
Run 1
20%
Run 2
87%
fig. 1
Task success after two runs: same six tasks, same script, one month apart. Three of the six shown; Add Charge Invoice still lagged, and it stays on the chart on purpose.
Benchmark · runs 1–2
01
A platform thousands trusted, never once tested
Recurly is a B2B2C subscription billing platform: the system merchants trust to charge their customers correctly. That trust cut both ways. Every rough edge shipped at scale, on tasks with real money in them, and in the platform’s history nobody had ever run a usability test on it. Quality was invisible; releases shipped on faith.
The invisibility was a design problem of its own. Value you can’t measure is easy to cut, and design at Recurly was spending its credibility on opinion. I wanted a different currency.
The thesis case
This is the practice behind the receipts. The habit of attaching a number to design work starts in projects like this one.
02
Don’t argue for design. Measure it.
The bet was simple: stop making the case for design in meetings and let the product make it. I picked six revenue-critical tasks (canceling a subscription, adding one, and issuing a refund among them), wrote one test script with an engineer, and made those tasks the yardstick we’d run every release.
One scorecard, every release: the same tasks and the same script, so every delta is directly comparable to what shipped in between.
Field note
A measurement practice doesn’t need a research org. It needs a script, a calendar, and some stubbornness.
Design impact stops being an opinion and becomes a time series.
Benchmark constants — held every run
The tasks
Six revenue-critical tasks — canceling a subscription, adding one, and issuing a refund among them — in the same order, every run.
The script
One script, re-run verbatim each release, so every delta is directly comparable to what shipped in between.
The reporting
Task success per task, per run. Laggards stay on the chart, Add Charge Invoice included.
The caveat
Small cohort, kept consistent: directional results, reported as exactly that.
fig. 2.1
The scorecard’s standing rules.
03
Three calls: same tasks, revenue stakes, laggards stay visible
The benchmark’s credibility came down to three decisions.
Honest ledger
Every one of these calls trades short-term shine for long-term believability. That’s the whole product.
04
Run 1 set the baseline. Run 2 moved it. Run 3 made it a practice.
Run 1 was uncomfortable on purpose: 49%, 29%, and 20% task success on flows merchants pay for. Run 2 repeated the identical script one month and one focused fix list later, and most key tasks jumped (fig. 1). By Run 3, the scorecard was simply how release quality got judged.
On the fixes
Nothing heroic moved these numbers: an actions menu, a link where people actually looked, fewer dead ends. Placement beat invention.
Outcome coding — run 1 · Nov 2018 · 15 testers
Add charge invoice
5
2
2
6
Add credit invoice
12
3
Add subscription
5
2
8
Edit subscription
9
2
3
Cancel subscription
7
4
3
Issue refund
5
5
5
Direct success
Eventual success
Unable to complete
Perceived complete — wasn’t
fig. 4.1
Run 1, coded four ways. The rightmost band is the dangerous one: testers who believed they’d finished a revenue task, and hadn’t.
Benchmark · run 1 · Nov 2018
Outcome coding — run 2 · Mar 2019 · 11 testers
Add charge invoice
5
1
5
Add credit invoice
8
1
1
1
Add subscription
9
1
1
Edit subscription
8
3
Cancel subscription
11
Issue refund
8
1
2
Direct success
Eventual success
Unable to complete
Perceived complete — wasn’t
fig. 4.4
Run 2, same tasks, same coding. Cancel Subscription went clean — eleven of eleven direct successes. Add Charge Invoice stayed stubborn, and stayed on the chart.
Benchmark · run 2 · Mar 2019
05
The numbers moved. The practice stayed.
Reported ease of task, 1–5 — testers’ own rating
Add charge invoice
3.67→4.18
Run 1
3.67
Run 2
4.18
Add credit invoice
4.33→4.36
Run 1
4.33
Run 2
4.36
Add subscription
3.73→4.91
Run 1
3.73
Run 2
4.91
Edit subscription
4.20→5.00
Run 1
4.20
Run 2
5.00
Cancel subscription
3.53→4.73
Run 1
3.53
Run 2
4.73
Issue refund
3.33→4.09
Run 1
3.33
Run 2
4.09
fig. 5.1
Reported ease, run over run. Every task moved up; nothing that shipped got harder.
Benchmark · runs 1–2
Task success — quarterly goal line
Goal (OKR)
Actual
Run 1
Q4 2018
—
55.6%
Run 2
Q1 2019
60%
77.3%
Run 3
Q2 2019
85%
TBD
fig. 5.2
The number that outlived the study: task success became a quarterly OKR with a goal line. Run 2 beat its 60% goal at 77.3%.
Artifact · roadmap OKR
06
Let the numbers do the arguing
What worked here travels anywhere: pick tasks the business already values, hold the script constant, and let the numbers carry the argument instead of the designer. What I’d change is sequencing. I stood the baseline up alongside the first fixes; next time it comes first, so every later gain is unambiguous. The practice’s last job is running without me, wired into every release cycle and owned by the team.
There’s more to the practice than fits one page.
Closing note
The best artifact this project produced wasn’t a screen. It was a habit the org kept.