LearnProduct Judgment

Reading the Numbers You're Given

Reference classes, definitions, and retention curve shape: the specific ways product metrics mislead the person reading them, and the questions that make a number mean something.

Chapter 3 · 3 ideas · 9 checks · 13 min

Start here

The slide says: Activation up 30%.

It is the third slide in a monthly review and nobody stops on it. Why would they? Thirty percent is a big number, it is pointed the right way, and the person presenting it did the work.

Two months later the company has hired two more people onto that team and shifted the quarter's roadmap toward the thing that produced the 30%. Revenue has not moved.

Here is what nobody asked in the room, and none of it is a hard question:

Thirty percent of what: a rise from 10% to 13%, or from 10% to 40%? Activation meaning what, exactly; who chose that definition and when? Compared to what: last month, last quarter, or a comparable company at this stage? And thirty percent of which users: everyone who signed up, or everyone who finished signing up, which is a smaller and much friendlier group.

The number is almost certainly true. Somebody computed it correctly. The problem is that a true number can be uninterpretable, and an uninterpretable number in a room full of busy people gets interpreted anyway.

So: what do you have to know about a number before it can support a decision?

Core concepts

Each idea is a short read, one picture and one quick check.

Idea 1 of 3

Compared to what

"Our month-two retention is 40%." Notice that you cannot tell whether that is good. Not because you lack context about the company: the sentence on its own cannot be good or bad.

A metric is uninterpretable without its reference class: the set of comparable situations you judge it against. Establishing the class is not context-setting that happens after the analysis. It is the analysis.

Picture it

Month-two retention: 40%

  • A consumer mobile app (highlighted)← excellent
  • A B2B tool people use for workunremarkable
  • Enterprise software, annual contracts (circled)an alarm

Same number. Three verdicts.

Illustrative verdicts. The number didn't change between these lines; the reference class did. Until you know which line you are on, arguing about the number is arguing about optimism.

Check yourself

A board update reads "month-two retention 40%, up from 34% last quarter". What do you need before calling it good?

Go deeper

How to use it. Before arguing about what to do about a number, establish what the number should be. Four questions, in order:

  1. Compared to what? Which products, at what stage, in what category?
  2. At what stage? A seed-stage number and a Series-C number of the same size mean different things.
  3. In what category? The category determines the whole distribution.
  4. Measured how? This one gets its own idea next, because it is where most comparisons quietly break.

How to spot it. Two failure modes are equally common and they look opposite: celebrating a mediocre number because it went up, and panicking about a perfectly normal number because it sounds low. Both come from the same place: arguing about the change without knowing the level.

One thing that has changed. Ask a model for a benchmark and you will get one, fluently, whether or not one exists. A fabricated benchmark is worse than no benchmark, because no benchmark leaves you searching and a fabricated one ends the search. A benchmark needs a citable source with a stated methodology, and if the source's definition differs from yours, you do not have a comparison. You have two numbers.

Where this comes from. The general machinery here (base rates, the outside view, why the vivid specifics of your case should move you less than you want) belongs to critical thinking as a discipline and is treated properly in that domain's course. What matters for product work is the operator-facing habit: find out what good looks like before the meeting, not during it.

Idea 2 of 3

Measured how

Two teams at the same company report engagement. One says it is up, one says it is flat. Both are honest and both are right, and the argument that follows will last forty minutes and resolve nothing.

Between a concept (engagement, productivity, activation, quality) and a number lies a definition somebody chose. That choice, not the concept, is what the number reports. So ask measured how before asking is it good.

Two things go wrong. The definition doesn't capture the concept: session count as "engagement" also rises when the product gets confusing and people have to come back. Or two compared numbers use different definitions. That is the most common way a comparison silently inverts, and it needs nobody to be careless.

Check yourself

Each is a team's definition of a goal. Could the number rise while the thing it stands for gets worse?

  1. Engagement: sessions per user per week.

  2. Support quality: tickets closed per agent per day.

  3. Collaboration: reports shared with a teammate who then opened them.

  4. Productivity: lines of code written with the AI assistant.

  5. Retention: accounts still paying three months after the trial.

Go deeper

How to use it. Ask for three things alongside any number: the definition, the denominator and the window. And where a metric is also a target, expect the definition to have been optimised against. Not fraudulently; by ordinary human attention flowing to the thing being measured. Ask who is measured on the number before trusting its rise.

How to spot it. Ask who wrote the definition and when. If the answer takes more than a minute to find, the number has been travelling on trust. When two numbers with the same name appear in one document, ask whether they were computed the same way before reasoning about the difference between them.

One thing that has changed. This is acute for anything measuring AI. "Faster" in an internal AI-productivity metric almost always means output, not outcome, and you already know from chapter one that those are different things.

Idea 3 of 3

The shape of the curve

Signups are up 25% month over month. The chart is beautiful. Everyone can see it from the back of the room.

Now ask for the same people drawn differently: take everyone who joined in January and plot what share were still active each week after. Do the same for February's group. That group, bucketed by when they joined, is a cohort, and its curve tells you what the signup chart cannot.

The clearest evidence that a product has found its market is a retention curve that flattens: a cohort whose usage stops falling and settles at some level, because a group of people found lasting value. Signup growth cannot show this, because signups can be bought.

Picture it

Share of each cohort still active

0%50%100%Wk 0Wk 3Wk 6Wk 9Wk 12✓ 38%✕ 7%
  • Product Aflattens; some people stay
  • Product Bkeeps falling; a leak

read the tail, not week three

At week three, B looks slightly better. By week twelve, A has a group of people who stay and B has almost no one. No single week's number shows this; the shape does.

Check yourself

Two products report week-twelve retention. A is at 12%, and has been level since week six. B is at 30%, down from 45% at week eight and still falling. Which is the stronger sign that people found lasting value?

Go deeper

How to use it. Look for the curve, not the total. Rising signups with a retention curve that falls toward zero describes a leaking bucket being filled faster, and the faster you fill it, the more it costs. Before doubling spend on fast signups, ask whether any cohort has stopped falling. Also check what "retained" means in this specific product: the definition is where this metric is most often quietly gamed, which is the previous idea arriving again in a new outfit.

How to spot it. Aggregate monthly actives, with no cohort chart anywhere. That single number hides decay behind acquisition and can rise for a year while the product gets worse.

One thing that has changed. Retention is harder to read for AI-native products, where novelty drives a large early usage spike that falls away regardless of fit. Telling habit from curiosity needs a longer window than teams want to wait, and a short-window dashboard will flatter almost any AI feature launch.

Back to the 30%. Run the three ideas over that slide and it dissolves. Measured how: activation is defined by someone, somewhere, and if the definition changed in the last quarter the comparison is void. Compared to what: 30% is a change, not a level, and nobody in the room knows the level. What shape: even a real activation improvement tells you nothing about whether those users stayed, which is the question the hiring decision turned on. None of that requires statistics. It requires refusing to accept a number until it arrives with the three things that make it mean something.

Worked examples

Make the call yourself first. The reasoning opens after you choose.

Example 1 of 3

The number that passed every check but one

Esther is a founder reviewing a proposal to double spend on one paid acquisition channel. The case: cost per signup on this channel is £14, against £31 across all channels. The recommendation is to move budget. She runs her checks. Compared to what? A practitioner benchmark puts comparable products at £20–40, so £14 is good. Measured how? Marketing spend divided by accounts created, both from systems she trusts. Is it settled? The channel has run eleven weeks, past the noisy start. Every answer is reassuring.

Check yourself

What should Esther ask for before moving the budget?

Example 2 of 3

The comparison that inverted

Two product teams present quarterly retention. Team A reports 62%. Team B reports 47%. The obvious reading is that A is doing better, and the planning doc proposes moving one of B's open headcount slots to A. You are asked to sign it off.

Check yourself

What do you check before signing off?

Example 3 of 3

The dashboard that was rising and wrong

A consumer app has monthly actives up and to the right for five straight quarters. The board deck leads with it, and growth spend has risen each quarter on the strength of it. Next quarter's plan raises it again. You sit on the board.

Check yourself

What do you ask for before the spend goes up again?

What great operators do

The rules this chapter's scenarios test for, and how to tell when someone follows them.

Rules to keep

Before you open each card, say what you'd look for.

Common failure patterns

How smart people get this wrong. Find each one in the wild.

Trap 1 of 2

Naked metric

What it looks like. A number offered as evidence with no reference class, no baseline, no definition and no denominator. "Activation improved 30%." "Retention is 40%." "NPS is 42."

Check yourself

Tap the numbers that can't support a decision as they stand.

Monthly product review, October

Found 0 of 2.

Why smart people do it. The number is real and the improvement may well be real. Supplying the context is extra work that can only make the number look worse, and there is rarely anyone in the room whose job is to ask for it.

The correction. Demand four things: compared to what, at what stage, in what category, measured how. Most naked metrics do not survive the fourth question.

Trap 2 of 2

Leaking bucket growth

What it looks like. Rising signups and aggregate actives presented as product/market fit, with no cohort retention curve anywhere. Growth spend is being increased on the strength of it.

Check yourself

Tap the lines that treat growth as proof that people stay.

Quarterly growth review, fitness app

Found 0 of 2.

Why smart people do it. Acquisition responds to money on a weekly timescale; retention responds to whether the product works for people, on a monthly one. The fast number captures attention, and the total genuinely is going up. Nobody has to be fooled for this to work.

The correction. Plot cohorts and look for flattening before scaling spend. If the curve has not flattened, additional acquisition spend is buying a larger version of the same leak.

Cheat sheet

The whole chapter on one card. Save it or share it.

Judgment GymCheat sheet

Reading the Numbers You're Given

  1. 1Ask "compared to what?" before "is it good?"Find what good looks like for this stage and category, from a source you can name.
  2. 2Ask how it was measured before arguing about it.The definition someone chose is what the number reports.
  3. 3Two numbers with one name may be two different measures.Check the definitions match before reasoning about the gap.
  4. 4Read the shape of the retention curve, not one week's number.A curve that flattens means some people found lasting value.
  5. 5Rising signups can hide a leaking bucket.Plot cohorts before scaling spend. Signups can be bought; staying is earned.
  6. 6Follow the denominator to what you care about.Cost per signup is not cost per customer who stays.
Train the judgment AI can't do for you.@judgmentgym
Download image

Memes

The failure patterns, one picture each. For the group chat.

Your turn

Make the call

Each practice case puts a decision behind a number. One engineering team is free and two honest proposals compete for it, or a fix plan rests on one headline conversion rate. You choose what to look at closely, then commit.

The trap. Opening the number that confirms the headline instead of the one that tests it. In every case the decisive figure sits behind a headline that looks complete, such as a cohort table behind a persuasive proposal or a traffic mix behind a conversion rate.