LearnProduct Judgment

Evidence, Not Encouragement

Customer discovery as evidence-gathering. Why compliments correlate with nothing, what revealed preference looks like on the page, and what a machine summary of an interview strips out.

Chapter 2 · 3 ideas · 10 checks · 14 min

Start here

Twelve interviews. Three weeks of scheduling, an hour each, notes written up the same evening. The summary doc runs to four pages and its conclusion is one line: strong validation — recommend we build.

Read the highlights. "This is exactly the kind of thing we've been looking for." "Honestly, if you had this today I'd use it." "You're solving a real problem." Nine of twelve rated the concept 4 or 5 out of 5.

The team is proud of this, and they should be. Twelve strangers gave up an hour each. That is genuinely hard to arrange and the notes are careful.

Now try to answer one question from the document: what does any of these twelve people currently do about this problem? Not what they said about your idea — what they do on a Tuesday, right now, before you existed. What they spend on it. What spreadsheet they built. Who they email.

The document does not say. Not because the team hid it, but because nobody asked, and nobody asked because every question was about the idea.

So the question this chapter answers is not "did we do enough research." Three weeks is plenty. It is: what kind of thing did these twelve conversations produce, and is it the kind that can support a decision?

Core concepts

Each idea is a short read, one picture and one quick check.

Idea 1 of 3

Evidence and encouragement

Describe your idea to a polite adult and what comes back is warm. Not because they are lying — because encouragement is what the situation calls for.

A customer conversation produces one of two things: facts about what the person has already done, or opinions about what they might do. Only the first is evidence. Compliments, hypothetical enthusiasm and "I'd definitely use that" correlate with nothing: not with future purchase, not with usage, not even with the person's own private view.

Picture it

Interview notes · clinic 4

  • “Love it. I'd definitely use this.” (struck out)a forecast
  • Rebuilds the recall list by hand every Monday. (highlighted)← something they did
  • “You're solving a real problem.” (struck out)a compliment
  • Pays £40 a month for a text tool and uses one feature. (highlighted)← money already spent
  • Offered to send us the spreadsheet. (highlighted)← something given up

3 facts. None came from the praise.

Evidence is what someone did, paid or gave up. Encouragement is what they said about your idea. A summary with nothing highlighted validated nothing, however many conversations it reports.

Check yourself

Each line is from an interview write-up. Evidence or encouragement?

  1. "Honestly, if you had this today I'd use it."

  2. She rebuilds the Monday report by hand every week. It takes about two hours.

  3. Rated the concept 5 out of 5.

  4. They pay £40 a month for a text service and use one of its five features.

  5. He offered to introduce us to their head of operations next week.

  6. "It's really frustrating, to be honest."

Go deeper

How to use it. Interrogate any discovery write-up (the summary of research done before building) for three things:

  1. Past behaviour, not predicted behaviour. Does it report what happened, or what someone says would happen?
  2. Specifics, not generalities. "It's frustrating" is a generality. "She rebuilds the report by hand every Monday morning, takes about two hours" is a specific.
  3. Something real given up. Time, an introduction, money, a scheduled follow-up with a colleague. Or only words.

How to spot it. Read only the quotes. If they are about your product, the conversation drifted to you and away from the person's life. If they are about spreadsheets, contractors, current spend and things that broke last month, the conversation stayed where it belonged.

Back to the twelve interviews. They produced encouragement. That is not a small failure to be fixed with more interviews, and this is the part most teams get wrong: twelve more of the same conversations produce twelve more compliments, because forward-looking questions have only one kind of answer. The fix is upstream of volume. It is in what gets asked.

One thing that has changed. Discovery synthesis is now routinely machine-summarised, and summarisation strips exactly the texture that distinguishes evidence from encouragement. A model will happily render "several users expressed strong interest" out of twelve polite deflections, and the sentence will be accurate as a summary and useless as evidence. Demand the underlying quotes and behaviours, not the synthesis.

Idea 2 of 3

The chain from outcome to experiment

A team has decided to build in-app messaging. Ask why and you get a fluent answer: users want better communication. Ask what else was considered and the room goes quiet. In-app messaging was not chosen from a set. It arrived, and the reason was assembled afterwards.

One structure makes this visible. It links four levels: the outcome you want to move, the opportunities that could move it (needs and pains people actually have), the solutions to each opportunity, and the experiments that test each solution. Its value is that the chain from "what we want to change" to "what we're building" becomes checkable.

Picture it

  1. Outcome

    Cut patient no-shows by 20% this quarter
  2. Opportunities

    Reminders arrive when shift workers can't read themThe recall list lives in a two-year-old spreadsheet
  3. Solutions

    Patient-chosen reminder timesTwo-way text to confirm or rebookRecall list built into the booking system

    Three options for the same opportunities. Something was compared.

  4. Experiments

    Send reminders by hand at chosen times, one clinic, two weeks
Trace any proposal up (which opportunity, which outcome?) and sideways (what else could solve the same opportunity?). One branch usually means the solution came first.

Check yourself

A product lead proposes in-app messaging "because users want better communication". Which question best shows whether it was chosen or just arrived?

Go deeper

How to use it. Trace any proposal in two directions. Upward: which opportunity does this serve, and which outcome does that opportunity move? A solution that cannot be traced up is unmoored. It may still be a good idea, but nobody can say why. Sideways: what other solutions to that same opportunity were considered?

How to spot it. Ask the sideways question out loud. Watch for surprise.

One thing that has changed. Generating the alternative branches is now nearly free, which removes the last excuse for a single-option proposal — and floods the tree with plausible options nobody has any evidence for. The scarce work moved from generating branches to pruning them against real opportunity evidence.

Idea 3 of 3

The riskiest assumption

Every plan rests on things that must be true. Most plans never list them, so nobody can tell which one, if false, takes the whole thing down.

The discipline: before building, list the assumptions. Place each one on two axes — how much it matters if we're wrong and how little we know. Test the one in the top-right corner first, with the smallest test that could actually change your mind.

Picture it

FatalMatters if wrong →Minor
KnownHow little we know →Unknown
  1. Clinics will pay £40 a month
  2. Receptionists will drop the paper list
  3. Texts can go out at any time
  4. Patients prefer texts to calls
  5. Clinics want a mobile app
  6. Shaded corner: Test first
Sana's clinic product. The top-right belief is the week's work. The bottom-left ones can wait, however interesting they are.

Check yourself

The riskiest belief is "clinics will pay £40 a month". Which test could actually come back "no"?

Go deeper

How to use it. Write the list. Mark each item evidenced or assumed. Find the one that is both load-bearing and unevidenced: that is this week's work. Then apply the second test, the one people skip: could the proposed test actually return "no"? A test with no failing outcome is theatre.

How to spot it. Look at the plan's risks section. If every entry is an execution risk (the timeline may slip, the vendor may be late) and none is a belief risk, the plan has not been examined. It has been scheduled.

One thing that has changed. Cheap building makes "just build it and see" superficially rational, and for genuinely small bets it now often is. What matters is whether the learning is cheap, not whether the building is. A week-long build that produces no interpretable signal is still a wasted week.

Worked examples

Make the call yourself first. The reasoning opens after you choose.

Example 1 of 3

The conversation that changed shape

Sana is building software for independent physiotherapy clinics. Her first eight interviews all went well and taught her nothing, which she only realised when she tried to write the summary and found she had no sentence to write. She has one more interview booked.

Check yourself

What should Sana change for interview nine?

Example 2 of 3

What the summary removed

Marcus runs product at a company selling inventory tools to small shops. His team recorded fourteen discovery calls and had a model write the synthesis. It is a good document: themes, frequencies, quotes, a clear recommendation. Theme two reads: "Users find the reorder workflow confusing (mentioned in 9/14 interviews)." The recommendation is a workflow redesign.

Check yourself

What should Marcus do before approving the redesign?

Example 3 of 3

The survey that answered a different question

A team wants to know whether to add a compliance-reporting module. They run a survey: 340 responses, 71% say they would find it valuable, 44% say they would pay extra for it. The survey was competently run and the numbers are real.

Check yourself

Can this survey support the decision to build the module?

What great operators do

The rules this chapter's scenarios test for, and how to tell when someone follows them.

Rules to keep

Before you open each card, say what you'd look for.

Common failure patterns

How smart people get this wrong. Find each one in the wild.

Trap 1 of 3

Compliment laundering

What it looks like. A discovery doc whose evidence is enthusiasm: "users loved the concept", "strong positive signal from 9 of 12 interviews". No past behaviour, no existing spend, no commitments.

Check yourself

Tap the lines that launder compliments into findings.

Discovery summary — scheduling tool, Q3

Found 0 of 2.

Why smart people do it. The conversations genuinely happened and genuinely felt good. Effort was spent, so it feels like evidence was gathered. And nobody enjoys re-reading twelve interviews to discover they learned nothing.

The correction. Re-derive the finding from behaviour only, discarding every opinion. If nothing survives, the honest write-up is "we have not yet validated demand" — which is itself a finding, and a cheap one next to finding out after the build.

Trap 2 of 3

Hypothetical validation

What it looks like. Demand established by asking about the future: survey results on willingness to pay, "would you use this", feature rankings from people who have never had the problem cost them anything.

Check yourself

Tap the lines that treat a forecast as demand.

Survey readout — compliance module

Found 0 of 2.

Why smart people do it. It scales, it produces charts, and it arrives fast. Behavioural evidence is slow and small, which looks weaker in a deck. A sample of nine feels embarrassing next to 340, even when the nine are the only ones who know anything.

The correction. Convert every forward-looking question into a backward-looking one and see how much of the case survives. Treat surveys as hypothesis generators, never as validation.

Trap 3 of 3

Premature automation

What it looks like. Automating onboarding, support or research before anyone understands the process, justified by efficiency. Increasingly: replacing customer conversations with generated summaries at exactly the point where the team was about to learn something.

Check yourself

Tap the lines that automate away the signal.

Plan — next sprint

Found 0 of 2.

Why smart people do it. Manual work feels like a failure of engineering, and automation is satisfying to build. It also removes the uncomfortable part of early-stage work: talking to people who don't want your product yet.

The correction. Automate what is understood and repetitive; protect direct contact longest. The temptation has inverted in the last few years: it is now easier to automate customer contact than to do it, so teams skip the manual phase by default and buy fluent throughput at the cost of the only signal that mattered.

Cheat sheet

The whole chapter on one card. Save it or share it.

Judgment GymCheat sheet

Evidence, Not Encouragement

  1. 1Facts about the past beat opinions about the future.Evidence is what someone already did, paid or gave up. Everything else is politeness.
  2. 2A compliment means the conversation drifted.Steer back from your idea to their last real week.
  3. 3Ask "walk me through the last time this happened."Never "would you use something that…" or "how much would you pay?"
  4. 4Workarounds and existing spend are the best cheap signal.Someone already paid to solve it. That ranks the problem.
  5. 5Trace every solution up and sideways.Which outcome does it move? What else was considered for the same need?
  6. 6Test the riskiest belief first, with a test that can say no.If it can only come back "yes", it isn't a test.
  7. 7Read the quotes behind any summary.Summaries keep the themes and drop the specifics that made it evidence.
Train the judgment AI can't do for you.@judgmentgym
Download image

Memes

The failure patterns, one picture each. For the group chat.

Your turn

Make the call

The checkpoint is the discovery memo from Start here, in full: enthusiasm quotes, themes, and a recommendation to build. You open one part of it, then decide whether to approve the build, ask for more, or fix how the research was done.

The trap. "The sample is too small" feels like the safe answer, and it is wrong. Twelve is plenty. Look at what the interviews asked: forward-looking questions, so more of them only produce more compliments.