There’s a belief that runs quietly (sometimes not so quietly) through a lot of research conversations we have had over the years: the more respondents you have, the more you can trust the data. Bigger sample, better research. It sounds mostly reasonable. However, it’s also leading a lot of teams to waste money, move slowly, and still make decisions they can’t fully defend.
Statistical relevance and rigour are not the same thing, and confusing them is one of the most expensive mistakes in product research. We know that getting statistically relevant data from qualitative interviews is basically impossible, so let’s not talk about that for now. But surveys are quant enough to be statistically relevant, right? Well… let’s unpack that.
What statistical relevance actually requires
Statistical relevance isn’t a vague aspiration. It has a specific mathematical definition, and hitting it costs more than most people realise.
Let’s consider the standard threshold most researchers cite: to get a 95% confidence level with a margin of error of ±5%, you need a minimum of 385 respondents for a large population (Cochran, 1977). That’s 385 screened (so they’re actually the right people in the population), recruited, incentivised, and completed responses. Not partial submissions. Completed, usable data.
For smaller populations or looser margins of error the numbers come down, but the point stands: genuine statistical relevance is a significant undertaking. It takes time, budget, and a level of methodological control that most commercial research projects simply don’t have.
From what we have experienced over the years, most organisations commissioning surveys are nowhere near that threshold. And many of them don’t realise what that means for the conclusions they’re drawing.
95% confidence, ±5% margin of error, assumes maximum variability (p = 0.5)
Where statistical relevance belongs
We are in no way saying statistical relevance doesn’t matter. It’s real, it’s important, and in some contexts it’s non-negotiable.
Medical research and clinical trials are designed around it. Scientific journals demand, and should demand, it before publication. Public health studies, drug efficacy research, and epidemiological surveys all depend on statistically robust samples because the stakes of being wrong are high, the findings will be scrutinised, and the decisions made from them affect large populations over a long time.
If you’re running that kind of research, this post is not for you. You need to be statistically relevant.
But most product teams aren’t running clinical trials. They’re trying to answer a more practical question: is this the right direction? Does this concept resonate? Where are users getting stuck? It would be great to answer those questions with statistically relevant data too, but it’s usually not worth the cost.
All pre-launch research is predictive
No product research tells you what will happen. It tells you what people say, think, or do right now, in their context, about a product or concept that doesn’t exist at scale yet. The market is the only real test.
A survey of 400 people telling you they would buy your product does not mean 400 people will buy your product. A survey of 50 people raising concerns about your onboarding doesn’t mean you’ll lose that proportion of users. Pre-launch research is directional. It reduces the risk of the next step, it doesn’t guarantee the outcome.
So if the job of research is to give you enough confidence to take the next step, to build a prototype, run an experiment, or launch an MVP, then the question isn’t “is this statistically relevant?” The question is “is this enough to move forward?”
Those are very different standards, and the second one is almost always cheaper and a far better return on investment.
Not statistically relevant does not mean unscientific
This is the part we want to stress.
Accepting that your survey results are directional rather than statistically definitive is not permission to cut corners, ignore bias, or report findings carelessly. The scientific method still applies. This means:
Clear objectives before you start. Know what question you’re actually trying to answer. Research that begins without a precise question usually ends without a useful answer, regardless of sample size.
Controlled, consistent methodology. Word questions neutrally. Randomise the order of options where relevant. Show the same stimuli to every participant. Sloppy methodology produces misleading data whether you have 30 respondents or 3,000.
Honest interpretation. If your sample is 60 people, frame your findings as directional. You found a strong pattern, not a verified truth. The language you use to report findings should match the strength of the evidence behind them.
Transparency about limitations. Good research names what it can’t tell you, not just what it can. A finding from a non-representative sample is still a finding. It just needs to be positioned correctly.
Rigour is about how carefully you design and run your research. Statistical relevance is about how large and representative your sample is. You can have one without the other. The goal is to have both where the context demands it, and to be honest when you’re prioritising one over the other.
Finding the balance: When bigger samples are worth the investment
There are situations in commercial research where larger samples start to earn their cost.
Segment comparisons are one. If you want to know whether 30 to 40 year olds behave differently from 50 to 60 year olds in your product, you need enough respondents in each group to detect a meaningful difference. Thin segments produce unreliable comparisons.
Brand tracking is another. If you’re measuring sentiment or awareness over time and comparing quarter on quarter, consistency and sample size both matter. Small samples produce noisy tracking data.
And anything that will be published, presented publicly, or used in a regulatory submission carries a higher standard of evidence, because others will scrutinise it.
Outside of these situations, the cost of chasing statistical relevance in a one-off project survey usually exceeds the value of the extra confidence it buys.
The better question
Most teams asking “is this statistically relevant?” are really asking “can I trust this?” Those are not the same question.
You can trust research that was designed carefully, run consistently, and reported honestly. You can’t trust a sloppy survey just because the sample was big. A well-run survey with 40 respondents, clear objectives, and neutral question design will produce more useful insight than a poorly designed survey of 1,000. And smaller, more frequent studies usually beat one massive engagement that tries to settle everything at once.
The goal isn’t statistical relevance. The goal is enough reliable signal to make a better decision than you would have made without the research, at a cost that makes sense for the product. That’s what rigour gives you, at a fraction of the price.
Reference
Based on Cochran’s sample size formula (W.G. Cochran, Sampling Techniques, 3rd ed., 1977) The best online reference for this is: https://www.tarleton.edu/academicassessment/wp-content/uploads/sites/119/2022/05/Samplesize.pdf
How Might We is a qualitative UX research studio based in Cape Town. We run hundreds of research sessions every year across finance, healthcare, retail, and services. If you are trying to work out what kind of research your project actually needs, get in touch.