How to right-size survey sample without undercutting data quality

By Jason Talwar, Principal VP, Methods & Innovation

Every research team has heard some version of the same request: can we get a bigger sample?

A bigger sample size (N) can buy you more precision. It can also make a public report easier to defend in a room full of stakeholders who want a number they can trust at a glance. Still, the jump from n=100 to n=1,000 does not automatically make a study better. If the wrong people take the survey, you just paid for more bad answers.

That is why I think about sample size as an optimization problem. You are trying to get the level of confidence you need at a price that makes sense, from respondents who actually match the audience you care about. Quality, quantity, and cost move together.

Start with the decision, not the number

Most sample-size conversations begin actually begin with the research decision they are meant to support. Then later, a somewhat arbitrary N is associated with that decision. 

But the N shouldn’t be arbitrary, it should change with the job. A fast directional read does not need the same design as a pricing study. A concept test does not need the same design as a brand tracker with cuts by region, segment, and role. If a client wants regression, key driver analysis, or other advanced modeling, the sample needs to support that work from the start.

That is where researchers often oversimplify the conversation. They jump straight to margin of error and stop there.

Margin of error matters. So do confidence intervals. So does statistical power. If you need to detect a real difference between two brands, two messages, or two waves, you need enough sample to pick up the size of effect you care about. If the effect is large, you may not need a massive sample. If the effect is subtle, you may. If the analysis is more complex, the bar moves again.

That is why I do not like blanket statements about sample size. The right N is relative to the decision, the method, and the stakes.

In B2B, audience quality drives the outcome

This issue gets sharper in B2B because you rarely need a generic respondent pool. You need a very specific kind of segment.

You may need enterprise cloud buyers. You may need cybersecurity leaders with budget authority. You may need physicians who use a specific workflow. You may need CMOs in a defined region who control spend.

Once the audience gets that tight, the real challenge changes. You stop asking how many people exist in theory. You start asking whether you can reach the right people consistently, verify who they are, and field the work without compromising the quality of the data.

Most teams are trying to balance sample size and audience quality. In practice, the decision can still tilt toward N because of stakeholder expectations, public-reporting needs, timelines, budget approvals, or simple organizational politics. Those pressures do not make the tradeoff irrational. They make it important to state clearly what the additional sample buys and what quality standards the study will protect. In B2B, audience fit deserves equal weight because a larger sample of the wrong people can make the business more confident in the wrong answer.

NewtonX has built its approach around that reality. The focus sits on custom recruiting, verified professional identity, and tighter targeting for hard-to-reach audiences.

Sometimes quant is still the right answer. Sometimes it is not. If the audience is low-incidence and the client needs direction more than formal significance, AI-moderated qualitative interviews at scale can be a smarter route. You get signal faster, you spend less, and you avoid forcing a quant design that does not fit the job.

Bigger samples can help the story. They cannot fix the sample.

This is the part people tend to dance around. I do not think we need to.

A larger sample can absolutely help the story you tell, especially in a public-facing report. Readers use N as a shortcut. A headline that says you surveyed 1,000 people carries a different kind of weight than one that says you surveyed 100, even when the smaller sample includes better respondents. That is real. It affects how research gets bought, read, and repeated.

Still, that does not change the underlying truth. A bigger sample does not repair weak inputs. It just gives weak inputs more surface area.

I had been pitched on the quality of panel data again and again. Then I looked at the results and kept finding the same problems: data-quality issues and numbers that did not line up with what we were seeing in the market. I got frustrated enough to dig into the data myself. At Tableau, I brought the evidence to Salesforce, showed the gap, and helped make the case for an additional $500,000 PO to use NewtonX exclusively for sample. The goal was not to minimize N. It was to find the right balance: enough sample to support the analysis, respondents we could trust, and a cost that made sense.

That experience shaped how I think about this category. Sample decisions can get political. Teams defend old vendors. Buyers defend old work. Large N values feel safe. None of that changes the quality of the data. If the wrong respondents make it into the study, they do not just add a little noise. They can push the entire conclusion in the wrong direction.

What I would ask before increasing N

Before I approve a larger sample, I would want clear answers to a few questions.

1. What business decision hangs on the result?

A high-stakes product, pricing, or brand decision deserves a different level of rigor than a quick pulse check.

2. What analysis do we need the data to support?

If the team wants heavy segmentation, regression, or key driver work, the sample plan needs to reflect that upfront.

3. Are we trying to prove significance or get directional signal?

Both can be useful. They are not the same thing.

4. Are we reaching the exact people who matter?

A real person and a qualified person are not interchangeable.

5. What are we willing to pay for added precision?

Extra completes only make sense when they improve the decision enough to justify the cost.

The goal is a smarter sample, not a larger one by default

Research teams do not win by maxing out N on every project. They win by matching the sample to the question.

That means defining the audience carefully, choosing the right method, setting the right statistical standard, and spending where the added precision actually matters. Sometimes that leads to a larger sample. Sometimes it leads to a smaller, tighter one. Sometimes it leads away from quant altogether.

What matters is whether the study gives the business a trustworthy answer. That is the standard I care about, and it is the one more teams should use when they talk about sample size.