Zurück

Probability vs Non-Probability Sampling Explained

Matthieu Saussaye

The sample decides what your survey can prove. Pick the wrong kind and no amount of analysis will fix it.

Probability vs non-probability sampling is the first real choice in any study: do you give every person in your population a known chance of being selected, or do you take the people you can reach and correct for the gaps? This guide covers both families, every common method with a worked example, the bias each one carries, how weighting works with real numbers, and when an online panel is perfectly fine. If you just need a well-built study fielded fast, SmartInterview handles the questionnaire, the sample and the analysis in one place.

The Short Answer

  • Probability sampling: every member of the population has a known, non-zero chance of selection (simple random, systematic, stratified, cluster, multistage). It is the only basis for a classic margin of error.

  • Non-probability sampling: selection depends on access, judgment or self-selection (convenience, quota, purposive, snowball, opt-in online panels). Selection probabilities are unknown, so accuracy depends on how well you model and correct the gaps.

  • Bias: probability samples mainly suffer from non-response and coverage gaps; non-probability samples add self-selection bias on top.

  • Online panels are non-probability, and they are fine for comparing concepts, messages or prices, tracking change over time and exploring reasons. They are weaker for estimating exact population shares.

  • Weighting aligns a sample to known population totals (age, gender, region), but it only fixes bias that correlates with the variables you weight on, and it reduces your effective sample size.

Probability vs Non-Probability Sampling at a Glance


Probability sampling

Non-probability sampling

Selection

Random, from a list or frame covering the population

By availability, quotas, referral or self-selection

Chance of selection

Known for every unit

Unknown

Needs a sampling frame

Yes (customer database, address register, employee list)

No

Margin of error

Valid design-based formula

Not valid in the classic sense; model-based "credibility intervals" instead (AAPOR)

Main bias risks

Non-response, coverage gaps in the frame

Self-selection, coverage, professional respondents, plus non-response

Typical cost and time

Higher, weeks to months for general populations

Lower, often days

Best for

Official statistics, prevalence, market sizing, satisfaction among a full customer base

Concept tests, message and price comparisons, exploratory and qualitative work, hard-to-reach groups

Method

Family

How it works

Example

Simple random

Probability

Every unit drawn at random from the frame

Draw 500 random IDs from 12,000 customers

Systematic

Probability

Every k-th unit after a random start

k = 12,000 / 500 = 24; start at 7, then 31, 55...

Stratified

Probability

Random sample inside each subgroup

Separate random draws per language region

Cluster

Probability

Randomly pick whole groups, survey everyone in them

Pick 20 of 200 stores, survey all staff there

Multistage

Probability

Random groups, then random units inside them

20 stores, then 15 random shoppers per store

Convenience

Non-probability

Whoever is easy to reach

Survey link posted on your LinkedIn page

Quota

Non-probability

Fill fixed targets per subgroup from available people

Panel survey capped at 50% women, 30% aged 18-34

Purposive

Non-probability

Researcher chooses people who fit a profile

12 hospital pharmacists for depth interviews

Snowball

Non-probability

Respondents recruit further respondents

Rare-disease patients referring others

Self-selection

Non-probability

People volunteer to answer

Feedback widget on a website

For a wider overview of how each method fits into a study design, see our guides to sampling methods and data collection methods.

How Each Approach Works, With Real Numbers

What probability sampling actually requires

Three things: a defined population, a sampling frame that lists it (or a way to reach every member, such as an address register), and a random selection mechanism. If one is missing, the sample is not a probability sample, even if a random number generator was involved somewhere.

The payoff is the margin of error. For a proportion, the 95% margin of error is 1.96 × √(p(1 − p) / n). With n = 400 and p = 0.5 (the worst case), that gives 1.96 × √(0.25 / 400) = 1.96 × 0.025 = ±4.9 points. With n = 1,000 it drops to ±3.1 points. It measures sampling error only, not bias.

Worked example: stratified sampling of a customer base

Say you have 12,000 customers: 7,200 in German-speaking Switzerland (60%), 3,600 in the French-speaking part (30%) and 1,200 in Ticino (10%). You want 600 completes. Proportional allocation gives 360, 180 and 60, drawn at random inside each region.

The problem is the 60 completes in Ticino: a margin of error of about ±12.7 points there. If you need to report Ticino on its own, oversample it (say 150), then weight it back down to 10% in the total. That is disproportionate stratification.

What non-probability sampling gives up, and what it keeps

In a non-probability sample, you do not know each person's chance of being included. People who join an online panel, click a social media link or answer a pop-up are different from those who do not, in ways you cannot fully observe. The AAPOR Task Force on Non-Probability Sampling (2013) concluded that the classic margin of error does not apply to these samples and that their accuracy depends on the assumptions used to adjust them.

The famous warning is the 1936 Literary Digest poll: about 2.4 million returned ballots, drawn largely from car registrations and telephone directories, predicted that Alf Landon would beat Franklin Roosevelt. Roosevelt won 46 of 48 states. A huge sample with a biased selection is still biased. The nuance people forget: George Gallup called the result correctly with a much smaller quota sample, which is also non-probability. Method matters less than how carefully the sample matches the population on what drives the answer.

Worked example: weighting an online panel sample

You run a concept test with 500 panel respondents. The sample is 60% women and 40% men, but your target population is 50/50. Weights are target share divided by sample share: women 0.50 / 0.60 = 0.833, men 0.50 / 0.40 = 1.25.

Group

Sample share

Target share

Weight

"Would buy" score

Women

60%

50%

0.833

70%

Men

40%

50%

1.25

50%

Total




Unweighted 62% / Weighted 60%

Unweighted, purchase intent is 0.60 × 70 + 0.40 × 50 = 62%. Weighted, it is 0.50 × 70 + 0.50 × 50 = 60%. That correction only holds if gender drives the bias. If the real bias is that panel members are more interested in new products than the average person, gender weights will not touch it.

Weighting also costs precision. Kish's approximation for the design effect is 1 + CV², where CV is the coefficient of variation of the weights. Here the weights have a CV of about 0.20, so the design effect is about 1.04 and the effective sample is 500 / 1.04 ≈ 481. With heavier weighting (CV of 0.6), the design effect is 1.36 and 500 respondents behave like about 368. Pew Research Center's 2018 study on weighting opt-in samples found that which variables you weight on matters more than the statistical method you use.

How to Choose and Build Your Sample, Step by Step

1. Write the decision and the number it depends on

"Which of three packaging designs do we launch?" depends on a relative comparison. "How many Swiss households would buy this?" depends on an absolute share. Comparisons tolerate non-probability samples well, because the same biases hit every concept. Absolute shares do not.

2. Check whether a frame exists

If you are surveying your own customers, employees or members, you have a frame: use it, and draw randomly or invite everyone. This is the cheapest probability sample available to most companies, and it is why customer satisfaction surveys and NPS programs can be statistically clean. If there is no frame (all Swiss adults who drink plant milk), you will almost always use a panel.

3. Define quotas or strata on what drives the answer

Age, gender and region are the defaults, but for a supplement concept, prior use of supplements may matter more. A quota on the wrong variable gives the right demographics and the wrong opinions. Example screener and quota setup:

  • S1. Have you bought any food supplement in the last 12 months? Yes / No / Not sure (No and Not sure: screen out).

  • S2. Which age group do you belong to? 18-34 / 35-54 / 55+ (quota: 30% / 35% / 35%).

  • S3. Where do you live? German-speaking Switzerland / French-speaking Switzerland / Ticino (quota: 65% / 27% / 8%).

4. Set the sample size from the smallest group you will report

If you will compare 18-34 against 55+, each needs enough completes on its own. 100 per cell gives roughly ±10 points in the probability case; 200 gives about ±7. For monadic concept tests, plan the cell size per concept, not the total; our guide to monadic and sequential concept testing shows the arithmetic.

5. Add quality checks and benchmark questions

Include one attention check, flag straight-liners in grids, and add one or two questions with a known answer from official statistics (for example household size from the Federal Statistical Office). If your sample is far off on those, it is probably off elsewhere too. More on spotting skew in our survey bias guide.

6. Weight, then report honestly

Weight to population totals on the variables you quota'd on, check the design effect, and say "based on 500 online panel respondents, weighted by age, gender and region" rather than quoting a margin of error you cannot justify.

7. Doing it in SmartInterview

In SmartInterview, you describe the objective and the AI drafts the full questionnaire, screener included, with a quality score before launch. You choose the source: your own customer list through a reusable link (your frame, tagged by source) or an audience recruited in Switzerland and Europe, or both in the same study so you can compare customers and non-customers. Screen-outs and quotas are enforced live. When an open answer is vague, the AI asks a follow-up; in our studies that gives up to 4x longer answers and 2.2x more topics. Open answers are coded into themes linked to verbatims, with sentiment and cross-tabs by quota group, and export to Excel or SPSS for weighting. Pharmalp, a Swiss natural health brand, used this setup to validate pricing and positioning of four supplement concepts; see the Pharmalp case.

Where I draw the line

I use a recruited panel without hesitation when the study compares options: which concept, which claim, which price point wins, and why. Every concept is seen by the same kind of people, so self-selection mostly cancels out, and the open answers tell you what the numbers cannot.

Where I stop is when someone takes "34% would buy" from an opt-in panel and multiplies it by the population of Switzerland to size a market. That number was never an estimate of the population; it is a comparison point. If the decision rests on an absolute share, use your own customer frame, a probability-based panel, or calibrate against real sales data before anyone puts it in a business plan.

Matthieu Saussaye, Founder of SmartInterview.

Practical Checklist Before You Field

  1. Name the population precisely: "Swiss residents aged 18-75 who bought a supplement in the past year", not "consumers".

  2. Decide whether you need an absolute estimate or a comparison. Only the first one requires probability sampling.

  3. Look for a frame you already own: customer list, CRM, member database.

  4. Pick quota or strata variables that relate to the answer, not only demographics.

  5. Size every subgroup you plan to report, aiming for at least 100 per cell.

  6. Write a strict screener and put screen-out questions first.

  7. Add an attention check and one benchmark question with a known population value.

  8. Plan the weighting and source the population totals before fieldwork.

  9. Write the methods note: source, dates, n, weighting variables, and whether a margin of error applies.

Common Mistakes

  • Quoting a margin of error for an opt-in sample. AAPOR advises against it unless you use and explain a model-based measure.

  • Confusing quota with stratified sampling. Both use subgroup targets, but only stratified sampling selects randomly inside each group.

  • Believing a big sample fixes bias. The Literary Digest had 2.4 million responses and still got it wrong.

  • Weighting on variables that do not drive the answer, then assuming the bias is gone.

  • Extreme weights. A respondent weighted at 8 counts like eight people; trim weights (a common cap is around 5) and watch the effective sample size.

  • Ignoring non-response in "random" customer surveys. A 10% response rate turns a probability sample into something close to self-selection; see why response rates are crashing.

Going further: our guides to panel research, qualitative vs quantitative research, market research methods, survey analysis tools and Pollfish alternatives. Plans and respondent volumes are on the pricing page.

Frequently Asked Questions

What is the main difference between probability and non-probability sampling?

In probability sampling, every member of the population has a known, non-zero chance of being selected, usually by random draw from a list. In non-probability sampling, selection depends on availability, quotas, referrals or self-selection, so those chances are unknown. That difference decides whether a classic margin of error is valid.

Is an online survey panel a probability sample?

Most commercial online panels are opt-in, so they are non-probability samples, even when quotas make them look representative. A few panels recruit members by random sampling of addresses or phone numbers (for example Pew Research Center's American Trends Panel); those count as probability-based.

Which is better, probability or non-probability sampling?

Neither is better in general. Probability sampling is better for estimating population shares such as prevalence or market size. Non-probability sampling is faster and cheaper, and works well for comparing concepts, messages and prices, for exploratory and qualitative research, and for hard-to-reach groups.

Can you calculate a margin of error for a non-probability sample?

Not in the classic design-based sense, because selection probabilities are unknown. AAPOR recommends model-based alternatives such as Bayesian credibility intervals, clearly labelled with the assumptions behind them.

Is quota sampling the same as stratified sampling?

No. Both divide the population into subgroups with target sizes, but stratified sampling draws randomly within each subgroup from a frame, while quota sampling fills each target with whoever is available. Stratified is probability sampling; quota is not.

What are the four main types of probability sampling?

Simple random, systematic, stratified and cluster sampling. Multistage sampling combines them, for example randomly selecting regions, then stores, then shoppers within each store.

When is convenience sampling acceptable?

For pilot tests of a questionnaire, early exploratory research and qualitative interviews where you need ideas rather than estimates. It should not be used to report what share of a population thinks or does.

Want to run a well-sampled study this week, with your own customers, a Swiss and European audience, or both? SmartInterview is free for your first 50 respondents.

Table of Contents

Verwandte Artikel

Kostenlos anmelden

Kostenlos anmelden

Kostenlos anmelden