Back
Probability vs Non-Probability Sampling Explained

Matthieu Saussaye

The sample decides what your survey can prove. Pick the wrong kind and no amount of analysis will fix it.
Probability vs non-probability sampling is the first real choice in any study: do you give every person in your population a known chance of being selected, or do you take the people you can reach and correct for the gaps? This guide covers both families, every common method with a worked example, the bias each one carries, how weighting works with real numbers, and when an online panel is perfectly fine. If you just need a well-built study fielded fast, SmartInterview handles the questionnaire, the sample and the analysis in one place.
The Short Answer
Probability sampling: every member of the population has a known, non-zero chance of selection (simple random, systematic, stratified, cluster, multistage). It is the only basis for a classic margin of error.
Non-probability sampling: selection depends on access, judgment or self-selection (convenience, quota, purposive, snowball, opt-in online panels). Selection probabilities are unknown, so accuracy depends on how well you model and correct the gaps.
Bias: probability samples mainly suffer from non-response and coverage gaps; non-probability samples add self-selection bias on top.
Online panels are non-probability, and they are fine for comparing concepts, messages or prices, tracking change over time and exploring reasons. They are weaker for estimating exact population shares.
Weighting aligns a sample to known population totals (age, gender, region), but it only fixes bias that correlates with the variables you weight on, and it reduces your effective sample size.
Probability vs Non-Probability Sampling at a Glance
Probability sampling | Non-probability sampling | |
|---|---|---|
Selection | Random, from a list or frame covering the population | By availability, quotas, referral or self-selection |
Chance of selection | Known for every unit | Unknown |
Needs a sampling frame | Yes (customer database, address register, employee list) | No |
Margin of error | Valid design-based formula | Not valid in the classic sense; model-based "credibility intervals" instead (AAPOR) |
Main bias risks | Non-response, coverage gaps in the frame | Self-selection, coverage, professional respondents, plus non-response |
Typical cost and time | Higher, weeks to months for general populations | Lower, often days |
Best for | Official statistics, prevalence, market sizing, satisfaction among a full customer base | Concept tests, message and price comparisons, exploratory and qualitative work, hard-to-reach groups |
Method | Family | How it works | Example |
|---|---|---|---|
Simple random | Probability | Every unit drawn at random from the frame | Draw 500 random IDs from 12,000 customers |
Systematic | Probability | Every k-th unit after a random start | k = 12,000 / 500 = 24; start at 7, then 31, 55... |
Stratified | Probability | Random sample inside each subgroup | Separate random draws per language region |
Cluster | Probability | Randomly pick whole groups, survey everyone in them | Pick 20 of 200 stores, survey all staff there |
Multistage | Probability | Random groups, then random units inside them | 20 stores, then 15 random shoppers per store |
Convenience | Non-probability | Whoever is easy to reach | Survey link posted on your LinkedIn page |
Quota | Non-probability | Fill fixed targets per subgroup from available people | Panel survey capped at 50% women, 30% aged 18-34 |
Purposive | Non-probability | Researcher chooses people who fit a profile | 12 hospital pharmacists for depth interviews |
Snowball | Non-probability | Respondents recruit further respondents | Rare-disease patients referring others |
Self-selection | Non-probability | People volunteer to answer | Feedback widget on a website |
For a wider overview of how each method fits into a study design, see our guides to sampling methods and data collection methods.
How Each Approach Works, With Real Numbers
What probability sampling actually requires
Three things: a defined population, a sampling frame that lists it (or a way to reach every member, such as an address register), and a random selection mechanism. If one is missing, the sample is not a probability sample, even if a random number generator was involved somewhere.
The payoff is the margin of error. For a proportion, the 95% margin of error is 1.96 × √(p(1 − p) / n). With n = 400 and p = 0.5 (the worst case), that gives 1.96 × √(0.25 / 400) = 1.96 × 0.025 = ±4.9 points. With n = 1,000 it drops to ±3.1 points. It measures sampling error only, not bias.
Worked example: stratified sampling of a customer base
Say you have 12,000 customers: 7,200 in German-speaking Switzerland (60%), 3,600 in the French-speaking part (30%) and 1,200 in Ticino (10%). You want 600 completes. Proportional allocation gives 360, 180 and 60, drawn at random inside each region.
The problem is the 60 completes in Ticino: a margin of error of about ±12.7 points there. If you need to report Ticino on its own, oversample it (say 150), then weight it back down to 10% in the total. That is disproportionate stratification.
What non-probability sampling gives up, and what it keeps
In a non-probability sample, you do not know each person's chance of being included. People who join an online panel, click a social media link or answer a pop-up are different from those who do not, in ways you cannot fully observe. The AAPOR Task Force on Non-Probability Sampling (2013) concluded that the classic margin of error does not apply to these samples and that their accuracy depends on the assumptions used to adjust them.
The famous warning is the 1936 Literary Digest poll: about 2.4 million returned ballots, drawn largely from car registrations and telephone directories, predicted that Alf Landon would beat Franklin Roosevelt. Roosevelt won 46 of 48 states. A huge sample with a biased selection is still biased. The nuance people forget: George Gallup called the result correctly with a much smaller quota sample, which is also non-probability. Method matters less than how carefully the sample matches the population on what drives the answer.
Worked example: weighting an online panel sample
You run a concept test with 500 panel respondents. The sample is 60% women and 40% men, but your target population is 50/50. Weights are target share divided by sample share: women 0.50 / 0.60 = 0.833, men 0.50 / 0.40 = 1.25.
Group | Sample share | Target share | Weight | "Would buy" score |
|---|---|---|---|---|
Women | 60% | 50% | 0.833 | 70% |
Men | 40% | 50% | 1.25 | 50% |
Total | Unweighted 62% / Weighted 60% |
Unweighted, purchase intent is 0.60 × 70 + 0.40 × 50 = 62%. Weighted, it is 0.50 × 70 + 0.50 × 50 = 60%. That correction only holds if gender drives the bias. If the real bias is that panel members are more interested in new products than the average person, gender weights will not touch it.
Weighting also costs precision. Kish's approximation for the design effect is 1 + CV², where CV is the coefficient of variation of the weights. Here the weights have a CV of about 0.20, so the design effect is about 1.04 and the effective sample is 500 / 1.04 ≈ 481. With heavier weighting (CV of 0.6), the design effect is 1.36 and 500 respondents behave like about 368. Pew Research Center's 2018 study on weighting opt-in samples found that which variables you weight on matters more than the statistical method you use.
How to Choose and Build Your Sample, Step by Step
1. Write the decision and the number it depends on
"Which of three packaging designs do we launch?" depends on a relative comparison. "How many Swiss households would buy this?" depends on an absolute share. Comparisons tolerate non-probability samples well, because the same biases hit every concept. Absolute shares do not.
2. Check whether a frame exists
If you are surveying your own customers, employees or members, you have a frame: use it, and draw randomly or invite everyone. This is the cheapest probability sample available to most companies, and it is why customer satisfaction surveys and NPS programs can be statistically clean. If there is no frame (all Swiss adults who drink plant milk), you will almost always use a panel.
3. Define quotas or strata on what drives the answer
Age, gender and region are the defaults, but for a supplement concept, prior use of supplements may matter more. A quota on the wrong variable gives the right demographics and the wrong opinions. Example screener and quota setup:
S1. Have you bought any food supplement in the last 12 months? Yes / No / Not sure (No and Not sure: screen out).
S2. Which age group do you belong to? 18-34 / 35-54 / 55+ (quota: 30% / 35% / 35%).
S3. Where do you live? German-speaking Switzerland / French-speaking Switzerland / Ticino (quota: 65% / 27% / 8%).
4. Set the sample size from the smallest group you will report
If you will compare 18-34 against 55+, each needs enough completes on its own. 100 per cell gives roughly ±10 points in the probability case; 200 gives about ±7. For monadic concept tests, plan the cell size per concept, not the total; our guide to monadic and sequential concept testing shows the arithmetic.
5. Add quality checks and benchmark questions
Include one attention check, flag straight-liners in grids, and add one or two questions with a known answer from official statistics (for example household size from the Federal Statistical Office). If your sample is far off on those, it is probably off elsewhere too. More on spotting skew in our survey bias guide.
6. Weight, then report honestly
Weight to population totals on the variables you quota'd on, check the design effect, and say "based on 500 online panel respondents, weighted by age, gender and region" rather than quoting a margin of error you cannot justify.
7. Doing it in SmartInterview
In SmartInterview, you describe the objective and the AI drafts the full questionnaire, screener included, with a quality score before launch. You choose the source: your own customer list through a reusable link (your frame, tagged by source) or an audience recruited in Switzerland and Europe, or both in the same study so you can compare customers and non-customers. Screen-outs and quotas are enforced live. When an open answer is vague, the AI asks a follow-up; in our studies that gives up to 4x longer answers and 2.2x more topics. Open answers are coded into themes linked to verbatims, with sentiment and cross-tabs by quota group, and export to Excel or SPSS for weighting. Pharmalp, a Swiss natural health brand, used this setup to validate pricing and positioning of four supplement concepts; see the Pharmalp case.
Where I draw the line
I use a recruited panel without hesitation when the study compares options: which concept, which claim, which price point wins, and why. Every concept is seen by the same kind of people, so self-selection mostly cancels out, and the open answers tell you what the numbers cannot.
Where I stop is when someone takes "34% would buy" from an opt-in panel and multiplies it by the population of Switzerland to size a market. That number was never an estimate of the population; it is a comparison point. If the decision rests on an absolute share, use your own customer frame, a probability-based panel, or calibrate against real sales data before anyone puts it in a business plan.
Matthieu Saussaye, Founder of SmartInterview.
Practical Checklist Before You Field
Name the population precisely: "Swiss residents aged 18-75 who bought a supplement in the past year", not "consumers".
Decide whether you need an absolute estimate or a comparison. Only the first one requires probability sampling.
Look for a frame you already own: customer list, CRM, member database.
Pick quota or strata variables that relate to the answer, not only demographics.
Size every subgroup you plan to report, aiming for at least 100 per cell.
Write a strict screener and put screen-out questions first.
Add an attention check and one benchmark question with a known population value.
Plan the weighting and source the population totals before fieldwork.
Write the methods note: source, dates, n, weighting variables, and whether a margin of error applies.
Common Mistakes
Quoting a margin of error for an opt-in sample. AAPOR advises against it unless you use and explain a model-based measure.
Confusing quota with stratified sampling. Both use subgroup targets, but only stratified sampling selects randomly inside each group.
Believing a big sample fixes bias. The Literary Digest had 2.4 million responses and still got it wrong.
Weighting on variables that do not drive the answer, then assuming the bias is gone.
Extreme weights. A respondent weighted at 8 counts like eight people; trim weights (a common cap is around 5) and watch the effective sample size.
Ignoring non-response in "random" customer surveys. A 10% response rate turns a probability sample into something close to self-selection; see why response rates are crashing.
Going further: our guides to panel research, qualitative vs quantitative research, market research methods, survey analysis tools and Pollfish alternatives. Plans and respondent volumes are on the pricing page.
Frequently Asked Questions
What is the main difference between probability and non-probability sampling?
In probability sampling, every member of the population has a known, non-zero chance of being selected, usually by random draw from a list. In non-probability sampling, selection depends on availability, quotas, referrals or self-selection, so those chances are unknown. That difference decides whether a classic margin of error is valid.
Is an online survey panel a probability sample?
Most commercial online panels are opt-in, so they are non-probability samples, even when quotas make them look representative. A few panels recruit members by random sampling of addresses or phone numbers (for example Pew Research Center's American Trends Panel); those count as probability-based.
Which is better, probability or non-probability sampling?
Neither is better in general. Probability sampling is better for estimating population shares such as prevalence or market size. Non-probability sampling is faster and cheaper, and works well for comparing concepts, messages and prices, for exploratory and qualitative research, and for hard-to-reach groups.
Can you calculate a margin of error for a non-probability sample?
Not in the classic design-based sense, because selection probabilities are unknown. AAPOR recommends model-based alternatives such as Bayesian credibility intervals, clearly labelled with the assumptions behind them.
Is quota sampling the same as stratified sampling?
No. Both divide the population into subgroups with target sizes, but stratified sampling draws randomly within each subgroup from a frame, while quota sampling fills each target with whoever is available. Stratified is probability sampling; quota is not.
What are the four main types of probability sampling?
Simple random, systematic, stratified and cluster sampling. Multistage sampling combines them, for example randomly selecting regions, then stores, then shoppers within each store.
When is convenience sampling acceptable?
For pilot tests of a questionnaire, early exploratory research and qualitative interviews where you need ideas rather than estimates. It should not be used to report what share of a population thinks or does.
Want to run a well-sampled study this week, with your own customers, a Swiss and European audience, or both? SmartInterview is free for your first 50 respondents.


