Back

Causal Research: Definition, Designs and Examples

Matthieu Saussaye

Causal research tests whether a change in one variable produces a change in another, and by how much. It isolates a cause, such as a price, a message or a feature, by changing it deliberately for some people and not others, usually through a randomized experiment or A/B test, then comparing outcomes. It answers: if we do X, will Y change?

The short version:

  • Three conditions must hold. The cause comes before the effect, the two move together, and no third factor explains the link.

  • Randomization is the strongest tool. Randomly assigning people to versions makes the groups alike on everything except the change you made.

  • Correlation is not enough. Customers who use a feature may renew more because they were already your most engaged customers, not because of the feature.

  • Business forms are everyday: A/B tests on websites, price tests, ad holdouts, split-sample survey experiments and monadic concept tests.

  • When you cannot randomize, quasi-experiments such as matched control regions or before-and-after with a comparison group get you part of the way.

  • It needs a hypothesis. You test a specific claim, which usually comes from exploratory or descriptive work done earlier.

Causal research is the most demanding of the purpose-based types of research. This guide shows how to design one, which design to choose when a pure experiment is impossible, and how surveys can run experiments too.

Causal research designs compared

Designs differ in how confidently they rule out other explanations. Researchers call that internal validity.

Design

How it works

Confidence in cause

Business example

Main weakness

Randomized controlled experiment

People randomly assigned to treatment or control

High

Two onboarding flows, random 50/50 split

Can be slow or impossible for big decisions

A/B test (online)

Randomized experiment run on live traffic

High for the measured metric

Two checkout page versions

Short-term metrics may not reflect long-term effects

Split-sample survey experiment

Each respondent sees one randomly assigned version of a question, price or concept

High for stated reactions

Monadic price test: each group sees one price

Measures stated intent, not real purchase

Quasi-experiment

Treatment and comparison groups not randomly assigned, such as test and control regions

Medium

New price in two regions, compare with similar regions

Groups may differ in ways that also affect the outcome

Before-and-after without control

Measure, change something, measure again

Low

Sales before and after a campaign

Seasonality and other events are mixed into the effect

Observational analysis

Statistical models on existing data, controlling for known factors

Low to medium

Regression of churn on feature usage

Unmeasured confounders can still drive the result

Move down the table only when the row above is impossible, and say in your report which design you used.

The three conditions for causality

Every causal claim rests on the same three tests. Fail any one and you have an association, not a cause.

  1. Temporal precedence. The cause happens before the effect. If satisfaction scores rose before the new support team started, the team cannot be the cause.

  2. Covariation. When the cause changes, the effect changes too. Customers exposed to the new message must behave differently from those who were not.

  3. No plausible alternative explanation. Nothing else changed at the same time that could explain the difference. This is the hard one, and the reason randomization matters.

In epidemiology, where experiments are often unethical, Sir Austin Bradford Hill set out nine viewpoints in 1965 for judging whether an observed association is likely to be causal, including strength, consistency, temporality, a dose-response gradient and experimental evidence (Bradford Hill criteria). They are a useful checklist for business analysts working with observational data too.

The confounder problem, with an example

Suppose customers who attended an onboarding webinar renew more often. Three stories fit that pattern:

  • The webinar teaches people to get value, so they renew. (Causal.)

  • Customers who were already committed are the ones who sign up for webinars. (Self-selection.)

  • Larger accounts both attend webinars and renew more, because they have dedicated admins. (Confounding by account size.)

Analytics cannot separate these. Randomly inviting half of new accounts to the webinar can: if the invited half renews more, the webinar is doing the work.

How to conduct causal research in 7 steps

  1. State one hypothesis with a direction and a size. 'Showing annual pricing first increases annual-plan uptake by at least 3 points.'

  2. Define the treatment exactly. What changes, for whom, from when. Everything else stays identical.

  3. Pick one primary outcome. Decide in advance which metric decides the test. Picking the best-looking metric afterwards inflates false positives.

  4. Randomize the assignment. Use a random process, not 'odd customer IDs' or 'the Monday cohort', which can hide patterns.

  5. Size the test. Small effects need large groups. Calculate the sample needed to detect the smallest effect worth acting on before you start.

  6. Run it long enough. Cover full weekly cycles and give novelty effects time to fade.

  7. Analyze, then explain. Report the effect with a confidence interval. If the effect is real, use explanatory research to find out why it worked, which tells you what to try next.

Causal research in surveys

Surveys alone do not prove causes when you simply ask people why they did something. People rationalize. But a survey becomes a true experiment once you randomize what each respondent sees.

  • Monadic concept tests. Each respondent sees one concept only, randomly assigned. Differences in appeal between groups reflect the concepts, not the order they were shown in. See concept testing for the full method.

  • Price experiments. Randomly assign different price points to comparable groups and compare purchase intent.

  • Message tests. Show one of several headlines or claims, then ask the same outcome questions to all groups.

  • Question wording experiments. Randomly vary wording to measure how framing changes answers.

  • Conjoint analysis. Respondents choose between product profiles with randomly varied attributes, letting you estimate the effect of each attribute on choice.

The limit: survey experiments measure stated reactions. A price that wins in a survey still needs a market test before you change your price list.

Outcome questions for survey experiments

  • How likely would you be to buy this product at the price shown? (5-point scale)

  • How appealing is this offer to you? (very unappealing to very appealing)

  • How believable is this claim? (not at all to completely)

  • Which of these would you choose if you were buying today? (profiles A, B, C, none)

  • How much would you expect to pay for this? (open numeric)

Ask the same outcome questions, in the same order, to every group. Only the stimulus should differ.

A real example: the Bing headline test

Online experiments are the purest business form of causal research. In a 2017 Harvard Business Review article, Ron Kohavi and Stefan Thomke describe a 2012 idea at Microsoft's Bing to change how ad headlines were displayed. Judged low priority, it sat in the backlog for more than six months until an engineer ran a simple A/B test. Within hours the new version triggered a “too good to be true” alert. It was not a bug: the change had raised revenue by 12%, which they estimate at more than $100 million a year in the United States alone, without hurting user-experience metrics (HBR, 2017).

The lesson is not that every test wins. It is that the team would never have known the size of the effect, or trusted it, from opinions or correlations. A randomized comparison turned a hunch into a measured cause.

Causal vs explanatory vs descriptive research

The terms overlap in textbooks, and some use causal and explanatory interchangeably. In practice they do different jobs.

Descriptive

Explanatory

Causal

Question

What is happening?

Why and how does it happen?

Does X cause Y, and by how much?

Main evidence

Measurements

Patterns, drivers, mechanisms

Controlled comparison

Typical method

Survey, analytics

Driver analysis, follow-up interviews

Randomized experiment

Output

Size and profile

A plausible mechanism

An effect size with a confidence interval

Common mistakes in causal research

  • Peeking and stopping early. Checking results daily and stopping the moment a difference looks significant inflates false positives. Fix the duration or sample size in advance.

  • Testing too many things at once. If the new page changes the headline, the image and the price, a win tells you nothing about which change mattered.

  • Too small a sample. An underpowered test will often show 'no difference' even when a worthwhile effect exists.

  • Contaminated groups. If control customers hear about the treatment offer from colleagues, the comparison is no longer clean.

  • Measuring only the short term. A discount can lift this week's conversion and lower next quarter's margins. Track a downstream metric too.

  • Generalizing beyond the test. An effect measured on new website visitors in one country may not hold for existing customers elsewhere.

Advantages and limitations of causal research

Advantages

Limitations

The only design that shows what an action will do

Requires a clear hypothesis up front

Gives an effect size you can put in a business case

Can be slow, costly or politically hard to run

Removes opinion from contested decisions

Lab and survey settings may not match real life

Repeatable: the same test can be rerun

Shows that something works, not why it works

Online tests can be cheap at scale

Ethical limits on what you can randomly withhold

Find out why your winning version won

A causal test tells you which version performed better. It rarely tells you what respondents noticed, believed or disliked, which is what you need to design the next version.

SmartInterview can run split-sample survey experiments, with each respondent randomly shown one concept, price or message, and then ask open follow-up questions by voice or text with AI probing on vague answers, in the respondent's language. Open answers are coded into themes automatically, so you can compare the reasons behind each version's score.

Start free and set up your first survey experiment.

Frequently Asked Questions

What is causal research in simple terms?

Causal research tests whether one thing causes another. You change one factor for some people and not for others, ideally assigned at random, and measure whether the outcome differs. If it does, and nothing else differs between the groups, the change caused the difference.

What is an example of causal research?

An A/B test that randomly shows half of website visitors a new checkout page and compares completed purchases is causal research. So is a survey where each respondent is randomly shown one of three prices and purchase intent is compared across the groups.

What are the three conditions for causality?

The cause must come before the effect, the cause and effect must vary together, and there must be no plausible alternative explanation such as a confounding variable. Randomized experiments are designed to satisfy the third condition.

Is causal research qualitative or quantitative?

Causal research is quantitative. It relies on controlled comparisons between groups and statistical tests of the difference. Qualitative research can suggest likely causes and explain mechanisms, but it cannot measure the size of an effect.

What is the difference between causal and correlational research?

Correlational research measures whether two variables move together without changing either one. Causal research deliberately changes one variable and controls other factors, so it can show that one variable produces the change in the other.

Can a survey establish causation?

A survey can establish causation when it includes an experiment, meaning respondents are randomly assigned to different versions of a stimulus and their answers are compared. Asking respondents why they did something does not establish causation, because people often misreport their reasons.

Table of contents

Related articles

Sign up for free

Sign up for free

Sign up for free