Two related but distinct problems: properly analyzing a genuine randomized A/B test (with the statistical rigor most informal analyses skip), and recovering a causal effect from observational data where randomization wasn't possible at all — the far more common real-world scenario, and a meaningfully rarer skill to demonstrate than predictive modeling.
The core theme across both halves: a naive comparison of two groups can be actively misleading, and this project shows concretely how and why, with synthetic ground-truth data used specifically to prove each method recovers (or fails to recover) the true effect.
Dataset: Marketing A/B Testing — Kaggle A genuinely randomized experiment: users were shown either an actual advertisement or a public service announcement (PSA) as the control, with conversion as the outcome.
This part covers what a rigorous A/B test analysis actually requires, beyond just running a t-test on whatever data exists:
- Sample size / power calculation, done before looking at results — determining how much data is actually needed to detect a meaningful effect, the step most informal A/B tests skip entirely
- A two-proportion z-test with confidence intervals, the standard approach for conversion-rate experiments
- The peeking problem, made visible — repeatedly checking significance as data accumulates inflates the false-positive rate well above the nominal 5%; this project explicitly simulates that by re-running the test at increasing sample sizes and plotting how the p-value fluctuates, rather than just describing the problem in prose
Data: synthetically generated with a known, deliberately injected ground-truth treatment effect, modeling a realistic scenario: a company rolls out a program that customers opt into rather than being randomly assigned to. Higher-engagement customers are more likely to opt in — this selection bias is built into the data on purpose, because it's exactly what makes the naive comparison fail and is ubiquitous in real business settings (almost nothing in industry is actually randomized).
Three methods are compared against the known true effect:
1. Naive comparison (the wrong way) — simply comparing post-period outcomes between the two groups. Included specifically to demonstrate how badly it overstates the effect when the groups differ systematically for reasons unrelated to treatment.
2. Difference-in-Differences — compares the change over time within each group rather than comparing levels directly, which cancels out any pre-existing, time-invariant difference between the groups. Implemented both as the classic four-cell manual calculation and as a regression (spending ~ treatment + post + treatment:post), the way it's typically run and reported in practice.
3. Propensity Score Matching — estimates each customer's probability of opting in based on observed covariates (age, baseline engagement), then matches treated customers to control customers with similar propensity scores, explicitly checking covariate balance after matching as the standard diagnostic for whether the matching actually worked.
| Method | Estimate | True Effect | Error |
|---|---|---|---|
| Naive comparison | ~29.8 | 15.0 | +14.8 |
| Difference-in-Differences | ~15.1 | 15.0 | +0.1 |
| Propensity Score Matching | ~14.6 | 15.0 | -0.4 |
The naive comparison nearly doubles the true effect because it never accounts for the fact that higher-engagement customers — who would have spent more regardless — were the ones who opted in. Both DiD and PSM recover an estimate close to the true injected effect, demonstrating they're actually correcting for the selection bias, not just producing a different number by chance.
causal-inference-ab-testing/
├── data/raw/ # marketing_ab_test.csv (not committed, download from Kaggle)
├── src/
│ ├── ab_testing/
│ │ ├── power_analysis.py # Sample size + achieved power calculations
│ │ └── analysis.py # Two-proportion z-test + peeking problem simulation
│ ├── causal_inference/
│ │ ├── generate_data.py # Synthetic observational data with known true effect
│ │ ├── diff_in_diff.py # Naive comparison + DiD (manual and regression)
│ │ └── propensity_matching.py # Propensity scores, matching, balance check, ATT
│ ├── evaluation/
│ │ └── visualize.py
│ └── pipeline.py # Runs both parts end-to-end
├── notebooks/
│ └── 01_full_walkthrough.ipynb
├── tests/
│ ├── test_power_analysis.py
│ ├── test_ab_analysis.py
│ ├── test_diff_in_diff.py # Verifies DiD recovers a known, constructed true effect
│ └── test_propensity_matching.py
└── requirements.txt
git clone https://github.com/armanesh/causal-inference-ab-testing.git
cd causal-inference-ab-testing
pip install -r requirements.txt
pip install -e .Get the data: download from Kaggle and save as data/raw/marketing_ab_test.csv. The causal inference half generates its own synthetic data and needs no download.
python -m src.pipelineRuns both parts end-to-end: the A/B test analysis (power calculation, significance test, peeking-problem simulation), then the observational causal inference comparison (naive vs DiD vs PSM, each scored against the known true effect).
pytest tests/ -vThe tests for DiD and PSM are deliberately built around synthetic data with a known, exactly constructed true effect — for example, a fixture where the treated group starts 20 points higher than control and the true causal effect is exactly 10, then asserting that DiD recovers 10 (not the naive 30-point gap). This is the correct way to validate a causal inference method: not just "does it run," but "does it give the right answer on data where the right answer is known by construction."
Power analysis before the test, not just significance testing after: Skipping sample size planning leads to one of two failures — stopping too early to detect a real effect, or running far longer than needed. Including this step reflects how a properly run experiment is actually planned, not just analyzed after the fact.
Demonstrating the peeking problem rather than just naming it: It's one thing to know that repeated significance checks inflate false positives; it's another to show a concrete plot of a p-value bouncing above and below 0.05 purely from sampling noise as data accumulates, with no real underlying effect having changed.
Synthetic ground truth for the causal inference half: With real-world observational data, there's no way to know the "true" effect to check an estimate against. Generating synthetic data with a known, injected effect makes it possible to directly verify each method's accuracy — which is also why the tests can make exact, specific assertions rather than just checking that code executes without errors.
Covariate balance checking after matching: Running propensity score matching without checking whether it actually balanced the covariates is a common mistake — the diagnostic step here makes the matching's success or failure visible and auditable, not assumed.
Ali Rahbarimanesh — Data Scientist & AI Engineer LinkedIn · GitHub