This project analyses the impact of a marketing experiment (A/B test) on user conversion behaviour. The goal is to determine whether a new campaign variant (Test group) improves performance compared to the Control group.
The analysis focuses on conversion efficiency across the marketing funnel, combining SQL-based data exploration with statistical validation in Python.
A company launched a new marketing campaign (Test group) to improve conversions.
Key question:
Should the company roll out the new campaign to all users?
-
Source: Kaggle A/B Testing Dataset
https://www.kaggle.com/datasets/amirmotefaker/ab-testing-dataset?select=test_group.csv -
Data is aggregated at campaign/day level
-
Two groups:
- Control group
- Test group
- Impressions
- Clicks
- View Content
- Add to Cart
- Purchases
- Spend
- SQL (MySQL) → Data cleaning & metric calculation
- Python (statsmodels) → Statistical testing
- Tableau → Data visualisation (optional)
Conversion is defined as:
Purchase / Clicks
| Group | Clicks | Purchases | Conversion Rate |
|---|---|---|---|
| Control | 154,303 | 15,161 | 9.83% |
| Test | 180,970 | 15,637 | 8.64% |
👉 The test group generated more traffic but had a lower conversion rate.
| Stage | Control | Test |
|---|---|---|
| Impressions | 3,177,233 | 2,237,544 |
| Clicks | 154,303 | 180,970 |
| View Content | 56,370 | 55,740 |
| Add to Cart | 37,700 | 26,446 |
| Purchases | 15,161 | 15,637 |
👉 Drop-off increases significantly after clicks in the test group.
A two-proportion z-test was conducted to evaluate whether the difference in conversion rates is statistically significant.
- Z-statistic: 11.84
- p-value: < 0.001
👉 Result: The difference is highly statistically significant.
- The test group drives more clicks, indicating higher initial engagement
- However, it performs worse in downstream conversion
- The issue likely occurs post-click (landing page, UX, or targeting quality)
The test variant should NOT be rolled out.
Although it increases traffic, it significantly reduces conversion efficiency. The negative impact is statistically significant and would likely reduce overall revenue performance.
This analysis highlights an important principle:
More traffic ≠ better performance
Optimising for clicks without considering downstream conversion can harm business outcomes.
- Investigate landing page experience for the test group
- Analyse user segments (device, location, traffic source)
- Run follow-up experiments focusing on conversion optimisation
ab-testing-product-analytics/
├── data/
├── sql/
├── notebooks/
├── images/
├── docs/
├── README.md
- A/B testing methodology
- Funnel analysis
- Statistical validation
- Business decision-making using data
- End-to-end analytics workflow
Rui Cristovam