Sample Size Calculator for A/B Tests for AI Recommendations Functionality

Statistical analysis for determining required sample sizes based on Monthly Active Users (MAU), coefficient of variation, expected uplift, and significance levels.

Understanding the Metrics and the Method

MAU	CV (SD/Mean)	Uplift (rel.)	n per group	n total	Days to sig.	Duration (mo)
1500	1	+5%	6,272	12,544	251	8.36
1500	1	+7%	3,200	6,400	129	4.27
1500	1	+10%	1,568	3,136	63	2.09
1500	1	+15%	697	1,394	28	0.93
1500	1	+20%	392	784	16	0.52
1500	1.5	+5%	14,112	28,224	565	18.82
1500	1.5	+7%	7,200	14,400	288	9.6
1500	1.5	+10%	3,528	7,056	141	4.7
1500	1.5	+15%	1,568	3,136	63	2.09
1500	1.5	+20%	882	1,764	36	1.18
1500	2	+5%	25,088	50,176	1004	33.45
1500	2	+7%	12,800	25,600	513	17.07
1500	2	+10%	6,272	12,544	251	8.36
1500	2	+15%	2,788	5,575	112	3.72
1500	2	+20%	1,568	3,136	63	2.09
1500	2.5	+5%	39,200	78,400	1569	52.27
1500	2.5	+7%	20,000	40,000	801	26.67
1500	2.5	+10%	9,800	19,600	393	13.07
1500	2.5	+15%	4,356	8,711	175	5.81
1500	2.5	+20%	2,450	4,900	99	3.27
2000	1	+5%	6,272	12,544	189	6.27
2000	1	+7%	3,200	6,400	96	3.2
2000	1	+10%	1,568	3,136	48	1.57
2000	1	+15%	697	1,394	21	0.7
2000	1	+20%	392	784	12	0.39
2000	1.5	+5%	14,112	28,224	424	14.11
2000	1.5	+7%	7,200	14,400	216	7.2
2000	1.5	+10%	3,528	7,056	106	3.53
2000	1.5	+15%	1,568	3,136	48	1.57
2000	1.5	+20%	882	1,764	27	0.88
2000	2	+5%	25,088	50,176	753	25.09
2000	2	+7%	12,800	25,600	384	12.8
2000	2	+10%	6,272	12,544	189	6.27
2000	2	+15%	2,788	5,575	84	2.79
2000	2	+20%	1,568	3,136	48	1.57
2000	2.5	+5%	39,200	78,400	1176	39.2
2000	2.5	+7%	20,000	40,000	600	20
2000	2.5	+10%	9,800	19,600	294	9.8
2000	2.5	+15%	4,356	8,711	131	4.36
2000	2.5	+20%	2,450	4,900	74	2.45
3000	1	+5%	6,272	12,544	126	4.18
3000	1	+7%	3,200	6,400	64	2.13
3000	1	+10%	1,568	3,136	32	1.05
3000	1	+15%	697	1,394	14	0.46
3000	1	+20%	392	784	8	0.26
3000	1.5	+5%	14,112	28,224	283	9.41
3000	1.5	+7%	7,200	14,400	144	4.8
3000	1.5	+10%	3,528	7,056	71	2.35
3000	1.5	+15%	1,568	3,136	32	1.05
3000	1.5	+20%	882	1,764	18	0.59
3000	2	+5%	25,088	50,176	502	16.73
3000	2	+7%	12,800	25,600	256	8.53
3000	2	+10%	6,272	12,544	126	4.18
3000	2	+15%	2,788	5,575	56	1.86
3000	2	+20%	1,568	3,136	32	1.05
3000	2.5	+5%	39,200	78,400	784	26.13
3000	2.5	+7%	20,000	40,000	400	13.33
3000	2.5	+10%	9,800	19,600	196	6.53
3000	2.5	+15%	4,356	8,711	87	2.9
3000	2.5	+20%	2,450	4,900	49	1.63
4000	1	+5%	6,272	12,544	95	3.14
4000	1	+7%	3,200	6,400	48	1.6
4000	1	+10%	1,568	3,136	24	0.78
4000	1	+15%	697	1,394	11	0.35
4000	1	+20%	392	784	6	0.2
4000	1.5	+5%	14,112	28,224	212	7.06
4000	1.5	+7%	7,200	14,400	108	3.6
4000	1.5	+10%	3,528	7,056	53	1.76
4000	1.5	+15%	1,568	3,136	24	0.78
4000	1.5	+20%	882	1,764	14	0.44
4000	2	+5%	25,088	50,176	377	12.54
4000	2	+7%	12,800	25,600	192	6.4
4000	2	+10%	6,272	12,544	95	3.14
4000	2	+15%	2,788	5,575	42	1.39
4000	2	+20%	1,568	3,136	24	0.78
4000	2.5	+5%	39,200	78,400	588	19.6
4000	2.5	+7%	20,000	40,000	300	10
4000	2.5	+10%	9,800	19,600	147	4.9
4000	2.5	+15%	4,356	8,711	66	2.18
4000	2.5	+20%	2,450	4,900	37	1.23
5000	1	+5%	6,272	12,544	76	2.51
5000	1	+7%	3,200	6,400	39	1.28
5000	1	+10%	1,568	3,136	19	0.63
5000	1	+15%	697	1,394	9	0.28
5000	1	+20%	392	784	5	0.16
5000	1.5	+5%	14,112	28,224	170	5.64
5000	1.5	+7%	7,200	14,400	87	2.88
5000	1.5	+10%	3,528	7,056	43	1.41
5000	1.5	+15%	1,568	3,136	19	0.63
5000	1.5	+20%	882	1,764	11	0.35
5000	2	+5%	25,088	50,176	302	10.04
5000	2	+7%	12,800	25,600	154	5.12
5000	2	+10%	6,272	12,544	76	2.51
5000	2	+15%	2,788	5,575	34	1.12
5000	2	+20%	1,568	3,136	19	0.63
5000	2.5	+5%	39,200	78,400	471	15.68
5000	2.5	+7%	20,000	40,000	240	8
5000	2.5	+10%	9,800	19,600	118	3.92
5000	2.5	+15%	4,356	8,711	53	1.74
5000	2.5	+20%	2,450	4,900	30	0.98
6000	1	+5%	6,272	12,544	63	2.09
6000	1	+7%	3,200	6,400	33	1.07
6000	1	+10%	1,568	3,136	16	0.52
6000	1	+15%	697	1,394	7	0.23
6000	1	+20%	392	784	4	0.13
6000	1.5	+5%	14,112	28,224	141	4.7
6000	1.5	+7%	7,200	14,400	72	2.4
6000	1.5	+10%	3,528	7,056	36	1.18
6000	1.5	+15%	1,568	3,136	16	0.52
6000	1.5	+20%	882	1,764	9	0.29
6000	2	+5%	25,088	50,176	251	8.36
6000	2	+7%	12,800	25,600	129	4.27
6000	2	+10%	6,272	12,544	63	2.09
6000	2	+15%	2,788	5,575	28	0.93
6000	2	+20%	1,568	3,136	16	0.52
6000	2.5	+5%	39,200	78,400	393	13.07
6000	2.5	+7%	20,000	40,000	201	6.67
6000	2.5	+10%	9,800	19,600	99	3.27
6000	2.5	+15%	4,356	8,711	44	1.45
6000	2.5	+20%	2,450	4,900	25	0.82
8000	1	+5%	6,272	12,544	48	1.57
8000	1	+7%	3,200	6,400	24	0.8
8000	1	+10%	1,568	3,136	12	0.39
8000	1	+15%	697	1,394	6	0.17
8000	1	+20%	392	784	3	0.1
8000	1.5	+5%	14,112	28,224	106	3.53
8000	1.5	+7%	7,200	14,400	54	1.8
8000	1.5	+10%	3,528	7,056	27	0.88
8000	1.5	+15%	1,568	3,136	12	0.39
8000	1.5	+20%	882	1,764	7	0.22
8000	2	+5%	25,088	50,176	189	6.27
8000	2	+7%	12,800	25,600	96	3.2
8000	2	+10%	6,272	12,544	48	1.57
8000	2	+15%	2,788	5,575	21	0.7
8000	2	+20%	1,568	3,136	12	0.39
8000	2.5	+5%	39,200	78,400	294	9.8
8000	2.5	+7%	20,000	40,000	150	5
8000	2.5	+10%	9,800	19,600	74	2.45
8000	2.5	+15%	4,356	8,711	33	1.09
8000	2.5	+20%	2,450	4,900	19	0.61
10000	1	+5%	6,272	12,544	38	1.25
10000	1	+7%	3,200	6,400	20	0.64
10000	1	+10%	1,568	3,136	10	0.31
10000	1	+15%	697	1,394	5	0.14
10000	1	+20%	392	784	3	0.08
10000	1.5	+5%	14,112	28,224	85	2.82
10000	1.5	+7%	7,200	14,400	44	1.44
10000	1.5	+10%	3,528	7,056	22	0.71
10000	1.5	+15%	1,568	3,136	10	0.31
10000	1.5	+20%	882	1,764	6	0.18
10000	2	+5%	25,088	50,176	151	5.02
10000	2	+7%	12,800	25,600	77	2.56
10000	2	+10%	6,272	12,544	38	1.25
10000	2	+15%	2,788	5,575	17	0.56
10000	2	+20%	1,568	3,136	10	0.31
10000	2.5	+5%	39,200	78,400	236	7.84
10000	2.5	+7%	20,000	40,000	120	4
10000	2.5	+10%	9,800	19,600	59	1.96
10000	2.5	+15%	4,356	8,711	27	0.87
10000	2.5	+20%	2,450	4,900	15	0.49
12000	1	+5%	6,272	12,544	32	1.05
12000	1	+7%	3,200	6,400	16	0.53
12000	1	+10%	1,568	3,136	8	0.26
12000	1	+15%	697	1,394	4	0.12
12000	1	+20%	392	784	2	0.07
12000	1.5	+5%	14,112	28,224	71	2.35
12000	1.5	+7%	7,200	14,400	36	1.2
12000	1.5	+10%	3,528	7,056	18	0.59
12000	1.5	+15%	1,568	3,136	8	0.26
12000	1.5	+20%	882	1,764	5	0.15
12000	2	+5%	25,088	50,176	126	4.18
12000	2	+7%	12,800	25,600	64	2.13
12000	2	+10%	6,272	12,544	32	1.05
12000	2	+15%	2,788	5,575	14	0.46
12000	2	+20%	1,568	3,136	8	0.26
12000	2.5	+5%	39,200	78,400	196	6.53
12000	2.5	+7%	20,000	40,000	100	3.33
12000	2.5	+10%	9,800	19,600	49	1.63
12000	2.5	+15%	4,356	8,711	22	0.73
12000	2.5	+20%	2,450	4,900	13	0.41
15000	1	+5%	6,272	12,544	26	0.84
15000	1	+7%	3,200	6,400	13	0.43
15000	1	+10%	1,568	3,136	7	0.21
15000	1	+15%	697	1,394	3	0.09
15000	1	+20%	392	784	2	0.05
15000	1.5	+5%	14,112	28,224	57	1.88
15000	1.5	+7%	7,200	14,400	29	0.96
15000	1.5	+10%	3,528	7,056	15	0.47
15000	1.5	+15%	1,568	3,136	7	0.21
15000	1.5	+20%	882	1,764	4	0.12
15000	2	+5%	25,088	50,176	101	3.35
15000	2	+7%	12,800	25,600	52	1.71
15000	2	+10%	6,272	12,544	26	0.84
15000	2	+15%	2,788	5,575	12	0.37
15000	2	+20%	1,568	3,136	7	0.21
15000	2.5	+5%	39,200	78,400	157	5.23
15000	2.5	+7%	20,000	40,000	80	2.67
15000	2.5	+10%	9,800	19,600	40	1.31
15000	2.5	+15%	4,356	8,711	18	0.58
15000	2.5	+20%	2,450	4,900	10	0.33
20000	1	+5%	6,272	12,544	19	0.63
20000	1	+7%	3,200	6,400	10	0.32
20000	1	+10%	1,568	3,136	5	0.16
20000	1	+15%	697	1,394	3	0.07
20000	1	+20%	392	784	2	0.04
20000	1.5	+5%	14,112	28,224	43	1.41
20000	1.5	+7%	7,200	14,400	22	0.72
20000	1.5	+10%	3,528	7,056	11	0.35
20000	1.5	+15%	1,568	3,136	5	0.16
20000	1.5	+20%	882	1,764	3	0.09
20000	2	+5%	25,088	50,176	76	2.51
20000	2	+7%	12,800	25,600	39	1.28
20000	2	+10%	6,272	12,544	19	0.63
20000	2	+15%	2,788	5,575	9	0.28
20000	2	+20%	1,568	3,136	5	0.16
20000	2.5	+5%	39,200	78,400	118	3.92
20000	2.5	+7%	20,000	40,000	60	2
20000	2.5	+10%	9,800	19,600	30	0.98
20000	2.5	+15%	4,356	8,711	14	0.44
20000	2.5	+20%	2,450	4,900	8	0.25
25000	1	+5%	6,272	12,544	16	0.5
25000	1	+7%	3,200	6,400	8	0.26
25000	1	+10%	1,568	3,136	4	0.13
25000	1	+15%	697	1,394	2	0.06
25000	1	+20%	392	784	1	0.03
25000	1.5	+5%	14,112	28,224	34	1.13
25000	1.5	+7%	7,200	14,400	18	0.58
25000	1.5	+10%	3,528	7,056	9	0.28
25000	1.5	+15%	1,568	3,136	4	0.13
25000	1.5	+20%	882	1,764	3	0.07
25000	2	+5%	25,088	50,176	61	2.01
25000	2	+7%	12,800	25,600	31	1.02
25000	2	+10%	6,272	12,544	16	0.5
25000	2	+15%	2,788	5,575	7	0.22
25000	2	+20%	1,568	3,136	4	0.13
25000	2.5	+5%	39,200	78,400	95	3.14
25000	2.5	+7%	20,000	40,000	48	1.6
25000	2.5	+10%	9,800	19,600	24	0.78
25000	2.5	+15%	4,356	8,711	11	0.35
25000	2.5	+20%	2,450	4,900	6	0.2
30000	1	+5%	6,272	12,544	13	0.42
30000	1	+7%	3,200	6,400	7	0.21
30000	1	+10%	1,568	3,136	4	0.1
30000	1	+15%	697	1,394	2	0.05
30000	1	+20%	392	784	1	0.03
30000	1.5	+5%	14,112	28,224	29	0.94
30000	1.5	+7%	7,200	14,400	15	0.48
30000	1.5	+10%	3,528	7,056	8	0.24
30000	1.5	+15%	1,568	3,136	4	0.1
30000	1.5	+20%	882	1,764	2	0.06
30000	2	+5%	25,088	50,176	51	1.67
30000	2	+7%	12,800	25,600	26	0.85
30000	2	+10%	6,272	12,544	13	0.42
30000	2	+15%	2,788	5,575	6	0.19
30000	2	+20%	1,568	3,136	4	0.1
30000	2.5	+5%	39,200	78,400	79	2.61
30000	2.5	+7%	20,000	40,000	40	1.33
30000	2.5	+10%	9,800	19,600	20	0.65
30000	2.5	+15%	4,356	8,711	9	0.29
30000	2.5	+20%	2,450	4,900	5	0.16

AI Recommendations A/B Test Example

Goal

Verify that AI recommendations increase average GGR per user (ARPU) compared to standard game presentation.

Test Design — Step by Step

1. Hypothesis

AI recommendations will increase average GGR/user by +20% compared to classic game list.

2. Metrics

Primary: Primary: GGR per player for 7 days after first visit in test.
Secondary: Secondary: deposit conversion, average session length, number of games played.

3. Randomization and Groups

50/50 split at user level (not session).
Control: Control: standard game presentation.
Test: Test: AI-powered game recommendations.
User is locked to their group for the duration of the experiment.

4. Sample Size and Duration

Per Miller's formula (95%/80%):

n_per group = 2(z₁₋α/₂ + z₁₋β)² · (CV/Uplift)²

For instance: MAU=8000, CV=1, Uplift=+20% → n/group = 392, n total = 784.

Recruitment duration (at 50/50 split):

Duration (mo) = 2 · n_per group / MAU ≈ 0.1 mo ≈ 3 days.

Observation window for Primary metric: 7 days.

Total calendar time: ~10 days (3 recruitment + 7 observation).

5. Thresholds and Success Criteria

Significance level α=0.05, power 80% (β=0.20).
Success criterion locked before launch: uplift ≥ +20% on Primary and p-value < 0.05.

6. Monitoring During Test

Daily: user flow into groups, basic metrics, deviation control.
Without sequential design, do not make decisions based on interim p-values.

7. Analysis Upon Completion

Check distribution; for heavy tails — bootstrap confidence intervals.
Two-sample t-test for average GGR/user over 7 days.
Report: uplift estimate, 95% CI, p-value, impact on secondary metrics.
Decision: if uplift ≥ +20% and p < 0.05 — roll out.

Important Notes

CUPED/covariates: use pre-period GGR/activity to reduce variance (lowers CV and required sample).
A/A-test: verify correct randomization and tracking.
Fixed protocol: do not change split, rules, observation windows, or promo activities during test.

Example Summary (table row)

Parameter	Real Application
MAU	8,000 players
Groups	392 per group (784 total)
Target uplift	+20% GGR
Timeline plan	~3 days recruitment + 7 days observation
Calendar time	≈ 10 days to result

Sample Size Calculator for A/B Tests for AI Recommendations Functionality

Understanding the Metrics and the Method

📊 Statistical Methodology

MAU (Monthly Active Users)

CV (SD/Mean)

Uplift (rel.)

n per group

n total

Days to significance (est.)

Duration (months)

AI Recommendations A/B Test Example

Goal

Test Design — Step by Step

Important Notes

Example Summary (table row)