A Bootstrap Proportion Test for Brand Lift Testing (Liu et al., 2023)

Description

This function generates binomial random samples for the control group (with sample size n_1 and success probability p_1) and the treatment group (with sample size n_2 and success probability p_2).

Usage

gen.simu.data(n1, n2, p1, p2, summary=TRUE)

Arguments

Details

The a 2x2 contingency table is of the following form

Value

A list of simulated data for the control group and the treatment group if summary=FALSE or a 2x2 contingency table if summary=TRUE

Examples

n1 <- 100; n2 <- 100; p1 <- 0.1; p2 <- 0.2
set.seed(1)
sim.data <- gen.simu.data(n1, n2, p1, p2)
sim.data

A Bootstrap Proportion Test for Brand Lift Testing (Liu et al., 2023)

Description

This function generates the asymptotic power of the proposed bootstrap test. Two methods are provided: the asymptotic power based on the relative lift and the asymptotic power the absolute lift. For more details, please refer to the paper Liu et al., (2023).

Usage

get.asymp.power(n1, n2, p1, p2, method='relative', alpha=0.05)

Arguments

Details

Let N = n_1 + n_2 and \kappa = n_1/N. We define

\sigma_{a,n} = \sqrt{n_1^{-1}p_1(1-p_1) + n_2^{-1}p_2(1-p_2)},

\bar\sigma_{a,n} = \sqrt{(n_1^{-1} + n_2^{-1})\bar p(1-\bar p)}.

where \bar p = \kappa p_1 + (1-\kappa) p_2. \sigma_{a,n} is the standard deviation of the absolute lift and \bar\sigma_{a,n} can be viewed as the standard deviation of the combined sample of the control and treatment groups. Let \delta_a = p_2 - p_1 be the absolute lift. The asymptotic power function based on the absolute lift is given by

\beta_{Absolute}(\delta_a) \approx \Phi\left( -cz_{\alpha/2} + \frac{\delta_a}{\sigma_{a,n}} \right) + \Phi\left( -cz_{\alpha/2} - \frac{\delta_a}{\sigma_{a,n}} \right).

The asymptotic power function based on the relative lift is given by

\beta_{Relative}(\delta_a) \approx \Phi \left( -cz_{\alpha/2} \frac{p_0}{\bar p} + \frac{\delta_a}{\sigma_{a,n}} \right) + \Phi \left( -cz_{\alpha/2} \frac{p_0}{\bar p} - \frac{\delta_a}{\sigma_{a,n}} \right),

where \Phi(\cdot) is the CDF of the standard normal distribution N(0,1), z_{\alpha/2} is the upper (1-\alpha/2) quantile of N(0,1), and c = {\bar\sigma_{a,n}}/\sigma_{a,n}.

Value

Return the asymptotic power

References

Wanjun Liu, Xiufan Yu, Jialiang Mao, Xiaoxu Wu, and Justin Dyer. 2023. Quantifying the Effectiveness of Advertising: A Bootstrap Proportion Test for Brand Lift Testing. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM ’23)

Examples

n1 <- 100; n2 <- 100; p1 <- 0.1; p2 <- 0.2
get.asymp.power(n1, n2, p1, p2, method='relative')

A Bootstrap Proportion Test for Brand Lift Testing (Liu et al., 2023)

Description

This function generates the minimum sample size required to obtain a statistically significant result for a given power. For more details, please refer to the paper Liu et al., (2023).

Usage

get.min.size(p1, p2, p_treat, method='relative', power=0.8, alpha=0.05)

Arguments

Details

The minimum required sample size is approximated by the asymptotic power function. Let N = n_1 + n_2 and \kappa = n_1/N. We define

\sigma_{a,n} = \sqrt{n_1^{-1}p_1(1-p_1) + n_2^{-1}p_2(1-p_2)},

\bar\sigma_{a,n} = \sqrt{(n_1^{-1} + n_2^{-1})\bar p(1-\bar p)}.

where \bar p = \kappa p_1 + (1-\kappa) p_2. \sigma_{a,n} is the standard deviation of the absolute lift and \bar\sigma_{a,n} can be viewed as the standard deviation of the combined sample of the control and treatment groups. Let \delta_a = p_2 - p_1 be the absolute lift. The asymptotic power function based on the absolute lift is given by

\beta_{Absolute}(\delta_a) \approx \Phi\left( -cz_{\alpha/2} + \frac{\delta_a}{\sigma_{a,n}} \right) + \Phi\left( -cz_{\alpha/2} - \frac{\delta_a}{\sigma_{a,n}} \right).

The asymptotic power function based on the relative lift is given by

\beta_{Relative}(\delta_a) \approx \Phi \left( -cz_{\alpha/2} \frac{p_0}{\bar p} + \frac{\delta_a}{\sigma_{a,n}} \right) + \Phi \left( -cz_{\alpha/2} \frac{p_0}{\bar p} - \frac{\delta_a}{\sigma_{a,n}} \right),

where \Phi(\cdot) is the CDF of the standard normal distribution N(0,1), z_{\alpha/2} is the upper (1-\alpha/2) quantile of N(0,1), and c = {\bar\sigma_{a,n}}/\sigma_{a,n}.

Given a power (say power=0.80), it is difficult to get a closed form of the minimum sample size. Note that when \delta_a > 0, the first term of the power function dominates the second term, so we can ignore the second term and derive the closed form for the minimum sample size. Similarly, when \delta_a < 0, the second term of the power function dominates the first term, so we can ignore the first term. In particular, the closed form for the minimum sample size is given by

N_{Relative} = \left( \frac{p_1(1-p_1)}{\kappa} + \frac{p_2(1-p_2)}{(1-\kappa)} \right) \left( \Phi^{-1}(\beta)p_1/\bar p + cz_{\alpha/2} \right)^2 / \delta_a^2,

N_{Absolute} = \left( \frac{p_1(1-p_1)}{\kappa} + \frac{p_2(1-p_2)}{(1-\kappa)} \right) \left( \Phi^{-1}(\beta) + cz_{\alpha/2} \right)^2 / \delta_a^2.

Value

Return the required minimum sample size. This is the total sample size of control group + treatment group

References

Wanjun Liu, Xiufan Yu, Jialiang Mao, Xiaoxu Wu, and Justin Dyer. 2023. Quantifying the Effectiveness of Advertising: A Bootstrap Proportion Test for Brand Lift Testing. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM ’23)

Examples

p1 <- 0.1; p2 <- 0.2
get.min.size(p1, p2, p_treat=0.5, method='relative', power=0.8, alpha=0.05)

A Bootstrap Proportion Test for Brand Lift Testing (Liu et al., 2023)

Description

This function implements several proportion tests that can be applied to Brand Lift Testing, including

1. \mathbf{clt}: Absolute lift based Z-test and relative lift based Z-test. The limiting distribution of Z-statistics are derived from the central limit theorem.

2. \mathbf{bootstrap}: Absolute lift based bootstrap test (BS-A) and relative lift based bootstrap test (BS-R), see Liu et al., (2023).

3. \mathbf{bootstrapmean}: Absolute lift based bootstrap mean test and relative lift based bootstrap mean test. (Efron and Tibshirani 1994).

4. \mathbf{permutation}: Absolute lift based permutation test and relative lift based permutation test. (Efron and Tibshirani 1994).

Learn more about the proportion tests in the section Details.

Usage

proportion.test(data, method, B)

Arguments

Details

\mathbf{clt}: the classic Z-test based on normal approximation. The absolute lift based Z-test is defined as

Z = \frac{\hat p_1 - \hat p_0}{\sqrt{{s_0^2}/{n_0} + {s_1^2}/{n_1}}},

and he relative lift based Z-test is defined as

Z_r = \frac{\hat p_1 / \hat p_0 - 1}{\sqrt{s_1^2/(n_1\hat p_0^2) + \hat p_1^2s_0^2/(n_0 \hat p_0^4)}},

where s_0^2 = \hat p_0(1-\hat p_0) and s_1^2 = \hat p_1(1-\hat p_1).

\mathbf{bootstrap}: the bootstrap proportion tests proposed in Liu et al., (2023), see Algorithm 1 in their paper. There are two bootstrap tests: the absolute lift based bootstrap test BS-A and the relative lift based bootstrap test BS-R. Note that this type of bootstrap test is testing whether the distribution of the control group is the same as the distribution of the treatment group. In the binomial distribution case, it is equivalent to test whether the mean of the control group is the same as the mean of the treatment group.

\mathbf{bootstrapmean} the bootstrap test to test whether the mean of the control group is the same as the mean of the treatment group. See Algorithm 16.2 of Efron and Tibshirani (1994).

\mathbf{permutation} the permutation test to test whether the distribution of the control group is the same as the distribution of the treatment group. See Algorithm 15.1 of Efron and Tibshirani (1994).

Value

A list of absolute lift, relative lift, standardized absolute lift and their corresponding p-values. Standardized absolute lift equals absolute lift divided by its standard deviation. Only absolute lift and relative lift are available for method clt.

References

Wanjun Liu, Xiufan Yu, Jialiang Mao, Xiaoxu Wu, and Justin Dyer. 2023. Quantifying the Effectiveness of Advertising: A Bootstrap Proportion Test for Brand Lift Testing. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management (CIKM ’23)

Efron, Bradley, and Robert J. Tibshirani. An introduction to the bootstrap. CRC press, 1994.

Examples

n1 <- 100; n2 <- 100; p1 <- 0.1; p2 <- 0.2
set.seed(1)
sim.data <- gen.simu.data(n1, n2, p1, p2, summary = TRUE)
result <- proportion.test(sim.data, method = "bootstrap", B = 1000)
relative.lift <- result$lift$relative
relative.lift.pval <- result$pvalue$relative