Miró's control group size calculator offers three methods of calculating possible minimum sizes for control groups given a binary outcome. It uses functionality from Miró, which is analytical software produced by Stochastic Solutions, currently in beta.
DISCLAIMER: The functionality is provided free of charge, on an "as is" basis, and with no warranty or guarantee of any kind. Terms of service may be found here.
Briefly, the three methods are:
The Fleiss Method (Treat with Caution)
The first method uses the de facto standard approach documented by Joseph L. Fleiss in his book Statistical Methods for Rates and Proportions, 2nd Edition, Wiley (New York) 1981. This produces (up to) four sizes based on the goal of making the control group large enough to establish a "statistically significant" treatment effect with a given probability. In order to do this, the user must supply three parameters. On the assumption that the goal of the treatment is to increase the rate at which some binary outcome occurs (most commonly a purchase of some kind), these are:
The Fleiss formula assumes that the test applied after the results have been gathered with be a test for a difference in proportions between the treated and control groups. The size required depends on two further parameters, α and β.
The user does not need to choose between these values of α and β; the control group sizes produced using all four combinations of the standard values are shown.
It is to be emphasized, once again, that Stochastic Solutions does NOT recommend basing control group sizes on the Fleiss formula, at least not with the uplift that you actually expect to see, as we believe that it typically suggests groups that are too small for proper and detailed analysis of the results of campaigns. It also encourages concentration on whether there is any detectable effect, rather than on the magnitude of that effect, though it is possible to use the calculator to more useful ends. There will be a discussion of this issue posted over the coming weeks at the blog.
Miró's adjusted Fleiss size for uplift modelling
The basic Fleiss formula is based on the assumption that the goal is simply to establish statistical significance. Stochastic Solutions advocates analyzing the results of marketing campaigns to try to improve targeting of future campaigns where possible. Such analysis can be ad hoc or systematic (using some form of incremental response modelling). The adjusted Fleiss size calculation is based on the experience of attempting uplift modelling on numerous occasions and finding that, for obvious reasons, a control group size that is barely adequate for measuring uplift is almost completely useful for modelling uplift. The adjusted size generally tries to make the control population at least 10 times as large as the Fleiss size corresponding to a statistical confidence of 99% and a power of 90%. Even this is not always adequate, but acts as a useful lower bound.
This approach to control group sizing is heuristic, based on the experience of uplift modelling over roughly six years. Obviously, from an analytical perspective, until the control group reaches 50% of the population, larger is always better, so even this figure should very much be treated as a lower bound.
Miró's population-only based recommendation
In reality, estimating the likely uplift and the highest outcome rate that is likely to be seen in the control group is hard. Miró also, therefore, provides a very some simple rule of thumb estimates for sensible control group sizes based solely on the total population available. There is no "science" behind them, but plenty of experience.