Miró Control Group Size Calculator Help

Miró's control group size calculator offers three methods of calculating possible minimum sizes for control groups given a binary outcome. It uses functionality from Miró, which is analytical software produced by Stochastic Solutions, currently in beta.

DISCLAIMER: The functionality is provided free of charge, on an "as is" basis, and with no warranty or guarantee of any kind. Terms of service may be found here.

Briefly, the three methods are:

• The Fleiss Formula for sizing control groups based on the goal of establishing a treatment effect with a given level statistical confidence given assumptions about likely outcomes (though standard, this approach to control group sizing is NOT recommended by Stochastic Solutions; it is provided simply because it is widely used as the basis for sizing control groups);
• Miró's adjusted Fleiss formula, which assumes that the goal is not mearly to establish a statistically significant effect, but to measure it and analyze its variation across a population using some form of incremental response modelling (uplift modelling);
• A simple heuristic based only on the population size, which makes no assumptions about outcome.

The Fleiss Method (Treat with Caution)

The first method uses the de facto standard approach documented by Joseph L. Fleiss in his book Statistical Methods for Rates and Proportions, 2nd Edition, Wiley (New York) 1981. This produces (up to) four sizes based on the goal of making the control group large enough to establish a "statistically significant" treatment effect with a given probability. In order to do this, the user must supply three parameters. On the assumption that the goal of the treatment is to increase the rate at which some binary outcome occurs (most commonly a purchase of some kind), these are:

• the total population available, to be randomly split into a treated group and a control group. This is normally straightforward
• the smallest uplift that we wish to be able to detect. The uplift is simply the difference in the outcome rate between the treated group and the control group. Strictly, it is measured in percentage points, but here we have simply marked it with a percentage sign. For example, if we are not interested in any result that shows an outcome rate at least one percentage point higher in the treated group than the control group, we would set this to 1%.
• The highest likely rate of the outcome in the control group. It is much easier to establish that a difference between outcome rates of 1% and 2% is significant than between 50% and 51%. So we also have to specify the highest outcome rate we expect to see in the control group.

The Fleiss formula assumes that the test applied after the results have been gathered with be a test for a difference in proportions between the treated and control groups. The size required depends on two further parameters, α and β.

• α controls the probability of false positives (so-called Type-I errors), i.e. the probability of declaring a result to be statistically significant when the treatment effect is not real. Standard levels for α are α=0.01, which corresponds to a 99% confidence level, and α=0.05, which corresponds to a 95% confidence level.
• β controls the probability of false negatives (so-called Type-II errors), i.e. the probability of failing to find a result to be statistically significant when the treatment effect is real. Standard levels for β are β=0.1, which corresponds to a power of 90% (meaning that if the effect is real, we should declare it to be statistically significant at least 90% of the time) and β=0.2, which corresponds to a power of 80%.

The user does not need to choose between these values of α and β; the control group sizes produced using all four combinations of the standard values are shown.

It is to be emphasized, once again, that Stochastic Solutions does NOT recommend basing control group sizes on the Fleiss formula, at least not with the uplift that you actually expect to see, as we believe that it typically suggests groups that are too small for proper and detailed analysis of the results of campaigns. It also encourages concentration on whether there is any detectable effect, rather than on the magnitude of that effect, though it is possible to use the calculator to more useful ends. There will be a discussion of this issue posted over the coming weeks at the blog.

Miró's adjusted Fleiss size for uplift modelling

The basic Fleiss formula is based on the assumption that the goal is simply to establish statistical significance. Stochastic Solutions advocates analyzing the results of marketing campaigns to try to improve targeting of future campaigns where possible. Such analysis can be ad hoc or systematic (using some form of incremental response modelling). The adjusted Fleiss size calculation is based on the experience of attempting uplift modelling on numerous occasions and finding that, for obvious reasons, a control group size that is barely adequate for measuring uplift is almost completely useful for modelling uplift. The adjusted size generally tries to make the control population at least 10 times as large as the Fleiss size corresponding to a statistical confidence of 99% and a power of 90%. Even this is not always adequate, but acts as a useful lower bound.

This approach to control group sizing is heuristic, based on the experience of uplift modelling over roughly six years. Obviously, from an analytical perspective, until the control group reaches 50% of the population, larger is always better, so even this figure should very much be treated as a lower bound.

Miró's population-only based recommendation

In reality, estimating the likely uplift and the highest outcome rate that is likely to be seen in the control group is hard. Miró also, therefore, provides a very some simple rule of thumb estimates for sensible control group sizes based solely on the total population available. There is no "science" behind them, but plenty of experience.