Statistical significance is the likelihood that the difference in conversion rates between the variants and baseline is not due to random chance. Marpipe uses G-Test to give each insight group a confidence measure. Eg. When you have 85% confidence level, you are confident that 85 out of 100 times the results will fall between the lowest and the highest value of the confidence interval. Review this presentation to understand the basics of statistical significance within Marpipe Intelligence.
High Confidence
The range of confidence is between 80 - 100%.
The insights group has a substantial influence over the selected goal. (Eg. Image variant group has substantial influence over purchases)
The fluctuation in conversion rate is most likely not due to chance.
Continue to challenge the winners and drop low performers in subsequent tests.
Mild Confidence
The range of confidence is between 56 - 79%.
There isn’t quite enough data to find a clear outcome but the results show promise that an outlier may emerge.
The fluctuation in conversion rate may be due to chance.
Continue testing the variant group in subsequent tests.
Low Confidence
The range of confidence is between 0 - 55%.
The test cannot verify the influence over the selected goal.
The fluctuation in conversion rate is not great enough to indicate a causal effect.
Try testing variants with more substantial differences or gathering more data.
Notes
The statistical significance of the insights group would change as more data is collected.
Statistical significance would be accurately representative of the insights group after the campaign is completed.