The Performance Column showcases how a variant is expected to perform relative to the rest of the group for a selected goal. It is a simplified measure based on conversion rate or action rate.
The currently selected goal will determine the metric that the performance column would use to give a verdict on the performance. Eg. If CTR is selected then [clicks] will be the action rate measured against [clicks/impressions]. If the cost per add to cart is the goal then [# of adds to cart] will be the action rate measured against [# of adds to cart/impressions]. Additionally, it utilizes the confidence of the variant group to calculate a confidence interval for each variant.
Positive Performance
Expected action rate of the Variant is significantly higher than the average expected action rate of the group.
Neutral Performance
Expected action rate of the Variant is within the average expected action rate of the group.
Negative Performance
Expected action rate of the Variant is significantly less than the average expected action rate of the group.
NA Performance
The insight group has only one variant or the variants does not have results for the selected goal to calculate statistical significance.
Interpret Results using Performance and Significance
Impact | Statistical Significance | Interpretation |
Positive Performance | High Confidence | There is high confidence that this row would continue to outperform the other rows in this group |
Positive Performance | Mild Confidence | This row is looking like a winner, with more data it's expected to obtain higher confidence. This may change as the test continues running |
Positive Performance | No/Low Confidence | You'll need to collect more data but this specific row is showing signs of being a positive outlier. |
Neutral Performance | High Confidence | There is high confidence that this row would continue to perform within the average of this group. Try challenging these rows to uncover a positive outlier |
Neutral Performance | Mild Confidence | There is not enough evidence to predict that this variant would perform outside of this group's average. This may change as the test continues running. |
Neutral Performance | No/Low Confidence | You'll need to collect more data or if you think you have enough data, then this row is not differentiated from the other things being tested |
Negative Performance | High Confidence | This is high confidence that this row would continue to underperform against the other rows in this group |
Negative Performance | Mild Confidence | This is looking like a loser but that still may change if the test continues running |
Negative Performance | No/Low Confidence | You'll need to collect more data but this specific row is showing signs of being a strong negative outlier |
NA Performance | No Confidence | Nothing being tested, or no results to base a statistical analysis off of |
FAQ
How does Marpipe measure confidence in a variant group?
You can read more about statistical significance in Marpipe here
What if all rows show average performance but with low confidence?
This occurs when every variant in a group is interacted with so evenly that we cannot verify with any confidence that one singular variant performed better or worse. You can action on this in one of two ways:
Spend more money to try and find an outlier
Determine that none of the tested variants influenced the ad overall to warrant an outlier
The top performing variants are different depending on how I rank my metrics. What now?
Determine what is a more important metric for the types of ads you're running. Take CTR (Link) and Purchases as an example. These two metrics will tell you what's getting users to enter the funnel initially and how much influence the creative has on users following through to conversions. The highest CTR doesn't always attribute to the highest purchases. We've seen that sometimes users who click through on one ad aren't always the most qualified purchasers. In the case that the components are the same through the funnel, you've hit the jackpot.
My performance column is all mixed around. Top performers are below average performers. What gives?
The performance column is not a column you can rank by in Marpipe, because it's not a hard and fast metric. In the event you see a top performing variant below an average performer (or even a negative performer), remember that the performance column is measured by the action you're ranking by divided by the impressions that variant received. A variant may have received more actions, moving it to the top of the ranking; but, because it received a significantly higher amount of impressions, the action rate was brought down.