All Collections
Testing Guide
A/B Tests
A/B Test Protocol & Performance Indicators
A/B Test Protocol & Performance Indicators

Tests and protocols

Benoit Bouteille avatar
Written by Benoit Bouteille
Updated over a week ago


1.Generate audiences with each targeting method

•Create your first audience with the volume you want with Tinyclues. It is important to keep in mind that the volume of population should be between 5% and 25-30% of the population.

•Create a second target with the same amount of volume with your own targeting method.

2. Separate your extract in 3 segments (5 if you want to have the Tinyclues volume extension)

In order to allow for a more precise analysis, you will need to send emails to three (or five) segments you have identified with their own respective tracking metrics:

•Tinyclues exclusive targets
•Targets using your own targeting methods
•The common targets identified by both targeting methods (Red and Orange)


The results of the test can now be analyzed using the results of the 3 (or 5) segments. This will help answer the question of “Which targeting method performed best?”

•The Chi Square test will allow you to validate that the gap between both targets are statistically significant or not.

•Furthermore, we recommend to our clients that the analysis should take place after gathering results on several campaigns. This is so we can establish a global trend of the results.


Here are several metrics we strongly recommend, in order to analzse the results of the campaigns:

Volume Sent

Total number of customers who were sent a message.

Volume received

 The volume of potential customers that received the message. 

Open Rate (#openers / #delivered)

The open rate is an indicator that isn’t 100% reliable. An email is counted as open when an image appears in your email (pixel tracking). This can be a genuine open from the receiver, but it can also be a “technical” open depending on the webmail the person is using, the device they opened it on, or the image can even be tracked regardless if the person has opened it or not. These “ghost openers” are taken into account, as well as some cases where an opened email doesn’t result in the image appearing at all! 

Click Rate (#clickers / #delivered)

This is generally a reliable rate, as the openers (the only risky metric) are excluded from the calculation.

CTOR (#clickers / #openers)

The CTOR metric is almost always intentional (rarely accidental). The only thing that tampers with this ratio is the fact that the calculation rests on the amount of openers (as discussed with the open rate, ‘opens’ aren’t very reliable)

⇒ The comparison of the open rate and CTOR is more pertinent by comparing email domains than as a whole (Some providers like Yahoo! Or Hotmail tend to block images by default, and therefore make it difficult to get credible data. This is contrary to some other providers)

Conversion Rate (#orders / #clickers)

Ratio that measures the quality of the generated visits. This ratio can vary in regard to the attribution model that is chosen (last click first click etc.). The better the targeting is, the better the conversion rate will be.

Revenue generated by the campaign

The main objective here is to measure the increase in revenue using Tinyclues. It is important to note that the volumes must be the same, in order to accurately measure the incremental revenue. 

Mean Basket (Revenue/#orders)

One thing to note: if you have a small amount of orders (a few dozen in certain cases), an order that’s out of the ordinary can greatly change this ratio without the targeting being a factor.


Revenue per 1,000 emails sent (Revenue / Volume Sent x 1,000)). It corresponds to the rate of return of the campaign.


Damage caused by the campaign. Do you know how to give value to the fact that you lost a potential buyer?


This indicator is the only one that allows for measuring the positive impact of a campaign (revenue) and its negative impact (unsubscribers)

⇒ When the RPM, revenue/unsubscribe and the unsubscribe/clicks are better for a campaign, generally this is a strong indicator that the campaign is perceived more positively by its receivers

Did this answer your question?