What is A/B testing?
An A/B test allows you to compare two different targeting methods, in order to determine which has performed best. Contextually the main objective is to measure the performance of the audience created through the Tinyclues platform and the client’s own selection.
Best practices for A/B Testing
Having a lot of experience conducting these types of tests with our clients, we have selected a few best practices to follow during the testing phase. The objective here is to gather results in order to draw concrete conclusions on the performance of the targeting technology.
It is important that both segments of the population (created by the two different targeting methods) receive the email at the same time. If there is a variation in the timing of the emails being sent, then other events could tamper with the results.
2. One single variable: Audiences
It is important to test using only one variable at a time. It can be tempting to want to test several variables at the same time: content of the email, object, sent date. However, testing more than one variable at a time will not let you gather result significance in order to conclude on the individual impact of each variable.
3. Data range available for each targeting method
In order to compare 2 targeting methods, both the methods need to pull from the same data range. If both methods do not draw from the same data range, the results will be incomplete and will not allow the analysis of the campaign that performed best.
4. Statistical Significance
Statistical significance is a concept that determines a certain level of confidence in the results. It allows for some comfort in knowing that the difference between the results between two segments really exists and is not just coincidental. The chi-square test is a tool that allows to measure statistical significance. It works by using the size of each population as well as the number of events with a positive outcome achieved for each population (number of conversions, clicks, etc.). In general, we consider that two methods are truly different if the chi-square test shows a statistical confidence greater than 95%.
WHEN TO USE AN A/B TEST
A/B tests are a decision-making tool that allows for analysis and comparison of two different population audience in order to determine the best performing one.
Contextually, A/B tests let you compare Tinyclues targets to targets generated via another targeting system. It is used to measure performance for the following case studies:
o Increase in revenue with the same population volume
o Increase in revenue for trade-marketing campaigns
o Increase in population volume and an increase in revenue
In the last use case, the test will incorporate not 3 but 5 segments. The last being a segment of additional volume identified by Tinyclues.