Overview
In today’s dynamic business landscape, marketing strategies have become increasingly complex and are constantly evolving. It has become very difficult to confidently differentiate between strategies & changes that truly drive positive outcomes and those that yield coincidental results. Yet, marketing budgets are often limited, demanding
judicious allocation of resources.
Incrementality testing provides an evidence-based approach to identify changes & strategies that provide the best return on investment. Businesses can focus their efforts on those that demonstrably contribute to their goals, avoiding wastage on ineffective tactics. By continually conducting these tests, businesses can accumulate a wealth of insights about their marketing campaigns, the effectiveness of different strategies, and how these evolve over time. These insights contribute to the development of more informed and refined long-term marketing strategies and provide concrete evidence of the impact of changes, making it easier to communicate the rationale behind decisions to internal teams, clients, investors, and other stakeholders.
Creating a proper incrementality test is a task fraught with challenges and intricacies. The nuances lie not only in the technical implementation but also in the strategic thinking that underpins the entire process. From defining suitable control groups to accurately tracking user behaviour, every step demands a meticulous approach to ensure the reliability and validity of the test results. In the face of uncertainties like data bleed and external noise, this in depth case study showcases the strategic decisions necessary to arrive at a test design that maximises insights while minimising confounding factors.
The Challenge
Hypus partnered with Client X, a highly successful British premium watch brand in mid 2021. Tasked with overhauling their digital marketing efforts, Hypus quickly identified the paid social channel as an area for short & medium term focus – given Client X’s unique offering, compelling brand story and library of high quality creative assets.
Despite success on paid social channels having eluded Client X previously, Hypus were very confident that with skillful management Client X could create a portfolio of paid social campaigns that could both run and be scaled quickly at a profitable CAC (customer acquisition cost). With immediate results and a platform-reported CAC that was clearly profitable,
Hypus advised Client X to allocate substantially more budget to the campaigns in the knowledge that efficiency improvements could counterbalance any natural increases in the CAC brought about by scaling. Over a period of 18 months,
spend on paid social increased massively, all while the platform-reported CAC was being maintained at the profitable level – much to the delight of all involved.
Despite all stakeholders being very comfortable in the knowledge that the paid social campaigns were very profitable, there still remained a nagging uncertainty. With notable product launches, favourable press coverage and a diverse advertising strategy – including TV ads and professional sports team sponsorship- no one was under any illusions that the platform-reported CAC, relying on very favourable attribution models, was in any way an accurate reflection of the state of play. The time was now right to build an incrementality testing framework that would more accurately ascertain the true CAC of the UK paid social campaigns, crucial for the next phase of investment in the channel.
The Solution
With deep experience running incrementality tests, Hypus were acutely aware that creating an effective incrementality test goes far beyond the surface, involving a profound comprehension of both the technical landscape and the underlying business dynamics.
A/B Test Design
It was clear that the best design for the testing was an A/B model with the creation of two distinct groups – one exposed to the paid social marketing campaigns (the treatment group) and the other not exposed (the control group). This would allow for a reliable benchmark to be established by which the effectiveness of the paid social campaigns could be assessed.
A & B
Creating two UK population groups, one to be targeted (treatment group) and the other to be suppressed (control group) raised the all important question: what should be the segregating feature/s? We needed a segregation that was possible within the paid social advertising platform, allowing us to target only one of the groups. Additionally, we required the same segregation feature to be applicable within Client X’s backend data, meaning we could assess the relative performance of the two groups. It soon became apparent that only a geographical split of the UK population met both criteria, but that there were some challenges to overcome.
Bleed & Noise
Despite there being location targeting settings in the chosen advertising platform it remained unclear both how the platform defined a user’s location, and how well it could identify the user’s location. Additionally, with movement between locations, a person could well see an ad in a targeted location and then subsequently purchase an item and have it delivered to a non-targeted location or vice versa.. All this could potentially lead to a “bleed” between the groups, where someone targeted would be identified as belonging to the non-targeted group and vice versa – bringing the behaviour of the two groups closer together than they ought to be.
Another point to consider was the potential for “noise” – whereby a particular area within either group was disproportionately affected by a certain factor leading to unbalanced performance data. In the case of Client X, a good example of this would be the area within close proximity of the professional sports team that they sponsored; disproportionately affected by their exposure and feelings towards the team and its
sponsors. The best way to resolve the “noise” issue would be to randomly split the entire UK into small areas and assign them to group A or B, but while it would be useful in negating the noise problem it would certainly compound the “bleed” problem by creating many, many more borders between the two groups.
After much thought it was decided that the best option would be to establish a single border between the two groups – splitting the UK into two, geographically – that was likely to have the least amount of “bleed”. What we were looking for was to draw up a virtual border that ran through areas with low population density and that had few commuter routes. Once establishing the virtual border we would take steps to mitigate the “noise” by analysing different regions within each group and comparing their performance to understand what was “noise- affected”, and to what degree.
Drawing a line
Our next step was to investigate UK population density. Using a population density map map 1 we began work on drawing up potential virtual borders for the two groups. We narrowed down the options to line “a” and “b” in the illustration below.
Whilst line “a” is a cleaner line with likely less “bleed”, only 9% of Client X’s sales were North of the line. This would mean that the testing would have to run for a very long time to get meaningful results. Line “b”, with a sales split of 60% to the south and 40% to the North, presented an opportunity to run the test for a much shorter period for conclusive results. Being far enough North and West of London to avoid the commuter areas, whilst being just far enough from the West Midlands metropolitan areas, we felt satisfied that this was the best virtual border for our project. We would suppress the areas to the North in our advertising campaigns, whilst maintaining advertising to the South.
Matching area codes
We now required a simple, congruous method to implement the new segregation within both the advertising platform and Client X’s backend data. To do so, we decided to draw our virtual line using area code (letter/s that preface postcodes) borders trying our best to keep it as aligned as possible with our line “b”. We could then compile a list of area codes that we would target and a list of those that we would suppress. The illustration below shows how we managed to create Line “b” using area codes.
Negating any lingering effects
With the UK population now divided into two, geographically, we were ready to implement our incrementality test. We configured the advertising platform to target only area codes South of Line “Y” and ensured we could segment backend performance data by area code, meaning we could now compare the relevant performance of the targeted and non-targeted population groups.
In order to gain meaningful insight we decided that we would disqualify the first two months of results to negate the lingering effect of previous targeting to the Control group (pre- suppression), which would have brought many consumers into Client X’s funnel only for them to purchase after ad-targeting had already stopped
Interpreting the results
Despite being finally satisfied that the test had been designed and implemented for reliable & meaningful results, we still remained very conscious that the results would not tell us unequivocally what the CAC for the paid social campaigns was, due to “bleed” and “noise”. But what it would tell us was a broad range of the CAC and whether it was close to the platform-reported CAC or several times higher – crucial information to define the strategy around the next phase of investment in the channel.
Conclusions
In an era where data-driven decisions are paramount, the adoption of an “incre-mentality” is essential for informed decision-making in the ever-evolving marketing landscape. It enables companies to allocate resources effectively, maximise returns on investment, and ensure that marketing strategies are grounded in evidence-based data.
However, improperly executed incrementality tests can lead to reinforced misguided assumptions and contribute to misinformed decision-making. Meticulous planning and strategic thinking in designing reliable experiments is required to gain truly meaningful and valuable insight that can drive a business forward.
