How To Properly Perform Statistically Relevant Split PPC Ad Testing
The crux of a solid paid search campaign indubitably is the ad copy that drives the user to make the click. When auditing paid search campaigns from other companies, it is always a shock to see the (lack of) ad testing that is used. An A/B split test is an extremely simple way to test a baseline ad against a new iteration, and use simple statistics and confidence testing to prove a theory with an exact confidence. This is a crucial element that we leverage with client recommendations and tactics -- as facts and science never lie. Want to see how easy it can be to calculate statistical relevancy? Follow along in the steps below.
1. Formulate a Test & Implement
In order to test, we'll need two different versions of an ad to test. More importantly, we need to identify what the success metric is. Maybe it is number of purchases, maybe it's number of clicks. For this example, let's assume that our success goal is a "sign-up" and we are testing the ad copy effects on visitors.We will start with one "championship" ad that is currently active that we'll call "Ad A". The variation will be "Ad B" with all elements the same as "Ad A", except for the ad headline.
2. Ensure You Are Collecting Accurate Data
As any statistician will tell you, the sample that you are using is key to cut down sample error. The goal is to pull our data from an unbiased sample that is representative of the population.
Ad Rotation In Google:
The way that AdWords is set up out of the gate won't serve us in the way we need, as it will try to optimize for clicks. We would rather use a systematic sampling model that will provide us equal results in our test. We recommend using "Rotate indefinitely" for your test within the campaign settings tab in AdWords:
Ad Rotation In Bing:
Like Google, Bing also automatically sets ads to show more often based on clicks. To turn this off you'll need to change each Ad Group by hopping into the Ad Group settings. You'll see the Ad Rotation option:
3. Collect Data
Ahhh, the easy part. Sit back and watch the impressions, clicks and conversions pile up. The size you'll need will depend on how accurate you'd like the test to be. Obviously the more data the better and you can see some tables on error percentages based on differing sample sizes. If you want to project a timeframe required to hit your ideal test sample size, Visual Website Optimizer has a nice free tool to help.
4. Analyze Data
If you aren't a savvy statistician, don't fret. There are plenty of calculators that can help display the data you require. I personally use Thumbtack's free A/B Test calculator for quick comparison. You'll want to insert your number of trials (visitors in this case) to the successes (sign-ups). You can adjust the confidence level if you'd like, but a 95% confidence interval is standard in stats.
The p-value displays our confidence that the improvement is in fact statistically relevant. The lower the p-value, the higher the confidence. We'd like to have a p-value of between 0.05 and 0.01 for "statistical significance." For the example above we can see a very low p-value that means the tests have reached statistical significance. Seeing that this is statistically relevant, we can then look at the results and the estimated improvement. This test shows that the variation would likely cause a decreased success rate that would actually be anywhere from a 24% - 143% decrease in conversions (with the average conversion decrease of 93%).
This example however shows that our test is currently not statistically significant as our p-value is a whopping 0.21. There is nothing that can be statistically recommended from this test. So from the two examples, we would strongly recommend keeping the Baseline ad in Test 1 as visitors were 93% more likely to convert, while we would not have any recommendations for the second test. If the success ratios held firm on the second test continue we'd need a little more than 2.5x the sample size to be statistically significant.
And that's it! Not only should this help you make more informed decisions for yourself and clients, but it also an easy method to back-up your work and tactics with science in case a client doesn't agree with your strategy..
Like this? Want more great info? Be sure to follow us on Google+:
Keep up with us on Twitter: Follow @cypressnorth
or of course Instagram and Facebook.
Meet the Author
Greg Finn
Greg is the Head of Performance and Innovation for Cypress North's digital marketing team and one of the founders of our agency. In 2010, he and Matt Mombrea started Cypress North in Buffalo. Greg oversaw the opening of our second office in 2022, located in his hometown of Rochester.
As Head of Performance and Innovation, Greg co-manages our digital marketing department and works closely with our team to ensure all our clients achieve the best possible results. He is always looking for ways to test new digital marketing techniques and technology, and oversees all teaching and training efforts to ensure our agency stays ahead of the curve.
Greg is also a co-host of our weekly Marketing O'Clock podcast, where he and the team provide updates, insights, and hot takes on the latest SEO, PPC, and social media marketing news. In addition to weekly news shows, Greg hosts our Marketing O'Torial digital marketing tutorials and often co-hosts bonus Marketing O'Talk episodes that bring together panels of digital marketing experts.
With nearly two decades of experience, Greg is a known and trusted voice in the digital marketing community. He’s a contributor for Search Engine Land, a member of the Search Marketing Expo (SMX) programming team, and has been a featured speaker at some of the largest search engine conferences, including SMX, eSummit, and Pubcon.
When he’s not working or staying updated on the latest trends, Greg enjoys watching his kids play sports and coaching their soccer team. He’s been named the runner-up “Greg of the Year” on Marketing O’Clock’s annual Clockscars Awards four years in a row. While the coveted award has evaded him for many years, Keanu Reeves has not. Greg once saw him at Gabriel's Gate tavern in Buffalo (and noted he was very tall.)
Related Resources
Google Ads Announces 3 Changes to Search Ad Creatives
Google Ads will be rolling out three changes this month to search ad creatives. We break down the three features, how to use them, and what marketers need to know.
How to Opt Out of Auto-Applied Recommendations in Google Ads
Learn more about what auto-applied recommendations are in Google Ads and how to opt out of them in your account.
Empowering Marketers: How CypresSEO Streamlines SEO Tasks
Discover how CypresSEO revolutionizes SEO for marketers, streamlining tasks and empowering your strategies. Enhance your SEO efficiency with our AI-powered tool.
Google Ads & Bidding Types: The Ultimate Source for Google Search Bidding
Learn about the different bid strategies you can use in Google, the pros and cons of each, and how to optimize them.
Google & Apple Business Profiles: A Comprehensive Overview
Having an accurate & up-to-date business profile is very beneficial to any business. Read on to learn how to set these up on Google & Apple, what each platform offers, and some things to watch out for.
B2B Marketing With the Rise of Gen Z: Navigating the Digital Native Workforce
Explore the unique challenges & opportunities of marketing to Gen Z in the B2B space. Discover how these digital natives are reshaping the professional world & what it means for marketing strategies. Stay ahead in the era of Gen Z with expert insights.
How to Create a High-Quality Email Sending List in HubSpot
The HubSpot lists tool allows you to create and edit lists of contacts, companies, or deals based on specific property values and attributes. This blog post will review the ins and outs of HubSpot lists and the different components of using and optimizing them for email marketing that works.
Should Google Ads Recommendations Guarantee Broad Match Success?
Google is being overly aggressive with recommendations for broad match. Here's what you should know.
How to Set Up Google Ads Automated Rules
Learn How To Set Up Automated Rules In Google Ads & When To Use Them In Your Account Automated rules in Google Ads are a feature that can make a digital marketer’s life a bit easier. Our agency uses automated […]
Importing Campaigns from Google Ads into Microsoft Advertising: Everything You Need to Know
Learn How To Import Campaigns Into Microsoft & Common Pitfalls To Watch Out For Building out campaigns in both Google and Microsoft Ads shouldn’t have to take up your whole day. This article will walk you through a comprehensive step-by-step […]
How to Set Up a Google Ads Campaign Experiment
Guide to Setting Up Campaign Experiments In 2022 Goodbye, drafts! Early in 2022, Google Ads overhauled their campaign experiments, making it easier for advertisers to run A/B tests within the platform. This article will outline how to set up an […]
How to Get Started with GA4: A Step-by-Step Guide
Need help setting up GA4 for your company or client’s website? Look no further! This post provides a step-by-step process for creating GA4 properties and best practices to make sure necessary events are tracked and the data flowing into GA4 are accurate.
Testing Website Accessibility: Five Testing Tools & Methods To Identify Accessibility Issues
Is your site compatible with website accessibility guidelines? Use these tools to benchmark and identify improvement opportunities. Website accessibility is often overlooked, but it shouldn’t be. According to CDC estimates, 61 million adults in the United States alone live with […]
How to Opt-Out of All Automated Google Ads Extensions
What you need to know about automated extension changes and how to turn them off. Automated extensions are one of the sneakiest forms of automation pushed on Google Ads advertisers, and they are soon going to be pushed even more. […]
23 Of The Best Resources For Starting A Digital Marketing Career
The best social media communities, podcasts, newsletters, and certifications for new digital marketers I didn’t know anything about digital marketing when I started as an intern at Cypress North earlier this year. Acronyms like SEO, CPA, and ROAS meant nothing […]
What to Include in a PPC Dashboard
Learn What Metrics to Include in PPC Reports. Then, Download Our Free Data Studio Dashboard Template! Let’s be real, pay per click advertising is all about data. What campaigns are bringing in the most revenue? What landing pages are converting […]
Sorry Google, There Are Benefits to Using Multiple Match Types when Using Broad Match with Smart Bidding
Over the years, advertisers have flocked to Google Ads (formerly Google AdWords) due to the creativity, flexibility, reach, and targeting offered by the ad giant. Google Ads was an empty canvas with unlimited opportunities. Over the years the platform has […]
Making the Case for Broad Match
Hear me out, I tested broad match keywords in Google Ads, and they’re actually converting. I know, I know, I’m the biggest hypocrite in history. If you’ve read my book or ever listened to me rant on Marketing O’Clock, you […]
A Marketer’s Website Launch Checklist
SEO, Google Tag Manager Tracking, & Content Planning for a New Website There’s nothing quite like launching a beautiful, brand new website, but it is something that can have ugly consequences if not performed correctly. While marketers aren’t typically responsible […]
Pinterest as a Search Engine: An Advertisers’ Guide to Keyword Targeting in Pinterest Ads
Pinterest – the land of dream wedding dresses, instant pot recipes, room inspo, and… SERPs? Pinterest is so much more than a social media platform. Yes, the platform allows users to like, share, and discover new content based on their […]
0 Comments