7 Things to Consider Before Your Next Nonprofit A/B Test

What nonprofit doesn’t want to offer the strongest, most efficient, and results-driven donation experience to supporters? One way to be confident that yours performs its best is to lean into new A/B testing ideas.

A/B testing is a reliable strategy for nonprofits but requires time and resources to do it right. We’re here to lay out exactly what goes into the A/B testing process to help your team prepare and budget accordingly. 

The Impact of A/B Testing Your Donation Experience

A common practice to inform the user experience, A/B testing identifies areas for improvement. At Classy, we run these tests often to inform our product innovation and help our nonprofit community optimize donation experiences.

When might a nonprofit complete an A/B test?

Nonprofit organizations may conduct an A/B test to:

• Identify optimization opportunities on donation websites or forms (like colors, buttons, fonts, branding, form fields, etc.).
• Evaluate the conversion rate of embedded versus standard donation forms. 
• Understand donors’ demographics and behaviors better.
• Increase specific campaign goals (like recurring giving revenue).
• Reduce friction in the checkout process (like removing particular contact information requirements).
• Evaluate campaign copy and creative elements to discover which resonates with donors the most.

Other Testing Methods

Sometimes, an A/B test isn’t the best fit, especially when you want to control variables like seasonality. A great example of this is one of the tests we ran at Classy on our embedded donation forms. 

An embedded donation form is a streamlined experience typically deployed as a primary website donation option. Visitors can access the form through a “Donate” call to action (CTA) in the nonprofit’s site header or the primary CTA in the body of the nonprofit’s site. 

Recently, we tested this use case to determine how traffic from more than 500 organizations’ websites fluctuates over nine months when using an embedded form compared to a standard donation form. We examined more than 500,000 unique sessions and compared the conversion performance to a standard donation from the same period in the prior year. 

Here’s a peek at the findings:

• The median conversion lift was three points with embedded donation experiences, which increased to 4.3 points on mobile.
• The revenue per visitor was 29% higher on embedded donation experiences. 

Our industry-wide and organization-specific research helps nonprofits make informed decisions about donation experiences. Then, when it’s appropriate to A/B test, we do plenty of that as well to offer insights into what resonates with donors in the space today. 

7 Things to Know Before You A/B Test Your Campaign

A/B testing can feel like a daunting process with several nuances. That’s why we outlined a few steps to guide your decisions and map out the best path forward.

1. Get Clarity Around Numbers to Set Yourself Up for Success

A clearly defined goal and hypothesis for the test you want to perform is where you want to start. There are several different metrics and key performance indicators (KPIs) that an A/B test can support. 

Determine your main priority, whether increasing conversion rate, total revenue, revenue per visitor, or any other specific metric that will increase revenue for your nonprofit organization. At Classy, we monitor all these KPIs and more. We often focus on increasing the revenue that each visitor brings into a donation form, which is a holistic metric that considers the conversion rate and donation size. 

Once you have clarity around your goals and the measurement of success, identify your statistical significance level of comfortability to base your decisions. 

Just remember that testing can have challenges, and you may need to test a few times to ensure you can rely on the results. Our team recommends achieving 95% statistical significance before taking the next steps to ensure the results are as accurate and representative as possible.

2. Test Specific Variables First

The variables you test matter, and you want to focus on avoiding testing too many variables simultaneously. For example, if you test the design of a donation experience, adding new fonts and spacing can distort your results and make it harder to know what genuinely drives the increase or decrease in performance.

The only thing that should change between the experiences you test is the test variable. That means keeping things like the logo, title, copy, font, and spacing the same if you want to see the impact of only changing the design. On that note, you want to prevent other A/B tests from running on your website that could interfere with your test results.

3. Consider the Sample Size

The sample size of your audience needs to be large enough to reach statistical significance to achieve reliable results. After all, a test of 15 people doesn’t represent a potential donor base of thousands. A more extensive group representing various thought patterns and scenarios will decrease the probability of randomness and increase your chances of accuracy. 

Additionally, the more sessions a site receives, the faster the A/B test can achieve statistically significant results and reliable insights. That means aiming for as much representation as possible from your testing audience in any scenario. It may take some time, but the bigger, the better.

A general rule is: for a highly reliable test, you need a minimum of 30,000 visitors and 3,000 conversions per variant. If you follow this guideline, you’ll generally achieve enough traffic and conversions to derive statistically significant results at a high level of confidence.¹

4. Produce More Accurate Results With Real-World Conditions

A/B tests should recreate your nonprofit website’s typical traffic patterns and conditions. Think about how much traffic your nonprofit website typically gets over a week and use that number to determine how long you’ll need to run a test to reach statistical significance. 

A minimum of two weeks is a general baseline to account for natural fluctuations. If you don’t see increased traffic daily, the test may need more time to ensure you capture everything accurately. 

5. Invest in a Trusted Testing Program

Several programs can assist with A/B testing. The most popular tool, Google Optimize, is free but will sunset on September 30, 2023. In its place, nonprofits can consider a paid subscription to another tool like Optimizely or Vanguard FTSE Emerging Markets ETF. 

These tools offer comprehensive functionality to assist you in launching, monitoring, and analyzing your test. However, it’s imperative to note that a number of resources, rigorous monitoring, and in-depth analyses are typically necessary to harness the potential of A/B testing tools fully. 

Classy makes it easy to duplicate a current donation website and make small tweaks without any back-end coding work. You can then A/B test each campaign URL against the other in your testing platform. We recommend doing a 50/50 split of traffic.

6. Test for Long-Term Data Trends to Capture the Full Picture

Your A/B test can provide insights into a specific change at a particular time. That means you may have to run a series of tests to see trends over time as you begin to think about the bigger picture of your data.

Classy’s product roadmap and path of innovation center on insights from several donation experiences throughout various seasons. We conduct our A/B testing this way for nonprofits to get the full picture and confidently use our findings to inform decision-making.

7. Consider How Each Test Will Impact Your Bottom Line

By now, you understand how much goes into a single A/B test. Naturally, you want to see a return on your cost and time investment, so consider the impact these experiments could have on your bottom line. 

Investing in a fundraising platform that conducts behind-the-scenes testing to ensure it offers the best tools and campaign experiences in the industry could save your staff (and finance team) from extra work or hardship. At Classy, we know these insights are valuable, so we take care of them for you.

When we tested embedded donation forms against standard donation forms on Classy, we saw nonprofits convert donors at a rate of 2x the industry standard. This is one example of the insights that drive our platform and inform the recommendations and coaching advice Classy customers can benefit from as they shape their fundraising strategies. 

When we saw the impact of embedded donation forms through our testing, the conversion results inspired many nonprofits to sign up to experience the impact. 

For example:

• Many Hopes saw a 56% improvement in donations year over year. 
• V Foundation increased conversion by 13 points.
• Feeding San Diego saw a 44% conversion rate.

Find a Fundraising Platform You Can Trust

An A/B test is a valuable tool for your nonprofit to make confident decisions. However, it’s also a costly and time-consuming process that can cause extra work or hardship when doing it alone. That’s where Classy comes in. 

At Classy, we’re committed to testing our products before and after they’re available to customers to ensure each product meets the demands of the sector in the most efficient and effective ways. Additionally, we aim to provide every nonprofit using Classy with the confidence to know every product in our fundraising suite undergoes rigorous testing to the fullest extent, then modified around the results to satisfy donor and nonprofit expectations.

We look forward to continuing to experiment with A/B testing across our platform as we gain more insight into what today’s donors resonate with and what drives them to take action in support of incredible missions like yours.

Copy Editor: Ayanna Julien

Article Source

1. “How to Correctly Calculate Sample Size in A/B Testing,” Guess the Test, last modified December 2021, https://guessthetest.com/calculating-sample-size-in-a-b-testing-everything-you-need-to-know.