A/B Test Tool
A/B tests allow you to compare two slightly different versions of something to see which performs better. You can easily run A/B tests on your contribution forms using our platform. Our A/B Test Tool randomly splits everyone who clicks on the link to your contribution form into two groups. One group receives the normal form (the control), and one receives the altered form (the variable). Once the test has run its course, you can compare the two versions and see which one received more dollars and donations.
One example of a good A/B test to try is comparing a plain pop-up recurring ask with a pop-up recurring ask that includes an added incentive for those who sign up, like a bumper sticker. We’ve found that this is one of the best ways to get recurring donations.
Setting Up an A/B Test
To set up an A/B test, click the "Edit" button to the right of your form's name on the Contribution Forms page of your Dashboard. This will open up the form editor for the form you’d like to A/B test.
Select "A/B testing" in the form editor.
Click the link in the “A/B Testing” tab.
You'll be taken to an A/B Test page, where you can set up your experiment.
Give your test a descriptive name. The name will only be seen internally, so choose something that works best for you.
Next, add a short hypothesis that describes what you are looking to find out with your A/B test.
Underneath these two fields, the “Automatic traffic allocation” option will automatically be selected (pictured above). This feature, also known as a multi-armed bandit algorithm, directs more traffic to the version of your form that is performing best. This way you won’t lose out on potential one-time or recurring contributions, depending on what you choose to optimize for, while still running a sound test. This is an easy, and highly recommended, way to run tests and raise funds at the same time. You can learn more about the multi-armed bandit on our blog.
Directly below the “Automatic traffic allocation” box, you can select which outcome you would like the algorithm to optimize for while your A/B test is running. If you select “Form conversions,” the multi-armed bandit algorithm will direct more traffic to the version of your form that is collecting the most contributions of any amount. If you select “Recurring upsell,” the algorithm will direct more traffic to the version of your form on which more donors are choosing to start a recurring donation after they complete their one-time contribution. If you select “Recurring conversions,” the algorithm will direct more traffic to the version of your form on which more donors are choosing to make recurring contributions, whether the donations are made on the form itself during the regular donation flow or after, through the recurring pop-up upsell.
If you uncheck the “Automatic traffic allocation” box, each of your variants will be allotted the same volume of traffic, even if one performs very poorly. Please note that even if you have “Form conversions,” “Recurring upsell,” or “Recurring conversions” selected, each version of your form will receive the same volume of traffic if you uncheck the “Automatic traffic allocation” box.
Next, select the feature that you would like to test:
This will be your variable. Each version of your form in the A/B test will be the same, except for the variable you are choosing to test. While you can test multiple features at once, if you want to run the most accurate test possible, you should pick just one. That way you will know it was the variation that caused a difference in visits, contributions, contribution amounts, or recurring contributions.
Once you’ve selected which feature you want to test, update the variation of that feature as needed. For example, if you are testing the form title, your current form title will populate and you’ll need to enter a second variation of the title (pictured below).
Once you’ve finalized your test variations, click the “Create Test” button. Congrats — your test is live!
You will now be able to view both variations of your form. Go to your form and open the form editor. Go to the "A/B testing" tab and select which variation of the form you would like to see from the “Preview A/B test variation” dropdown menu.
Tracking Your A/B Test
Once you create the test, you’ll be able to track each variation's progress by returning to your form, opening the form editor, and going to the "A/B testing" tab. Then follow the link.
You will see the name of the A/B test, the hypothesis, and which admin at your campaign or organization started the test at the top of the page.
Next, you'll see the traffic allocation, form conversion rate (people who donated divided by people who only visited the form), average contribution, and dollars per visit for each version of the form. You will also see the recurring conversion rate, which is the percentage of donors who visit the form and choose to make any type of recurring donation (during the regular donation flow or after with a pop-up upsell).
If you optimized your traffic allocation for recurring upsell or recurring conversions, you will also see the pop-up recurring results, which is the percentage of donors who see your form’s recurring pop-up after giving a one-time contribution and choose to also give a recurring donation. If you optimized your traffic allocation for recurring conversions but turned the pop-up recurring feature off, you will not see the pop-up recurring results on your Active A/B Test page.
Please note: Donors who contribute through an ActBlue Express Lane link will not be included in the statistics you see on your Active A/B Test page.
If you did NOT use the automatic traffic allocation (multi-armed bandit) option, you will need to click “Make winner” on the variation that performed best in order to stop the less-successful form from being distributed. If you did use the multi-armed bandit method, the A/B test will do this on its own over time, but if you see a clear winner you can click “Make winner” to speed up the process.
Data on your A/B test will be included in the CSV detailing your form’s contributions, which can be found on your form's Statistics page in the “Additional tools” dropdown near the bottom.
You can get to the Statistics page when you are viewing your form by clicking on the "Stats" button on the form navigation bar.
You can also get to the Statistics page by clicking either the number of contributions or the donation total for your form on the Contribution Forms page of your Dashboard.
If you've offered any additional incentives, like a bumper sticker on a pop-up recurring test, information for all of the donors that signed up will be in the CSV as well.
If you have further questions about A/B tests or best practices, let us know.