A/B Test Tool

In this article:

1. Setting up an A/B test
2. Tracking your A/B test
3. Viewing completed A/B tests

---

A/B tests allow you to compare two slightly different versions of something to see which performs better. You can easily run A/B tests on your contribution forms using our platform. Our A/B Test Tool randomly splits everyone who clicks on the link to your contribution form into two groups. One group receives the normal form (the control), and one receives the altered form (the variable). The variables you can test include: 

• Form title
• Donation ask
• Preset amounts
• Branding
• Hide thermometer
• Hide embedded video
• Recurring promotions, like a recurring pitch or recurring incentive
• Pop-up recurring title
• Pop-up recurring ask
• Smart Boost upsell

Plus, you have multiple optimization options, so you can run an A/B test without missing out on one-time, recurring, or Smart Boost contributions!

Once the test has run its course, you can compare the two versions and see which one received more dollars and donations.

---

Setting Up an A/B Test

To set up an A/B test, click the “Edit” button to the right of your form’s name on the Contribution Forms page of your Dashboard. This will open up the form editor for the form you’d like to A/B test.

Select “A/B testing” in the form editor.

Click the link in the “A/B Testing” tab.

You’ll be taken to an A/B Test page, where you can set up your experiment.

Give your test a descriptive name. The name may be seen externally, so choose something that is not sensitive or private.

Next, add a short hypothesis that describes what the outcome of your A/B test will be. For example: “We expect the form with the custom title written by the content team will have more form conversions.”

Underneath these two fields, the “Automatic traffic allocation” option will automatically be selected (pictured above). This feature, also known as a multi-armed bandit algorithm, directs more traffic to the version of your form that is performing best. This way you won’t lose out on potential one-time or recurring contributions, depending on what you choose to optimize for, while still running a sound test. This is an easy, and highly recommended, way to run tests and raise funds at the same time. You can learn more about the multi-armed bandit on our blog. 

Directly below the “Automatic traffic allocation” box, you can select which outcome you would like the algorithm to optimize for while your A/B test is running. 

If you select “ Form conversions,” the multi-armed bandit algorithm will direct more traffic to the version of your form that is collecting the most contributions of any amount. If you select “Recurring conversions,” the algorithm will direct more traffic to the version of your form on which more donors are choosing to make recurring contributions, whether the donations are made on the form itself during the regular donation flow or after, through the recurring pop-up upsell. If you select “Recurring upsell,” the algorithm will direct more traffic to the version of your form on which more donors are choosing to start a recurring donation after they complete an initial one-time contribution. If you select “Smart Boost,” the algorithm will direct more traffic to the version of your form on which more donors are choosing to increase their initial contribution in the post-donation upsell. You cannot change what you choose to optimize for after the test has begun.

If you uncheck the “Automatic traffic allocation” box, each of your form variants (versions) will be allotted the same volume of traffic, even if one performs very poorly. Please note that even if you have “Form conversions,” “Recurring upsell,” “Recurring conversions,” or “Smart Boost” selected,  each version of your form will receive the same volume of traffic if you uncheck the “Automatic traffic allocation” box. 

 Next, select the feature that you would like to test:

This will be your variable. Each version of your form in the A/B test will be the same, except for the variable you are choosing to test. While you can test multiple features at once, if you want to run the most accurate test possible, you should pick just one. That way you will know it was the variation that caused a difference in visits, contributions, contribution amounts, recurring contributions, or Smart Boost contributions.

If you choose to optimize for form conversions, you can test any element of your form except for recurring promotions, and you cannot test any part of the recurring pop-up. If you choose to optimize for recurring conversions, you can test any element of your form or recurring pop-up. If you choose to optimize for recurring upsell, you can only test the recurring pop-up title and ask. To test a recurring pitch or recurring incentive, select “Recurring promotion.” If you choose to optimize for Smart Boost, you can only test the Smart Boost pop-up title and ask. 

Once you’ve selected which feature you want to test, update the variation of that feature as needed. For example, if you are testing the form title, your current form title will populate and you’ll need to enter an additional title to test (pictured below). 

If you chose “Recurring promotion,” you have the option of testing different variations of the recurring pitch and recurring incentive tools. The control will be your form’s current settings, whether that be a recurring pitch or a recurring incentive (both can be configured in the recurring settings tab of the form editor!). 

For the variation, you have the option to test a recurring pitch or a recurring incentive. That means you have the option to test the same type of recurring promotion with different copy, or a different type of recurring promotion! 

If you would like to add another variation, click “Add another variation” and repeat the process.

Once you’ve finalized your test variations, click the “Create Test” button. Congrats — your test is live! You’ll be brought to the Active A/B Test page, where the results and analysis of your A/B test will live. 

You will now be able to view both variations of your form in the form editor. Go to your form and open the form editor. Go to the “A/B testing” tab and select which variation of the form you would like to see from the “Preview A/B test variation” dropdown menu. 

---

Tracking Your A/B Test

Once you create the test, you’ll be able to track each variation’s progress by returning to your form, opening the form editor, and going to the “A/B testing” tab. Then follow the link.

You will see the name of the A/B test, which admin created the test, what the test is optimizing for, and the hypothesis. 

Then, you’ll see an option to receive a weekly email update about your A/B test. Check the box to get an A/B test analysis in your email inbox! 

Next, you’ll see analysis of the control and each of your form variations. First, you’ll see the traffic allocation, which is the percentage of traffic that is being directed to that form version based on the multi-armed bandit algorithm. If you have automatic traffic allocation turned off, each version of your form will have equal traffic allocation. You will also see the total money raised on that form variation beneath the traffic allocation. 

No matter what you optimized your test for, you will always see the form conversion rate (people who donated divided by people who only visited the form), average contribution, and dollars per visit for each version of your form.

If your test is optimizing for recurring conversions, you will also see the recurring conversion rate, which is the percentage of donors who visit the form and choose to make any type of recurring donation (during the regular donation flow or after with a pop-up upsell), and the average recurring amount. 

If your test is optimizing for recurring upsell, you will also see the upsell conversion rate in addition to the recurring conversion rate and average recurring amount. The upsell conversion rate is the percentage of donors who see your form’s recurring pop-up after giving a one-time contribution and choose to also give a recurring donation. 

If your test is optimizing for Smart Boost, you will see the Smart Boost conversion rate and the average Smart Boost contribution in addition to the dollars per visit, the form conversion rate, and the average contribution. The Smart Boost conversion rate is the percentage of donors who see your form’s Smart Boost pop-up after making a contribution and choose to increase their donation. The average Smart Boost contribution is the average amount that donors choose to increase their contribution. 

Please note: Donors who contribute through an ActBlue Express Lane link will not be included in the statistics you see on your Active A/B Test page. 

If you did NOT use the automatic traffic allocation (multi-armed bandit) option, you will need to click “Make winner” on the variation that performed best in order to stop the less-successful form from being distributed. If you did use the multi-armed bandit method, the A/B test will do this on its own over time, but if you see a clear winner you can click “Make winner” to speed up the process. 

You can find data on your A/B test in the spreadsheet of all your form’s contributions, which you can find on your form’s Statistics page. You can navigate to the Statistics page from the Active A/B Test page while previewing your form or by clicking either the number of contributions or the donation total for your form on the Contribution Forms page of your Dashboard. 

The report of all of your form’s contribution info, including data from your A/B test, can be found when you download a report from the Stats page. Simply click the blue “Export all contributions” button.

---

Viewing Completed A/B Tests

Once you’ve completed your A/B test, either by letting the multi-armed bandit algorithm pick a winner or picking a winner yourself, you can view the results at any time on the A/B test page of your form. You can always navigate to the A/B test page through the link in the form editor or the Form Tools menu. 

If you have further questions about A/B tests or best practices, let us know.