Email AB Testing with Customer.io

Email campaigns are a critical tool for any Marketing team. Finding the best performing Email template is a perfect use-case for an A/B test. Statsig allows you to run simple but powerful A/B tests on different parts of your email content. Since Statsig can integrate seamlessly with product analytics, you can run email experiments and understand deeper business level impact on product metrics easily.

This guide assumes you have an existing Statsig account. Please go here to create a new free account if you don’t already have one: https://statsig.com/signup

Step 1: Create an experiment

Start by creating a new Experiment on Statsig console. Put in a name and leave the rest of the fields empty/default. For the purposes of this walkthrough, that should do.

Step 2: Start the experiment

Since you can’t start an experiment without a parameter, let’s go ahead and add a dummy parameter.

Save the experiment setup and Start it. We’re all set with the experiment set up.

While you’re at it, copy the Experiment Name. We’ll use this in a bit.

Step 3: Set up Exposure Webhooks

In your Customer.io campaign, create a Random Cohort Branch which a flow similar to the following:

In the Webhook actions, put in a post request similar to the following, with your api key and experiment name filled in:

Pass in any other custom IDs and user attributes inside the post body.

For each webhook, make sure to expose the correct group that you’d like to attribute your branch to. In the above webhook, we are exposing the “Control” group.

In Statsig, you’ll now have exposures for each of your experiment groups.

Holdouts

To use Statsig Holdouts with Customer.io, it’s recommended to identify users that are part of a holdout via customer.io’s identify function: https://customer.io/docs/sdk/ios/identify/

Where you call Customer.io’s identify method, you could check a Statsig holdout gate, and add an attribute to the user to mark that user as being in a holdout.

In your campaign, you’ll be able to create a True/False branch to check whether a user is in the holdout.