A/B Testing
A/B Testing allows advertisers to segment the users they're reaching on Twitter so that they can understand how best to optimize for campaign performance and gather learnings to inform their marketing strategies.
These segments—referred to as user group splits—are randomized and mutually exclusive. With randomization, factors that influence outcomes are equally distributed. In other words, there are no inherent differences between groups or their expected behaviors. Because of this, when a single variation is applied to a user group and not the others, the difference in campaign performance can be attributed to that variation.
While it's possible to test many variations at once, we strongly recommend testing a single variation at a time. This isolates the causal factor for the observed difference in campaign performance.
Variations are set at either the campaign level or the ad group level. The ad group is set through line item in Ads API, As an example of an ad group level variation, if the advertiser wants to test the efficacy of a new creative, they should create one campaign with 2 identical ad groups where the only difference is the creative.
Use Cases
A/B testing is most often used to support (1) optimization use cases for performance customers who want to understand what works best on Twitter in order to optimize their investment and (2) learning use cases for brand advertisers who want to use learnings to inform their marketing strategy.
The API will support A/B testing for any campaign variable, including:
Creative
Targeting
Bid type
Bid unit
Attributes
A/B Tests are represented as nested structures. There are top level fields for the A/B Test itself and an array of user group objects, each with a set of fields describing it.
At a high level, every A/B Test must include the following information.
The test duration, represented by the start_time and end_time fields
The level at which the split will occur, represented by the entity_type field
At least two (and at most 30) user groups, each represented as an object in the user_groups array
Each user group must include the following information.
The percentage of users that should be allocated to the given user group, represented by the size field
The campaign IDs/line item IDs that should make up the pool of users for the given user group, represented by the entity_ids array
Optionally, name and description values can be set for A/B Tests and for user groups. Information about validation rules and other constraints can be found below.
Other metadata, such as the ID or the created at time, are also included, but are automatically set by Twitter.
An example A/B Test entity for campaign level is shown below.
{
"created_at": "2020-12-01T00:00:00Z",
"created_by": {
"user_id": "756201191646691328",
"username": "apimctestface"
},
"deleted": false,
"description": "documentation example",
"end_time": "2020-12-05T01:00:00Z",
"entity_type": "CAMPAIGN",
"id": "hr7l0",
"name": "first AB test",
"start_time": "2020-12-01T01:00:00Z",
"status": "SCHEDULED",
"user_groups": [
{
"id": "p1bcx",
"name": "first group",
"description": null,
"size": "50.0",
"entity_ids": [
"f2qcw",
"f2tht"
]
},
{
"id": "p1bcy",
"name": "second group",
"description": "second AB test group",
"size": 50,
"entity_ids": [
"f2rqi",
"f2tws"
]
}
],
"updated_at": "2020-12-01T00:00:00Z",
"updated_by": {
"user_id": "756201191646691328",
"username": "apimctestface"
}
}
An example A/B Test entity for line item level is shown below.
{
"created_by":{
"user_id":"756201191646691328",
"username":"apimctestface"
},
"name":"Test2e",
"start_time":"2022-08-15T00:00:00Z",
"updated_by":{
"user_id":"756201191646691328",
"username":"apimctestface"
},
"description":"My Second AB test",
"entity_type":"LINE_ITEM",
"end_time":"2022-08-30T00:00:00Z",
"id":"1ul",
"user_groups":[
{
"name":"first group",
"size":"50.0",
"description":"first group description",
"entity_ids":[
"ij9dh"
],
"id":"4xe"
},
{
"name":"second group",
"size":"50.0",
"description":"second group description",
"entity_ids":[
"ihng8"
],
"id":"4xf"
}
],
"status":"SCHEDULED",
"created_at":"2022-08-11T00:10:50Z",
"updated_at":"2022-08-11T00:10:50Z",
"deleted":false
}
Usage
The subsections below describe creating and updating A/B Tests. Reading and deleting work like they do with all other Ads API endpoints.
Creating
Create an A/B Test using the POST accounts/:account_id/ab_tests endpoint. The endpoint only accepts JSON POST bodies. The Content-Type must be set to application/json.
After the advertiser sets up two or more campaigns, an A/B Test can be created. As stated above, A/B Tests must include: test duration, split level, and at least two user groups. Each user group must declare the percentage of users that should be allocated to it as well as the campaign IDs that should make up its pool of users. Each of these is described in further detail below.
Test duration:
The start_time and end_time values must
Be in the future (relative to when the A/B Test is created)
Overlap with the campaign/line item flight dates
The test must last at least one day for non-app-based campaigns and last at least five days for app-based campaigns
Split level:
The entity_type can be set to CAMPAIGN or LINE_ITEM
User groups:
Each user group is represented as an object in the user_groups array
A minimum of two user groups is required
A maximum of 30 user groups is allowed
The size for each user group is set using a string representation of a numeric value between 1.00 and 99.00
Note: The size values across objects must add up to 100.00
The campaign IDs should be specified in each user group's entity_ids array
Optionally, set the name and description for the A/B Test or for one of more user groups.
The following request creates an A/B Test on the campaign level that lasts four days and has two user groups with 50% of users in each group. The first user group is based on campaigns f2qcw and f2tht; the second user group is based on campaigns f2rqi and f2tws. The request also adds names and descriptions to some parts of the entity.
twurl -X POST -H ads-api.twitter.com "/8/accounts/18ce54d4x5t/ab_tests" -d '{"end_time": "2020-12-05T01:00:00Z", "entity_type" : "CAMPAIGN", "start_time": "2020-12-01T01:00:00Z", "user_groups": [{"entity_ids": ["f2qcw", "f2tht"], "size": "50.00", "name": "first group"},{"entity_ids": ["f2rqi", "f2tws"], "size": "50.00", "name": "second group", "description": "second AB test group"}], "name": "first AB test", "description": "documentation example"}'
{
"request": {
"params": {
"account_id": "18ce54d4x5t",
"end_time": "2020-12-05T01:00:00Z",
"entity_type": "CAMPAIGN",
"start_time": "2020-12-01T01:00:00Z",
"user_groups": [
{
"entity_ids": [
"f2qcw",
"f2tht"
],
"size": "50.0",
"name": "first group"
},
{
"entity_ids": [
"f2rqi",
"f2tws"
],
"size": "50.0",
"name": "second group",
"description": "second AB test group"
}
],
"name": "first AB test",
"description": "documentation example"
}
},
"data": {
"created_at": "2020-12-01T00:00:00Z",
"created_by": {
"user_id": "756201191646691328",
"username": "apimctestface"
},
"deleted": false,
"description": "documentation example",
"end_time": "2020-12-05T01:00:00Z",
"entity_type": "CAMPAIGN",
"id": "hr7l0",
"name": "first AB test",
"start_time": "2020-12-01T01:00:00Z",
"status": "SCHEDULED",
"user_groups": [
{
"id": "p1bcx",
"name": "first group",
"description": null,
"size": "50.0",
"entity_ids": [
"f2qcw",
"f2tht"
]
},
{
"id": "p1bcy",
"name": "second group",
"description": "second AB test group",
"size": "50.0",
"entity_ids": [
"f2rqi",
"f2tws"
]
}
],
"updated_at": "2020-12-01T00:00:00Z",
"updated_by": {
"user_id": "756201191646691328",
"username": "apimctestface"
}
}
}
For A/B Testing on line item level
The major difference between A/B testing on the campaign and the line item levels is the entity_type. We need to set it to ‘entity_type’ = ‘LINE_ITEM’ for A/B tests on the line item level. This applies to all the actions on an already created A/B test below.
Requirements:
- All line items of the A/B testing campaign must be included in the split test.
- Only equal splitting is allowed for Line item level.
- Number of user groups line items allowed in 1 split test must be less than or equal to 5.
- Only 1 line item per user group.
Updating
Update an A/B Test using the PUT accounts/:account_id/ab_tests/:ab_test_id endpoint. The endpoint requires that a JSON blob be sent in the request. The Content-Type must be set to application/json.
Like other update endpoints, the PUT accounts/:account_id/ab_tests/:ab_test_id endpoint requires that the A/B Test ID be referenced in the URL. In general, A/B Tests can only be updated while the status is SCHEDULED. There is one exception: it's possible to update the A/B Test's end_time while it's LIVE.
This endpoint supports partial JSON with object IDs. The following principles apply:
To add or remove objects or elements, pass in the entire array (and its substructures); this is a replacement operation
Otherwise, modify (change, add, remove) existing fields by referencing key names or IDs
To remove a field, set its value to null
Fields that are not passed in are not modified
For example, adding a third user group to the previously created A/B Test would require us to send the user_groups array with the two existing user group objects as well as with the new one we'd like to add. Think of this as recreating the user_groups array; pass the data in as if you were creating it this way to begin with (do not pass in user group object IDs). The user_groups array in the update request could be represented as follows.
[
{
"entity_ids": [
"f2qcw",
"f2tht"
],
"size": "30.0",
"name": "first group"
},
{
"entity_ids": [
"f2rqi",
"f2tws"
],
"size": "30.0",
"name": "second group",
"description": "second AB test group"
},
{
"entity_ids": [
"i1vwr",
"i1xre"
],
"size": "40.0"
}
]
Notice that the size values across objects still add up to 100.00. If we would not have updated them for the first two objects—previously set at 50.00 each—the request would have failed.
If, instead, we simply wanted to add a description to the first user group, the user_groups array in the update request would be represented as follows.
[
{
"id": "p1bcx",
"description": "updated using a PUT request"
}
]
We reference the user group object by ID and only include the field we'd like to modify.
Request Examples
This section provides additional example update requests. Think of these as being called sequentially. JSON blobs are formatted for readability. Responses are omitted.
To make the following modifications, the request would be represented as follows. (This is the same as the previously used example above.)
Adds a third user group without a name or description
Changes the percentage of users in each user group
twurl -X PUT -H ads-api.twitter.com "/8/accounts/18ce54d4x5t/ab_tests/hr7l0" -d '
{
"user_groups": [
{
"entity_ids": [
"f2qcw",
"f2tht"
],
"size": "30.00",
"name": "first group"
},
{
"entity_ids": [
"f2rqi",
"f2tws"
],
"size": "30.00",
"name": "second group",
"description": "second AB test group"
},
{
"entity_ids": [
"i1vwr",
"i1xre"
],
"size": "40.00"
}
]
}'
To make the following modifications, the request would be represented as follows.
Removes the A/B Test description
Adds a description to the first user group
Add an entity ID (f2syz) to the second user group
twurl -X PUT -H ads-api.twitter.com "/8/accounts/18ce54d4x5t/ab_tests/hr7l0" -d '
{
"description": null,
"user_groups": [
{
"id": "p1bcx",
"description": "first AB test group"
},
{
"id": "p1bcy",
"entity_ids": [
"f2rqi",
"f2tws",
"f2syz"
]
}
]
}'
The third modification requires that we pass in the two existing entity IDs along with the new one. Notice that no changes were made to the third user group.
To make the following modifications, the request would be represented as follows.
Removes the second user group
Changes the percentage of users in each user group
twurl -X PUT -H ads-api.twitter.com "/8/accounts/18ce54d4x5t/ab_tests/hr7l0" -d '
{
"user_groups": [
{
"entity_ids": [
"f2qcw",
"f2tht"
],
"size": "55.00",
"name": "first group"
},
{
"entity_ids": [
"i1vwr",
"i1xre"
],
"size": "45.00"
}
]
}'