PowerTrack API migration to X API v2 filtered stream

Use this migration guide to understand the similarities and differences between PowerTrack API and X API v2 filtered stream, and to help migrate a current PowerTrack API integration to v2 filtered stream.

• Similarities
  • Streaming delivery method
  • Integration process
  • Persistent stream connection with separate rules management endpoints
  • Rule syntax
  • Rule operators (with exceptions)
  • Rule matching logic
  • Support for Post edit history and metadata
• Differences
  • Rule length
  • Rule volume
  • Endpoint URLs
  • App and Project requirement for access
  • Authentication method
  • Request parameters
  • Usage tracking
  • Multiple streams, redundant conections, backfill and Replay recovery
  • Request parameters and response format
  • Response JSON data structure

Similarities

Streaming delivery method

Both PowerTrack and X API v2 filtered stream use streaming data delivery, which require the client to establish an open connection to an endpoint and keeping a very long lived HTTP request, and parsing the response incrementally from the server in real time.  Both PowerTrack and X API v2 filtered stream filter publicly available Posts matching rules that exist on the stream in real time, and use keep-alive signals as new line characters (\r\n) to signal the connection is still active. Both PowerTrack and X API v2 filtered stream endpoint connections deliver data in real time and should be read by the connecting client quickly.  

Integration process

Integrating with filtered stream is similar to integrating with PowerTrack, using the general process below:

1. Establish a streaming connection.
2. Asynchronously send separate requests to add and delete rules from the stream.
3. Reconnect to the stream automatically when connection is disconnected.

Persistent stream connection with separate rules management endpoints

Similar to the PowerTrack API and Rules API, the new X API v2 filtered stream endpoints allows you to apply multiple rules to a single stream and add and remove rules to your stream while maintaining the stream connection.

Rule syntax, operators, and matching rules logic

The X API v2 filtered stream uses a subset of the same rule operators currently used for PowerTrack rules. These operators are used to create boolean based rule syntax used for filtering desired matching Posts from the live stream.  Both PowerTrack and filtered stream use the same rule syntax for building rules and matching logic is the same. While the majority of the operators are available for both PowerTrack and filter stream, there are a few notable differences and net new operators listed below.  For more details and example uses for each operator see current PowerTrack operators and current X API v2 filtered stream operators. 

Please note that many operators (noted as 'advanced operators') are reserved for those users who have been approved for Academic Research access or Enterprise access.

Operators available with both PowerTrack and X API v2 Filtered stream:

Support for Post edit history and metadata

Both versions provide metadata that describes any edit history. Check out the filtered stream API References and the Edit Posts fundamentals page for more details. 

Differences

Rule length

Rule length is measured the same way (by character count) for both PowerTrack and filtered stream rules, however the maximum length for PowerTrack rules is 2048 characters and the maximum rule length for rules on X API v2 filtered stream varies by access level. 

Enterprise access -  2048 characters (please contact your designated account manager regarding your specific account)

Rule volume

The PowerTrack maximum rule volume per stream is defined within the enterprise account contract.  X API v2 filtered stream rule volume varies by access level.

Enterprise access - 25000+ rules (please contact your designated account manager regarding your specific account)
 

Endpoint URLs

• PowerTrack endpoints:
  • https://gnip-stream.twitter.com/stream/powertrack/accounts/{account_name}/publishers/twitter/{stream_label}.json
  • https://gnip-api.twitter.com/rules/powertrack/accounts/{account_name}/publishers/twitter/{stream_label}.json
  • https://gnip-api.twitter.com/rules/powertrack/accounts/{account_name}/publishers/twitter/{stream_label}/validation.json
• X API v2 endpoint:
  • https://api.twitter.com/2/tweets/search/stream
  • https://api.twitter.com/2/tweets/search/stream/rule

App and Project requirements for v2 access

PowerTrack access is granted through a contracted annual subscription for data, and set up through console.gnip.com by your account manager at X.  PowerTrack does not require a X developer App to access.  In order to use the X API v2 filter stream, you must have signed up for a X developer account, and a X developer App associated with a Project. The developer App and Project setup for X API v2 access is all done through the developer portal.  

Authentication method

The PowerTrack API endpoints use Basic Authentication set up in console.gnip.com. The X API v2 filtered stream endpoints require a X developer App and an OAuth 2.0 App Access Token (also referred to as Application-only or Bearer Authentication). To make requests to the X API v2 version you must use your specific developer App's Access Token to authenticate your requests.

In the process of setting up your developer account, developer App and Project, an App Access Token is created and shared within the dev portal user interface, however, you can generate a new one by navigating to your app's “Keys and tokens” page on the developer portal. If you’d like to generate/destroy the App Access Tokens programmatically, see this OAuth 2.0 App-Only guide.

Usage tracking

PowerTrack usage can be retrieved programatically using the Usage API, or can be seen in console.gnip.com on the usage tab.  Post consumption across all PowerTrack streams is deduplicated each day and volume consumption is defined within the enterprise contract. 

X API v2 filtered stream usage can be tracked within the developer portal at the Project level. Post consumption is set at the Project level and is shared across several different X API v2 endpoints, including filtered stream, recent search, user Post timeline and user mention timeline.  Currently with Basic Access, the monthly Post consumption limit is 500,000 Posts per month total.

Multiple streams, redundant conections, backfill and Replay API for recovery

There are several recovery and redunancy features available via PowerTrack, some of which are not yet available for X API v2 filtered stream.  For PowerTrack, all recovery and redundancy features are configured within the enterprise contract. PowerTrack API currently has the flexibility to offer multiple PowerTrack streams (commonly "dev" and "production") with unique rulesets.  Currently, the X API v2 filtered stream is only available with a single stream.

PowerTrack allows you to connect to have multiple connections to a single stream, generally used for redundant connections to different data centers or clients. 

If a PowerTrack stream is disconnected breifly, a reconnection can be made using the backfillMinutes parameter to reduce the chance of data loss within five minutes of the disconection. While we have added this functionality to the X API v2 version, it is currently only available with Academic Research access, and has been renamed to backfill_minutes.

If a PowerTrack stream is disconnected for longer than a 5 minute period, the separate Replay API can be used to recover data for up to a 2 hour period in the recent 5 day past.  Currently, the X API v2 filtered stream does not have a replay feature.

Request parameters and response format

One of the biggest differences between PowerTrack API and X API v2 filtered stream is the parameter functionality.

Using the X API v2 filtered stream, there are several parameters used on the connection request to identify which fields or expanded objects to return in the Post payload.  This is common for all v2 endpoints. By default, only the Post id and text are returned for matching Posts but additional parameters, fields and expansions described below, can be added in order to recieve more detailed data per matching Post. 

fields: X API v2 endpoints enable you to select which fields are provided in your payload. For example, Post, User, Media, Place, and Poll objects all have a list of fields that can be returned (or not). 

expansions: Used to expand the complementary objects referenced in Post JSON payloads. For example, all Retweets and Replies reference other Posts. By setting expansions=referenced_tweets.id, these other Post objects are expanded according to the `tweet.fields` setting.  Other objects such as users, polls, and media can be expanded.

https://api.twitter.com/2/tweets/search/stream?tweet.fields=created_at,public_metrics,author_id,entities&expansions=attachments.poll_ids

You can learn more about how to use fields and expansions by visiting our guide, and by reading through our broader data dictionary.

curl --compressed -v -uexample@customer.com "https://gnip-stream.twitter.com/stream/powertrack/accounts/:account_name/publishers/twitter/:stream_label.json"

curl "https://api.twitter.com/2/tweets/search/stream?tweet.fields=attachments,author_id,context_annotations,conversation_id,created_at,entities,geo,id,in_reply_to_user_id,lang,possibly_sensitive,public_metrics,referenced_tweets,reply_settings,source,text,withheld&user.fields=created_at,description,entities,id,location,name,pinned_tweet_id,profile_image_url,protected,public_metrics,url,username,verified,withheld&expansions=author_id,referenced_tweets.id,referenced_tweets.id.author_id,entities.mentions.username,attachments.poll_ids,attachments.media_keys,in_reply_to_user_id,geo.place_id&place.fields=contained_within,country,country_code,full_name,geo,id,name,place_type&poll.fields=duration_minutes,end_datetime,id,options,voting_status" -H "Authorization: Bearer $ACCESS_TOKEN"

The PowerTrack API data format is set within console.gnip.com at the stream settings level, which can be set to either the X native enriched format or Activity streams format. 

PowerTrack API only uses one optional parameter on connection, to reconnect using backfill (backfillMinutes=5). This optional parameter is also available to filtered stream, but is called backfill_minutes, and is only available via Academic Research access.
 

https://gnip-stream.twitter.com/stream/powertrack/accounts/{account_name}/publishers/twitter/{stream_label}.json?backfillMinutes=5

Response structure and data format

As described above, the request parameters set at the connection request for X API v2 filtered stream determine the response data returned.  There are several different response possibilites using different fields and expansions which can range from the most simple default response with only the Post id and text, to an extremely detailed and expanded data payload.

The data format for PowerTrack is set within console.gnip.com at the stream settings level, which can be set to either the X Native Enriched format or Activity Streams format. 

The following table references Post response examples in each different format:

If you would like to know more about how the enterprise data formats map to the X API v2 format, please visit our following guides:

• Native Enriched to X API v2 format migration guide
• Activity Streams to X API v2 format migration guide