Get Tweet timelines

Guides contents

Introduction

This guides looks at alternative API solutions for developers impacted by the recently announced request limit changes to /statuses/mentions_timeline and /statuses/user_timeline.

What’s changing?

On June 19, 2019, we will begin limiting total GET requests to the v1.1 /statuses/mentions_timeline and /statuses/user_timeline endpoints to 100,000 requests per day. This is a total request limit (per endpoint) applied across both user-auth and app-auth requests. This means that in a 24-hour period, a single app can make up to 100,000 requests to /statuses/mentions_timeline and/or 100,000 requests to /statuses/user_timeline (with either app or user auth) before hitting this new app-level rate limit. The existing default user-auth and app-auth rate limits will not change.

Am I impacted?

Before diving into this guide, it’s worth confirming that you will indeed be impacted by the forthcoming changes. Please review the criteria below to check if you will be impacted by this change:

  • A single client app exceeds 100,000 requests per day (24-hour period) to the /mentions_timeline endpoint.
  • A single client app exceeds 100,000 requests per day (24-hour period) to the /user_timeline endpoint.

If either condition above is true, and you want to migrate your usage of the mentions / user timeline endpoints to an enterprise API alternative, please continue reading through this guide.

Alternative APIs

The following enterprise API solutions can be functional replacements for the most common use cases of the /mentions_timeline and /user_timeline endpoints endpoints. While offering methods for getting “mentions” and Tweets from users, they do differ in terms of authentication, delivery mechanism (streaming, webhooks), and data availability depending on the API. Follow the links below to explore the API documentation directly.

  1. Account Activity API (webhooks) – delivers both mentions and Tweets posted by the authenticating user
  2. PowerTrack API (streaming) – supports the ability to stream mentions (@ operator) and/or Tweets posted by any public user (from: operator) as well as many other filter operators.
  3. Enterprise Search API (REST) – supports the ability to query for mentions (@ operator) and/or Tweets posted by any public user (from: operator) as well as several other filter operators. Depending on your historical data needs, there are two options to choose from:
    • 30-Day Search (past 30 days)
    • Full-Archive Search (complete access back to March 2006)

 

Choosing an API

To help evaluate which API is right for you, please see the comparison table below that outlines the technical characteristics, unique features, and limitations of each API.

  /mentions_timeline* /user_timeline* Enterprise Search APIs PowerTrack API Account Activity API
Type REST REST REST Streaming Webhooks
Auth Oauth 1.0 (user context)

Oauth 1.0 (user context)

and

OAuth 2.0 (app-only)

HTTP basic (username / password) HTTP basic (username / password)

Oauth 1.0 (user context)

and

OAuth 2.0 (app-only)

Pagination 1-200 (per request) 1-200 (per request) 10-500 (per request) N/A N/A
Max Tweets Up to 800 Up to 3200 No max, but volume limits determined by contract No max, but volume limits determined by contract Unlimiited (total subscribed users determined by contract)
Rate limit 75/15 min (per authenticated user)

900/15 min (user context)

and

1500/15 min (app-only)

Custom N/A (streaming) Unlimited (data is pushed to webhook)
Unique features

- Ability to trim user object

- Ability to trim entities

- Get mentions by protected accounts

- Ability to trim user object

- Ability to trim entities

- Access historical Tweets from accounts that recently changed to unprotected

- Minimum latency

- Data Enrichments

- Cursor-based pagination

- 30 filter operators

-Counts endpoint (volume associated with query)

- Real-time delivery

- Data Enrichments

- Over 50 filter operators

- Ideal for heavy duty data processing

- Real-time delivery

- Push capability (passive listening)

- Retries

- Delivers all account activity (Tweets, mentions, DMs, Likes, etc.)

Limitations

- Limited to 800 most recent Tweets

- Doesn't return Tweets deemd "low quality"

- Limited to 3200 most recent Tweets

- 30-Day Search API limited to past 30 days

- Limited to Tweets from public users

- Always returns full Tweet payload

- Limited to Tweets from public users

- Always returns full Tweet payload 

- Twitter app permissions must be set to (or updated) to “Read, write and access Direct Messages”
Integration effort Low Low Medium Medium/High High

*The /mentions_timeline and /user_timeline endpoints are listed for easy comparison.

 

Which API is right for me?

Based on the comparison table above, you should have a better idea of which API(s) will better serve your needs and use case. If you’re still not sure, below are some additional aspects to consider:

API Consider this API if you....
Enterprise Search API
  • Require historical access to Tweets
  • Do not need Tweets in realtime (e.g., seconds after it’s published)
  • Prefer to continue using REST endpoints to request Tweets
  • Do not currently use HTTP streaming or webhook APIs today
  • Want to retrieve historical Tweets from users, topics, events, etc.
PowerTrack API
  • Require Tweets in realtime (seconds after they’re published)
  • Do not need historical access to Tweets
  • Are familiar with http streaming APIs (e.g., always-on stream of data)
  • Require high volume, low-latency Tweet data processing
  • Your use case involves realtime monitoring of mentions, conversations, events, and topics on Twitter  
Account Activity API
  • Require Tweets in realtime (seconds after they’re publishing)
  • Do not need historical access to Tweets
  • Want to receive all activities related to an account (e.g., Tweets, Retweets, mentions, DMs, Likes, and more)
  • Power customer service solutions on Twitter, build chatbots, or monitor activity for managed Twitter accounts
  • Use the Direct Message endpoints today and prefer a faster, more efficient way to receive inbound DMs

Enterprise Search API

The enterprise Search APIs offer low-latency, reliable, query-based access to publicly available Tweets ~30 seconds after they are published on the platform. We offer two different enterprise Search options:

  1. 30-Day Search: provides historical Tweets from the past 30 days.
  2. Full-Archive Search: provides complete access to the full corpus of historical Tweets dating back to the first Tweet in March 2006.

These REST APIs support complex boolean queries of up to 2,048 characters using the 30 different filter operators available – including the @ operator to capture mentions and the “from:” operator to capture Tweets from specific users.

Use cases: Retrieve historical brand mentions, perform analysis (e.g., how many Tweets mentioned a brand this year vs last year?), or get recent Tweets related to an event, topic, or trend.

Key characteristics

  • Historical access: retrieve past mentions, Tweets from users, or search on keywords, hashtags, and other attributes within the past 30 days or the full archive.
  • Equivalent functionality:
    • /mentions_timeline – “@” operator retrieves all mentions of an account.
    • /user_timeline – “from:” operator retrieves all Tweets from the specified user.
  • Infrastructure: minimal infrastructure changes will be required to migrate to the Search APIs as they provide a similar REST interface. The request structure will change slightly and the response payload may be larger as the ability to trim response objects goes away.
  • Limitations:
    • Date range: The mentions / user timeline endpoints were limited by a static (max) number of Tweets returned; whereas, the Search API will return all Tweets matching your query from within the specified time period (minute granularity available).
    • Public Tweets: the Search API only returns Tweets published by public users. The mentions / user timeline endpoints can return Tweets from protected accounts (user token must be provided) and Tweets that were created when an account was protected, but has since switched to public (unprotected).

 

Migration considerations

Below are some key technical considerations you’ll need to account for when planning your migration:

I. Authentication

Change authentication mechanism to HTTP basic auth (see docs).

II. Request structure

Adjust code to make requests to a new endpoint with a different JSON request body and required parameters. Both GET and POST requests are supported. A new query parameter is required.

III. Response structure

The Tweet payload is largely the same, but you will need to account for the full response payload (no ability to trim user/entities) and additional metadata enrichments. You may need to adjust your parser to account for, or simply ignore, new objects in the payload.

IV. Pagination

You will need to implement new pagination logic (see docs). When there is more data than can be returned in a single response, a “next” token will be present in the response that is needed to make the next request. This pattern should continue until there is no “next” token in the subsequent response(s).

Additional references:

 

PowerTrack API

PowerTrack provides filtered access to the full stream of publicly available Tweets in realtime. With over 50 different operators—including the @ mentions and from: operators—customers can ingest a targeted stream of only the data they care about.

Use cases: listen for brand mentions in realtime, monitor topics with complex, boolean filters, or detect emerging trends and anomalies to keep your customers informed.

Key characteristics

  • Realtime: delivers all Tweets to your stream that match your rules (filters) in realtime. Tweets are delivered seconds after they are published.
    • This is a significant change in the data delivery mechanism to an always-on, streaming service, rather than a pull-based, paginating endpoint.
  • Equivalent functionality:
    • /mentions_timeline – “@” operator retrieves all mentions of an account.
    • /user_timeline – “from:” operator retrieves all Tweets from the specified user.
  • Infrastructure: You will need to create a streaming client/infrastructure that can handle potentially high volumes of inbound data. A common architecture will include a “stream reader” that reads inbound data off of the stream and loads it into a queue for further processing (parsing, storing, etc). For more details, please see the Consuming streaming data tutorial.
  • Limitations: PowerTrack will only deliver Tweets that match your specified rules. It is up to you to create boolean rules using the available operators to match the Tweets that you want to receive through the stream.
    • Rate of inbound Tweets: the API does not offer a setting to control or limit the rate at which matching Tweets are delivered to your stream. In the event that your rule(s) matches a large volume of Tweets, they will be sent down the stream accordingly. It is important that your client code can handle this.
    • Public Tweets: PowerTrack only returns Tweets published by public users. The mentions / user timeline endpoints can return Tweets from protected accounts (user token must be provided) and Tweets that were created when an account was protected, but has since switched to public (unprotected).

 

Migration considerations

Below are some key technical considerations you’ll need to account for when planning your migration:

I. Authentication

Change authentication mechanism to HTTP basic auth (see docs).

II. Streaming infrastructure

You will need to create a streaming client/infrastructure to work with PowerTrack. This client will need to connect to and sustain an http streaming connection so that data can flow properly. Tweets that match your rule(s) are delivered in realtime, so the client will need to read potentially high volumes of inbound data and be flexible to bursts of data. It is recommended to have code that reads data off of the stream and loads it into a queueing system to separate this from the parsing, processing, and storing of data.

III. Rule management

PowerTrack provides a separate “rules” endpoint where you add and remove filters for your stream. This is managed independent of the stream endpoint, so that you can add / remove filters while maintaining your streaming connection.

IV. Request structure

To receive data, you must first create a rule (POST request to /rules endpoint) on your stream (e.g., from:twitterdev). Then, connect to the stream (GET request to /stream endpoint) and you will start seeing Tweet payload flow through as they are published in realtime.

V. Response structure

The Tweet payload is largely the same, but you will need to account for the full response payload (no ability to trim user/entities) and additional metadata enrichments. You may need to adjust your parser to account for, or simply ignore, new objects in the payload.

Additional resources:

 

Account Activity API

The Account Activity API (AAAPI) delivers activities related to an account in real-time, including “mentions”. Tweets from authenticated users, and all activity related to an account such as: Tweets, replies, likes, follows, direct messages, and more.

Use cases: power customer service interactions, build chatbot experiences, or monitor account activity for brands that you manage on Twitter.

Key characteristics

Realtime: delivers all activity related to a subscribed account, including “mentions”. Activities will be delivered (“posted”) to your registered webhook almost instantaneously.

  • It will substantially change the request pattern to a push-based service, rather than a pull-based, paginating endpoint.

Equivalent functionality:

  • /mentions_timeline – “mention” events of subscribed user (pushed to webhook in realtime)
  • /user_timeline – “Tweet” events from subscribed user (pushed to webhook in realtime)

Infrastructure: you will need to create a web app to serve as your “webhook” for Twitter to post events to. You will only receive events related to accounts (users) that are subscribed to your webhook.

Limitations: as is the case with /statuses/mentions_timeline, you must have a user token (user context) to retrieve mentions and other account activity.

  • Important: Account Activity API delivers DMs, so your Twitter app permissions must be set to “Read, write, direct messages”. If this is not the case, you will need to change your app permissions to this setting and regenerate existing access tokens to alter permissions levels.

 

Migration considerations

Below are some key technical considerations you’ll need to account for when planning your migration:

I. Authentication

Unchanged for new endpoints, but requires implementation of OAuth 2.0 for others.

II. Webhook infrastructure

Developers will need to build and host their own endpoints (webhooks). This requires creating a web app to serve as the “webhook” for Twitter to deliver (POST) events to. In more detail, you will need to:

  1. Create a webhook consumer app
  2. Secure your webhook (CRC)
  3. Register your webhook URL
  4. Make sure your web app supports POST requests for incoming events

III. App permissions

Check your Twitter app permissions to ensure the “read, write, and direct messages” is set. If this is not set already, you will need to change it at the app-level, then regenerate existing access tokens to alter the permissions levels (Note: this will likely involve re-authenticating users at the new permissions level)

IV. Subscribing accounts

You will need to make a POST request (3-legged OAuth - whitelisted consumer key and subscribing user's access token) to the Twitter subscriptions endpoint (see docs).

V. Data objects (response)

Data objects are JSON formatted “events”. There are 19 event types that flow through the API. Your HTTP service (webhook) must respond to inbound “events” within 3 seconds.

Additional resources: