Search Tweets: recent search
Standard v1.1 compared to Twitter API v2
If you have been working with the v1.1 search/tweets, the goal of this guide is to help you understand the similarities and differences between the standard and Twitter API v2 search Tweets endpoint.
- OAuth 1.0a User Context and OAuth 2.0 Bearer Token
- Endpoint URLs
- App and Project requirements
- Request parameters
- New JSON format
OAuth 1.0a User Context and OAuth 2.0 Bearer Token authentication
Therefore, if you were previously using the standard v1.1 search endpoint you can continue using the same authentication method if you migrate to the Twitter API v2 version.
Depending on your authentication library/package of choice, Bearer Token authentication is probably the easiest way to get started and can be set with a simple request header. To learn how to generate a Bearer Token, see this OAuth 2.0 Bearer Token guide.
If you would like to take advantage of the ability to pull private or advertising metrics with the Twitter API v2 endpoint, you will need to use OAuth 1.0a User Context, and pass the user access tokens related to the user who posted the Tweet for which you would like to pull metrics.
- Standard v1.1 endpoints:
- Twitter API v2 endpoint:
App and Project requirements
The Twitter API v2 endpoints require that you use credentials from a developer App that is associated to a Project when authenticating your requests. All Twitter API v1.1 endpoints can use credentials from standalone Apps or Apps associated with a project.
The following standard v1.1 request parameters have equivalents in Twitter API v2:
|Standard search v1.1||Search Tweets v2|
|until (YYYY-MM-DD)||end_time (YYYY-MM-DDTHH:mm:ssZ)|
|Response provides search_metadata.next_results||next_token|
One of the biggest differences between standard v1.1 and Twitter API v2 endpoint versions is how you select which fields return in your payload. For the standard endpoints, there are several parameters that you could use to identify which fields or sets of fields would return in the payload, while the Twitter API v2 version simplifies these different parameters into fields and expansions.
fields: Twitter API v2 endpoints enable you to select which fields are provided in your payload. For example, Tweet, user, Media, Place, and Poll objects all have a list of fields that can be returned (or not).
expansions: Used to expand the complementary objects referenced in Tweet JSON payloads. For example, all Retweets and Replies reference other Tweets. By setting expansions=referenced_tweets.id, these other Tweet objects are expanded according to the tweet.fields setting. Other objects such as users, polls, and media can be expanded.
There are also a set of standard search Tweets request parameters not supported in Twitter API v2:
Standard v1.1 parameter
Search Tweets at the Basic Access level does not support geo operators.
With standard search, this was used to specify the language of the query but never fully implemented.
Search Tweets endpoints provide a lang query operator for matching on languages of interest.
Tweet entities are always included.
Search Tweets endpoints deliver all matching Tweets, regardless of engagement level.
Twitter API v2 is built from the ground up to support Tweets with up to 280 characters. With v2, there is no concept of 'extended' Tweets.
Here is an example request that shows the difference between a Standard request and a Search Tweets request:
Twitter API v2
These requests will both return the 50 most recent Tweets that contain the keyword snow. The v2 request will return the default id and text fields of the matching Tweets. Here is an example of specifying additional Tweet and user fields to include in the JSON payload:
Twitter API v2
Common query operators
- keyword - matching Tweets that contain a specific word:
- exact phrases - matching on Tweets with words in a specific order:
- # - Matches Tweets with a hashtag:
- @ - Matches Tweets that mention a Twitter account:
- from: - MatchesTweets from a Twitter account:
- to: - Matches Tweets directed to a Twitter account:
- lang: - Matches Tweet bodies detected as being written in a specific language:
- url: - Matches on link tokens included in the Tweet body:
New query operators
Search Tweets introduces new operators in support of two new Twitter API v2 features:
- Conversation IDs - As conversations unfold on Twitter, a conservation ID will be available to mark Tweets that are part of the conversation. All Tweets in the conversation will have their conversation_id set to the Tweet ID that started it.
- Twitter Annotations provide contextual information about Tweets, and include entity and context annotations. Entities are comprised of people, places, products and organizations. Contexts are domains, or topics, that surfaced entities are a part of. For example, people mentioned in a Tweet may have a context that indicates whether they are an athlete, actor, or politician.
- context: matches on Tweets that have been annotated with a context of interest.
- entity: matches on Tweets that have been annotated with an entity of interest.
AND / OR operator precedence
The basic building block for building search queries is the use of OR and AND logical groupings. The standard search API applies ORs before ANDs, while Search Tweets endpoints (as well as the premium and enterprise versions) applies ANDs before ORs. This difference is critical when translating queries between the two.
For example, with standard search, if you wanted to match Tweets with the keyword ‘storm’ that mention the word ‘colorado’ or the #COwx hashtag, you could do that with the following search query:
storm #COwx OR colorado
With the Search Tweets operator precedence, ANDs are applied before ORs. So the above query is equivalent to:
(storm #COwx) OR colorado
However, the above rule would match on any Tweets that mentions ‘Colorado’, regardless if the Tweet mentions ‘storm’ or the #COwx hashtag. In addition it would also match Tweets that mentioned both the keyword ‘storm’ and the #COwx hashtag.
To make the query behave as originally intended, the OR clauses need to be grouped together. The translation of the original standard query to Search Tweets would be:
storm (#COwx OR colorado)
These two rules have very different matching behavior. For the month of October 2019, the original rule matches over 1,175,000 Tweets while the correctly translated rule matches around 5,600 Tweets. Be sure to mind your ANDs and ORs, and use parentheses where needed.
New JSON format
- At the JSON root level, the standard endpoints return Tweet objects in a statuses array, while Twitter API v2 returns a data array.
- Instead of referring to Retweeted and Quoted "statuses", Twitter API v2 JSON refers to Retweeted and Quoted Tweets. Many legacy and deprecated fields, such as contributors and user.translator_type are being removed.
- Instead of using both favorites (in Tweet object) and favourites (in user object), Twitter API v2 uses the term like.
- Twitter is adopting the convention that JSON values with no value (for example, null) are not written to the payload. Tweet and user attributes are only included if they have a non-null values.
In addition to the changes that we made to our new JSON format, we also introduced a new set of fields to the Tweet object including the following: