Comparing Labs and standard search endpoints

If you have been working with the standard search endpoint, the goal of this guide is to help you understand the similarities and differences between the standard and Labs search endpoints. If you have code that you want to use with Labs, keep reading to learn more about updating that code to work with Labs. 

When comparing the standard and Labs query languages, note that while there is a set of common operators between the two, there are also important differences. Most notably is that the OR / AND precedence, or the order these logic booleans are applied, are the opposite. 

Here we are comparing these Twitter search endpoints:

  • Standard recent search: https://api.twitter.com/1.1/search/tweets.json
  • Labs recent search: https://api.twitter.com/labs/1/tweets/search
     

See these sections to learn more about the similarities and differences between Labs and standard search endpoints.

 

Similarities

Authentication

The standard search endpoint supports both OAuth 2.0 Bearer Token and OAuth 1.0a. The Labs recent search only supports app-only OAuth 2.0 Bearer Token authentication. If you have been working with the standard search endpoint and have been making app-only requests, then your code is likely ready for making requests to the Labs recent search endpoint. 

Depending on your authentication library/package of choice, OAuth 2.0 Bearer Token is probably the easiest way to get started and can be set with a simple request header. To learn how to generate a Bearer Token, see this Application-only authentication guide. Also, the Quick Start guide includes example code in Python, Ruby, Node.js, and Java that generates Bearer Tokens.
 

Supported HTTP request methods

The Labs recent search and standard search endpoints support GET requests and do not support POST requests.

 

Common query operators

  • keyword - matching Tweets that contain a specific word:
    winter
  • exact phrases - matching on Tweets with words in a specific order:
    "freezing rain"
  • # - Matches Tweets with a hashtag:
    #COwx
  • @ - Matches Tweets that mention a Twitter account:
    @TwitterDev
  • from: - MatchesTweets from a Twitter account:
    from:TwitterDev
  • to: - Matches Tweets directed to a Twitter account:
    to:TwitterDev
  • lang: - Matches Tweet bodies detected as being written in a specific language:
    lang:en
  • source: - Matches Tweets from a specified source:
    source:ios OR source:android OR source:”Twitter Web Client”
  • url: - Matches on link tokens included in the Tweet body:
    url:TwitterDev
     

Differences

Request parameters

The following standard request parameters have equivalents in Labs:

Standard

Labs

q

query

 

start_time  (YYYY-MM-DDTHH:mm:ssZ)

until (YYYY-MM-DD)

end_time  (YYYY-MM-DDTHH:mm:ssZ)

since_id

since_id

max_id

until_id

count

max_results

Response provides search_metadata.next_results

next_token


There are also a set of standard search request parameters not supported in Labs:

Standard

Comment

geocode

While Labs does not currently support geo Operators, these are available in other Tweet search APIs. 

locale

With standard search, this was used to specify the language of the query but never fully implemented. 

lang

The Labs recent search endpoint provides a lang query operator for matching on languages of interest. 

include_entities

Tweet entities are always included.

result_type

The Labs recent search endpoint delivers all matching Tweets, regardless of engagement level. 

extended

Labs is built from the ground up to support Tweets with up to 280 characters. With Labs, there is no concept of 'extended' Tweets. 


AND / OR operator precedence 

The basic building block for building search queries is the use of OR and AND logical groupings. The standard search API applies ORs before ANDs, while the Labs recent search endpoint (as well as the premium and enterprise versions) applies ANDs before ORs. This difference is critical when translating queries between the two. 

For example, with standard search, if you wanted to match Tweets with the keyword ‘storm’ that mention the word ‘colorado’ or the #COwx hashtag, you could do that with the following search query:

storm #COwx OR colorado

With the Labs operator precedence, ANDs are applied before ORs. So the above query is equivalent to: 

(storm #COwx) OR colorado

However, the above rule would match on any Tweets that mentions ‘Colorado’, regardless if the Tweet mentions ‘storm’ or the #COwx hashtag. In addition it would also  match Tweets that mentioned both the keyword ‘storm’ and the #COwx hashtag. 

To make the query behave as originally intended, the OR clauses need to be grouped together. The translation of the original standard query to Labs would be:  

storm (#COwx OR colorado)

These two rules have very different matching behavior. For the month of October 2019, the original rule matches over 1,175,000 Tweets while the correctly translated rule matches around 5,600 Tweets. Be sure to mind your ANDs and ORs, and use parentheses where needed. 

Tweet and User object JSON

Labs is introducing new JSON designs for the objects returned by the APIs, including Tweet and User objects. The Labs JSON formats represent the future JSON formats that Twitter APIs will return. We are taking this opportunity to update the JSON key names of common objects and their attributes. Here are some examples of these changes:

  • At the JSON root/top-level, the standard search endpoint returns Tweet objects in a statuses array, while Labs search returns a data array.
  • Instead of using the term statuses, the term tweet will be used. 
  • Many legacy and deprecated fields, such as contributors and user.translator_type are being removed.
  • Instead of using both favorites (in Tweet object) and favourites (in user object), Labs uses the term like.  
  • Lab formats are adopting the convention that JSON values with no value (e.g., null) are not written to the payload. Tweet and User attributes are only included if they have a non-null value. 

To see the new JSON field names, see the response section of the Labs recent search API Reference guide.

Response pagination metadata

With standard search, response payloads include a search_metadata section with next_results URLs for cases when more pages of data are available, and refresh_url URLs, since_id, and max_id  metadata for polling use cases. 

With Labs recent search, response payloads include a meta section with a next_token when another page of data is available, and newest_id and oldest_id Tweet IDs to support polling use cases. 
 

Next steps