Tutorials / Getting historical Tweets using the full-archive search endpoint

Getting historical Tweets using the full-archive search endpoint

Introduction

The Twitter API allows you to analyze the organic performance of Tweets by giving you the ability to see a series of metrics associated with a given Tweet. This can be helpful in making data-driven decisions around how specific Tweets are performing. Tweet lookup is a GET method that returns information about a Tweet or group of Tweets. Metrics are available as part of the Tweet lookup endpoint.

In this tutorial, you will learn how to:

  • Determine what questions you are trying to solve for
  • Connect and authenticate to the Twitter API
  • Decide what data you need and make a GET request
  • Add in any additional logic needed to answer the question you are trying to solve
     

Prerequisites


Steps to consider

Step 1: Determine what questions you are trying to solve for

In order to measure Tweet performance, you may want to take a moment to determine what you are looking for while analyzing data. This may be a hypothesis you are trying to prove or disprove, a problem you are trying to look deeper into, or a question you are attempting to learn the answer of. 

Examples of questions you can ask are:

  • Do Tweets with positive sentiment scores perform better than Tweets with negative sentiment scores?
  • Which type of language resonates the best with my audience?
  • Are my recent Tweets performing below or above my personal average?
  • What were the top 10 Tweets of the year from our organization?
     

Step 2: Connect and authenticate to the Twitter API

In order to connect to the Twitter API, you first will need to authenticate by passing in your credentials before you can get any data back. In order to do so, you should use a library to help you with OAuth. 

You will want to make sure to store your credentials in a secure way. Some options include using environment variables or setting up a configuration file that you can store in your .gitignore file on GitHub. You can learn more about authentication and read more about security best practices. 
 

Step 3: Decide what data you need and make a GET request

The data returned in your GET request is determined by the fields and expansions you include in your request. You will want to consider how many Tweets you want to learn more about and determine if it’s a single Tweet or multiple Tweets. Public metrics are what is available for all Tweets whereas non-public metrics are only available for your own or authorized accounts. Public and non-public fields return the sum of organic metrics which are posted and viewed in a regular manner on Twitter and promoted which are posted and viewed as part of an Ads campaign. Additionally, if you are looking for public or non-public metrics around media and engagement, identify what fields the data you are looking for is located in. Metrics that are available include impressions, retweets, quote Tweets, likes, URL link clicks, user profile clicks, replies, video views, and video view quartiles. Learn more about what is available with the API reference guide for Tweet lookup.

An example of a cURL request to a single Tweet is:

      curl --request GET 'https://api.twitter.com/2/tweets/1263150595717730305?tweet.fields=public_metrics' --header 'Authorization: Bearer <Bearer Token>'

The payload you get in return is:

      {"data":{"text":"Do you \uD83D\uDC40our new Tweet settings?\n\nWe want to know how and why you’d use a feature like this in the API. Get the details and let us know what you think\uD83D\uDC47\nhttps://t.co/RtMhhfAcIB https://t.co/8wxeZ9fJER","id":"1263150595717730305","public_metrics":{"retweet_count":12,"reply_count":13,"like_count":50,"quote_count":7}}}

You will need to replace <Bearer Token> with your own bearer token. An example of a cURL request for multiple Tweets:

      curl --request GET 'https://api.twitter.com/2/tweets?tweet.fields=public_metrics&ids=1263150595717730305,1262485275348885504' --header 'Authorization: Bearer <Bearer Token>'

The payload you get in return is as follows:

      {"data":[{"text":"Do you \uD83D\uDC40our new Tweet settings?\n\nWe want to know how and why you’d use a feature like this in the API. Get the details and let us know what you think\uD83D\uDC47\nhttps://t.co/RtMhhfAcIB https://t.co/8wxeZ9fJER","public_metrics":{"retweet_count":12,"reply_count":13,"like_count":50,"quote_count":7},"id":"1263150595717730305"},{"text":"Starting today, we’re rolling out a new look for embedded Tweets.\n\nCheck it out and share your feedback in the forum post ⬇️\nhttps://t.co/gkMD0w7mFs","public_metrics":{"retweet_count":42,"reply_count":6,"like_count":85,"quote_count":12},"id":"1262485275348885504"}]}

Using a REST client such as Postman or Insomnia can be helpful for seeing what data you get back and making adjustments before you start writing code for an iterative approach to adjusting your URL. We also have a Postman collection which might provide a good starting point. 

You may want to check out our sample code for Tweet lookup at this point:


Step 4: Add in any additional logic needed to answer the question you are trying to solve.

After you have a payload that contains the proper fields you can make sure your code has built-in logic, timing, or API connections to perform the task you identified in the first step.

To learn if Tweets with positive sentiment scores perform better than Tweets with negative sentiment scores, you will need to get the Tweet IDs, and text of the Tweets and metrics from all the Tweets from the length of time you are looking to measure. You may also need to connect to a library or an API to determine sentiment scores for each of your Tweets. 

To determine what type of language resonates the best with my audience you will need to pull all the Tweets from the timeframe you have in mind. You may need to run natural language processing on your Tweets to create a starting point to the compare metrics

In order to see if your recent Tweets performing below or above your personal average. To determine your personal average you need to determine what metrics you are looking for the average of and perform a calculation. For example, this could be the average of likes plus retweets. You will need to determine the average for your recent Tweets based on the same calculations and you can compare the differences. 

To see what the top 10 Tweets of the year from your organization are, you need to pull all your Tweets for the year and sort them by the metrics you determine to be the most important to your organization.

 

Next steps

Ready to build your solution?

Apply for developer access to get started

Apply for access