Tutorials / Analyze past conversations
Twitter data is used by developers, students, and researchers to study various topics. You may want to study the conversation around a topic using hashtags, set of keywords, etc. from the last few days. For example, maybe you want to study the sentiment of Tweets about a football game over the weekend. You can get data for such use cases using Twitter’s recent search endpoint. This endpoint gives you conversations from Twitter for the last 7 days.
In this guide, you will learn how to:
Identify which past conversation you wish to study
Connect and authenticate to the recent search endpoint to receive Tweets
Analyze the data for past conversations
As a first step, you need to identify the type of conversation you wish to study. For example, you may decide to study all Tweets about the climate for the last few days. In order to do this, you might be looking for Tweets with a certain hashtag or words. Once you have identified these words, you will need to convert these into a query (also referred to as rule or filter). You can define the rule to include/exclude keywords, images, etc. You can also specify certain account names to get Tweets from. Check out this guide that shows you how to use the operators that are supported by the recent search endpoint that lets you get that data you need.
Once you have decided what data you want and what operators you will use, you can connect to the recent search endpoint. The example below shows how you can use cURL to do so. To authenticate, replace
$BEARER_TOKEN with the bearer token from your App in the developer portal and
$QUERY using your query.
curl -X GET -H "Authorization: Bearer $BEARER_TOKEN" "https://api.twitter.com/2/tweets/search?tweet.fields=created_at,author_id,lang&query=$QUERY"
So for example, if you wish to get all Tweets with images about the hashtag
#Caturday, excluding retweets your query will look like:
“#caturday has:images -is:retweet”
And thus, your cURL call will be as follows:
curl -X GET -H "Authorization: Bearer $BEARER_TOKEN" "https://api.twitter.com/labs/2/tweets/search?query=caturday%20has:images%20-is:retweet&tweet.fields=created_at,author_id,lang"
You can also connect to the recent rearch endpoint using one of our code samples:
This will return 10 Tweets objects by default. You can also specify a maximum number of results you want to receive per API call. To do so, you can use the max_results query parameter with your API call. In order to get additional Tweets, you need to handle pagination. To do so, you will need to access the next_token from the object called meta and connect to get the additional Tweet objects back for the next 10 Tweets (by default unless you specify a number with the max_results parameter) until there is no longer a next_token available.
By default, you will get back the id and text of each Tweet. However, you can customize what additional fields you want to return as part of the Tweet payload. You can include these additional fields in the payload by adding additional fields and expansions to your query.
Instead of making curl calls from your terminal, you can use a REST client such as Postman or Insomnia to get data from the recent search endpoint. Our Postman collection will help you get started quickly in calling the recent search endpoint.
The Twitter API provides a good opportunity to study historical data. Examples of analysis that you might be interested in doing with Twitter data include:
Mapping Tweets by location
Sentiment analysis on Tweets about past events
Identifying influencers on Twitter
There are various libraries available in programming languages such as Python, R, etc. that let you analyze the data obtained from Twitter API.
In R, you can use something like Tidyverse (which is a collection of R packages for data science) for data analysis. You can use ggplot2, which is part of Tidyverse to build data visualization using Twitter data.
There are a variety of visualizations that you can build using these libraries mentioned above. Some examples of such visualizations that can be built using Twitter data include:
Histograms for displaying Tweet frequency
Map for displaying Tweets by Geo-Location
Time series analysis of Trends
Check out this sample App that demonstrates how to use the recent search endpoint to analyze sentiments of your own Tweets.
Ready to build your solution?