Twitter API / Tutorials
Get customized Tweet notifications where you want them
Relevant Endpoints
Introduction
Many times people want to be notified when someone Tweets about a certain topic of interest to them, or when an account of interest to them Tweets something. For example, I might want to be notified whenever a person Tweets about Academic Research with Twitter data, so that I can learn about how that person is using Twitter APIs for their research and what kind of research they are doing. Similarly, somebody might want to be notified about breaking news stories or conversations pertaining to certain brands. They may want to be notified via email and SMS, or they may want to receive the links to the Tweets of interest as a message in their Slack or Amazon Chime channel.
In this tutorial, you’ll learn how to build an app in Java that publishes links to Tweets based on user defined interests. Specifically, the app will publish Tweets from some news channel’s handles containing the words "breaking news". We’ll use Twitter’s recent search endpoint to push Tweets to tools such as Slack, Chime, or even SMS, using Twilio. This is a serverless app and can be configured quickly to AWS Lambda without requiring code-changes.
In order to start, you will need a Twitter developer account and an AWS account. Slack, Chime and Twilio are optional (If these are not setup, then the app will simply print the links to the Tweets to your CloudWatch logs).
As I mentioned above, this app will use v2 of Twitter’s recent search endpoint. It supports over 30 filtering operators, and can use search queries up to 512 characters in length when using the Standard product track, and 1024 characters long when using the Academic Research product track. The v2 version of this endpoint also supports requesting specific fields and objects to return with each Tweet using the fields and expansions parameters, and return a new Tweet JSON design.
Additionally, the app is ‘serverless’ and also demonstrates:
- AWS Lambda, Secrets Manager & CloudWatch
- Storing Tweet IDs in DynamoDB
- Publishing messages to Slack & Amazon Chime
- Sending SMS using Twilio using Java SDK
The complete source code is available on GitHub.
Architecture
An AWS CloudWatch event triggers a Lambda function every hour. The Lambda function then calls the Twitter recent search endpoint and gets the Tweets of interest for that time period. It stores the new Tweet IDs in DynamoDB and publishes the links to Tweets to Slack, Amazon Chime and sends it as SMS to the phone number configured with Twilio.
Step 1: Creating a Twitter App
In order to use the Twitter APIs in this app, you will need the following:
- A Twitter developer account
- A Twitter App associated with a Project
- Your Twitter API keys, which can be retrieved from the App
Step 2: Identifying Tweets of interest
You can search by Tweets from a particular account, Tweets that include images, media etc. and also filter on excluding certain Tweets or conditions. More information on building queries for recent search can be found on our documentation page.
In this example, we have set SEARCH_STRING to:
"(from:@CNN OR from:@BBCWorld) lang:en \"breaking news\" -is:retweet"
... which means we are looking for all Tweets from the CNN or BBC World accounts that are not a Retweet, are in English, and contain the string breaking news.
Additional details on implementing the recent search endpoint in Java can be found in the TwitterService.java file. Make sure to set the value of the SEARCH_STRING appropriately in your Lambda environment variable with your own search string for Tweets of interest to you. I use the start_time query parameter to ensure that I am getting Tweets from one hour ago. The query parameters are added as shown below (from the TwitterService.java file):
URIBuilder uriBuilder = new URIBuilder("https://api.twitter.com/2/tweets/search/recent");
ArrayList<NameValuePair> queryParameters;
queryParameters = new ArrayList<>();
queryParameters.add(new BasicNameValuePair("query", searchString));
queryParameters.add(new BasicNameValuePair("start_time", startTime));
queryParameters.add(new BasicNameValuePair("expansions", "author_id"));
uriBuilder.addParameters(queryParameters);
Step 3: Setting up the Slack webhook
In order to send Tweets of interest to a Slack channel, you will need to create a webhook by following the instructions on their documentation. Copy the webhook url as you will store this later in AWS Secrets Manager and use this to publish Tweets that match your criteria to your slack channel. Make sure you specify the name of your slack bot by setting it as an environment variable BOT_NAME in your Lambda function. In this sample app, I have used a third-party Java SDK for Slack that makes it easy for me to publish Tweets to a channel. More information on the implementation can be found in the SlackService.java file.
Step 4: Setting up the Amazon Chime webhook
In order to send Tweets to a Chime channel, you will need to add a webhook to your chat room – just follow the Chime documentation to set it up. Copy the webhook URL, so that you can specify it in AWS Secrets Manager later. A POST call is required to the webhook url to publish messages to the chime channel as shown below:
public void postMessage(String url) throws IOException {
if(validParameters(webhook)) {
HttpClient httpClient = HttpClients.custom()
.setDefaultRequestConfig(RequestConfig.custom()
.setCookieSpec(CookieSpecs.STANDARD).build())
.build();
HttpPost request = new HttpPost(webhook);
StringEntity params =new StringEntity(String.format("{\"Content\":\"%s\"}", url));
request.addHeader("content-type", "application/json");
request.setEntity(params);
httpClient.execute(request);
}
}
More information on the implementation can be found in the ChimeService.java file.
Step 5: Setting up Twilio to send and receive SMS
In order to send SMS using Twilio, you will need to create an account and set up a Twilio app. Please refer to the Twilio documentation on getting started in Java. You will need the sid, token and the phone numbers you want to send and receive messages from. I have used Twilio’s Java SDK and the implementation can be found in the TwilioService.java file.
Step 6: Setting up a table in DynamoDB to store Tweet IDs
In this tutorial, we will store the Tweet IDs of the published Tweets in DynamoDB. In order to do that, you will have to create a table in DynamoDB with a primary key (String) to store the Tweet IDs. More information on creating tables in DynamoDB can be found in their docs.
Copy the table name and primary key id, because you will save them in AWS Secrets Manager. Implementation of storing the Tweet IDs in DynamoDB can be found in the DynamoService.java file.
Step 7: Storing keys in AWS Secrets Manager
Instead of storing API Keys, Secrets, Webhooks etc. in properties files, we will store them in AWS Secrets Manager and use them in our code programmatically. AWS has a nice tutorial on adding secrets to Secrets Manager.
First create a secret in your Secrets Manager console.
Select Other type of secrets.
In the section where it says Specify the key/value pairs to be stored in this secret, add the following keys (and their corresponding values from the previous steps):
- twitter-api-key
- twitter-api-secret
- slack-webhook-url
- chime-webhook-url
- dynamo-table-name
- dynamo-table-pk
- twilio-sid
- twilio-token
- twilio-from-number
- twilio-to-number
Click next and give the secret a name and save it. Make sure to set the environment variable SECRET_NAME for your Lambda function with this secret name. Also make sure that the right AWS region environment variable REGION is set (by default it is set to us-east-1). Implementation of retrieving secrets from Secrets Manager can be found in the SecretsManagerService.java file.
Step 8: Creating a CloudWatch Events rule
In order to invoke the Lambda function periodically, we will need to create a CloudWatch Trigger rule that will invoke the Lambda function (every hour in this example). If you need more details, you can follow a tutorial for setting CloudWatch rule in the AWS docs. Make sure to set the trigger interval to 1 hour.
Step 9: Building the project
Ensure you have cloned the code from the Github repository.
Using maven: go to the directory containing pom.xml, and run:
mvn assembly:assembly -DdescriptorId=jar-with-dependencies package
This will generate a zip file named "tweet-notifier-1.0-jar-with-dependencies.jar" in the target directory.
Step 10: Deploying to AWS Lambda
In order to deploy the code to Lambda:
- Go to the AWS Console and create a Lambda function.
- Select author from scratch and give your function a name (e.g. twitter-notifier) and select Java 8 as the runtime.
- Select appropriate role and click create function.
- For Code entry type, select "Upload a .ZIP file" and then upload the jar file created in the previous step (step 9) from the build directory to Lambda.
- Set the Handler as LambdaHandler
- Increase the Timeout to 30 seconds and make sure memory is set to at least 512 MB under Basic Settings.
- Click add trigger, and select Cloudwatch Events/EventBridge. Under rules, click on the rule you created (in step 8) and click add.
- Click manage environment variables and add the following environment variables
- Click manage environment variables and add the following environment variables
- BOT_NAME (with the value of name for your Slack chatbot)
- REGION (with the value of your AWS region)
- SECRET_NAME (with the value of the name of your secret that has your keys)
- SEARCH_STRING (with the search query string for the Tweets you wanted to be notified about)
- Click save to complete setting up the Lambda function.
Once you have deployed the app to Lambda, you will receive links to your chosen Tweets of interest every hour delivered to your Slack or Chime channels or via SMS.
I used several libraries and tools beyond the Twitter API to create this code sample, but you may have different needs and requirements and should evaluate whether those tools are right for you. Twitter does not operate or manage the third party tools mentioned above and those tools may have separate terms and limits regarding use of any webhooks and other features. Try it out and reach out to me on Twitter @suhemparack or @TwitterDev with feedback or questions.
Ready to build your solution?