Twitter API Toolkit for Google Cloud: Enterprise API

• Twitter API Toolkit for Google Cloud: Enterprise API -  framework can be used to detect macro and micro-level trends across domains and industry verticals

• Designed to horizontally scale and process higher volumes of Tweets in the order of millions of Tweets/day

• Automates the data pipeline process to ingest Tweets into Google Cloud

• Visualization of trends in an easy-to-use dashboard

• Below are a few real-time dashboard illustrations that are built with the Filtered Stream toolkit. 

• Fig 1, depicts a real-time ‘Gaming’ dashboard that depicts the conversations on Video games on Twitter. You can get insights on the trending topics in Gaming, popular hashtags, and the underlying Tweets that are streamed in real-time.

• Similarly, Fig 2 depicts the real-time analysis of the ‘Cryptocurrencies and Digital Assets’ 

• This toolkit helps you to build a real-time trend detection dashboard 

• Monitor real-time trends for configured rules as they unfold on the platform

This toolkit is comprised of 5 core components that are responsible for listening to real-time Tweets, pushing Tweets to a topic/queue, a CRON job that triggers a Tweet loader service which pulls the Tweets from the topic/queue to store them in a database, and, finally, Tweet visualization in a dashboard that connects to the database via a SQL query.

Tweet streamer service (nodeJs component)

The Tweet streamer service is responsible for listening to the real-time Twitter PowerTrack API and pushes the Tweets temporarily to a Topic that is based on Google PubSub. The PowerTrack rules are governed by the PowerTrack Rules API.

Stream topic based on Google PubSub

The stream topic based on Google PubSub acts as a shock absorber for this architecture. When there is a sudden surge of Tweets, the toolkit can handle the increased volume of Tweets with the help of PubSub. The PubSub topic will act as temporary storage for the Tweets and batch them for the Tweet loader service to store the Tweets in the database. Also, this will act as a shield to protect the database from a huge number of ingesting calls to persist the Tweets.

CRON job based on Google Cloud Scheduler

A CRON job based on Google Cloud Scheduler will act as a poller that will trigger the Tweet loader service at regular intervals.

Tweet loader service (nodeJs component)

The Tweet loader service that gets triggered based on the CRON job will pull the Tweets in a batch mode (i.e. 500 Tweets per pull, configurable via config.js file), and store the Tweets in a BigQuery database.

Google DataStudio as a Dashboard for analytics

Google DataStudio is used to create a dashboard for trend detection and connects to the BigQuery via a SQL query with a time interval as a parameter. The time interval is a range (30 minutes, 60 minutes) that will be used to query the Tweets based on their creation time. Trends can be analyzed based on the time interval which can range from minutes to hours. For example, you can analyze trends “60 minutes ago”, passing the time interval variable to the SQL query.

As a user of this toolkit, you need to perform four steps:

1. Add rules to the stream with the PowerTrack rules API endpoint

2. Install and involve the toolkit from GitHub in your Google Cloud project

3. Configure the CRON job - Google Cloud Scheduler

4. Configure the dashboard, by connecting to the BigQuery database with DataStudio

• Twitter Enterprise API access(pricing info will be provided by your account manager); Sign up here.

• [Optional] Get a Twitter API bearer token for accessing Engagement API. Refer to this doc.

• A Google Cloud Account. Sign up here

1. Add rules to the stream. You can add/update/delete the rules via the PowerTrack Rules API

2. Alternatively, you can add rules via the GNIP console. You should also have access to the GNIP console. Navigate to the GNIP console and select the PowerTrack stream you want to listen to with this application and click “Rules”.

3. In the “Edit Rules” dialog section, enter the rules and click “Update”. The rules must be compliant with the PowerTrack rules language

4. An example PowerTrack rule below for monitoring keywords related to Cryptocurrencies and NFT.

4. Ensure you have access to the BigQuery data owner role. Navigate to Google Cloud Console -> Choose IAM under the main menu -> Add a new IAM permission

Principal: Your Google account email address

Role: BigQuery Data Owner

5. From the “Cloud shell terminal”  command prompt, download the code for the toolkit be executing the command:

6. Navigate to the source code folder

7. Make changes to the configuration file. Use your favorite editor like vi or emacs
Once you’ve made the following changes, save them and quit the editor mode.

a. Update the “pt_stream_path” with the “Stream Path” from the GNIP console under the “Overview” section. Copy the stream URL from the string “/stream”

b. Update the GNIP username and password. These are the credentials required to log into the GNIP console

c. Update the Twitter Bearer token

d. Update the Google project ID

e. Update the “topicName” and “subscriptionName” with any unique string. Recommend something like “trends-topic” for topic name and “trends-sub” for subscription name

f. The “messageCount” attribute is the maximum number of messages that would be pulled from the topic by the stream loader service

g. The “streamReconnectCounter” represents the number of times stream loader service will withstand empty message pulls from topics before reconnecting. Recommend to leave it as 500 (max) and 3, however, would require calibration based on the topic you are listening to.

h. Update the “dataSetId” attribute with a unique string. Recommend “TrendsDS”

i. Update the “tweets” and “engagement” attributes with unique table names. Recommend “TweetTrends” and “Engagements”

8. Back in the cloud shell, deploy the code in AppEngine by executing the below command: Note that the deployment can take a few minutes.

9. After the deployment, get the URL endpoint for the deployed application with the command:

10. Use the endpoint URL from the output of step h) and make a CURL or browser request with “/stream” appended to the request path. This will invoke the toolkit and it starts listening to the Tweets as defined by the rules in the “stream/rules” endpoint.

11. Start tailing the log file for the deployment application

12. If the stream is not connected, reconnect to the stream with the below command:

1. Create a Google Cloud Scheduler Job by navigating to the Google Cloud console and clicking the Cloud Scheduler option under the main menu.

2. Create a new Cloud Scheduler job and define the schedule with frequency as below: Ensure a space between each asterisk like “* * * * *”

This will ensure that the Cloud Scheduler job triggers every minute.

3. Configure the execution with “Target Type” as “HTTP” and insert your application endpoint URL as below:

The request “/stream/poll” points to the “Tweet loader” service. Parameters 2 and 30000 refer to the invocation frequency of the Tweet loader service and the time interval in milliseconds. For example “2/30000” will invoke the “Tweet loader” service 2 times within a minute with a delay of 30000 milliseconds or 30 seconds. If you anticipate more Tweets for a topic, increase the invocation frequency and decrease the delay to increase the consumption. This is a calibration that can be fine-tuned based on monitoring of a specific topic like “Crypto” or “Doge coin”  

1. Create a Google Cloud Scheduler Job by navigating to the Google Cloud console and clicking the Cloud Scheduler option under the main menu.

2. Create a new Cloud Scheduler job and define the schedule with frequency as below: Ensure a space between each asterisk like “5 * * * *”

3. Configure the execution with “Target Type” as “HTTP” and insert your application endpoint URL as below:

Set the HTTP method to GET and add an HTTP header of name User-Agent and value Google-Cloud-Scheduler.

The request “/stream/engagement” triggers the Engagement API /totals endpoint and saves the engagement metrics in the Engagement table defined in the config.js file. You can get the most “liked” Tweet or Tweet with the most video views by joining the Engagement table with the Tweet table.

1. SQL query to be used for Trend detection in DataStudio

2. Replace your <<datasetId.table_name>> in the below SQL. The ‘datasetId’ and ‘tweet_table_name’ can be determined from the config.js file

3. SQL to be used for engagement metrics

It is crucial that any developer who stores Twitter content offline ensures the data reflects user intent and the current state of content on Twitter. For example, when someone on Twitter deletes a Tweet or their account, protects their Tweets, or scrubs the geoinformation from their Tweets, it is critical for both Twitter and our developers to honor that person’s expectations and intent. The batch compliance endpoints provide developers an easy tool to help maintain Twitter data in compliance with the Twitter Developer Agreement and Policy.

Optional - Delete the Google cloud project to avoid any overage costs