Twitter API Toolkit for Google Cloud: Filtered Stream

• Twitter API Toolkit for Google Cloud: Filtered Stream -  framework can be used to detect macro and micro-level trends across domains and industry verticals

• Designed to horizontally scale and process higher volumes of Tweets in the order of millions of Tweets/day

• Automates the data pipeline process to ingest Tweets into Google Cloud

• Visualization of trends in an easy-to-use dashboard

• Below are a few real-time dashboard illustrations that are built with the Filtered Stream toolkit. 

• Fig 1, depicts a real-time ‘Gaming’ dashboard that depicts the conversations on Video games on Twitter. You can get insights on the trending topics in Gaming, popular hashtags, and the underlying Tweets that are streamed in real-time.

• Similarly, Fig 2 depicts the real-time analysis of the ‘DogeCoing Cryptocurrency’ 

• This toolkit helps you to build a real-time trend detection dashboard 

• Monitor real-time trends for configured rules as they unfold on the platform

• The dashboards below is an example built with the toolkit that illustrates real-time trends in gaming

This toolkit is comprised of 5 core components that are responsible for listening to real-time Tweets, pushing Tweets to a topic/queue, a CRON job that triggers a Tweet loader service which pulls the Tweets from the topic/queue to store them in a database, and, finally, Tweet visualization in a dashboard that connects to the database via a SQL query.

• Tweet streamer service (nodeJs component)

The Tweet streamer service is responsible for listening to the real-time Twitter Filtered Stream API and pushes the Tweets temporarily to a Topic that is based on Google PubSub. The Filtered Stream rules are governed by the Filtered Stream Rules API.

• Stream topic based on Google PubSub

The stream topic based on Google PubSub acts as a shock absorber for this architecture. When there is a sudden surge of Tweets, the toolkit can handle the increased volume of Tweets with the help of PubSub. The PubSub topic will act as temporary storage for the Tweets and batch them for the Tweet loader service to store the Tweets in the database. Also, this will act as a shield to protect the database from a huge number of ingesting calls to persist the Tweets.

• CRON job based on Google Cloud Scheduler

A CRON job based on Google Cloud Scheduler will act as a poller that will trigger the Tweet loader service at regular intervals.

• Tweet loader service (nodeJs component)

The Tweet loader service that gets triggered based on the CRON job will pull the Tweets in a batch mode (i.e. 25 Tweets per pull, configurable via config.js file), and store the Tweets in a BigQuery database.

• Google DataStudio as a Dashboard for analytics

Google DataStudio is the dashboard for trend detection and connects to the BigQuery via a SQL query with a time interval as a parameter. Trends can be analyzed based on the time interval which can range from minutes to hours. For example, you can analyze trends “60 minutes ago”, passing the time interval variable to the SQL query.

As a user of this toolkit, you need to perform four steps:

1. Add rules to the stream with the Filtered Stream rules API endpoint

2. Install and involve the toolkit from GitHub in your Google Cloud project

3. Configure the CRON job - Google Cloud Scheduler

4. Configure the dashboard, by connecting to the BigQuery database with DataStudio

• Twitter Developer account! Sign up here

• Get a Twitter API bearer token. Refer to this doc

• A Google Cloud Account. Sign up here

1. Add rules to the stream. Let’s listen to Tweets related to “DogeCoin”, however, we only want to listen to 10% of random Tweets of “DogeCoin”. This can be accomplished by 

2. Validate the rules

3. You should get an output like the below if there are no rules previously added. If you have previously added rules, they will also be returned here; ensure you delete them.

4. Ensure you have access to the BigQuery data owner role. Navigate to Google Cloud Console -> Choose IAM under the main menu -> Add a new IAM permission

Principal: Your Google account email address

Role: BigQuery Data Owner

5. From the “Cloud shell terminal”  command prompt, download the code for the toolkit be executing the command:

6. Navigate to the source code folder

7. Make changes to the configuration file. Use your favorite editor like vi or emacs
Once you’ve made the following changes, save them and quit the editor mode.

8. Back in the cloud shell, deploy the code in AppEngine by executing the below command: Note that the deployment can take a few minutes.

9. After the deployment, get the URL endpoint for the deployed application with the command:

10. Use the endpoint URL from the output of step 8) and make a CURL or browser request with “/stream” appended to the request path. This will invoke the toolkit and it starts listening to the Tweets as defined by the rules in the “stream/rules” endpoint

11. Start tailing the log file for the deployment application

12. If you don’t see the messages in the logs console as “Received Tweet” or “~~ Heartbeat Payload ~~”, the stream may be disconnected. To reconnect to the stream, make a curl/browser request as below:

1. Create a Google Cloud Scheduler Job by navigating to the Google Cloud console and clicking the Cloud Scheduler option under the main menu.

2. Create a new Cloud Scheduler job and define the schedule with frequency as below: Ensure a space between each asterisk like “* * * * *”

This will ensure that the Cloud Scheduler job triggers every minute.

3. Configure the execution with “Target Type” as “HTTP” and insert your application endpoint URL as below:

The request “/stream/poll” points to the “Tweet loader” service. Parameters 2 and 30000 refer to the invocation frequency of the Tweet loader service and the time interval in milliseconds. For example “2/30000” will invoke the “Tweet loader” service 2 times within a minute with a delay of 30000 milliseconds or 30 seconds. If you anticipate more Tweets for a topic, increase the invocation frequency and decrease the delay to increase the consumption. This is a calibration that can be fine-tuned based on monitoring of a specific topic like “Crypto” or “Doge coin”  

1. SQL query to be used for Trend detection in DataStudio

2. Replace your <<datasetId.table_name>> in the below SQL. It should look something like “sixth-hawk.tweets”

It is crucial that any developer who stores Twitter content offline ensures the data reflects user intent and the current state of content on Twitter. For example, when someone on Twitter deletes a Tweet or their account, protects their Tweets, or scrubs the geoinformation from their Tweets, it is critical for both Twitter and our developers to honor that person’s expectations and intent. The batch compliance endpoints provide developers an easy tool to help maintain Twitter data in compliance with the Twitter Developer Agreement and Policy.

Optional - Delete the Google cloud project to avoid any overage costs