Building rules for filtered stream

The filtered stream endpoints deliver filtered Tweets to you in real-time that match on a set of rules that are applied to the stream. Rules are made up of operators that are used to match on a variety of Tweet attributes.

Multiple rules can be applied to a stream using the POST /tweets/search/stream/rules endpoint. Once you’ve added rules and connect to your stream using the GET /tweets/search/stream endpoint, only those Tweets that match your rules will be delivered in real-time through a persistent streaming connection. You do not need to disconnect from your stream to add or remove rules. 

To learn more about how to create high-quality rules, visit the following tutorial:
Building high-quality filters for getting Twitter data
 

Table of contents

• Building a rule
  • Rule limitations
  • Operator availability
  • Operator types: standalone and conjunction-required
  • Boolean operators and grouping
  • Order of operations
  • Punctuation, diacritics, and case sensitivity
  • Specificity and efficiency
  • Iteratively building a rule
• List of operators
  
   

Building a rule

Rule limitations

Your rules will be limited depending on which product track you are using. 

If you are using the Standard product track at the Basic level of access, you are able to add 25 concurrent rules to your stream, and each rule can be 512 characters long.

If you are using the Academic Research product track, you are able to add 1000 concurrent rules to your stream, and each rule can be 1024 characters long. 

Operator availability

While most operators are available to any developer, there are several that are reserved for those that have been approved for the Academic Research product track. We list which product tracks each operator is available to in the list of operators table using the following labels:

• All: Available when using any Project.
• Academic Research only: Available when using an Academic Research Project 
   

Operator types: standalone and conjunction-required

Standalone operators can be used alone or together with any other operators (including those that require conjunction).

For example, the following rule will work because it uses the #hashtag operator, which is standalone:

#twitterapiv2

Conjunction required operators cannot be used by themselves in a rule; they can only be used when at least one standalone operator is included in the rule. This is because using these operators alone would be far too general, and would match on an extremely high volume of Tweets.

For example, the following rules are not supported since they contain only conjunction required operators:

has:media
has:links OR is:retweet

If we add in a standalone operator, such as the phrase "twitter data", the rule would then work properly. 

"twitter data" has:mentions (has:media OR has:links)

Boolean operators and grouping

If you would like to string together multiple operators in a single rule, you have the following tools at your disposal:

Order of operations

When combining AND and OR functionality, the following order of operations will dictate how your rule is evaluated.

1. Operators connected by AND logic are combined first
2. Then, operators connected with OR logic are applied
   
   

For example:

• apple OR iphone ipad would be evaluated as apple OR (iphone ipad)
• ipad iphone OR android would be evaluated as (iphone ipad) OR android
  
  

To eliminate uncertainty and ensure that your rule is evaluated as intended, group terms together with parentheses where appropriate. 

For example:

• (apple OR iphone) ipad
• iphone (ipad OR android)
   

Punctuation, diacritics, and case sensitivity

If you specify a keyword or hashtag rule with character accents or diacritics, it will match Tweet text honoring the diacritics (hashtags or keywords). Rules with a keyword Diacrítica or hashtag #cumpleaños will match Diacrítica or #cumpleaños but not Diacritica or #cumpleanos without the tilde í or eñe.

Characters with accents or diacritics are treated the same as normal characters and are not treated as word boundaries. For example, a rule with the keyword cumpleaños would only match activities containing the word cumpleaños and would not match activities containing cumplea, cumplean, or os.

All operators are evaluated in a case-insensitive manner. For example, the rule cat will match all Tweets that include the following: cat, CAT, Cat.
 

Specificity and efficiency

When you start to build your rule, it is important to keep a few things in mind.

• Using broad, standalone operators for your rule such as a single keyword or #hashtag is generally not recommended since it will likely match on a massive volume of Tweets. Creating a more robust rule will result in a more specific set of matching Tweets, and will hopefully reduce the amount of noise in the payload that you will need to sift through to find valuable insights. 
  • For example, if your rule was just the keyword happy you will likely get anywhere from 200,000 - 300,000 Tweets per day.
  • Adding more conditional operators narrows your search results, for example (happy OR happiness) place_country:GB -birthday -is:retweet
• Writing efficient rules is also beneficial for staying within the characters rule length restriction. The character count includes the entire rule string including spaces and operators.
  • For example, the following rule is 59 characters long: (happy OR happiness) place_country:GB -birthday -is:retweet
    

Iteratively building a rule

Test your rule early and often

Getting a rule to return the "right" results the first time is rare. There is so much on Twitter that may or may not be obvious at first and the rule syntax described above may be hard to match to your desired search. As you build a rule, it is important for you to periodically test it out with the stream endpoint to see what data it returns. You can also test with one of the Search Tweet endpoints, assuming the operators that you are using are also available via that endpoint. 

For this section, we are going to start with the following rule and adjust it based on the results that we receive during our test: 

happy OR happiness

Use results to narrow the rule

As you test the rule, you should scan the returned Tweets to see if they include the data that you are expecting and hoping to receive. Starting with a broad rule and a superset of Tweet matches allows you to review the result and narrow the rule to filter out undesired results.  

When we tested the example rule, we noticed that we were getting Tweets in a variety of different languages. In this situation, we want to only receive Tweets that are in english, so we’re going to add the lang: operator:

(happy OR happiness) lang:en

The test delivered a number of Tweets wishing people a happy birthday, so we are going to add -birthday as a negated keyword operator. We also want to only receive original Tweets, so we’ve added the negated -is:retweet operator:

(happy OR happiness) lang:en -birthday -is:retweet

Adjust for inclusion where needed

If you notice that you are not receiving data that you expect and know that there are existing Tweets that should return, you may need to broaden your rule by removing operators that may be filtering out the desired data. 

For our example, we noticed that there were other Tweets in our personal timeline that expressed the emotion that we are looking for and weren’t included in the test results. To ensure we have greater coverage, we are going to add the keywords, excited and elated.

(happy OR happiness OR excited OR elated) lang:en -birthday -is:retweet

Adjust for popular trends/bursts over the time period

Trends come and go on Twitter quickly. Maintaining your rule should be an active process. If you plan to use a single rule for a while, we suggest that you periodically check in on the data that you are receiving to see if you need to make any adjustments.

In our example, we notice that we started to receive some Tweets that are wishing people a “happy holidays”. Since we don’t want these Tweets included in our results, we are going to add a negated -holidays keyword.

(happy OR happiness OR excited OR elated) lang:en -birthday -is:retweet -holidays