Introduction 

Products utilizing premium operators deliver social data to you based on filtering rules you set up. Rules are made up of one or more ‘clauses’, where a clause is a keyword, exact phrase, or one of the many premium operators. Before beginning to build rules with premium operators, be sure to review the syntax described below, look through the list of available operators, and understand the restrictions around building rules. You should also be sure to understand the nuances of how rules are evaluated logically, in the ‘order of operations’ section.

Multiple clauses can be combined with both ‘and’ and ‘or’ logic. ‘And’ logic is specified with a space between clauses, and ‘or’ logic is specified with an upper-case OR. See below for more details…

Premium operator rule format:

  • Each rule can be up to 2,048 characters long with no limits on the number of positive clauses (things you want to match or filter on) and negative clauses (things you want to exclude and not match on).

Building rules with premium operators

Keyword match

Keyword matches are similar to queries in a search interface (e.g. Google). For example, the following premium operator rule would match activities with ‘happy’ in the text body.

    happy

ANDing terms with white space

Adding another keyword is the same as adding another requirement for finding matches. For example, this rule would only match activities where both ‘happy’ and ‘party’ were present in the text, in either order – having a space between terms operates as boolean AND logic. If you include an explicit AND in your rule, it will rejected by the rules endpoint.

    happy party

ORing terms with upper-case OR

Many situations actually call for boolean OR logic, however. This is easily accomplished as well. Note that the OR operator must be upper-case and a lower-case ‘or’ will be treated as a regular keyword.

    happy OR party

Negating terms

Still other scenarios might call for excluding results with certain keywords (a boolean NOT logic). For instance, activities with ‘happy’, but excluding any with ‘birthday’ in the text.

    happy -birthday

Grouping with parentheses

These types of logic can be combined using grouping with parentheses, and expanded to much more complex queries.

    (happy OR party) (holiday OR house) -(birthday OR democratic OR republican)

This is just the beginning though – while the above examples rely simply on tokenized matching for keywords, premium products also offer operators to perform different types of matching on the text.

Exact match

    "happy birthday"

Substring match

    contains:day

Proximity match

    "happy birthday"~3

Further, other operators allow you to filter on unique aspects of social data, besides just the text. For example:

The user who is posting a Tweet

    from:user

Geo-tagged Tweets within 10 miles of Pearl St. in Boulder, CO

    point_radius:[-105.27346517 40.01924738 10.0mi]

Putting it all together

These can be combined with text filters using the same types of logic described above.

    (happy OR party) (holiday OR house OR "new year's eve") point_radius:[-105.27346517 40.01924738 10.0mi] lang:en -(birthday OR democratic     OR republican)

 

Boolean Syntax

The examples in the previous section, utilized various types of boolean logic and grouping. See the table below for additional detail regarding the syntax and requirements for each.

Logic type premium operator syntax Description
AND social data Whitespace between two operators results in AND logic between them

Matches activities containing BOTH keywords ('social', 'data').

Do NOT use AND explicitly in your rule. Only use whitespace. An explicit AND will be treated like a regular keyword.
OR social OR data To OR together two operators, insert an all-caps OR, enclosed in whitespace between them

Matches activities with EITHER keyword ('social' OR 'data')

Note that if you combine OR and AND functionality in a single rule, you should understand the order of operations described here, and consider grouping operators together using parentheses as described below to ensure your rule behaves as expected. 

You must use upper-case 'OR' in your rule. Lower-case 'or' will be treated as a regular keyword.
NOT social -data
apple -(fruit OR orange)
apple -(android phone)
Insert a - character immediately in front of the operator or group of operators.

The example rule shown matches activities containing keyword 'social', but excludes those which contain the keyword 'data')

Negated ORs are not allowed where the rule would request "everything in the firehose except the negation." E.g., apple OR -ipad is invalid because it would match all activities except those mentioning 'ipad'.
Grouping (social OR data) -(gnop OR ping) Parentheses around multiple operators create a functional "group".

Groups can be connected to clauses in the same manner as an individual clause via whitespace (AND) or ORs, and can be negated. However, note that the same restriction described above regarding negation/OR combination also applies to groups. For example, the following are examples of invalid syntax using groups:
ipad OR -(iphone OR ipod)
ipad OR (-iphone OR ipod)

Grouping is especially important where a single rule combines AND and OR functionality, due to the order of operations used to evaluate the rule. See below for more details.

Note that operators may be either positive or negative.

Positive Operators define what you want to include in the results. E.g. the ‘has:hashtags’ operator says “I want activities containing hashtags.”

Negative Operators define what you want to exclude from the results, and are created by using the Boolean NOT logic described above. E.g. ‘-has:hashtags’ says “Exclude any activities containing hashtagss, even if they otherwise match my rule.”

Premum operator products have no restriction on the number of positive and negative clauses, subject to a maximum length of 2,048 characters.

Order of Operations

When combining AND and OR functionality in a single rule, the following order of operations will dictate how your rule is evaluated.

  1. Operators connected by AND logic are combined first
  2. Then, operators connected with OR logic are applied

Example:

  • apple OR iphone ipad would be evaluated as apple OR (iphone ipad)
  • ipad iphone OR android would be evaluated as (iphone ipad) OR android

To eliminate uncertainty and ensure that your rules are evaluated as intended, group terms together with parentheses where appropriate. For example:

  • (apple OR iphone) ipad
  • iphone (ipad OR android)

Punctuation, Diacritics, and Case Sensitivity

If you specify a keyword or hashtag rule with character accents or diacritics for premium operators, it will match Tweet text honoring the diacritics (hashtags or keywords). Rule with a keyword Diacrítica or hashtag #cumpleaños will match Diacrítica or #cumpleaños but not Diacritica or #cumpleanos without the tilde í or eñe.

Characters with accents or diacritics are treated the same as normal characters and are not treated as word boundaries. For example, a rule of cumpleaños would only match activities containing the word cumpleaños and would not match activities containing cumplea, cumplean, or os.

All operators are evaluated in a case-insensitive manner. For example, the rule Cat will match all of the following: cat, CAT, Cat.

Rule Tags

As described here, each rule may be created with a tag. These tags have no effect on filtering, but can be used to create logical groupings of rules within your app. Each rule may have only one tag, with a maximum of 255 characters. Tags are included with the JSON formatted rule at the time of creation via the API, as described in our documentation.

Putting Rules in JSON Format

In order to add or delete a rule from a stream via the API, the rules must utilize JSON format. Essentially, this requires putting each rule into the following structure:

{"value":"insert_rule_here"}

Rules with Double-quotes

If the ‘rule’ contains double-quote characters (“) associated with exact-match or other operators, they must be escaped using a backslash to distinguish them from the structure of the JSON format. For example, if your rule is:

"social data" @gnip

The JSON formatted rule would be:

{"value":"\"social data\" @gnip"}

Rules with Double-quote String Literals

To include a double-quote character as a string literal within an exact-match, it must be double-escaped. For example, for a rule matching on the exact phrase ‘Toys “R” Us’, including the double-quotes around R, the plain-text representation of this would look like the following:

"Toys \"R\" Us"

Translating this to JSON format, you should use the following structure:

{"value":"\"Toys \\\"R\\\" Us\""}

Rules with Tags

To include an optional Tag with your rule, as described above, simply include an additional “tag” field with the rule value:

{"value":"\"social data\" @gnip","tag":"RULE-TAG-01"}

Formatting for API Requests

When adding or deleting rules from the stream via the API, multiple JSON formatted rules should be comma delimited, and wrapped in a JSON “rules” array, as shown below:

{"rules":[{"value":"from:gnip"},{"value":"\social data\" @gnip","tag":"RULE-TAG-01"}]}

Operators that Match Quote Tweets

In terms of filtering, the operators below will match on content from both the original quoted Tweet and the new “comment” Tweet.

  • Keywords
  • Phrases
  • Proximity
  • #hashtags
  • @mentions
  • $cashtags
  • url:
  • url_contains:
  • has:links
  • has:mentions
  • has:hashtags
  • has:media
  • has:symbols
  • is:quote