Getting started with the sampled stream endpoint

The endpoints we release in Labs will be previews and are likely to change before they are released broadly, so we encourage you to take that into consideration as you build. Before getting started, we encourage you to read more about Twitter Developer Labs.

The sampled stream endpoint allows developers to stream, in real-time, roughly a 1% sample of all public Tweets. 

Before you can start, you will need the following:

Authentication

This endpoint is authenticated using OAuth 2.0 Bearer token (also known as app-only authentication). This means you will need to generate a Bearer token, and pass this token in all of your requests. 

You can use the Run in Postman button to download a collection that will handle the generation of a Bearer token for you. Other clients like curl and Insomnia will require you to generate a token manually.

 

REST client

REST applications such as Postman can be used for organizing, testing, and debugging HTTP requests.

  • Python 3
  • Ruby
  • JavaScript (Node.js)

To run this example, you will need to add your consumer key and secret to this example. You will also have to enter one or several comma-seperated Tweet ID(s) from owned/authorized account(s) as indicated towards the bottom of this example. 

To add your consumer key and secret:

  1. Navigate to your app dashboard.
  2. Select the app you've enabled with the Filtered stream preview, then click Details.
  3. Select the Keys and tokens tab.
  4. In the Consumer API keys section, copy the values for API key into consumer_key and API secret key into consumer_secret.
     

Important: Never check consumer keys and secrets into source control. Learn how to secure your credentials.

import os
import requests
import json
from pprint import pprint
from requests.auth import AuthBase
from requests.auth import HTTPBasicAuth

consumer_key = "" # Add your API key here
consumer_secret = "" # Add your API secret key here

stream_url = "https://api.twitter.com/labs/1/tweets/stream/sample"

# Gets a bearer token
class BearerTokenAuth(AuthBase):
  def __init__(self, consumer_key, consumer_secret):
    self.bearer_token_url = "https://api.twitter.com/oauth2/token"
    self.consumer_key = consumer_key
    self.consumer_secret = consumer_secret
    self.bearer_token = self.get_bearer_token()

  def get_bearer_token(self):
    response = requests.post(
      self.bearer_token_url, 
      auth=(self.consumer_key, self.consumer_secret),
      data={'grant_type': 'client_credentials'},
      headers={"User-Agent": "TwitterDevSampledStreamQuickStartPython"})

    if response.status_code is not 200:
      raise Exception(f"Cannot get a Bearer token (HTTP %d): %s" % (response.status_code, response.text))

    body = response.json()
    return body['access_token']

  def __call__(self, r):
    r.headers['Authorization'] = f"Bearer %s" % self.bearer_token
    return r

def stream_connect(auth):
  response = requests.get(stream_url, auth=auth, headers={"User-Agent": "TwitterDevSampledStreamQuickStartPython"}, stream=True)
  for response_line in response.iter_lines():
    if response_line:
      pprint(json.loads(response_line))

bearer_token = BearerTokenAuth(consumer_key, consumer_secret)

# Listen to the stream. This reconnection logic will attempt to reconnect as soon as a disconnection is detected.
while True:
  stream_connect(bearer_token)

To run this example, you will need to add your consumer key and secret to this example. You will also have to enter one or several comma-seperated Tweet ID(s) from owned/authorized account(s) as indicated towards the bottom of this example. 

To add your consumer key and secret:

  1. Navigate to your app dashboard.
  2. Select the app you've enabled with the Filtered stream preview, then click Details.
  3. Select the Keys and tokens tab.
  4. In the Consumer API keys section, copy the values for API key into @consumer_key and API secret key into @consumer_secret.
     

Important: Never check consumer keys and secrets into source control. Learn how to secure your credentials.

require 'typhoeus'
require 'base64'
require 'json'

@consumer_key = "" # Add your API key here
@consumer_secret = "" # Add your API secret key here

@bearer_token_url = "https://api.twitter.com/oauth2/token"
@stream_url = "https://api.twitter.com/labs/1/tweets/stream/sample"

def bearer_token
  return @bearer_token unless @bearer_token.nil?

  @credentials = Base64.encode64("#{@consumer_key}:#{@consumer_secret}").gsub("\n", "")
  
  @options = {
    body: {
      grant_type: "client_credentials"
    },
    headers: {
      "Authorization": "Basic #{@credentials}",
      "User-Agent": "TwitterDevSampledStreamQuickStartRuby",
    },
  }

  @response = Typhoeus.post(@bearer_token_url, @options)
  @body = JSON.parse(@response.body)
  @bearer_token = @body["access_token"] ||= nil
end

def stream_connect
  @options = {
    timeout: 20,
    method: 'get',
    headers: {
      "User-Agent": "TwitterDevSampledStreamQuickStartRuby",
      "Authorization": "Bearer #{bearer_token}",
    },
    params: {
      format: 'compact',
    },
  }

  @request = Typhoeus::Request.new(@stream_url, @options)
  @request.on_body do |chunk|
    puts chunk
  end
  @request.run
end

# Listen to the stream.
# This reconnection logic will attempt to reconnect when a disconnection is detected.
# To avoid rate limites, this logic implements exponential backoff, so the wait time
# will increase if the client cannot reconnect to the stream.
timeout = 0
while true
  stream_connect
  sleep 2 ** timeout
  timeout += 1
end

To run this example, you will need to add your consumer key and secret to this example. You will also have to enter one or several comma-seperated Tweet ID(s) from owned/authorized account(s) as indicated towards the bottom of this example. 

To add your consumer key and secret:

  1. Navigate to your app dashboard.
  2. Select the app you've enabled with the Filtered stream preview, then click Details.
  3. Select the Keys and tokens tab.
  4. In the Consumer API keys section, copy the values for API key into consumer_key and API secret key into consumer_secret.
     

Important: Never check consumer keys and secrets into source control. Learn how to secure your credentials.

const https = require('https');
const request = require('request');
const util = require('util');

const get = util.promisify(request.get);
const post = util.promisify(request.post);

const consumer_key = ''; // Add your API key here
const consumer_secret = ''; // Add your API secret key here

const bearerTokenURL = new URL('https://api.twitter.com/oauth2/token');
const streamURL = new URL('https://api.twitter.com/labs/1/tweets/stream/sample');

async function bearerToken (auth) {
  const requestConfig = {
    url: bearerTokenURL,
    auth: {
      user: consumer_key,
      pass: consumer_secret,
    },
    form: {
      grant_type: 'client_credentials',
    },
    headers: {
      'User-Agent': 'TwitterDevSampledStreamQuickStartJS',
    },
  };

  const response = await post(requestConfig);
  return JSON.parse(response.body).access_token;
}

function streamConnect(token) {
  // Listen to the stream
  const config = {
    url: 'https://api.twitter.com/labs/1/tweets/stream/sample?format=compact',
    auth: {
      bearer: token,
    },
    headers: {
      'User-Agent': 'TwitterDevSampledStreamQuickStartJS',
    },
    timeout: 20000,
  };

  const stream = request.get(config);

  stream.on('data', data => {
    try {
      const json = JSON.parse(data);
      console.log(json);
    } catch (e) {
      // Keep alive signal received. Do nothing.
    }
  }).on('error', error => {
    if (error.code === 'ETIMEDOUT') {
      stream.emit('timeout');
    }
  });

  return stream;
}

(async () => {
  let token;

  try {
    // Exchange your credentials for a Bearer token
    token = await bearerToken({consumer_key, consumer_secret});
  } catch (e) {
    console.error(`Could not generate a Bearer token. Please check that your credentials are correct and that the Sampled Stream preview is enabled in your Labs dashboard. (${e})`);
    process.exit(-1);
  }

  const stream = streamConnect(token);
  stream.on('timeout', () => {
    // Reconnect on error
    console.warn('A connection error occurred. Reconnecting…');
    streamConnect(token);
  });
})();