Interested in exploring Labs?
The endpoints we release in Labs are previews of tools that may be released more broadly in the future, but will likely undergo changes before then. We encourage you to take that into consideration as you explore. Before getting started, please read more about Twitter Developer Labs.

Getting started with the sampled stream endpoint

The sampled stream endpoint allows developers to stream, in real-time, roughly a 1% sample of all public Tweets. 

Before you can start, you will need the following:

Authentication

This endpoint is authenticated using OAuth 2.0 Bearer token (also known as app-only authentication). This means you will need to generate a Bearer token, and pass this token in all of your requests. 

You can use the Run in Postman button to download a collection that will handle the generation of a Bearer token for you. Other clients like curl and Insomnia will require you to generate a token manually.

 

REST client

REST applications such as Postman can be used for organizing, testing, and debugging HTTP requests.

  • Python 3
  • Ruby
  • JavaScript (Node.js)
  • Java

To run this example, you will need to add your consumer key and secret to this example.

To add your consumer key and secret:

  1. Navigate to your app dashboard.
  2. Select the app you've enabled with the Filtered stream preview, then click Details.
  3. Select the Keys and tokens tab.
  4. In the Consumer API keys section, copy the values for API key into consumer_key and API secret key into consumer_secret.
     

Important: Never check consumer keys and secrets into source control. Learn how to secure your credentials.

import os
import requests
import json
from pprint import pprint
from requests.auth import AuthBase
from requests.auth import HTTPBasicAuth

consumer_key = "" # Add your API key here
consumer_secret = "" # Add your API secret key here

stream_url = "https://api.twitter.com/labs/1/tweets/stream/sample"

# Gets a bearer token
class BearerTokenAuth(AuthBase):
  def __init__(self, consumer_key, consumer_secret):
    self.bearer_token_url = "https://api.twitter.com/oauth2/token"
    self.consumer_key = consumer_key
    self.consumer_secret = consumer_secret
    self.bearer_token = self.get_bearer_token()

  def get_bearer_token(self):
    response = requests.post(
      self.bearer_token_url, 
      auth=(self.consumer_key, self.consumer_secret),
      data={'grant_type': 'client_credentials'},
      headers={"User-Agent": "TwitterDevSampledStreamQuickStartPython"})

    if response.status_code is not 200:
      raise Exception(f"Cannot get a Bearer token (HTTP %d): %s" % (response.status_code, response.text))

    body = response.json()
    return body['access_token']

  def __call__(self, r):
    r.headers['Authorization'] = f"Bearer %s" % self.bearer_token
    return r

def stream_connect(auth):
  response = requests.get(stream_url, auth=auth, headers={"User-Agent": "TwitterDevSampledStreamQuickStartPython"}, stream=True)
  for response_line in response.iter_lines():
    if response_line:
      pprint(json.loads(response_line))

bearer_token = BearerTokenAuth(consumer_key, consumer_secret)

# Listen to the stream. This reconnection logic will attempt to reconnect as soon as a disconnection is detected.
while True:
  stream_connect(bearer_token)

To run this example, you will need to add your consumer key and secret to this example.

To add your consumer key and secret:

  1. Navigate to your app dashboard.
  2. Select the app you've enabled with the Filtered stream preview, then click Details.
  3. Select the Keys and tokens tab.
  4. In the Consumer API keys section, copy the values for API key into @consumer_key and API secret key into @consumer_secret.
     

Important: Never check consumer keys and secrets into source control. Learn how to secure your credentials.

require 'typhoeus'
require 'base64'
require 'json'

@consumer_key = "" # Add your API key here
@consumer_secret = "" # Add your API secret key here

@bearer_token_url = "https://api.twitter.com/oauth2/token"
@stream_url = "https://api.twitter.com/labs/1/tweets/stream/sample"

def bearer_token
  return @bearer_token unless @bearer_token.nil?

  @credentials = Base64.encode64("#{@consumer_key}:#{@consumer_secret}").gsub("\n", "")
  
  @options = {
    body: {
      grant_type: "client_credentials"
    },
    headers: {
      "Authorization": "Basic #{@credentials}",
      "User-Agent": "TwitterDevSampledStreamQuickStartRuby",
    },
  }

  @response = Typhoeus.post(@bearer_token_url, @options)
  @body = JSON.parse(@response.body)
  @bearer_token = @body["access_token"] ||= nil
end

def stream_connect
  @options = {
    timeout: 20,
    method: 'get',
    headers: {
      "User-Agent": "TwitterDevSampledStreamQuickStartRuby",
      "Authorization": "Bearer #{bearer_token}",
    },
    params: {
      format: 'compact',
    },
  }

  @request = Typhoeus::Request.new(@stream_url, @options)
  @request.on_body do |chunk|
    puts chunk
  end
  @request.run
end

# Listen to the stream.
# This reconnection logic will attempt to reconnect when a disconnection is detected.
# To avoid rate limites, this logic implements exponential backoff, so the wait time
# will increase if the client cannot reconnect to the stream.
timeout = 0
while true
  stream_connect
  sleep 2 ** timeout
  timeout += 1
end

To run this example, you will need to add your consumer key and secret to this example.

To add your consumer key and secret:

  1. Navigate to your app dashboard.
  2. Select the app you've enabled with the Filtered stream preview, then click Details.
  3. Select the Keys and tokens tab.
  4. In the Consumer API keys section, copy the values for API key into consumer_key and API secret key into consumer_secret.
     

Important: Never check consumer keys and secrets into source control. Learn how to secure your credentials.

const https = require('https');
const request = require('request');
const util = require('util');

const get = util.promisify(request.get);
const post = util.promisify(request.post);

const consumer_key = ''; // Add your API key here
const consumer_secret = ''; // Add your API secret key here

const bearerTokenURL = new URL('https://api.twitter.com/oauth2/token');
const streamURL = new URL('https://api.twitter.com/labs/1/tweets/stream/sample');

async function bearerToken (auth) {
  const requestConfig = {
    url: bearerTokenURL,
    auth: {
      user: consumer_key,
      pass: consumer_secret,
    },
    form: {
      grant_type: 'client_credentials',
    },
    headers: {
      'User-Agent': 'TwitterDevSampledStreamQuickStartJS',
    },
  };

  const response = await post(requestConfig);
  return JSON.parse(response.body).access_token;
}

function streamConnect(token) {
  // Listen to the stream
  const config = {
    url: 'https://api.twitter.com/labs/1/tweets/stream/sample?format=compact',
    auth: {
      bearer: token,
    },
    headers: {
      'User-Agent': 'TwitterDevSampledStreamQuickStartJS',
    },
    timeout: 20000,
  };

  const stream = request.get(config);

  stream.on('data', data => {
    try {
      const json = JSON.parse(data);
      console.log(json);
    } catch (e) {
      // Keep alive signal received. Do nothing.
    }
  }).on('error', error => {
    if (error.code === 'ETIMEDOUT') {
      stream.emit('timeout');
    }
  });

  return stream;
}

(async () => {
  let token;

  try {
    // Exchange your credentials for a Bearer token
    token = await bearerToken({consumer_key, consumer_secret});
  } catch (e) {
    console.error(`Could not generate a Bearer token. Please check that your credentials are correct and that the Sampled Stream preview is enabled in your Labs dashboard. (${e})`);
    process.exit(-1);
  }

  const stream = streamConnect(token);
  stream.on('timeout', () => {
    // Reconnect on error
    console.warn('A connection error occurred. Reconnecting…');
    streamConnect(token);
  });
})();

To run this example, you will need to add your consumer key and secret to this example.

To add your consumer key and secret:

  1. Navigate to your app dashboard.
  2. Select the app you've enabled with the Filtered stream preview, then click Details.
  3. Select the Keys and tokens tab.
  4. In the Consumer API keys section, copy the values for API key into CONSUMER_KEY and API secret key into CONSUMER_SECRET.
     

Important: Never check consumer keys and secrets into source control. Learn how to secure your credentials.

/*
 * Sample code to demonstrate the use of the Labs Sample Stream endpoint
 * */
public class SampleStreamDemo {

  //FIXME Replace the keys below with your own keys and secret
  private static final String CONSUMER_KEY = "";
  private static final String CONSUMER_SECRET = "";

  public static void main(String args[]) throws IOException, URISyntaxException {
    String accessToken = getAccessToken();
    connectStream(accessToken);
  }

  /*
   * This method calls the sample stream endpoint and streams Tweets from it
   * */
  private static void connectStream(String accessToken) throws IOException, URISyntaxException {

    HttpClient httpClient = HttpClients.custom()
        .setDefaultRequestConfig(RequestConfig.custom()
            .setCookieSpec(CookieSpecs.STANDARD).build())
        .build();

    URIBuilder uriBuilder = new URIBuilder("https://api.twitter.com/labs/1/tweets/stream/sample");

    HttpGet httpGet = new HttpGet(uriBuilder.build());
    httpGet.setHeader("Authorization", String.format("Bearer %s", accessToken));

    HttpResponse response = httpClient.execute(httpGet);
    HttpEntity entity = response.getEntity();
    if (null != entity) {
      BufferedReader reader = new BufferedReader(new InputStreamReader((entity.getContent())));
      String line = reader.readLine();
      while (line != null) {
        System.out.println(line);
        line = reader.readLine();
      }
    }

  }

  /*
   * Helper method that generates bearer token by calling the /oauth2/token endpoint
   * */
  private static String getAccessToken() throws IOException, URISyntaxException {
    String accessToken = null;

    HttpClient httpClient = HttpClients.custom()
        .setDefaultRequestConfig(RequestConfig.custom()
            .setCookieSpec(CookieSpecs.STANDARD).build())
        .build();

    URIBuilder uriBuilder = new URIBuilder("https://api.twitter.com/oauth2/token");
    ArrayList<NameValuePair> postParameters;
    postParameters = new ArrayList<>();
    postParameters.add(new BasicNameValuePair("grant_type", "client_credentials"));
    uriBuilder.addParameters(postParameters);

    HttpPost httpPost = new HttpPost(uriBuilder.build());
    httpPost.setHeader("Authorization", String.format("Basic %s", getBase64EncodedString()));
    httpPost.setHeader("Content-Type", "application/json");

    HttpResponse response = httpClient.execute(httpPost);
    HttpEntity entity = response.getEntity();

    if (null != entity) {
      try (InputStream inputStream = entity.getContent()) {
        ObjectMapper mapper = new ObjectMapper();
        Map<String, Object> jsonMap = mapper.readValue(inputStream, Map.class);
        accessToken = jsonMap.get("access_token").toString();
      }
    }
    return accessToken;
  }

  /*
   * Helper method that generates the Base64 encoded string to be used to obtain bearer token
   *
   * */
  private static String getBase64EncodedString() {
    String s = String.format("%s:%s", API_KEY, API_SECRET);
    return Base64.getEncoder().encodeToString(s.getBytes(StandardCharsets.UTF_8));
  }
}

 


 

Additional resources