Interested in exploring Labs?
The endpoints we release in Labs are previews of tools that may be released more broadly in the future, but will likely undergo changes before then. We encourage you to take that into consideration as you explore. Before getting started, please read more about Twitter Developer Labs.
Frequently asked questions
How many Tweets can I get from the Labs streaming endpoint?
- You can stream up to 500,000 Tweets per month (months are calculated starting from the day you activate this preview).
- You can stream up to 500,000 Tweets per day.
- You will receive no more than 50 Tweets per second.
How many times can I connect, and how many simultaneous connections can I make?
Each app is allowed one connected client at a time. You can attempt reconnecting up to 50 times per 15 minute period. You can make up to 450 rule changes per 15 minute period.
How can I check my usage and rate limits?
To check connection limits response will return three headers: x-rate-limit-limit, x-rate-limit-remaining and x-rate-limit-reset headers. This is useful to understand how many times you can use the rule endpoint, and how many reconnections attempts are allowed for the streaming endpoint.
- x-rate-limit-limit indicates the number of allotted requests your client is allowed to make during the 15-minute window.
- x-rate-limit-remaining indicates the number of requests made so far in the 15-minute window.
- x-rate-limit-reset is a UNIX timestamp indicating when the 15-minute window will restart, resetting x-rate-limit-remaining to 0.
The streaming endpoint does not currently report usage data. To check how many Tweets have been delivered, your code can implement a metering logic, so that consumption can be measured and paused if needed.
How do I consume a stream?
Realtime streams are similar to any other REST API request, except they do not close the connection when done, or when no response is available.
Given the potential of high volumes of Tweets delivered in a stream, it is highly recommended that incoming content is processed in an asynchronous fashion. Your code that hosts the client side of the stream simply inserts incoming Tweets into a first in, first out (FIFO) queue, or a similar memory structure; a separate process/thread should consume Tweets from that queue to parse and prepare content for storage. With this design, you can implement a service that can scale efficiently in case incoming Tweet volumes changes dramatically.
I am having problems authenticating. How can I avoid authorization errors?
All the filtered stream endpoints are authenticated using OAuth 2.0 Bearer token (also known as application only authentication). Make sure your OAuth token is correct and valid. In order to use this preview, you need to activate from the Labs dashboard. Make sure “Filtered Stream” is listed under “Your active previews”.
My stream disconnected. Why did it happen, and how can I avoid disconnections?
Your stream can disconnect for a number of reasons. Inspect the error message returned by the stream to understand the reason for the failure. Common reasons for disconnections are:
- An authentication error (such as a wrong token or a wrong authentication method being used).
- A temporary server issue, scheduled maintenance and updates.
- Your client is not keeping up with the volume of Tweets the stream is delivering.
- Your account exceeded your monthly quota of Tweets.
I’m receiving an “HTTP 429 Too Many Requests” error, but I have not exceeded my rate limit yet. What does it mean and how do I avoid getting this error?
If you’re receiving this error on your stream, it may mean one of the following:
- You tried to establish more connections than allowed during the 15-minute rate limit window.
Solution: wait until the rate limit window resets. You can inspect the x-rate-limit reset to understand how much time is left in the rate limit window.
- You are already connected through another client, and the stream will not accept any additional connections.
Solution: disconnect other clients, and ensure you are connecting with only one client at the time
- The stream did not disconnect in a clean way, which means it may still look connected to Twitter. This can happen when your connection experiences packet loss, or when your service can’t handle a sudden data burst. When this happens, you may notice the client resetting a TCP/IP connection.
Solution: these disconnections should timeout automatically. If this error persists, switch network connections if possible, or contact your ISP.
- You are updating rules concurrently, or multiple requests to the rules endpoints are being made in parallel.
Solution: call the rules endpoint from a single client, and send all your rules in a single request.
I’m trying to reconnect from a single client but I’m still getting an “HTTP 429 Too Many Requests” error. How do I connect?
If your stream is receiving a low volume of Tweets, it may take up to 20 seconds to detect a disconnection. This can happen for example when your client hasn’t received new Tweets, and it disconnects before attempting an immediate reconnection.
Why isn't my rule appearing on the stream right away?
Most rule additions take effect in about 20 seconds or less. immediately. It’s unlikely, but depending on external factors (for example, network connectivity), it may take longer before you start receiving matching Tweets. If you can’t find one a rule in the list rule endpoint, make sure that the rule creation request succeeded; this can be done by checking your logs for any error messages.
What if some Tweets are missing? I was expecting them to be returned by the stream, but they weren't.
Most of the time, missing Tweets indicate that your filters are more restrictive than intended. We suggest to check your rules:
- Make sure you are using the correct operators, as some operators are more restrictive than others. For example, -has:media filters out all media; sometimes, an exact phrase match can filter out more Tweets than intended.
- If you’re grouping operators, check that the grouping is logically correct, and that operators do not invalidate the entire rule. You can test this by breaking down those groups into separate rules, and check if a Tweet matches that subset.
Your connection and rules also determine why some Tweets might not have been delivered:
- Because we deliver Tweets in real-time, you must connect to the stream before you can receive current Tweets. The streaming endpoint does not deliver Tweets retroactively.
- Your rule must be already in place. The streaming endpoint does not apply rules retroactively. Also remember that rules may take up to 20 seconds to match Tweets.
- The stream does not deliver Tweets that are deleted very shortly after they are created. If a Tweet is deleted within a few seconds, it will not be streamed.
- The stream leverages some of our content filtering technology where we exclude high-confidence noisy content.
Lastly, you can check the following:
- The missing Tweet needs to originate from a public account. The streaming endpoint can only return Tweets from accounts that were public before the time of sending, and must remain public by the time it is delivered to your stream.
- You may have received more than 50 Tweets in a one second interval. In this case, no further Tweets will be sent.
What if I get disconnected from the stream? How can I collect any Tweets that was missed while disconnected?
When streaming Tweets, the goal is to stay connected for as long as possible, recognizing that disconnects may occur. In Labs, the streaming endpoint does not include a way to recover Tweets that were missed while disconnected. Instead, the endpoint provides a 20-second keep alive heartbeat (it will look like a new line character). Use this signal to detect if you’re being disconnected.
- Your code should detect when fresh content and the heartbeat stop arriving.
- If that happens, your code should trigger a reconnection logic. Some clients and languages allow you to specify a read timeout, which you can set to 20 seconds.
- Your service should detect these disconnections and reconnect as soon as possible.
The code samples on the "Quick start" page provide an example of this reconnect logic.