Troubleshooting full buffer disconnect issues
You may get one of the following errors when your stream is not keeping up with the speed at which we are delivering data and your App isn't consuming the data from the stream fast enough:
This stream has been disconnected because your client was unable to keep up with us.
This stream has been disconnected for operational reasons.
We allow delivery to get behind for a period of time, and we have a temporary staging buffer amount for each stream on our side; but if you don't catch up, we initiate a disconnect to allow you to reconnect at the current point in time. Please note that this may lead to data loss (for data that is within the buffer at the time of the full buffer disconnect).
These can occur around large spikes in data. Generally, we recommend using a buffer process for consuming data quickly that is separate from the processing process.
If you are an enterprise customer using v1.1 endpoints, you can find out more about optimizing your App to prevent disconnects like this in our articles on connection and on consuming streaming data here and here.
There is a range of tools available for retrieving missed Tweets due to a disconnect, including the ones listed below. Note that the following tools are only available with v1.1 endpoints at enterprise level of access.
Redundant Connections - With multiple connections, consume the stream from multiple servers to prevent missed data when one is disconnected.
Replay - Recover data from within the last 5 days using a separate stream.
Backfill - Reconnect within 5 minutes and start from where you left off.
- Historical PowerTrack - Recover data from the entire Twitter archive.