Consuming Twitter Stream updates with HTTPClient
31/05/2009A couple of weeks ago Twitter announced a new Stream API . Instead of having to poll for updates, Twitter is sending them your way on HTTP, in real-time.
The streams come into several flavors, some of which providing a sample of the “firehose” - the stream of all updates, some other taking a list of user id in argument and returning updated for those ids only.
One easy way to test the service as twitter suggest is to use curl on spritzer, a feed that will provide you with 400-500k updates everyday. Which is already quite enough to have a good sample of data to make some stats and other analytics.
%curl http://stream.twitter.com/spritzer.json -uYOURUSERNAME:YOURPASSWORD
Here is the sample code if you want to retrieve and process those updates from Java using HTTP Client 4 from Apache.
This code is not optimized at all and should not be used for production! You’ll also need to handle disconnection / reconnection as the feed can suddenly be d/c ( I got it running for more than 5 days without having to reconnect ).
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;
import org.apache.http.HttpResponse;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.ClientProtocolException;
import org.apache.http.client.ResponseHandler;
import org.apache.http.client.methods.HttpGet;
import org.apache.http.impl.client.DefaultHttpClient;
public class StreamConsumer {
public StreamConsumer() throws Exception {
DefaultHttpClient httpclient = new DefaultHttpClient();
httpclient.getCredentialsProvider().setCredentials(
new AuthScope("stream.twitter.com", 80),
new UsernamePasswordCredentials("username", "password"));
HttpGet httpget = new HttpGet("http://stream.twitter.com/spritzer.json");
ResponseHandler<String> responseHandler = new ResponseHandler<String>() {
public String handleResponse(HttpResponse resp)
throws ClientProtocolException, IOException {
BufferedReader r = new BufferedReader(new InputStreamReader(
resp.getEntity().getContent(), "US-ASCII"));
while (true)
System.out.println(r.readLine());
}
};
httpclient.execute(httpget, responseHandler);
}
public static void main(String[] args) {
try {
StreamConsumer consumer = new StreamConsumer();
} catch (Exception e) {
e.printStackTrace();
}
}
}
