HTTP Persistent Connections keep on opening new connections

There is an odd behavior with Java HttpUrlConnection in which excessive amount of connection is opened. I can see this by doing netstat on my Windows box.

The setup is like this:

ExecutorService es = Executors.newFixedThreadPool(16);

Callable c = new Callable() {
public void call() throws Exception {
... do something that uses HttpUrlConnection
}
}

for(int i=0; i < 160; i++) {
es.submit(c);
}

es.shutdown();

If you run this and run netstat,you will end up seeing a lot more than 16 connections with TIME_WAIT status. In fact I'd bet there are probably more than 50 connections in that status to your destination.

There are many reasons why this shouldn't happened.
1. Keep-Alive should have reused connections.
2. Eventually one of the connection will timeout. After all you do establish that many connections to the remote system, and it might not like that you are doing that.

This page has excellent example of how to make sure keep-alive is being used and connections do get reused.
http://java.sun.com/j2se/1.5.0/docs/guide/net/http-keepalive.html

I follow everything there including making sure that my streams are closed. No effect.

However, I do see an interesting change when the number of parallel threads are reduced to 4. With that few parallel threads, not more than 4 connections are shown by netstat.

It turns out that this has to do with the default number of connections for keep alive.
According to this article, the number if 5.
http://java.sun.com/j2se/1.5.0/docs/guide/net/http-keepalive.html

I'd guess that the implementation of HttpUrlConnection is such that a connection is bound to a thread. If we have 4 threads, then there will be 4 connection recycled (after all it is the same 4 threads using it). However, when the number of threads is 16, then there are 16 connections being opened which is more than the maxConnection of 5. At this point odd behavior starts to happen. Instead of reusing, it starts to keep on spawning new connection.

The workaround is to actually start the jvm with system properties to change the max connections.

When I repeat the same operation with -Dhttp.maxConnections=16, netstat would give me exactly 16 connections. Nice!

So next time, if you need to issue X http calls in parallel, start your JVM by giving it maxConnections=X.

Preparing Ubuntu with Developer's essential

Like compiler, make, etc.

sudo apt-get install linux-kernel-headers
sudo apt-get install build-essential