Dog and Color

HTTP Persistent Connections keep on opening new connections

There is an odd behavior with Java HttpUrlConnection in which excessive amount of connection is opened. I can see this by doing netstat on my Windows box.

The setup is like this:

ExecutorService es = Executors.newFixedThreadPool(16);

Callable c = new Callable() {
public void call() throws Exception {
... do something that uses HttpUrlConnection
}
}

for(int i=0; i < 160; i++) {
es.submit(c);
}

es.shutdown();

If you run this and run netstat,you will end up seeing a lot more than 16 connections with TIME_WAIT status. In fact I'd bet there are probably more than 50 connections in that status to your destination.

There are many reasons why this shouldn't happened.
1. Keep-Alive should have reused connections.
2. Eventually one of the connection will timeout. After all you do establish that many connections to the remote system, and it might not like that you are doing that.

This page has excellent example of how to make sure keep-alive is being used and connections do get reused.
http://java.sun.com/j2se/1.5.0/docs/guide/net/http-keepalive.html

I follow everything there including making sure that my streams are closed. No effect.

However, I do see an interesting change when the number of parallel threads are reduced to 4. With that few parallel threads, not more than 4 connections are shown by netstat.

It turns out that this has to do with the default number of connections for keep alive.
According to this article, the number if 5.
http://java.sun.com/j2se/1.5.0/docs/guide/net/http-keepalive.html

I'd guess that the implementation of HttpUrlConnection is such that a connection is bound to a thread. If we have 4 threads, then there will be 4 connection recycled (after all it is the same 4 threads using it). However, when the number of threads is 16, then there are 16 connections being opened which is more than the maxConnection of 5. At this point odd behavior starts to happen. Instead of reusing, it starts to keep on spawning new connection.

The workaround is to actually start the jvm with system properties to change the max connections.

When I repeat the same operation with -Dhttp.maxConnections=16, netstat would give me exactly 16 connections. Nice!

So next time, if you need to issue X http calls in parallel, start your JVM by giving it maxConnections=X.

Preparing Ubuntu with Developer's essential

Like compiler, make, etc.

sudo apt-get install linux-kernel-headers
sudo apt-get install build-essential

Installing Lighttpd (Ubuntu)

sudo apt-get install libpcre3-dev
sudo apt-get install zlib1g-dev
sudo apt-get install libglib2.0-dev
sudo apt-get install libbz2-dev

wget for the source (see lighttpd website)
unzip

./configure
make
make install

sed -e 's/FOO/lighttpd/g' doc/rc.lighttpd > /etc/init.d/lighttpd
chmod a+rx /etc/init.d/lighttpd
mkdir /etc/sysconfig
cp -p doc/sysconfig.lighttpd /etc/sysconfig/lighttpd
install -Dp ./doc/lighttpd.conf /etc/lighttpd/lighttpd.conf
update-rc.d lighttpd defaults

Access MySQL from Another Server (Ubuntu)

Apparently, my.cnf (/etc/mysql/my.cnf) shipped with
bind-address = 127.0.0.1

The reason being
# Instead of skip-networking the default is now to listen only on
# localhost which is more compatible and is not less secure.

Okay, but I need to access mysql from another server. Anyway, just change this to your IP and it should be good.

Don't forget to restart mysql
sudo /etc/init.d/mysql restart

source: http://dev.mysql.com/doc/refman/5.0/en/can-not-connect-to-server.html
third comment.

Slice Setup

Login as root.

apt-get update
apt-get dist-upgrade

adduser
# just in case sudo is not installed yet
apt-get install sudo
visudo
# add following line
ALL=(ALL) ALL

chmod 1777 /tmp

test login with your user
make sure your user can run sudo

sudo vi /etc/ssh/sshd_config
PermitRootLogin no

/etc/init.d/ssh restart

Login as your user

sudo apt-get install build-essential

# install Ruby
sudo apt-get install ruby1.8-dev ruby1.8 ri1.8 rdoc1.8 irb1.8 libreadline-ruby1.8 libruby1.8 libopenssl-ruby

sudo ln -s /usr/bin/ruby1.8 /usr/bin/ruby
sudo ln -s /usr/bin/ri1.8 /usr/bin/ri
sudo ln -s /usr/bin/rdoc1.8 /usr/bin/rdoc
sudo ln -s /usr/bin/irb1.8 /usr/bin/irb


# setting up mysql
sudo apt-get install mysql-server mysql-common mysql-client libmysqlclient15-dev libmysqlclient15off

sudo mysqladmin -u root password newrootsqlpassword

Here Comes the Boss-man

I was part of a lucky group of employees who got invited to sit down with one of the big boss in my company. I worked for a massively huge company making enterprise software.

The big boss looks like a regular Joe but he doesn't act like one. He's calm, steady, straight-talking and completely in control of the situation. I guess it helped that he was talking to people about 4 levels under him, but he did have a commanding presence.

The session was an hour of q&a. I was surprised that he just came in, sit down, introduced himself, ask for a round table introduction, and went straight to q&a. No morale rousting speeches, no corporate propaganda. Just answering questions.

One thing to note is that unlike all the managers I've ever had, this guy actually listen. He waited until someone completely finish a question before even starts answering (he only cut off those folks can't seem to control themselves). Then he actually answered the question.

For example.
Q: Why is it so darn difficult to get IT to help us to get anything done? Like giving us a replacement for our broken workstations??

Big-boss-man answer: Yeah, they sit on a padestal trying to streamline IT while not giving us the service we need. I am doing my best to change that, however I have to pick my battle. I am working with my boss to get to the situation where IT will be providing services again instead of going about their own pet-projects.

My first-line-manager answer: You have a broken workstation? Did you call IT?

Another favorite of all managers I have worked with is to pick one word out of 20 that you said and skewered you with it.

Me: I'm not interested in taking Oracle DB Administration classes. I would like to focus my learning on server-side programming and get better at those side of the application.

Any of my managers: They are not just classes. They are excellent on-hands training program that will help you in your career.

So few things I take from listening to the Big-Boss-Man:
1. Listen listen listen.
2. Don't bullshit. It barely works at the low level (with people like me), and it most definitely won't work with the big boys (his level).
3. Pick your fights carefully. Apparently when you run a business division which are competing with other division for resources, you need to make sure you put your effort where you got the best return. It such common sense, yet some of us wants to fight all the time for all kinds of worthless thing.
4. Big-boss-man often has to go around appeasing people for all the screw-ups that all kinds of people under him do.
5. It is a better idea to pay conservatively and keep your employees during down-turn rather than pay extravagantly and lay off people when situation go worse.
6. Count your money. Apparently with a budget of a few hundred millions, you still do need to count the money. Maybe even more carefully!
7. Develop stuff in house. It's cheaper than buying innovations from outside.

Big-boss-man is a good man.

Java 1.6 got HttpServer!

I am totally excited!

I always hated the idea that I need to put in a Tomcat or Resin server for my Java web app. They are excellent web servers by the way, but I need to get a few megabytes and a dozen jars just to do request/response handling? Cmon! I was looking at Simple for awhile, but then I found out that Sun's Java 1.6 has HttpServer built it.

These past few days, I've been writing some test codes around it. The bottom line is that it is fast, dead simple, but it doesn't offer much else. While this is perfect for my cache-to-SDB app, this is probably too bare bones for web app.

It is an intriguing project to build a Rails-like web app framework around this Http server. I don't even need a Java version of ActiveRecord at this point, since the future is not RDBMS (or is it?). A convention-driven httpserver -> controller -> template mechanism is all I need. Looks like Velocity is a great fit. Will have to investigate that possibility.

Management Star Trek Style

I love Star Trek, and I wish that I actually live in a starship. However, would I run a software development team like Kirk ran Enterprise?

1. A lot of orders come from the top. Even automatic ones like to raise shield when a major energy waves are coming, what choice of engine to use when leaving starbase dock.

2. Log everything. While we mostly hear the captain's log, almost every officers seems to have a log. First officer got one, chief engineer's got one, chief medical got one. I bet everybody can log anything to the computer. Even private logs.

3. A lot of face-to-face time. I'd think they would use intercoms and computers more to convey information. Turns out a lot of them walk around and ride the elevators just to bring message and have conversation.

4. Most captain's decision are made alone with very limited discussion with their chiefs.

5. Commands are readily transferable. Whenever the captains are not on the helm, first officer take over and knows what to do immediately. This goes down the chain. Every chief officers can delegate responsibility to next in line immediately.

What will Persistence Layer Looks Like

Data sharding is the way to go. We need a decent node that has the latest data. They layer must be decoupled from web app. Each node (or a manager) can inform requestor where to get the right data. Must support affinity transfer smoothly.

So it will be like this:
1. REST interface.
2. Cache data (like memcached just less awesome).
3. A node only need to handle configured silos (data that it handled was configured in).
4. Probably need a manager node (if a node is asked for data it doesn't have, pass it to manager who knows where to get it from).
5. Support seamless transition. Say if a node should no longer handle silo A, then it will cease taking in updates (redirecting to new node) and slowly phase out silo A cache.
6. Updates SimpleDB on update request. This way, I do not need to perform complicated async SimpleDB update. If SimpleDB is not available, update cannot be performed.

To make it easy, it should be strictly key-value system (like memcached). No query allowed.

Now the question is how to define the data object, or if it should be defined at all. It would suck to have to define the data object on web app AND this layer.

Persistence as a Service

Can this be applied with SimpleDB as the backend?
Looks like it can help to resolve the issues that I was thinking about.

http://natishalom.typepad.com/nati_shaloms_blog/2008/03/scaling-out-mys.html

Why use Amazon AWS?

While the use of SQS is too obvious to even be considered more than once, SimpleDB, S3 and even EC2 is highly contentious.

Instead of EC2, you can get a VPS from slicehost for reasonable sum (depending on how you compare). But if you use S3 or SimpleDB, EC2 is probably a must.

So then why S3. I find serving assets (images) from S3 to be too slow. You also cannot do CNAMEs tricks to maximize browser open connections. So think of S3 as a ultimate data storage, and backup data server, but never main data server. However, Dreamhost can store and serve your data at ridiculously cheap price. So even if you use S3, you still need to do something to actually serve your data.

SimpleDB is even more troublesome. It is not RDBMS, so you have limited query capability (no SQL!), no foreign keys, no transactions, no data constraints. And you have to deal with "eventual consistency".

Why use Amazon AWS at all then?

The short answer is that because it is there. I recently re-started a project that uses S3 as data storage. While I need to spent a few hours setting up my VPS, installing MySQL, installing RMagick and ImageMagicks, ruby, etc, etc (why something always go wrong?), the data upload to S3 just... works. When the image server piece wasn't running right, I just need to tell my code to start using S3 URLs and it... works. It's just so nice that it's always there, no need to back it up, no need to move it around, etc.

Using Amazon AWS is really come to down to the fact that you do not need to do admin work. Since I use S3, I do not need to worry about moving my data, backing it up and splitting it between nodes due to size constraints.

Similarly with SimpleDB, I'd foresee that I no longer need to muck around with MySQL settings and figuring out how to setup clusters, etc.

But, no transactions! "Eventual consistency"! How???

Well, all those things do have a code workaround. And guess what, I am better at programming than doing admin work. Not to mention I also like programming more than admin work! I'd rather spent a few weeks figuring out a slick way to deal with "eventual consistency" and end up with a supremely reliable data storage. The alternative is having to create cron jobs to backup MySQL, install MySQL whenever I need one... *shudder*

Of course there's also the whole scalability benefit, but with 500 visitor/day... I do not need to worry about that as yet.

SimpleDB Fetch Test in Java

While 25ms is not bad, it is also not impressive.
So either SimpleDB is slow, Ruby is slow, or my code is wrong.

I check and recheck my code, profile different portions of the code, but nothing yield.

David Kavanagh wrote in his article (near the bottom)
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1292

Even his number from his laptop to SimpleDB at 0.0537s/item at 10 threads is very expected (I got 0.04s/item from my desktop). However at 30 threads he is clocking 0.0390s/item while I still clock 0.04s/item.

So I wrote a test in Java using typica (http://code.google.com/p/typica/).

The number I got was even better than David's.

Threads	Java seconds/item	Ruby seconds/item
1	0.32	0.5
2	0.17	0.13
5	0.065	0.06
10	0.034	0.047
20	0.019	0.047
30	0.015	0.041
50	0.013	0.048

What impresses me is how linear the improvement is up to 20 threads. Luckily I seem to have a faster connection than David's (or just less latency).

I do find it odd that at lower thread count, Ruby is keeping up with Java, but at around 10 threads it starts losing. What gives? Possibly Ruby's green threads cannot keep up with Java native thread which can utilize my dual-core better.

I will have to try this Java test on EC2. David can got it down to less than 20ms. That's awesome. If that is the case, maybe I need to rethink my programming language preference, at least for this purpose.

Update (a few minutes later)

Got an Ubuntu AMI running and quickly drop in JDK 1.6 in there.

Okay the numbers

Threads	Java s/item	Ruby s/item
1	0.026	0.043
2	0.013	0.031
5	0.007	0.024
10	0.06	0.025
20	0.007	0.025
30	0.006	0.025
50	0.007	0.025

Now this is better! Even at single-thread it is performing at the same level as Ruby with 5 threads. And then it is linear improvement until around 5 threads, at which point it bottoms out at 6ms. I guess that's not much you can ask at that point.

SimpleDB Fetch Test

It is a simple test. Put 100 items into SimpleDB, each holding 4 attributes containing String of random content (no more than 10 characters).

Then we do a query on the domain to get the 100 item_ids, and then load their attributes one by one.

Of course it is all done in an EC2 instance to save me some money and faster network.

The goal is to see how many parallel threads do we need to achieve optimum load time.
For statistical purposes, we ran with 1, 2, 5, 7, 10, 12, 15, 17, 20 and 25 threads.
Each thread run was done 10 times and average. Then we ran this 5 times and average the averages for each thread.

Here's what we got.



Y-axis is time in seconds to fetch the attributes of 100 items.
X-axis is the number of parallel threads.

Noticed that single-thread is definitely a bad idea.
Two is better.
Five is even better, but anymore than that doesn't make it any faster.
At 20 threads it is even getting worse. Maybe ruby's thread management gets weird at this point?

It is possible that 100 items is too small (afterall at 20 threads, this means each thread is only loading 5 items). Not to mention that the test code includes thread spawning in the timing, which means that the cost of spawning 20 threads might no longer do justice to load only 5 items).

So let's try again with more items. This time 250 at once (the maximum number of items that can be queried from SimpleDB). Now, at 20 threads we are loading 12 per thread, slightly more than double the previous test.



Again, Y-axis is time in seconds to load attributes of 250 items.
X-axis is the number of thread.

The two lines are two different runs. This is even more curious because in one of the run, there is no noticeable improvement with more than 2 threads. But in general anything between 5 and 15 is good. Again there is a rise around 20 threads.

At this point I do not know if the limitation is on SimpleDB, ruby or networking.

The bottom line is that it takes 25 ms to fetch this data from SimpleDB. Each set of attributes (item) averages 32 bytes at 5 or more threads. At 1 thread, it takes 46ms, which is fairly acceptable to achieve sub-1-second web app response time.

Quick Setup Ruby on Amazon EC2 to test SimpleDB speed

Start ec2 instances (pick developer image).

Login as root.

Install zlib (rubygems need this)

cd /usr/local/src
wget http://www.zlib.net/zlib-1.2.3.tar.gz
tar -xvf zlib-1.2.3.tar.gz
cd zlib-1.2.3
./configure --prefix=/usr
make
make test
make install


Setup Rubygems

cd /usr/local/src
wget http://rubyforge.org/frs/download.php/29548/rubygems-1.0.1.tgz
tar -xvf rubygems-1.0.1.tgz
cd rubygems-1.0.1
ruby setup.rb


Get a few gems
cd /
gem install uuid
gem install rake
gem install needle
gem install aws-sdb

Diagramm!

I've been reading up on memcached and it is perfect for what I need.

Thinking of a setup like this:

|------------|     |-------------------|    |----------|
| memcached  |<--->| persistence layer |<-->| simpleDB |
|------------|     |-------------------|    |----------|
                   |my awesome web app |
                   |-------------------|

so instead of doing cache on my persistence layer, let memcached do it. Even better multiple webapp instances (which now will have persistence layer embedded) will be able to talk to the same set of memcached servers (instead of maintaining their own cache).

To keep things simple, only documents will be in memcached. query will go to simpleDB.

Persistence layer will do this:
1. given an ID
2. check memcached, if yes return document
3. get from simpleDB
4. put in memcached
5. return document.

Put Version in every SimpleDB entries

This way we can always do stale-ness check easily.

But then do we need to keep around the old stuff? Do we need to do real versioning?

If we do then this also means we'll then have two more "columns".

Business key and version. (Business Key??!! Ugh! Can't get away from Enterprise-software mentality). Why the need for business key? Well, the ID won't help because we'll have different ID per entries even if they are the same "thing". But business key is the same for different versions of the same "thing".

Question is what to do with old entries? Should they be moved to another space. Would it even save anything? Probably will confuse me more than help me.

On RDMBS, I would have another table or two for archives. But here?

Update 3/22/08 12:13pm

This wont' work. Keeping old version in the same space will make it really complicated to perform query (I always need to make sure I pick up the max version, etc). So either I do not do this, or move the item into another space.

Either way I do not need business key anymore.

Persistence Layer to SimpleDB

How would this layer cache?

Should it cache by query? The answer is no.
The grand idea is that we only store documents (thanks couchDB). Anything else it extraneous. So all we need is to cache by document ID (key).

So when asked for a document, given a key.
1. Check cache if the key is in the cache, if yes return the the document.
2. No? Get it from SimpleDB
3. Cache it and return the document.

When asked to run a query given a simpleDB query:
1. pass it to simpleDB.
2. return the data.
No caching!

How about invalidation?
Hmm.. this is a tough one.

ActiveRecord on SimpleDB

Using ActiveRecord against SimpleDB doesn't make sense. ActiveRecord was fitted for RDBMS systems where the columns are defined. In the case of SimpleDB, the app must know what data it wants to get and how.

Need to make a layer that talk to simpleDB.
It must be able to:
1. take ruby command and return data objects.
2. cache data (see earlier post).
3. take in a definition of an document type(?)
4. accessible from both internal (within the same ruby vm) and external (rack interface?).

Can it deduce the "content" of the data from the data it get from SimpleDB. While this is neat idea, most likely we'll have problem figuring out whether 00001234 is a number 1234 or it is a String.

Something like this can be made fairly simple. Probably borrow some idea from ActiveRecords and other persistence layer implementation.

SimpleDB and Eventual Concurrency

So we have a system that takes in user input, stores it in SimpleDB and then query SimpleDB to display it back to the user. Very easy. But guess what, when you query SimpleDB, that data you just put in is not there! Yikes!

That's actually expected because the nature of SimpleDB. Data is propagated between "nodes" and you do not know which node you will hit for save and which node for query. They call this "eventual concurrency". You can be assured that your data will eventually be propagated to all nodes but you do not know when and how fast.

This limitation make our simple use case impossible.

The easiest solution is to maintain our own aware-node. A node that we talk to for update and a node that we talk to for query. Whether this is a standalone app or built-in doesn't matter as long as it can have the latest value and the app (and other nodes) know its location and can talk to it.

Assuming that we have a distributed system. A cluster of nodes running the same app. When user made an entry, one of the app received it and update SimpleDB.

What's next?

How would user see latest value? Well, we can keep this user on the same node. Since this node is the node that is aware of the latest data, then we can see the latest data from here. Other nodes will hit SimpleDB to get data and that's okay. In all likelyhood, User B wouldn't mind if it takes a few minutes before seeing User A's update.

This requires sticky session. Any user session must go to the same server. That's probably not too hard to do. Designing an aware node is definitely harder to do. Basically it is a cache against SimpleDB and somehow it must know when to ask SimpleDB and when to rely on what's cached.

Now the next level of use case.

User enter data. We store it in SimpleDB. Alas, SimpleDB doesn't have much in terms of arithmetic operation, so we need a job that aggregate data. No problem can be done. We put an entry in SQS for the job to pick it up.

First problem. When the job picks it up and query SimpleDB, it might not see the data yet. So it will aggregate without the new data. Pointless.

To fix that we can stick information in SQS entry on where to look for latest data. Easy enough. We tell the job to look for latest data in the aware node.

What if two different users enter data one after another. Then we have two entries in SQS.

The problem is that we cannot be sure which entries are picked up first, and each entry is not aware of the other.

Let's say the last entry is picked up first by the job. Then the job will hit the aware node for that one latest data and get the rest from SimpleDB (maybe by asking the aware node or by itself, it doesn't matter). But if the second latest data has not been propagated, it will not know where to get it from.

One solution would be for the job to get all SQS entries in the queue. The sort them into buckets of similar data (say it knows that User A and User B are both doing update on the thing that has to be computed together). Then it takes the two SQS entries, pick up the latest data from respective aware nodes, and process data.

The drawback is that this one job will consume this entire queue and we cannot have another job on this same queue. This is okay as we can create unlimited queues.

Thread and Cores: Java vs. Ruby

Tested Pentium D. Java 1.5 can utilize both cores simply by creating another Thread. I assume 1.6 can do same too.

Ruby 1.8.6 on Windows does not. It only consume one core.

Looks like Ruby 2.0 uses native thread, but where is Ruby 2.0? (The next version is still 1.9).
http://en.wikipedia.org/wiki/Green_threads

Restart SSH

Usually this is done after disabling root or blocking access to users.

/etc/init.d/ssh restart

Figuring out Ubuntu version

cat /etc/lsb-release

cat /etc/issue