Why use Amazon AWS?

While the use of SQS is too obvious to even be considered more than once, SimpleDB, S3 and even EC2 is highly contentious.

Instead of EC2, you can get a VPS from slicehost for reasonable sum (depending on how you compare). But if you use S3 or SimpleDB, EC2 is probably a must.

So then why S3. I find serving assets (images) from S3 to be too slow. You also cannot do CNAMEs tricks to maximize browser open connections. So think of S3 as a ultimate data storage, and backup data server, but never main data server. However, Dreamhost can store and serve your data at ridiculously cheap price. So even if you use S3, you still need to do something to actually serve your data.

SimpleDB is even more troublesome. It is not RDBMS, so you have limited query capability (no SQL!), no foreign keys, no transactions, no data constraints. And you have to deal with "eventual consistency".

Why use Amazon AWS at all then?

The short answer is that because it is there. I recently re-started a project that uses S3 as data storage. While I need to spent a few hours setting up my VPS, installing MySQL, installing RMagick and ImageMagicks, ruby, etc, etc (why something always go wrong?), the data upload to S3 just... works. When the image server piece wasn't running right, I just need to tell my code to start using S3 URLs and it... works. It's just so nice that it's always there, no need to back it up, no need to move it around, etc.

Using Amazon AWS is really come to down to the fact that you do not need to do admin work. Since I use S3, I do not need to worry about moving my data, backing it up and splitting it between nodes due to size constraints.

Similarly with SimpleDB, I'd foresee that I no longer need to muck around with MySQL settings and figuring out how to setup clusters, etc.

But, no transactions! "Eventual consistency"! How???

Well, all those things do have a code workaround. And guess what, I am better at programming than doing admin work. Not to mention I also like programming more than admin work! I'd rather spent a few weeks figuring out a slick way to deal with "eventual consistency" and end up with a supremely reliable data storage. The alternative is having to create cron jobs to backup MySQL, install MySQL whenever I need one... *shudder*

Of course there's also the whole scalability benefit, but with 500 visitor/day... I do not need to worry about that as yet.

SimpleDB Fetch Test in Java

While 25ms is not bad, it is also not impressive.
So either SimpleDB is slow, Ruby is slow, or my code is wrong.

I check and recheck my code, profile different portions of the code, but nothing yield.

David Kavanagh wrote in his article (near the bottom)
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=1292

Even his number from his laptop to SimpleDB at 0.0537s/item at 10 threads is very expected (I got 0.04s/item from my desktop). However at 30 threads he is clocking 0.0390s/item while I still clock 0.04s/item.

So I wrote a test in Java using typica (http://code.google.com/p/typica/).

The number I got was even better than David's.

Threads	Java seconds/item	Ruby seconds/item
1	0.32	0.5
2	0.17	0.13
5	0.065	0.06
10	0.034	0.047
20	0.019	0.047
30	0.015	0.041
50	0.013	0.048

What impresses me is how linear the improvement is up to 20 threads. Luckily I seem to have a faster connection than David's (or just less latency).

I do find it odd that at lower thread count, Ruby is keeping up with Java, but at around 10 threads it starts losing. What gives? Possibly Ruby's green threads cannot keep up with Java native thread which can utilize my dual-core better.

I will have to try this Java test on EC2. David can got it down to less than 20ms. That's awesome. If that is the case, maybe I need to rethink my programming language preference, at least for this purpose.

Update (a few minutes later)

Got an Ubuntu AMI running and quickly drop in JDK 1.6 in there.

Okay the numbers

Threads	Java s/item	Ruby s/item
1	0.026	0.043
2	0.013	0.031
5	0.007	0.024
10	0.06	0.025
20	0.007	0.025
30	0.006	0.025
50	0.007	0.025

Now this is better! Even at single-thread it is performing at the same level as Ruby with 5 threads. And then it is linear improvement until around 5 threads, at which point it bottoms out at 6ms. I guess that's not much you can ask at that point.

SimpleDB Fetch Test

It is a simple test. Put 100 items into SimpleDB, each holding 4 attributes containing String of random content (no more than 10 characters).

Then we do a query on the domain to get the 100 item_ids, and then load their attributes one by one.

Of course it is all done in an EC2 instance to save me some money and faster network.

The goal is to see how many parallel threads do we need to achieve optimum load time.
For statistical purposes, we ran with 1, 2, 5, 7, 10, 12, 15, 17, 20 and 25 threads.
Each thread run was done 10 times and average. Then we ran this 5 times and average the averages for each thread.

Here's what we got.



Y-axis is time in seconds to fetch the attributes of 100 items.
X-axis is the number of parallel threads.

Noticed that single-thread is definitely a bad idea.
Two is better.
Five is even better, but anymore than that doesn't make it any faster.
At 20 threads it is even getting worse. Maybe ruby's thread management gets weird at this point?

It is possible that 100 items is too small (afterall at 20 threads, this means each thread is only loading 5 items). Not to mention that the test code includes thread spawning in the timing, which means that the cost of spawning 20 threads might no longer do justice to load only 5 items).

So let's try again with more items. This time 250 at once (the maximum number of items that can be queried from SimpleDB). Now, at 20 threads we are loading 12 per thread, slightly more than double the previous test.



Again, Y-axis is time in seconds to load attributes of 250 items.
X-axis is the number of thread.

The two lines are two different runs. This is even more curious because in one of the run, there is no noticeable improvement with more than 2 threads. But in general anything between 5 and 15 is good. Again there is a rise around 20 threads.

At this point I do not know if the limitation is on SimpleDB, ruby or networking.

The bottom line is that it takes 25 ms to fetch this data from SimpleDB. Each set of attributes (item) averages 32 bytes at 5 or more threads. At 1 thread, it takes 46ms, which is fairly acceptable to achieve sub-1-second web app response time.

Quick Setup Ruby on Amazon EC2 to test SimpleDB speed

Start ec2 instances (pick developer image).

Login as root.

Install zlib (rubygems need this)

cd /usr/local/src
wget http://www.zlib.net/zlib-1.2.3.tar.gz
tar -xvf zlib-1.2.3.tar.gz
cd zlib-1.2.3
./configure --prefix=/usr
make
make test
make install


Setup Rubygems

cd /usr/local/src
wget http://rubyforge.org/frs/download.php/29548/rubygems-1.0.1.tgz
tar -xvf rubygems-1.0.1.tgz
cd rubygems-1.0.1
ruby setup.rb


Get a few gems
cd /
gem install uuid
gem install rake
gem install needle
gem install aws-sdb

Diagramm!

I've been reading up on memcached and it is perfect for what I need.

Thinking of a setup like this:

|------------|     |-------------------|    |----------|
| memcached  |<--->| persistence layer |<-->| simpleDB |
|------------|     |-------------------|    |----------|
                   |my awesome web app |
                   |-------------------|

so instead of doing cache on my persistence layer, let memcached do it. Even better multiple webapp instances (which now will have persistence layer embedded) will be able to talk to the same set of memcached servers (instead of maintaining their own cache).

To keep things simple, only documents will be in memcached. query will go to simpleDB.

Persistence layer will do this:
1. given an ID
2. check memcached, if yes return document
3. get from simpleDB
4. put in memcached
5. return document.

Put Version in every SimpleDB entries

This way we can always do stale-ness check easily.

But then do we need to keep around the old stuff? Do we need to do real versioning?

If we do then this also means we'll then have two more "columns".

Business key and version. (Business Key??!! Ugh! Can't get away from Enterprise-software mentality). Why the need for business key? Well, the ID won't help because we'll have different ID per entries even if they are the same "thing". But business key is the same for different versions of the same "thing".

Question is what to do with old entries? Should they be moved to another space. Would it even save anything? Probably will confuse me more than help me.

On RDMBS, I would have another table or two for archives. But here?

Update 3/22/08 12:13pm

This wont' work. Keeping old version in the same space will make it really complicated to perform query (I always need to make sure I pick up the max version, etc). So either I do not do this, or move the item into another space.

Either way I do not need business key anymore.

Persistence Layer to SimpleDB

How would this layer cache?

Should it cache by query? The answer is no.
The grand idea is that we only store documents (thanks couchDB). Anything else it extraneous. So all we need is to cache by document ID (key).

So when asked for a document, given a key.
1. Check cache if the key is in the cache, if yes return the the document.
2. No? Get it from SimpleDB
3. Cache it and return the document.

When asked to run a query given a simpleDB query:
1. pass it to simpleDB.
2. return the data.
No caching!

How about invalidation?
Hmm.. this is a tough one.

ActiveRecord on SimpleDB

Using ActiveRecord against SimpleDB doesn't make sense. ActiveRecord was fitted for RDBMS systems where the columns are defined. In the case of SimpleDB, the app must know what data it wants to get and how.

Need to make a layer that talk to simpleDB.
It must be able to:
1. take ruby command and return data objects.
2. cache data (see earlier post).
3. take in a definition of an document type(?)
4. accessible from both internal (within the same ruby vm) and external (rack interface?).

Can it deduce the "content" of the data from the data it get from SimpleDB. While this is neat idea, most likely we'll have problem figuring out whether 00001234 is a number 1234 or it is a String.

Something like this can be made fairly simple. Probably borrow some idea from ActiveRecords and other persistence layer implementation.

SimpleDB and Eventual Concurrency

So we have a system that takes in user input, stores it in SimpleDB and then query SimpleDB to display it back to the user. Very easy. But guess what, when you query SimpleDB, that data you just put in is not there! Yikes!

That's actually expected because the nature of SimpleDB. Data is propagated between "nodes" and you do not know which node you will hit for save and which node for query. They call this "eventual concurrency". You can be assured that your data will eventually be propagated to all nodes but you do not know when and how fast.

This limitation make our simple use case impossible.

The easiest solution is to maintain our own aware-node. A node that we talk to for update and a node that we talk to for query. Whether this is a standalone app or built-in doesn't matter as long as it can have the latest value and the app (and other nodes) know its location and can talk to it.

Assuming that we have a distributed system. A cluster of nodes running the same app. When user made an entry, one of the app received it and update SimpleDB.

What's next?

How would user see latest value? Well, we can keep this user on the same node. Since this node is the node that is aware of the latest data, then we can see the latest data from here. Other nodes will hit SimpleDB to get data and that's okay. In all likelyhood, User B wouldn't mind if it takes a few minutes before seeing User A's update.

This requires sticky session. Any user session must go to the same server. That's probably not too hard to do. Designing an aware node is definitely harder to do. Basically it is a cache against SimpleDB and somehow it must know when to ask SimpleDB and when to rely on what's cached.

Now the next level of use case.

User enter data. We store it in SimpleDB. Alas, SimpleDB doesn't have much in terms of arithmetic operation, so we need a job that aggregate data. No problem can be done. We put an entry in SQS for the job to pick it up.

First problem. When the job picks it up and query SimpleDB, it might not see the data yet. So it will aggregate without the new data. Pointless.

To fix that we can stick information in SQS entry on where to look for latest data. Easy enough. We tell the job to look for latest data in the aware node.

What if two different users enter data one after another. Then we have two entries in SQS.

The problem is that we cannot be sure which entries are picked up first, and each entry is not aware of the other.

Let's say the last entry is picked up first by the job. Then the job will hit the aware node for that one latest data and get the rest from SimpleDB (maybe by asking the aware node or by itself, it doesn't matter). But if the second latest data has not been propagated, it will not know where to get it from.

One solution would be for the job to get all SQS entries in the queue. The sort them into buckets of similar data (say it knows that User A and User B are both doing update on the thing that has to be computed together). Then it takes the two SQS entries, pick up the latest data from respective aware nodes, and process data.

The drawback is that this one job will consume this entire queue and we cannot have another job on this same queue. This is okay as we can create unlimited queues.

Thread and Cores: Java vs. Ruby

Tested Pentium D. Java 1.5 can utilize both cores simply by creating another Thread. I assume 1.6 can do same too.

Ruby 1.8.6 on Windows does not. It only consume one core.

Looks like Ruby 2.0 uses native thread, but where is Ruby 2.0? (The next version is still 1.9).
http://en.wikipedia.org/wiki/Green_threads