I have just completed moving all my database related code from SimpleDB to MongoDB. Maybe I am just the latest in a trend since Netflix is migrating to Cassandra (http://www.slideshare.net/adrianco/global-netflix-hpts-workshop). This has been a two stage process.

The first stage was to move my write dominant tables. Specifically I first moved the table that holds url meta data for our web crawler. As I scaled up the crawler SimpleDB started to have large numbers for problems with write pressure. Crawlers generate considerable write pressure and while a few dropped writes is not a real concern SimpleDB returned large numbers of errors. The transition to Mongo was smooth and Mongo has handled ever increasing write pressure. I also found SimpleDB to be too expensive and it does not have good tools for analyzing queries to analyze the load they create.

The second stage which I completed this weekend was cutting over all remaining metadata. This was also completed without fanfare and took just a couple of hours to copy and redeploy all executables.

Why did we choose Mongo? Well it has an excellent Haskell package. I very much enjoy increased type safety, which SimpleDB does not have since it stores everything as strings. I also like its operators for mutating lists atomically. The query model is pretty sweet. In fact MongoDB has become a kind of data swiss army knife for us. We use it in almost everything we do. Not that I would not use other types of solutions such as Cassandra in the future.