MongoDB at Craigslist: 1 Year Later

May 3 • Posted 1 year ago

Update: watch the video of Jeremy Zawodny and Chris Mooney’s talk on A Year of MongoDB at Craigslist at MongoSF ‘12

Last year, Craigslist moved their archive to MongoDB from MySQL. After the initial set up, we spoke with Jeremy Zawodny, software engineer at Craigslist and the author of High Performance MySQL (O’Reilly), and asked him some questions about their cluster. In advance of their talk at MongoSF tomorrow, we caught up with Jeremy to get the scoop on what’s happening at Craigslist one year later. 

Last time we spoke you were building a MongoDB store for 5 Billion Documents. What do your numbers look like now?

We’re currently approaching the 3 billion mark. The 5 billion number was our target capacity when building the system. Back then we had about 2.5 billion documents that we migrated into MongoDB, and we’ve continued to add documents ever since then.

Can you share an anecdote on the benefits of replica sets/sharding and something you’d like to change/improve in that feature set?

The sharding has made it easy for handling growth. We know that when the day comes, we can add an additional replica set to our cluster and it will help ease any space crunch. The replica sets have been great for handling machine failures. We’ve had several machines lock-up on us and require unplanned reboots or service. Throughout that time, the worst thing we’ve seen is some read-only time for the cluster metadata (when a config server dropped) but we’ve been able to serve requests without stopping.

Can you share some anecdotes about how your team adjusted to working with MongoDB?

There was a bit of adjustment that our systems administration team performed to the original deployment and configuration to make it better mesh with our home-grown management and deployment tools. But other than that, MongoDB has been pretty hands-off for most of the team. As long as it behaves well (which it does), we don’t need to touch it that often.

Any exciting plans for your MongoDB clusters?

We’ve been testing MongoDB in a few new roles at Craigslist and plan to present some of those challenges at MongoSF on May 4th. 

Thanks to Jeremy for giving us some insight into how MongoDB powers Craigslist!