The MongoDB NoSQL Database Blog

It’s not too late to register for NoSQL Live in Boston on March 11^th. We have an exciting lineup of speakers and panelists who will discuss real use cases for NoSQL in production systems.

Session topics at NoSQL Live will include scaling with NoSQL, NoSQL in the cloud, schema design with document-oriented databases, the evolution of graph data structure from research to production, the enterprise adoption of NoSQL, and toward web standards for NoSQL. In addition to speakers and panels, the conference will also include lightning talks and a NoSQL Lab for practical exploration of working with specific NoSQL products.

Here’s the confirmed list of speakers, panelists, and moderators:

— Dwight Merriman, CEO, 10gen

— Eliot Horowitz, CTO, 10gen

— Adam Kocoloski, CTO, Cloudant

— Alan Hoffman, CEO, Cloudant

— Durran Jordan, Senior Developer, Hashrocket

— Les Hill, Software Adventurer, Hashrocket

— Marko Rodriguez, Graph Systems Architect, AT&T Interactive

— Ryan King, Technical Lead, Storage Team, Twitter

— Alex Feinberg, Senior Software Engineer, LinkedIn

— Jonathan Ellis, Systems Architect, The Rackspace Cloud

— Sandro Hawke, Software Developer and Systems Architect, W3C

— Benjamin Day, Microsoft MVP

— Ryan Rawson, Systems Architect, StumbleUpon

— Bryan Fink, Senior Software Developer, Basho Technologies

— Rusty Klophaus, Senior Software Engineer, Basho Technologies

— Adam Wiggins, Co-founder, Heroku

— Mark Atwood, Director of Community Development, Gear6

— Sourav Mazumder, Principal Technology Architect, Infosys Technologies Limited

— Tim Anglade, CTO, GemKitty

— Bradford Stephens, Founder, Drawn to Scale

— Doug Judd, CEO, Hypertable

— Daniel Rinehart, Chief Software Architect, Allurent

— Emil Eifrém, CEO, Neo Technology

— Paul Davis, Research Assistant, New England Biolabs

— Borislav Iordanov, Software Architecture Consultant, Kobrix

— Jim Wilson, Lead Software Engineer, Vistaprint

— Ryan Angilly, Senior Developer, Punchbowl Software

Sign ups for the $40 early bird registration end today. If you’re interested in presenting or sponsoring, contact Meghan Gill, event coordinator, at meghan@10gen.com. Check out http://nosqlboston.eventbrite.com for more information and to register.

In October, 10gen hosted a webinar where we heard from 10gen CEO Dwight Merriman and The Business Insider Lead Developer Ian White about the basics of developing applications with MongoDB and about how MongoDB is used in production at TBI.

We’d like to follow up with a webinar focused on how MongoDB works “under the hood.” Please join us on March 8 at 12:30 PM Eastern Time. 10gen software engineer Mike Dirolf will lead the session. Registration is free but limited to 125 attendees.

Register now.

A couple weeks ago we asked people on Twitter, IRC, and the mailing list to fill out a survey on how they were using MongoDB.  About 120 people responded (thanks guys!).

Here is what we gleaned:

Everyone’s a noob

How long people have been using Mongo:

Most people haven’t been using Mongo for very long.  Exactly 0% said they’d been using Mongo for a year or more (which makes sense, given our first official release was ~12 months ago).

Interesting things being stored in Mongo

Lots of people are storing log data, analytics, user info… the usual.  Some less usual stuff:

• Game title development info
• Patents
• Crime reports and warrants

And quite a few people said: ”Everything.”

So, how big is it?

One person said they were testing up to 40 billion documents, but I wasn’t clear on if they had actually put in 40 billion or were going to.  So, we’ll ignore the outlier, but we can pretty safely say people are storing ~70 million documents.

On a scale of 1-10, would you recommend Mongo to a friend?

Happily, the average was 9.64!  If you are happy with MongoDB, please consider tweeting, writing a blog post, or giving a talk at a conference or meetup… the biggest obstacle we’re facing right now is letting people know we exist!

If you were below average (haha), I’d encourage you to hit the list or IRC.  We’d love to help out (or at least find out why you’re unhappy).

And, finally, most importantly, religious wars:

Kyle has the most users

Ruby wins handily with over 40% of users.

“Other” contains mainly C#, Perl, and Groovy users.

OS X: the universal dev environment

OS people are using for development:

OS people are using for production:

Go Linux go!

If you feel left out, feel free to fill out the survey now.  Thanks to everyone for you input!

Update (3/16/2011): MongoDB v1.8 includes journaling

Update (4/9/2010): real time replication and blocking for replication are implemented in v1.6

We get lots of questions about why MongoDB doesn’t have full single server durability, and there are many people that think this is a major problem.  We wanted to shed some light on why we haven’t done single server durability, what our suggestions are, and our future plans.

To start, there are some very practical reasons why we think single server durability is overvalued.  First, there are many scenarios in which that server loses all its data no matter what.  If there is water damage, fire, some hardware problems, etc… no matter how durable the software is, data can be lost.  Yes - there are ways to mitigate some of these, but those add another layer of complexity, that has to be tested, proofed, and adds more variables which can fail.

In the real world, traditional durability often isn’t even done correctly.  If you are using a DBMS that uses a transaction log for durability, you either have to turn off hardware buffering or have a battery backed RAID controller.  Without hardware buffering, transaction logs are very slow.  Battery backed raid controllers will work well, but you have to really have one.  With the move towards the cloud and outsourced hosting, custom hardware is not always an option.

Requirements for web applications are also changing.  99.99% uptime is no longer the goal, people want 100% uptime as much as possible.  If you have durability through a transaction log, then you have to replay it to come back up.  If you have a master and slave in the same data center and you lose power, both will have to recover which could take 5-30 minutes.[1]

Another feature of new non-relational databases is horizontal scalability.  While MongoDB’s auto-sharding is still in Alpha, we still feel this is a core component. With horizontal scalability comes many servers.  If you have a 100 node cluster, worrying about every machine is a liability.  If a machine goes down in the middle of the night, you want the system to recover as fast as possible, without human intervention.  Given that, and that a high percentage of failures are hardware, the best thing is to just mark that server as inactive, and ignore it until someone can look at it easily (could be hours or days).

Given all this, we’re not saying durability isn’t important, we just think that single server durability isn’t the best way to get true durability.  We think the right path to durability is replication (local and remote) and snapshotting.  That’s why we’ve spent so much time making replication fast and easy and work over wide area networks in MongoDB.

We are currently planning many more enhancements to replication to make it better.

• psuedo real-time with optional blocking for writes until on multiple servers
• replica sets instead of replica pairs
• easier to create new slaves with large data sets

Now - there are definitely some cases where single server durability is the best option.  It is on our road map, its just not on the short list right now.  We know what we want to do and how we want to do it, it’s just a matter of code :)

[1] Some databases such as CouchDB use an append only model that allows for instantaneous restarts. However, this type of design usually requires compaction routines to be run periodically, so can be costly in high update scenarios.

10gen is offering day-long MongoDB training sessions in San Francisco and New York City! Kyle Banker, a software engineer at 10gen, will be leading both sessions. Kyle has presented MongoDB in numerous forums, most recently at Chicago Ruby, and is excited to share his expertise. Kyle is preparing several interesting and challenging projects so that attendees can really get their hands dirty. Whether you are brand new to MongoDB or you’ve played with it already, you will leave this course with a comprehensive understanding of how to build applications with MongoDB.

More details available on the 10gen website.

Update on supported hosting options:

• Dreamhost is now offering instant configuration and deployment of MongoDB to DreamHost PS customers
• Webfaction and Linode have recently published instructions for installing MongoDB on their respective systems

Check out the Hosting Center for more details. If you’re interested in support from other hosting providers, please let us know which ones you’d like to see in the comments.