The MongoDB NoSQL Database Blog

Version 1.8 of MongoDB supports journaling in the storage engine for crash safety and fast recovery. An interesting question arises then regarding how journaling interacts with replication. A traditional approach might be to wait for the commit (i.e., journal physical write confirmed) before replicating any data. MongoDB does not do this. Instead, it allows data to replicate even if the journal write has yet to occur or be confirmed. We then must ask “but what happens if we crash before journaling but the data replicated out?” With replica sets, it turns out this is ok. In a replica set the rule is that the freshest node will be elected primary. Thus if the crashed node comes back up, but the node which received the unjournaled data is ahead, it will be primary. We might then ask about a cascade of failures. This is ok too as replica sets have a notion of rolling back to a consistent point of view. How do we know our data won’t be rolled back? The answer is that a write is truly committed in a replica set when it has been written at a majority of set members. We can confirm this with the getLastError command.

Read More →

Update: You can view a video of Jeremy Zawodny’s talk at MongoSF on 10gen.com.

MongoDB is now live at Craigslist, where it is being used to archive billions of records.

Craiglist has kept every post anyone has ever made in a large MySQL cluster. A few months ago, they began looking for alternatives: schema changes were taking forever (Craigslist’s schema has changed a couple times since 1995) and it wasn’t really relational information. They wanted to be able to add new machines without downtime (which sharding provides) and route around dead machines without clients failing (which replica sets provide), so MongoDB was a very strong candidate. After looking into a few of the most popular non-relational database systems, they decided to go with MongoDB.

Jeremy Zawodny is a software engineer at Craigslist and an author of High Performance MySQL (O’Reilly). He kindly agreed to answer some questions about their MongoDB cluster (editor’s comments in italics).

Any numbers you can give us?

We’re sizing the install for around 5 billion documents. That’s from the initial 2 billion document import we need to do plus room to grow for a few years to come. Average document size is right around 2KB. (Five billion 2KB documents is 10TB of data.) We’re getting our feet wet with MongoDB so this particular task isn’t high throughput or growing in unpredictable ways.

We can put data into MongoDB faster than we can get it out of MySQL during the migration.

Read More →

An interview with Jeff Yemin, Director of Content Management Systems at MTV Networks and a presenter at the upcoming MongoNYC conference.

Read More →

Improving scalable Java application development with MongoDB + Morphia:

Over the last year I have seen a significant rise in the number of questions and interest from both the greater Java community and enterprise Java shops about MongoDB. Coming from the MongoDB and Java worlds (among others), this is something I have watched with great interest and excitement.

As one of the authors and project leads for Morphia(MongoDB Java ORM) I have seen a lot of questions relating to both the core driver and how to build Java applications with MongoDB. A lot of these questions arise from the paradigm shift users experience when moving from the standard SQL/JPA/Hibernate platforms/frameworks to the document oriented world of MongoDB.

Read More →

In light of last week’s EBS issues, we wanted to make sure MongoDB users on EBS are configured to be as robust as possible. A basic setup would consist of a 3 node replica set. The nodes would be roughly laid out like this: * A: us-east-1a priority 1 * B: us-east-1b priority 1 * C: us-west-1a priority 0 

Read More →

Listen to Jared Rosoff’s June 2 webinar “VMware Cloud Foundry with MongoDB”.
Read about getting started with Cloud Foundry, MongoDB, and Node.js.

Last week, VMware launched Cloud Foundry: an open-source platform as a service. It’s pretty radical in that not only can you run your apps on infrastructure operated by VMware, you can also download Cloud Foundry itself and run it on your own machines!

But what’s most awesome about Cloud Foundry is that it supports MongoDB right out of the box! In today’s post, we’re going to walk through the creation of a Rails application using MongoDB and Cloud Foundry.

Here’s what we’re going to need to do:

1. Create a new rails project (skipping Active Record)
2. Add the dependencies for MongoMapper
3. Build a simple app that interacts with the database
4. Set up our credentials so we can talk to Cloud Foundry’s MongoDB
5. Push our application to CloudFoundry

Read More →

10gen is happy to announce support for the official C# driver for MongoDB. Several preview releases have already been made available, and the latest, Version 0.11, was released January 25, 2011. Version 1.0 has just been released and includes support for the new features in MongoDB 1.8.

Read More →

We are happy to announce that MongoDB v1.8.0 is now available. 1.8 is the stable follow-up release to 1.6, which came out in August of 2010. Version 1.8 introduces many new features, along with bug fixes and other improvements. Some of the highlights:

• Journaling
• Sharding performance improvements
• Replica set enhancements, including support for authentication
• Spherical geo search
• Covered and sparse indexes
• B-tree index self-compaction
• New map/reduce options for incremental updates
• Tab completion in the shell
• mongostat --discover

Read More →

The state of Ruby and MongoDB is strong. In this post, I’d like to describe some of the recent developments in the Ruby driver and provide a few notes on Rails and the object mappers in particular.

We just released v1.2 of the MongoDB Ruby driver. This release is stable and supports all the latest features of MongoDB. If you haven’t been paying attention to the driver’s development, the Cliff’s Notes are below. (Note that if you’re an using older version of the driver, you owe it to your app to upgrade).

If you’re totally new to the driver, you may want to read Ethan’s Gunderson’s excellent post introducing it before continuing on.

Read More →

Here’s a rundown of some of the most useful features added recently. These are all available in 1.7.4 and will, of course, be in 1.8.

1. Initial sync from a secondary

   You can now set an initialSyncsource for each member, which controls where the new guy will sync from. For example, if you wanted to add a new node and force it to sync from a secondary, you could do:

   > rs.add({"_id" : num, "host" : hostname, "initialSync" : {"state" : 2}}) 
   

   You can choose a sync source by its state (primary or secondary), _id, hostname, or up-to-date-ness. For the last, you can specify a date or timestamp and the new member will choose a source that is at least that up-to-date.

   By default, a new member will attempt to sync from a random secondary and, if it can’t find one, sync from the primary. If it chooses a secondary, it will only use the secondary for its initial sync. Once it’s ready to sync new data, it will switch over and use the primary for “normal” syncing.

Read More →

Someone recently pointed out to me, rather insightfully, that MongoDB is a good fit for archival of relational data.  

I had not really considered this before, but it is a good point : flexible schemas are very helpful for archival.  How do we keep an archive of data, say, 10 years or more of data history, when over that time period the schema will undergo significant changes?  It is not so easy.

One approach would be to apply any schema changes from the online / operational database at the archival database too.  However, there are some issues.  First, the archival database may be huge, making schema migrations impractical.  But more importantly, these changes may not be what we want in an archive.  Imagine we decide to drop a column in the online db.  It may now be deprecated and unneeded.  However, a true and complete archive would still have that data.  Dropping the column in the archive is not what we want.

Document-oriented databases, with their flexible schemas, provide a nice solution.  We can have older documents which vary a bit from the newer ones in the archive.  The lack of homogeneity over time may mean that querying the archive is a little harder.  However, keeping the data is potentially much easier.

—dm

MongoDB 1.6.0 is the fourth stable major release (even numbers are “stable” : 1.0, 1.2, 1.4, …) and is the culmination of the 1.5 development series.

Scale-out

The focus of the 1.6 release is scale-out.  Sharding is now production-ready.  The combination of sharding and replica sets allows one to build out horizontally scalable data storage clusters with no single points of failure.

A single instance of mongod can be upgraded to a distributed cluster with zero downtime when the need arises.

A big thanks to all the 1.5.x beta testers of sharding (including foursquare and bit.ly who have been using sharding in production for a while now).

Read More →

Visit the more recent post, Getting Started with VMware Cloud Foundry, MongoDB, and Node.js. Listen to the recorded Node.js Panel Discussion webinar.

Node.js is turning out to be a framework of choice for building real-time applications of all kinds, from analytics systems to chat servers to location-based tracking services. If you’re still new to Node, check out Simon Willison’s excellent introductory post. If you’re already using Node, you probably need a database, and you just might have considered using MongoDB.

The rationale is certainly there. Working with Node’s JavaScript means that MongoDB documents get their most natural representation — as JSON — right in the application layer. There’s also significant continuity between your application and the MongoDB shell, since the shell is essentially a JavaScript interpreter, so you don’t have to change languages when moving from application to database.

Read More →

We’re pleased to announce the winner’s of the MongoDB blogging contest!

Grand Prize

• Why (and How) I Replaced Amazon SQS with MongoDB - Matt Insler

Runners Up

• Running MongoDb on Microsoft Windows Azure with CloudDrive - Simon Green
• Using MongoDB for CodeIgniter Logs - Tom Schlick
• Integrating MongoDB with Spring Batch - Baruch Sadogursky

The winners should contact meghan@10gen.com to claim their prizes.

You check out all the awesome entries at mongodb.slinkset.com

Thanks to everyone who submitted!

On May 21, 10gen organized the second conference dedicated to MongoDB. Like MongoSF, MongoNYC included a great line-up of speakers. One of the more popular talks was Kyle Banker’s Schema Design session, which was so crowded that many attendees sat on the floor! Both the video and slides from the talk are now available. 

Read More →

Valentin Kuznetsov just presented a paper at the International Conference on Computational Science on CERN’s use of MongoDB for Large Hadron Collider data. The paper, The CMS Data Aggregation System, is available as a PDF at ScienceDirect.

“CMS” stands for Compact Muon Solenoid, a general-purpose particle physics detector built on the Large Hadron Collider. The CMS project posted a few comics which provide a nice, simple (if somewhat cheesy) explanation of what the CMS/LHC does.

The LHC generates massive amounts of data of all different varieties, which is distributed across a worldwide grid. It sends status messages to some of the computers, job monitoring info to other computers, bookkeeping info still elsewhere, and so on.

This means that each location has specialized queries it can do on the data it has, but up until now it’s been very difficult to query across the whole grid. Enter the Data Aggregation System, designed to allow anything to be queried across all of the machines.

Read More →

10gen has a ticket to OSCON that we’d like to give to a MongoDB user.

How to Enter

• Write a blog post.  It has to be about MongoDB, but within that it can be anything: a how-to, an experience you had, a review, a rant, a rave, a technical piece, a humorous piece… whatever you want.
• Post your blog at mongodb.slinkset.com and include your contact information in the description.  You must submit by June 1st.  Your post must be publicly accessible (not behind a pay wall or members-only site).
• We’ll announce the winners June 7th.

Prizes

Grand prize

• 1 ticket to OSCON
• MongoDB swag package: shirt, coffee mug, and stickers
• Option to do a guest post on blog.mongodb.org

Read More →

MongoSF, the first full-day conference dedicated to MongoDB, featured 35+ sessions and even produced a few surprises along the way. Over 200 people attended the April 30 conference. Slides and video from many sessions now available on the 10gen website.

The Sharding presentation was one of the major highlights of the event. Eliot Horowitz, the CTO of 10gen, demoed a 25-node cluster on EC2. Check out the video of the session.

MongoHQ made a major announcement during their session, launching their add-on to all Heroku users as a public beta. For more details, check out the Heroku blog.

The conference was held at Bently Reserve, where we hope many future tech conferences will take place. Not only is the venue beautiful and historic, but it is fully equipped for conferences. The wireless actually worked!

More MongoDB conferences coming soon! Use the discount code “blog” when registering.

MongoNYC - Friday, May 21
MongoUK - Friday, June 18 
MongoFR - Monday, June 21 

MongoDB conferences are coming to Europe! MongoUK is on Friday, June 18 at Skills Matter and MongoFR is on Monday, June 21 at La Cantine. Each conference will feature sessions from the 10gen team on schema design, replication, sharding, indexing, and map/reduce. In addition, attendees will learn about MongoDB in production through presentations by companies like Boxed Ice, Silentale, OCW Search, and Novelys.

10gen is organizing MongoFR in conjunction with the NoSQL Paris User Group, with the help of NoSQL enthusiast Tim Anglade. (Pour les francophones : nous devons vous avertir que certaines sessions de MongoFR seront en anglais et d’autres en français. La journée sera également suivie d’une rencontre spéciale de l’UG — entrée gratuite à partir de 19h à La Cantine. Pour plus d’informations en français, contactez timanglade@gmail.com par email ou @timanglade sur Twitter.)

We look forward to seeing you there!

The MongoDB team is very excited about how the developer community is building around MongoDB, and we wanted to share some numbers.

These are download numbers for the core server for January through March.  It is exactly the number of downloads of the core database from downloads.mongodb.org minus all bots (all known plus anything with bot in the user-agent) and all other crawlers we determine.  We use these numbers internally, so we do try and keep them accurate.

January    15647
February  23226
March      37144

We are very excited about these numbers — please spread the word and help us continue growth of MongoDB!

-Eliot