The MongoDB NoSQL Database Blog

MongoDB and Twilio Contest

Wed, 18 Aug 2010 13:49:00 -0400

MongoDB and Twilio are teaming up this week to do a contest! You have until midnight on August 22nd to create an application using Twilio and MongoDB. The best application wins:

A netbook
$100 of Twilio credit
MongoDB Timbuk2 Laptop Bag (the only other ways to get one are to be a major contributor or write a MongoDB book)
MongoDB T-shirt
MongoDB Coffee Mug
MongoDB Stickers

It’s a very open ended contest: you can create any application, so long as it uses MongoDB to store some of its data and the Twilio API. When you’re done, submit it on the Twilio website. Some resources to get you started:

Twilio

MongoDB

Good luck!

MongoDB 1.6 Released

Thu, 05 Aug 2010 12:28:00 -0400

MongoDB 1.6.0 is the fourth stable major release (even numbers are “stable” : 1.0, 1.2, 1.4, …) and is the culmination of the 1.5 development series.

Scale-out

The focus of the 1.6 release is scale-out. Sharding is now production-ready. The combination of sharding and replica sets allows one to build out horizontally scalable data storage clusters with no single points of failure.

A single instance of mongod can be upgraded to a distributed cluster with zero downtime when the need arises.

A big thanks to all the 1.5.x beta testers of sharding (including foursquare and bit.ly who have been using sharding in production for a while now).

Replica Sets

Replica sets allow you to setup a high availability cluster with automatic fail over and recovery. Replica pair users should, when convenient, migrate to replica sets.

Other Improvements in v1.6

acknowledged replication: The w option (and wtimeout) force writes to be propagated to N servers before returning success (works well with replica sets).
$or queries
Up to 64 indexes/collection
Improved concurrency
$slice operator
Support for UNIX domain sockets and IPv6
Windows service improvements
The C++ client is a separate tarball from the binaries

Downloads: http://www.mongodb.org/display/DOCS/Downloads

Release Notes: http://www.mongodb.org/display/DOCS/1.6+Release+Notes

Full change log

Please report any issues to http://groups.google.com/group/mongodb-user (support forums) or http://jira.mongodb.org/ (bug/feature db).

What’s Next

Now that 1.6 is out, we’re going to be focusing on 1.8. Help us prioritize features for this release by voting for your key needs at jira.mongodb.org. The #1 feature queued for v1.8 is single server durability.

More Information

Please join 10gen CEO and Co-Founder Dwight Merriman for the webinar What’s New in MongoDB v1.6 on Tuesday, August 10 at 12:30pm ET / 9:30am PT.

Node.js and MongoDB

Wed, 14 Jul 2010 15:49:00 -0400

Node.js is turning out to be a framework of choice for building real-time applications of all kinds, from analytics systems to chat servers to location-based tracking services. If you’re still new to Node, check out Simon Willison’s excellent introductory post. If you’re already using Node, you probably need a database, and you just might have considered using MongoDB.

The rationale is certainly there. Working with Node’s JavaScript means that MongoDB documents get their most natural representation — as JSON — right in the application layer. There’s also significant continuity between your application and the MongoDB shell, since the shell is essentially a JavaScript interpreter, so you don’t have to change languages when moving from application to database.

Node.js MongodB Driver

Especially impressive to us at 10gen has been the community support for Node.js and MongoDB. First, there’s Christian Kvalheim’s excellent mongodb-node-native project, a non-blocking MongoDB driver implemented entirely in JavaScript using Node.js’s system libraries. The project is a pretty close port of the MongoDB Ruby driver, making for an easy transition for those already used to the 10gen-supported drivers. If you’re just starting, there’s a helpful mongodb-node-native mailing list.

Hummingbird

Need a real-world example? Check out Hummingbird, Michael Nutt’s real-time analytics app. It’s built on top of MongoDB using Node.js and the mongodb-node-native driver. Hummingbird, which is used in production at Gilt Groupe, brings together an impressive array of technologies; it uses the express.js Node.js app framework and sports a responsive interface with the help of web sockets. Definitely worth checking out.

Mongoose

Of course, one of the admitted difficulties in working with Node.js is dealing with deep callback structures. If this poses a problem, or if you happen to want a richer data modeling library, then Mongoose is the answer. Created by Learnboost, Mongoose sits atop mongodb-node-native, providing a nice API for modeling your application.

Node Knockout

All of this just to show that the MongoDB/Node.js ecosystem thrives. If you need a good excuse to jump into Node.js or MongoDB development, be sure to check out next month’s Node Knockout. It’s a weekend app competition for teams up to four, and registration is now open.

Blog Contest Winners!

Tue, 08 Jun 2010 15:50:04 -0400

We’re pleased to announce the winner’s of the MongoDB blogging contest!

Grand Prize

Why (and How) I Replaced Amazon SQS with MongoDB - Matt Insler

Runners Up

Running MongoDb on Microsoft Windows Azure with CloudDrive - Simon Green
Using MongoDB for CodeIgniter Logs - Tom Schlick
Integrating MongoDB with Spring Batch - Baruch Sadogursky

The winners should contact meghan@10gen.com to claim their prizes.

You check out all the awesome entries at mongodb.slinkset.com

Thanks to everyone who submitted!

Highlights from MongoNYC

Mon, 07 Jun 2010 11:15:03 -0400

On May 21, 10gen organized the second conference dedicated to MongoDB. Like MongoSF, MongoNYC included a great line-up of speakers. One of the more popular talks was Kyle Banker’s Schema Design session, which was so crowded that many attendees sat on the floor! Both the video and slides from the talk are now available.

Also interesting were the many talks on MongoDB production deployments. Kushal Dave, the CTO at Chartbeat, gave an excellent talk on how Chartbeat came to use MongoDB after trying many solutions to store historical data analytics (see slides & video). Jay Ridgeway talked about bit.ly user history, which is auto-sharded using MongoDB (slides & video). Gilt Groupe demoed their real-time analytics tool Hummingbird, which is built with MongoDB and node.js. Avery Rosen, the CTO of ShopWiki, wrote a recap of his talk “Finding a Swiss army data store” on the ShopWiki dev blog.

Another big hit was Harry Heymann’s presentation on MongoDB at foursquare, the video of which is included below.

Videos from all the talks at MongoNYC are available at mongodb.blip.tv.

Thanks for making the event such a success! We’re getting really excited about MongoUK and MongoFR, which are only a few weeks away.

Holy Large Hadron Collider, Batman!

Thu, 03 Jun 2010 09:53:00 -0400

Valentin Kuznetsov just presented a paper at the International Conference on Computational Science on CERN’s use of MongoDB for Large Hadron Collider data. The paper, The CMS Data Aggregation System, is available as a PDF at ScienceDirect.

A summary

“CMS” stands for Compact Muon Solenoid, a general-purpose particle physics detector built on the Large Hadron Collider. The CMS project posted a few comics which provide a nice, simple (if somewhat cheesy) explanation of what the CMS/LHC does.

The LHC generates massive amounts of data of all different varieties, which is distributed across a worldwide grid. It sends status messages to some of the computers, job monitoring info to other computers, bookkeeping info still elsewhere, and so on.

This means that each location has specialized queries it can do on the data it has, but up until now it’s been very difficult to query across the whole grid. Enter the Data Aggregation System, designed to allow anything to be queried across all of the machines.

How it works

The aggregation system uses MongoDB as a cache. It checks if Mongo has the aggregation the user is asking for and, if it does, returns it, otherwise the system does the aggregation and saves it to Mongo.

They query the system using a simple, SQL-like language which they transform into a MongoDB query. So, something like file="abc", run>10 becomes {"file" : "abc", "run" : {"$gt" : 10}}. (It’s not the same as SQL, but the code for this might be interesting to people who want to use SQL queries with MongoDB.)

If the cache does not contain the requested query, the system iterates over all of the places in the world that could have this information and queries them, gathering their results. It then merges all of the results, doing a sort of “group by” operation based on predefined identifying key and inserts the aggregated information into the cache.

It was built using the Python driver.

Goals

They’re looking forward to field testing it and horizontally scaling the system with sharding. As this is a general grid aggregation/querying tool, they’re also interested in applying it to problems outside of the LHC and CERN.

We wish them luck and hope they’ll keep us informed on future progress!

Edit: the slides from Valentin’s presentation are available at http://www.slideshare.net/vkuznet/das-iccs-2010.

Kristina Chodorow maintains the MongoDB PHP and Perl drivers. She blogs at www.snailinaturtleneck.com and tweets as @kchdorow.

Write a blog post on MongoDB for a chance to win a ticket to OSCON 2010!

Mon, 17 May 2010 11:08:22 -0400

10gen has a ticket to OSCON that we’d like to give to a MongoDB user.

How to Enter

Write a blog post. It has to be about MongoDB, but within that it can be anything: a how-to, an experience you had, a review, a rant, a rave, a technical piece, a humorous piece… whatever you want.
Post your blog at mongodb.slinkset.com and include your contact information in the description. You must submit by June 1st. Your post must be publicly accessible (not behind a pay wall or members-only site).
We’ll announce the winners June 7th.

Prizes

Grand prize

1 ticket to OSCON
MongoDB swag package: shirt, coffee mug, and stickers
Option to do a guest post on blog.mongodb.org

There will also be 3 runners up who get MongoDB mugs and stickers, as well as mentions/links on blog.mongodb.org.

Judging

50% public judging - vote up your favorite blogs at mongodb.slinkset.com
50% will be done by a panel of MongoDB core developers. We’ll be looking for a really great piece on MongoDB, a piece that is really interesting. Please don’t forget to proofread!

Rules

It must be written. No screencasts or podcasts, sorry. You can use images but no audio or video are allowed.
It must be posted at mongodb.slinkset.com, and include your contact information in the description.
Entries will be accepted between Monday May 17 and Tuesday June 1st.
You must provide for your own travel, we’re just giving you the OSCON ticket. You have to get your own plane ticket, hotel reservation, visa (if you’re outside the US) etc.

If you don’t have a blog, you can get one in about 3 seconds from Posterous or tumblr. Make sure you’re identifiable if you go this route!

Good luck everyone!

MongoSF Slides & Video; Discounts on upcoming MongoDB conferences

Mon, 10 May 2010 09:40:17 -0400

MongoSF, the first full-day conference dedicated to MongoDB, featured 35+ sessions and even produced a few surprises along the way. Over 200 people attended the April 30 conference. Slides and video from many sessions now available on the 10gen website.

The Sharding presentation was one of the major highlights of the event. Eliot Horowitz, the CTO of 10gen, demoed a 25-node cluster on EC2. Check out the video of the session.

MongoHQ made a major announcement during their session, launching their add-on to all Heroku users as a public beta. For more details, check out the Heroku blog.

The conference was held at Bently Reserve, where we hope many future tech conferences will take place. Not only is the venue beautiful and historic, but it is fully equipped for conferences. The wireless actually worked!

The main banking hall at Bently Reserve at 8:30 on the morning of MongoSF - it filled up shortly thereafter!

More MongoDB conferences coming soon! Use the discount code “blog” when registering.

MongoNYC - Friday, May 21
MongoUK - Friday, June 18
MongoFR - Monday, June 21

MongoDB Conferences in London and Paris in June

Wed, 05 May 2010 06:00:00 -0400

MongoDB conferences are coming to Europe! MongoUK is on Friday, June 18 at Skills Matter and MongoFR is on Monday, June 21 at La Cantine. Each conference will feature sessions from the 10gen team on schema design, replication, sharding, indexing, and map/reduce. In addition, attendees will learn about MongoDB in production through presentations by companies like Boxed Ice, Silentale, OCW Search, and Novelys.

10gen is organizing MongoFR in conjunction with the NoSQL Paris User Group, with the help of NoSQL enthusiast Tim Anglade. (Pour les francophones : nous devons vous avertir que certaines sessions de MongoFR seront en anglais et d’autres en français. La journée sera également suivie d’une rencontre spéciale de l’UG — entrée gratuite à partir de 19h à La Cantine. Pour plus d’informations en français, contactez timanglade@gmail.com par email ou @timanglade sur Twitter.)

We look forward to seeing you there!

MongoDB Q1 Download Numbers

Mon, 26 Apr 2010 10:32:11 -0400

The MongoDB team is very excited about how the developer community is building around MongoDB, and we wanted to share some numbers.

These are download numbers for the core server for January through March. It is exactly the number of downloads of the core database from downloads.mongodb.org minus all bots (all known plus anything with bot in the user-agent) and all other crawlers we determine. We use these numbers internally, so we do try and keep them accurate.

January    15647
February  23226
March      37144

We are very excited about these numbers — please spread the word and help us continue growth of MongoDB!

-Eliot

On Distributed Consistency - Part 6 - Consistency Chart

Thu, 15 Apr 2010 11:32:12 -0400

See also:

The following diagram (click for large version) shows the various consistency models that have been discussed in this blog post series. Stronger consistency modes generally meet the requirements of weaker modes, and are thus shown as subsets in this Venn-like diagram.

Keep in mind that for many products, consistency is tunable: a product doesn’t necessarily belong to a particular rectangle, but a given operation certainly does.

Eventual Consistency - eventual consistency as defined by Amazon in the Dynamo paper.
Monotonic read consistency - a stricter form of eventual consistency.
Read-your-own-writes consistency - a stricter form of eventual consistency.
MRC + RYOW - a system with both monotonic read plus read-your-own-writes properties. A master-master replication system, where a given client always interacts with a single master, would have these properties.
Immediate Consistency - a system which is immediately consistent but which does not support atomic operations. Strict quorum systems, where R+W>N, meet this criteria (and theoretically could do more, depending on the design).
Strong Consistency - a system which supports read/write atomic operations on single data entities. This is the default mode for MongoDB.
Full Transactions - Oracle!

On Distributed Consistency - Part 5 - Many Writer Eventual Consistency

Wed, 14 Apr 2010 10:18:00 -0400

See also:

In part 2 we primarily discussed “single writer” eventual consistency. Here we will discuss many-writer, and define that term more precisely.

By many-writer, we mean a system where different data servers can receive writes concurrently (and asynchronously). Examples of many-writer eventually consistent systems include

Amazon Dynamo
CouchDB master-master replication

With multi-writer eventual consistency, we need to address the phenomenon of conflicting writes. Writes to two servers at the same time may be updates for the same object. We must resolve the conflict in a way that is acceptable for the use case in question. Some ways to do this are:

last write wins
programmatic merge
commutative operations

Last Write Wins

Last write wins is a popular default in many systems. If we receive an operation that is older, we simply ignore it. In a distributed system the definition of “last” is hard as clocks can’t be perfectly synchronized. Thus many systems use vector clocks.

Inserts

Surprisingly, a traditional insert operation is tricky with many writers. Consider these operations performed at about the same time at different servers:

op1: insert( { _id : 'joe', age : 30 } )
op2: insert( { _id : 'joe', age : 33 } )

If we naively apply these two operations in any order, we get an inconsistent result. insert typically means:

if( !already_exists(x._id) ) then set( x );

However, with eventual consistency we do not have real-time global state. Checking already_exists() is thus hard.

The best solution is to not support insert, but rather set() - i.e. “set a new value”. Sometimes this is called an upsert. Then, if we have last-write-wins semantics, everything is fine.

Deletes

Deletes require special handling in cases of object rebirth. Consider this sequence:

op1: set( { _id : 'joe', age : 40 } }
op2: delete( { _id : 'joe' } )
op3: set( { _id : 'joe', age : 33 } )

If op2 and op3 are reversed in execution order, we would have a problem. Thus we need to remember the delete for a while, and apply last-operation-wins semantics. Some products call the remembrance of the delete a tombstone.

Updates

Updates have a similar issue as insert, so for updates, we use the set() operation we described above instead.

Note that partial object updates can be tricky to replicate efficiently. Consider a set() operation where we wish to update a single field:

update users set age=40 where _id=’joe’

This is no problem with eventual consistency if we replicate a full copy of the object. However, what if the user object was 1MB in size? It would be really nice to just send the new age field and the _id, rather than the whole object. However, this is difficult. Consider:

op1: update users set age=40 where _id='joe'
op2: update users set state='ca' where _id='joe'

Wen can’t simply replicate the partial update and use last-write-wins; the database will need more sophistication to handle this efficiently.

Programmatic Merge

Last-write-wins is great, but is not always sufficient. Having the client application resolve the conflict via a merge is a fine alternative. Let’s consider an example mentioned in the Amazon Dynamo paper: manipulations of shopping carts. With eventual consistency it would not be safe to do something like:

update cart set this[our_sku].qty=1 where _id='joe'

If there are multiple manipulations of the cart, some may get lost using last-write-wins. Instead, the Dynamo paper talks of storing the operations in the cart object, rather than the actual data state. We could store something like:

update cart append { time : now(), op : 'addToCart', sku : our_sku, qty : 1 }
  where _id='joe'

When a conflict occurs, cart objects can be merged. We do not lose any operations. When it is time to check out, we replay all the operations, which might include quantity adjustments and removes from cart. After replay we have the final cart state.

Note the above example uses a timestamp field — in a real system a vector clock might be used to order the operations in the cart.

It’s interesting to note that not only have we avoided conflicts, we are also able to do operations where atomicity would be required.

Commutative Operations

If all operations are commutative (more precisely, foldable), we will never have any conflicts. Operations can simply be applied in any order, and the result is the same. For example:

// x starts as { }
x.increment('a', 1);
x.increment('a', 3);
x.addToSet('b', 'foo');
x.addToSet('b', 'bar');
result: { a : 4, b : {bar,foo} }

// x starts as { }
x.addToSet('b', 'bar');
x.increment('a', 3);
x.increment('a', 1);
x.addToSet('b', 'foo');
result: { a : 4, b : {bar,foo} }

Note however that composition of addToSet and increment would not be foldable; thus, we have to use only one or the other for a particular field of the object.

On Distributed Consistency - Part 4 - Multi Data Center

Mon, 12 Apr 2010 18:01:00 -0400

See also:

Eventual consistency makes multi-data center data storage easier. There are reasons eventual consistency is helpful for multi-data center that are unrelated to availability and CAP. And as mentioned in Part 3, some common types of network partitions, such as loss of an entire data center, are actually trivial network partitions and may not even effect availability anyway.

Here are a few architectures for multi-data center data storage:

DR
Single Region
Local reads, remote writes
Intelligent Homing
Eventual consistency

By DR we mean a traditional disaster recovery / business continuity architecture. It’s pretty simple: we serve everything from one data center, with replication to a secondary facility that is offline. In a failure we cut over.

Availability can be quite high in this model as on any issue with the first data center, including internal network partitions, we cut over, and with the whole first data center disabled, the partition is trivial.

This model works fine with strong consistency.

Multi Data Center, Single Region

This option is analogous to using multiple data centers within a single region. Amazon and DoubleClick have used this scheme in the past. We have multiple data centers, physically separated, but all within one region (such as the Northwest). The latency between data centers is then reasonable: if we stay within a 150 mile radius, we can have round trip times of around 5ms. We might have a fiber ring among say, 3 or 4 data centers. As the latency is reasonable, for many problems, a WAN operation here is fine. With a ring topology, a non-trivial network partition is unlikely.

Single region is useful both for strong consistent and eventually consistent architectures. With a Dynamo style product, when N=W or N=R, this is a good option, as otherwise when using multiple data centers we will have a long wait time to confirm remote writes.

Local Reads, Remote Writes

For read-heavy use cases, this is a good option. Here we read eventually consistent data (easy with most database products including RDBMS systems) but do all writes back to the master facility over the WAN. A dynamo style system in multiple data centers with a very high W value and low R value can be thought of this way also.

This pattern would work great for traditional content management: publishing is infrequent and reading is very frequent.

Using a Content Delivery Network (CDN), with a centralized origin web site serving dynamic content, is another example.

Intelligent Homing

We discussed “Intelligent Homing” a bit in Part 3. The idea is to store the master copy of a given data entity near its user.

This model works quite well if data correlates with the user, such as the user’s profile, inbox, etc.

We have fast locally confirmed writes. If a data center goes completely down, we could still fail over master status to somewhere else which has a replica.

Eventual consistency

Many-writer eventual consistency gives us two benefits with multiple data centers:

higher availability in the face of network outages;
fast locally confirmed writes

In the diagram below, a client of a dynamo-style system writes the data to four servers (N=4). However, it only awaits confirmation of the writes from two servers in its local data center, to keep write confirmation latency low.

Note however that if R+W > N, we can’t have both fast local reads and writes at the same time if all the data centers are equal peers.

Combinations

Combinations often make sense. For example, it’s common to mix DR and Read Local Write Remote.

On Distributed Consistency - Part 3 - Network Partitions

Thu, 08 Apr 2010 10:46:00 -0400

See also:

It’s fascinating that the formal theorem statement for CAP, in the first proof (that I know of), doesn’t use the word partition!

Theorem 1 It is impossible in the asynchronous network model to implement a read/write data object that guarantees the following properties:
• Availability
• Atomic consistency in all fair executions (including those in which messages are lost).

That said, let’s talk about partitions, as “messages lost…in the asynchronous network model” is directly analogous.

Let’s look at an example:

In our diagram above, the network is partitioned. The left and right halves (perhaps these correspond say to two continents) cannot communicate at all. Four clients and four data server nodes are shown in the diagram. So what are our options?

Deny all writes. If we deny all writes when the network is partitioned, we can still read fully consistent data on both sides. So this is one option. We give up write availability, and keep consistency.
Allow writes on one side. Via some sort of consensus mechanism, we could let one side of the partition “win” and have a master (as shown by the “M” in the diagram). In this case, reads and writes could occur on that side. On the other non-master partitions, we could either (a) be strict and allow no operations, or (b) allow eventually consistent reads, but no writes. So in this situation we have full consistency in one partition, and partial operation in all others.
Allow reads and writes in all partitions. Here, we keep availability, but we must sacrifice strong consistency. One partition will not see the operations and state from the other until the network is restored. Once restored, we will need to a method to merge operations that occurred while disconnected.

A mitigation technique also comes to mind. Suppose a particular client C has a much higher probability of needing an entity X than other clients. If we store the master copy of X on a server close to C, we increase the probability that C can read and write X in option (2) above. Let’s call this “intelligent homing”. A real world example of this would be to “store master copies of data for east coast users on servers on the east coast”. Intelligent homing doesn’t solve our problems, but would likely significantly decrease their frequency — that’s good, we just want more nines anyway.

Hopefully the above is a good informal “proof” of CAP. It really is pretty simple.

Trivial Network Partitions

Many common network partitions are what we might term trivial. Let’s consider from the perspective of option (2) above. We define a trivial network partition is one such that on all non-master partitions, there are either

no live clients at all, or
no servers at all

For example, if we have many data centers and our clients are Internet web browsers, and one of our data centers goes completely dark (and we have more left), that is a trivial network partition (we assume here that we can fail over master status in such a situation). Likewise, losing a single rack in its entirety is often a trivial network partition.

In these situations, we can still be consistent and available. (Well, for the partitioned client, we are unavailable, but that is of course a certainty if it cannot reach any servers anywhere.)

On Distributed Consistency - Part 2 - Some Eventual Consistency Forms

Mon, 05 Apr 2010 09:42:00 -0400

On Distributed Consistency -- Part 1

Fri, 26 Mar 2010 15:32:00 -0400

See also:

For distributed databases, consistency models are a topic of huge importance. We’d like to delve a bit deeper on this topic with a series of articles, discussing subjects such as what model is right for a particular use case. Please jump in and help us in the comments.

We’ll start here with a basic introduction to the subject.

CAP

The CAP theorem states that one can only have two of consistency, availability, and tolerance to network partitions at the same time. In distributed systems, network partitioning is inevitable and must be tolerated, so essential CAP means that we cannot have both consistency and 100% availability.

Informally, I would summarize the CAP theorem as:

If the network is broken, your database won’t work.

However, we do get to pick the definition of “won’t work”. It can either mean down (unavailable) or inconsistent (stale data).

More precisely what do we mean by “consistency”? The academic work in this area is referring to “one copy serializability” or “linearizability”. If a series of operations or transactions are performed, they are applied in a consistent order. One less formal way of thinking about the trade-off is “Could I be reading and manipulating stale/dirty data? Can I always write?”

Embodiments

We have two classes of architectures: a C class (strongly consistent) and an A class (higher availability looser consistency). Let’s consider some real-world distributed systems and where they are classified.

Amazon Dynamo is a distributed data store which implements consistent hashing and is in the A camp. It provides eventual consistency. One may read old data.

CouchDB is typically used with asynchronous master-master replication and is in the A camp. It provides eventual consistency.

A MongoDB auto-sharding+replication cluster has a master server at a given point in time for each shard. This is in the C camp. Traditional RDBMS systems are also strongly consistent (as typically used) - a synchronous RDBMS cluster for example.

It’s worth noting that alternate configurations of these products sometimes alter their consistency (and performance) properties. For our discussion here, we’ll assume these products are configured in their common case setup, unless otherwise specified.

Write Availability, not Read Availability, is the Main Question

With most databases today, it’s easy to have any number of asynchronous slave replicas distributed about the world. If networks partition, we would then still have access to local slave data. As the replication is asynchronous, this data is eventually consistent, so this result is not surprising — we are now in the A class of systems. However, almost all designs, even from the C class, can add on asynchronous read capabilities easy. Thus, the critical design decisions are around write availability.

The Trade-offs

even load distribution is easier in eventually consistent systems
multi-data center support is easier in eventually consistent systems
some problems are not solvable with eventually consistent systems
code is sometimes simpler to write in strongly consistent systems

We will discuss these pros in cons in more details in subsequent articles.

—dwight

MongoDB 1.4 Ready for Production

Thu, 25 Mar 2010 13:22:00 -0400

The MongoDB team is very excited to announce the release of MongoDB 1.4.0. This is the culmination of 3 months of work in the 1.3 branch and has a large number of very important changes.

Many users have been running 1.3 in production, so this release is already very thoroghly vetted both by our regressions systems and by real users.

Some highlights:

Core server enhancements

concurrency improvements
indexing memory improvements
background index creation
better detection of regular expressions so the index can be used in more cases
performance numbers

Replication & Sharding

better handling for restarting slaves offline for a while
fast new slaves from snapshots
configurable slave delay
replication handles clock skew on master
$inc replication fixes
sharding alpha 3 - notably 2 phase commit on config servers

Deployment & production

configure “slow” for profiling
ability to do fsync + lock for backing up raw files
option for separate directory per db
http://localhost:28017/_status to get serverStatus via http
REST interface is off by default for security (—rest to enable)
can rotate logs with a db command, logRotate
enhancements to serverStatus - counters/replication lag
new mongostat tool and db.serverStatus() enhancements

Query language improvements

$all with regex
$not
partial matching of array elements $elemMatch
$ operator for updating arrays
$addToSet
$unset
$pull supports object matching
$set with array indices

Geo

2d geospatial search
geo $center and $box searches

Downloads: www.mongodb.org/display/DOCS/Downloads

Full Change Log: jira

Release Notes: http://www.mongodb.org/display/DOCS/1.4+Release+Notes

Thanks for all your continued support, and we hope MongoDB 1.4 works great for you.

As always, please let us know of any issues,

-Eliot and the MongoDB Team

MongoDB 1.4 Performance

Thu, 25 Mar 2010 13:21:42 -0400

We generally avoid posting benchmarks and suggest people create their own targeting their use cases. However, we have decided to publish a few of our internal micro-benchmarks comparing 1.2 with 1.4RC2 (aka 1.3.5) to show that in almost all cases performance is the same or better (sometime significantly so), even though we’ve added many new features.

The test works by spawning N threads and having them hammer the DB with one operation as fast as they can. It is probably best to ignore the raw numbers and only look at the relative performance. In particular, these numbers shouldn’t be compared against other databases.

Results

Code for benchmarks

A few highlights (and one lowlight):

Single threaded query performance increased slightly
Query performance increases linearly or super-linearly as more cores are used
Insert performance increases by 10-30% vs 1.2
It is usually faster to create an index after importing your data than before. More so in 1.4.
Not shown, but performance for both 1.2 and 1.4 held steady from 10 threads up to at least 500 threads. Even where it looks like the lines will cross, they do not.
Update and Upsert performance is the same or higher until using more than 4 hammering threads. In practice, this shouldn’t effect users unless they are already doing more updates per second than the server can handle. Even so, we will look into solutions to this in the 1.5.x series.

This test was run on an Intel Core i7 Quad 860 2.8GHz with 8GB of RAM and an Intel G2 SSD. Even though they improved performance, HyperThreading and TurboBoost were disabled as they can skew results in nondeterministic ways.

MongoDB Day Austin coming up on Saturday, March 27

Tue, 23 Mar 2010 16:00:00 -0400

If you’re in the Austin area on March 27, you won’t want to miss MongoDB Day at Cospace in Austin. MongoDB Day Austin will be hosted by GeekAustin, a Slashdot style news site, and sponsored by 10gen.

This conference will have something for anyone interested in using MongoDB, from introductory sessions to more advanced discussions on sharding and MapReduce. Presenters at the conference will include Four Kitchens co-founder David Strauss and 10gen’s Mathias Stearn.

Tickets are running out fast! Visit http://mongodbday.eventbrite.com/ to learn more and register.

Are you going to Structure?

Fri, 19 Mar 2010 14:00:00 -0400

GigaOm’s Structure is one of the most interesting conferences in the Bay Area this year. In 2010, we’re excited that Eliot Horowitz from 10gen / MongoDB will be speaking. GigaOm, who were media sponsors at the recently completed NoSQL Live event in Boston, has provided a special discount code for friends of MongoDB to register for the conference at a $100 savings. Hope to see you there!

Details, including the discount code from our partner GigaOm:

GigaOM’s Structure conference is back for 2010! Get your ticket now!

Your $100 discount on this year’s conference is available now!

http://structure2010.eventbrite.com/?discount=NOSQL100

GigaOM’s flagship conference, Structure, returns on June 23rd and 24th for two days of deep insight on the Cloud Computing industry. Taking place at the Mission Bay Conference Center in San Francisco, Structure 2010 promises to be our best Structure conference yet.

**Save the dates**

June 23rd and 24th

Mission Bay Conference Center, San Francisco  http://events.gigaom.com/structure/10/

The $1.4 trillion IT market is undergoing a massive shake-up due to Cloud Computing. From the chips that power the compute clouds to the broadband that transports the computation and the software that ties it all together, Cloud Computing is creating a fundamental shift in how we think about and buy computing services. And at each point in the chain, such disruption creates another opportunity. At Structure 2010 you will learn about those opportunities and how to profit from them.   Our speaker list is growing every day. Confirmed speakers include:

Erich Clementi - VP, Strategy & General Manager, Enterprise Initiatives, IBM

Marc Benioff - Chairman and CEO, salesforce.com

Werner Vogels - CTO, Amazon.com

Dr. Amr Awadallah - CTO and co-founder, Cloudera

Eliot Horowitz – CTO and co-founder, 10gen / MongoDB

Nick McKeown - Professor, Stanford University

William Forrest - Principal, McKinsey

For the most up-to-date list, see our web site:  http://events.gigaom.com/structure/10/

When you attend Structure 2010 you will learn:

Why Cloud Computing is important — The scenarios in which it reduces cost, improves collaboration, speeds the real-time enterprise and increases enterprise agility.

Why new computing architectures are needed to support Cloud Computing and what they are — Hint: It’s not what’s currently in your data center.

Why Big Data means Big Problems — How do you make sense of exascale data in a timely and cost-effective manner? What new opportunities exist to improve this?

Why we might need to re-invent Internet technologies — The Internet is now asked to transport vast chunks of computation rather than small pieces of text, as it was designed to do.

What impacts “Real Time” has on the cloud — What extra considerations does real-time business infrastructure require?

…and much, much more. Check out the full schedule on our web site.   So join us on June 23rd and June 24th in San Francisco to be part of the discussion at the Cloud Computing industry’s premier conference: Structure 2010.  Register now and save $100 off the early-bird ticket price!   http://structure2010.eventbrite.com/?discount=NOSQL100

Structure 2010 also represents a great way to directly address one of the most influential tech audiences anywhere. Call Mike Sly at (415) 235-0358 to find out how your company can exhibit.