The MongoDB NoSQL Database Blog

MongoDB Driver Releases: April

Tue, 08 May 2012 01:01:38 -0400

We’ve had a big month with updates and improvements to our drivers. Here’s a summary:

We released v1.0 of the Mongo-Hadoop connector, which provides working input and output adapters for MongoDB on Hadoop’s MapReduce.
We released Node.js Driver v1.0.
Aaron Heckmann made a bunch of bug fixes and feature updates to Mongoose 2.6.x
The 1.4 release of the C# Driver now supports LINQ queries.
The 1.6.2 release of the Ruby driver now implements socket timeout using non-blocking IO. This change improve performance when timeouts are enabled.
There are a number of bug fixes in the PHP driver’s 1.2.10 release
The PyMongo team pushed PyMongo 2.2, which has support for Python 3, Gevent and improved pooling.

MongoDB at Craigslist: 1 Year Later

Thu, 03 May 2012 13:38:00 -0400

Update: watch the video of Jeremy Zawodny and Chris Mooney’s talk on A Year of MongoDB at Craigslist at MongoSF ‘12

Last year, Craigslist moved their archive to MongoDB from MySQL. After the initial set up, we spoke with Jeremy Zawodny, software engineer at Craigslist and the author of High Performance MySQL (O’Reilly), and asked him some questions about their cluster. In advance of their talk at MongoSF tomorrow, we caught up with Jeremy to get the scoop on what’s happening at Craigslist one year later.

Last time we spoke you were building a MongoDB store for 5 Billion Documents. What do your numbers look like now?

We’re currently approaching the 3 billion mark. The 5 billion number was our target capacity when building the system. Back then we had about 2.5 billion documents that we migrated into MongoDB, and we’ve continued to add documents ever since then.

Can you share an anecdote on the benefits of replica sets/sharding and something you’d like to change/improve in that feature set?

The sharding has made it easy for handling growth. We know that when the day comes, we can add an additional replica set to our cluster and it will help ease any space crunch. The replica sets have been great for handling machine failures. We’ve had several machines lock-up on us and require unplanned reboots or service. Throughout that time, the worst thing we’ve seen is some read-only time for the cluster metadata (when a config server dropped) but we’ve been able to serve requests without stopping.

Can you share some anecdotes about how your team adjusted to working with MongoDB?

There was a bit of adjustment that our systems administration team performed to the original deployment and configuration to make it better mesh with our home-grown management and deployment tools. But other than that, MongoDB has been pretty hands-off for most of the team. As long as it behaves well (which it does), we don’t need to touch it that often.

Any exciting plans for your MongoDB clusters?

We’ve been testing MongoDB in a few new roles at Craigslist and plan to present some of those challenges at MongoSF on May 4th.

Thanks to Jeremy for giving us some insight into how MongoDB powers Craigslist!

MongoDB: Powering the Magic and the Monsters at Stripe

Wed, 02 May 2012 18:41:00 -0400

Update: Watch the video of Greg Brockman’s talk on MongoDB for High Availability at MongoSF ‘12

Stripe offers a simple platform for developers to accept online payments. They are a long-time user of MongoDB and have built a powerful and flexible system for enabling transactions on the web. In advance of their talk at MongoSF on MongoDB for high availability, Stripe’s engineer, Greg Brockman spoke with us about what’s going on with MongoDB at Stripe.

Stripe has a heavy write load with large query volumes. Can you give us some insight into your tips and tricks for wrangling with MongoDB’s replica sets on your system?

Getting replica sets up and running is actually incredibly easy. I used to run MySQL clusters where configuring and maintaining replication was a pain, and it was a joy to just be able to run “rs.add(node)” and watch it join the cluster.

In order to avoid losing any operations even if we lose our database primary, we structure our application such that all writes are idempotent. We then wrap our calls to the MongoDB driver in a retry block. If the call fails because our MongoDB cluster is currently reconfiguring, we try the operation again (with the usual backoff and timeout you’d expect from a scheme like this). We’re very careful to avoid operations which could result in evicting hot data from the cache. Running unindexed queries is an obvious example of this, but we’ve also found that running a large multi-update can have production impact.

So when we need to change our schema for an entire collection of documents, we’ll usually run a slower (but non-impactful) document-by-document migration at the application level.

Let’s take a step back to your past talk at MongoSV ‘11 — what are you doing with Monster (Stripe’s native events processing system for payments)?

Monster is our framework for event production and event consumption, which uses MongoDB as a highly-available, persistent queue. With Monster, our engineers can start logging a new type of event with only a few lines of code, and at any time in the future can add a consumer that will automatically be passed relevant events (possibly even historical ones). We use it for a variety of purposes: structured logging, incremental updating of state (such as people’s graphs of payment volume), and background jobs.

Lots of people are innovating in the financial space — in particular building APIs for mobile payments. For those just starting up, why should they use MongoDB?

As a payments processor, our uptime is incredibly important. We were initially drawn to MongoDB because replica sets make it incredibly easy to run your database in a highly-available fashion. I came from a world where my database master could never be rebooted, since there was no zero-downtime failover strategy even for routine maintenance — MongoDB gives you this almost out of the box.

MongoDB also makes it easy to do zero-downtime migrations, with features such as background index builds and allowing multiple schemas in a single collection. Anyone caring about their availability should look very hard at MongoDB.

How are you guys using the Ruby driver in your system? Anything interesting?

We’ve banged on the Ruby driver in a variety of configurations, ensuring that it behaves properly when exposed to all the possible failures we can imagine (or have noticed) our database servers experiencing. These days, we’re very happy with how robust the Ruby driver is against the wide variety of failure modes of the distributed MongoDB nodes.

What’s your wish list for the Ruby Driver?

I wish there were a configuration option for forcing reads from a secondary. (Right now, you can request that reads be on a secondary if one is available, but they’ll start reading from the primary if no secondary is available.)

What’s on stripe’s engineering roadmap?

While making Stripe available outside the US is our top priority, our biggest engineering challenges at the moment are scaling our systems to keep up with the phenomenal growth we’ve been experiencing.

Many thanks to Greg for taking the time to tell a bit about the magic at Stripe.

Revamp of MongoDB's Documentation

Tue, 01 May 2012 12:59:19 -0400

We’re revamping MongoDB’s documentation. The new design in the MongoDB Manual has an improved reference section and an index for simplified search. It will also eventually support multiple MongoDB versions at the same time.

This project is a work in progress, and things are changing quickly. Our goal is to consolidate, sharpen, organize, and continue to improve the documentation in support of MongoDB. For now, the new docs will live alongside the original MongoDB Wiki. But over the next few months, we’ll be transitioning everything to the new manual.

In the spirit of open source, the docs are housed on Github. Feedback is welcome! Feel free to fork the repository and issue pull requests. You can also open tickets in JIRA, and we’ll promptly address any suggestions.

Meet Variety, a Schema Analyzer for MongoDB

Fri, 27 Apr 2012 14:46:00 -0400

Variety is a lightweight tool which gives a feel for an application’s schema, as well as any schema outliers. It is particularly useful for

• quickly learning how data is structured, if inheriting a codebase with a production data dump

• finding all rare keys in a given collection

An Easy Example

We’ll make a collection, within the MongoDB shell:

db.users.insert({name: "Tom", bio: "A nice guy.", pets: ["monkey", "fish"], someWeirdLegacyKey: "I like Ike!"});
db.users.insert({name: "Dick", bio: "I swordfight."}); 
db.users.insert({name: "Harry", pets: "egret"});
db.users.insert({name: "Geneviève", bio: "Ça va?"});

Let’s use Variety on this collection, and see what it can tell us:

$ mongo test --eval "var collection = 'users'" variety.js

The above is executed from terminal.”test” is the database containing the collection we are analyzing.

Variety’s output:

{ "_id" : { "key" : "_id" }, "value" : { "types" : [ "object" ] }, "totalOccurrences" : 4, "percentContaining" : 100 }
{ "_id" : { "key" : "name" }, "value" : { "types" : [ "string" ] }, "totalOccurrences" : 4, "percentContaining" : 100 }
{ "_id" : { "key" : "bio" }, "value" : { "types" : [ "string" ] }, "totalOccurrences" : 3, "percentContaining" : 75 }
{ "_id" : { "key" : "pets" }, "value" : { "types" : [ "string", "array" ] }, "totalOccurrences" : 2, "percentContaining" : 50 }

{ "_id" : { "key" : "someWeirdLegacyKey" }, "value" : { "type" : "string" }, "totalOccurrences" : 1, "percentContaining" : 25 }

Every document in the “users” collection has a “name” and “_id”. Most, but not all have a “bio”. Interestingly, it looks like “pets” can be either an array or a string. The application code really only expects arrays of pets. Have we discovered a bug, or a remnant of a previous schema?

The first document created has a weird legacy key I’ve never seen before- the people who built the prototype didn’t clean up after themselves. These rare keys, whose contents are never used, have a strong potential to confuse developers, and could be removed once we verify our findings.

For future use, results are also stored a varietyResults database.

Learn More!

Learn more about Variety now, including

• How to download Variety

• How to set a limit on the number of documents analyzed from a collection

• How to contribute, and report issues

Variety is free, open source, and written in 100% JavaScript. Check it out on Github.

-by James Cropcho

MongoDB and Node.js at 10gen

Wed, 25 Apr 2012 09:21:00 -0400

With their strong roots in JavaScript, Node.js and MongoDB have always been a natural fit, and the Node.js community has embraced MongoDB with a number of open source projects. To support the community’s efforts, 10gen is happy to announce that the MongoDB Node.js driver will join the existing set of 12 officially supported drivers for MongoDB.

The Node.js driver was born out of necessity. Christian Kvalheim started using Node.js in early 2010. He had heard good things about MongoDB but was disappointed to discover that no native driver had yet been developed. So, he got to work. Over the past two years, Christian has done amazing work in his driver, and it has matured through the contributions of a large community and the rigors of production. For some time now, the driver has been on par with 10gen’s officially supported MongoDB drivers. So we were naturally thrilled to welcome Christian full time at 10gen to continue his work on the Node.js driver.

If you’ve used MongoDB with Node.js, you’re probably familiar with Mongoose, a popular object-document mapper that’s used atop the Node.js driver. We’re really excited to announce that Aaron Heckmann, one of the authors of Mongoose, has also joined the 10gen team.

10gen is fully committed to making MongoDB a first class citizen in the Node.js ecosystem. We’ve partnered with Node.js advocates Joyent and Microsoft. We’ve seen fantastic companies built on Node.js and MongoDB like Learnboost, Ordr.in, and Trello. And with Aaron and Christian on board, we see a very promising future for developers wanting to use MongoDB with Node.js.

Curious about the driver and how it works? Check out the documentation and watch some sessions on MongoDB’s integration with Node.js.

Github
Node.js Language Center
MongoDB and Node.js Presentations

Interested in working on MongoDB and Node.js? Drop us a line. We continue to look for other great engineers to join us in this effort.

link to the press release

Grails in the Land of MongoDB

Wed, 29 Feb 2012 16:31:00 -0500

Groovy and Grails’ speed and simplicity are a perfect match to the flexibility and power of MongoDB. Dozens of plugins and libraries connect these two together, making it a breeze to get Grooving with MongoDB.

Using Grails with MongoDB

For the purpose of this post, let’s pretend we’re writing a hospital application that uses the following domain class.

class Doctor { 
  String first 
  String last 
  String degree 
  String specialty 
}

There are a few grails plugins that help communicate with MongoDB, but one of the easiest to use is the one created by Graeme Rocher himself (Grails project lead). The MongoDB GORM plugin allows you to persist all your domain classes in MongoDB. To use it, first remove any unneeded persistance-related plugins after you’ve executed the ‘grails create-app’ command, and install the MongoDB GORM plugin.

Using MongoDB GORM for Persistence

Step 1: Remove the hibernate dependency from the BuildConfig.groovy file.
Step 2:Install the MongoDB GORM

plugin grails install-plugin mongodb

Step 3:Edit DataSource.groovy to specify connection settings. You can entirely remove the content of this configuration file and replace it with the following, or you can specify different connection settings for each of your environments (dev, test, prod):

grails { 
  mongo { 
    host = “localhost” 
    port = 27107 
    username = ”user” 
    password=”secretpassword” 
    databaseName = “physicians” 
  } 
}

And that’s it! You can continue building your Grails application as usual by generating your controllers, views, services, etc.

GORM and Mongo

Although GORM was specifically created to persist and manipulate objects in a ‘relational’ way, you can still use some of its powerful features with MongoDB. These include:

Basic GORM:

Doctor.list() 
Doctor.get(5) 
Doctor.getAll(5,6,7) 
Doctor.list(max:5,sort:'first',order:'desc')
Doctor.listOrderBySpecialty()

Dynamic Finders:

Doctor.findByLast(“House”) 
Doctor.findAllBySpecialtyAndDegree(“Pediatrics”,”MD”)

Criteria Queries:

 Doctor.withCriteria{ eq("specialty", "Pediatrics") }

Projections:

Doctor.withCriteria{ projections {countDistinct('specialty')} }

Query by Example:

def sampledoc = new Doctor(specialty:"Pediatrics") 
def docs = Doctor.findAll(sampledoc)

Some of the GORM features that are not supported include:

Criteria queries on associations
HQL
Groovy SQL

The Schemaless Universe in a GORM World

This is where it gets exciting: with MongoDB, you can make domain objects do things that are impossible with a relational database. For example, you can now programmatically add properties on the fly based on your run time needs.

Dynamic Attributes

Let’s suppose that you want to add a ‘city’ property to your domain object. You can easily accomplish this without having to modify the original domain class using the subscript operator:

def doc = new Doctor(first:”Julius”,last:”Hibbert”, degree:”MD”,specialty:”General Medicine”) 

// Add a city dynamically to this doctor 
doc["city"]="Springfield" 

// Saving it creates an instance of Doctor with the new field 
doc.save()

Dynamic Finders even work with dynamically added fields:

def d = Doctor.findByCity(“Springfield”)

Grails Identifiers and MongoDB Identifiers

A word of caution about ID generation. Grails automatically assigns an ID as an integer value. When a Doctor object is saved, the plugin creates an _id within the document as follows:

{ "_id" : NumberLong(1),...}

However, a normal insert in MongoDB, creates it as a BSON ObjectId:

{ "_id" : ObjectId("4f2ca36b25f39d121957cebb"),...}

An extra collection is also created, doctor.next_id, to keep track of the next id.

To make ID generation compatible with MongoDB, you need to declare your ID explicitly as an ObjectID in your domain class:

import org.bson.types.ObjectId
class Doctor {
     ObjectId id
     …
}

Groovy and MongoDB

You can use Groovy and its scripting and metaprogramming powers with the help of Gmongo.

Gmongo, written by Paulo Poiati, is a convenient wrapper around the MongoDB Java driver. Using Gmongo in your Groovy programs is the closest thing to writing commands directly into Mongo’s interactive shell.

In version 0.9.3, Gmongo can be instantiated by calling it through Grapes:

@Grab(group='com.gmongo', module='gmongo', version='0.9.3') 
import com.gmongo.GMongo 

// Instantiate gmongo.GMongo object 
// It defaults to localhost, but you can pass connection settings through the constructor 
def mongo = new GMongo() 

// Get a reference to the db
def db = mongo.getDB("physicianfinder")

Now you have a database object, which you can attach to your collection to run MongoDB commands. Here’s an example of the syntax:

// [MongoDB database(usually just db)].[name of your collection].[command] 

// The following example returns the number of documents in a collection
Integer howmanydoctors = db.doctor.count()

// Finding documents with Gmongo
// Returns a com.mongodb.DBCursor class
def doctors = db.physician.find() 

// output each document to screen 
doctors.each { 
    println it 
} 

// Finds and returns one doctor
db.doctor.findOne()

// Find all doctors with last name 'Wu' 
db.doctor.find(last:"Wu") 

// Return last and specialty fields only
db.hospitals.find([:],[last:1,specialty:1]) 

// Using a regex, find all docs whose last name starts with 'H'
db.doctor.find([last: ~/^H/]) 

// Find and sort by specialty 
db.doctor.find().sort([specialty:1]) 

// Skip by 10, limit results by 5
db.doctor.find(specialty:”Pediatrics”).skip(10).limit(5)

Saving data to MongoDB

Writing documents to MongoDB with Groovy is again very similar to how it’s done through the shell. You can also pass multiple types of collections to the insert method and have it magically saved correctly as a document.

 // Inserting a Doctor with two fields
db.doctor.insert([first:"Charles",last:"Darwin"]) 

// Specifying write concern
db.doctor.insert([first:"Gregory",last:"House"], com.mongodb.WriteConcern.NORMAL)

// Inserting nested documents: 
def hospital = [name:"My City Big Hospital", city:"Chicago", 
                buildings:[[address:"123 Main St.", name:"Feinberg",num_of_beds:300], 
                           [address:"345 Hope Ln.", name:"Galter",num_of_beds:100]]] 

// Saves the map of maps as a nested document
db.hospitals.insert(hospital)

// The following document is then created in the hospitals collection: 
{ "_id" : ObjectId("4f2c98d53b7e58ee6ea885d5"), 
  "name" : "My City Big Hospital", 
  "city" : "Chicago", 
   "buildings" : [ { "address" : "123 Main St.", "name" : "Feinberg", "num_of_beds" : 300 }, 
                   { "address" : "345 Hope Ln.", "name" : "Galter", "num_of_beds" : 100 } ] }

You can also use Groovy builders to generate your documents.

 // Create a doctor with a JSON builder 
def builder = new groovy.json.JsonBuilder() 
builder.physicians { 
  doctor { first 'Julius' last 'Hibbert' 
          
           // Named arguments are valid values for objects too   
           address( city: 'Springield', 
                    country: 'USA', 
                    zip: 02105, ) 
           medicalschool: "Johns Hopkins University School of Medicine" 
           mensamember true 
           practice 'Dr. Julius Hibbert, M.D. Family Practice' } 
          } 

// Parse JSON string into a com.mongodb.util.JSON object 
def parsed = com.mongodb.util.JSON.parse(builder.toString()) 

// Save to Mongo 
db.physicians.insert(parsed)

Updating and removing are also straightforward:

// Look for 'Darwin' and update first name to 'Charles'
db.doctor.update([last:"Darwin"],[$set:[first:"Charles"]]) 

// Delete all 'Darwins' 
db.doctor.remove(last:"Darwin")

If instead of collections you want to persist Plain Old Groovy Objects(POGOs), you can. Let’s start with our Doctor object from earlier:

class Doctor { 
     def first 
     def last 
     def specialty }

You can insert an instance of it by doing the following:

def doc = new Doctor(first:"Richard",last:"Kimble",specialty:"Pediatrics")

The doc instance also contains properties that we do not need to save into the document, so we remove the class and metaClass property before inserting:

 // Get only the properties we want to save 
def doctor = doc.properties.findAll { !['class', 'metaClass'].contains(it.key) }

// Insert it into the doctor collection
db.doctor.insert(doctor)

And we can retrieve it back as a POGO as well:

 // Ignore the _id property 
Doctor docFromMongo = db.doctor.findOne(last:"Kimble").findAll { it.key != '_id' }

And if you don’t have a POGO declared, you can use the magic of the Expando class to retrieve any document into a Groovy class:

Expando docFromMongo = db.doctor.findOne(last:"Kimble")

We’ve only covered the basics of interacting with MongoDB, but there are many more features you can leverage for making the most of the NoSQL world. To get started, check out the MongoDB GORM plugin.

— Ariel Gamino

Operations in the New Aggregation Framework

Tue, 17 Jan 2012 14:04:00 -0500

Available in 2.1 development release. Will be stable for production in the 2.2 release

Built by Chris Westin (@cwestin63)

MongoDB has built-in MapReduce functionality that can be used for complex analytics tasks. However, we’ve found that most of the time, users need the kind of group-by functionality that SQL implementations have. This can be implemented using map/reduce, but doing so is more work than it was in SQL. In version 2.1, MongoDB is introducing a new aggregation framework that will make it much easier to obtain the kind of results SQL group-by is used for, without having to write custom JavaScript.

MongoDB provides an easy way to store documents that aren’t just flat records like SQL, but that also contain arrays and nested documents. Another common problem users have told us about is that it can be difficult to manipulate these sub-parts of documents easily. The new aggregation mechanism is also taking on making this easier, by providing operators that can be used to disassemble and re-assemble document sub-parts.

After many requests, we have also included expression evaluation to return computed values—for virtual fields.

The new aggregation framework is declarative. Rather than writing code to aggregate data, you specify a chain of operations, called a “pipeline,” to apply to your documents. This is similar to a pipe in a unix shell.

We’re now going to show you a few short examples (using the mongo shell) of this new functionality to get you started.

$match is a pipeline operator that filters out documents that don’t match the specified condition. For example

db.runCommand({ aggregate : "article", pipeline : [ { $match : { author : "dave" } } ]});

This very simple one step pipeline doesn’t introduce any new functionality yet, but shows how to put a query at the beginning of your pipeline. For those of you familiar with MongoDB, this is the equivalient of

> db.article.find({ author : "dave" });

$project is a pipeline operator that let’s you select which fields you want to include or exclude from a result. You can also create new virtual fields from computed values. Here is a simple example, building on the previous one:

db.runCommand({ aggregate : "article", pipeline : [ { $match : { author : "dave" } }, { $project : { _id : 0, author : 1, tags : 1 }} ]});

As before, this doesn’t introduce anything radically new; this is the equivalent of

> db.article.find({ author : "dave" }, { _id : 0, author : 1, tags : 1);

While this functionality was available before from find(), what it does demonstrate here is how to put together more than one aggregation pipeline operator. The value of pipeline is an array consisting of the operators to apply. Conceptually, the aggregation framework behaves as if it was feeding the result of scanning the target collection through that pipeline.

Now let’s take a look at some new functionality that wasn’t available before.

$unwind hands out the elements of an array one at a time; each element is surrounded by the document that contains the original array. For this example, we’ll first show what a document from the article collection looks like. Suppose we had saved this:

db.article.save( { title : "this is your title" , author : "dave" , posted : new Date(4121381470000) , pageViews : 7 , tags : [ "fun" , "nasty" ] , comments : [ { author :"barbara" , text : "this is interesting" } , { author :"jenny" , text : "i like to play pinball", votes: 10 } ], other : { bar : 14 } });

Now, we can unwind the tags array as follows:

db.runCommand({ aggregate : "article", pipeline : [ { $unwind : "$tags" } ]});

The result looks like this:

{ "result" : [ { "_id" : ObjectId("4eeeb5fef09a7c9170df094b"), "title" : "this is your title", "author" : "dave", "posted" : ISODate("2100-08-08T04:11:10Z"), "pageViews" : 7, "tags" : "fun", "comments" : [ { "author" : "barbara", "text" : "this is interesting" }, { "author" : "jenny", "text" : "i like to play pinball", "votes" : 10 } ], "other" : { "bar" : 14 } }, { "_id" : ObjectId("4eeeb5fef09a7c9170df094b"), "title" : "this is your title", "author" : "dave", "posted" : ISODate("2100-08-08T04:11:10Z"), "pageViews" : 7, "tags" : "nasty", "comments" : [ { "author" : "barbara", "text" : "this is interesting" }, { "author" : "jenny", "text" : "i like to play pinball", "votes" : 10 } ], "other" : { "bar" : 14 } } ], "ok" : 1 }

Notice that the single document has been duplicated (see the identical _id values). However, the tags array in each copy of the document has been replaced with one value from the original tagsarray.

$unwind isn’t useful on its own, but is useful when combined with $match, which can be used to filter out copies of the document without values of interest, and then the array can be reassembled using other operators such as $push or $addToSet. $unwind is also useful when combined with $group.

$group groups elements with a common key together, and allows the application of aggregation functions. For the purposes of a small example, let’s add a second article to our collection:

db.article.save( { title : "this is some other title" , author : "jane" , posted : new Date(978239834000) , pageViews : 6 , tags : [ "nasty" , "filthy" ] , comments : [ { author :"will" , text : "i don't like the color" } , { author :"jenny" , text : "can i get that in green?" } ], other : { bar : 14 } });

If these articles represent content on a web site, I might want to know what tags users are applying to these articles. I can use $group to find that out like so:

db.runCommand({ aggregate : "article", pipeline : [ { $unwind : "$tags" }, { $group : { _id : "$tags", count : { $sum : 1 }, authors : { $addToSet : "$author" } }} ]});

The grouping key for a $group is defined by specifying a field (or subdocument of multiple fields) for the _id. The grouping key is used to define buckets into which documents are put. Other fields are aggregate functions that can compute values across the entries in each bucket.

In this case, one is added to count for each document that matches the group key. That counts up the number of items in each bucket. authors will be an array, and each unique author in the bucket will be added to it.

The result of all this is

{ "result" : [ { "_id" : "filthy", "count" : 1, "authors" : [ "jane" ] }, { "_id" : "fun", "count" : 1, "authors" : [ "dave" ] }, { "_id" : "nasty", "count" : 2, "authors" : [ "jane", "dave" ] } ], "ok" : 1 }

One document has the “filthy” tag, and it was written by jane. One document has the “fun” tag, and it was written by dave. Two documents have the “nasty” tag, and both jane and dave wrote documents with that tag.

The new aggregation functionality is available only by compiling from source. It is available in the 2.1 development release but is not meant for production. MongoDB 2.2 will be the first stable release with the aggregation framework. For more examples and information, check out the talk and demo from MongoSV 2011, or the documentation on the MongoDB wiki.

by Chris Westin

MongoSV Recap

Fri, 16 Dec 2011 14:57:00 -0500

Last week over 1,100 developers came together for MongoSV, the largest MongoDB conference to date. 10gen kicked off MongoSV with our inaugural MongoDB Masters program, which brought together MongoDB evangelists from around the world.

At the opening keynote, 10gen CTO Eliot Horowitz demoed a twitter app for #mongoSV tweets, featuring the new aggregation framework expected for the MongoDB 2.2 release. These gather all the tweets sent out with the hashtag #mongoSV and organizes them in by recency and most retweets. Get the source code for the demo app here

Highlights from MongoSV include presentations on X.commerce’s new open source developer platform, MongoDB’s integration with Azure, MongoDB’s new aggregation framework, How Disney manages their deployment of 1400 Mongo instances and more

the 10gen booth at MongoSV

10gen President Max Schireson welcomes the Speakers and Masters to MongoSV

MongoDB On Microsoft Azure

Thu, 01 Dec 2011 12:54:00 -0500

A new preview release of the MongoDB controller for Azure is available. This release includes support for replica sets, and over the coming months, we’ll be adding support for MongoDB’s sharding facilities. We’ll also be working to more tightly integrate MongoDB with the features of Azure platform. Each member of a replica set is hosted by an instance of an Azure worker role, so the size of the replica set is determined by the number of instances configured for the the replica set worker roles. Each replica set worker role creates a child process to run the mongod server process. The controller defines an Azure worker role which represents a MongoDB cluster. Currently the cluster consists of a single MongoDB replica set. Each role instance participates as a member of the replica set. On deployment the controller configures the replica set and initiates it. Role instances handle restarts, including assuming identity for a replica set member and replacing old instances. The worker role also integrates with Azure Diagnostics to capture trace and performance counter information. The MongoDB Azure wrapper release is currently delivered as a Visual Studio 2010 solution with associated source files, and we’ll follow that up in the future with packaging and deployment tools. To get started, take a look at the documentation or browse through the source. We’ve included a small sample application to get you up and running quickly. If you have questions, head over to our user mailing list or file a bug report on our JIRA system. Microsoft will be presenting at the MongoSV conference in Santa Clara on December 9th. It will be a great opportunity to understand the solution in depth, as well as understand the future direction of this project. We hope to see you there.

MongoDB Monitoring Service Docs Available

Thu, 03 Nov 2011 14:53:48 -0400

MongoDB Monitoring Service (MMS) documentation now available: http://mms.10gen.com/help/

Mongo Boston Recap

Tue, 11 Oct 2011 13:42:56 -0400

Last week 250 developers converged at the Microsoft New England Research and Design Center for Mongo Boston. Highlights from the event include presentations on MongoDB 2.0, how MTV leverages MongoDB for CMS, rapid prototyping, and more.

More photos from the event are available on the MongoDB Flickr page.

If you missed the event, join the Boston MongoDB User Group, which also meets at NERD. The next meeting is on November 15.

2.0 Presentation at New York MongoDB User Group

Tue, 20 Sep 2011 15:46:00 -0400

On Thursday MongoDB core committer Eliot Horowitz presented to the New York MongoDB User Group on the latest features in v2.0. The event was hosted by Sailthru, a MongoDB-powered startup doing intelligent email marketing. The meetup was announced Monday night and within a day was oversubscribed. After the presentation, we all went out for drinks to celebrate the release.

The NY MUG has over 1,000 members and meets monthly. There are also MongoDB user groups in San Francisco, Washington DC, London, Boston, Japan, and more. And if there isn’t a MongoDB meetup in your city, we’re happy to support you if you would like to start one.

In case you missed the meetup, video and photos are posted below. Enjoy!

Cache Reheating - Not to be Ignored

Mon, 19 Sep 2011 13:56:00 -0400

An important aspect to keep in mind with databases is the cost of cache reheating after a server restart. Consider the following diagram which shows several cache servers (e.g., memcached) in front of a database server.

This sort of setup is common and can work quite well when appropriate; it removes read load from the database and allows more RAM to be utilized for scaling (when the database doesn’t scale horizontally). But what happens if all the cache servers restart at the same time, say, on a power glitch in a data center?

We then have a cache reheating scenario. After the bounce the full load of all read requests will hit the database server (for a while) given the caches are empty. The server won’t be able to handle it (that’s why the cache servers were there in the first place). Now if the reheat time is short, this is not a big problem : we did go down after all. But if reheat takes a long time, it’s a big problem. Imagine 20 cache servers with 64GB RAM each. 1.2 terabytes of data must be queried from the database to be fully reheated!

Even without a cache server, the same issue exists for almost all databases. Imagine a server restart on a system with 64GB RAM. If loading 100MB/sec, reheat will take 10 minutes. However, if the queries are coming in randomly, it could take much much longer — only with sequential I/O can we get anywhere near that speed.

One could just sequentially read data in and fill the cache. However for databases much larger than RAM, loading in the right portion is difficult. “Hot” data could be at very different locations in the terabyte(s) of on disk data.

A very nice attribute of the MongoDB storage engine is its use of memory-mapped files. In this model the cache is the operating system’s file system cache. Restart the mongod process, and there is no reheat issue at all. Some databases use their own page cache which causes a reheat scenario even on just a process restart. Of course on a full server reboot MongoDB must reheat too.

A few points to keep in mind:

Think about reheating and how you will operationally handle it if a scenario involving it occurs. For schedules OS maintenance restart the server during off peak hours to minimize the load during reheat.
Restarting the mongod process is safe with respect to reheating.
Remounting a volume likely loses all file system cache info for the volume.
On a server restart, copy datafiles to /dev/null to force reheating to be sequential and thus much faster. This can be done even if the mongod process is already running. If the database is larger than RAM, copy only the newest datafiles (ones with the highest numbers); while this isn’t perfect, the latest files likely contain the largest percentage of frequently used data.

MongoDB 2.0 Released

Mon, 12 Sep 2011 11:32:00 -0400

The MongoDB development team is pleased to announce the release of version 2.0.0. Version 2.0 is the latest stable release, following the March 2011 release of version 1.8. This release includes many new features, improvements to existing features, and performance enhancements.

Please note version 2.0 is a significant new release, but is 2.0 solely because 1.8 + 0.2 = 2.0; for example the upgrade from 1.6 to 1.8 was similar in scope.

Highlights of the 2.0 release:

Concurrency improvements
Index enhancements to improve size and performance
Authentication with sharded clusters
Replica Set Data Center Awareness
Journaling is now enabled by default. Journal files are now compressed.
User defined getLastError options
Smaller default stack size to accommodate more connections
Compact Command
Geo-spatial features
- Multi-location documents
- Polygon searches
Output map/reduce results to sharded collection

Concurrency improvements in 2.0 are just the start of a much larger concurrency roadmap we are working on. In 2.0, we are beginning to address one of the biggest issues: holding locks during a page fault. 2.0 tracks memory caching and has the ability to yield locks and fault outside. This is hooked in a number places, notably: updates by _id, remove, and long table scans.

Being able to keep the working index set in memory is an important factor in overall performance, and we overhauled indexes to make this easier. Indexes in 2.0 are about 25% smaller and faster, meaning that you can fit more in memory.

Replica sets get two new important features in 2.0: priorities and tagging. Priorities let you have nodes that you prefer to be primary if you have a non homogeneous environment. Tagging lets you guarantee writes hit certain groups of servers. One use case for this is guaranteeing a new user registration is written to two data centers before acknowledging to a user.

There were many other improvements, so we encourage those interested to look at the change log. Overall, this release added a large number of small performance and concurrency improvements, greater stability to sharding, and better replica set operations.

Downloads: http://www.mongodb.org/downloads

Release Notes: http://www.mongodb.org/display/DOCS/2.0+Release+Notes

Full change log: https://jira.mongodb.org/secure/IssueNavigator.jspa?requestId=10140

For the full scoop on what’s new in MongoDB version 2.0, register for our live webinar on Thursday, September 15th. We will have two sessions: the first at 10am GMT, and another at 1:30pm ET/10:30am PT.

Thank you to the MongoDB community for your continued feedback and testing through the 1.9 development series.

-Eliot and the MongoDB Team

BSON and Data Interchange

Wed, 24 Aug 2011 09:52:00 -0400

There’s a lot of good things about JSON — it’s a standards based, language independent, representation of object-like data. Also, it’s easy to read (for users and programmers alike). Each document is only about data, not complex object graphs and links. Thus it’s easy to inspect without knowing all the code of an application.

Further, JSON is “schemaless”. We do not have to predefine our (protocol) schema. This can be quite helpful: imagine RPC’ing data from client A to server B with a fixed schema for the messages. On a schema change both need to be ‘updated’ with the new schema. If there are many components to the system it’s even more complicated of course. There is some analogy here to XML, which can (optionally) be schemaless.

It would be nice to have a binary representation of JSON. That is what BSON is all about.

So what are the goals of BSON? They are:

Fast scan-ability. For very large JSON documents, scanning can be slow. To skip a nested document or array we have to scan through the intervening field completely. In addition as we go we must count nestings of braces, brackets, and quotation marks. In BSON, the size of these elements is at the beginning of the field’s value, which makes skipping an element easy.
Easy manipulation. We want to be able to modify information within a document efficiently. For example, incrementing a field’s value from 9 to 10 in a JSON text document might require shifting all the bytes for the remainder of the document, which if the document is large could be quite costly. (Albeit this benefit is not comprehensive: adding an element to an array mid-document would result in shifting.) It’s also helpful to not need to translate the string “9” to a numeric type, increment, and then translate back.
Additional data types. JSON is potentially a great interchange format, but it would be nice to have a a few more data types. Most importantly is the addition of a “byte array” data type. This avoids any need to do base64 encoding, which is awkward.

One thing that is not a goal of BSON: compactness. The JSON document {“field”:7} represents the number seven as a single byte. That’s pretty good.

Perhaps the best example to date of usage of BSON is MongoDB. MongoDB uses it heavily — for sending documents over the network, persisting them to disk, as well as for internal data manipulations. In fact this is where BSON originated, although today it is a separate spec that should not be considered coupled to any one particular project.

There is BSON serialization and deserialization code for most languages; implementations are open source and mostly available under Apache 2 license. Quite a few implementations originated from a MongoDB drivers; work is underway in most drivers to fully decouple, although independent use works fine today.

If you or someone you know is using BSON in a project, please let us know by posting on the BSON mailing list. Check out bsonspec.org for more information.

— Spencer & Dwight

Master Detail Transactions in MongoDB

Mon, 11 Jul 2011 11:17:00 -0400

In relational databases, transactions let you make reliable atomic updates to your data. Because relational schemas are often highly normalized, most logical transactions span multiple tables, so it is important to be able to do multiple updates atomically (all or nothing).

While MongoDB does not have multi-document transactions, it makes up for this in many use cases through its document oriented data model. In this post, we’ll talk about the Master-Detail design pattern that comes up very often in data modelling that almost always requires multi-statement transactions in an RDBMS, but is easily handled without cross-statement transactions in MongoDB.

Master-Detail transactions in an RDBMS

As an example of the Master-Detail pattern, consider a Purchase Order with multiple line items. In an RDBMS, we might model this as a Purchase Order table (the Master) and a Line Item table (the Detail). To get a purchase order, I need to join Purchase Order and Line Item tables together to get all of the info in the purchase order.

I might model my Purchase Orders as follows in an RDBMS:

If I want to make atomic updates to a purchase order and its line items, I need a multi-statement transaction. For example, if I am going to create a purchase order, I might follow these steps:

With this update, there is never a time where the Purchase Order exists but has no Line Items in it. The whole object and its details are committed in a single transaction.

Now if I need to update that Purchase Order, say to add a few more line items, then I would perform another transaction.

This time I’ve ensured that my two new Line Items appear at the same time (or not at all) and that the total field of the Purchase Order is updated at the same time. No client will ever see a state in which only one of those Line Items exists nor any state where the total does not match the sum of the line items.

Master-Detail in MongoDB

Working with MongoDB is a bit different. While we don’t have the ability to perform multi-documents transactions (at least so far). However this Master-Detail pattern can be handled without multi-statement transactions thanks to MongoDB’s richer data model.

MongoDB can store data as documents with nested objects, arrays, and other rich data types. Thus we don’t need to store Master-Detail entities in multiple collections, or even in more than one document. A common way of modeling such objects with MongoDB is using nested documents. So our Purchase Order model might look like this:

Let’s look at how we can get the same level of atomicity from MongoDB without needing multi-statement transactions!

First, we want to be able to create a new purchase order and its first line items atomically.

This atomically creates the purchase order and its initial items in a single operation. Just as with the SQL scenario, clients will never see a point in time where the purchase order is empty. It all succeeds in a single step.

Now what about modifying that purchase order? If we want to add some items to the PO, we can do so like this:

The $pushAll operator atomically appends values onto an array attribute. Just as with our RDBMS scenario, this update is atomic and the whole command either succeeds or fails. Meanwhile the $inc operator atomically increments the “total” field of the purchase order. All of these updates happen atomically and they succeed or fail as a group so another client will never see an inconsistent state of the order.

Summary

It turns out that most of the time where you find yourself with a Master-Detail pattern in an RDBMS, you can achieve the same level of consistency in MongoDB by modelling your object as a rich, nested document, rather than multiple joined tables. Combine this with MongoDB’s atomic update operators, and you can solve most of what you would traditionally do with multi-statement transactions in an RDBMS.

An RDBMS needs multi-statement transactions for these scenarios because the only way it has to model these types of objects is with multiple tables. By contrast, MongoDB can go much further with single-statement transactions because there’s no need to join on simple updates like this.

This is not to say that multi-statement transactions are not useful. If you need to perform cross-entity transactions (e.g. move a line item from one purchase order to another) or if you need to modify a purchase order and inventory objects in a single step, then you may still need multi-statement transactions. Or else you would have to take some alternate approach.

But it turns out that many of the cases where we traditionally need multi-statement transactions go away when we can model objects as documents and perform atomic updates on those documents.

Another aspect of transactions and ACID is isolation. MongoDB does not support fully generalized snapshotting. We haven’t discussed that here; it’s probably a good topic for another blog post in the future.

— Jared Rosoff (@forjared)

Design of the Erlang Driver for MongoDB

Tue, 05 Jul 2011 14:27:00 -0400

Since November 2010, I have been writing an Erlang driver for MongoDB. After many months of work, I would consider the driver production-ready, and wanted to take this opportunity to introduce the driver and highlight a few of the design decisions. For detailed documentation with code examples please see the links at the end of this article.

BSON
At the highest level, the driver is divided into two library applications, mongodb and bson. BSON is defined independently of MongoDB at bsonspec.org. One design decision was how to represent BSON documents in Erlang. Conceptually, a document is a record, but unlike an Erlang record, a BSON document does not have a single type tag. Furthermore, the same MongoDB collection can hold different types of records. So I decided to represent a BSON document as a tuple with labels interleaved with values, as in {name, <<"Tony">>, city, <<"New York">>}. An alternative would have been to represent a document as a list of label-value pairs, but I wanted to reserve lists for BSON arrays.

A BSON value is one of several types. One of these types is the document type itself, making it recursive. Several value types are not primitive, like objectId and javascript, so I had to define a tagged tuple type for each of them. I defined them all to have an odd number of elements to distinguish them from a document which has an even number of elements. For example, a javascript value looks like {javascript, {x,2}, <<"x + 1">>}. Finally, to distinguish between a string and a list of integers, which is indistinguishable in Erlang, I require BSON strings to be binary (UTF-8). Therefore, a plain Erlang string is interpreted as a BSON array of integers, so make sure to always encode your strings, as in <<"hello">> or bson:utf8("hello")

Var
The mongodb driver has a couple of objects like connection and cursor that maintain mutable state. The only way to mutate state in Erlang is to have a process that maintains its own state that it updates when it receives messages from other processes. The Erlang programmer typically creates a process for each mutable object and defines the messages it may receive and the action to take for each message. They usually provide helper functions for the clients to call that hide the actual messages being sent and returned. Erlang OTP provides a small framework called gen_server to facilitate this process definition but it is still non-trivial. To alleviate this burden I created another abstraction on top of gen_server called var. A var is an object (process) that holds a value of some type A that may change. Its basic operation is modify (var(A), fun ((A) -> {A, B})) -> B which applies the function to the current value of the var then changes the var’s value to the first item of the result while returning the second item of the result to the client. This is done atomically thanks to the sequential nature of Erlang processes. The function may perform side effects (sending/receiving messages or doing IO), in which case the var acts like a mutex since only one function can execute against the var at a time. In essence, using var or even just gen_server changes the programming paradigm from message passing to protected shared state, which is more like Haskell for example.

With var it is now very easy to create objects that protect a shared resource or have internal mutable state. A TCP connection to a MongoDB server is one such resource that needs protection. The connection object in mongodb is implemented as a var holding a TCP connection. Every read and write operation gets exclusive access to the TCP connection when sending and receiving its messages to and from the server. This allows multiple user processes to read and write to the same mongodb connection concurrently.

DB Action
Every read/write operation may throw a DB exception. Furthermore, every read/write operation requires a DB context that includes the connection to use, the database to access, whether slave is ok (read_mode), and whether to confirm writes (write_mode). We group a series of read/write operations that perform a single high-level task into a function called a DB action. We then execute the action within a single exception handler and with a single DB context in dynamic scope (using Erlang’s process dictionary). mongo:do (write_mode(), read_mode(), connection(), db(), action(A)) -> {ok, A} | {failure, failure()} sets up the context, runs the action, and catches and returns any DB failure. Note, it will not catch and return other types of exceptions like programming errors.

You may notice that a DB action is analogous to a DB transaction for a relational database in that the action aborts when one of its operations fails. However, for scalability reasons, MongoDB does not support ACID across multiple read/write operations, so the operations before the failed operation remain in effect. Your failure handler must be prepared to recover from this intermediate state. If your DB action is conceptually a single high-level task, then it should not be too hard to undo and redo that task even from an intermediate state.

Documentation
Detailed documentation with examples can be found in the ReadMe’s of the two libraries, mongodb and bson, and in their source code comments and test modules.

- Tony Hannan

Getting started with VMware CloudFoundry, MongoDB and Node.js

Thu, 16 Jun 2011 10:00:00 -0400

Listen to the recording of the Node.js Panel Discussion webinar.

Overview

Following up from our previous post we’re posting up a quick how-to for using Node.JS, CloudFoundry and MongoDB together.

Our end goal here is to build a simple web app that records visits and provides a reporting screen for the last 10 visits.

Tools We Need

Sign up for a Cloud Foundry account.
Local installation of MongoDB & Node.JS.
Cloud Foundry VMC tools.
All of the code is available on github.

Follow the links to install & configure the various tools.

Getting Started

Start the mongod process on your local computer. Use the default port.
Confirm that node.js is correctly installed. You should be able to run node from the command-line and receive a basic javascript shell. You should also have NPM (node package manager) installed.
Make a directory for the project and then ensure that CloudFoundry is correctly configured. Mine looked as follows:

<pre > mongo@ubuntu:~$ sudo gem install vmc mongo@ubuntu:~$ vmc target api.cloudfoundry.com Succesfully targeted to [http://api.cloudfoundry.com] mongo@ubuntu:~$ vmc login Email: gates@10gen.com Password: ******** Successfully logged into [http://api.cloudfoundry.com]

Step 1: Hello world

In your directory create a file called app.js. In that file, type the following code. This will create a basic web server on localhost:3000 or on the assigned host:port combination on the CloudFoundry server.

var port = (process.env.VMC_APP_PORT || 3000);
var host = (process.env.VCAP_APP_HOST || 'localhost');
var http = require('http');

http.createServer(function (req, res) {
  res.writeHead(200, {'Content-Type': 'text/plain'});
  res.end('Hello World\n');
}).listen(port, host);

Test our file locally:

$ node app.js
$ curl localhost:3000
Hello World
# kill node with CTRL+C

Push our file to CloudFoundry and test. CloudFoundry automatically picks up that we’re using node.js,but it will ask some other configuration questions, including a name and the services we want to have running. I have named mine gvp_node_test and requested that MongoDB be run as a service.

The commands & output:

$ vmc push 
Would you like to deploy from the current directory? [Yn]: Y
Application Name: gvp_node_test
Application Deployed URL: 'gvp_node_test.cloudfoundry.com'? 
Detected a Node.js Application, is this correct? [Yn]: Y
Memory Reservation [Default:64M] (64M, 128M, 256M, 512M, 1G or 2G) 
Creating Application: OK
Would you like to bind any services to 'gvp_node_test'? [yN]: y
The following system services are available::
1. mongodb
2. mysql
3. redis
Please select one you wish to provision: 1
Specify the name of the service [mongodb-55516]: 
Creating Service: OK
Binding Service: OK
Uploading Application:
  Checking for available resources: OK
  Packing application: OK
  Uploading (0K): OK   
Push Status: OK
Staging Application: OK                                                         
Starting Application: OK

At this point you should have a simple working web app. Note the URL: your_app_name.cloudfoundry.com, we can test it is working with curl.

$ curl your_app_name.cloudfoundry.com
Hello World

Step 2: getting mongo configs

CloudFoundry has now configured a MongoDB service with its own user name, password, ip and port. To access these on the server, we will need to parse the environment variables coming into the node process.

To do this we do the following, note that we’ve added an else clause so that we can run this locally.

if(process.env.VCAP_SERVICES){
  var env = JSON.parse(process.env.VCAP_SERVICES);
  var mongo = env['mongodb-1.8'][0]['credentials'];
}
else{
  var mongo = {"hostname":"localhost","port":27017,"username":"",
    "password":"","name":"","db":"db"}
}

The code wraps this up in a generate_mongo_url function that provides a “connection string” of the form mongodb://username:password@host:port/db_name.

Copy in the rest of the code from step 2 on github and test locally.

$ node app.js
$ curl localhost:3000
# connection string for localhost

Once that’s working push the update to the cloud. Notice that we add the name of the project and we don’t get asked any questions:

$ vmc update your_app_name
Uploading Application:
...
Stopping Application: OK
Staging Application: OK                                                         
Starting Application: OK 
# test again
$ curl your_app_name.cloudfoundry.com
# bunch of environment variables

Step 3: now with drivers

First we need to install the node-mongodb-native driver. To do this, we use NPM.

$ npm install mongodb

You should see a new directory at the end of this process: node_modules. To enable use to include these module on the cloud we add this path to the require variable at the top of the code.

require.paths.unshift('./node_modules');

if(process.env.VCAP_SERVICES){
...

Our goal here is to build a function for recording a visit. Let’s build that function.

var record_visit = function(req, res){
  /* Connect to the DB and auth */
  require('mongodb').connect(mongourl, function(err, conn){
    conn.collection('ips', function(err, coll){
      /* Simple object to insert: ip address and date */
      object_to_insert = { 'ip': req.connection.remoteAddress, 'ts': new Date() };

      /* Insert the object then print in response */
      /* Note the _id has been created */
      coll.insert( object_to_insert, {safe:true}, function(err){
        res.writeHead(200, {'Content-Type': 'text/plain'});
        res.write(JSON.stringify(object_to_insert));
        res.end('\n');
      });
    });
  });
}

Notice the .connect and .collection('ips'...). We’re telling it to store data in the ips collection.

Another nice feature is the object_to_insert. Saving a document with Node+MongoDB is as simple as creating the object and inserting it.

Let’s fix up the main createServer function.

http.createServer(function (req, res) {
  record_visit(req, res);
}).listen(port, host);

Then we can test locally and push with vmc. If this is working locally, you should be able to connect to your local mongod instance and see some data in the ips collection.

$ mongo localhost:27017/db
> db.ips.find()
...

Step 4

At this point, you’ve probably tested a few times and you’ve successfully put data in the database. Now it’s time to get that data out.

Let’s create a function to print the last 10 visits:

var print_visits = function(req, res){
  /* Connect to the DB and auth */
  require('mongodb').connect(mongourl, function(err, conn){
    conn.collection('ips', function(err, coll){
      /*find with limit:10 & sort */
      coll.find({}, {limit:10, sort:[['_id','desc']]}, function(err, cursor){
        cursor.toArray(function(err, items){
          res.writeHead(200, {'Content-Type': 'text/plain'});
          for(i=0; i < items.length; i++){
            res.write(JSON.stringify(items[i]) + "\n");
          }
          res.end();
        });
      });
    });
  });
}

Let’s update the createServer method to print when we request /history.

http.createServer(function (req, res) {
  params = require('url').parse(req.url);
  if(params.pathname === '/history') {
    print_visits(req, res);
  }
  else{
    record_visit(req, res);
  }
}).listen(port, host);

Again, we test locally and then upload with vmc. If it all works, you should be able to do this:

$ vmc update your_app_name
...
$ curl your_app_name.cloudfoundry.com
{"ip":"172.30.49.42","ts":"2011-06-15T20:14:18.977Z","_id":"4df9129af354f8682d000001"}
$ curl your_app_name.cloudfoundry.com
{"ip":"172.30.49.43","ts":"2011-06-15T20:14:21.745Z","_id":"4df9129df354f8682d000002"}

# now let's test history
$ curl gvp_node_test.cloudfoundry.com/history
{"ip":"172.30.49.43","ts":"2011-06-15T20:14:21.745Z","_id":"4df9129df354f8682d000002"}
{"ip":"172.30.49.42","ts":"2011-06-15T20:14:18.977Z","_id":"4df9129af354f8682d000001"}
...

Going further

Check out our upcoming Node.js Panel Discussion webinar.
For some MongoDB wrappers take a look at
- Mongoose, an ORM / ODM wrapper
- MongoSkin, a layer over node-mongodb-native to help reduce callbacks.
For building more complex web sites take a look at the Express framework.

— Gates Voyer-Perrault
@gatesvp

A reminder about MongoDB "office hours"

Wed, 08 Jun 2011 11:21:00 -0400

Very casual whiteboard type chat sessions on a regular basis in SF, NYC, Redwood Shores, Mountain View, and Atlanta. Stop by!

Also check out the meetup users groups in lots of other cities.