$push to sorted array

Aug 20 • Posted 8 months ago

By Sam Weaver, MongoDB Solutions Architect and Alberto Lerner, MongoDB Kernel Lead

MongoDB 2.4 introduced a feature that many have requested for some time - the ability to create a capped array.

Capped arrays are great for any application that needs a fixed size list. For example, If you’re designing an ecommerce application with MongoDB and want to include a listing of the last 5 products viewed, you previously had to issue a $push request for each new item viewed, and then a $pop to kick the oldest item out of the array. Whilst this method was effective, it wasn’t necessarily efficient. Let’s take an example of the old way to do this:

First we would need to create a document to represent a user which contains an array to hold the last products viewed:

db.products.insert({last_viewed:["bike","cd","game","bike","book"]})
db.products.findOne()
{
    "_id" : ObjectId("51ff97d233c4f2089347cab6"),
    "last_viewed" : [
        "bike",
        "cd",
        "game",
        "bike",
        "book"
    ]
}

We can see the user has looked at a bike, cd, game, bike and book. Now if they look at a pair of ski’s we need to push ski’s into the array:

db.products.update({},{$push: {last_viewed: "skis"}})
db.products.findOne()
{
    "_id" : ObjectId("51ff97d233c4f2089347cab6"),
    "last_viewed" : [
        "bike",
        "cd",
        "game",
        "bike",
        "book",
        "skis"
    ]
}

You can see at this point we have 6 values in the array. Now we would need a separate operation to pop “bike” out:

db.products.update({},{$pop: {last_viewed: -1}})
db.products.findOne()
{
    "_id" : ObjectId("51ff97d233c4f2089347cab6"),
    "last_viewed" : [
        "cd",
        "game",
        "bike",
        "book",
        "skis"
    ]
}

In MongoDB 2.4, we combined these two operations to maintain a limit for arrays sorted by a specific field.

Using the same example document above, it is now possible to do a fixed sized array in a single update operation by using $slice:

db.products.update({},{$push:{last_viewed:{$each:["skis"],$slice:-5}}})

You push the value ski’s into the last_viewed array and then slice it to 5 elements. This gives us:

db.products.findOne()
{
    "_id" : ObjectId("51ff9a2d33c4f2089347cab7"),
    "last_viewed" : [
        "cd",
        "game",
        "bike",
        "book",
        "skis"
    ]
}

Mongo maintains the array in natural order and trims the array to 5 elements. It is possible to specify to slice from the start of the array or the end of the array by using positive or negative integers with $slice. It is also possible to sort ascending or descending by passing $sort also. This helps avoid unbounded document growth, and allows for the event system to guarantee in-order delivery.

There are lots of other applications for this feature:

  • Keeping track of the newest messages in a messaging system
  • Managing recent clicks on a website Last accessed/viewed products
  • Top X users/comments/posts

The list goes on.

This feature is available in MongoDB 2.4, but there are many extensions requested, such as full $sort and $slice semantics in $push (SERVER-8069), and making the $slice operation optional (SERVER-8746). Both of these are planned for the 2.5 development series.

Special thanks to Yuri Finkelstein from eBay who was very enthusiastic about this feature and inspired this blog post.