Enhancing the F# developer experience with MongoDB

Aug 28 • Posted 7 months ago

This is a guest post by Max Hirschhorn, who is currently an intern at MongoDB.

About the F# programming language

F# is a multi-paradigm language built on the .NET framework. It is functional-first and prefers immutability, but also supports object-oriented and imperative programming styles.

Also, F# is a statically-typed language with a type inference system. It has a syntax similar to Ocaml, and draws upon ideas from other functional programming languages such as Erlang and Haskell.

Using the existing .NET driver

The existing .NET driver is compatible with F#, but is not necessarily written in a way that is idiomatic to use from F#.

Part of the reason behind this is that everything in F# is explicit. For example, consider the following example interface and implementing class.

[<Interface>]
type I =
    abstract Foo : unit -> string

type C() =
    interface I with
        member __.Foo () = "bar"

// example usage
let c = C()
(c :> I).Foo()

So in order to use any of the interface members, the class must be upcasted using the :> operator. Note that this cast is still checked at compile-time.

In a similar vein, C# supports implicit operators, which the BSON library uses for converting between a primitive value and its BsonValue equivalent, e.g.

new BsonDocument {
    { "price", 1.99 },
    { "$or", new BsonDocument {
        { "qty", new BsonDocument { { "$lt", 20 } } },
        { "sale", true }
    } }
};

whereas F# does not. This requires the developer to explicitly construct the appropriate type of BsonValue, e.g.

BsonDocument([ BsonElement("price", BsonDouble(1.99))
               BsonElement("$or", BsonArray([ BsonDocument("qty", BsonDocument("$lt", BsonInt32(20)))
                                              BsonDocument("sale", BsonBoolean(true)) ])) ])

with the query builder, we can hide the construction of BsonDocument instances, e.g.

Query.And([ Query.EQ("price", BsonDouble(1.99))
            Query.OR([ Query.LT("qty", BsonInt32(20))
                       Query.EQ("sale", BsonBoolean(true)) ]) ])

It is worth noting that the need to construct the BsonValue instances is completely avoided when using a typed QueryBuilder.

type Item = {
    Price : float
    Quantity : int
    Sale : bool
}

let query = QueryBuilder<Item>()

query.And([ query.EQ((fun item -> item.Price), 1.99)
            query.Or([ query.LT((fun item -> item.Quantity), 20)
                       query.EQ((fun item -> item.Sale), true) ]) ])

What we are looking for is a solution that matches the brevity of F# code, offers type-safety if desired, and is easy to use from the language.

New features

The main focus of this project is to make writing queries against MongoDB as natural from the F# language as possible.

bson quotations

We strive to make writing predicates as natural as possible by reusing as many of the existing operators as possible.

A taste

Consider the following query

{ price: 1.99, $or: [ { qty: { $lt: 20 } }, { sale: true } ] }

we could express this with a code quotation

bson <@ fun (x : BsonDocument) -> x?price = 1.99 && (x?qty < 20 || x?sale = true) @>

or with type safety

bson <@ fun (x : Item) -> x.Price = 1.99 && (x.Quantity < 20 || x.Sale = true) @>
Breaking it down

The quotations are not actually executed, but instead are presented as an abstract syntax tree (AST), from which an equivalent BsonDocument instance is constructed.

The ? operator

The ? operator is defined to allow for an unchecked comparison. The F# language supports the ability to do a dynamic lookup (get) and assignment (set) via the ? and ?<- operators respectively, but does not actually provide a implementation.

So, the F# driver defines the ? operator as the value associated with a field in a document casted to a fresh generic type.

// type signature: BsonDocument -> string -> 'a
let (?) (doc : BsonDocument) (field : string) =
    unbox doc.[field]

and similarly defines the ?<- operator as the coerced assignment of a generically typed value to the associated field in the document.

// type signature: BsonDocument -> string -> 'a -> unit
let (?<-) (doc : BsonDocument) (field : string) value =
    doc.[field] = unbox value |> ignore
Queries

Unchecked expressions have the type signature Expr<BsonDocument -> bool>.

// $mod
bson <@ fun (x : BsonDocument) -> x?qty % 4 = 0 @>

Checked expressions have the type signature Expr<'DocType -> bool>.

// $mod
bson <@ fun (x : Item) -> x.Quantity % 4 = 0 @>
Updates

Unchecked expressions have the type signature Expr<BsonDocument -> unit list>. The reason for the list in the return type is to perform multiple update operations.

// $set
bson <@ fun (x : BsonDocument) -> [ x?qty <- 20 ] @>

// $inc
bson <@ fun (x : BsonDocument) -> [ x?qty <- (+) 1 ] @>
Mmm… sugar

A keen observer would notice that (+) 1 is not an int, but actually a function int -> int. We are abusing the fact that type safety is not enforced here by assigning the quantity field of the document to a lambda expression, that takes a single parameter of the current value.

Note that

// $inc
bson <@ fun (x : BsonDocument) -> [ x?qty <- x?qty + 1 ] @>

is also valid.

Checked expressions either have the type signature Expr<'DocType -> unit list> or Expr<'DocType -> 'DocType>, depending on whether the document type has mutable fields (only matters for record types).

// $set
bson <@ fun (x : Item) -> [ x.Quantity <- 20 ] @>

// $inc
bson <@ fun (x : Item) -> [ x.Quantity <- x.Quantity + 1 ] @>

mongo expressions

Uses the monadic structure (computation expression) to define a pipeline of operations that are executed on each document in the collection.

Queries
let collection : IMongoCollection<BsonDocument> = ...

mongo {
    for x in collection do
    where (x?price = 1.99 && (x?qty < 20 || x?sale = true))
}

or with a typed collection

let collection : IMongoCollection<Item> = ...

mongo {
    for x in collection do
    where (x.price = 1.99 && (x.qty < 20 || x.sale = true))
}
Updates
let collection : IMongoCollection<BsonDocument> = ...

mongo {
    for x in collection do
    update
    set x?price 0.99
    inc x?qty 1
}

or with a typed collection

let collection : IMongoCollection<Item> = ...

mongo {
    for x in collection do
    update
    set x.Price 0.99
    inc x.Quantity 1
}

Serialization of F# data types

Now supports

Conclusion

Resources

The source code is available at GitHub. We absolutely encourage you to experiment with it and provide us feedback on the API, design, and implementation. Bug reports and suggestions for improvements are welcomed, as are pull requests.

Disclaimer. The API and implementation are currently subject to change at any time. You must not use this driver in production, as it is still under development and is in no way supported by MongoDB, Inc.

Acknowledgments

Many thanks to the guidance from the F# community on Twitter, and my mentors: Sridhar Nanjundeswaran, Craig Wilson, and Robert Stam. Also, a special thanks to Stacy Ferranti and Ian Whalen for overseeing the internship program.

Today’s News

Aug 27 • Posted 7 months ago

A: I met someone from 10gen the other day…

B: From where?

A: 10gen. The company that makes MongoDB.

B: Ohhh.

As of today, the above conversation will never happen again, because we are now called “MongoDB, Inc.”

MongoDB CEO, Max Schireson, published a post that details why we made the decision to rebrand. See. If you have any questions or concerns, please let us know.

-Eliot and the MongoDB Team

Surviving Success at Matchbook: Using MMS To Track Down Performance Issues

Aug 22 • Posted 7 months ago

This is a guest post from Jared Wyatt, CTO of Matchbook, an app for remembering the places you love and want to try.

I joined Matchbook as CTO in January with the goal of breathing new life into an iOS app that had a small, but very devoted following. For various reasons, we decided to start fresh and rebuild everything from the ground up—this included completely revamping the app itself and totally redesigning our API and backend infrastructure. The old system was using MySQL as a datastore, but MongoDB seemed like a better fit for our needs because of its excellent geospatial support and the flexibility offered by its document-oriented data model.

We submitted Matchbook 2.0 to the App Store at the end of June and within a few days received an email from Apple requesting design assets because they wanted to feature our app. So, of course we were all, like, “OMG OMG OMG.”

Read more

Aggregation Options on Big Data Sets Part 1: Basic Analysis using a Flights Data Set

Aug 21 • Posted 7 months ago

By Daniel Alabi and Sweet Song, MongoDB Summer Interns

Flights Dataset Overview

This is the first of three blog posts from this summer internship project showing how to answer questions concerning big datasets stored in MongoDB using MongoDB’s frameworks and connectors.

The first dataset explored was a domestic flights dataset. The Bureau of Transportation Statistics provides information for every commercial flight from 1987, but we narrowed down our project to focus on the most recent available data for the past year (April 2012-March 2013).

We were particularly attracted to this dataset because it contains a lot of fields that are well suited for manipulation using the MongoDB aggregation framework.

Read more

$push to sorted array

Aug 20 • Posted 8 months ago

By Sam Weaver, MongoDB Solutions Architect and Alberto Lerner, MongoDB Kernel Lead

MongoDB 2.4 introduced a feature that many have requested for some time - the ability to create a capped array.

Capped arrays are great for any application that needs a fixed size list. For example, If you’re designing an ecommerce application with MongoDB and want to include a listing of the last 5 products viewed, you previously had to issue a $push request for each new item viewed, and then a $pop to kick the oldest item out of the array. Whilst this method was effective, it wasn’t necessarily efficient. Let’s take an example of the old way to do this:

First we would need to create a document to represent a user which contains an array to hold the last products viewed:

db.products.insert({last_viewed:["bike","cd","game","bike","book"]})
db.products.findOne()
{
    "_id" : ObjectId("51ff97d233c4f2089347cab6"),
    "last_viewed" : [
        "bike",
        "cd",
        "game",
        "bike",
        "book"
    ]
}

Read more

Mongoose 3.7.0 (Unstable) Released

Aug 15 • Posted 8 months ago

By EJ Bensing, MongoDB intern for Summer 2013

I’ve spent the last 2 months interning at 10gen, the MongoDB company, working on Mongoose. It has been a lot of fun and I’ve learned a ton about Node.js, MongoDB, and building open source libraries. I’m going to save all of that for a different post though, and instead talk about the newest release of Mongoose.

Unstable

To start things off, this is an unstable release. This means that it contains potentially breaking changes or other major updates, and thus should probably not be used in production. You can tell this is an unstable release because of the middle digit. Starting from 3.7, odd middle digits mean unstable, even mean stable. This is identical to the Node.js and MongoDB versioning schemes.

Read more

The MongoDB Web Shell

Aug 14 • Posted 8 months ago

About

The MongoDB Web Shell is a web application designed to emulate some of the features of the mongo terminal shell. This project has three main uses: try.mongodb.org, 10gen Education online classes, and the MongoDB API documentation.

In these three different contexts, users will be able to familiarize themselves with the MongoDB interface and basic commands available both independently and as part of a 10gen education homework assignment in the education program.

See a screenshot of the state of the browser shell prior to this summer below:

Read more

The MongoDB Java Driver 3.0

Aug 13 • Posted 8 months ago

By Trisha Gee, MongoDB Java Engineer and Evangelist

You may have heard that the JVM team at 10gen is working on a 3.0 version of the Java driver. We’ve actually been working on it since the end of last year, and it’s probably as surprising to you as it is to me that we still haven’t finished it yet. But this is a bigger project than it might seem, and we’re working hard to get it right.

So why update the driver? What are we trying to achieve?

Well, the requirements are:

  • More maintainable
  • More extensible
  • Better support for ODMs, third party libraries and other JVM languages
  • More idiomatic for Java developers
Read more

Improving Driver Documentation: The MongoDB Meta Driver

Aug 8 • Posted 8 months ago

This is a guest post, written by Mario Alvarez, a MongoDB intern for Summer 2013

This summer, I worked on developing the Meta Driver project, an effort to re-work the drivers documentation, creating an implementable specification of driver functionality that could be applied to different drivers in different languages.

Read more

MongoDB Connector for Hadoop

Aug 7 • Posted 8 months ago

by Mike O’Brien, MongoDB Kernel Tools Lead and maintainer of Mongo-Hadoop, the Hadoop Adapter for MongoDB

Hadoop is a powerful, JVM-based platform for running Map/Reduce jobs on clusters of many machines, and it excels at doing analytics and processing tasks on very large data sets.

Since MongoDB excels at storing large operational data sets for applications, it makes sense to explore using these together - MongoDB for storage and querying, and Hadoop for batch processing.

The MongoDB Connector for Hadoop

We recently released the 1.1 release of the MongoDB Connector for Hadoop. The MongoDB Connector for Hadoop makes it easy to use Mongo databases, or MongoDB backup files in .bson format, as the input source or output destination for Hadoop Map/Reduce jobs. By inspecting the data and computing input splits, Hadoop can process the data in parallel so that very large datasets can be processed quickly.

Read more

Securing MongoDB on Windows Azure

Aug 2 • Posted 8 months ago

By Sridhar Nanjundeswaran, Windows Azure lead at 10gen

I have used the MongoDB Installer for Windows Azure to deploy my MongoDB instance on a Windows Virtual Machine on Windows Azure. It is not my production environment but I would still like to secure it. What do I need to do to secure this standalone instance?

Let us take a look at the possible issues and how you would resolve each of them.

  • Password
  • Administrator username
  • Endpoints

Read more

The Most Popular Pub Names

Jul 30 • Posted 8 months ago

By Ross Lawley, MongoEngine maintainer and Scala Engineer at 10gen

Earlier in the year I gave a talk at MongoDB London about the different aggregation options with MongoDB. The topic recently came up again in conversation at a user group, so I thought it deserved a blog post.

Gathering ideas for the talk

I wanted to give a more interesting aggregation talk than the standard “counting words in text”, and as the aggregation framework gained shiny 2dsphere geo support in 2.4, I figured I’d use that. I just needed a topic…

What is top of mind for us Brits?

Two things immediately sprang to mind: weather and beer.

Read more

Introducing the MongoDB Driver for the Rust Programming Language

Jul 25 • Posted 8 months ago

Discuss on Hacker News

This is a guest post by Jao-ke Chin-Lee and Jed Estep, who are currently interns at 10gen. This summer they were tasked with building a Rust driver for MongoDB.

Today we are open sourcing the alpha release of a MongoDB driver for the Rust programming language. This is the culmination of two months of work with help from the Rust community, and we’re excited to be sharing our initial version. We are looking forward to feedback from both the Rust and MongoDB communities.

Read more

NeDB: a lightweight Javascript database using MongoDB’s API

Jul 17 • Posted 9 months ago

This is a guest post by Louis Chatriot

Sometimes you need database functionality but want to avoid the constraints that come with installing a full-blown solution. Maybe you are writing a Node service or web application that needs to be easily packageable, such as a continuous integration server. Maybe you’re writing a desktop application with Node Webkit, and don’t want to ask your users to install an external database. That’s when you need NeDB.

Read more

Morphia Version 0.101 Released

Jul 11 • Posted 9 months ago

The Java Team has released Morphia, version 0.101. Rumors of Morphia’s demise have been greatly exaggerated.

This release formalizes the 0.99-SNAPSHOT code that many have been using for years. Apart from some formatting and minor style changes, there is no functional change from what’s been on Google Code for the past few years.

Read more