Posts tagged:

drivers

Announcing the Server Discovery and Monitoring Spec

Oct 3 • Posted 2 weeks ago

By Jesse Davis, Python Engineer at MongoDB

Space Shuttle Discovery

[Space Shuttle Discovery]

Last week, we published a draft of the Server Discovery And Monitoring Spec for MongoDB drivers. This spec defines how a MongoDB client discovers and monitors a single server, a set of mongoses, or a replica set. How does the client determine what types of servers they are? How does it keep this information up to date? How does the client find an entire replica set from a seed list, and how does it respond to a stepdown, election, reconfiguration, or network error?

In the past each MongoDB driver answered these questions a little differently, and mongos differed a little from the drivers. We couldn’t answer questions like, “Once I add a secondary to my replica set, how long does it take for the driver to discover it?” Or, “How does a driver detect when the primary steps down, and how does it react?”

From now on, all drivers answer these questions the same. Or, where there’s a legitimate reason for them to differ, there are as few differences as possible and each is clearly explained in the spec. Even in cases where several answers seem equally good, drivers agree on one way to do it.

The server discovery and monitoring method is specified in five sections. First, a client is constructed. Second, it begins monitoring the server topology by calling the ismaster command on all servers. (The algorithm for multi-threaded and asynchronous clients is described separately from single-threaded clients.) Third, as ismaster responses are received the client parses them, and fourth, it updates its view of the topology. Finally, the spec describes how drivers update their topology view in response to errors.

I’m particularly excited about the unittests that accompany the spec. We have 37 tests that are specified formally in YAML files, with inputs and expected outcomes for a variety of scenarios. For each driver we’ll write a test runner that feeds the inputs to the driver and verifies the outcome. This ends confusion about what the spec means, or whether all drivers conform to it.

The Java driver 2.12.1 is the spec’s reference implementation for multi-threaded drivers, and I’m making the upcoming PyMongo 3.0 release conform to the spec as well. Mongos 2.6’s replica set monitor is the reference implementation for single-threaded drivers, with a few differences. The upcoming Perl driver 1.0 implements the spec orthodoxly.

Once we have multiple reference implementations and the dust has settled, the draft spec will be final. We’ll bring the rest of our drivers up to spec over the next year.

You can read more about the Server Discovery And Monitoring Spec at these links:

We have more work to do. For one thing, the Server Discovery And Monitoring Spec only describes how the client gathers information about your server topology—it does not describe which servers the client uses for operations. My Read Preferences Spec only partly answers this second question. My colleague David Golden is writing an improved and expanded version of Read Preferences, which will be called the Server Selection Spec. Once that spec is complete, we’ll have a standard algorithm for all drivers that answers questions like, “Which replica set member does the driver use for a query? What about an aggregation? Which mongos does it use for an insert?” It’ll include tests of the same formality and rigor as the Server Discovery And Monitoring Spec does.

Looking farther ahead, we plan to standardize the drivers’ APIs so we all do basic CRUD operations the same. And since we’ll allow much larger replica sets soon, both the server-discovery and the server-selection specs will need amendments to handle large replica sets. In all cases, we’ll provide a higher level of rigor, clarity, and formality in our specs than we have before.

This was originally posted to Jesse’s blog, Empty Square

Motor: Asynchronous Driver for MongoDB and Python

Sep 5 • Posted 2 years ago

Tornado is a popular asynchronous Python web server. Alas, to connect to MongoDB from a Tornado app requires a tradeoff: You can either use PyMongo and give up the advantages of an async web server, or use AsyncMongo, which is non-blocking but lacks key features.

I decided to fill the gap by writing a new async driver called Motor (for “MOngo + TORnado”), and it’s reached the public alpha stage. Please try it out and tell me what you think. I’ll maintain a homepage for it here, including basic documentation.

Status

Motor is alpha. It is certainly buggy. Its implementation and possibly its API will change in the coming months. I hope you’ll help me by reporting bugs, requesting features, and pointing out how it could be better.

Advantages

Two good projects, AsyncMongo and APyMongo, took the straightforward approach to implementing an async MongoDB driver: they forked PyMongo and rewrote it to use callbacks. But this approach creates a maintenance headache: now every improvement to PyMongo must be manually ported over. Motor sidesteps the problem. It uses a Gevent-like technique to wrap PyMongo and run it asynchronously, while presenting a classic callback interface to Tornado applications. This wrapping means Motor reuses all of PyMongo’s code and, aside from GridFS support, Motor is already feature-complete. Motor can easily keep up with PyMongo development in the future.

Installation

Motor depends on greenlet and, of course, Tornado. It is compatible with CPython 2.5, 2.6, 2.7, and 3.2; and PyPy 1.9. You can get the code from my fork of the PyMongo repo, on the motor branch:

pip install tornado greenlet pip install git+https://github.com/ajdavis/mongo-python-driver.git@motor To keep up with development, watch my repo and do

pip install -U git+https://github.com/ajdavis/mongo-python-driver.git@motor when you want to upgrade.

Example

Here’s an example of an application that can create and display short messages:

Other examples are Chirp, a Twitter-like demo app, and Motor-Blog, which runs this site.

Support For now, email me directly if you have any questions or feedback.

Roadmap In the next week I’ll implement the PyMongo feature I’m missing, GridFS. Once the public alpha and beta stages have shaken out the bugs and revealed missing features, Motor will be included as a module in the official PyMongo distribution.

A. Jesse Jiryu Davis

MongoDB Driver Releases: April

May 8 • Posted 2 years ago

We’ve had a big month with updates and improvements to our drivers.  Here’s a summary:

blog comments powered by Disqus