Introducing Mongo Connector

Aug 10 • Posted 1 year ago

MongoDB is a great general purpose data store, but for some workflows, you may want to use another tool or integrate data from MongoDB into another system. To address this common interest, we built Mongo Connector, which is a generic connection system that you can use to integrate MongoDB with another system with simple CRUD operational semantics (i.e. insert, update, delete, and search operations.)

Consider the following use cases for this system, which could include:

  • Connecting MongoDB to search engines for more advanced search.
  • Creating a secondary, backup MongoDB cluster that uses Mongo Connector to keep both clusters in sync.
  • Storing specific collections or specific information in other, possibly relational, database systems.
  • Connecting MongoDB to integration platforms such as Mule
  • Dumping your data from MongoDB to any other storage systems, with support to stop and restart the dump at any point.

On startup, Mongo Connector copies your documents from MongoDB to your target system. Afterwards, it constantly performs updates on the target system to keep MongoDB and the target in sync. The connector supports both Sharded Clusters and standalone Replica Sets, hiding the internal complexities such as rollbacks and chunk migrations. Mongo Connector abstracts the MongoDB internals so you only have to implement one class: the DocManager.

The DocManager is a simple, lightweight, and most importantly, simple to write class that defines a limited number of CRUD operations for the target system. The DocManager API explains what functions must be implemented, and Mongo Connector uses those functions to link up MongoDB and the target system.

For the first release, we have implementations of the Doc Manager for Solr, ElasticSearch and, of course, MongoDB (if you want to connect your MongoDB to another MongoDB instance).

To install Mongo Connector, issue the following command at your systems shell:

pip install mongo-connector

After that, start the Mongo Connector. For example, suppose there is a Sharded Cluster with a mongos running on localhost:27217, a Solr search server running on localhost:8080, and the Solr access URL being http://localhost:8080/solr. Then, use the following command to have Mongo Connector sync the MongoDB cluster with Solr:

python mongo_connector.py -m localhost:27217 -t http://localhost:8080/solr

The connector will start syncing the data to the Solr connection at http://localhost:8080/solr

Check out our github repo for requests for new doc managers, bug reports, and documentation on Mongo Connector: https://github.com/10gen-labs/mongo-connector

About us: Mongo Connector was designed, coded, tested, packaged, and released by Leonardo Stedile and Aayush Upadhyay, two of 10gen’s summer interns. Special thanks to Spencer Brody and Randolph Tan, our two mentors. We hope you find Mongo Connector useful, and that it helps you build awesome things with MongoDB.

blog comments powered by Disqus
blog comments powered by Disqus