Microsoft has fired a shot heard around the globe, so to speak, in data management with the debut of Azure Cosmos DB at the recent Microsoft Build 2017 developer conference in Seattle. The cloud database is positioned for elasticity and globally available data, supported on the Azure cloud. The project was founded in 2010 by Microsoft’s Dharma Shukla, who holds the title of distinguished engineer at the company.
InfoWorld Editor at Large Paul Krill spoke with Shukla during the conference to get his perspectives on the technology.
InfoWorld: Why is this project, which began more than six years ago, going to the public now?
Shukla: It’s a very complex system. The goal we had was to build a globally distributed database system, which makes the data automatically available wherever the users are so that you can be in any part of the world and you get really low latency, really fast performance when you access your data. You can scale throughput and storage elastically, pay only for what you need across the world. Cloud is perfect for it. When you write your app, your app is automatically available everywhere, wherever the datacenters are.
But when it comes to data, data is locked. Data doesn’t get distributed automatically, so that was the mission for Cosmos DB, and the reason it took so long is it’s like climbing Mount Everest. All along, we had internal applications using it. We also released a service called DocumentDB in 2015, which was a milestone along the journey. Since then, we’ve been adding more and more stuff, so now it’s culminated into Cosmos DB, the service that we released.
InfoWorld: With DocumentDB, those customers were moved over to Cosmos?
Shukla: Yes, they are automatically Cosmos DB customers.
InfoWorld: How is Cosmos DB an advance over DocumentDB?
Shukla: There is significant enhancement. We’ve been steadily adding features, and as a cloud database we hardened the features. We tested it with internal customers, then selected external customers before opening it to everyone. We’ve been working on many capabilities, which are the database engine capabilities, as well as distributed systems capabilities. DocumentDB was a snapshot in that journey. A lot has been added.
To give you some examples, [there is] support for multiple data models in the database engine to support key value, graph, and other data types, support for elastically scaling throughput at different granularities. In DocumentDB, it was only at a second granularity. These are hard problems.
For example, if you’re scaling throughput, say I want a million transactions per second across four different geographical regions. The next line of code you write is you say instead of a million transactions per second I need 20 million transactions per second. As a customer, you provision these. You scale your system, a single table or database, with this much throughput.
But what if you get a spike? A lot of deep engineering has gone into creating a system that offers battle-tested SLAs. In Cosmos DB, we are constantly adding more consistency models and not all are exposed. In today’s launch we have five. DocumentDB had initially three, then four. There are many, many features that we keep adding.
InfoWorld: What are the five consistency models at this point?
Shukla: Strong, bounded staleness, session, consistent prefix, and eventual.
InfoWorld: This is obviously something for large, global enterprises, correct?
Shukla: It could be for any app that wants to reach the users. If you’re an ISV, if you’re a startup building an iPhone app, you need to find a database that can make that data automatically available everywhere, and I think that is a fundamental, differentiated capability. Click on the map of the world and add regions, and your data magically appears.
InfoWorld: Is any of Cosmos DB open source at all?
Shukla: No, it is not open source.
InfoWorld: Does Cosmos DB support containers and Docker?
Shukla: You can use a version of Cosmos DB on your laptop. We have a Dockerized version of it. You can put it in a Docker container, run it on your laptop or on a VM somewhere.
InfoWorld: How were you able to make Cosmos DB scale worldwide?
Shukla: It took us many years and we are not done yet. There are many aspects. In my blog post, I talk about at a very level a list of things. But for each one it is probably worth 20, 30 pages of design documents. We stand on the shoulders of giants. Underneath us, we have millions of lines of C++ code, but underneath us is also a lot of Azure infrastructure that has helped us.
InfoWorld: Do you anticipate Cosmos DB ever running on other clouds? Would you ever let this run on the Amazon cloud or the Google cloud or is this specifically going to be Azure only?
Shukla: We want to be on Azure Stack, so that is one path. In terms of other cloud providers, one of the things we have done is that we have opened up all APIs. You can use Mongo APIs, and over time, we’re going to add other APIs. We add Gremlin APIs [which are in preview mode]. We support DocumentDB, SQL APIs, we support Azure table storage APIs. Over time we are going to support other APIs. You can use these APIs, and you don’t have to change your app. That’s the direction we have.
InfoWorld: How do you hope to improve Cosmos DB in the future? What capabilities do you plan to add to it that aren’t there now, and when might we see these?
Shukla: We are constantly evolving. We have many optimizations in the area of how we [work with] different ways of tiering data so you can, if you haven’t used your data for some period of time, you can archive it. If you want the freshest data, we prioritize those over the data that you’re not accessing recently—things like that are in the path forward. In terms of APIs we go by the [level of] interest.
InfoWorld: Did SQL Server provide any of the basis for Cosmos DB, or are they two different technologies?
Shukla: Two completely different technologies. The common thread between them is that they both use Azure components underneath. That’s a common layer, but they’re both different technologies.
InfoWorld: How has Cosmos DB worked out at different user sites that have had preview versions of it? Jet.com was mentioned as an early user.
Shukla: They’ve been in production with this for, I think, close to a year.
InfoWorld: Is there anything else you want to say about Cosmos DB?
Shukla: I would say a couple of things worth noting. One is that it is the world’s first globally distributed database that supports multiple data models, and it’s extensible, so we’ll keep adding. It is the first globally distributed database that makes global distribution of data turnkey, meaning you can go with a single click of a button or an API call. There is no setup of machines or datacenters or replication topologies. It’s simple. The mission for us is to enable developers to write globally distributed apps easily because we think users would benefit from low latency around the world, high availability, consistency choices, scaling throughput worldwide, all of these things.
And you can count on it because it is backed by very comprehensive SLAs. It’s the first time since the arrival of cloud that a service has SLAs that are beyond high availability. Everyone has high availability SLAs, but this has low-latency SLAs, as well as consistency SLAs and throughput SLAs. This is something that a lot of engineering has gone into, so large companies or startups who care about at that scale like Jet.com or some of the other names we have, they can count on it.