Financial information behemoth Bloomberg is a big fan of Kubernetes, and is using it for everything from serving up Bloomberg.com to complex data processing pipelines.
Rather than use a managed Kubernetes service or employ an outsourced provider, Bloomberg has chosen to invest in deep Kubernetes expertise and keep the skills in-house. Like many enterprise organizations, Bloomberg originally went looking for an off-the-shelf approach before settling on the decision to get involved more deeply with the open source project directly.
Steven Bower, Data and Infrastructure Lead, Bloomberg photographed at Bloomberg World Headquarters... [+]
"We started looking at Kubernetes a little over two years ago," said Steven Bower, Data and Infrastructure Lead at Bloomberg. "We had built our own orchestration system that pre-dated Docker and Kubernetes, and we were using that to manage our search infrastructure in the team I originally lead."
"We were looking at the future of our orchestration framework and trying to figure out if there was a way for us not to have all this custom code," he said. Bower's team looked at Mesos, Yarn, Rancher, and similar offerings but nothing seemed to fit.
Around the same time, other parts of Bloomberg started to adopt Kubernetes. "Things like Bloomberg.com are running it," he said, "and some other non-infrastructure engineering teams were using it for things like stateless content processing pipelines." Eighteen months ago Bower's team started using Kubernetes, driven largely by data science workloads.
"It's a great execution environment for data science," says Bower. "The real Aha! moment for us was when we realized that not only does it have all these great base primitives like pods and replica sets, but you can also define your own primitives and custom controllers that use them."
"I have always said that people contribute to open source in a selfish way," says Dan Kohn, Executive Director of the CNCF, "but everyone in the ecosystem benefits from the collective selfishness." This doesn't happen automatically, and the role of the CNCF in shepherding the development of Kubernetes, and ensuring that individual selfishness doesn't overwhelm the collective good, shouldn't be overlooked.
"It's been a complicated growing process," says Kohn. "Nobody is born knowing how to run a TOC [Technical Oversight Committee] meeting." The members of the Foundation and its 'three ring circus' governance structure have tried to learn from history and their personal experience with other governance efforts. "We've gotten clearer about a set of principles," says Kohn. These principles are captured, in typical developer fashion, in a GitHub repo.
"The Cloud Native Computing Foundation (CNCF) has done a fantastic job with the Kubernetes ecosystem," Bower said. "It has a well managed roadmap, so we can plan ahead."
Bloomberg has moved away from relying on vendors to drive innovation and has become heavily involved in open-source software communities instead. "Culturally, Bloomberg is like lots of small startups attached to a central spine," Bower says. "A core value is delivering capability quickly."
Bower has advice for other organizations considering a similar approach. "You need to have your organization in a good place for managing software in this way already," he says. "If people are just randomly building stuff in a disorganized way, you're going to have the same thing with open source software."
Organizations also need to carefully consider how involved they want to be with Kubernetes. "With a cloud-hosted service, a lot of the complexity of Kubernetes is hidden from you. On-site you have to have people who understand how it works at the core, not just interfacing with it." This isn't a decision to be taken lightly, because it becomes a critical operational constraint.
Leaving support up to an external vendor makes you beholden to the vendor's responsiveness if there are issues. Bloomberg decided that a typical vendor response time wasn't going to be enough. "No one ever talks about nines of uptime here," Bower says. "This thing cannot go down."
The traders relying on Bloomberg's data to inform their trades stand to lose big money if they can't access the information they need due to an outage, and Bloomberg is often on the hook to make them whole if they miss out on an opportunity. These financial incentives have reinforced Bloomberg's decision to be in tight control of its data supply-chain.
Other organizations should look carefully at their own circumstances before following Bloomberg's lead.
