Mesosphere DC/OS, the datacenter automation solution built atop the Apache Mesos orchestration system to provide one-click management for complex applications, has now hit its 1.9 revision.
With this release, Mesosphere is once again emphasizing DC/OS as a solution for deploying and maintaining large, complex data-centric applications. Version 1.9 adds out-of-the-box support for several major data services and a passel of improvements for DC/OS's existing container support.
Everyone into the pool!
DC/OS manages a datacenter's worth of Linux machines as if they were a single pooled resource maintained by high-level commands from a CLI and GUI. Apps like Apache Cassandra, Kafka, Spark, and HDFS -- many of them not known for being easy to manage -- can be deployed with a few command-line actions and scaled up or down on demand or automatically.
Among the new additions are two major stars of the modern open source data stack: The database/in-memory caching store Redis and the NoSQL database solution Couchbase. Redis in particular has become a valuable component for big data applications as an accelerator for Apache Spark, so being able to accelerate other DC/OS apps with it is a boon.
Version 1.9 also adds support for Elastic; DataStax Enterprise, the commercial offering based on the Apache Cassandra NoSQL system; and Alluxio, a data storage acceleration layer specifically designed for big data systems like Spark.
Managing applications like these through DC/OS makes better use of the utilization in a given cluster. Each application supported in DC/OS has its own scheduling system, so apps with complementary behaviors can be packed together more efficiently and automatically migrated between nodes as needed. DC/OS also ensures apps that upgrade frequently (like scrappy new big data frameworks) can be rolled out across a cluster without incurring downtime.
There's barely a data application these days that isn't tied into machine learning in some form. Given that Mesosphere was already promoting DC/OS for data-centric apps, it only makes sense the company is also pushing DC/OS as a management solution for machine learning apps built on its supported solutions. This claim has some validity with GPU resources, as DC/OS can manage GPU as simply another resource to be pooled for application use.
Container conscious
Because DC/OS also manages containers with Google's Kubernetes project, it's been described as a container solution, but only in the sense that containers are one of many kinds of resources DC/OS manages.
Containers have long been criticized for being opaque. Prometheus, now a Cloud Native Computing Foundation project, was originally developed by Soundcloud for getting insight into running containers, and DC/OS 1.9 supports Prometheus along with Splunk, ELK, and Datadog as targets for managing the logs and metrics it collects from containers.
Version 1.9 also introduces a feature called container process injection. With it, says the company, developers "remotely run commands in any container in the same namespace as the task being investigated." Containers are not simply opaque by nature, but also ephemeral, so being able to connect to them and debug them directly while they're still running will be useful.