Review: SaltStack shifts devops into high gear

The only sane and efficient way to manage a large numbers of servers—or even a few dozen, if they change often—is through automation. Automation tools have to be learned and mastered, so they exact a significant up-front cost, but they dramatically reduce the administrative burden in the long run. Perhaps most important, they provide a staunch line of defense against the fatal fat-fingered mistake, which even the most sophisticated cloud operators struggle to avoid.

Configuration management tools such as Puppet, Chef, Ansible, and SaltStack hold the key to automating server provisioning and management tasks. Puppet and Chef are long-established, feature-rich tools that handle the complete lifecycle of IT operations regardless of the hardware platform, OS, application, and configuration model. Ansible is a relative newcomer distinguished by its simplicity and ease of use. SaltStack is neither as convenient as Ansible, nor as feature-rich as Puppet or Chef. But with a highly scalable architecture, powerful orchestration of services, and reliable performance, SaltStack stands tall among these competitors.

SaltStack is an enterprise-grade, commercially supported configuration management tool based on open source Salt. To the Salt core, SaltStack brings a management GUI, role-based access control, LDAP integration, dashboards and reports, auditing, and other advanced features. Salt and SaltStack support a range of platforms, including AIX, Solaris, MacOS, Linux, and Windows.

This review is based on SaltStack Enterprise 5.0, which comes with a completely new GUI, custom dashboards and reports, an integrated database for encrypting sensitive data, and a new multimaster architecture that brings greater scalability and redundancy. SaltStack 5.0 also includes a comprehensive library of modules for handling a wide range of orchestration and automation tasks. You’ll find SaltStack easy to integrate with enterprise IT solutions such as HPE, Nutanix, VMware, Zenoss, and ServiceNow.

Salt and SaltStack were created by Thomas S. Hatch, who had worked with both Puppet and Chef and become dissatisfied with the heavy reliance on Ruby and the slow communication in those earlier tools. Using the power of the ZeroMQ messaging system, Hatch built a fast and highly scalable Python-based system that is now giving Puppet and Chef stiff competition. LinkedIn, NASA, Comcast, and Rackspace are a few notable customers of SaltStack.

The Salt architecture

Unlike Puppet and Chef, which use Ruby to script deployments and configurations, Salt relies on Python as the working language. Like Puppet and Chef, Salt uses a client-server architecture wherein different client machines in the network are controlled by one or more masters. Whereas Salt is limited to a single master, SaltStack supports any number of masters and allows masters to be tiered to scale the system across large environments.

In the Salt architecture, the central command server is called the Salt master and the clients are called minions. The Salt master performs tasks such as authenticating minions, sending commands to minions, and receiving requests from minions, while interacting with admins via the Salt CLI. The Salt master contains a process list for various role categories such as publisher, event publisher, mworker, and reqserver.

A publisher sends commands over the designated transport (ZeroMQ by default) to the connected minions. Each Salt minion has a connection to the master publisher, which uses TCP port 4505. An event publisher is responsible for publishing events to the event bus. The mworker manages back-end processes for the Salt master. At least one mworker process is required. You can specify the number of mworker processes in the Worker_threads configuration file. The reqserver distributes requests among mworkers and receives replies from the minions.

When the Salt master starts up, all of these processes begin as well. As soon as the network boots, Salt minions request a connection to the master that is defined in the minion config file. A cryptographic key exchange is used to authenticate the minion and establish a persistent TCP connection to the master. This persistent connection with the master not only results in faster communications, but also allows for dynamic interactions, such as triggering provisioning or configuration actions in response to events on the minion.

You don’t have to deploy a Salt master to manage Salt minions. They can also be managed as standalone systems. The local salt-call command is used to run a standalone minion.

Salt components

In addition to Salt masters and minions, the key components of Salt are Salt states, Salt pillars, Salt grains, and of course the modules that are used to apply states to minions. Salt states are the most basic building block in the Salt environment. Like the manifest in Puppet, playbook in Ansible, and cookbook in Chef, the Salt state is the file that represents the configuration to be applied to the managed system.

Salt states are written in YAML, which has a number of advantages. First, YAML format is clean, designed for human readability. Second, YAML is language-agnostic and hence portable. Third, YAML has seen widespread adoption that is only growing. On the downside, YAML is verbose (the price of readability) and processing it involves some overhead.

Salt uses Jinja2 templates to define loops and variables. Jinja2 works well within HTML templates because the curly brackets of syntax can be easily distinguished from HTML code. The code becomes clear and concise. However, it should be noted that Jinja2 doesn’t allow you to include Python code in templates, which means you need to code the functionality into your app if you require behavior beyond standard Jinja2 filters.

Salt states gather information from common data types such as dictionaries, lists, and strings. However, regardless of the template data format, template data is translated into Python data types. By default, state files are rendered using Jinja2 templates, which are then parsed as YAML documents. Mako and Wempy templates and the JSON document format are also supported.

The underlying file format can be altered using a different renderer. To manage Salt states, you can use HTML files, Puppet files, XML files, or any other format that is translatable to Python data structures.

Salt pillars are database systems that store global variables, including sensitive data such as passwords and access keys. This information is securely shared with minions on-demand. Although Salt doesn’t allow you to declare the variables on the go, the structured order is an advantage in that it allows pillar data to be easily mapped to minions based on defined characteristics called grains.

Salt grains are facts. They include specific pieces of information such as CPU type, number of cores, and system architecture. Universal states are created using this information.

Salt modules are of two types: namely, state modules and execution modules. While a state module defines a state and tries to fulfill that end state, execution modules simply execute a function. In general, state modules use execution modules to determine state and make changes if necessary. The complete list of state modules can be found on the SaltStack website.

Salt includes several built-in modules to perform common tasks such as installing packages, sending notifications, extracting archives, and accessing Git repositories. With the out-of-box modules, you can save a considerable amount of time. In addition, you will find more than 150 official Salt formulas on GitHub for popular apps such as Docker, PHP, MySQL, Apache, and Node. Salt formulas are prewritten Salt states.

Salt advantages

There are three important aspects of Salt that make it a powerful and comprehensive tool capable of meeting enterprise server management needs. Firstly, Salt enables administrators to manage Linux and Unix server systems while also providing significant support for Windows Server management. Most Salt modules have been ported to Windows, and many Salt states work on Windows. With one tool, you can manage most devices in the network.

Secondly, Salt supports push and pull mechanisms. Push makes it easy to bootstrap the system, so even a bare-metal server can be set up quickly. While Salt uses push by default to send commands to agents, it can also operate in the other direction, allowing minions to query masters for updates.

Thirdly, the ZeroMQ messaging system delivers greater levels of performance while efficiently scaling resources to suit enterprise-class networks and efficiently handle large systems or clients.

While Salt’s working model is similar to those of Puppet, Chef, and Ansible, the biggest difference lies in the communications protocol. For instance, Ansible uses the slower SSH protocol. To compensate, Ansible maintains persistent connections with hosts; that means you always have at least one Ansible process running for each managed node. Similarly, Puppet and Chef use the HTTPS protocol. Puppet includes a “Connection: Keep-Alive” header in the HTTP request to maintain persistent connections, while Chef doesn’t maintain persistent connections by default.

The ZeroMQ messaging library used by Salt scales extremely well. In fact the primary focus of Salt is to facilitate real-time communication with as many servers as possible to support an event-driven infrastructure. Salt helps you monitor your entire network and trigger reactions based on events. With reactive provisioning, configuration, and management across all machines in the infrastructure, you can perform tasks such as instantly restoring configurations to systems after unauthorized changes or automatically scaling cloud resources in response to usage spikes.

Evaluating SaltStack

Capability. SaltStack is designed to perform at peak levels while efficiently managing huge deployments. By default, Salt uses the push method wherein the Salt master is used to remotely execute commands to modify minion states. At the same time, minions can query the master for commands. This asynchronous push and pull communications make Salt extremely flexible and powerful.

Salt state runs are also quite fast. In addition to the high-speed ZeroMQ messaging, Salt takes advantage of persistent connections between the master and minions data caches on minions to make communication faster. And where Puppet, Chef, and Ansible are “static” solutions—meaning you write a manifest, cookbook, or playbook, then execute it—Salt facilitates dynamic communications to respond more quickly to changes in the infrastructure.

By default, Salt executes states in the order in which they are defined in the state files. However, when an action is dependent on or requires another action, you can change this order using a requisite system. Salt provides requisite statements that let you name a state that should be evaluated before the state requiring it. Salt does not hard-fail by default when an error is detected. It will still run to check all the states and create a list of successes and failures for your reference. However, this can be configured. If you wish to halt an execution when a change is discovered or trigger an event such as restarting a service, you can do so.

Scalability. The biggest advantage of SaltStack is its high scalability and resilience. Salt was built with a focus on efficiently managing large deployments, and its speed of communications for remote execution sets it apart from other configuration management tools. Although Salt can access SSH for communication, using minion agents greatly enhances the scalability of the network.

With SaltStack, Salt masters can be configured into multiple levels to form a tiered architecture for redundancy and load distribution. In this way Salt can easily scale from hundreds to tens of thousands of managed nodes. At the same time, you can easily isolate or target a group of minions running under a specific kernel version. In addition, the process of serving files to minions is expedited with the use of asynchronous file services.

Operating system and application coverage. Salt was built primarily to manage Unix and Linux servers. Salt users can also draw on significant Windows Server management capabilities, though Puppet and Chef may have more to offer Windows shops. Both Puppet and Chef integrate well with Windows PowerShell DSC and automate Azure environments.

SaltStack provides a REST API for integrating the system with other tools and clouds, including Amazon EC2, Microsoft Azure, Google Compute Engine, Rackspace, and SoftLayer. Salt Cloud is the provisioning tool used to deploy virtual machines to public clouds and virtualization platforms including VMware, Parallels, VirtualBox, and OpenStack. As Salt Cloud manages virtual machines based on maps and profiles, VM management becomes quick and easy. Salt also enables you to easily provision and manage Docker containers.