Quick! Name a log analysis service. If the first word that popped out of your mouth was “Splunk,” you’re far from alone.
But Splunk’s success has spurred many others to up their log-analysis game, whether open source or commercial. Here is a slew of contenders that have a lot to offer sysadmins and devops folks alike, from services to open source stacks.
Elasticsearch (ELK stack)
The acronym “LAMP” is used to refer to the web stack that comprises Linux, the Apache HTTP web server, the MySQL database, and PHP (or Perl, or Python). Likewise, “ELK” is used to describe a log analysis stack built from Elasticsearch for search functionality, Logstash for data collection, and Kibana for data visualization. All are open source.
Elastic, the company behind the commercial development of the stack, provides all the pieces either as cloud services or as free, open source offerings with support subscriptions. Elasticsearch, Logstash, and Kibana offer the best alternative to Splunk when used together, considering that Splunk’s strength is in searching and reporting as well as data collection.
Other companies also offer commercially supported editions of the ELK stack, or ELK as a service:
Logsene
Sematext’s Logsene product is ELK as a service: a hosted ELK stack, available either in the cloud or behind the firewall, that works with any log-shipping service. The platform integrates with 40-plus services and apps to generate contextual information about what’s going on inside your organization. Plans begin at $50 per month, with free 30-day trials for paid plans. A free basic tier is available, although it’s restricted to 500MB per day of logs and seven days of retention.
Logsene also offers Logagent, an open source project for ingesting logs from a variety of sources and piping them to Sematext’s cloud or to an Elasticsearch instance. One of Logagent’s handier out-of-the-box features is data masking, so that sensitive data can be concealed before being shipped. Logagent is also available in a 30-day trial.
Logz.io
Logz.io offers ELK as a service with features like “live tail” (the ability to see logs in real time from a console) and automatic archiving to Amazon S3 object storage. Time-series analysis via Kibana and Grafana is also now available in an early form.
Paid plans starting at $289 per month for 5GB of storage and up to one year of retention. A free community tier provides up to 3GB of daily capacity and three-day retention.
Qbox
Qbox provides hosted editions of every piece of the ELK stack on a variety of cloud infrastructures (AWS, IBM Cloud, Rackspace). Each implementation can be scaled across nodes, with an adjustable amount of RAM, deployment in various geographic regions, and optional failover between nodes. Qbox also offers a hosted version of the full ELK stack.
Graylog
Graylog uses Elasticsearch as the central component, but it also relies on the MongoDB data store and the Apache Kafka streaming system. Event data and on-the-wire data can be ingested from most any source, including third-party connectors like Fluentd. Graylog also comes with its own browser-based front-end UI, but its APIs in theory will allow any front end.
The core product is free open source. The enterprise edition, which adds functions like archiving, is free for users processing less than 5GB a day. Editions are available for most every virtualized environment, including Docker, and scripts for major orchestration and automation tools (Chef, Puppet, Ansible, Vagrant) are also provided.
InsightOps
InsightOps is part of Rapid7’s cloud-hosted suite of analytics, visibility, and automation product line. Data can be ingested from a wide variety of formats and platforms—container systems like Docker and CoreOS; events from Logstash, PagerDuty, and New Relic; and alerts from notification and messaging systems like Slack. Most anything else can be integrated via webhooks and the API. “Synthetic” logs can be generated from endpoints that don’t normally produce them. Both live dashboards and static reports can be generated from gathered data.
Pricing starts at $48 per month for 30GB of data and 30 days of data retention, with a free 30-day trial period.
Loggly
Loggly is a cloud service that collects logs from a broad range of defined services, but anything that has a syslog-compatible agent (anything that uses RFC 5424, basically) works as an ingestion source. Ingested data is made available for fast searching and analysis via a RESTful API.
The results can be examined through a web-based dashboard and configured to trigger alerts in Slack based on certain conditions. Users can see live tail results with chosen logs. It’s also possible to automatically extract details from logged data, such as session IDs, for further insight.
Paid plans begin at $79 per month, and have a 14-day free trial period. The free tier limits ingestion to 200MB per day and seven days of data retention.
Papertrail
Papertrail has many features familiar from other competitors, including live views of collected logs, convenient search functions, and contextual links in a log’s history, all delivered as a cloud service with a highly granular pricing structure.
Paid plans start at $6 per month with 1GB per month of storage and a one-year retention period, with plans highly customizable after that up to 1,500GB per month. An introductory tier lets you collect up to 50MB of logs per month for free (plus a bonus 16GB in the first month), with 48 hours of logs searchable and seven days of logs archived.
SolarWinds Log Analyzer
SolarWinds offers a broad variety of IT management products for security, databases, infrastructure management, and—you guessed it—event log analysis. SolarWinds Log Analyzer takes in data from many common event-generation systems (system logs in the syslog format, as well as Windows and VMware events), provides a search-and-filter front end, offers real-time stream views of events, can generate reports, and forwards or exports logs to other destinations like SIEM systems, databases, or flat-text files. Prices for Log Analyzer begin at $1,495, with a free 30-day trial available.
Sumo Logic
Sumo Logic—one of Network World’s 10 big data startups to watch in 2014—is a cloud-native log-analysis service that uses machine learning and predictive analytics to discover anomalies and outliers in the data and help users anticipate potentially disruptive events.
Sumo Logic comes preconfigured with searches and dashboards for many common enterprise products, from web servers (Apache, IIS, Nginx) to infrastructure (Cisco, Kubernetes, Docker) to operating systems. It also supports native ways for gathering metrics directly from hosts—for example, on AWS by way of Amazon CloudWatch. Users can also roll their own data collection service by using tools like Graphite.
Paid tiers start at $270 per month for 3GB of ingestion per day and up to 30GB of storage. The free tier lets you ingest up to 500MB per day with 4GB of data retention.