By Tomer Levy, InfoWorld |

Is open source overtaking Splunk?

No conversation on log analytics can occur without beginning with Splunk. But is open-source ELK overtaking the company? A different point of view

Trying to understand open source adoption is a challenging task. In contrast to public companies, the metrics of open-source projects mostly rely on the number of GitHub stars (which is public) or the number of downloads (which is often unknowable).

As a co-founder and CEO of Logz.io, I've been heavily involved in the open source log analytics domain through working with with the community and focusing on the ELK Stack.

The background: The ELK Stack is the combination of Elasticsearch, Logstash, and Kibana that is used specifically in log analytics. Logstash ships log data to Elasticsearch, which indexes the information in a searchable datastore. Kibana then takes the datastore and shows the information in graphical format for log analysis.

I have been looking into creative ways to understand the magnitude of the ELK Stack's adoption. GitHub stars are soaring, downloads of the ELK Stack are reportedly exceeding 500,000 monthly downloads, and key, tech-savvy companies like Google, Netflix, and LinkedIn heavily rely on it.

No doubt, the ELK Stack has been seeing tremendous adoption.

However, the question remains: Is open source ELK overtaking Splunk?

Log analytics started with Splunk

No conversation on log analytics can occur without beginning with Splunk -- the company that has become the top enterprise log analysis software since its founding in 2003.

Splunk's Brian Gilmore, however, recently told Diginomica's Derek du Preez that "we can't ignore the fact that people are aware that [competing open source products] exist."

Clearly, there is a growing demand for open-source log analysis software. But what can we learn about how Splunk's alternatives are taking advantage of that demand?

Comparing Splunk's alternatives

Google Trends is a tool that compares the number of searches for given keywords over time. While it will not tell you the specific number of searches, the tool will show you relative changes in search volumes.

What digital marketers call "organic search" is one of the best sources of objective data because people type search queries to find what they really want while thinking that no one is watching them (although we all know on some level that we are being tracked online).

So, I used Google Trends as another indication of the number of people who are using, learning, and discovering this open source stack.

Take a look at the global Google Trends report for Splunk versus the components of the ELK Stack versus Sumo Logic:

(The terms I've used are: [splunk] versus ["elk stack" + "logstash" + "kibana" + "elasticsearch"] versus ["sumo logic"].)

Splunk was founded in 2003. The ELK Stack was mostly created in 2010. In terms of the number of people who are searching Google, the components of the ELK Stack have gained as much brand awareness as Splunk in only five years. (Of course, Elasticsearch can be used for search in applications, but one of the most common uses is for log analysis.)

This analysis indicates that that early 2014 was the inflection point at which the ELK Stack as a whole had gained more traction than Splunk. From that point on, the ELK Stack has been seeing roughly 40 percent more organic searches than Splunk.

These results are pretty amazing when you come to think of them.

The reach of the competition

A Google Trends geographical analysis also presents an interesting take on Splunk, the ELK Stack, and Sumo Logic:

Splunk is a global product that is used worldwide -- and the only alternative to Splunk that has global reach is the ELK Stack, while Sumo Logic is geographically focused on the United States. Even a heavily funded, global company like Splunk is unable to cope with a successful open-source stack. (We have compared some other solutions like Loggly and Logentries, but Google Trends showed similar and smaller numbers to Sumo Logic.)

My takeaways:

Splunk is a global product with reach across Europe, the United States, and Asia, and it is seeing consistent increases in adoption and the number of Google searches

The only real rival of Splunk is the ELK Stack, which is seeing massive global attention and has rapidly grown enough to overtake Splunk

A successful open-source project can propagate faster than most heavily-funded proprietary software

Having said all that, very often, the challenge for such open source projects is transforming from a project to a real enterprise-grade solution.

If these market trends continue, then the log analytics industry will look very different very soon. Just see this tweet that I saw not too long ago:

Next read this:

Tomer Levy is co-founder and CEO of log analytics software company Logz.io. Before founding Logz.io, Tomer was the co-founder and CTO of Intigua and an executive for six years at CheckPoint, managing its Intrusion Prevention System (IPS) Software Blade.