Is Hadoop Replacing the Data Warehouse? Survey Says Not So Much

Snowflake Computing, a cloud data warehousing company that only recently emerged from startup stealth mode, has announced the results of an independent, national survey of more than 315 technology and analytics professionals with responsibility for corporate data initiatives. Conducted by Dimensional Research, the goal of the research was to understand the state of the data warehouse and Big Data initiatives – including experiences, challenges and trends in data warehousing and data analytics.

The survey turned up some surprises. For example, nearly two thirds of the respondents said that they believe Hadoop will not have any impact on their legacy data environments.

“The State of the Data Warehouse” research findings indicate that data warehousing is still viewed as a critical business component, in spite of its challenges. An overwhelming majority of data professionals characterize their data warehouse as very important for business operations (72 percent), and expect an increasing level of investment in data warehousing at their organization (70 percent). However, 97 percent of respondents stated major challenges when it comes to data warehousing.

Despite these challenges, Hadoop is not viewed as a practical replacement for the data warehouse. A majority of data professionals (64 percent) would not consider Hadoop as a possible replacement for any portion of their data warehouse. In fact, 91 percent expressed concerns about using Hadoop, with 71 percent citing lack of specialized expertise as a key concern.

In addition, interest in Big Data looks to ber high but still in its early stages. The study showed that while 70 percent of respondents were interested in a Big Data initiative, only five percent of respondents actually had a fully deployed initiative and just 11 percent are in pilot.

“Hype has created confusion about what is really happening with Big Data initiatives,” said Diane Hagglund, principal at Dimensional Research. “Although interest in Big Data is high, this study found that most organizations still have not implemented Big Data initiatives. Further, this study found that Big Data initiatives and Hadoop have not diminished the importance of the data warehouse–data warehousing remains critically important and will not be replaced by Hadoop.”

Here is a summary of some of the key survey findings:

- 99 percent of respondents say their data warehouse is important to business operations

- 70 percent are increasing their investment in data warehousing

- 97 percent face challenges with their current data warehousing solutions

- 91 percent have considered a Big Data investment

- Yet only 11 percent have a pilot in place, and only five percent have fully deployed their Big Data initiative

- 96 percent say Hadoop will not replace their existing data warehouse

- Just 12 percent have easy access to Hadoop expertise (by contrast, 93 percent have easy access to SQL expertise)

-  93 percent see value in the potential benefits of cloud data warehousing

- 32 percent have a cloud-based data warehouse today

- 79 percent of those with data in the cloud bring it into the data warehouse, although only nine percent bring all data in

“Although the Big Data buzz has led to speculation about the future of data warehousing, this survey confirms that the ‘death of the data warehouse’ has been greatly exaggerated,” said Bob Muglia, CEO of Snowflake Computing. “The data warehouse remains critically important, but as this survey shows the market has a critical need for a modern data warehouse, one designed to take advantage of the cloud and eliminate the complexity of current solutions, without requiring scarce new skills and expertise.”

To access the full Dimensional Research survey report on “The State of the Data Warehouse,” you can go to: http://info.snowflake.net/StateOfTheDataWarehouse.html