Everyone agrees that big-data is big business. But how big is it? That's the question.
Big-data, as just about everyone knows by now, is all about sifting through mountains of digital information to find telltale patterns and correlations. Harness up a team of servers, give each one as much RAM as it can handle, wire up some disk drives, add software with names like Hadoop, tk, and Cassandra, and off you go.
Nobody is arguing with the notion that this is a business poised to take off like a rocket. The world is brimming with data right now, and the cost of the computing infrastructure needed to grind through it is continually falling. In short, there's a perfect storm of supply and demand, of opportunity and motive.
But how big is big? International Data Corp. (IDC) has just come out with a report concluding that revenue for the worldwide Hadoop-MapReduce ecosystem software market (aka big-data) will grow from $77 million in 2011 to $812.8 million in 2016, for a compound annual growth rate of 60.2 percent. In my book, that certainly registers as big and getting bigger.
Yet Internet Research Group, another market researcher, finds that big-data revenue easily topped $300 million in 2011 and is on its way to reaching $1.45 billion in 2016. Those numbers are a good deal higher than IDC's. The reasons for that are not entirely clear. Perhaps IDC is measuring solely software-related revenue while IRG is including hardware, as well.
But guess what? For the purposes of those using or simply eyeing big-data in enterprise IT, the exact amount of spending matters little.
What's obvious, even with no figures at all to report, is that big-data techniques are establishing themselves as standard operating procedure. What we're witnessing right now is the emergence of a whole new set of technologies, services, companies, and techniques that will root themselves in enterprise IT much like the ecosystem that has grown up around the relational database management system.
Quoting IDC's figures, IRG forecasts that toward the end of next year, enterprise spending on big-data will start to exceed that of its original users, big Websites like Yahoo and eBay.
As IDC researchers point out, revenue in the big-data marketplace will grow as it has in the Linux business. As with Linux, the main pillars of big-data are open-source programs like Hadoop and MapReduce, which overall will keep software revenue lower than it might be in a market dependent on commercial products. In short, don't expect another Oracle or Sybase to emerge, as happened in the relational database market.
Dan Vesset, vice president of business analytics solutions at IDC, said in a press relase:
Over the next decade, much of the revenue will be accrued by hardware, applications, and application development and deployment software vendors -- both established IT providers and start-up, which in aggregate have raised more than $300 million in venture capital funding.
Much venture capital has been flowing into this area. Late last year, for instance, Accel Partners launched a $100 million fund dedicated solely to investment in big-data startups.
How has your organization been using or planning for big-data?