Everyone agrees that big-data is big business. But how big is it? That's the question.
Big-data, as just about everyone knows by now, is all about sifting through mountains of digital information to find telltale patterns and correlations. Harness up a team of servers, give each one as much RAM as it can handle, wire up some disk drives, add software with names like Hadoop, tk, and Cassandra, and off you go.
Nobody is arguing with the notion that this is a business poised to take off like a rocket. The world is brimming with data right now, and the cost of the computing infrastructure needed to grind through it is continually falling. In short, there's a perfect storm of supply and demand, of opportunity and motive.
But how big is big? International Data Corp. (IDC) has just come out with a report concluding that revenue for the worldwide Hadoop-MapReduce ecosystem software market (aka big-data) will grow from $77 million in 2011 to $812.8 million in 2016, for a compound annual growth rate of 60.2 percent. In my book, that certainly registers as big and getting bigger.
Yet Internet Research Group, another market researcher, finds that big-data revenue easily topped $300 million in 2011 and is on its way to reaching $1.45 billion in 2016. Those numbers are a good deal higher than IDC's. The reasons for that are not entirely clear. Perhaps IDC is measuring solely software-related revenue while IRG is including hardware, as well.
But guess what? For the purposes of those using or simply eyeing big-data in enterprise IT, the exact amount of spending matters little.
What's obvious, even with no figures at all to report, is that big-data techniques are establishing themselves as standard operating procedure. What we're witnessing right now is the emergence of a whole new set of technologies, services, companies, and techniques that will root themselves in enterprise IT much like the ecosystem that has grown up around the relational database management system.
Quoting IDC's figures, IRG forecasts that toward the end of next year, enterprise spending on big-data will start to exceed that of its original users, big Websites like Yahoo and eBay.
As IDC researchers point out, revenue in the big-data marketplace will grow as it has in the Linux business. As with Linux, the main pillars of big-data are open-source programs like Hadoop and MapReduce, which overall will keep software revenue lower than it might be in a market dependent on commercial products. In short, don't expect another Oracle or Sybase to emerge, as happened in the relational database market.
Dan Vesset, vice president of business analytics solutions at IDC, said in a press relase:
Over the next decade, much of the revenue will be accrued by hardware, applications, and application development and deployment software vendors -- both established IT providers and start-up, which in aggregate have raised more than $300 million in venture capital funding.
Much venture capital has been flowing into this area. Late last year, for instance, Accel Partners launched a $100 million fund dedicated solely to investment in big-data startups.
How has your organization been using or planning for big-data?
sohaibmasood 6/3/2012 9:38:40 AM User Rank Basic Coder
Re: P_redicting Behaviour
I agree, but then prediction a groups behaviour stems from predicting an individuals behaviour.
Like you said we cannot account for free will in our results and therefore we have to focus on accuracy of the existing data set (inclusive of free will) and hope that the tolerance is a bare minimum.
I guess the people who "do" big-data aren't aiming to predict any particular person's behavior, they're just trying to tilt the roulette wheel a little in their own favor. If you can be X% more accurate in your predictions, you can make that much more money and be that much more effective. Free will will always intervene, but statistically people's behavior can be roughly predicted.
sohaibmasood 5/21/2012 11:09:30 AM User Rank Basic Coder
Re: P_redicting Behaviour
Indeed, prediction is an interesting subject. I think predicting human behaviour is a difficult task because of the element of free will. Even with heuristic based search algorithms predicting exact human behaviour in a given situation is difficult.
nasimson 5/21/2012 10:02:02 AM User Rank Management GUI
Re: Big-Data: Big & Getting Bigger
I believe Big data has potential for great use in transportation and logistics. Regions and countries with thick populations and growing middle classes have ever increasing car owners, Big data can be used in these situations to map out a way of decreasing their carbon footprint and at the same time manage growing road structures. Plus theft can also be monitored and curbed this way, as increase in car usage increases the rate of theft also.
Prediction is an interesting notion, for sure, but in the context of big-data, it seems to assume that the world, or people, are mere deterministic machines. Identify the force vectors affecting them and you can predict their future trajectories, like so many billiard balls. Free will? No such thing!
@Seth, that's an interesting application of big data, figuring out where ambulances should go. I had never thought of that.
Long ago, my father, who knew his way around computers, told me how he reckoned that one day they'd figure out what causes cancer, for instance, by just building a program that did a regression analysis on piles of data about what people ate, where they lived, what chemicals were in their lives, and so forth and so on, and this would reveal the most significant factors. I am not sure that is possible, even now, but the idea must surely be kicking around in some researcher's brain. The difficult, I suppose, is the inaccuracy of people's reporting on themselves. Besides which, it's pretty clear to me that many risk factors are fairly obvious, and no computer is needed to identify them - even if certain parties don't want to admit that they are contributing to the rise of such factors.
Toby 5/10/2012 4:46:10 AM User Rank Management GUI
P_redicting Behaviour
@John: This would seem to point towards better and more precise means to gather and analyze predictors of user behaviour, key to marketing the next gizmo or gadget, also any number of services. The longer term benefits of analyzing behaviour of peoples and nations as a whole would seem to be an obvious step for governments who, doubtless in their own good time, will pour even more money into this area of analytics.
Henrisha 5/9/2012 10:39:13 PM User Rank Basic Coder
Re: Big-Data: Big & Getting Bigger
Now that we're all aware that big data is now here--and that it really is big--then it's time to figure out how to store, handle, and manage it to prevent resources from getting overwhelmed so that we can really put it to good use.
Like Seth said, it has a huge impact on the healthcare industry because they're one of the largest consumers of data. Eager to know where this road will eventually take them when it comes to innovating their services.
cpafern 5/8/2012 11:37:34 PM User Rank Basic Coder
Re: Health Care
John and Seth - do you think with Big Data, there is a possiblity of reducing the information overload we are all suffering by doing a more concise job of filtering the information we are getting overloaded by?
SethGB 5/8/2012 6:04:00 PM User Rank Management GUI
Health Care
Our hospital is using big data in many ways. For instance, we are using big data to research if procedures are really obtaining the benefits intended, if medicines are combining in harmful ways, or even in unintended beneficial ways. One of the greatest benfefits of data is we are now obtaning how lifestyle choices over large populations are affecting people's outcomes after procedures and affecting medicines effectiviness.
Big data can help in unsupecting ways. For example an ambulance company can see which local hospital may be best to treat a patients medical condition, not just based on expertise but current staffing and capacity.
To save this item to your list of favorite IT Software Community content so you can find it later in your Profile page, click the "Save It" button next to the item.
If you found this interesting or useful, please use the links to the services below to share it with other readers. You will need a free account with each service to share an item via that service.
Tweet This
