Home    Bloggers    Messages
Tw  |  Fb  |  In  |  Rss
John W. Verity

Graphs: Big-Data's Next Frontier

John W. Verity
Page 1 / 2   >   >>
@AlanMorrison
@AlanMorrison
7/26/2012 7:49:13 PM
User Rank
Basic Coder
See our July 2012 interview with YarcData
Hey, John. Nice post. We had the opportunity to interview Arvind Parthasarathi of YarcData recently. See http://www.pwc.com/en_US/us/technology-forecast/2012/issue1/interviews/interview-arvind-parthasarathi-president-yarcdata.jhtml for the excerpt.


What we tried to do was ferret out some use cases that exemplify how data in graph form lends itself to certain kinds of discovery. Arvind, for instance, gave this example:


"Think about what constitutes a typical weapon—a gun, for instance. A gun basically has a control mechanism—I can pick it up and aim it. It has a targeting mechanism to facilitate the aiming. It has a guidance mechanism—I can pull the trigger, and it can project something in a particular direction. And it has a delivery mechanism—the ability to hit a target. So we know intuitively that general description of a gun also fits a rocket-propelled grenade or bazooka or missile.

But if you think about it from a different perspective, a plane actually has all those characteristics, too. It has a control mechanism—a pilot sitting in a plane. It has the ability to propel itself and to target because it can be guided. And it has detonation capability. So if you take the gun's characteristics, relationship analytics will help you find things such as planes that could be used as weapons, things you hadn't thought about before in that context."

Discovery of unknown unknowns.... No wonder intelligence agencies and medical research institutes are interested.

Another aspect of this is purely on the high-performance computing side. The claim is that distributed memory and x86 architectures cannot process large graphs efficiently--shared memory OS and purpose-built processors from a supercomputer provider are needed. This Sandia Labs paper explores the claims of the basic architecture (reviewing the performance of some earlier Cray products) and provides some degree of confirmation: http://www.sandia.gov/~bahendr/papers/graphs-and-machines.pdf

This is supercomputer technology, but customers lease, rather than buy it. Opex rather than capex, marketed as an "in-memory appliance".... A horizontal sales model more accessible to mainstream enterprises.... Uses the open source semantic web stack and any visualization tool designed to work with that stack, so you could imagine integration of a range of heterogeneous datasets with this approach.

 

50%
50%
techie
techie
5/31/2012 10:29:12 AM
User Rank
Basic Coder
Re: Special hardware
Ae they still into hardwae memory systems ? I thought they gave it up long time back.

50%
50%
John W. Verity
John W. Verity
5/30/2012 10:47:10 PM
User Rank
Blogger
Re: Special hardware
Actually, it's not clear what Cray builds anymore. I think their main contribution in hardware is in memory systems and interconnects, to join together hundreds of processors. And software to make it all work, of course. 

50%
50%
techie
techie
5/28/2012 10:27:05 AM
User Rank
Basic Coder
Re: Special hardware
Yes when everyone is on the lookout for profits why not them isnt it ?

50%
50%
John W. Verity
John W. Verity
5/25/2012 5:39:13 PM
User Rank
Blogger
Special hardware
It's still not clear to me how Cray's hardware has been specialized to work with graph data. I suppose it's a matter of I/O, or the fabric connecting the many processors in this box. I can't see that they would build a specialized processor chip just for this limited market.

50%
50%
philtheswguy
philtheswguy
5/22/2012 10:42:36 AM
User Rank
Basic Coder
Re: Big Weather
Maybe the accuracy past 2-3 days will improve too!

50%
50%
techie
techie
5/22/2012 10:27:25 AM
User Rank
Basic Coder
Re: Rough Weather at Facebook
Yes true enough I think they wanted to create some sort of a hype in the market to generate some more traffic. Greedy FB

50%
50%
munira
munira
5/21/2012 12:51:12 AM
User Rank
Basic Coder
Re: Big Weather
Thanks John. The index cards example explains it well.

50%
50%
John W. Verity
John W. Verity
5/20/2012 7:17:50 PM
User Rank
Blogger
Re: Big Weather
Yes, this is all about marketing, not number crunching. Think of it as a giant box of index cards, all cross-indexed in many different ways, and this appliance is able to sort and sift through the cards at blinding speed.

50%
50%
Innovator
Innovator
5/20/2012 3:04:17 PM
User Rank
Management GUI
Social Media Analytics and SuperComputers

John It is really amazing the computing power of Supercomputers, and thank you for bring up the Grandaddy of all supercomputer makers - Cray.  I can see the usefulness of these types of machines as it relates to the measurement and understanding of these vast networks comprised of millions/billions of nodes.    But when you apply this power to the tasks of real-time processing of tweets (for example) and other social media based data, I agree I am not so sure there will be much value.

 

But one can rest assured that elements of this will be applied to "everyday big data"  once the cost point has lowered.   Hopefully by that time, the field of (Social Media ) Analytics will have evolved enough to make sense of the supercomputer based findings.


50%
50%
Page 1 / 2   >   >>
More Blogs from John W. Verity
The famous novelist Ray Bradbury is dead, leaving behind one of the most memorable analyses of information technology ever written.
A company called Coraid has figured out how to build storage networks on Ethernet, saving users serious money.
News out of Inner Mongolia gives hope that some IT interface standards issues may actually get resolved in this millennium.
Securing APIs and IT services is crucial, and of all people, Niccolo Machiavelli (author of The Prince) had some important things to say about it.
The Flame virus, which has been discovered attacking Middle Eastern computers, is the stuff of good spy novels.
flash poll
ITSC on twitter
like us on facebook
IT Software Community    About Us     Contact Us     Help     Register     Twitter     Facebook     RSS