By Theresa Caragol
Our society is creating an unprecedented and massive data resource in cyberspace with each passing second. However, this growing mountain only turns into a resource when there is some means to mine or exploit it — and that is where big data comes in. Big data is already a hot topic, and more and more organizations are looking at their mounting data and wondering if there might be “gold in those hills." Some of the biggest data mountains belong to media, manufacturing, and government. However, these massive databases are building up everywhere such as online retail giants like Amazon and the less obvious smaller retail operations, especially those with loyalty cards.
Currently all this data is not moving anywhere, and there is a growing appreciation that this massive burden might contain within it valuable insights, but if only there was some way to extract them.
A serious commitment to big data in most cases requires a rethink of the network architecture, and technology organizations will be looking for help and guidance towards agile, scalable and cost-effective solutions spelling big business for the channel. The first thing organizations looking to capitalize on this potential need to know is that big data is not a single problem needing to be solved, so it demands a strategy more than any one solution to overcome its challenges.
The Challenges of Big Data
The first challenge with big data is the growing demands for compliance making it mandatory for organizations to preserve records as proof they have conformed to financial, fiduciary, health, safety and other regulations. This data often must be stored infinitely and, what’s more, stored at least twice to ensure backup for disaster recovery creating a huge burden.
The second difficulty in mining big data is managing this sheer volume of data — not just the storage of it, but also the processing power needed to analyze it. An extreme example is the CERN Large Hadron Collider where physicists need to sift through 15 petabytes every year to see if the collisions have thrown up any significant results. Not even CERN has the computing power to do this, and so they rely on a worldwide grid of computers to crunch those numbers.
The third obstacle is speed. CERN cannot wait forever for those results, but a delay of days or weeks makes little difference in the progress of science. Compare that with a cybersecurity operation where an hour of delay could give the criminal a window of opportunity or a financial system where even one second could lose a fortune.