(Image Credit: iStockPhoto/MaFelipe)
Big data can mean different things to different people, but it is commonly defined by the three Vs: volume, variety and velocity. Today organizations face not only growing volumes of data but also an expanding range of data sources and faster data creation rates. Managing all three dimensions at once presents a significant challenge.
Many businesses have progressed unevenly through these dimensions. While each V is important, most organizations remain in the earliest stage: dealing with the sheer volume of data. They are still focused on collecting and reporting massive datasets and have not yet fully addressed the next challenges—normalizing diverse data sources and responding fast enough to turn incoming data into timely insight.
How can businesses move beyond the first stage and take advantage of the full potential of big data?
The first wave: the data tsunami
Data management becomes more difficult each year. According to IDC, 90% of all existing digital data was created in the last two years, and the total volume of digital data is projected to double roughly every two years. That explosion of information forces companies to rethink how they store and report on data, since legacy data warehouses were not designed for such scale. Many big data initiatives begin with the need to replace or augment those traditional systems, and the initial focus is often on bringing order to burgeoning volumes and managing the influx of information.
Hadoop remains central to these early projects, enabling organizations to store and process vast amounts of data on commodity hardware at lower cost. Investments in Hadoop-centric platforms (from vendors such as Hortonworks, Cloudera and MapR) are typically driven by a clear business case: reduce storage costs and plan for the escalating volumes of data. This makes it possible to launch projects and capture more data, but it can also lead to a narrow focus on collection over insight. Many organizations get stuck here—storing and aggregating data without yet extracting the intelligence needed to drive meaningful business outcomes.
The second wave: data diversity
Once volume is addressed, attention must turn to variety. The number and types of data sources continue to grow—Internet of Things sensors, smartphones, social media feeds and video streams are just a few examples. Many of these sources did not exist five years ago, yet they now influence networks and provide important contextual information.
IDC reports that 90% of current digital data is unstructured, often arriving in incompatible formats that are difficult to integrate into conventional analytics systems. As connected devices and IoT sensors multiply, both the variety and complexity of data will increase along with volume. Different data streams also introduce noise: not all data is equally valuable. Organizations that progress to the second wave learn to evaluate data at the source. They use streaming analytics to decide what to store and analyze and what to ignore—sifting for the most valuable signals rather than retaining everything indiscriminately.
The third wave: surf’s up!
The final wave brings velocity to the forefront. While many companies collect and report data and a growing number apply streaming analytics to filter variety, relatively few can operate in real time. Velocity is about how quickly organizations can convert data into actionable insight and then embed those insights into business processes. Companies that master this wave create automation triggers and integrate analytics directly into operational workflows, accelerating decision-making and improving responsiveness.
Apache Spark is one example of a technology that supports this transition, enabling real-time processing that complements Hadoop’s batch-oriented capabilities. As technologies such as Software Defined Networking (SDN) and Network Function Virtualization (NFV) mature, real-time automation supported by operational intelligence from big data platforms will become increasingly important.
On the crest of a wave
As the market matures, organizations will move from one wave to the next, and the real benefits of big data will emerge. The next stage will likely emphasize predictive analytics and machine-driven control loops, advancing a more data-driven, value-focused future. The swell is coming; companies that prepare their leadership and infrastructure now will be better positioned to capitalize on what’s next.
Do you think businesses should “ride on the crest” of the big data wave? Let us know in the comments.
If you’d like to learn more about Big Data & Security, consider attending IoT Tech Expo Europe at London’s Olympia, 2–3 December 2015.