CIOInsider India Magazine
Dark Data In A Digital Revolution Era!
Established in 1991, Shoppers Stop is one of the pioneers of modern retailing in India. Shoppers Stop offers more than 400 of the finest international and national brands across categories.
Today, we live in a digital society and our distinct imprints are in every interaction we make. In fact data formation is a default today. Buzzword these days: “Data is the new Oil”.
Deep within the enormous volumes of information generated by business transactions, social media, customer connects IoT, sources of intelligence are waiting to be discovered and explored to derive insights. A lot of dark data exist in unstructured formats too, thus making it difficult to categorize and analyse further in its raw form.
Given the gigantic amount of data organizations increasingly generate and hold, it should come as no surprise that most of the organizations do not fully utilize their data's value. In fact, many of these companies may not even necessarily know what kind or how much information they truly have under their control unless they face a scenario where this data could have been used.
Most organizations also knowingly retain dark data for a variety of reasons like regulatory compliance, record keeping, audit logs etc. while others retain from a future usage perspective i.e. once they have acquired better analytics and business intelligence technology to process the information.
With the optimization opportunities coming up in the storage cost, storing data has become flexible, however storing and securing the data usually entails greater expenses (or even risk) than the potential returns.
From a global perspective, there is definitely a room for improvement when it comes to the adoption of data across the functions. This however raises the question, how can organizations take advantage of all their data if they really don’t know what they have or even where it resides?
In a lot of instances, at the time of data collection, the people assume that the data is going to provide value and end up creating a huge chunk of data elements. This results in a high investment both monetarily and otherwise.
Take the example of a bank analysing online applications for credit cards. The credit card marketing team would focus primarily on customer details and eligibility, but if no attention is paid to the data on how the customer arrived at the application page there might be a missed opportunity depending on the priority assigned to this aspect.
The unattended data could have provided valuable insights on the usability of the bank website and the application page. Another very prominent example across organizations would be the data collection and storage process in silos which may not be known to other departments. So, data, even if relevant to other departments, remain unused.
With artificial intelligence and machine learning getting more popular, future of dark data usage looks much brighter
Perspective of untapped opportunity - The image illustrates the notional percentage of dark data that is present at any time across technology solutions.
Dark data represents a huge opportunity for organizations to gain valuable insights which can drive their business. Take a look at the following examples:
• Server log files can provide website visitor behaviour
• Customer call/email records reveal customer sentiments and feelings
• Mobile geo-location data can provide traffic patterns
Not all companies have started tapping this aspect of data yet. It is also a fact that there is a need for strategic vision, better processes, coordination and technologies to appropriately use dark data.
Some food for thought:
• Does dark data have a shelf life?
Yes, it does! Useful data may become dark data after it becomes irrelevant, as it is not processed fast enough during its relevant life time. For example, if the geo-location of a customer is known, the business can make offer based on the location while the customer is there, however if this data is not processed immediately, it may be irrelevant in the future for the said purpose.
• Can storing dark data impact organizations?
Yes and No, depends on variety of reasons like infrastructure cost impacts, risk of data loss due to sensitive nature of data, volume etc.
• Does data have nationality?
A new combat and a point to debate! Should the origin of the entities handling personal data of individuals matter when deciding their rights over processing the data?
Solutions to the Dark Data Problem
With Artificial Intelligence and Machine learning getting more popular, future of dark data usage looks much brighter than what it was until now. As we advance the AI/ML adoption curve, dark data would gain higher value and momentum.
Software’s that converts dark data to graphics or programs that automatically organizes data into easy-to-understand graphics are being explored and are in process of adoption by organizations who have advanced in their analytical curve.
In this race to acquire and utilize the newest and most valuable data, new and better technologies will continue to emerge. Data and analytics will be the foundation of the modern business revolution in the next few years. Most organizations have also started creating a case for adding the role of Chief Data Officer, or CDO, in their organization structure for improving business outcomes by making business more data-driven. Given the matrix structure in most organizations today, CDO role is expected to be strategic, influential and a problem solver with primary objectives being managing information assets, delivering insights to improve decision making and generating incremental business value by managing data like any other revenue-generating asset.
So, here’s the bottom line: If you have not considered all your data to be an asset, it’s probably time to start afresh! Treat all your data the way you would have done with any other valuable asset. Figure out exactly what you have, where you have it, how you manage it and how you can leverage it to benefit your bottom line.
