What's the Big Deal about Big Data?

Big Data is simply the collection, processing, and analysis of huge data sets from traditional or digital means which deliver business intelligence for companies to enhance their services or create a new competitive advantage.

Raw data which provides at best basic information is generated at an alarming rate of multiplication since 2000. This is fuelled by the emergence of social media and the proliferation of mobile devices leading to mobile computing. 

The crucial challenge for long-term survival and relevance for every business is to convert information to insight and finally to intelligence as quickly and efficiently as possible.

Five Attributes of Big Data 

In order to explain Big Data to many companies, we need to study its attributes and how it affects the future relationship between business and its information collected, analyzed and processed. 

1. Volume

If we take all the information generated by human beings since the beginning of time till 2018, the same amount of data is being generated every minute right now. This results in most dataset becoming too large to store and analyze using traditional database technology.   

2. Velocity

The speed of data generation forces some companies to analyze data on the fly without storing into any system. Data that enters the system become input to other bigger and more complex data analysis.  

3. Value

Collecting and analyzing Big Data is futile unless we can generate massive value in converting information to intelligence. It is not about the speed or volume of the input or processing, it is about the size of the value of the data output. 

4. Veracity

The diversity and credibility of data may cause companies to have less control of the reliability of the content. Data technology now enables information to be mined meaningfully even if there are typo-errors or colloquial speeches. 

5. Variety

In the past, we are geared to analyze structured data like relational databases and financial data. Now, 80% of the data we generate is non-structured like text, images, videos, or voice. With the emergence of social media, most of these unstructured data are messages, conversation, and other media recordings. 

Leverage the 4 Layers of Big Data

Data Source Layer:

Every input to a system can become a source layer for Big Data. It can be as simple as emails and audit trails to as complex as customer service interaction like calls, frequency, duration, locations, etc. The key questions for any business to ask are if new sources of monitoring are needed, or what else do they wish to find out from the existing layer source. The more relationships and co-relationships you see from the data trends, the more important selection of source layers are.

Data Storage Layer:

This is where your Big Data will reside after it is compiled from your diverse sources. As the volume of data generated and stored by companies start to escalate, complex but accessible systems and tools– such as Apache Hadoop DFS (distributed file system), and Google File System – have already been developed to help with this task.

Data Processing/Analysis Layer:

When you want to utilize the data you have stored to filter out something useful, you will need processing and analysis capabilities. A common method is to use a MapReduce tool. Essentially, the tool is used to select the elements of the data that you want to and put it into a format from which insights can be shown. 

Data Out Layer:

This is where the insights gleaned from the analysis are submitted to the corporate users who will see new connections and leverage from them. Clear and concise communication (particularly if your decision makers do not have a background in statistics) is essential, and this output can take the form of reports, charts, figures and key recommendations.

Six Key Skills of Future Specialist of Big Data

To harness the power of Big Data for the future, we will require the new skills of the future. Executives at different levels need to sharpen the following skills in order to stay relevant and competitive. 

Analytical Skills:

The ability to determine what data to collect and how to point out new relationships between different . It is about identifying patterns and links to create connection between data.

Creativity Skills:

The ability to generate new ways of collecting, and interpreting data from current data system performance. It is the art of seeing the behind the analytics. 

Mathematics and Statistics:

The ability to simply crunch traditional numbers will not be sufficient for Big Data. One needs to be able to create and perform scenario planning models. It will also include company, competitors and industry practices. 

Information Technology Skills:

The age of programming may return because of Big Data. New and smarter algorithms will have to be written from systems like Hadoop, Python, Pig to deliver more robust business applications. 

Business Acumen:

In the past, it is sufficient to possess the ability to identify business opportunities.  Now we need to be able to identify future business and information needs and objectives to constantly create opportunities for the company. 

Communication Skill:

The ability to deliver clarity from complexity in different communication modes (presentation, white paper, social media conversation) is key to providing relevant insights to different stakeholders for call-to-action.

In conclusion, Big Data will be a big opportunity for companies who understand its value and application. Just like an army needs intelligence about its enemies to win the war, companies need business intelligence to defeat the speed of business irrelevance to survive.