Big Data: A State of the Union

Big Data. The keyword of the decade. Yet, it’s barely understood or often confused by so many. The idea behind Big Data is simple. Collect all the data you can (Volume), from multiple sources in any and all formats (Variety), as fast as you can (Velocity), and try to be sure the source provides useful, accurate and clean data (Veracity). The 4 V’s just mentioned, form the basis of Big Data.

A simple spreadsheet with 100 million rows and 1000 columns doesn’t make Big Data. Big Data is the steady flow of relevant data, in any format like texts, numbers, audio clips, video clips, tweets, pokes, or anything else that can be used to make business decision.

Current Market

Since its introduction, Big Data has taken the world by a storm. The market for Big Data stood at $23 billion for 2015, and $27 billion in 2016. Another report by Cap Gemini found that 56% of companies expect to increase their spending on big data in the next three years.

All this investment and business finds applications in government regulations, medicine and health, finance, security, customer service, advertisement optimization, business intelligence and lastly, but largely, into The Internet of Things (IoT).

Future Opportunities

All the predictions for Big Data show exponential growth over next decade. The current market is just at the initial stages of exploding. It is estimated that Big Data will drive $48.6 billion in annual spending by 2019 and $92.2 billion by 2026. This business and revenue will largely be directed towards IoT into the future as the applications out of IoT expand and merge a lot of other fields of application.

The basis for this success lies in the fact that the size of data will keep going up, as companies collect more and more data. This data will need to provide relevant ROI. At the same time, the cost of analyzing this data will reduce as new technologies emerge to do it better, faster, and in a more secure way. This will eventually push for in-depth and real-time analytics.

Hadoop and its relating technologies are creating their own niche in the Big Data market, becoming the leading technology for dealing with raw unstructured data. According to Allied Market Research, the Hadoop Market was valued at $1.5 billion in 2012 and is expected to grow up to $50.2 billion in 2020.

Social Good

Most of the focus of Big Data has been towards business development and progress. But an important vein out of Big Data will be aiming for social good. The two big concentration points will be the education field, and the government.

For education, the idea is to create a learning environment that feeds from already existing information on the students. A student that has been known to have issues grasping words (through previously recorded data) will be provided special tools that matches the student’s skill level. Students with health issues may be given a tracking device like the ones that are already available, to monitor their health at all times and make real-time decision if something were to happen. Global learning can be made easy through live data streaming, and responses can be captured to make sure students actually grasped what was being presented. All this knowledge and use of technology will create a customized experience for individual students, helping them learn better and eventually create a better social environment.

For Government, improving analytics from Big Data in Government field will have good impacts on a lot of other fields like medicine and health, security, and crime. Being able to track outlier transactions can help catch money launderers, terrorist groups, and stop the transactions in record time. Sensors and cameras with better tracking ability can help track criminals. Big Data can also improve emergency response time by tracking social media trends. Providing wearable tracking devices that are already available, to the elderly, can monitor their health in real-time, and can efficiently direct social workers. Also tracking of previous treatments, allergies, and other basic information can reduce risk of errors in treatment.

Despite all the current challenges of Big Data, it will be where the future lies for us.