Considering the effect big data has made on society today, I think it’s important to inform people of society’s misconceptions concerning the amount of data we have and the processes we’ve implemented to use such data. Michael Healey points out a couple of the commonly misperceived ideas, which I’ve summarized below:
Lie 1: We understand how much data we have today
People don’t consider the information floating around on the cloud or on mobile devices, which account for about 35% of the data the public uses. We can gain a lot of insight by looking at such data, but instead it lies untouched because people do not know it is there to tamper with.
Lie 2: The data we have is good
Not many people working with big data check the data for accuracy. For example, in many company data reports, no confidence intervals or bias results are reported. Data accuracy and precision is important to make sure the data we’re using is tackling the issues we’re trying to uncover.
Lie 3: Everything will be OK if we can just get more tools
Many new data manipulation tools such as Hadoop and NoSQL have become popular analysis tools to tackle large data and seek out relationships among differing market variables; however, such tools will not do the work alone. Instead, we need trained individuals who are comfortable working with discovering trends in data to uncover meaning out of what there is to work with. Yes, Hadoop and similar tools will certainly help trained individuals, but these individuals must also understand the industry to gain any sort of headway.
Lie 4: There’s an expertise shortage
This can be related to what was discussed above, but ultimately, IT not only has to understand the data itself, but IT teams also must work at identifying and growing the centralized talent pool of data analysts capable of working with large quantities of data. The talent is there, but the training is not. Healey’s solution to this is that IT departments should “put more emphasis on training and talent development.
Lie 5: We know what data we need
Many companies today are not taking advantage of the data that they have because they do not have designated people to keep track of such data. Unbeknownst to them though, a lot can be discovered just by looking at a couple of key variables such as CRM, phone, email, and Web analytics data. Tying such data together can provide insight as to how email relates to phone calls and Web traffic. Healey mentions that, “all of that intelligence exists today, yet few companies have this level of analysis integrated into their big data strategies.”
Lie 6: We do something with our analysis
In using big data, companies often only focus on the good things that the data highlights, rather than looking at the negative things that big data can help a company improve upon. In reality, big data will find some negative things (not just positive)–about a company’s sales team’s effectiveness, their online presence, or their true costs of operations. Knowing that big data analysis can uncover such things, a company should use it to help find and prioritize the problems they need to fix, instead of just using this data to report positive data trends concerning the company.
The lies about big data I discussed above can be changed once we discover ways to work with the data and get around the misconceptions. For now, I think it’s important to let IT teams know of the existing problems many companies have regarding big data, in addition to enforcing the importance of continually working with the data rather than only looking at it during discrete time intervals.