In this class, we’ve talked a lot about how “big data” is now the new big thing, and it certainly creates a buzz in conversation these days. In our final projects, we are getting exposure to using and manipulating big data with ratemyprofessor.com. Companies are starting to use their big data too in order to provide a foundation that leads to making the correct economical decisions to benefit the business. Big data can open the door for new discoveries as companies can analyze relationships between variables that they’ve never assessed in the past.
Big data provides incredible amounts of data, however it’s important to still question the data and the relationships that big data shows. Eric Lundquist comments on big data and enterprise in this blog. He notes that although big data provides tremendous gains for industry growth and opportunity, we need to make sure we don’t get rid of the human factor in making important business decisions. He notes that “listening” to the human gut is important, and that we can’t just rely on the statistical methods that we implement on this big data alone.
I think the human element in analysis is really important in trend discovery on for purposes of error checking and ensuring the quality and precision of the data. I generally agree that modeling data can provide new insights into data relationships unnoticeable to the human eye, however we shall not undermine the power of the human when it comes to qualitative analysis. When tackling big data, we can check for precision by first implementing our model on the data. The computers and models will spit out trend information that tries to best fit the data numerically. Once that is all complete, the human is then encouraged to observe the output and take it with a grain of salt. Is the model really displaying a phenomenon that is possible/observable in the outside world (the source of the data)? When we subset the data smaller in trying to make our models precise as possible, we can look at the trends in the larger subsets in order to check that the smaller subsets are showing trends we more or less expected to see in the data.
We are encouraged to the do same error checking measures in assessing our easiness vs. quality relationships. First we can compare results across subject levels, and from there we can delve further into our data and look at differing schools and subject levels and see how the relationships compare, ensuring that the additional findings we unlock in the smaller subsets make sense compared to the relationships we discovered in the larger subsets. Here, we are utilizing human analytics while still incorporating help and new insights given to us from our models.
Lundquist’s advice to enterprise in working with big data and knowing where to start is this: “Pay attention to Silver’s process, but be equally assertive in looking at how your company operates. You will probably find lots of silos of activity where each group tends to use the same measures and methods year after year. Your job is to think outside the box, think like a customer and consider all the influences that would go into a purchasing decision. Understand the influences and you will be on your way to developing a prediction model that actually works for your business.” From this advice, we can take away that predictive models and big data opens the door to huge gains in industry, however, we still must not make our decisions too fast, and we should always be skeptical of the model and ensure not to completely eliminate the analytic power of the human brain regarding trend discovery.