The “Human Element” in Big Data and Predictive Analytics

 

In this class, we’ve talked a lot about how “big data” is now the new big thing, and it certainly creates a buzz in conversation these days.  In our final projects, we are getting exposure to using and manipulating big data with ratemyprofessor.com.  Companies are starting to use their big data too in order to provide a foundation that leads to making the correct economical decisions to benefit the business.  Big data can open the door for new discoveries as companies can analyze relationships between variables that they’ve never assessed in the past.

Big data provides incredible amounts of data, however it’s important to still question the data and the relationships that big data shows.  Eric Lundquist comments on big data and enterprise in this blog.  He notes that although big data provides tremendous gains for industry growth and opportunity, we need to make sure we don’t get rid of the human factor in making important business decisions.  He notes that “listening” to the human gut is important, and that we can’t just rely on the statistical methods that we implement on this big data alone.

I think the human element in analysis is really important in trend discovery on for purposes of error checking and ensuring the quality and precision of the data.  I generally agree that modeling data can provide new insights into data relationships unnoticeable to the human eye, however we shall not undermine the power of the human when it comes to qualitative analysis.  When tackling big data, we can check for precision by first implementing our model on the data.  The computers and models will spit out trend information that tries to best fit the data numerically.  Once that is all complete, the human is then encouraged to observe the output and take it with a grain of salt.  Is the model really displaying a phenomenon that is possible/observable in the outside world (the source of the data)?  When we subset the data smaller in trying to make our models precise as possible, we can look at the trends in the larger subsets in order to check that the smaller subsets are showing trends we more or less expected to see in the data.

We are encouraged to the do same error checking measures in assessing our easiness vs. quality relationships.  First we can compare results across subject levels, and from there we can delve further into our data and look at differing schools and subject levels and see how the relationships compare, ensuring that the additional findings we unlock in the smaller subsets make sense compared to the relationships we discovered in the larger subsets.  Here, we are utilizing human analytics while still incorporating help and new insights given to us from our models.

Lundquist’s advice to enterprise in working with big data and knowing where to start is this: “Pay attention to Silver’s process, but be equally assertive in looking at how your company operates. You will probably find lots of silos of activity where each group tends to use the same measures and methods year after year. Your job is to think outside the box, think like a customer and consider all the influences that would go into a purchasing decision. Understand the influences and you will be on your way to developing a prediction model that actually works for your business.”  From this advice, we can take away that predictive models and big data opens the door to huge gains in industry, however, we still must not make our decisions too fast, and we should always be skeptical of the model and ensure not to completely eliminate the analytic power of the human brain regarding trend discovery.

Source:

http://www.informationweek.com/global-cio/interviews/nate-silvers-big-data-lessons-for-the-en/240062638

Advertisements

About cbaughma

Senior at University of Michigan studying Informatics: Data Mining & Information Analysis
This entry was posted in Uncategorized. Bookmark the permalink.

One Response to The “Human Element” in Big Data and Predictive Analytics

  1. hoffalex says:

    It is interesting to think about the limitations of big data, considering how much of this course is focused on its applications. While big data analysis can be a useful tool for understanding statistical trends, I believe that it is also important not to forget that, as of now, there are still certain tasks that can be better accomplished by humans. It is important to note, of course, that this may not be true forever. With the advancement of artificial intelligence, it may eventually be possible to have the data checked and analyzed by another computer program without requiring the intervention of humans at all.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s