Destructive Analytics

In this era of big data, we have become infatuated with the promise of how we can improve every aspect of our lives by better collecting data, analyzing it, and managing our lives to various data points. In business if you haven't heard someone in your organization parrot Peter Drucker's mantra that "you can't manage what you don't measure," then you must be running short on fresh MBA's.

While I won't argue that there isn't some truth to this, this path of managing by measurement seems to repeatably lead us to the land of unintended consequences.  Have some patience with me as I go a bit abstract for a minute. The pattern seems to be something like the following:

1. We want situation X to get better
2. We examine various situations in which X gets better
3. We identify the history of this case where X got better, and we identify data points that we can follow like the yellow brick road that led to the improvement of X
4. We move to the new case where we want to see X improve and get a baseline on the data points
5. We then take actions to see progress on those data points, with the expectation that this will lead us to X improving

We expect to see (6) 'X improved,' but this is where we end up in the land of unintended consequences. The people involved in the process become aware that what they need to do is hit the various data points, and what they do is bend the process in whatever way they need to in order to hit those data points.  Unfortunately, the data points are hit, but 'X is not improved'.

It is as if there is some social version of the Heisenberg Uncertainty Principle at work where once we're consciously measuring for particular data points as indicators as progress to 'X improved,' those data points are no longer a true indicator of that progress.  Instead achieving the data points themselves become the end in focus, and in so doing, they are no longer the means to the true improvement initially desired.

Let me give a few concrete examples of this.

1. Testing in schools.  We've wanted to improve the quality of our schools, so we've looked to standardized testing to provide concrete measuring blocks of progress.  Instead of improving education however, its led to the extreme bending of educational process to "teach to the test."  Those whose livelihood is tied to this performance now focus on developing good test takers rather than students with good fundamental skills because they have been given that as their new explicit goal.  I'm sure you have heard of numerous examples of this for yourself.  Here is a recent blog entry of a long-standing teacher in Texas lamenting the impact this has had in her school. http://marybethlee.wordpress.com/2012/03/02/a-plea-from-a-teacher/
   
2. CompStat in policing. In 1994, New York started tracking crimes with a system called CompStat for the first time.  Following the data and modifying policing to address trends initially led to major improvements with serious crime apparently dropping by as much as 77%.  The emphasis on data tracking led to a focus on the number of tickets officers were writing and other minor policing action. This created an environment were police officers were directly or indirectly pressured to ticket and detain individuals without justification.  In worse cases, it led to police intentionally mis-classifying serious crimes as minor crimes or encouraging victims not to officially report them at all.  For one police officer who was interested in truly taking care of his neighborhood and not concerned about the measured numbers, it led to harrassment and even unwilling "hospitalization".  If you haven't heard about the case of Adrian Schoolcraft, I strongly encourage you to pop over and listen to this episode of This American Life.
3. Rebuilding in Iraq. If you like horror stories, you should read: We Meant Well: How I Helped Lose the Battle for the Hearts and Minds of the Iraqi People. OK, it is not technically a horror story.  But in a time of economic troubles we have not witnessed since the Great Depression and the number of families who have lost their homes to foreclosure, to learn of the excessive waste of money our government has committed without positive impact in Iraq is a true horror in its own way.  Peter Van Buren chronicles how the focus in Iraq became on how many projects were being completed, and not what type of projects nor whether they had any positive impact on rebuilding Iraq.  A great example was the creation of a multimillion dollar automated chicken processing plant which created few jobs (because of the automation) and yet was so expensive to operate, that the chickens processed at the plant could not be brought to market in a less expensive means than traditional processes.  Thus millions were spent but ultimately no jobs were created (except those temporary positions to pay people to staff the plant when reporters visited).  I encourage you to read the book, but you can also get a sense of this by listening to an interview of Van Buren on NPR's Fresh Air.
4. Medical Treatment. In our seriously dysfunctional medical industry in the United States, physicians are being measured not on patient outcomes, but on how many patients they are seeing or tests they are ordering. Similarly, technology is being implemented to track both staff and patients literally wherever they are in hospitals.  Now there are good reasons to use this technology for things like patient safety, locating needed resources and facilitating critical fast responses.  But from personal experiences, I know of health care professionals who are also being monitored on how much time they are spending with patients.  The focus once again shifts from what we want improved, operational efficiency in hospitals as evidenced by getting patients healthy, and moves to management data points, namely do the data points show you're an efficient health care provider.
5. Call Centers. Dilbert provides us a humorous example of this taken to the extreme in a call center, but I'm sure those working in call centers where their time is monitored by the minute and calls are recorded could provide some real-life examples as well.  By managing through measurement, the focus gets distorted away from the true purpose of a call center, customer support, and instead gets placed on measurements that show how many callers we're serving.

What's the moral of the story? I don't have any great solution to share on all of this.  I hope that by drawing attention to this problem that has cropped up in such diverse professions as teaching to health care to policing that we'll all be more aware of this common trap. As they say, humans are the only animal that will trip over the same stone twice.  As you look to implement management by measurement in your endeavors, be careful that you are measuring what is truly meaningful, and that by implementing the measuring and management process you aren't actually heading away from your ultimate goal.