Big Data’s Biggest Challenge? Convincing People NOT to Trust Their Judgment
Here’s a simple rule for the second machine age we’re in now: as the amount of data goes up, the importance of human judgment should go down. It reads like heresy, doesn’t it? Management education today is largely about educating for judgment. We’re told to trust our guts and instincts, and that (especially after we gain experience) we can make accurate assessments in a blink.
This is the most harmful misconception in the business world today (maybe in the world full stop). Human intuition is real, but it’s also really faulty. Human parole boards do much worse than simple formulas at determining which prisoners should be let back on the streets. Highly trained pathologists don’t do as good a job as image analysis software at diagnosing breast cancer. Purchasing professionals do worse than a straightforward algorithm predicting which suppliers will perform well. America’s top legal scholars were outperformed by a data-driven decision rule at predicting a year’s worth of Supreme Court case votes. Read entire article here>