Navigating the Out-of-Sample Dilemma

Suppose that you’re a very good driver. Almost everyone thinks they’re a good driver, but you have the track record to prove it: just two minor fender benders in thirty years behind the wheel, during which time you have made 20,000 car trips.

You’re also not much of a drinker, and one of the things you’ve never done is drive drunk. But one year you get a little carried away at your office party. A good friend of yours is leaving the company, and you’ve been under a lot of stress: one vodka tonic turns into about twelve. You’re blitzed, three sheets to the wind. Should you drive home or call a cab?

That sure seems like an easy question to answer: take a taxi. And cancel your morning meeting. But you could construct a facetious argument for driving yourself home that went like this: out of a sample of 20,000 car trips, you’d gotten into just two minor accidents, and gotten to your destination safely the other 19,998 times. Those seem like pretty favorable odds. Why go through the inconvenience of calling a cab in the face of such overwhelming evidence?

The problem, of course, is that of those 20,000 car trips, none occurred when you were anywhere near this drunk. Your sample size for drunk driving is not 20,000 trips but zero, and you have no way to use your experience to forecast your accident risk. This is an example of an out-of-sample problem.