I\’d like to share and comment on a case that recently came up in another department. The case is this. The department was doing a hire, and they had two main finalists. One of the finalists was a woman a few years out of a very good program, with a ton of first-authored publications (well upwards of ten), several in top-ranked journals in the field, despite having a full-teaching load. At the flyout, this rather dimunitive woman (standing 5 feet tall) did not \”impress.\” Her research was judged to be very good, but many on the hiring committee were not impressed by her presentation: she seemed nervous, her voice was too quiet, etc. The second candidate was a 6-foot-2 male with no first-authored publications. Although he had come out of the #1 program and been in prestigious post-doc for an entire year, he only had three publications, all of which were co-authored with his (very famous) graduate school advisor. He had no teaching duties at his post-doc, very few papers under review, and had struggled to finish his graduate program, taking a full two years longer to graduate than the female candidate. Despite all this, many of the faculty were \”blown away\” by his brilliance during the flyout. He spoke in a deep, confident voice and was considered by many faculty to \”obviously be a future star.\”

The faculty were torn between the two candidates. Some thought the brilliant man was the obvious choice. Others thought this to be completely absurd. In their view, the female candidate was far and away the better candidate. Unlike the male candidate, she actually had a proven track record of success publishing her own work in top-journals and in the classroom. Alas, after several days of drama and deliberations, the department decided to offer the position to the male candidate. Members of the faculty who preferred the female candidate were irate, feeling that the committee had favored someone with few accomplishments over a person with many.

In my view, this is the kind of case that should lead us to question the value of interviews and such. There is this common belief–a belief which there is a great deal of empirical evidence against–that we are capable of detecting \”talent\” better through personal experience than through purely statistical resources (i.e. resumes alone). Study after study has that purely algorithmic processes (i.e. counting publications) are systematically better predictors of success than so-called \”eyeball tests\” (i.e. personal impressions)–and yet, as we see in the case above, people continue to base hiring decisions on personal impressions. Why?

The reason why \”eyeball tests\” don\’t work very well seem obvious enough (to me, at any rate). Generally speaking, the best predictor of future success is past success. But, if you want to know about someone\’s past success, all you have to do is look at their work. Looking at the person, on the other hand, introduces all kinds of additional noise into the process: preferring people on the basis of things that might seem impressive (e.g. confidence, a bellowing voice, mental quickness, etc.) but which don\’t necessarily have any reliable relation to actual production. To see how, consider just a few notable examples from professional NFL football:

• Despite only having one good collegiate season, JaMarcus Russell blew everyone away at the 2007 scouting combine with his physical talents. He was selected #1 overall in the draft and went onto become one of the worst draft \”busts\” in history.
• In 1998, pro scouts were split between two college quarterbacks: Peyton Manning, who has gone onto become one of the greatest pro quarterbacks ever, and Ryan Leaf, who is one of the worst draft \”busts\” of all time. Many scouts preferred Leaf over Manning because of his \”obvious\” physical advantages–this despite the fact that manning had a much more successful collegiate career.
• In 1999, Akili Smith was selected in the first round of the NFL draft despite having only one good season, because of his \”obvious physical talents\” at the scouting combine.
• In 2000, a man named Tom Brady was selected in the 6th round of the NFL draft, due to perceptions that he lacked sufficient physical tools to succeed.
• In 1979, Joe Montana was selected in only the 3rd round of the NFL draft, being judged by scouts to be too slight of build and not having a strong enough throwing arm–this despite being famous for incredible comeback victories while at Notre Dame. Montana is now judged to be perhaps the best NFL quarterback of all time.

There\’s an obvious pattern here: a pattern of people systematically ignoring actual accomplishments in favor of personal judgments about \”talent.\” And there\’s another obvious pattern: this not working very well! Time and time again, individuals who are profoundly impressive in person do not go onto succeed…precisely because their personal impressiveness is not backed by a past record of actual success. Conversely, time and again, individuals with actual records of success continue to succeed even if they are \”not that impressive\” in person.

So why, after all of the studies, and all of the trends, do people continue to \”trust their judgments\” of talent? Why indeed!