Uncovering Bias: Interviewing the Interviewers

When I started interviewing at Amazon let’s just say there was a steep learning curve. In my previous life as a book editor most decisions were made using intuition. At Amazon, intuition doesn’t get you very far. If you want to get things done at Amazon you’re going to need data.

There’s a good reason for this. Intuition resides in the subconscious, the same place all our hopes, fears, and biases reside. To know whether your intuition is good judgement versus a discriminatory bias you need evidence to support your decision. 

In the hiring process, this idea of evidence is critically important. Afterall, most hiring decisions amount to people judging people. The cost of making the wrong decision can cost tens or even hundreds of thousands of dollars. Conversely, consistently making good hiring decisions will result in building an elite team. Without evidence it’s very hard to know which decision you’re making until it’s too late.

I got a masterclass in all this when I started interviewing at Amazon. I still recall the first interview loop I participated in:


Sitting in a drab conference room in the Columbia Tower in Seattle, six of us were silently reading each other’s feedback on the candidate we interviewed earlier in the week. The position was entry-level and the candidate was a recent grad from Harvard. After about 5 minutes of reading, the Bar Raiser in charge of facilitating the debrief started asking questions. 

“Tim, you wrote that this candidate has a lot of intellectual horsepower. Why?”

“Well, he’s graduating from Harvard and he had smart answers,” the interviewer responded. 

“Give us an example of a ‘smart answer’?” the Bar Raiser asked.

“Well…” Tim started, glancing down at his feedback, then at the wall, then at his computer. 

It quickly became clear that while Tim perceived the candidate as having “intellectual horsepower” (whatever that means), he didn’t have strong evidence of it. I don’t blame Tim for making the mistake. Like me, he was new and this candidate had an impressive pedigree and charisma. Many companies would have hired the candidate on resume and first impressions alone. But at Amazon, the Bar Raiser wanted evidence and when the interviewers couldn’t offer any we passed on the candidate.

Interviewing the Interviewers

No matter how good of an interviewer you think you are, you have subconscious biases. We all do. While some of us are better than others at self-interrogation, one of the best ways to uncover these biases is by being interviewed yourself.

At Amazon this is done in the debrief by the Bar Raiser. 

In the debrief, after everyone has had a chance to read each other’s written feedback (which is another critical part of the interview process btw), the Bar Raiser will get to work interviewing the interviewers. 

The Bar Raiser’s goal here is to evaluate the evidence being used to support a hiring decision. Is there actual evidence that hiring this candidate is a good/bad idea or is it interviewer bias. 

I observed a perfect example of this during a recent debrief. The team had just given their initial opinions on a candidate and the consensus was tepid. On one hand, the candidate had the experience and qualifications they were looking for. But, no one was inclined to hire. The group didn’t think the candidate would be a fit. 

As the Bar Raiser interviewed the interviewers he uncovered that the group was making some assumptions about the candidate based on the candidate’s style of dress. When the group got to re-weighing the pros and cons after their bias had been uncovered, they unanimously voted to hire the candidate and the rest is history. 

How to Interview the Interviewers

Interviewing is a skill that needs constant practice and honing so I won’t pretend that this one blog post is definitive. In fact, our clients ask us about interview training so frequently that we’ve had to start offering it as a service. 

What I will say is that probably the single most important question to ask your interviewers is, “Can you provide a specific example?”

What you quickly want to determine is the foundation of that interviewer’s opinion. Is it based on evidence or assumption?

How an interviewer responds to that questions will tell you a lot about the quality of their interview. Were they able to get one or multiple examples that support their opinion or are they making inferences. 

For instance, if an interviewer says a candidate has a lot of “intellectual horsepower” can they provide a specific example where say the candidate solved a really hard problem with original thinking. Or did they just enjoy talking to them (sidenote: one of my favorite papers on this phenomenon is here). 

If you don’t have someone interviewing the interviewers it’s very likely that you’re making decisions based on the latter. Which when it comes to hiring is about as good as flipping a coin. 

For more thoughts on the arts and science of hiring, check out our Hiring Insights blog. Have questions or feedback, send them our way at talent@perfectloop.org.