Robin Cannon's Blog

Don't Rank Your Candidates Against Each Other

March 05, 2019

One of my Carbon team had her last day at IBM last week. There’s always sadness in seeing people leave. But I also love seeing developers grow and move on to exciting opportunities. We’re not going to “replace” Mari. She’s going to achieve great things as a Senior UI Engineer at Netflix. But we’ll be starting the hiring process to find a new developer who can join the team and succeed in their own way.

The hiring process is always tough. I’m feeling positive. I have a good record of identifying talent and building great teams. Like many companies, there’s pressure to fill open headcount fast. But I won’t be ranking candidates against each other and hiring the best of that pool. There are dangers in that approach.

The “Best” Might Not Be Good Enough

In a broad sense, yes. I want to hire excellent people, with great teamwork and development skills. But “best” is a relative term. The best of a poor selection may still not be good enough.

This is definitely true during the candidate selection and interview process. Ranking candidates means that somebody becomes the “number one” or “best candidate”. Given the pressures to fill a job role fast, there’s going to be pressure to hire that number one choice.

But what if the pool they’re drawn from is mediocre? Sure, they’re the best of that pool. But that might not mean they have the skills for the job. I’d have pressured myself into a hire by giving myself a “number one” candidate.

Absolute vs Relative

Instead of ranking candidates relative to each other, I need to create a baseline. Then the team has a constant reference that everyone is scored against.

When we have a job opening, we define the skills we need. Those are the same skills we list in our job description. But we also put those skills into a candidate spreadsheet.

Everyone who interviews a candidate will score them 1-5 against each of those skills. They’ll put those scores into our spreadsheet. They’ll also score 1-5 against the cultural values we’ve defined (see The Dangers of ‘Culture Fit’ for more info). Those scores also have a weighting. “Required Skills” scores are weighted 100%, “Desired Skills” and cultural values scores are weighted 50%.

Candidates interview at least three people on the team. At the end of the interview process they have an average score. We also have a baseline score that is the minimum we will accept as good enough to hire. So, even if someone has the best score of all the candidates, they could fall below that minimum. If that’s the case, we don’t hire anyone - we need to keep looking!

Here’s an example Excel spreadsheet.

There are more things we could do to make the process even more objective. We haven’t done these yet.

  • Have all our interviewers submit their scores independently. This would avoid confirmation bias. Right now, everyone can see what everyone else has scored someone.
  • Have a set list of questions that must be asked.

Final Thoughts

Rating candidates against a baseline has brought more consistency to our hiring process. And it makes sure we’re choosing based on the skills for the job, not because someone is only the “best available”. It can make our hiring process slower. But I’d prefer no hire to a bad hire.

Robin Cannon

A Brit in Texas. I manage the dev team for IBM's Carbon Design System.
My blog for work, tech, and industry related musings. Thoughts my own.
@shinytoyrobots on Twitter