When Evan Carmi posted his Google job interview experience (http://ecarmi.org/writing/google-internship/ ) on HN, I felt reminded of my bygone startup days. In over a decade of "modern" IT startup job interviews we have made no progress whatsoever. If anything, I was part of the problem there for a few years. I simply copied a hiring mechanism that seemed like a standard at the time, and in doing so I failed miserably at the most important goals a company should observe when looking for new developers. Today the tech front pages are full of Larry Page's efforts to turn around the company, but I think performance problems at developer-centric companies may to a large part be burned into their DNA by a deeply faulty hiring process.
How We Did It
My cofounder and I were running a small web development shop in Germany. We had started working literally out of my friend's basement. Over time, we grew, and moved into real office space. At first it was easy to find new employees, we could just ask our friends to come in and work for us. Of course, that model didn't scale, but it performed a very important function: it made sure we hired people that were a good fit for the company, both on a personal and a professional level. Then came the day when we needed to fill positions by bringing in people from the outside.
One of the redeeming features of the German regional unemployment offices is they will send you a large stack of CVs on demand, within a few hours of calling them on the phone. I was pleasantly surprised that we didn't have to hire an agency to do this. Together with the CVs we already had from people who applied to the job posting on our website, we now had some sifting to do. In the end, we agreed on about a dozen of the best and invited them for an interview. This is the part where everything went wrong:
The Standard Dev Interview
A candidate would come in, usually all dressed up in their best suit and tie, we'd sit down and have a talk. That talk was essentially like an oral exam in college. I would ask them to code algorithms for all the usual cute little CS problems and I'd get answers with wildly varying qualities. Some were shooting their pre-canned answers at me with unreasonable speed. They were prepared for exactly this kind of interview. Others would break under the "pressure", barely able to continue the interview.
To be honest, when we first started doing this I had to look up these puzzles in advance, mainly to make sure I didn't embarrass myself. This should have been the first warning sign that maybe we weren't exactly testing for skills that were most relevant to our requirements. If these doubts occurred to me, I must have dismissed them very quickly. After all, it was the way everyone approached the interview process.
Of course, we ended up hiring the candidate with the smoothest answers. Inevitably, the next job openings came, we did it again and again in the same fashion, for the rest of the company's lifetime. If this sounds familiar to you, you are clearly not alone.
Actual Job Performance
But how did the candidates we selected measure up? The truth is, we got very mixed results. Many of them were average, very few were excellent, and some were absolutely awful fits for their positions. So at best, the interview had no actual effect on the quality of people we were selecting, and I'm afraid that at worst, we may have skewed the scale in favor of the bad ones.
What does bad and good even mean in this context? Let's have a look some of the benchmarks that I consider important:
Company Culture. In hindsight, one of the most important features a new employee should have is compatibility with the spirit of the people who already work there. The Standard Dev Interview performed worst in this area, for obvious reasons. It's difficult to judge people's personalities in interviews because they are not exactly themselves. In fact, they're incentivized not to be themselves.
Programming Competence. Somewhat counterintuitively, the code examples done during the interview were a bad indicator for actual competence on the job. Real world projects rarely consist of implementing binary searches without access to a parser or literature. It turned out that employees who had done very well in the code examples were not always able to transfer theoretical knowledge into practical solutions. Having candidates write sorting algos on the whiteboard is a method that rewards people with great and precise short-term memory who come prepared for exactly these kinds of questions. In our case, we needed resourceful coders who could write neat, stable, and elegant software - and the interview process wasn't delivering them to us.
Project Management. People who did well in the interview were not necessarily good team mates or even good presenters in front of our customers. This result, too, was surprising to me. Turns out, sucking up to an interviewer for an hour is a completely different skill set than, say, being good at coordinating with your coworkers or the people who pay our bills. Nor was their interview performance indicative of the ability to write good documentation or how to behave in online communications.
The results of a hiring process such as this may be one of the factors responsible for a company to lose its startup spirit and its creative soul. This was certainly the case with our company. As the CEO our ultimate failure was certainly my fault, however, having the wrong people on the job was a large part of the company's inability to deliver the quality and quantity of output needed to sustain it. Infighting poisoned our teams. Incompetence was covered up with good presentation skills and ass-kissing. Good people left the company because they hated the new atmosphere.
Though I had to let go many people for different reasons over the years and in the end had to deliver the hardest speech of my life on the morning I dissolved the company, I only went "full Trump" once on an employee. It was the one who had displayed the best interview performance and the best academic references of them all only a year before.
Sure, that's an extreme example. Most companies succeed regardless. But I still believe we can vastly improve the chances of finding candidates that are good fits by radically changing the way we do interviews. And in our case, that would probably have made all the difference in the world.
So what should a developer job interview look like then? Simple: eliminate the exam part of the interview altogether. Instead, ask a few open-ended questions that invite your candidates to elaborate about their programming work.
- What's the last project you worked on at your former employer?
- Tell me about some of your favorite projects.
- What projects are you working on in your spare time?
- What online hacker communities do you participate in?
- Tell me about some (programming/technical) issues that you feel passionately about.
These questions are designed to reveal a great deal about the person you have in front of you. They can help you decide whether the candidate is interested in the same things as you, whether you like their way of thinking, and where their real interests lie. It's tougher for them to bullshit their way through here, because the interviewer can drill deeper into a large number of issues as they present themselves.
What about actual coding ability? Well, take a few moments after the interview and look into some code the candidate wrote. Maybe for an open source project, maybe they have to send you something that's not public, doesn't matter. Looking at actual production code tells you so much more than having them write contrived fiveliners on the whiteboard.
I'm sure you can come up with even more questions and other ways to engage the interviewee. At this point, pretty much any idea will be an improvement.
Most people are quick to defend the status quo, and sure, that's a rewarding position to hold. It's risk free and you can always fall back on the old argument "a lot of smart, rich and successful people do it the old way, so my money is on whatever they are doing".
❜❜Nice, but that doesn't work for large, successful companies. Your idea doesn't scale.❛❛
Sure it scales, in terms of effort per interview there is no difference. There is no reason this couldn't work in larger companies. In the end, the interviewer always makes a personal and deeply subjective decision, I'm merely suggesting a way that delivers more relevant information for that purpose.
❜❜The best programmers have no spare time projects.❛❛ or: ❜❜The most talented people I know work from 9 to 5 and then go home to watch football/be with their families/whatever.❛❛
This is not my experience. I'm not saying that a good programmer should not have a life. But I do believe that a certain amount of enthusiasm for programming is called for. And really, if you have such a great skill, not using it for fun seems kind of wasteful to me.
❜❜In my spare time I'm working on making the next million for my company. Oh, when I'm not working for my company? I'm with my family or friends.❛❛ (verbatim from http://news.ycombinator.com/item?id=2385148)
That's great, those people can surely show me something they have been working on. I would, however, consider the lack of any hobby projects a warning sign for _some_ development jobs.
It has been my experience that the traditional developer interview is insufficient at finding good candidates. While the typical whiteboard coding exercises correlate somewhat with general CS competence, they are poor indicators of actual programming performance. It is my contention that we have been doing them this way for years simply because they're easy to administer, but the data that's coming out of these interviews is largely irrelevant at best. We as an industry should move to more personalized interview questions that focus on the entirety of a developer's skill set. Also, I believe it is more productive to judge production code as opposed to abstract modular puzzles that have no real connection to the actual job in question. Most importantly, I am convinced that gaining insight into the developer's real personality is just as important as checking for professional competence, because one bad fit can destroy an entire team.