Interviewing software engineers - what questions should I ask?

Interviews are a very vulnerable display of ourselves and I am extremely humbled by the fact that some 400 odd engineers sat with me to interview, and taught me how to do it right.

I have previously put together a team of brilliant engineers from the likes of Google, Facebook & Amazon for a seed-funded startup as the product manager and set up an entirely new tech team from scratch for FusionCharts as a product consultant. Now I am doing it at scale, as the co-founder at Adaface where we automate first round technical screening with our chatbot, Ada. Since it has been a core part of my job for several years now, I have been working on designing an interview process that I would want to go through myself, one that helps us find the engineers best suited for the role, and most importantly an interview where candidates leave the building with a smile. This post is a summary of what I have learned so far, and I will lay out actionable insights that hiring managers can incorporate into their hiring process.

I will divide my learnings into 3 sections:

1. Technical interview
2. Making a decision
3. Interview Etiquette

Technical Interview

1. Ask questions that help you score candidates granularly.

So how do we decide what questions to ask? Do not ask trick questions just for the sake of a filter. Candidates get pissed. And it doesn't give you any meaningful signal about the ability of the candidate. No one is happy.

Just because there is a question that only 5% of engineers can solve, doesn't mean that those candidates are the top 5% most suited for the role. Also, this way of measuring a developer's skill has an inherent bias against more experienced developers.

• Don't let each interviewer decide what questions to ask candidates randomly.
• Have questions for all levels of scoring 1, 2, 3, 4 (increasing levels of difficulty/ years of expertise).
• Each question needs to be solved by all interviewers before they start interviewing candidates. Then allow candidates a generous allowance of 3x the time to solve that question.
• The idea behind the asking particular questions in an interview is to understand the candidate's ability in a given skill. This inherently means that we can't ask everyone all questions/ the same questions- that would be a waste of time. If a candidate couldn't solve 2 questions of medium difficulty, there is no point in discussing a difficult question just for the sake of it. Asking an easier question will give you useful data about where the candidate stands on that skill.

2. Have a consistent scoring mechanism.

After a bit of iteration, I chanced upon what I like to call assume-the-candidate-is-exceptional way to score. Here's how it works:

• When you present a question, assume that the candidate will solve it in the best possible way and give them a 5.
• Throughout the discussion, add negative points for things that matter and the candidate doesn't do well on.
• E.g. for a role which requires a good grip on algorithms, -1.5 for being able to arrive only at an unoptimized answer on attempt #2, -1.5 if we had to give the candidate a very revealing hint to move forward, -2 if they give up too soon, -0.5 if they miss edge cases and so on.
• Add the numbers at the end of the discussion for that question, and pick the next question accordingly.

3. Candidates are the best judges of their skills.

Most candidates know how good they are with a particular skill, but they usually self-report incorrectly for 2 reasons: bias and misaligned scales.

There's an easy way to fix them both:
Bias → ask them to prove it with data.
Misaligned scales → tell them about your scale and ask them to grade to your scale.

Most engineers are smart to recalibrate if you explain your scale.

I typically explain my skill expertise scale to candidates:

1. You know the syntax and keywords enough to google, use Stack Overflow and get the job done.
2. You have experience handling an end to end project using the skill. This can be verified by going through their code, screenshots, understanding scale/ impact/ challenges of the project.
3. You know the internals of the skill, design patterns and currently perform code reviews for other engineers in your company for that skill. This can be verified by going through their code reviews (if possible) and asking what design patterns they look for.
4. Zen mode: You are writing libraries and rolling out new versions of this skill.

And I ask candidates how they would grade themselves in this scale for all the skills they know. Almost all of the candidates score themselves within an error range of -0.5 of what I end up scoring them at. Use this to your advantage to know what skills to focus more on and what to spend less time on.

4. Ask at least one question where they struggle.

This might sound mean to some extent and maybe it is. But if you ask a candidate 10 questions and they solve all 10 easily, they will perceive it to be an easy interview and will associate that with how challenging the work might be. That might lead to them not being very excited about the role. The harder it is to get somewhere, the more we appreciate it. Also, it gives you a lot of meaningful data on how they react in a difficult/ stressful situation.

Obviously, that does not mean that we have to ask the hard questions and make them feel like they lost. The intention is quite the opposite. The candidate and the interviewer are evaluating each other. Let's push each other to have at least one challenge that's new so that the discussion has taught us something new, irrespective of the result.

So I end up asking at least one question (I have a handy list of some questions that are really good/ tough) to good candidates and leave them with that.

Making a decision

1. Create a scorecard as you go and check for bias later.

Don't write the scorecards post-interview. You will forget a lot of what happened during the interview, the scoring won't be as granular. At the same time, make sure you check for bias  at the end. Did you score someone higher just because they are from a tier 1 college? Did you score them higher or lower because they are of the opposite/ same gender? Or because they were very similar to you?

2. Force yourself to say yes or no by choosing better scales.

We don't like to take ownership of things we didn't signup for. It is hard to take a stance of rejecting or hiring someone and a lot of the times we end up saying 'maybe' so that someone else needs to make that decision. Force yourself to not do that. One way to do that is by choosing a good scale. If you choose a scale of 1-5, most of the interviewers choose 2 or 3 or 2.5 or 2.75 (anything that makes them comfortable to be in the zone of 'I don't know let someone else make the decision').

Here's the scale I think interviewers should use for the final yes/ no decision: 1 - 4

1. If they're hired, I quit.
2. Don't hire them.
3. Hire them.
4. Hire them, else I quit.

The pros of this scale are that there are no maybe's and there are polarizing yes's or no's. When every interviewer follows this scale, it makes easy to discuss a candidate in the context of hiring them.

3. Hire. Don't get too caught up in finding the absolute best. Be okay with making calculated "maybe" hires.

There are hiring managers who never compromise on the "bar"/ "quality" of candidates. But at the end of the day, we need to execute. We need to move closer to the company's mission. So how do we decide whether to hire or not? Build a decision framework that gives results (actual hires) and course-correct as you go. To this end, I believe a humane version of hire fast, fire fast strategy is better than not hiring anyone.

If we always hire everyone with a rigid framework, we'll end up with similar thinking, non-diverse and eventually a boring group. Sometimes, go with the gut and make few 'maybe' or even 'no' hires if the project needs that but be very mindful of how many such hires you're making. Who will their manager be? What are their responsibilities? Setup 1-1s to check how they're performing. Set up processes/ training to help them turn their negatives around or double down on their positives.

Here's a hiring strategy that has worked quite well for me in the past:

1. If any interviewer said 4, even if someone else said 2, hire.
2. If no one said 4 and at least one interviewer said 1, don't hire.
3. If at least 2 interviewers said 3, hire.

4. Set processes to revisit your hiring framework.

We need to set meetings on the calendar to visit our hiring framework and its results. Analyze if our assumptions have been correct. Are the "perfect" hires performing as expected? What happened to the "maybe" hires- how are they performing?

5. Get your team to sit with the candidate to see if there's a mutual cultural fit.

Your friends probably know if your current partner is good for you (or not) and might foresee your breakup way before it happens. It is the same with a team and a new hire. Your team might give you good data on what they think about the hire culturally. It also gives the candidate a chance to experience the company's culture before making a final decision.

Interview Etiquette

1. Create a short document on the pre-requisite skills and how you evaluate them. Share it with the candidates beforehand.

What's a pre-requisite or a must-have skill? If a candidate doesn't have this skill, we are not going to hire them. Ideally, we won't even interview them if we can help it.

This is a bit of work. But do it. Ideally, you need to have this locked down with all interviewers. Do not let each interviewer decide for themselves what's important for the role.

Share it in the interview invitation email that you send to candidates. Once I started doing this, the conversation with candidates was much more effective. Remember that this might be constantly changing for the same role, so talent acquisition teams and hiring managers need to keep iterating this.

For example: hiring the first iOS engineer? Probably want them to must-have knowledge of Swift. Second iOS engineer? Maybe Swift isn't a must-have this time? They can learn on the job.

Tricky spot: Hiring for a role with no must-have skills like a full-stack engineer. In cases like these dig deeper into what kind of work they should have previously done. How many projects/ delivered-impact/ time did the person work on any particular skill? If the person kept switching between languages every quarter without any high-impact projects, they might be a beginner in everything.

Once we have narrowed down on the must-have skills, we also need to define the expertise range. Should they be experts in the skill, and be able to deploy new libraries by themselves? Should they already be doing code reviews in that skill? Should they have experience with one production application in that? What is the bare minimum we need?

2. Break the ice. You first.

It takes time for candidates to settle in. So we shouldn't expect them to start talking about themselves. Have a pitch about yourselves, the company and the role handy (write it down). Preferably the short, medium and long versions. Start with a short introduction of yourself and set the stage for them.

3. ASK them for their communication style and adapt to it.

This is probably one of the most important things in an interview, and what most of us get wrong. Especially because every how-to-smash-a-tech-interview guide out there tells candidates that they need to think out loud. But not everyone works that way. Some interviewers expect candidates to walk them through their thought process as they solve a question, and that's detrimental for the category of engineers who don't work that way.

I understand why that is, and it makes the interviewer's job easy if the candidate talks out loud about what they are thinking, it's much easier to see how they are thinking about optimizing step by step and score accordingly. But, we need to learn how to work in a way that's most conducive for the candidate to perform at their best.

I believe the best way is to give the candidates all options upfront. "Here's a few ways we can do this, what would you prefer?" They usually ease up immediately.

4. The interview should end with leaving the candidate excited and inspired.

Ideally everyone, more so if the candidate could be a great fit. Talk about what impact YOU created in the company so far and what your next projects are. Nothing beats honesty and personal experience. Tell them why are you excited to be in the company and what inspires you. Prove with data. If you are not excited/ inspired every day about your job, you probably shouldn't interview. That is one reason, I would never ask someone who has spent only 3-4 months so far in the company to interview.

5. Please be kind.

I'll end with this, if you do nothing else right, please make sure you're not insulting a candidate irrespective of circumstances. An investor might have pulled out at the last moment, you might be going through a breakup or whatever else, nothing is a good enough justification to make a candidate feel bad in an interview setting.

The part of your life where you're interviewing for roles can be extremely stressful. And a bad interview experience can kill someone's confidence, you never know what someone is already going through. Irrespective of how qualified (or not) the candidate is for the role, please make sure each candidate is leaving the interview with a smile. That's the least we can do for someone who took so much time out just to interview with us.

If you would like a step-by-step procedure to conduct great interviews, you should definitely read TalentLyft's Guide: How to be a good Interviewer, which explains the importance of being a good interviewer and the best methods to prepare for an interview.

There are enough companies in the world that have ridiculous interview processes that cause undue stress to candidates. Please be an example of how it can be better.