The evolution of design hiring at Stack Overflow

In July 2020, Stack Overflow closed a round of funding that enabled hiring across the company to flourish. Senior Leadership looked across the company and worked to come up with a 18-month hiring strategy. One of the highest priority hires was a designer for my team. Because of this, I had the opportunity to reevaluate how we hired designers. I wanted to share how I evaluated our old hiring practices, refactored them, and what I learned along the way — including why using classic product design approaches like iteration can help you build your hiring practices.

What we used to do

Having interviewed with Stack Overflow over 5 years ago, I didn’t really recall what my own personal journey was. I vaguely remembered talking to the design manager, the head of engineering, and a design test. So my first step was to dig up documentation on the latest hiring practices. The last time we had hired a designer was over a year before our fundraising and it looked a little something like this:

1. Application screening
2. Recruiter interview
3. Skills interview
4. Workstyle interview
5. Design Test
6. Final CPO interview

First change: Shorten process, change roles

This process seemed very long given our urgent need for a new designer. The first step in our evolution was to combine the skills interview and workstyle interview. What did this mean?

In our previous skills interview, one designer sat with the candidate to go through a project in their portfolio and a critique exercise. While the portfolio review is pretty typical for a design candidate, our critique exercise was a bit unique. The exercise helped to measure the candidates ability to give critique on something that they may have been seeing for the first time — an actual feature on Stack Overflow.

Meanwhile, our work style interview covered generic questions about how the designer worked, product thinking, and strategy.

When combining the two, I cringed at realizing that we would have one less Stack Overflow designer evaluating a candidate, so we decided to have this interview run by both the hiring manager and a product designer. We also re-evaluated the questions being asked in the workstyle interview and simplified the feature that was being critiqued. The original feature was hard to explain, let alone understand, especially for someone unfamiliar with Stack Overflow. It felt like an unfair way to judge someone’s ability to critique so we picked something easier and something that gave visibility to different product groups at Stack Overflow versus only exposing candidates to just our main Q&A features.

Another change that happened immediately was in regards to our design test. Previously, the presentation involved two Product Designers, the Design Manager, and a Product Manager. We felt that this didn’t really reflect how the Stack Overflow Product team worked anymore, and so we decided to shift this process to involve one Product Designer, one Product Researcher and one Director of Product — preferably the Director of the product for which we thought the candidate would be the best fit.

To sum up our first round of changes, our process now looked more like this:

1. Application screening
2. Recruiter interview
3. Peer panel interview: Portfolio review and critique
4. Design Test
5. Final CPO interview

Second change: Organizing and strengthening our evaluation process

After running through this process with only a handful of designers, we quickly discovered a few problems:

• Two people meeting a candidate on their first interview was time consuming
• A candidate meeting two people on their first interview was intimidating
• We were unable to get through our list of questions for the candidate.
• We had difficulty gauging candidate seniority

We decided to add back in an additional 1:1 interview. How was this different from the original plan you may ask? Well, we moved the hiring manager interview to the front of the process, so that candidates met their hiring manager first. Hiring is a part of the manager’s job and therefore taking the time to talk to candidates was seen as more relevant to their role. This also gave us the opportunity to talk to candidates up front about what they want to work on and what we were working on. We also took the time to refactor our questions once again. At this point we started categorizing our interview questions by the standard job competencies we had defined for the role. We felt that this helped better structure the interview process for the candidate and helped us better understand if the candidate would thrive in our environment.

Our hiring manager interview was now aligned with the competencies of Personal Leadership and Execution, while our peer interview was aligned with Domain Expertise and Problem Solving.

The overall format for the peer panel interview remained largely the same–we still reviewed one portfolio item, performed a critique of our product, and this was done by two product designers. We expected that adding the additional interview back in was going to help us better determine fit and improve our candidate experience.

After our second round of changes, our process looked like this:

1. Application screening
2. Recruiter interview
3. Hiring manager interview: Personal Leadership and Execution
4. Peer panel interview: Portfolio review, critique, Domain Expertise and Problem Solving
5. Design Test
6. Final CPO interview

Third change: Applying what we’ve learned

We were excited for the changes we had just made. We thought we surely had found our sweet spot, but alas, this was not the case. We were able to move nearly 5 designers through to our design test kickoff with only one coming out the other side. It was back to the drawing board.

When we got together to chat about how to fix the problem of having so many failed design tests, it was important to note why each person failed their design test. We found it came down to two reasons

• We lacked the ability to assess how a candidate would work alongside other team members
• We wanted to know more about their expertise before this point.

Our highest priority fix was adding a way to see how our candidates worked alongside team members. After reflecting on this for a while, we felt the best way to judge how a candidate was going to work with others was to simulate. Luckily, our engineering interviews were going through a similar hurdle, so we were able to align efforts.

Introducing “Snack Overflow”, our made up company meant to challenge not only our candidate, but the entire panel participating in the interview. What does this mean? This new interview takes our candidate on a journey alongside another designer, PM, and developer to solve problems that our imaginary company is facing. We ran through this panel in multiple mock scenarios and adapted it for several roles and I truly enjoyed the experience. I also found it incredibly interesting how different the solutions we came up with when our candidate changed, but our panel didn’t.

This new panel interview covered our remaining competencies: Collaboration and Interpersonal Skills, something we were lacking before. However, a new interview round put a strain on our small design team for time. We also still felt that this didn’t necessarily close the gap on better understanding their expertise so we made even more changes.

Our portfolio review stage, which was once a panel, would now be a 1:1 interview. We hoped that this would also make the candidate more comfortable before taking part in two panel steps after this interview.

Next, we increased the number of portfolio items the candidate would present from one to two. Critique was not something we wanted to remove completely. We felt, and still feel, that our critique portion of the interview is a great method to assess seniority and cultural fit, so we simply decided to move the critique to an earlier stage to better balance time.

Now, our interview process runs like this:

1. Application screening
2. Recruiter interview
3. Hiring manager interview: Critique, Personal Leadership and Execution
4. 1:1 Peer Interview: Portfolio Review, Domain Expertise and Problem Solving
5. Peer Panel Interview: Scenario, Collaboration and Interpersonal Communication
6. Design Test
7. Final CPO interview

Final change: Remove the design test?

We realize that this process is lengthy. We are also aware of the stigma that design tests have. Our ultimate goal is for the rest of our process to deem the design test unnecessary — that is our North Star in this journey. But for now, it remains. And I want to defend it.

At Stack Overflow, we do design tests a bit differently than other companies. To start, our design tests are paid. And because they are paid, they are time boxed. We understand that the candidate’s time is precious and we don’t expect the candidate to give us their time for free, especially for a job opportunity that they may not necessarily get.

The second thing I think our design test does well is that it’s actually based on a real problem. We don’t ask candidates to solve some unrealistic or unrelatable problem, but instead focus on real problems we face with our products. We feel that this allows our candidates to get a better understanding for the work that they’d be doing if they were to get and accept a position with us. It also allows us to supply real research and data and see how the candidate uses these.

Lastly, we give our candidates a lifeline. We set up our candidates with a Slack channel with the panel that will be interviewing them — a designer, a researcher, and a Director of Product. We tell candidates that this channel is available to them to use however they see fit. We try to allow for our candidates to have every opportunity to not only feel comfortable in a typically uncomfortable interview stage, but to thrive in their task.

What did I learn?

One thing I learned when refactoring our hiring practices was that grouping interviews into “themes”, or in our case competencies, made for a much better experience for both the interviewer and candidate. Each interview had a specific purpose that an interviewer had to evaluate against, and the experience for the candidate had a much better flow — questions never felt out of place or shocking.

The panel interview was another area where I learned a lot. This interview step was new for us and extremely successful. It put our candidate in a situation with potential coworkers to work through problems together. We are able to see the candidate’s process in various parts of a project, as well as how they work with others — something typically hard to judge in an interview setting. It removed the awkwardness of white boarding exercises — no solutions needed here — just conversations around a problem space.

The biggest lesson I learned through this process is that iterations are okay and encouraged! Much like the design process, you release with a v01 and build from there based on what you learned. Do this for your interview process! Don’t be afraid to interview those who went through the process (interviewers and hires), add steps, remove steps, reorganize, talk to other hiring managers in your company, in your industry. Use your processes as a designer towards creating the best hiring practices for your company and your needs.

As we continue to hire heavily at Stack Overflow over the next year, we will continue to look at our processes and refine them in ways that benefit our candidates, our team, and our products.

Image credit: Clem Onojeghuo