People applying for software engineering positions at companies are often asked to solve problems on a whiteboard, under the watchful eye of an interviewer, as a way to assess technical problem solving skills.
But recent research suggests that whiteboard technical tests – so daunting to job seekers that there are books on how to deal with them – often fail to assess technical skill, according to new research. Instead, they’re all about pressure.
In a paper [PDF] to be presented later this year at the ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, researchers from North Carolina State University (NCSU) and Microsoft in the US argue that whiteboard sessions test for stage fright rather than, y’know, coding competency.
The title of the paper hints at its conclusion: “Does Stress Impact Technical Interview Performance?” NCSU authors Mahnaz Behroozi, Shivani Shirolkar, and Chris Parnin, with Titus Barik from Microsoft, say it most certainly does.
“Through a happy accident, the software industry has seemingly reinvented a crude yet effective instrument for reliably introducing stress in subjects, which typically manifests as performance anxiety,” the paper explains.
“A technical interview has an uncanny resemblance to the Trier Social Stress Test, a procedure used for decades by psychologists and is the best known ‘gold standard’ procedure for the sole purpose of reliably inducing stress.”
As a consequence, whiteboard interviews may fail to assess coder competency. Rather, the researchers argue, they measure how well job candidates handle anxiety.
Using 48 graduate and undergraduate students with programming experience, the researchers conducted a randomized controlled trial to compare the traditional technical interview (done while being watched) with a private session evaluation (done without being observed). The experiment was designed to measure cognitive load and stress through the collection of eye tracking metrics, specifically fixation duration and pupil dilation.
The researchers found that stress hinders interview performance, with participants in the traditional technical interview exhibiting higher cognitive load, lower scores, and higher stress levels. In essence, social anxiety took otherwise qualified job candidates out of the running because of the circumstances of the interview.
Further flaws
What’s more, whiteboard technical interviews appear to favor men over women.
“We also observed that no women successfully solved the problem in the public setting, whereas all women solved it correctly in the private setting,” the paper says.
In a phone interview with The Register, Christopher Parnin, assistant professor at NC State University and one of the paper’s co-authors, said he doesn’t have a conclusive reason why this might be the case. He said there’s some support in academic literature to indicate the women have more performance anxiety than men, but he stressed that’s a gross oversimplification because men experience performance anxiety too.
For Parnin, the problem is whiteboard tests themselves. “It all comes down to the fact that the test is designed to make almost anyone fail,” he said. “You’re basically having to interview tons of people just to find those who can pass it.”
Parnin took issue with the way the industry has dealt with the difficulty of its tests. Rather than coming up with a fair way to evaluate software engineers, companies like Google advise at least 40 practice sessions – a time commitment that’s not an option for everyone. This amounts to stress inoculation training and it does help people pass whiteboard tests, he said, but it doesn’t make the tests an effective skill assessment tool.
As an alternative, the paper points to the way devops biz Honeycomb (Hound Technology) – overseen by a female CEO, CTO, CMO and VP of engineering – approaches hiring. The company provides interview questions in advance so it’s not a Trier Social Stress Test.
As the company explains on its website, its goal is to avoid surprises. “The research is clear: unknowns cause anxiety, and people don’t perform well when they’re anxious,” the company says.
“The big picture is to provide more accessible alternatives,” said Parnin. “There are a lot of ways to test for the same thing without putting all this pressure on people.”
What Parnin forgets, is that pressure is actually a great part of a software developers’ life and so a very valid thing to test for.
