Getting the Interview Phone Screen Right

blog
101

Well, I’m afraid to say I disagree with a lot of the detractors who’ve posted here. Great post Jeff.

I’ve done a fair bit of interviewing in my time and what I’ve noticed from people who’ve been hired is that there are a lot of people who can solve most problems, but there are a few people that can solve any problem, no matter how hard it is.

Example of “any” : There’s a customer on the phone who’s fed up because once every two months the app we sold them intermittently trashes a bit of data and causes their system to be unusable, but testing can’t reproduce the problem in house. It happens.

The “average” developer will throw their arms up in despair at this and hope somebody else will come along to rescue them. Now this isn’t actually a bad thing itself, if the developer is in a junior position, provided they can then learn and get better as a result of somebody higher up helping them out. (Focus on the first derivative!)

The “superstar” developer might not know exactly what the problem is, and may well not be able to reproduce it in testing, but they’ll be able to look at the code, and think about the symptom, and come up with some ideas. It might take a while, but they’ll never give up.

As a business, you need people who can solve all the problems that get thrown at you. It’s no good having developers who can solve 95% of the problems if the last 5% means you have to loose your largest and longest standing customer.

Yes, I don’t need to worry about writing my own code to reverse a string in my day job. I don’t use regexs that often, and I rarely need to worry about bits and bytes, and when to use a binary tree over a list. I don’t deal with these things for 95% of my working time. But you know what, sometimes, you’ve just got to know about these things to solve a really hard problem.

And, all else being equal, people who’ve at least heard of the sort of areas Jeff and Steve are talking about in their phone screen are more likely than not to be the sort of people who can tackle the really hard problems when the business needs them to.

From reading Steve’s original post, I don’t think he’s even expecting competence in these areas during the phone screen. You might not know the exact regex to extract phone numbers from HTML, but you’ve at least heard of grep, right? And you know it can be used to filter output on a file? I think that’s enough to get through the phone screen.

Also, bear in mind Steve’s phone questions are tailored towards a job at Amazon. If your business is 100% ASP.NET development and is never going to diverse away from that (and how sure are you that’s the case), you might want to ask some different questions instead. Ask the candidate if they’ve heard of view state and session state. Don’t necessarily worry about if they know the difference between them during the phone screen, chances are if they recognise the names they might be able to do a bit more than just drag and drop!

I’m sorry if I’ve antagonised anybody by saying this, or you think I’m looking down on you, but sometimes the “superstar” things are what you need to do to do your job! Maybe working as an internal dev in a bank’s different and you don’t get these problems - I dunno.

Rejecting a candidate is not personal (or at least shouldn’t be). I’ve interviewed for places and been rejected. It happens. Get over it.

102

I think that the questions with string reversals, malloc, btrees, anything related to hex - n conversion are totally outdated and not productive, because they mesure “math” skills and not the real world knowledge

They are also favoring recent graduates without experiences (because those structural alogorythm tasks are part of exams in my country) and desfavor the experienced developers. I haven’t done nothing of the above mentioned things after 1998 when I graduated and I code at least 40 hours per week.

Another thing here to be mentioned is that all of those stupid questions are very well documented on various sites so an smart candidate could memorize all those commonly asked “puzzles” and trick the intervier

To me much better approach is giving a smelly code to candidate to refactor asking them to debug something, asking some of the gotchas we have faced etc

Summarized: IMHO, this style of interviewing candidates is not acceptable and I personally feel somehow insulted when participating in that kind of interviews

103

@Steve: Um, wasn’t that kind of presumed whatever the candidate wrote was inside of a class since he said it was in C#? The class in that instance was of no importance; the METHOD was.

104

Speaking of interviews, I just came up with a new concept: Have a candidate read and comment on a post just like this one.

Maybe a majority of the commentators here can program their asses off, but personality conflicts can bring a team to a grinding halt as fast as total incompetence. I would much rather take a humble-smart developer on board than a genius-asshole.

Snarky != Intelligent

105

Well, I’d be pretty happy with answering all those questions (although maybe a bit weak on Data Structures, and the answers might not always be ‘perfect’), but I do think that those questions do cover most of the range of a general programmer. And good developers will have some holes; this is to be expected. There’s no such thing as the perfect recruit. Someone who is baffled by all of the questions though, well, I’d avoid.

It’s interesting the number of negative comments that seem to be along the lines of “I have no degree / a BA and I don’t think these things are important”. I did a degree that was more about electronics and mathematics than programming (Cybernetics). When I got hired into my first job, I knew nothing about regexs, databases, uml. So, I very deliberately went and learned these things. And they are useful, even if only used from time to time. I totally get why folks would look for these things in a hire.

That’s not to say I’d only hire CS graduates. Computer Science degrees aren’t perfect (the students are usually rotten or rarely brilliant at mathematics, for example), and CS isn’t really about Software Engineering. But just saying “I don’t know these things and I can program, therefore they’re not important” sounds a bit like Pointy-Haired-Boss-ism

106

"I think that the questions with string reversals, malloc, btrees, anything related to hex - n conversion are totally outdated and not productive, because they mesure “math” skills and not the real world knowledge"
Surely tree-related searching is real world knowledge. I mean, it’s a core part of search-engines, isn’t it?
Yes the debug approach is good as well, but isn’t it hard to do down the phone?

107

I’m sorry but if you cannot show you know at least what the interviewer is talking about and make a stab at the answer, ten what are you doing in a programming job?

The RegEx question which so many people seem to think is irrelevant is basic pattern matching and even if you don’t know regex’s well (I don’t) you should at least recognise that pattern matching is the answer?

The .NET people who seem to think they do not have to understand data structures… How do you choose which objects/routines to use for efficiency if you don’t know the underlying datastructures? (Although it does explain why some .NET programs are so slow…)

The java library question as many have said should be tailored to the in-house language of choice

But as Jeff said these are not questions to be solved but to weed out the no-hopers who don’t understand the question, and don’t know how to find the answer…

108

Asking these kind of questions can be useful, but it all boils down to what is that you’re after. I’d rather have someone who can fix the problems than a mathematician, I have a co-worker who’d be definitely unable to answer most of these, however, everytime we ask him to do stuff, he excels at it or knows when/where to ask if he’s blocked. He’s got great attitude and is very valuable for the company as such. I guess it’s a matter of preference.

I agree with both parties, those who religiously claim that “if you don’t know this then you don’t deserve…” and then those who bash this saying stuff like “we know how to google”.

I understand them both. :slight_smile: Me? I’d have some trouble with some, but I definitely would be able to fix it and/or find a decent solution to all of them given a box, internet and some time. I wouldn’t be afraid to ask or say I don’t know this right now.

As Jeff always says: You don’t want to the the smartest guy in the room. I’d love to work in a place where I could learn. We are not building a rocket :slight_smile:

109

“It seems to me that a lot of the readers of this article have got the wrong idea. It’s not like if you don’t know this stuff, you will be screened out because you can’t program. The idea is to get a feel for the kind of person you are.”

Asking these test wil fail to get the good programmers with lots of experiences but let the good student programmers that just came from school get passed the test.

If I get a telephone interview and people ask me these questions. Than I start to have doubts about this company if it is worthwhile to work there. First of all they fail to ask the really important questions how to implement software in such a way that it is easily installable, easily configured, user friendly and when it fails, it does not detonate the computer. The questions about the OO and lists are only minor details in the grand scheme of your software product.
Experience shows that you cannot predict how a program will behave, on paper every- thing works but once you start to implmement and test then some technology that would be very logical and fast to use might nowt work. So the best thing to do is to create a framework that could switch technology or at least make it transient.

They fail to ask how I operate in groups, if I program quick and dirty or create quality code???

I have created very complex stuff that just works reliably and for many years. But I would probably fail the tests and I would probably not be interested in that company anymore if I get these type of questions.

110

Let me just say that as a programmer right out of college, I would have had no difficulty proving at least basic competency on those questions. They seem like an entirely reasonable set of quick weed-out problems.

He’s not asking you to come up with the regular expression for the phone search off the top of your head - he’s asking if you know what a regular expression is, and that it can be used for that sort of problem, and that in a day or two you can figure out how to write one into your favorite text editor and/or scripting language and figure out how to solve that problem.

After a year of experience (I know, still not much) I’ve found that the biggest time wastes I’ve experienced were projects where I or someone else didn’t know the right term/concept to begin researching previous solutions. Programmers need to be more than cut and pasters - but they also shouldn’t be trying to reinvent the wheel as it will often end up as a Reuleaux triangle.

Notice that he didn’t say that they failed the test immediately because they only knew java, or forgot about trees - he said it was an early warning that they will most likely fail other skills questions on the test, based on his experience. I’m entirely willing to believe that a competent programmer could miss a few questions - I know I’d probably miss some of the technical OOP jargon stuff just because they didn’t use those names in my classes - but I agree completely failing one area and/or failing multiple areas is a huge red flag.

111

One thing Steve mentioned in his original article that we haven’t touched on yet is another “failure mode” where the candidate can’t remember what they did in their degree two years ago.

Let’s see, in my degree I remember … C programming under Unix, shell programming, functional programming in Lisp, trees, quicksort, multithreading, PC internals, Minix, human computer interaction, and networking.

Haven’t used Minix ever since my degree but I can still remember it’s a cut down OS and the way it allocates inodes for files does something funny when they get above a certain size.

And that was 15 years ago.

I’ve hardly ever touched Lisp, but every now again I write or refactor something that makes me think “oh yeah, anonymous functions in C# - they’re a bit like lamda expressions.”

If you know your stuff, you don’t just forget these things!

112

From what I can tell, the goal of Steve’s interview isn’t to find a programmer that is ‘the right fit’ for the job opening that is currently posted. He’s developed something that he believes will find top tier programmers. The kind of people that Joel Splosky is always looking for, or even Martin Fowler (http://martinfowler.com/bliki/PreferDesignSkills.html).

Most professional programmers aren’t highly technical, top-tier individuals, just as most basket ball players aren’t Michael Jordan. That’s just the the facts. You can still use an ‘MJ’ test to hire an entire team and find out who is your MJ. Whether or not that’s a good idea is totally up for debate.

Personally, I prefer the methodology laid out by YerFired above since it gets to the root of the problem and can find the senior staff you need without this kind of rigmarole.

g

113

This is a very interesting phone screen technique. I’m certainly interested in using parts of it myself, although I think my application of it would be much lighter.
I guess it depends on the level of the candidate that you’re looking for too.

My one concern is that this just filters out those who can’t remember their CS 101.
It’s has the faint resemblance of a test designed by people who took CS degrees to remove people who don’t have CS degress to justify the fact that they spent x years getting their CS degree as opposed to spending that time in the industry.
Both of those choices have their pros and cons and I don’t think anyone can categorically state that either option results in better developers than the other. Arguably sure, but i’ve met idiots with degress and competant people without and visa-versa, so its not so black and white.

There are a bunch of great developers who just don’t work in the CS mode.
It seems to me like for some people computers start with bits and bytes but to others they start with a problem of a potential client.
I like solving problems, the quicker and more efficient the better that’s why I started coding. Bytes? Sure they’re an important underpinning but I rarely need to use them because most of my clients want business software and I use .NET because it wraps low level things up and makes me more productive.

Before the low level community goes up in arms; what is the yard stick for a good developer? I’ve got software running every day across the world that helps people perform their jobs more effectively, it’s clean, well architected and maintainable code. Some of my deliveries have zero bugs. But I’ve never written a XOR operation in my life.
So do I suck? Is everyone else writing business software in bytes or something?

I sometimes wonder if Yegge’s wizards at Amazon actually failed because they wrote the core in LISP and make it so incrediably “good” that no-one could understand it. Is that good or unmaintainable?

114

This really rings true for me, and I’d definitely want to screen out the guys who need to Google this stuff. Sure, if you don’t know what a hashtable is you can look it up on Google and find an implementation.

But if you never learned what it is, you won’t understand its weakness and you’ll wind up repeating the beginner mistakes. The worst part of it is the attitude that embraces ignorance. A good programmer is constantly learning which means adding to a body of knowledge and applying those lessons learned to make the software fundamentally better, not just perpetually patching over bugs.

115

“Write a function to count all the bits in an int value; e.g. the function with the signature int countBits(int x)”

This was my favorite question of the bunch.

int countBits(int x)
{
return sizeof(int) * 8;
}

Oh wait did you mean all the bits which are set to 1? :wink:

116

Those questions aren’t too hard. I was able to figure out 4 of them with a few minutes. The fibonacci sequence I did as part of brainteaser on the eulerproject. Break string into an array and, recursively pop it back into a string. OO questions I hate because its just more textbook junk. The bitwise is what threw me but I could brush up on it by the second interview.

Usually, when you are looking for work it’s good to do a lot of interviews. Each one will make you a stronger candidate. I interviewed twice with the same company. By the end of the first I new I wasn’t prepared and told the interviewer that. He kept my CV and called me a few months later. But by then I had found work elsewhere.

The questions shouldn’t be treated as an all or none deal. If you don’t know try to steer the interviewer towards your strengths. I’ve let a few interviews get out of hand (example: one diverged in coldfusion because they saw it listed on my CV, but I hadn’t used it in years). And try to learn another language.

117

G’day.

I just took your test as a dare from a mate of mine.

I got 40%. Almost certainly a fail.

So given the fact that I’ve co-founded a software company (I’m the chief software guy) and we’re turning over in excess of 80 million dollars a year, how do you validate the usefulness of this metric?

118

Here’s a better question

You have n trees.

You need to produce a list of arrays of nodes of all combinations of the trees, where only one node can appear from each tree.

How would you approach this problem?

119

Nice post Jeff, spot on. I sent the URL around to my coworkers.

As Remy in “Ratatouille” might have said, anyone CAN program, but that doesn’t mean that anyone SHOULD.

120

I think this post misses the point of the interview process. When I interview someone, I concentrate on their interpersonal skills, how they will interact with the team, and what experience they have, what their educational background is, what projects they have worked on, and what type of other programmers they have worked with and how they got along with them. As long as I am confident that they know how to use Google and can research answers then I am confident that they can do the job. As a previous poster said, I’m not looking for a math wiz or savant, I just need someone to develop decent eCommerce solutions and a web application every now and again.

I’ve been to interviews like this in the past and there wasn’t a single question that couldn’t be answered with five minutes of work on the internet.