Monday, 6 April 2009

Cheating on interview tests ...

I never like rejecting resumes or people. I wish I could truly give everyone a chance to prove themselves. Unfortunately, the limited resources of a for-profit organization dictate that that some people get rejected without a thorough assessment of their abilities. We have to use a lot of different reasons to whittle down the huge list of applicants to a number we can actually interview.

Today I rejected 11 people (out of 43 finalists) for cheating stupidly.

We just finished a "programming party" 2 weekends ago. This is a recruitment event where we bring something like 110 students in over the weekend, run them through automated programming tests then hold a small party afterwards. The point is to give students a chance to see where we work and talk to our developers in an informal environment to find out what life is like in Exoweb.

We ran the tests using our older Exobench system, so the programming problems had been used for a while. Therefore I wasn't really that surprised that some of the candidates knew of the problems in advance. What I was surprised at was that so many of them blindly copied answers they obtained elsewhere without thinking it through. If they had, they would have found the errors that made it so easy to disqualify them.

The most glaring example is one of the test problems called Dominator. Given an array of numbers, the problem is to find out if there is a single number that occurs in more than 50% of the array. If it does, return it. It's a simple enough a problem and yet very useful for testing since it has both an easy (exponential complexity) and an ideal (linear complexity) answer. This gives us lots to talk about in the interview.

Apparently someone posted the code for the easy solution, except with one enormously glaring error - the code only searched half of the array, stopping its search at length/2. If you're looking for a number that takes up half the array, you can't evaluate only half the array since you don't know which half of the array this number might be in! This mistake showed up the most and no doubt there were some that honestly made this mistake. However, the cheaters were obvious because there were also other small mistakes that made their code identical.

An even more obvious copy was an ideal solution for another problem. Getting an ideal solution is rare and statistically, we see the ideal solution for this problem once every 400 tests or so. This time, we got 8 out of 110. That's almost 30 times what we would normally expect. However, 7 of the solutions had one glaring error - they forgot to consider an array of all negative numbers. In every single other case, if the programmer was smart enough to come up with the ideal solution, they would always consider the all-negative array boundary case. Somehow we got 7 people who suddenly made the same mistake at once ...

4 out of the candidates made both of these mistakes together (candidates do 4 random problems at a time). The rest made either one or the other. All have been rejected. I have been told that this is standard practice in local universities when a batch of identical exam answers are received.

There are still a few others that have suspicious answers (identical answers, including mistakes, etc) but they're not conclusive enough to reject. We'll take the chance with interviewing them. If they really did cheat, then we'll find out soon enough in the interviews.

I'm bummed out because I recognized a couple of the people rejected. They made very positive impressions while chatting during the party and we were looking forward to interviewing them. Finding their names in the list of bad cheaters was really sad.

Such is life.