Creating a meaningful test that can differentiate between candidates

I was preparing a new set of question papers for the placement tests for Zyxware. This is probably the 4th or 5th time I am doing this. I haven't quite got the hang of the whole process yet. There are several key aspects involved in setting up a question paper for placement tests.

The questions should not be too tough or too easy. They should be a mix of everything from easy to difficult. If the questions are all too easy then the marks of the candidates would be segregated around a high value with little variance. Everybody will be able to answer most of the questions and there would not be too many mistakes and the variations in marks would be minimal. Similarly if the questions are all too tough the marks would be segregated around a low value with little variance again. Only with an optimal distribution of the toughness of questions will you be able to differentiate among the candidates. You wouldn't want too many candidates to clear the cut-off and you wouldn't want too few either.

Another issue to be considered is allocation of marks per question. Ideally each question should be given marks corresponding to the toughness of the question. However it is practically very difficult to do this. An alternative strategy is to divide the question set into sections where questions of similar complexity and type are grouped together. Now if each section is timed, then the time allowed per question can be varied to give tougher questions more time and the easier ones lesser time. This would ensure that even with identical marks you could make toughness/unit time a constant and have a meaningful test. However, if the timing of the section cannot be enforced - like when you have all the answers written on a single sheet - then the allocation of time does not make sense anymore.

Even after all these efforts if the set of candidates who turn up for the test has an expected average score much less than the fixed cut off then all the effort put in would be in vain. In fact one of the tests I had conducted I had to give extra time for the candidates during the interview to try out some of the problems they had skipped during the regular test.

Another issue that has been troubling me was whether I should publish the question sets or not. We follow an open documentation policy at zyxware. However it is tough to open out our question sets. Until we have a strong HR team that can generate new sets of questions it will be a big mis-utilization our resources if we have to keep creating new tests. So for the time being our tests are closed :).