Test creation guidelines

Principals / Overview

This article will outline the guiding principles for the creation of the screening tests that are available in evaluate. All of our tests are built with certain requirements:

• Tests should assess a range of different experience levels
• Tests should be practical. We’re looking to assess if candidates have the real world skills required to be a productive member of your team. A minimum of 50% of questions should be coding or practical in nature
• Each test is specific to a certain skill set or language. Tests can be combined together if multiple skills need to be assessed
• Avoid trick questions. We’re not looking to trip people up
• Tests should take between 30 mins to 1 hr to complete. This gives enough time to assess a variety of topics, but won’t burden the candidate with an overly lengthy screening process
• Tests should assess as many different aspects of the skill set as possible, and shouldn’t focus on any particular methodology or frameworks
• Questions that have googleable answers should be avoided. We want to see people’s experience and talent coming through

Structure of a test

The following are the test creation guidelines. Tests should:

• Have between 20 to 30 questions
• Be structured in thirds:
  • First batch of questions is basics / fundamentals
  • Second batch is intermediate
  • Third batch is challenging and geared towards seniors
• Questions should be designed to take 1-2 minutes each. Questions that are expected to take longer than 5 mins should be avoided
• All questions where possible should be auto-scoring, as we want test results to be auto-generated without manual scoring
• Marks should be evenly distributed across all questions. We want to avoid overly weighting any aspects of the skill set
• Question types include single choice, multiple choice, short answer, long answer (avoid as it can’t be auto-scored) and code (only use auto-scoring code questions)

Sourcing tests

Whenever possible, we build these tests in-house. This helps to maintain consistency and quality standards. In cases where we don’t have the applicable skill set in-house, we source subject matter experts carefully using open source contributions, stackoverflow and recommendations.

Verifying tests

Before making tests available to the general public, we invite and get feedback from a minimum of 5 subject matter experts. Any questions that are controversial / confusing or that more than 3 of the reviewers got wrong are removed and replaced.

Closing notes

The above guidelines are geared towards making tests: 

• Generally applicable
• Assess as much of the skill set as possible and 
• Do so in as time efficient a manner as possible