Test creation guidelines

Principals / Overview

This article will outline the guiding principles for the creation of the screening tests that are available in evaluate. All of our tests are built with certain requirements:

  • Tests should assess a range of different experience levels
  • Tests should be practical. We’re looking to assess if candidates have the real world skills required to be a productive member of your team. A minimum of 50% of questions should be coding or practical in nature
  • Each test is specific to a certain skill set or language. Tests can be combined together if multiple skills need to be assessed
  • Avoid trick questions. We’re not looking to trip people up
  • Tests should take between 30 mins to 1 hr to complete. This gives enough time to assess a variety of topics, but won’t burden the candidate with an overly lengthy screening process
  • Tests should assess as many different aspects of the skill set as possible, and shouldn’t focus on any particular methodology or frameworks
  • Questions that have googleable answers should be avoided. We want to see people’s experience and talent coming through

Structure of a test

The following are the test creation guidelines. Tests should:

  • Have between 20 to 30 questions
  • Be structured in thirds:
    • First batch of questions is basics / fundamentals
    • Second batch is intermediate
    • Third batch is challenging and geared towards seniors
  • Questions should be designed to take 1-2 minutes each. Questions that are expected to take longer than 5 mins should be avoided
  • All questions where possible should be auto-scoring, as we want test results to be auto-generated without manual scoring
  • Marks should be evenly distributed across all questions. We want to avoid overly weighting any aspects of the skill set
  • Question types include single choice, multiple choice, short answer, long answer (avoid as it can’t be auto-scored) and code (only use auto-scoring code questions)

Sourcing tests

Whenever possible, we build these tests in-house. This helps to maintain consistency and quality standards. In cases where we don’t have the applicable skill set in-house, we source subject matter experts carefully using open source contributions, stackoverflow and recommendations.

Verifying tests

Before making tests available to the general public, we invite and get feedback from a minimum of 5 subject matter experts. Any questions that are controversial / confusing or that more than 3 of the reviewers got wrong are removed and replaced.

Closing notes

The above guidelines are geared towards making tests: 

  • Generally applicable
  • Assess as much of the skill set as possible and 
  • Do so in as time efficient a manner as possible