The development of high-quality assessments is complex

The development of high-quality assessments consisting of open or closed questions, or assignments (such as professional products or tests) is a complex matter. In addition, it turns out that teachers have little time for the development of assessments. Even if teachers have had specialist training, it is clear that they regularly have difficulty with the development of assessments due to lack of time (Draaijer, 2016).

Restricted and unstructured availability of guidelines

There is a multitudinous variety of guidelines for the development of assessments. Various sources emphasise guidelines for the development of closed questions. There has, however, been little attention given to guidelines aimed at the development of learning objectives, open questions, assignments and assessment/item analysis, and even less to other stages of the development process, such as the composition of an assessment, the ascertaining of the cut-off scores and the provision of some sort of rating such as a mark. Guidelines tend to feature either in books and published articles or in separate, specific listings, but a structured overview of guidelines which follow the development process of assessments is lacking; a lacuna that hardly encourages the use of guidelines. The 180 guidelines in this overview are arranged on the basis of twelve assessment subjects, and provide tools to operate in a structured way which will be of benefit to assessment developers and assessors, as well as those carrying out analyses or providing assurance such as the members of examination committees.

Formation and validation of 180 guidelines

Based on the analysis of 60 scientific and other sources, a total of 1,513 guidelines were identified for the development and analysis of assessments. These were grouped under twelve assessments subjects. After de-duplication and, where necessary, supplementation and re-formulation, 371 guidelines remained. These were submitted to twelve expert and experienced assessment professionals, six with doctorates, for purposes of validation (ascertaining the validity of the result). These twelve assessment experts were: Jan Adema, Monique Altemühl-Booltink, Irene Biemond, Joost Dijkstra, Silvester Draaijer, Tom Erkens, three collaborating assessment experts of Examination Services, Desirée Joosten-ten Brinke, George Moerkerke, Bart Roosenboom, Tamara van Schilt-Mol and Gerard Straetmans.

The validation required relevance and clarity of the 371 guidelines. Intraclass correlation calculations were used to evaluate consensus among the assessment experts, who between them made a total of 1,905 remarks. The processing of the information obtained from the validation resulted in 208 guidelines which were subsequently dealt with during a session conducted by three assessment experts (Irene Biemond, Gerard Straetmans and Harry Molkenboer), eventually resulting in 180 validated guidelines.

Guidelines were only gathered and validated if they referred to a concrete assessment product such as a learning objective, assessment matrix, open question or an assessment. Guidelines to do with the process of arriving at assessments, such as having an assessment evaluated by a colleague or the organisation of taking a test were not gathered and validated. Admittedly, such processes do contribute to the quality of assessment, but they are secondary in nature. The best results can be achieved through guidelines for assessment products.

Overview of guidelines

This edition contains the overview of 180 validated guidelines for assessments. These are divided into the assessment subjects as follows (in brackets the number of guidelines per subject).

• Guidelines for the basic design of an assessment (11)
• Guidelines for language aspects (5)
• Guidelines for learning objectives (9)
• Guidelines for an assessment matrix (4)
• Guidelines for cases (6)
• Guidelines for closed questions, especially multiple choice questions (32)
• Guidelines for ‘other’ types of closed questions (16)
• Guidelines for open questions (22)
• Guidelines for assignments (35)
• Guidelines for cut-off scores and marking (5)
• Guidelines for the assessment (25)
• Guidelines for assessment and item analysis (10)

Cost

The English edition contains 32 pages and cost € 24,95 including 9% Tax, excluded shipping costs (€ 3,90 for Holland and € 8,70 for abroad).
Order

The English edition is realised in co-operation with Chhetri Ober, director of Onderwijs.pro.