A formative usability evaluation (formative evaluation) is a development-accompanying assessment of the user interface or the user-product interaction with the aim of identifying strengths, weaknesses and unforeseen use errors early. It does not serve as the final proof of the safe usability of the user interface but as the basis for its iterative improvement. It is a central component of the usability engineering process according to IEC 62366-1.
Role in the usability engineering process
Formative evaluations are typically carried out several times during development. They close the gap between use requirements, risk assessment and the later summative evaluation. Results feed directly into the further development of the user interface, instructions for use, training materials or system architecture. In contrast to a final validation, the goal is not to demonstrate safety and usability but to uncover design problems. This allows risks to be reduced before the final design is fixed.
Delimitation from summative evaluation
The most common confusion is between formative and summative evaluation. Both can use the same methods but pursue different goals. Formative evaluations are exploratory: Moderation, follow-up questions, design variants and spontaneous adjustments are permissible. Summative evaluations, by contrast, serve to obtain objective evidence for the selected hazard-related use scenarios that the user interface can be used safely; there, the study procedure and data collection must be controlled much more strongly. For regulatory acceptance, formative evaluation is no substitute for a summative evaluation.
| Formative evaluation | Summative evaluation | |
|---|---|---|
| Purpose | Exploratory | Objective evidence for the selected hazard-related use scenarios that the user interface can be used safely |
| Conduct | Moderation, follow-up questions, design variants and spontaneous adjustments are permissible | Study procedure and data collection must be controlled much more strongly |
| Methods | Both can use the same methods | Both can use the same methods |
Typical methods
Formative evaluations can be carried out with different methods:
- Contextual use simulations
- Moderated usability tests
- Cognitive walkthroughs
- Expert reviews
- Heuristic evaluations
- Mock-up and prototype tests
- Wizard-of-Oz scenarios
- Focus groups with concrete use scenarios
Especially for medical devices, early prototypes are often examined with users from the relevant user groups in order to identify potential use errors, use difficulties and weaknesses of the user interface. The choice of method depends on the stage of development and the identified use-related risks.
Link to the use-related risk analysis
Formative evaluations are closely linked to the use-related risk analysis (URRA). They serve to check suspected use errors, uncover previously unknown use errors, better understand hazardous situations and assess risk control measures. In practice, many relevant insights arise not from the theoretical risk analysis but only from observing real users. Formative studies therefore often provide input for updating the use-related risk analysis.
Iterative design optimization
The actual benefit of formative evaluation lies in iteration. A single formative study usually has only limited value; only repeated execution after design changes enables systematic improvement of the user interface. A typical cycle comprises identifying a problem, deriving a design measure, implementing it and a renewed formative check. The cycle is continued on a risk basis until the user interface is mature enough to carry out the planned summative evaluation. The acceptability of the remaining risk is assessed in the risk management process.
Common weaknesses in practice
The following deficits occur regularly in audits and projects:
- Carrying out formative studies only shortly before the summative evaluation
- Missing link to the use-related risk analysis
- Documentation merely as a general usability test report
- Concentration on operating comfort instead of safety-relevant use
- Selection of non-representative user groups
- No follow-up of identified problems and design changes
Particularly problematic is carrying out a single formative investigation without recognizable iteration. This loses the actual purpose of the method.
Regulatory reference
IEC 62366-1 requires the planning and execution of formative assessments as part of the usability engineering process; the goal is to evaluate and improve the user interface during development in order to identify design weaknesses, potential use errors and risks early. In its human factors guidance, the FDA recommends carrying out several formative investigations during product development and defines formative evaluation as an assessment of the user interface to identify strengths, weaknesses and unexpected use errors. The EU MDR does not explicitly require formative evaluations but does require evidence that ergonomics, user characteristics and the reduction of use-related risks have been taken into account. Implementation usually takes place through the usability engineering process according to IEC 62366-1.
Formative usability evaluation is the iterative engine of design optimization, not the proof of effectiveness. Even a single formative evaluation provides valuable insights; it unfolds its full potential, however, through repeated execution after design changes and through consistent feedback into the use-related risk analysis.
Frequently asked questions (FAQ)
Does every formative evaluation have to be carried out with real users?
No. Early formative assessments can also be carried out as an expert review, cognitive walkthrough or prototype assessment. For safety-critical questions, however, representative users usually provide considerably more robust findings.
Are there regulatory requirements for the sample size?
There are no fixed regulatory sample size requirements for formative evaluations. The number of participants should be based on the development goal, the risk and the expected depth of insight.
Can results of a formative evaluation be used directly as validation evidence?
No. Formative evaluations serve to improve the design. The regulatory evidence of safe use is normally provided through a separate summative evaluation or human factors validation.
Does every design change have to be checked by a new formative evaluation?
Not necessarily. Significant changes to safety-relevant user interfaces should, however, be assessed on a risk basis and, where applicable, examined again formatively.
Do you want to test your product design early and iteratively with representative users before costly design decisions are made? We support you throughout the usability engineering process according to IEC 62366-1.
More about our usability engineeringSources
- IEC 62366-1:2015+AMD1:2020, Medical devices, Part 1: Application of usability engineering to medical devices
- Regulation (EU) 2017/745 on medical devices (MDR)
Related terms
- Summative usability evaluation
- Use-related risk analysis (URRA)
- Cognitive walkthrough
- Representative users