The new ISO 5060 standard was published in February 2024. It “is applicable to translation service providers […], including individual translators, translation companies or in-house translation services, their clients and other interested parties in the translation sector, such as translator education and training institutions.” Aimed at providing guidance for human evaluation of the translation output, ISO 5060 can be applied to workflows involving both human and machine translation with or without subsequent post-editing.
The standard combines hands-on advice with the strategic considerations and general principles for translation quality evaluation. It is built on the analytic approach that involves a “bilingual examination of target language content against source language content while classifying any errors with respect to translation evaluation specifications and for the purpose of reaching a quality rating.” Focus on the translation requirements and treating any deviation from it as an error allows for a high degree of inter-rater reliability (IRR; guaranteeing that the results of evaluation don’t differ significantly depending on evaluator) and evaluation objectivity.
“The standard enables the user to produce reliable, trustworthy translation quality data and is therefore something completely different than MT predictive quality estimation.” — Christopher Kurz, member of the ISO/TC 37/SC 5 committee
ISO 5060 introduces a framework of concepts related to the evaluation of translation quality, and provides a set of requirements for all steps of the evaluation process, covering the pre-evaluation, evaluation, and post-evaluation phases. Error typology that has been harmonized with the first level Multidimensional Quality Metrics Framework (MQM), as well as guidelines on severity, weight, or the approach to repeated errors, form an important part of the standard.
According to the proposed typology, errors can fall into one of seven main categories:
- Terminology;
- Accuracy;
- Linguistic conventions;
- Style;
- Locale conventions;
- Audience appropriateness; and
- Design and markup.
Each of the categories contains several sub-categories and every error can be classified as Critical, Major, or Minor, depending on its severity.
Additionally, the standard outlines the competencies of evaluators, enumerating aspects such as: understanding of the source language, fluency in the target language, cultural ability, knowledge of the domain, as well as formal qualifications, among others.
Annexes provide readers with a set of questions to guide them through the process of analyzing the evaluation needs and to help them develop a translation evaluation strategy and system. Practical recommendations on sampling, a detailed explanation of the concepts of cohesion and coherence, as well as ready-to-go scorecard templates are also included.
The next revision of the standard is planned for 2029.
Interested parties can obtain a copy of the ISO 5060 standard and the other standards related to language services from the ISO website.
