Humans and AI Navigate Translation Challenges Similarly, Research Finds

In a paper published on December 19, 2023, researchers from the University of Melbourne, explored the efficacy of neural machine translation (NMT) models in predicting human translation difficulty and underscored the predictive capabilities of specific NMT-derived features in identifying challenges inherent in translation.

Drawing inspiration from Translation Process Research (TPR), a field that analyzes cognitive processes in translation by studying eye movements and keystrokes, Zheng Wei Lim, Ekaterina Vylomova, Charles Kemp, and Trevor Cohn focused on cognitive processing difficulty. Unlike conventional approaches that center on translation errors, they defined a word or phrase as challenging if it demanded extended processing time.

To quantify this difficulty, the researchers employed three variables: source reading time, target reading time, and translation duration, drawing data from the CRITT TPR-DB database. Their analysis encompassed data from 312 translators across 13 language pairs, with the researchers claiming it to be “the most comprehensive investigation of human translation difficulty to date.” 

The central goal, as highlighted by the researchers, was to “bring translation process research into contact with modern work on NMT.”

Translation Difficulty Predictors

The researchers utilized surprisal and attention features from NMT models — where surprisal measures the unexpectedness of a word or phrase and attention tracks the model’s focus allocation during translation — and tested whether these features could effectively predict which words or phrases are difficult for human translators.

In the realm of NMT, models are trained on large datasets to learn the relationships between words and their translations. During translation, the model predicts the probability distribution of possible target words given the source context. Surprisal quantifies how surprising or unexpected a particular word is based on the context provided by the preceding words. A word with higher surprisal indicates that it deviates from the model’s expectations, making it more challenging or unexpected in that specific translation context.

The researchers acknowledged that surprisal has been proposed as a predictor of translation difficulty in the past, with more contextually surprising words incurring higher cognitive load and being slower to process. However, they noted the relatively limited evaluations of this proposal.

State-of-the-art NMT models, relying on the transformer architecture, incorporate three kinds of attention: encoder self-attention, cross-attention, and decoder self-attention. The researchers considered all three sets of attention weights as potential predictors of translation difficulty, aligning them with the three stages of reading, transferring, and writing in the human translation process.

They found that attention and surprisal can effectively predict which words or phrases are challenging for human translators. Specifically, surprisal emerged as a “strong predictor” of translation difficulty, indicating that the more surprising a word, the more likely it is to be challenging for human translators. However, the most accurate predictions were achieved by combining surprisal and attentional features.

This suggests a strong alignment between these computational measures and human cognitive processes during translation, emphasizing the potential of NMT models in advancing TPR. “Our results support the prevailing view that current NLP models, including NMT, align partially with human language usage and are predictive of language processing complexity,” said the researchers.

Additionally, the demonstrated correlation between surprisal and translation difficulty reinforced the pivotal role of cognitive processing in translation, providing valuable insights into the underlying cognitive processes in translation.