According to researchers behind the project, the output of their machine translation (MT) engine has “reached the level of professional translators specialized for the financial sector.” Yes, we’ve heard it before. With Google (2016), Microsoft (2018), and a few others. Based on the very limited information released by the two agencies in English, this is what we know.
Prior to launching the engine, researchers conducted a study into translation quality. The process entailed gathering translation documents owned by the JFSA and financial industry groups. The NICT then refined these documents into MT training data (i.e., a standard process).
Slator Machine Translation Expert-in-the-Loop Report
60-page report on the interaction between human experts and AI in translation production, including AI-enabled workflows, adoption rates, postediting, pricing models.
When comparing the quality of translation for the same text produced by a “general-purpose translation engine” (unspecified in the English press release) versus the NICT’s newly developed “high-precision AI translation engine,” researchers found the percentage of what would be deemed a professional translation by financial translators had risen from 21% to 49%. Those deemed “NG” (No Good) was reduced from 21% to 12%.
As usual, claims of human parity would behoove anyone working today in language technology, or AI for that matter, to raise questions on how extensive (or narrow) the test samples used for evaluation were. And how context and other test parameters were selected.
That said, it is remarkable that Japan has chosen machine translation as an important pillar in its attempt to further internationalize its financial markets. Moreover, the Japanese government conducted a fairly large-scale research project to this end, which is now being deployed for corporate use.
Now, the claim of MT having “reached the level of professional translators” will be put to the test daily and at scale.
