Evaluating the Health in my Language project - a tale of two translation approaches

Blog
Evaluating the Health in my Language project

For the past three years, Cochrane was a partner in the EU-funded project called Health in my Language (HimL), aiming to use advances in machine translation to make health content available in different languages. Here, Hayley Hassan, Translation Support Officer, presents the results of an evaluation task led by Cochrane as part of the project, which investigated whether post-editing a machine translation was faster than translating “from scratch”, and a follow-up interview with some of the translators involved.

What was the aim of Health in my Language?

HimL aimed to address the need for reliable health information in multiple languages by developing health-domain adapted machine translation systems.  This means, these machine translation systems were specifically trained to translate health information, which made them potentially superior to generic machine translation services like Google Translate.

HimL focused on building systems to translate from English into Czech, German, Polish and Romanian. The results of HimL aim to inform the potential use of machine translation for other languages and domains.

HimL

What role did Cochrane play in the project?

As part of HimL, Cochrane Plain Language Summaries and health information from NHS24 Scotland served as use cases to evaluate the quality and usefulness of the HimL machine translations. Accuracy is very important in the context of health information, as mistakes in translations could lead to patient harm. Cochrane therefore participated in a range of activities to carefully evaluate Plain Language Summaries that were translated using the HimL translation systems. We wanted to know whether they would be acceptable and useful to the public, or whether they could support the translation process.

Evaluating the quality of machine translations

In one of the evaluation tasks, we investigated whether it is quicker to post-edit a HimL machine translation or to translate “from scratch”, which usually constitutes our standard human translation process in Cochrane. 

We chose ten Plain Language Summaries at random and had them translated twice by two different translators – once translating from scratch and once post-editing the HimL machine translations.  We measured the time it took translators to complete either task.

What were the results?

Post-editing was clearly quicker than standard human translation for Czech, German, and Romanian. For Czech, post-editing reduced the required translation time by about 30% on average compared to standard human translation, for German by about 40%, and for Romanian by about 60%. For Polish, however, post-editing and standard human translation took about the same time.

Upon completion of the task, we invited the Polish and German translators to take part in a debrief: Joanna Zajac and Malgorzata Kolcz (Polish translators), and Andrea Puhl, (German translator), to learn about their experience in more detail.

Did you prefer to translate from scratch, or post-edit, and why?

Andrea Puhl: “I really preferred the post-editing. For German, the quality was already so high, translation was quite easy to do, and the focus could be on fine-tuning, instead of finding words. In some instances, the translation was so good we didn’t have to do anything.”

Joanna Zajac: “The Polish language is quite tricky. Automatic translations in Polish often do not account for the context, which some words strongly depend on. You have to be careful, read each word, since one wrong word could change the whole meaning. Long sentences in Polish do not work at all and cannot be translated using automatic translation. Good machine translations were probably only about 30%.”

Malgorzata Kolcz: “I found that for some simple sentences, it was easy, and the translation was almost perfect, but for more complex sentences, about 90% of the time, the translation was incorrect. It is because of the structure of the Polish language.”

We have been hearing a lot lately about the rise of machine translation. What do you think is the future of machine translations in your language?

MK: “I think in the future, machine translation will act as an aid to the translator.”

JZ: “Sentences have to be broken down. If we break them down into smaller pieces, maybe that could help improve the quality of machine translation.”

MK: “However, this would have to be done by the author, to avoid changing the meaning of the text. In the Cochrane context, we cannot change the original version of the text we want to translate.”

AP: “The role of the translator is not what it used to be but it is not obsolete. Fine-tuning is very important. As the Polish team notes, often one word can change things, if it’s not quite the right meaning, so you still need the translator. It could speed up translation, but you can’t fully replace a translator.”

Do you think the public trust the quality of machine translations?

AP: “As long as it is high quality, the public should not be suspicious.”

MK: “The future is automatic translation, there is so much information that needs to be translated nowadays. It will definitely be the future - it’s unavoidable. People will not see the difference between a human translation and a machine translation.”

JZ: “It will become even more and more useful. People are not suspicious, if they speak two languages they can test it themselves. From feedback we have had about our machine translations, people are not suspicious about them, especially if they are post-edited.”

Do you think machine translation could provide a sustainable approach for organisations such as Cochrane who rely primarily on volunteers to do translations?

MK: “There is so much Cochrane content to translate – it is a solution. But you would still need someone to check it. You cannot publish it without somebody checking the translated text. This is a very good option for Cochrane, it would probably be a quicker and easier way to translate all the information.”

AP: “I agree. It is definitely a very helpful option for somewhere like Cochrane, where there is a high volume of new information, and making this information available in other languages in a timely manner. Wrong information could have serious consequences so it is highly important to have editors to approve it before it is published.”

JZ: “This cooperation between human and machine is necessary. Sometimes it is worth splitting the text into shorter sentences to make it easier for the machine to translate.”

How could Cochrane change its Plain Language Summaries to result in better machine translation?

JZ: “As I have said, the shorter sentences make things so much easier. Sometimes they are three or four lines long, which is impossible for machine translation.”

MK: “Shorter sentences would be better all round, since it is meant to be in plain language, that can be written using normal words, understood by someone who is not a healthcare professional. At the same time, it would make it easier for a machine to translate.”

AP: “Cochrane could include patients or other stakeholders at a final edit stage to get their opinion.”

Any final comments?

AP: “I found it to be a pleasant experience and I was struck by the quality.  I didn’t expect it to be like that.”

 

With thanks to Joanna Zajac, Andrea Puhl and Malgorzata Kolcz for their participation and our HimL partners:

  • The University of Edinburgh (project coordinator)
  • Charles University, Prague
  • Ludwig Maximilian University of Munich

 

For more information about the project please visit www.himl.eu. The full set of project results will be available on the website soon.

 

January 24, 2018

The Cochrane Official Blog is curated and maintained by the Knowledge Translation Department. To submit items for publication to the blog or to add comments to a blog, please email news@cochrane.org.

The Cochrane Blog presents commentary and personal opinion on topics of interest from a range of contributors to the work of Cochrane. Opinions posted on the Cochrane Blog are those of the individual contributors and do not necessarily reflect the views or policies of Cochrane.