• EMMA: European Multiple MOOC Aggregator
• Period: 1/2/2014 – 31/7/2016
• Project website: http://www.europeanmoocs.eu/
• Project supported by the European Union’s Competitiveness and Innovation Framework Programme (CIP) under grant agreement no. 621030.

EMMA – Providing multilingual access to European MOOCs

Project partners: Università de Napoli Federico II (Italy; coordinator), Universitat Politècnica de València (Spain), ATOS (Spain), IPSOS srl (Italy), Universitat Oberta de Catalunya (Spain), University of Leicester (UK), Open Universiteit (Netherlands), Universidade Aberta (Portugal), Université de Bourgogne (France), Tallinn University (Estonia), CSP – Innovazione nelle ICT S.C.A R.L (Italy), ATiT (Belgium).

The EMMA project was born out of research into MOOCs in Europe. Results of an expert survey launched in 2013 showed that European linguistic and cultural diversity were non-negotiable values to reach audiences in other parts of world, and that major aims were to create a European brand on the international stage, and to connect learning to instructional design through learning analytics. These findings became the basis for the EMMA project proposal.

EMMA’s aim has been to integrate and refine separate technological components to provide a cross-lingual platform for the delivery of diverse MOOCs. To achieve this goal, EMMA addressed the linguistic challenge directly by supporting transcription and translation services that provide direct translation between the original languages in which courses are developed and the learner’s language (via English).

The EMMA pilot provides in-domain automatic transcription systems for English, Italian, Spanish, French, Portuguese, Dutch and Estonian, and automatic translation systems from Italian, Spanish, French, Portuguese, Dutch and Estonian into English, and from English into Italian and Spanish. Transcription and translation services have been fully embedded into the EMMA platform, becoming a unique and innovative feature on a MOOC platform.

Scientific evaluations

The scientific evaluation of automatic transcription and translation services was carried out in terms of Word Error Rate (WER) and Bilingual Evaluation Understudy (BLEU), respectively. When reading the chart below, the lower the WER, the higher the transcription quality; the higher the BLEU, the higher the translation quality.

The EMMA automatic transcription and translation systems created by the MLLP were compared to mainstream providers, such as YouTube for transcription and Google Translate for translation. The table below shows the percentage of improvement of EMMA systems over mainstream providers.

Reviewing time

Even state-of-the-art automatic transcriptions and translations are error prone. However, they can be reviewed by course designers, teachers or even volunteers to produce accurate enough materials for students with little effort. The post-editing of automatic transcriptions and translations has been shown to significantly reduce the effort needed compared to a completely manual approach without automatic systems.

The time required for reviewers to post-edit automatic video transcriptions and translations is measured in terms of Real Time Factor (RTF). This measure is the video duration-normalized time required for the reviewer to post-edit the whole video transcription or translation. For instance, if a video lasts 6 minutes and its review takes one hour (60 minutes), then the RTF will be 10.

The generation ex novo of transcription and translation for a video is estimated in average at 10 RTF and 30 RTF, respectively. However, as shown in the table below, the time devoted to this process is significantly reduced when they are generated from post-editing automatic draft transcriptions and translations.

User satisfaction

On the user satisfaction side, more than 90% of learners felt that the translation feature added value to EMMA and that it made EMMA a truly European experience. 74% of learners used the translation function to access course content, and a large percentage of the learners who were surveyed appreciate that the platform is offered in multilingual version, 12% of the learners having chosen to switch between one language and another when following a MOOC to improve their language skills. Survey data indicates that automatic transcription and translation systems represent both an asset to be exploited by the EMMA consortium, and is appreciated as an option by the users because it increases their degree of freedom.

Indeed, the translation of the EMMA courses to a second, or even a third, language attracted a large number of students. This fact is quantified for the courses delivered in each language in terms of relative increase of enrolled students with respect to enrolled students taking the course in the original language.

Further details of scientific evaluations, reviewing times and user satisfaction will be published in a forthcoming article of the Journal of Educational Technology & Society: “Multilingual videos for MOOCs and OER” (Juan Daniel Valor Miró, Pau Baquero-Arnal, Jorge Civera, Carlos Turró, Alfons Juan).

EMMA has been funded in part under the CIP (Competitiveness and Innovation Programme) Framework Programme of the European Union with the rest of the funding necessary to make the platform operational coming from the partners themselves. You can find out more about the project in the EMMA website.