Many of the texts that are translated today are multimodal in one way or another: they consist of various interrelated modes. In multimodal texts, such as films or illustrated texts, verbal language is accompanied by other kinds of visual and/or auditive information that interact with words and influence the eventual interpretation of the multimodal text. The translation and the translational enquiry of these texts needs to include a careful consideration of all of the modes involved and the ways in which the modes combine to make meaning. In addition to the multimodal forms of interlingual translation, the study of multimodal meaning construction is notably relevant in the field of accessible translation: audio description, for instance, entails converting visually represented information into a verbal form, and requires an in-depth understanding of the multimodal entity.

The MULTI (Multimodality in Translation and Interpreting) research group investigates the implications of multimodal meaning construction in translation and interpretation. The group examines how different aspects of multimodality affect the work of translators and interpreters on the one hand, and the receivers of these products on the other. The research interests of the group cover a variety of languages (Finnish, English, Russian, German, Hungarian, Italian, and Spanish), and areas of multimodal intercultural communication, such as interpretation, audiovisual translation, audio description, and the translation of illustrated texts and comics.

MULTI’s research focuses on

  • examining the communicative functions of the different modes in relation to each other and evaluating how a translation interacts with other modes as a part of a multimodal orchestration
  • reviewing the current state of research on multimodal translation within translation studies and promoting awareness of the importance of such research to the discipline
  • developing novel research approaches that can address the diversity of multimodal data
  • envisioning and encouraging new ways of thinking about and conceptualizing multimodal translation
  • developing interdisciplinary research approaches that allow for the emergence of new and broadened perspectives into multimodal translation

 

Members

Chiara Galletti (UTA)

Melissa Garavini (Independent Researcher)

Maija Hirvonen (UH)

Anne Ketola (UTA)

Sari Kokkola (UTA)

Riitta Oittinen (UTA)

Eliisa Pitkäsalo (UTA)

 

Projects

Eliisa Pitkäsalo & Annikki Liimatainen: Sarjakuvan monikielisyys ja monikulttuurisuus yhteiskunnallisen muutoksen ilmiönä

Anne Ketola: Word–Image Interaction in Technical Translation. Students Translating an Illustrated Text (PhD Dissertation)

Chiara Galletti: An Author’s Adventures in Translating for Children: Bianca Pitzorno and the Poetics of Translation (PhD Dissertation)

Sari Kokkola: Embodied Multimodal Film Analysis for Translators: A Phenomenologically Informed Approach (PhD Dissertation)

Maija Hirvonen: MUTABLE: Multimodal Translation with the Blind. Academy of Finland (2017-2019)

 

Recent Publications (Selection)

2017. Ketola Anne. Using Translation Research to Model Word Image Interaction. New Voices in Translation Studies 17, 82–104.

2017. Oittinen, Riitta, Anne Ketola & Melissa Garavini. Translating Picturebooks. Revoicing the Verbal, the Visual and the Aural for a Child Audience. New York: Routledge.

2017. Pitkäsalo, Eliisa. Ääniä ja liikettä elokuvan ja sarjakuvan hybrideissä. Ääniä, Röster, Voices, Stimmen. Eds. Nicole Keng, Anita Nuopponen & Daniel Rellstab.  VAKKI Publications N:o 8. Vaasa, 169-180.

2017. Pitkäsalo, Eliisa. Conveying Nonverbal Messages in Translated Comics. Folia Uralica Debreceniensia 23. Eds. Sándor Maticsák & László Keresztes. Debrecen: A Debrecen Egyetemi Kiadó. 233–243.

2016. Ketola, Anne.  Translating picturebooks: Re-examining interlingual and intersemiotic translation. Proceedings of DRS 2016, 1179–1190.

2016. Ketola, Anne. Towards a multimodally oriented theory of translation: A cognitive framework for the translation of illustrated technical texts. Translation Studies, 9 (1), 67–81.

2016. Pitkäsalo, Eliisa & Nina Isolahti (eds.) Kääntäminen, tulkkaus ja multimodaalisuus. Menetelmiä monimuotoisten viestien tutkimiseen. Tampere Studies in Language, Translation and Literature.

2016.  Pitkäsalo, Eliisa. Tulkitse kuva! Eleet ja ilmeet sarjakuvassa. Teksti ja tekstuaalisuus. Vakki Publications N:o 7. Eds. Hirvonen, Pia, Daniel Rellstab & Nestori Siponkoski. Vaasa: Vakki. 438–449.

2016. Tuominen, Tiina, Maija Hirvonen, Anne Ketola, Eliisa Pitkäsalo ja Nina Isolahti. Katsaus multimodaalisuuteen käännöstieteessä. Kääntäminen, tulkkaus ja multimodaalisuus. Menetelmiä monimuotoisten viestien tutkimiseen. Eds. Eliisa Pitkäsalo & Nina Isolahti. Tampere Studies in Language, Translation and Literature, Series B3. 11–24.

2015. Kokkola, Sari & Ketola, Anne. Thinking outside the “Methods Box”: New Avenues for Research in Multimodal Translation. Vakki publications N:o 4, 219–228.

2014. Kokkola, Sari. The Role of Sound in Film Translation: Subtitling Embodied Aural Experience in Aki Kaurismäki’s Lights in the Dusk. Traduction, textes, médias, 27 (2), 17–47.

 

Forthcoming Publications

Jimenez Hurtado, Catalina, Anne Ketola and Tiina Tuominen (eds.). Methods for the Study of Multimodality in Translation. Linguistica Antverpiensia – Themes in Translation Studies (17/2018).

Oittinen, Riitta & Eliisa Pitkäsalo. Creating Characters in Visual Narration: Comics and Picturebooks in the Hands of the Translator.  In Search of Meaning and Identity Through Language; Literary, Linguistic, and Translational perspectives. Eds. Hanne Juntunen, Kirsi Sandberg & Kübra Kocabaş. Tampere Studies in Language, Translation and Literature, Series A5.

Ketola, Anne. Picturebook Translation as Transcreation. In Search of Meaning and Identity Through Language; Literary, Linguistic, and Translational perspectives. Eds. Hanne Juntunen, Kirsi Sandberg & Kübra Kocabaş . Tampere Studies in Language, Translation and Literature, Series A5.

Pitkäsalo, Eliisa. Images without words. Focus group as a method of experiencing comics. Methods for the Study of Multimodality in Translation. Eds. Catalina Jimenez Hurtado, Anne Ketola and Tiina Tuominen. Linguistica Antverpiensia -Themes in Translation Studies (17/2018).

 

Other

Introduction to Multimodality – International Online Course for Translation Students (Fall of 2016)

In the fall of 2016, MULTI organized and hosted an international online course in translation, covering topics such as the translation of different types of illustrated texts (ads, children’s picturebooks, comics and illustrated medical texts), and audiovisual translation in television and opera, as well as accessible translation, including audiodescription and subtitling for the deaf and hard-of-hearing. The course featured 13 lecturers from eight European universities, and 28 students from the universities of Tampere and Helsinki.