Many of the texts that are translated today are multimodal in one way or another: they consist of various interrelated modes, for example, image and word or speech and body language. In multimodal texts, such as films or illustrated texts, verbal language is accompanied by other kinds of visual and/or auditive information that interact with words and influence the eventual interpretation of the multimodal text. The translation and the translational enquiry of these texts needs to include a careful consideration of all of the modes involved and the ways in which the modes combine to make meaning. In addition to the multimodal forms of interlingual translation, the study of multimodal meaning construction is notably relevant in the field of accessible translation: audio description, for instance, entails converting visually represented information into a verbal form, and requires an in-depth understanding of the multimodal entity.
The MULTI (Multimodality in Translation and Interpreting) research group investigates the implications of multimodal meaning construction in translation and interpretation. The group examines how different aspects of multimodality affect the work of translators and interpreters on the one hand, and the receivers of these products on the other. The research interests of the group cover a variety of languages (Finnish, English, Russian, German, Hungarian, Italian, and Spanish), and areas of multimodal intercultural communication, such as interpretation, audiovisual translation, audio description, and the translation of illustrated texts and comics.
MULTI’s research focuses on
- developing novel research approaches that can address the diversity of multimodal data
- reviewing the current state of research on multimodal translation within translation studies and promoting awareness of the importance of such research to the discipline and the disciplines close to it
- envisioning and encouraging new ways of thinking about and conceptualizing translation from the viewpoints of multimodality, materiality and intermodal and intersemiotic translation
- developing interdisciplinary research approaches that allow for the emergence of new and broadened perspectives into multimodal translation
MULTI studies the communicative functions of the different modes in relation to each other and how
- a translation interacts with other modes as a part of a multimodal orchestration
- meanings transfer from one mode to another
- material media shapes the contents of the multimodal orchestration
Co-PIs
Maija Hirvonen (tenure track professor)
Eliisa Pitkäsalo (university lecturer, docent, Sarjis)
Members
Anne Ketola (university researcher, PhD, Sarjis)
Riitta Oittinen (emerita university lecturer, docent)
Laura Hekanaho (post-doctoral researcher)
Chiara Galletti (doctoral researcher)
Riku Haapaniemi (doctoral researcher)
Tero Ikävalko (doctoral researcher)
Mira Kainulainen (doctoral researcher, Sarjis)
Betta Saari (doctoral researcher)
Affiliated groups
Eliisa Pitkäsalo, Laura Kalliomaa-Puha, Anne Ketola, Vaula Haavisto, Kirsi Günther, Mira Kainulainen, Jan Pitkäsalo
Sarjis is searching for ways to make legal documents more accessible and find out whether the visualisation of documents can improve the accessibility of information. Making diffcult content (e.g. concepts) easier to grasp makes the text easier to understand and, therefore, supports the client’s autonomy and participation in society. In its current projects, Sarjis is translating various kinds of documents into comics.
Funded projects
Guided Audio Captioning for Complex Acoustic Environments
Annamaria Mesaros (PI), Irene Martin Morato, Manu Harju (Audio Research Group), Maija Hirvonen, Riku Haapaniemi (MULTI). Jane and Aatos Erkko Foundation (2022-2023)
The project develops a novel approach for machine learning to automatically produce linguistic descriptions of everyday acoustic environments. The general aim is to increase the understanding of how people (sighted and blind) interpret the world through sound and to teach intelligent machines to simulate it.
Using Language to Interpret Unstructured Data
Tuomas Virtanen (PI, Audio Research Group), Parthasaarathy Ariyakulam Sudarsanam, Huang Xie (Audio Research Group), Maija Hirvonen, Laura Hekanaho, Frank Emmert-Streib (Predictive Medicine and Data Analytics Lab), Nadeesha Perera, Pavel Davidson (Data Science). Emil Aaltonen Foundation (2021-2023)
The interdisciplinary project developed deep learning methods to compute unstructured data, such as images and sounds, into linguistic representations by learning from human multimodal communication and intermodal translation. This multidisciplinary research project combined signal processing, machine learning, health technology, translation studies and linguistics. The project was carried out at Tampere University, the Faculty of Information Technology and Communication Sciences (ITC).
Graphic Justice
Eliisa Pitkäsalo (PI) and the Sarjis research group. Academy of Finland (2020–2024)
Our multidisciplinary research group examines how the comprehensibility of different types of legal documents – such as contracts or court rulings – could be improved by using comic-style communication. Official documents can often be difficult for clients to understand, due overly complicated language, unfamiliar terms, etc. Misunderstandings can impede people from accessing their social rights. The Graphic Justice research project aims to tackle this problem by examing whether a comic-style document can be legally binding and whether visual presentation of information can make the document easier to understand and hence improve the client’s access to justice. The project is carried out in collaboration between researchers from the Faculty of Social Sciences (SOC) and the Faculty of Information Technology and Communication Sciences (ITC). The Principal Investigator of the project is Eliisa Pitkäsalo (ITC).
Word to Image
Laura Kalliomaa-Puha (PI), Eliisa Pitkäsalo, Anne Ketola, Vaula Haavisto, Kirsi Günther, Mira Kainulainen, Jan Pitkäsalo. Kone Foundation (2020–2023)
The Word to Image research project examines the visualization of social welfare documents. The language of institutional texts can often be difficult to comprehend, especially for clients with limited language or reading skills or different types of cognitive challenges that affect reading. The research project examines if the comprehensibility – and, hence, accessibility – of social welfare documents could be improved by converting the documents into comics. Further, the project sheds light on what the creators of comic-style documents need to take into account in order to produce documents that are legally binding also in a visual format. One example of a comic-style document the project is working on is an agreement for a supervised exchange, created in collaboration with the Finnish Federation of Mother and Child Homes and Shelters. The project is carried out in collaboration between researchers from ITC and SOC. The Principal Investigator of the project is Laura Kalliomaa-Puha.
MUTABLE: Multimodal Translation with the Blind
Maija Hirvonen (PI). Academy of Finland (2017–2020)
MUTABLE studies the collaboration and interaction in audio description teams that consist of blind and sighted members. The project sheds new light on collaborative translation processes, shared cognition, and multimodal interaction.
Ongoing dissertations and other research projects
The One Who Speaks Last Speaks for All. Mediating Mechanisms at Play in the Collaborative Indirect Translation of Multimodal Children’s Storybooks (doctoral dissertation)
Chiara Galletti
The dissertation project deals with a relatively common practice in children’s literature: the indirect translation of multimodal storybooks. This practice involves the collaboration between a professional in charge of producing a preliminary word-for-word translation crib and a storyteller in charge of shaping the voice of the story in view of the read-aloud situation. Applying Andrew Chesterman’s (2019) notion of consilience, the theoretic framework includes the studies on indirect translation carried out by the IndirecTRans international research network (Assis Rosa, Pięta and Bueno Maia 2019) and Marco Dallari’s narrativity studies in the field of multimodal children’s literature (2012). The empirical investigation concentrates on a small-scale corpus of multimodal storybooks translated indirectly by the Italian children’s author Bianca Pitzorno. The methodology involves a combination of contextual and textual descriptive analysis.
Materiality of texts in translation studies (doctoral dissertation)
Riku Haapaniemi, Finnish Cultural Foundation grant
This dissertation project discusses the concept of materiality in the study of translation, utilising perspectives from semiotics and textual theory. Materiality is identified as a common concern across all kinds of textual communication: modes of expression must always have material forms that can be identified by recipients, and the materiality of these forms affects the meaning constructed from the text. Translation Studies has a unique perspective on material processes of textual communication, and a material approach is useful in studying a variety of different translational phenomena.
Comic documents as an instrument of improving accessibility and social inclusion (doctoral dissertation)
Mira Kainulainen, Sarjis research group
The topic of the doctoral thesis is comic documents as an instrument of accessibility and social participation. Comic documents are comic versions of text documents. The goal of the research is to further develop functional translation theory for intersemiotic translation and accumulate information on the characteristics and functionality of comic documents from the viewpoint of their users. Accessible communication improves users’ social participation because it helps them to understand communication about and for them.
The role of interaction and language use in the construction of audio described visual art museum experiences of visually impaired people (doctoral dissertation)
Betta Saari, Emil Aaltonen Foundation grant
The dissertation project studies how social interaction and language use construct visual art museum experiences of visually impaired people on audio described and guided art museum tours. The multidisciplinary project is part of fields of linguistics, accessibility research and experience research. From the viewpoints of social interaction and language use, I examine how experiences of visual art are formed as cooperation on guided, interactive and audio described art museum tours. The project increases knowledge about the collaborative construction of a visual art experience of visually impaired people through social interaction.
The role of the audio describer’s voice in an art experience mediated by audio description
Tero Ikävalko
In this project, statistical analysis of voice features is used to analyse how the characteristics of an audio describer’s voice affect the art experience of audio description users.
Recent Publications
You can find the up-to-date research listings on each researcher’s Tunicris profile. You can access Tunicris by clicking on the researcher’s name at the section titled Research community.
2023. Haapaniemi, Riku. Translation as meaning-construction under co-textual and contextual constraints: A model for a material approach to translation. Translation Studies. DOI: 10.1080/14781700.2022.2147988
2023 (forthcoming). Anne Ketola, Eliisa Pitkäsalo & Robert de Rooy. Contracts via comics: Migrant workers and Thai fishing vessel employment contract. Teoksessa Comics and Migration: Practices and Representation, toim. Ralf Kauranen, Olli Löytty, Aura Nikkilä & Anna Vuorinne. London: Taylor & Francis.
2023 (forthcoming). Pitkäsalo, Eliisa & Riitta Oittinen. On Translation and Censorship in Children’s Literature. Teoksessa Routledge Handbook of Translation and Censorship, toim. Denise Merkle & Brian Baer. London: Taylor & Francis.
2022. Hirvonen, Maija & Betta Saari. Scripted or spontaneous? Two approaches to audio describing visual art in museums. Perspectives: Studies in Translatology, (E-pub ahead of print) https://doi.org/10.1080/0907676X.2022.2046816
2022. Pitkäsalo, Eliisa, Anne Ketola, Vaula Haavisto & Laura Kalliomaa-Puha. Image analysis as a visualization tool – Translating contracts into comics. Teoksessa Research Handbook on Contract Design, toim. Marcelo Corrales, Helena Haapio & Mark Fenwick. 347–364. Cheltenham: Edward Elgar Publishing.
2021. Ketola, Anne. Visual explicitation in intersemiotic translation. STRIDON: Studies in Translation and Interpreting 1 (1): 103–122.
2020. Hirvonen, Maija & Tuija Kinnunen (toim.). Saavutettava viestintä: Yhteiskunnallista yhdenvertaisuutta edistämässä. Helsinki: Gaudeamus. Sisältää artikkelit: Hirvonen, Kinnunen & Tiittula; Pitkäsalo & Kalliomaa-Puha; Hirvonen & Tiittula
2020. Hirvonen, Maija & Mari Wiklund. From image to text to speech – The effects of speech prosody on information sequencing in audio description. Text & Talk 41, 3, p. 309-334.
2020. Pitkäsalo, Eliisa (2020). Traduction intersémiotique et contexte : des contrats en bande dessinée en tant que documents juridiques accessibles. Meta: Translators’ Journal 65:1
2020. Pitkäsalo, Eliisa & Laura Kalliomaa-Puha (2020). Sarjakuvasopimukset – esimerkki oikeudellisen tekstin visualisoinnista. In Maija Hirvonen and Tuija Kinnunen (eds.): Saavutettava viestintä: Yhteiskunnallista yhdenvertaisuutta edistämässä. Gaudeamus.
2019. Pitkäsalo, Eliisa & Laura Kalliomaa-Puha. Democratizing access to Justice: the comic contract as intersemiotic translation. Translation Matters 2 (2019).
2018. Hirvonen, Maija. Audiodeskription und Sichtdolmetschen: Translation über Sinnesgrenzen. In Kvam, Sigmund, Parianou, Anastasia, Schopp, Jürgen & Solfjeld, Kåre (eds.): Spielräume der Translation: Sprach- und translationswissenschaftliche Zugänge. Münster: Waxmann, 100–124.
2018. Hirvonen, Maija & Tiittula, Liisa. How are translations created? Analysis of multimodal interaction as a methodology to study a team translation process. Linguistica Antverpiensia, Special issue “Methods for the Study of Multimodality in Translation” 17, 157–173.
2018. Hirvonen, Maija & Schmitt, Reinhold. Blindheit als Ressource: Zur professionellen Kompetenz eines blinden Teammitglieds bei der gemeinsamen Anfertigung einer Audiodeskription. Gesprächsforschung 19, 449-477.
2018. Jimenez Hurtado, Catalina, Anne Ketola &Tiina Tuominen (toim). Methods for the Study of Multimodality in Translation. Linguistica Antverpiensia – Themes in Translation Studies (17/2018).
2018. Ketola, Anne: Word–Image Interaction in Technical Translation. Students Translating an Illustrated Text (väitöskirja)
2018. Oittinen, Riitta & Eliisa Pitkäsalo. Creating Characters in Visual Narration: Comics and Picturebooks in the Hands of the Translator. In Search of Meaning and Identity Through Language; Literary, Linguistic, and Translational perspectives. Toim. Hanne Juntunen, Kirsi Sandberg & Kübra Kocabaş. Tampere Studies in Language, Translation and Literature, Series A5.
2018. Pitkäsalo, Eliisa. Images without words. Focus group as a method of experiencing comics. Methods for the Study of Multimodality in Translation. Toim. Catalina Jimenez Hurtado, Anne Ketola and Tiina Tuominen. Linguistica Antverpiensia -Themes in Translation Studies (17/2018).
2017. Hirvonen, Maija & Igareda, Paula. Imaginary places – Verbalization of setting in the English, German and Spanish audio descriptions of Slumdog Millionaire. mtm – A translation journal 9, 341–364.
2017. Ketola Anne. Using Translation Research to Model Word Image Interaction. New Voices in Translation Studies 17, 82–104.
2017. Oittinen, Riitta, Anne Ketola & Melissa Garavini. Translating Picturebooks. Revoicing the Verbal, the Visual and the Aural for a Child Audience. New York: Routledge.
2017. Pitkäsalo, Eliisa. Conveying Nonverbal Messages in Translated Comics. In: Sándor Maticsák & László Keresztes (eds.) Folia Uralica Debreceniensia 23. Debrecen: A Debrecen Egyetemi Kiadó. 233–243.
2016. Ketola, Anne. Towards a multimodally oriented theory of translation: A cognitive framework for the translation of illustrated technical texts. Translation Studies, 9 (1), 67–81.
2016. Pitkäsalo, Eliisa & Nina Isolahti (toim.) Kääntäminen, tulkkaus ja multimodaalisuus. Menetelmiä monimuotoisten viestien tutkimiseen. Tampere Studies in Language, Translation and Literature.
Other activities
MULTI participates in organising a digital study module called “Accessibility in a Digital Society” that has been available since 2021 under the life-long learning program of Tampere University. More information at the TACCU website (Tampere Accessibility Unit).
Contact
Maija Hirvonen, tenure track professor
maija.hirvonen@tuni.fi
+358 50 318 25 03
Eliisa Pitkäsalo, university lecturer, docent
eliisa.pitkasalo@tuni.fi
+358 50 318 1238
Anne Ketola, university researcher, PhD
anne.ketola@tuni.fi