In daily life, when we tell someone a story, report on an event or try to convince somebody of something, it is rarely only abstract information that we convey with our words. Rather our whole body takes part in the conversation: the sound of our voice, our facial expression and our gestures impart feelings, thoughts, opinions and actions.

This principle of embodiment can also be found in audiovisual media. News programs, feature films and TV series also communicate with their viewers on levels other than the merely lexical. At the same time, and to the same extent, they articulate moods and meanings by means of the multimodal interplay of cinematographic elements such as camera, mise-en-scène, shot composition, acting, sound design and editing—as sensory qualities and as patterns of movement.

But how exactly are processes of perception, feeling and understanding connected, in face-to-face and in media communication?

How do words and images interact multimodally and how can we describe the way in which, for example, this relationship directs our attention? 

These questions can be studied best by looking at the phenomenon of verbal images, since metaphors are an exemplary case of the bundling of sense and sensuousness. Metaphors produce imagery that make abstract thoughts concrete and perceptible to the senses.

Conceptual understanding and corporeal experience are almost completely conflated in metaphors.

We are therefore researching the concrete act and formation of the connection between emotional sensation and understanding in human communication by looking at varying multimodal metaphors—metaphorical processes in which different levels of expression work together.

Publications

Kappelhoff, H., Bakels, J.-H. (2011, Im Druck): Das Zuschauergefühl - Möglichkeiten empirisch orientierter Filmanalyse. Zeitschrift für Medienwissenschaft Nr. 5 (2/2011).

Kappelhoff, H., Müller, C. (2011, Im Druck). Embodied meaning construction. Multimodal metaphor and expressive movement in speech, gesture, and in feature film. Metaphor and the Social World (2).

Müller, C. (2011, Im Druck). Reaction Paper. Are ‘Deliberate’ Metaphors really Deliberate. A Question of Human Consciousness and Action. Metaphor and the Social World (1).

Müller, C., Tag, S. (2010, Im Druck). The Dynamics of Metaphor. Foregrounding and Activating Metaphoricity in Conversational Interaction. Cognitive Semiotics (Special Issue No 6, Spring 2010).

Lectures and Papers

Greifenstein, S., Rook, S., Kappelhoff, H., Müller, C., Schmitt, C., Tag, S. (2010). Multimodal metaphors and expressive movement in film. Audiovisual aesthetic structures modelling emotions. 4th Conference of Language, Culture and Mind (LCM4). Åbo Akademi University, Turku (Finland).

Kappelhoff, H., Müller, C. (2010). Multimodal metaphors and expressive movements in speech, gesture, and audiovisual images – A cognitive-linguistic and a film-analytic approach to multimodal metaphors. 8th International Conference on Research and Applying Metaphor (RaAM8): Metaphor and Domains of Discourse. Vrije Universiteit Amsterdam (Netherlands).

Schmitt, C., Tag, S., Kappelhoff, H., Müller, C., Greifenstein, S., Rook, S. (2010). Audiovisual orchestrations of metaphoric gestures in a TV report. 4th conference of the international society for gesture studies (ISGS'10) - gesture. evolution, brain, and linguistic structures. Europa-Universität Viadrina, Frankfurt/Oder (Germany).

Workshops

"Ausdrucksbewegung und Verkörperung" (21.-22. Mai 2010, FU Berlin)

"Dynamic Metaphor Analysis" (19. Oktober 2010, FU Berlin)

"Cognitive-linguistic Film Analysis - Multimodal Metaphor and Blending" (17.-18. November 2010, FU Berlin)