Cross-Modal Iconicity in Various Types of Discourse

Abstract

The paper presents a study of cross-modal iconicity which is viewed as a semiotic process of mutual assimilation of signs from different modalities via similarity. The research questions are as follows: Are there differences in cross-modal iconicity that can be observed between discourses with different combinations of semiotic systems and channels? How is iconicity revealed across signs belonging to verbal and non-verbal modalities in oral speech with gestures and in written texts with visual components. Relying on multimodal discourse analyses, the author investigates the interplay of verbal and non-verbal (gestural and pictorial) modalities and compares iconicity in two types of Russian and English discourse — spoken discourse (narratives, descriptions of paintings, and recipes) and written discourse (hybrid texts for news apps). It is demonstrated that irrespective of the type of discourse cross-modal iconicity follows the general principles outlined earlier for iconicity in language — i.e., “form mimes form and structure”, and “form mimes form”. The specific nature of iconicity in the two types of discourse (i.e., “form mimes function”) is linked to the character of semiotic heterogeneity, and, for the news app texts, to the affordances of the gadget, namely, to the scroll technology that determines the vertical movement of the main story, the limited nature of hypertextuality, and the compression of some parts of discourse into headings.

References

1. Loginova E.G. 2019. Polimodal’nye stsenarii semioticheskogo rezonansa v diskurse dramy [Multimodal scenarios of semiotic resonance in theatre discourse]. Kognitivnye issledovaniya yazyka, no. 38, pp. 120–141. (In Russ.)

2. Molchanova G.G. 2015. Sinergiya vizual’nogo i verbal’nogo v khromatike (zhivopisi) kak semioticheskii kod kommunikatsii [Synergy of the visual and the verbal in chromatics (paintings) as a semiotic code of communication]. Moscow State University Bulletin. Series 19. Linguistics and Intercultural Communication, no. 2, pp. 7–19. (In Russ.)

3. Sigal K.Ya. 2005. Preskriptornye pravila linearizatsii v kognitsii i tekste (na materiale russkikh sochinitel’nykh konstruktsii) [Prescriptor rules of linearization in cognition and text]. Voprosy kognitivnoi lingvistiki, no. 3 (4), pp. 11–25. (In Russ)

4. Sokolova O.V. 2014. Tipologiya diskursov aktivnogo vozdeistviya. Poehticheskii avangard, reklama i PR. Moscow, Gnozis. (In Russ.)

5. Cienki A., Müller C. (Еds.). 2008. Metaphor and Gesture. Amsterdam, John Benjamins.

6. DuBois J.W. 2014. Towards a dialogic syntax. Cognitive Linguistics, no. 25 (3), pp. 359–410.

7. Forceville C. 2009. Non-verbal and multimodal metaphor in a cognitivist framework: Agendas for research. In Forceville C., Urios–Aparisi E. (eds.). Multimodal metaphor. Berlin; New York, Mouton de Gruyter, pp. 19–42.

8. Freeman M.H. 2006. From metaphor to iconicity in a poetic text. In Benczes R., Csábi S. (eds.). The metaphors of sixty: Papers presented on the occasion of the 60th birthday of Zoltán Kövecses. Budapest, Eötvös Loránd University, pp. 127–135.

9. Haiman J. 1985. Iconicity in Syntax. Amsterdam, John Benjamins.

10. McNeill D. 1992. Hand and mind: What gestures reveal about thought. Chicago, Univ. of Chicago Press.

11. Mittelberg I. 2008. Peircean semiotics meets conceptual metaphor: Iconic modes in gestural representations of grammar. In Cienki A., Müller C. (eds.). Metaphor and Gesture. Amsterdam, John Benjamins, pp. 115–154.

12. Müller C. Iconicity and gesture. 1998. In Santi S. et al. (eds.). Oralite ́ et gestualite ́: Communication multimodale et interaction. Paris, L’Harmattan, pp. 321–328.

13. Müller C., Cienki A. 2009. Words, gestures and beyond: Forms of multimodal metaphor in the use of spoken language. In Forceville Ch., Urios-Aparisi E. (eds.). Multimodal Metaphor. Berlin, Mouton de Gruyter, pp. 297–328.

14. Nänny M., Fischer O. 1999. Iconicity in Language and Literature. Amsterdam, J. Benjamins Publishing Company.

15. Tabakowska E., Ljungberg C., Fischer O. (Eds). 2005. Insistent Images. Amsterdam, J. Benjamins Publishing company.

16. Zhao X. 2015. Multimodal poetic iconicity in Jimmy’s The Moon Forgets. Language and semiotic studies, vol. 1, no. 3, pp. 117–129.