Multimodal transcription as a tool of understanding visual metaphors

The articles presents an ongoing study of visual metaphors in terms of cognitive approach. Visual metaphor is viewed by the authors as a cognitive tool that structures the perception of the world. The perception of visual metaphors is analysed by means of multimodal transcription that allows to decipher the semiotic codes that produce the meaning. Multimodal transcript is made applying the system of 14 semiotic codes represented by verbal and non-verbal elements of the visual metaphor. The metaphors are divided into motionless and motion ones depending on the type of channel that is engaged for perceiving and processing the information. Depending on the type of the visual metaphor different semiotic codes are transcribed in the structure of the metaphor.


Introduction
Sight has always been one of the essential senses of the human being. It is considered that about 80 % of the information people get and process is obtained by means of sight. This can be proved by the fact that nowadays most of the information the modern society perceives is provided via visual aids: pictures, memes, gif-images, short videos, motivational videos in linear presentationformat, vlogs etc. Hence, it is natural that in order to enhance the processes of perception and cognition visual metaphor is used both for educational purposes and for those of entertaining, advertising, informing and (in some cases even) manipulating people.
Visual metaphor has been in the focus of research in recent years due to the so called 'visual turn' in terms of which the metaphor in general is viewed not as a rhetorical device but as a fundamental tool that structures how people think about abstract ideas. Charles Forceville divides visual metaphors into monomodal and multimodal ones. The modality in Forceville's interpretation is defined by the source of information rather than by the mode of perception (audial, visual, etc). Thus, monomodal metaphors are represented in the same modality whereas in multimodal metaphors the source and the target are represented in different modalities [1,2].
The above mentioned modalities, according to Forceville, are represented by eight sources of information: x oral communication; x written communication; x images; x sounds; x non-verbal sounds; x smell; x taste; x and touch [2]. Monomodal metaphors are divided by Forceville into four types: 1)hybrid pictorial metaphor where one object is made up of two parts belonging to different spheres, and the meaning is perceived by interpreting one part in terms of the other; 2)contextual pictorial metaphor in which the target domain is not represented in the visual image but is interpreted with the reference to the visual context; 3)pictorial similes, images combining the target and source domains presented close to each other or interlaced; 4)integrated metaphor where the target domain is represented in a posture or form similar to the source domain [1].
Speaking about audiovisual communication, Yves Gambier [9] sums up 14 semiotic codes that produce the meaning of a multimodal discourse element. The same classification of semiotic codes can be applied to visual metaphors in all their variety. These codes are active to different degrees in different types of metaphors (monomodal or multimodal ones). They include verbal and non-verbal elements and can be described as follows: As is seen from the table, eight out of fourteen semiotic codes refer to visual ones, that is the information is perceived via the visual channel. These codes apply to all the graphic elements of the visual metaphor, colours, perspective, facial features and postures, the use of space and distance, visual environment signs, the font and style of the inscriptions, etc. The relationship between these codes in all their complexity, as defined by Yves Gambier [9], can be summarized as follows: x redundancy (where one element of the visual metaphor repeats or emphasizes another one); x complementarity; x autonomy (in cases when one object is zoomed); x contradiction (one element of the metaphor is opposed to another one or to the whole meaning of the metaphor); x distance; x criticism; x help (one element of the visual metaphor aid in understanding of why the meaning is conveyed in a given way).
Different methods have been applied to studying visual metaphor in different fields of its application varying from market studies to sociology and psychology [10][11][12]. One of the common academic practices for research is transcription -turning a verbal text into writing for further analysis. It is widely used in a range of disciplines, such as discourse analysis, sociolinguistics, pragmatics, linguistic anthropology, etc. In recent years the traditional transcription has undergone certain changes. The emergence of multimodal transcripts was prompted by increasing availability of digital video technology bringing about the necessity to construct multimodal transcripts [13]. The use of multimodal transcription in the research of different types of visual metaphors can allow for the different modalities or semiotic codes to be dissected and analysed separately as parts of meaning production.

Materials and methods
The study of visual metaphors was carried out with the application of multimodal transcription -a new academic method applicable to elements referring to several modalities, or several semiotic codes.
The method of multimodal transcription was applied separately to monomodal and multimodal visual metaphors with reference to the semiotic codes as described by Yves Gambier.
The material for the study was chosen using the basic conceptual metaphor TIME IS MONEY represented in different sources as monomodal and multimodal visual metaphors.

Hybrid pictorial metaphor
The table below transcribes the verbal and non-verbal elements of the visual metaphor representing the basic conceptual metaphor TIME IS MONEY.  Verbal elements (signs) graphic code numbers on the clock, the inscription 'T$me is Money' with the sign of dollar substituting letter 'i' Non-verbal elements (signs) iconographic code the image of the clock representing the idea of time photographic code the green colour of the dollar sign and the word 'Money' scenographic code the inscription above the picture film code kinesic code proxemic code the verbal elements are situated above the image dress code - The combination of verbal and non-verbal elements is enhanced by the use of the green colour that is typically associated with the colour of money by means of reference to dollars as one of the most frequently used currencies. Another element enhancing the idea of money is the symbol of the American dollar substituting the letter 'i' in the word 'time'. The dollar sign is specifically accentuated by the use of the green colour that stands out on the background of the rest of the letters in the word. The contrast of the black and green colours draws the recipient's attention to the idea of money. The verbal element that is in some studies considered a tool of identifying the visual metaphor [14] presents the basic conceptual metaphor itself.

Pictorial simile
The given table presents a multimodal transcript of a monomodal visual metaphor -a pictorial simile representing the basic conceptual metaphor TIME IS MONEY.  The metaphor is visualized by means of such non-verbal elements as a red alarm-clock and a golden dollar symbol on a teeterboard. An unstable cylinder serves as a foundation of the teeterboard, yet, the board is absolutely stabilized by the alarm-clock and the dollar symbol which emphasizes the idea of the equation TIME IS MONEY.

Discussion
As is seen from the three examples of multimodal transcription, the iconographic code used in different types of visual metaphors is visualized by means of an image of the clock, with the image of the alarm-clock in two out of three cases. Visual metaphors may or may not include verbal elements of the graphic code, but this fact does not prevent interpretation of the meaning due to the abundance of such non-verbal elements as colour, position, background, etc.
Motionless visual metaphors, as described above, are interpreted with reference to the following semiotic codes: graphic (numbers, verbal inscriptions); iconographic (images inducing cognitive processes); photographic (colours); scenographic (organizing the elements of the metaphor in the picture); proxemic (positioning and distancing the elements of the metaphor). Motion visual metaphors (represented by gif-images, boomerang pictures, videos, etc) will also apply such semiotic codes as film and kinesic ones. If the motion visual metaphor is followed by an audio channel, all possible audio semiotic codes may be applicable to both structuring the metaphor and interpreting it.
Multimodal transcription can be used as an independent tool for analyzing visual metaphors, but can also be used in combination with advance methods applied in electrophysiological studies that can help uncover the principles of brain electrical activity while processing metaphoric images [15].
The reported study was funded by RFBR and FWF according to the research project № 25-512-14002