Jewitt, Carey, and Gunther R. Kress. Multimodal Literacy. New York: P. Lang, 2003.
“An explicit assumption in much multimodal work is that language is partial” (3). As multimodal theorists believe that there are always may modes in a communicational ensemble, the meaning of the message is distributed across all of these. The authors also explain “each mode is partial in relation to the whole of the meaning” (3) In their search for a better understanding of the multimodal, the authors ask: “What do we need to understand about the facilities offered by new communication technologies, in their configuration of modes and users of the media? What do we need to understand about new forms of message arrangements on the ‘page’ or the ‘screen’ and their role in learning?” (4) The authors here bring a social semiotic theory to multimodal theories of literacy, a theory which inherently emphasizes the “role of people in meaning-making , to their social agency” (4).
The authors believe: “social semiotics views the agency of socially situated humans as central to sign-making. “From a social semiotic perspective, people use resources that are available to them in the specific socio-cultural environments in which they act to create signs, and in using them, they change these resources…signs are constantly newly made, in a process in which the signified (what is to be meant) is realised through the most apt signifier (that which is available to give retaliation to that which is to be meant) in a specific social context” (10). And reading, according to a social semiotic theory, is also about sign making, but here, the process is reversed: “The process starts not from a person wanting to signify to the world outside, but from wanting to represent signs in the world outside (made by some other) to their inner self “ (13).
I find the authors discussion and research studies on meaning making and learning (how learning is a multimodal process) very informative and useful. What is needed, however, is a more thorough discussion of a social semiotic theory of the multimodal.