Jewitt, Carey, and Gunther R. Kress. Multimodal Literacy. New York: P. Lang, 2003.
“An explicit assumption in much multimodal work is that language is partial” (3). As multimodal theorists believe that there are always may modes in a communicational ensemble, the meaning of the message is distributed across all of these. The authors also explain “each mode is partial in relation to the whole of the meaning” (3) In their search for a better understanding of the multimodal, the authors ask: “What do we need to understand about the facilities offered by new communication technologies, in their configuration of modes and users of the media? What do we need to understand about new forms of message arrangements on the ‘page’ or the ‘screen’ and their role in learning?” (4) The authors here bring a social semiotic theory to multimodal theories of literacy, a theory which inherently emphasizes the “role of people in meaning-making , to their social agency” (4).
The authors believe: “social semiotics views the agency of socially situated humans as central to sign-making. “From a social semiotic perspective, people use resources that are available to them in the specific socio-cultural environments in which they act to create signs, and in using them, they change these resources…signs are constantly newly made, in a process in which the signified (what is to be meant) is realised through the most apt signifier (that which is available to give retaliation to that which is to be meant) in a specific social context” (10). And reading, according to a social semiotic theory, is also about sign making, but here, the process is reversed: “The process starts not from a person wanting to signify to the world outside, but from wanting to represent signs in the world outside (made by some other) to their inner self “ (13).
I find the authors discussion and research studies on meaning making and learning (how learning is a multimodal process) very informative and useful. What is needed, however, is a more thorough discussion of a social semiotic theory of the multimodal.
Diana George. “From Analysis to Design: Visual Communication in the Teaching of Writing.” College Composition and Communication 52.1 (September 2002): 11-39.
Diana George discusses the importance of bringing issues of visual literacy into the writing classroom. Primarily, George addresses the history of the visual within the field of composition studies from the 1940’s to the present. She claims that due to the history of composition studies, we have limited the possibilities for the visual in the teaching of writing. Reflecting on some examples of interesting student work George claims: “The work of these students and others like them has convinced me that current discussions of visual communication and writing instruction have only tapped the surface of possibilities for the role of visual communication in the composition class” (George 2002:12) She says: “Our students have a much richer imagination for what we might accomplish with the visual than our journals have yet to address” (12)
Attention visual rhetoric people! This comment of George’s must be addressed:
“Within the tradition of verbal/visual communication I am outlining here, only certain kinds of “visual” assignments seem possible for a writing course. Primarily, these would be assignments that use visual images as prompts for essay writing” (George 2002: 20).
Computers and Composition, volume 22, number 1 (Special Issue on New Media). Esp. Grigar, Dene. “Kineticism, Rhetoric, and New Media Artists.”
In this article, Grigar argues that while rhetorical analysis of the visual and aural have gained prominence, it is also important to consider the rhetorical kinetic act. She claims the kinetic activity refers to motion, however, in terms of new media it can be viewed as “new media works that turn readers into users and create characters who are no longer described as moving, but indeed dart, jump, run, roll…”(Grigar 105). She describes the work of two female new media artists, Jill Scott and Magerete Jahrmann, whose work has an element of the interactive, oftentimes through the use of their bodies. Grigar claims that without a rhetoric of the kinetic, the artists’ message and impact are lost. Girgar points out that the artists rhetoric breaks decorum and this is accomplished through use of the body. She makes the distinction that when the female artist uses her body this break with decorum occurs kinetically. When it is the audience manipulating the work of art it (the break of decorum) occurs kinesthetically. This insight implies that any and all media is a kind of rhetoric–it can take any form–“sound, action, body action–in addition to writing and orality and the visual” (Grigar 112). She concludes that although we talk about visual rhetoric, we should also be talking about rhetoric associated with auditory, kinetic, and the like” (112).
Grigar argues that we should be thinking widely about the modes of expression used for communicating: “Along with words or even images, we may be more thorough with our research if we come to see rhetoric as “media rhetoric”-that is, communication that crosses and encompasses all media” (112). This is so simple. This sounds a lot like multimodal literacy without any reference to Kress and the gang. This also reminds me of Bloom’s taxonomy of educational objectives. There was supposed to be: the cognitive, the affective and the kinesthetic/psychomotor. They were envisioned as a triad, although it was only the cognitive that became well known (because it was the easiest to research). Bloom et al’s theory of kinesthetic was about doing/producing things…why does no one ever use Bloom’s taxonomy? I mean the affective and the psychomotor– I think it’s fascinating stuff.