Moving Pictures

Carl Plantinga explores how new approaches to cognition are changing how we understand film

At the end of Ridley Scott’s Blade Runner (1982), the Replicant Roy Batty (Rutger Hauer) decides to save his enemy Deckard (Harrison Ford), rather than allow him to fall to his death. As Deckard lies injured in the rain, he sees the formidable Batty deliver his dying soliloquy, where Batty grieves the fact that all of his memories will be lost, ‘like tears in rain’. The camera rests on Batty’s sorrowful face in close-up. ‘Time to die’, he says, and then expires. What is going on in our response to this scene?

For more than one hundred years, scholars have been attempting to make sense of the human experience of movies; Hugo Munsterberg’s The Photoplay: A Psychological Study was first published in 1916. Munsterberg, along with William James, Henry Bergson, and, more recently, Oliver Sacks have all remarked on the similarity of films to human streams of consciousness. Movies are carefully constructed to provide an evolving series of represented events, conveyed to audiences through complex streams of images and sounds. As film theorist V. F. Perkins put it, movies are like ‘mind recorders’: they mimic human consciousness.

Since the 1980s, the central approach to understanding movies psychologically has been cognitive film and media theory, practiced by both film scholars and philosophers interested in aesthetics. The cognitivists have formed a lively scholarly organization (The Society for Cognitive Studies of the Moving Image) with an associated annual conference and professional journal (Projections: The Journal for Movies and Mind).

Early cognitive theory modelled thought on computer-like information processing. This kind of cognitive approach is not without its detractors, however. Recent times have witnessed an increased attention to emotion and to the body. Film and media theorists have turned to the senses, the body, and body metaphors, moods, features of brains such as mirror neurons, and neurological activity as measured by fMRIs. This turn may be interpreted as threatening to eclipse an interest in (what is taken to be) cognition.

To understand what’s at stake here, consider the Blade Runner example above. Early cognitive approaches would have pointed to the ‘computational’ aspects of audience response, such as inference-making and the testing of hypotheses about the narrative. Yet as I have argued elsewhere, this ‘scene of empathy’ is designed to elicit strong empathetic reactions and powerful emotions. How can we make sense of the experience of this scene while taking both mental ‘computation’ and emotion into account?

At the heart of these debates, in my opinion, is a fundamental misunderstanding of just what cognition is. We experience movies with our bodies, as we sway with the action, erupt in laughter, or feel that knot of suspense in our stomach. These sorts of observations about the bodily nature of our experience sometimes lead theorists to conclude that certain theories or accounts of films are ‘too cognitive’. How can the word ‘cognition’, with its suggestion of logic-like information processing, account for moods, emotions, and bodily experiences such as flinching, swaying, and weeping? Furthermore, ‘cognition’ seems to imply conscious, self-directed mental activity, rather than the automatic and involuntary processing that dominates our bodily responses to the world and to movies in particular.

Once we understand what cognition is and how it is studied, however, these objections fail to apply. Cognition, simply put, is the mental activities of gaining knowledge and understanding through thought, experience, and the senses, plus the results of such activities—comprehension, intuition, insight, perception, and so on. Cognition is neither singular nor linear, with multiple cognitive processes occurring simultaneously and connected in complex ways. Neither is cognition necessarily either conscious or rational. Psychologists write of the ‘cognitive unconscious’ to differentiate it from the Freudian unconscious. Much of cognition occurs beneath consciousness, and is seemingly automatic rather than voluntary. Moreover, cognition is neutral with respect to practical rationality, just as emotion is; in themselves, cognition and emotion are neither rational nor irrational.

The word ‘cognition’ has two different senses that ought to be differentiated at this point. Sometimes ‘cognition’ is used to designate rational and deliberative mental processes, such as inference-making, hypothesis-testing, and more generally, thinking. As such, cognition can in principle be separated from affects such as moods and emotions; it can be conceived of as a ‘cool’ phenomenon. Increasingly, however, we are coming to understand that cognition itself is affected by emotions and moods, for example, and has a firm grounding in bodily experience. With this in mind, cognition ‘proper’ is ‘hot’. Making sense of the world around us, we employ body metaphors, we gauge physiological feedback, and we are influenced by emotion.

The viewer’s understanding of characters, narrative events and progression, and narrational point of view is fundamentally cognitive in nature. What we should say now is not that cognition is separate from affect, the unconscious, and bodily and automatic processes; rather, a full account of cognition encompasses them. Cognition itself is infused with emotion, embodied, and in part automatic. To see how this is true, consider a central means by which viewers experience movies: sympathy and/or empathy with fictional characters. One of the strongest forces in generating spectator affective responses is through the characters in a story. If one aspect of our everyday consciousness is an awareness of the people around us, the movies have the capacity to focus and strengthen this kind of attention to others by offering virtual relationships with fictional characters that we can see and hear, as in the Blade Runner example. The kind of spectator–character relation that theorists have been most interested in is sometimes called ‘identification’, and typically involves some combination of sympathy (feeling for) and empathy (feeling with) the character.

Sympathy and empathy for characters can be driven by pre-conscious, largely automatic psychological processes. The use of extended close-ups, facial expressions, and various mood-inducing stylistic choices (in the Blade Runner scene, darkness, rain, low-key lighting, and plaintive music, for example) may all have their effect. The use of extended close-ups in such ‘scenes of empathy’ may elicit facial feedback and emotional contagion, thus transferring some of Roy Batty’s supposed sadness to the spectator. The mood of the scene is somber, and the spectator may ‘catch’ some of that somber quality merely by viewing the scene. The mysterious workings of the brain and its mirror neurons may lead to various mirror effects that partially escape conscious control. The scene has a ‘feel’, with textural and tonal qualities that strongly affect the audience. Much recent phenomenological film theory attempts to make sense of the spectator’s sensuous, tactile experience of the movies.

This attention to the body and to the sensuous film experience is absolutely vital. Still, cognition is also vitally important. As soon as the spectator relates this scene to other scenes, differentiates one character from another, evaluates a character’s behaviour in relation to narrative context, or in any other way makes sense of the scene in context, cognition becomes the means by which these impressions, feelings, and responses are united into an experience of the film as a whole. Cognition is ‘hot’, but it is still cognition. Consider The Silence of the Lambs (1991), in which the villainous Hanibal Lecter matches wits with FBI ingénue Clarice Starling (Jody Foster). During key scenes of the film, Lecter and Starling are filmed in shot/reverse shot scenes, alternating between close-ups of their two faces. These extended close-ups may initiate various sorts of automatic mirror effects in spectators, such as primitive forms of empathy. Yet the spectator’s more complex responses to these very different characters are governed by cognition, including the spectators’ prior knowledge of the characters and their role in the narrative, evaluation of their motives and behaviour, and various emotional responses and reactions. Hanibal Lecter may fascinate us, but most will have stronger sympathy and empathy for Clarice Starling.

The second observation is this: Even if we were to consider the Blade Runner scene apart from its narrative context (which we typically do not do), cognition would be called into play. In part, this has to do with cognitive frames or schemas that spectators appeal to, such as character stereotypes, visual conventions and associations, and even the basic knowledge of the ‘institutions of fiction’—in this case Hollywood’s—and of genre conventions. Beyond this, however, even basic affects such as moods have cognitive implications, such that the somber mood of the scene can actually influence thought. For example, the mood of the initial scenes in which Clarice speaks with Hanibal is always carefully calibrated. The cellblock resembles a medieval dungeon, with rough rock walls. The music is suspenseful, the lighting low-key, the ambient sound full of intimations of unseen horrors that echo in the cave-like setting. Moods have strong associations, and simply having or eliciting a mood can suggest these associations. The mood of these scenes from The Silence of the Lambs suggests danger, insanity, violence, and unspeakable horror, leading the viewer to anticipate and fear these sorts of narrative events. Affects, including both moods and emotions, are not incidental to cognition, but are an inseparable aspect of how we come to understand the world around us, including movies.

It is this broader sense of cognition—‘hot’ cognition—that characterizes our experience of movies. Cognition, properly conceived, includes emotions and body responses as well as ‘computational’ activities. Cognition is at the heart of human experience and at the heart of movie viewing. We should not minimize the centrality of cognition, but we do need to consistently monitor evolving ideas about just what cognition is. Both in and outside the movie theater, cognition must account for the body and emotion.