Actually Submersive (Not Totally Immersive)

Vince Dziekan, Monash University, Australia

Abstract

Keywords: Virtual Reality, Exhibition, Curatorial Design, Immersion, Submersive experience, Experience Design

To be a migrant means to be an explorer; it means movement, this is our shared condition.

—Tania Bruguera, International Migrant manifesto (Queens Museum, 2011)

Preface

Existing at the intersection of immersive storytelling, documentary filmmaking, and Visual FX cinematography, Alejandro Iñárritu’s celebrated Carne y Arena exemplifies the potential of immersive digital technologies, such as virtual reality, to tell stories in powerful, new and affective ways. While contemporary creative practitioners and practice-based researchers continue to extend the boundaries of this art form, the challenges associated with exhibiting VR artworks as part of immersive location-based experiences in museums and galleries remains relatively uncharted territory. Inspired and informed in equal measure by my first-hand viewing encounter with this celebrated work and observations gleaned from the opening plenary given at last year’s MuseWeb conference in Vancouver by Vicki Dobbs Beck (executive-in-charge of Lucasfilm’s immersive entertainment division, ILMxLAB), this paper explores what this exemplary artwork tells us about the respective ways that VR and museums function as extra-dimensional, meta-narrative spaces in which cultural experiences are embedded today.

While Beck’s presentation featuring Carne y Arena at MW18 focused primarily upon the immersive qualities of the VR film, she made an important observation that what is most revolutionary about developments associated with the advent of VR in conjunction with other forms of immersive entertainment isn’t their technological advancement of high-fidelity, real-time computer-generated graphics, but rather the novel forms of cross-platform, mixed reality storytelling they enable. In order to pursue this line of inquiry, it is crucial to recognize a deceptively obvious fact: that the viewing experience of virtual images actually takes place in real space. My critical reflections will develop upon this point in an effort to identify the way in which the mediated experience of Carne y Arena was found to be integrated into the exhibition, encompassing curatorial and meta-compositional staging as I personally experienced the work at Fondazione Prada in Milan. It is important to reiterate that any interpretation of Carne y Arena’s immersive experience should not be limited to the visual and affective qualities of the simulated, virtual image, but instead be responded to more critically and speculatively by contextualizing its viewing as part of the broader, expansive, and unfolding cinematographic—as distinct from cinematic— experience that results from the work’s museum installation. As such, the museum determines the work’s “exhibition space” by providing a distinctive museological “framing” for the film itself.

This paper seeks to develop upon Beck’s closing remark—that “in time, we will design in all shades of reality, allowing us to truly move from storytelling to story-living” (Dobbs Beck 2018). In an effort to take up from where this assertion leaves off, I put forward the proposition that it is, actually, the successful calibration of the viewing experience associated with VR cinema and curatorial design— responsible for activating the theatrical, performative, scenographic, and choreographic aspects of the overall exhibition encounter—that underpins the full “realization” of Iñárritu’s creative vision, resulting in what might more effectively be described as a submersive, not (totally) immersive experience.

Actually Submersive, (Not) Totally Immersive

The films of Mexican director Alejandro Iñárritu are celebrated for portraying themes of human connection, often through complex, evolving and intersecting story lines, told in a non-linear structure. Carne y Arena (2017) premiered as the first VR official selection at the 2017 Cannes Film Festival and was subsequently bestowed a special Oscar from the Academy of Motion Picture Arts and Sciences in recognition for achieving “a visionary and powerful experience in storytelling.” Featuring as one of the illustrative cases of immersive entertainment that Beck focused upon in her presentation at MW18, Carne y Arena certainly exemplifies the visual and narrative potentials of virtual reality. VR can rightly be held up as a new cinematic paradigm that, by exceeding the pre-defined boundaries of the moving image, breaks the dictatorship of “the frame” within which things are justobserved by allowing the viewer to claim the space and integrate their viewing engagement into the filmic experience. Produced in association with Lucasfilm’s ILMxLAB, Legendary Entertainment and Fondazione Prada, Carne y Arena extends the application of digital technologies associated with computer-generated imaging (CGI) beyond the virtual screen and into the real space of the installation. As I will develop from here, it is by successfully calibrating the VR viewing experience—described by Iñárritu as blurring the lines between “a dream or reality composed within a frame” (Celant and Iñárritu 2017)—through curatorial design, as it is applied to the overall exhibition concept, that supports the director’s intent to enable the viewer to identify empathetically with the immigrants’ experience.

But before doing so, it is worth taking a moment to conceptualize the legacy of the virtual image before offering a brief synopsis of the filmic narrative in order to identify key cinematic aspects that characterize its distinctive viewing experience.

Virtual images act as idealized visual representations of reality. Art historian David Summers asserts that the basis for achieving virtuality is rooted to the notion of extra-dimensionality—that being the capacity to see, for instance, three dimensions in two, or four dimensions in three. Tracing the development of the “image arts” across Western art history, Summers identifies that successful representation of reality is wedded to the frame and its functioning as a window, allowing you (the viewer) to see into another, virtual world, thus enabling “narratives, spaces and times separate from the present, which may be past or future” to be brought together (Summers, 2003). In the case of VR, this “scopic regime” (Jay, 1988) is reinforced by the reliance that its associated viewing experience has to head-mounted display technologies, such as the Oculus Rift, which support a highly individuated and constrained (or tethered) form of cinematic spectatorship, in part due to computational demands of processing high-resolution imagery at convincingly fast frame-rates. Technological and design limitations have exerted a delimiting effect on the creative application of virtual and augmented reality in the visual arts to date; however, the latest developments in headset technology (such as the Oculus Go and Magic Leap) are pointing towards a future in which (according to Oculus marketing), the user will be liberated to “go virtually everywhere.”

In the case of Carne y Arena, the narrative trajectory of the film itself leads the viewer through an initial stage of orientation and acclimatization to the perceptual experience of VR—adjusting to the hazy pre-dawn darkness; gaining one’s bearings within a (seemingly) vast and (almost) boundless space, before finding oneself assimilating with a caravan of Latin American refugees, trudging wearily forward through the Chihuahuan Desert. The unfolding narrative (sketched out below ) sets in motion a transformational process that “moves” the viewer, literally and figuratively, from a physical to an emotional level of engagement, from spatial orientation to intra-personal identification. Having effectively registered the participant’s relationship to the scene, optically and psychologically, the next passage of the VR film signals a palpable shift in viewing attitude: from the onus being placed on establishing, or locating, one’s physical position in the virtual world to eliciting a more critical awareness or consciousness on the part of the viewer to their moral position in relation to the precarious situation they find themselves in. Scene:

Desert, in dawn’s early light.

The gritty low-light level resolution of image-world is deftly reinforced by the sensation of sand under your feet.

At first, the contours of a single, shadowy figure can be discerned, emerging from the depths of the desert. Then, as more bodies continue to rise out of the darkness, you (the viewer) begin to identify the distinct physiognomies of men and women, younger and older as they surge forward through the undulating scrub, draw near and continue past.

Discarded bits of clothing and strewn detritus litter the ground underfoot. You stop, and drop down to more closely inspect one such item: a small backpack that may have belonged to a child.

In the wake of the ambient sound of wind, blowing sand and rustling movement, discernable voices of men, women and children materialize; the hushed whispers of members of the nomadic group interrupted by the sharp, shouted instructions of coyotes– a colloquial Mexican-Spanish term given to recruiters and transporters involved in people smuggling.

This cacophony of natural and man-made sounds are soon drowned out by a rapidly approaching helicopter, and as its search light pierces the darkness, the sound of its rotor blades becomes deafening and you feel yourself pummelled by waves of air and the ground shakes.

In the midst of this sensory onslaught, a patrol vehicle crests a nearby hill and blocks the pass ahead. A group of policeman carrying machine guns emerge from out of the smoke and dust hanging in the air to scan the group with their flashlights.

Then, following a cut-scene of a “magic realist” vignette (in which a table-top diorama of a ship laden with migrants slowly capsizes), the plot turns to a dramatic confrontation between the group and US Border Protection authorities. This episode acts as the film’s climax.

Following the group’s capture, a process of interrogation unfolds in which certain members of the group presumed to be the “coyotes” are singled out by the law enforcement officers for questioning, while others (predominantly women and children) are corralled together and ordered to get down, stay quiet, and remain still.

At this critical juncture, the viewer becomes directly implicated in this scenario and faced with an immediate dilemma: to submit along with their compatriots to the directions being forcibly issued by these border police, on the one hand, or willfully break the bounds of engagement established by the diegic story space. The gravitasof this decision, reinforced by the spotlit intensity of this tense situation, seems to hang in the balance. The viewer’s awareness to their own moral decision-making is amplified expertly by Iñárritu by treating this pivotal episode as a scenescape: a cinematic trope drawn from game design that allows the viewer alternate vantages upon scenes and their unfolding dramaturgy. The scenescape establishes a form of immersion, at once emotional and spatial. As games researchers Vosmeer & Schouten develop: “On one hand, the story that the scene is set in may invite the user to be immersed through narrative. On the other hand, the physical device that the scene is viewed by may offer additional spatial immersion” (Vosmeer & Schouten, 2014). In effect, the viewer is permitted not only to engage with the scene by choosing where to direct and concentrate their attention in the visual environment, but they can elect to remain within the boundaries of the illusory experience or choose to step outside of the “magic circle” of the plot, even momentarily, to engage with the event portrayed in a more circumspect or even intentionally subversive way. As such, the scenescape adds an important element to the repertoire of transmedia storytelling, in which different mediums convey distinctive parts of a story or provide supplementary information that when combined together form a composite, multi-faceted story world (Jenkins 2006).

In the process of having his/her point-of-view unframed in such a way by the VR experience, the participant is moved from individual consciousness to a social conscience. However, it is also important to reiterate that this immersive viewing experience is nested within a broader, expansive and unfolding exhibition experience. Not dissimilar to how the scenescape functions within the VR film itself, the curatorial design of the exhibition at large—including its array of media content (described in the following section), installation environments and gallery spaces situated before and after, including the “arena” in which the immersive VR experience itself exists within—provides an extra-dimensional layer to the film, creating in turn what might more effectively be described as a submersive (as distinct from immersive) experience.

Found in the Museum, (Not) Lost in the Fun House

It was easy to be awe-struck and inspired in equal measure by Carne y Arena and the other innovative applications of immersive technology that Beck showcased in her presentation at MW18—including bespoke entertainment experiences developed by ILMxLAB from the Star Wars canon, such as Trials of Tatooine (2016) and Star Wars: Secrets of the Empire (2017), and an augmented performance for London Fashion Week (2018) developed in collaboration with designer Steven Tai. Nonetheless, I recall being left with a lingering sense of uncertainty about how seductive examples like these might integrate themselves into the ready-made platform that museums offer. As mixed-reality storytelling develops through its own version of the Gartner-Hype cycle, how might museums draw upon their innate qualities and features to create and compose distinctive and meaningful encounters wherein virtual elements are introduced proportionally into its real-world setting? Recognizing the museum as a constructed space for story-living (“a living world where stories unfold” according to Beck), how might the different “shades of reality” from which the museum is constituted be curated and optimized for world-building and place-making?

In the following section, I will briefly discuss the curatorial design involved in the staging of Carne y Arena at Fondazione Prada by describing the choreography and scenography of its exhibition-based encounter, emphasizing the importance of how the VR work is effectively contextualized and “framed” by installation and display strategies employed in its adjacent gallery spaces.

At is dramatic core, Carne y Arena revolves upon the confronting experience faced by a nomadic group of men, women, and children seeking to cross the border clandestinely from Mexico into the United States. Iñárritu describes Carne y Arena as a “semi-fictionalized ethnography” (Iñárritu, 2017) that draws upon five years of research involving interviews with immigrants from Mexico, Honduras, El Salvador, and Guatemala, documenting their personal border crossing experiences. This “ground truth” was reworked into the narrative basis for the screen adaptation of the VR film while also factoring directly into the film’s production design and ultimately translating into the curatorial design of the exhibition at large. As part of the film’s production process, live shoots were re-enacted with the same people who’d actually experienced them; and in an effort to achieve even greater authenticity, additional steps were taken to have these (non) actors wear the same pieces of clothing as worn during their border crossings. In studio, 3-D scans were executed and the resulting digitized physiognomies applied to the cast of characters whose avatars populate the VR film. Here, “world-building” takes place at all perceptual scales (from fidelity of imagery associated with all elements that make up the visual environment, including volumetric lighting, shadows and dust effects; the soundscape, including background noise, chatter or atmospherics; to simulated haptics and the relatively simple means of laying the exhibition space with sand. Of course, it is important to be reminded that the “real-time” VR viewing experience of only six and one half minutes in length is nested within an expansive and progressively unfolding exhibition experience combining artefacts, installations, and video “portraits.” To date, this groundbreaking installation has been exhibited at both leading international venues, such as Fondazione Prada in Milan and the Los Angeles County Museum of Art (LACMA), as well as temporary installations in Mexico City and Washington, DC.  The curatorial design of Carne y Arena as gallery-based exhibition, I argue, should be factored into any discussion of the “immersive” impact achieved within its celebrated VR experience and its meta-composition, recognized as a exercise in immersive design in its own right.

The exhibition scenography of Carne y Arena is composed in three main “acts” which are designed choreographically to be experienced alone as a sole viewer/participant. Heralding this, however, is an “on boarding” stage that while apparently innocuous, plays an important role in establishing the terms of engagement for the experience that follows. The protocols that surround the exhibition (i.e. administrative processes of booking a personal viewing time, transactions with docents upon admission to the space, the prohibition on photography) reinforce the institutionalized terms of the encounter.

First Act: “The Freezer”

Upon entering the first gallery, the solitary viewer finds themselves confined to a caustically lit grey cell. As related by the first-hand account of Manuel (a Guatemalan immigrant whose personal story is represented in Carne y Arena), detention cells were colloquially referred to as “freezers”:

For 3 days and 3 nights, we crossed the desert. The cold almost killed me. My feet were filled with sores. A thorn was buried in my foot. I could not walk. The coyote threatened me with his gun. I walked, or he killed me, he said. I tied my shirt to my feet and walked. One night, a helicopter and 3 patrol cars caught us. They took my backpack with my food and tied our hands behind our backs. They took us to the holding cells known as “the freezers” because of how cold they kept them. I couldn’t communicate with them because I didn’t speak Spanish, only Ki’che, a Guatemalan dialect, and there was no translator.

A single metallic bench is located in the centre of the small room; an assortment of weathered footwear (running shoes, sandals and mud-caked boots) line the base of the walls. These items are representative of thousands of artefacts collected from the desert borderlands between Mexico and the U.S. Border by Casa Libre (a Los Angeles-based not-for-profit organization that assisted Iñárritu with his research and development). Printed instructions direct the participant to remove their own pair of shoes and set them aside in a wall locker. Left to sit in isolation for some minutes, the viewer becomes gradually sensitized to what it feels like to find oneself incarcerated and powerless. Subliminally, this experience introduces a physical, tactile register to the anticipated “virtual” experience, as the cold of the concrete floor begins to work its way up from the soles of your feet up. Finally, the air of uncertainty and suspended expectation is broken by a shrill alarm, and a signal light flashes insistently; a side door opens.

Transition

In contrast to the claustrophobic confines of the waiting cell, the passageway leads to a surprisingly large, expansive space bathed in an otherworldly orange glow. In the middle of a field of sand, a pair of assistants await you. These docents explain the extent of interaction available with the VR experience before fitting you with a backpack (containing a Nomadic Modular System, tethered by a long cable that constrains the limits of movement in the environment) and an Oculus Rift headset. Once kitted out with this apparatus, the viewer is dropped into the middle of the cinematic narrative, described previously.

Second Act: Hyper-reality Experience

The cinematic VR experience is mapped into a whole-body, fully immersive entertainment platform. Supported by ILMxLAB’s research and development into location-based immersive adventures, the spectacular quality of the real-time, Digital Effects (FX) filmmaking combines with auditory, visual and haptic “special effects” simulated with advanced technologies developed by Disney Imagineering (notably responsible for the creation, design, and construction of its theme parks and attractions). Together, this results in a multi-sensory experience that creates a strong sense of convergence between otherwise separated virtual and physical realms.

Transition

At the conclusion of this embodied and performative encounter, the viewer exits the arena and follows a long passageway lined by rusted metal sheets. The overwhelming materiality of the barrier impresses itself upon your tactile sensibility. Correspondence is established between the participant’s own movement along the corridor with that of the border crossing aspirations of the immigrants represented in the film, giving this otherwise mundane “threshold” experience added poignancy.

Third (Final) Act: Portrait Gallery

The exhibition concludes in a somber gallery lined with a deeply affective set of video portraits in which the personal stories of a group of Latin American immigrants and refugees are documented. Interspersed throughout the installation are a series of video monitors recessed into the wall surface. Mounted at head height, each frame is neatly individuated from the collective display. This tightly focused installation strategy reinforces the confronting of personal stories, recounted through this series of video testimonials. With stark, unadorned and direct immediacy, the life stories of these people reveal the fateful circumstances and motivations which drove them to seek refuge from oppressive conditions, gang violence, as well as tales of suffering endured as part of their migratory “crossings.”

For example, Amaru (20, El Salvador) recounts their lived experience of leaving home in El Salvador at age of 15 to escape oppressive cycle of gang recruitment:

Crammed into a room at Reynosa waiting to cross. I went to the rooftop and saw the lights of a U.S. city. I will never forget that. We crossed the river on a raft, and I decided to go alone from there. We were suddenly surrounded by a helicopter, quadbikes, horses, and dogs. They put some of us in dog cages and let the coyote go free.

In the case of Lina (a 53 year-old Guatemalan woman) forced to leave her children behind in order to escape a civil war:

We walked for 3 days in the desert. We slept inside plastic trash bags, because it was brutally cold. We hid from helicopters, and because we were in a hurry, the coyotes abandoned a young boy who could not walk anymore. We jumped the wire fence at the border, and I cut my hand. We walked with sponges tied to the soles of our shoes to not leave a trace. In the distance, we saw a highway and ran. I lost a shoe in the sand. We climbed into a van, and the coyotes arranged us in it like logs. I was lucky to be the last to arrive because they put me on top of all of the other bodies.

In the midst of watching these arresting accounts of the personal toll resulting from social upheaval and political dispossession, the viewer comes to the realization that the “real” people telling them are, in fact, the very same “characters” encountered earlier as avatars in the VR film. The resulting process of identification is both empathetic and cathartic.

Section IV: Conclusion

My curatorial research continues to be interested in understanding the nature of virtuality in the context of cultural production and ways in which exhibitionary spaces are responding to the distinctive post-digital conditions of aesthetic experience. The quest for participatory and immersive experience will continue to extend museum design, placing ever-more demand on its digital infrastructure and programme architecture. As the case of Carne y Arena illustrates, greater collaboration between the cultural and creative industries challenges the ways and means by which representation, narrative, viewer engagement, and user interaction are understood and practiced by museums.

With spectacular forms of immersive entertainment becoming normalized as part of theme parks and cinema experiences, museums must “scale up” in order to compete for their share of audience attention. According to respected museum director Seb Chan (Chief Experience Officer of the Australian Centre for the Moving Image), the challenges faced within today’s “experience economy” place even greater onus on providing the gallery-going public with “performative and performed museum experience[s]” (Chan, in Levent, et al., 2014) that are engaging as well as critically engaged. “Blockbuster” exhibitions and an increased emphasis on design at all levels illustrate the museum sector’s efforts to respond to such a challenge, while museums such as Tasmania’s Museum of Old and New Art (MONA) and Tate Modern in London provide indications of how a kind of performativity associated with theatre can be successfully merged with the museum experience. Drawing upon the trust vested in them as public institutions, museums are instrumental to not only fostering the media and communication literacies associated with new technologies—ranging beyond virtual and augmented reality to include locative and context-aware technologies, 3-D fabrication and artificial intelligence—but also championing their civic value and purposing. Reinforcing this observation from the vantage point of an artist, renowned electronic artist Rafael Lozano-Hemmer proposes that by raising the inherent performative qualities of the visual arts (through situated “location-based experiences” in particular), museums are especially well equipped to offer full and rich participatory experiences for their audiences since they are already connective spaces where disparate places and experiences actually get in touch” (Lozano-Hemmer, in Levent et al., 2014).

In their own respective ways, VR and museums function as multi-dimensional spaces in which narrative experiences are embedded. It is here, on this point of connection where episodic scenographic elements, curatorially designed into the exhibition’s meta-compositional experience meet their audience, and the worlds inside and outside the museum collide (as it did most tellingly during Carne y Arena’s recent staging by the Emerson Collective in Washington, DC when revelations emerging about the forced separation of migrant children from their families as part of the Trump administration’s border protection policies, imbued the work with an even greater sense of poignancy and urgency)—that the empathic perspective extolled by Carne y Arena assumes a social and political purpose. By constructing a reflexive narrative that foreground personal memory and first-hand accounts of dispossession and humanitarian crises, Iñárritu’s adoption of documentary strategies as artistic method reflects what cultural critic TJ Demos describes as the “new politics of truth” (Demos, 2013). Works like Carne y Arena exemplify the process of museum mediatization (Drotner et al., 2019) and nurture critical literacy about the way that “meaning” is produced and consumed and claims of “truth” contested. By taking account of this situation (and holding communication to account), a meta-dialogue is instigated that not only responds to difficult subjects (in this particular case, the global migratory crises being felt acutely in many parts of the world today), but also to the media spectacle created by the “culture industry” that these subjects and events find themselves circulating within and co-existing with. This places renewed importance on the role of the museum today: to deploy the fullest array of spaces that it is constituted from—institutional, communal, architectural, networked—as staging grounds, sites where we find ourselves simultaneously placed (situated) and displaced (existing somewhere between borders), caught up in the flow between real and virtual, psychological and perceptual, aesthetic and political states—a museum of mobilities at once authored and designed to take its audiences some place, as open and free for them to go virtually anywhere.

References

Beck, Vicki Dobbs. (2018). “Step Inside Our Stories: The Power of Immersive Storytelling.” Opening Plenary, Museums and the Web 2018 conference, Vancouver, 19-21 April 2018. Available at: https://mw18.mwconf.org/session/keynote-step-inside-our-stories-the-power-of-immersive-storytelling/

Celant, G. and Iñárritu, A.G. (2017). “Alejandro González Iñárritu in Conversation with Germano Celant.” Alejandro G. Iñárritu, Carne y Arena, Quaderno Fondazione Prada #12, Milan.

Demos, TJ. (2013). The Migrant Image: The Art and Politics of Documentary during Global Crisis. Duke University Press, Durham, NC.

Drotner, K., Dziekan, V., Parry, R. and Schroder, K. (2019). “Media, mediatization and museums,” Drotner, K.,  V. Dziekan, R. Parry and K. Schroder (eds.), The Routledge Handbook of Museums, Media and Communication, Routledge, London.

Iñárritu, A.G. (2017). “Carne y Arena (Virtually Present, Physically Invisible).” Alejandro G. Iñárritu, Carne y Arena, Quaderno Fondazione Prada #12, Milan.

Jay, M. (1988). “The Scopic Regimes of Modernity.” Foster, H. (eds), Vision and Visuality, Bay Press, Seattle.

Jenkins, H. (2006). Convergence Culture: Where Old and new Media Collide, New York University Press, New York.

Levent, N., Knight, H., Chan, S. and Lozano Hemmer, R. (2014). “Technology, Senses, and the Future of Museums.” Levent, N. and A. Pascual-Leone (eds.), Multisensory Museum: Cross-Disciplinary Perspectives on Touch, Sound, Smell, Memory and Space, Rowman & Littlefield, Lanham, MA.

Queens Museum (2011). “The International Migrant Manifesto.” Available at: https://queensmuseum.org/immigrant-movement-international

Summers, D. (2003). Real Spaces: World Art History and the Rise of Western Modernism, Phaidon, London.

Vosmeer, M. and Schouten, B. (2014). “Interactive Cinema: Engagement and Interaction.” Mitchell, A., C. Fernandez-Vara and D. Thue (eds.), Proceedings of the 7th International Conference on Interactive Digital Storytelling, ICIDS 2014, Singapore, 3-6 November 2014, Springer International Publishing Switzerland, Basel.