VR and Narrative

Narrative in VR, at core is a problem of agency.  There is not only a problem of exactly who is TELLING the story in an interactive environment, there is the issue of whose conflict is DRIVING the story—the hero or the viewer?  Traditional novel form of the hero’s journey is one of the hero being in action, not the audience.  The primacy of Western, Aristotelian plot-driven structural narrative strategies will have to diminish.  VR will also require a radical restructuring of traditional 2D film making since in that case the director is the storyteller, not the viewer.  These are issues many have pointed out from academia to the latest conference on interactive storytelling, but warrant discussion because although much of what is out there covers the challenges, few offer any suggestions.  The game industry is where most of the story development has happened since the 1990’s.

There is much that can be learned from the extensive research done in game story development.  Games are the big money makers now.  VR gear has grown out of desire from gamers for more realism, not moviegoers, and so it makes sense that their narrative construction insights might be useful.  But game narrative is like porn or Greek myth—the point of everything is to get to the next action moment to do battle (or whatever).  Most game “story” I’ve seen (which admittedly isn’t all that much) is very basic, despite multi-level branching plots and characters.  Part of this is determined by the economics of production and the impossibility of accounting for the infinity of possible narrative lines.  Another is that players are generally concerned about the action in the game, not the plot used to get there.  Although we’ve come a long ways from Space Invaders, that is not a storytelling bar anyone wants to be referencing.

VR and Joseph Campbell

The first casualty of narrative in virtual reality will be the hero and his (usually HIS, still even in 2017) journey.  The point of the hero in Western narrative is to be a flawed character who makes choices we would find too bold to make, and then to suffer for those choices until, when life is at its worst, make the enlightened choice and transcend themselves.  I have a lot of issues with that story structure, but even still when it comes to VR and narrative, it is not a simple matter of putting the viewer in the scenes of such a journey and expecting the audience to respond the same as if viewing it from a distance in 2D. 

The basic formal structure of current storytelling codified by Aristotle demands a beginning, middle, and end, following a singular hero burdened with a singular issue existing for the sole purpose of transcending that issue and thus generating catharsis in the audience.  Narrative experience, even if passive, is/wiil be very different in an immersive environment.

One challenge is the experience of place and time.  Traditional storytelling accomplishes this via a complex construction of interactions between structural and visual rhetorical tropes.  At its best, narrative is the compression and expansion of place and time, a distillation and condensation of reality in order to express an idea.  VR is experienced in real time.  Can we find ways to compress and expand time that will be understandable in VR?  Sure.  They just might not look/feel like what we’ve done in 2D to date.

Not all will want to shoot bad guys or go flying through the air.  Many (in fact, I would say MOST) will still want stories TOLD TO them.  Stories that they don’t have to do anything but sit back and experience.

I do expect viewers will want the option of choosing being automatically placed in pre-determined places for the best view of the scene.  I would also expect more interest in ensemble stories, where multiple characters have their own arc that could be explored individually.  This has many natural connections to current game story structures that might be useful.  But this sort of narrative still requires “following” a character.  As a consequence, very quickly some acceptable visual standards and conventions will be established that have nothing to do with storytelling in 2D.  These cannot be pre-determined because those practices have to do with the way many, many VIEWERS, not the creators, over time respond to VR. 

Not all conventions will be replaced, some will simply be repurposed in a different way with a new emphasis, e.g. portraiture has been around since the beginning of photography, but very few photogs turned the camera on themselves.  Ten years ago no one knew what a “selfie” was.  Now it is the most used genre on the internet (okay, maybe cat videos still lead, but not by much, I’m sure).

The hero’s journey will always have an audience, but new experiences of narrative will open up in VR.

VR and Cinema

It only takes 24 frames a sec to fool your brain into thinking it is seeing continuous motion.  Today, our brain is also oblivious to the raster lines of display diodes that are tinier than we can see and scanning faster than we can notice.  Both are tricks on your brain.  But in 2D we are always in the third person because we are not in the action of the scene.

There is a language of 2D cinema that only lets us see what the director wants us to see, when they want us to see it.  Something as simple as a Guy Ritchie cross-cut between two locations would be almost incomprehensible in VR.  The immediate shift from one location to another is a visual rhetorical device that everyone who was born after the invention of television has understood since childhood and works great in 2D.  However, the experience of abruptly shifting from one reality to another, then back and forth, will be annoying at minimum, at least as we can imagine it being experienced at this point.

In 2D we do not see what the camera does not see.  This has two implications: pragmatic and aesthetic.  On the pragmatic side, the simple illusion of “natural light” requires multiple set lights, generators, crews, etc. all outside the camera frame.  If the consumer can see 360 there is no place to put those support items.  You could shoot 360 with available light and digitally remove unwanted camerapersons, equipment, etc.  Or shoot like Mr. Cameron, in 360 greenscreen with actors moving around a built set (remember the “jungle” scenes in 40’s black and white movies—real trees and real water in a pond, but with a fake backdrop?  Same, same—but WAY more expensive).

Certain genres, let’s say horror and thrillers, depend greatly on what the camera is NOT showing you—the monster, the thing hovering just behind the character but they don’t turn around, the thing/monster/killer just out of frame as the victim backs up.    Other genres, like action, depend on frenetic editing and the perfect camera shot to keep you interested.  If all you have is the experience of riding in the car as Vin drives his fast and furious supercar like a maniac, that holds a certain amount of excitement, but will quickly become annoying and/or boring as the novelty wears off. 

But more importantly, what those stories are about are the characters experiencing those moments FOR the audience, not the audience themselves having the experience and related emotions.  Successful narrative at the emotional level doesn’t work by showing the audience the character crying—it is about making the audience cry, that is, catharsis.  It is generated by vicarious emotional manipulation, not personal lived experience.  Sometimes, once in a while life gives us a moment so intense we get that same emotional discharge, but audiences today expect that cathartic feeling every time they watch a feature film.

We are at a point where we are creating a monster by placing this old way of storytelling onto a new tech and wonder why it isn’t quite working.  This is quite normal.  Eventually the tech itself will reveal its own best practices different from 2D.

VR and Immersive Experience

Page 1, MA thesis, 1995

VR and Immersive Interactive Experience

When I say VR in this blog, I mean immersive interactive experience.  I don’t really know that virtual reality means anything, really.  This is what I wrote in my 1995 MA thesis:

Virtual reality is an existence in the domain of cyberspace . . . defined variously from Michael Heim’s critique of VR and cyberspace as an ontological Zen dream, and Arthur Kroker/Michael A. Weinstein’s pseudo-Marxist rant of VR as the ultimate paranoiac’s nightmare, to the hip, pop writing of R. U. Sirius’s version of VR as techno-voodoo funk.  One of the most inclusive definition is Marcos Novak’s . . . This definition is useful in that it defines the limits of cyberspace as a domain, an environment fabricated via computer software and hardware . . . This is merely another in a series of computer interfaces from punch cards and paper tape, to . . . data gloves, headgear and CAVE’s—yet one with exciting possibility.  (pg 32)

So who knows, really?  However, immersive tech is branded today as virtual reality and that’s the way it has been since the glove/visor combo appeared in the 1980’s.  Perhaps once we get away from that particular hardware configuration we can stop calling it VR.  It is not a virtual reality.  It is an extension of 2D computer graphical user interface display into 360 degrees of view with depth and interactivity.  It doesn’t have to remotely resemble reality.  What we call a VR, an immersive interactive experience, is just another part of reality.  When confronted with the technology of photography, then later silent film, people mistook them to be reality for a while.  We will move on.  Until then, I’ll use the vernacular term VR so people will understand what I’m talking about.

VR and Place

Still from “How the West was Won” 1998 Interactive QuickTime VR format

This is an example of using VR to experience the idea of the American western landscape.  The interface offered representations of the west, both appropriated and my own work as a “menu.”  By touching 2D images in the VR space you could move to an experience more of what you might call “natural” videos of Yosemite.  All of this was still in a fairly crude 2D display.  It was a decade before Oculus visor was in development.  You can imagine the looks I received from gallery owners when I tried to explain the medium I was working in.  Ah well . . .

VR and place

What I was commenting on in that piece was the importance of a sense of place within VR, that is, the relation between sensory experience and subjectivity.  Instead of placing the viewer in Yosemite Valley, I placed them in representations of that space.  They still could experience the videos I shot around the valley, but how they got to those experiences was via an interface without instructions or menus.  Which is not what most people were doing with QTVR.

The easiest way make a VR experience understandable is to make the VR space look like the reality already experienced (including conventions such as animation).  Back in the 90’s most QTVRs were basically what you see with people’s 360 camera posts today—360 views of the world around us.  These are shared so others can “experience” the place and are thus left with a sense of having “been there.” That makes perfect sense and it makes the extension of that, immersive VR, an easier sell.  There are animated product and combos of real image and animation (which includes augmented reality), as well as simply recorded places from prison cells to rain forests. 

Unfortunately, at the moment any ability to invite others in VR into that space is via simplistic avatars, at best.  But that will change very quickly.  Facebook has already announced this capability and as people use it, they will want more options, finer grain detail.  This ability to create a space and invite others will bring on a major shift in web and social media interactions.  Additionally, news, pro sports, concerts, and other live events will become very energizing in VR. On the opposite end, dreamy, relaxing meditative spaces will also be wanted.

In all of those spaces, however, the user is extremely limited by the physical space in the real world and the complexity of interface device in use—and thus, very passive.   Many are going to want active experiences.  Adventures.  Stories told.  But the old narrative models dependent on unique rhetorical structures will be ineffective in virtual reality, primarily due to our current relation to place and time in VR.

VR is (almost) Here

Forgive me blog-gods for it has been a year since my last post (almost an eon in blog years).  I thought the VR Expo event in Los Angeles warranted comment.

Ghost of VR Past

I was introduced to virtual reality due to a chance meeting 30 years ago with Scott Fisher and the headset/data glove he developed for NASA AMES.  He was creating a control system for robots working in space to do certain tasks and thus avoid dangerous and expensive human spacewalks.

All this was pretty cool in the 80’s, but also very crude tech compared to today.  Fisher was using some HP 3000 computer the size of a mini-bar refrigerator with a whole 8MB of RAM.  All I saw were simple wireframe renderings.  But it was VR.

Everyone was very excited.  There were artists claiming to be working in VR, although I was never really convinced.

Academics were studying it.  My master’s thesis considered ontology and VR.

And then . . . not much happened.

Ghost of VR Future

Although it was clear VR would replace 2D mediums such as photography and film, as both spaces to create in and experience, it was not clear when.  30 years later we are still waiting, but getting much closer.

At the point when the cost of a simple VR headset is low enough that it can be given away with your smartphone purchase, there will be a redesign of the OS for those phones to use a VR GUI.  There will then be a crushing demand for 360 experiences at minimum for everything from advertising to scripted production.

2D entertainment will begin an accelerated decline into oddity, then relic, “that old, retro, 20^th century tech”.  This will happen soon.  Nothing drives tech like consumer demand.  Television will stay around—radio is still alive and kicking—but all visual production will have to be geared for VR, even if only in 2D.

For example, imagine watching your favorite 2D television re-run in a VR room and television appropriate to the time period.  Or as a producer, you could purchase a pre-fab VR drive-in environment for your no-budget 2D horror feature distribution package.

A continuum of product will develop from consumer-produced 2D only at the low end to complex, interactive, live and scripted entertainment at the expensive high-end.

Ghost of VR Present

When it comes to revenue the product will generate, that continuum will not be a smooth curve.  There will be a massive amount of production out there—most of it bad and done for free or minimal pay.

And face it, audiences like it.  Ask this question in the present: How much time do people already spend consuming expensive scripted entertainment versus on Snapchat watching free video of their friend vacationing at Venice Beach?  There will very quickly develop two tiers of production—the very high-end, involving only a few players making 90-95% of the industry income, and everyone else, struggling at bottom dollar rates, regardless of visual quality.

This began happening in the music industry in the 1980’s, then photography in the 1990’s, and is now happening to 2D production.

In 1975, 29 different music acts held the number one hit slot.  Digital home recording was “studio quality” by 1980 and got cheaper—and better.  By 1995, only 10 acts had number one hits. In 2015 it was 8 acts—and yet there are far more bands with high-quality recordings out there than ever before.

Photography as a profession is now almost impossible.  Editorial day rates are half what they were when I quit commercial shooting in the late 1980’s—in real dollars.  At the same time, the cost of professional camera gear/tech is far beyond what it used to be, forcing everyone into keeping up with the new tech.  I used my $800 Nikon F3 rig professionally for over a decade and no client thought anything of it.  If you were to show up today at a commercial shoot with camera and computer from a decade ago, you would be fired.

Most filmmakers who actually get a project completed and out there are still “one and out” players due to either being unable to raise the funds for a second project, or the second project is a failure and loses money.  Storytelling is a time consuming process, no matter how good you are at it.  And you need to get paid for your time or you will be forced to do something else.

Twenty-somethings already know it is challenging to earn a living producing content, and not likely to get easier.

VR Reality

One thing is for certain if you are doing production: you need to include VR in your planning, at least.  For a time, the tech to do VR at the high-end will be expensive, especially design and integration of interactivity.  But all of that will drop in price and/or become part of production software.

My phone already shoots 4k video.  Phones are taking advantage of the fact that there is a camera on both sides, creating 360 views.  Most clients for still advertising shoots also require video documentation; soon a VR component will be added.

The bottom line: if you are doing production, get on board with VR or risk becoming irrelevant.