How Disney is using artificial intelligence to figure out exactly how much you enjoy its films

Movie studios have a long tradition of testing out new films to see how audiences react before launching them in wide release. But with their latest research innovation, Disney is taking it to a whole new level.

Now as you're settling in to watch the latest Disney blockbuster, the movie could also be watching you.

And while this could signal an exciting new era of responsive storytelling in which movies are shaped around our likes and dislikes in real time, it also raises some red flags about yet another frontier in personal data collection.

At a conference in July, Disney Research presented a new process called factorized variational autoencoders (FVAEs). Put in plain English, it measures complex audience reactions by assessing facial expressions. 

This deep-learning system has been trained to watch an audience of hundreds of faces in a darkened theatre, and to track their reactions: Are they smiling or crying? Bored or asleep, even? 

Disney Research is able to generate far more data than human intelligence is able to process. In their tests, they generated 16 million data points derived from 3,179 viewers.

And that's where artificial intelligence (AI) plays a role. As Disney Research scientist Peter Carr explained to Phys.org, computers can much more easily synthesize that massive yield of data, allowing Disney to measure the success of a film with a granularity that goes far beyond a subjective "did you like the movie?"

In fact, not only can FVAE measure reactions, Disney says the process can reliably predict them, too.

After observing an audience member's reactions for just a few minutes, the system is able to predict his or her facial expressions for the rest of the film using a pattern-recognition technique that functions similarly to a recommendation engine; it can generalize the reactions of an entire audience, and measure those reactions against an input that states how viewers "should" be reacting.

So you know all of those times you started watching a movie, thinking you'd hate it, but actually ended up loving it? It could now be possible for Disney to predict your enthusiasm for the flick before you were even consciously aware of your change of heart.

Why not? Our data is already being used to predict how we'll vote in major elections; it only seems fitting that it could also be used as a predictor of our love of Wonder Woman or The Secret Life of Dogs.  

For Disney, it has big advantages over the old way of testing, which was qualitative, meaning the audience would be asked what it thought.

After all, we're not that reliable; we often don't know or can't explain why we feel the way we do. And it's likely that after two hours of watching a film, we're unable to remember our exact engagement at the 30-minute mark, or recall the parts where we might have dozed off.

But using this new process, Disney is able to measure the audience's facial gestures so they can match those reactions to specific scenes — even frames — of a film.

"If data analysis of audience response in real time is used to customize content, either as revision or as new unfolding content, it will create a tighter loop between what studios and producers view as successful, marketable content in the context of audience demand," says Siobhan O'Flynn, a digital storytelling consultant and University of Toronto instructor.

O'Flynn points out a data-informed strategy is already being used by the likes of Disney-owned Marvel to determine their next movies.