April 13, 2016

Why the Algorithm That Promised to Save Hollywood Destroyed Relativity Media

Relativity Media claimed they'd cracked the box office code. Then it all went horribly wrong.

Editor's note: author Jack Zhang is the CEO of an analytics company, Greenlight Essentials, and the following post represents his opinions on Relativity Media and the overall role analytics platforms play in the film industry. This is in no way sponsored content and no money changed hands; for more, see our comment below.

In 2004, Ryan Kavanaugh and Lynwood Spinks founded Relativity Media. Kavanaugh drew in billion-dollar investments from the hedge fund Elliott Management and claimed that Relativity could reduce the risk in the uncertain business of filmmaking by using a Monte Carlo model. They said it was the future of Hollywood. In reality, it was an epic failure.

After producing and releasing films for nearly a decade, on July 30th, 2015, Relativity filed for Chapter 11 bankruptcy following protracted lawsuits and missing loan payments.

Why did Relativity Media’s Monte Carlo model not work? Is there no way to use algorithms to make better decisions in the film industry? We discuss these questions and more below.

​Relativity Media found a non-risky way to steadily lose investors' money.

How does the Monte Carlo method work?

Simply put, the Monte Carlo helps us think about and observe risk by incorporating probability instead of point estimate.

Let me explain these two concepts in an example: imagine there are two people, Owner A and Owner B, who share the ownership of a restaurant. They need to have an understanding of how much profit their restaurant is going to make in the next month so that they can decide whether or not they will close the business.

Owner A will use a point estimate (a single value as the best estimate) and Owner B will use Monte Carlo methods. For simplicity's sake, let’s assume that the profit of the restaurant is equal to its revenue minus its expense (Profit = Revenue - Expense) and that they are not related at all.

The way Owner A will estimate the profit for the next month is as follows. First, he looks at the revenue from past 6 months:

He takes an average of them ($500), then uses that average as the estimate of next month’s revenue. He does exactly the same thing for the expenses of last 6 months:

His estimate of expense for next month is $400. Finally, he subtracts the expense estimate from the revenue estimate to get his profit estimate. Owner A, through a point estimate, concludes that there will be a profit of $100 and decides that they should stay open.

Now, let’s look at Owner B. He sees owner A’s estimate and says, "Your estimate is not bad, but it doesn’t tell us about the risks that we are taking. Let me do a Monte Carlo simulation." 

He goes on to look at the past 6 months of revenue and pulls a dice out of his pocket. Each face of the dice represents each past month’s revenue. If he rolls it 10 times, he has 10 possible outcomes for the next month. Note: the following simulation is random. He rolls 10 times and gets the following 10 simulated revenue numbers:

He does the same thing with expenses and ends up with 10 more values:

Finally, he calculates the profit for each pair of the above revenue and expenses. The simulated profit calculation with Revenue and Expense (Possible Revenue - Possible Expense = Possible Next Month’s Profit) is as follows:

Owner B says to his partner, “We could lose $300 at the most or make $500 at the most. We have about a 30% chance of losing money and 70% chance of making money based on Monte Carlo simulation. I think the risk level is reasonable, so we should stay open.”

The problem with Monte Carlo

From the above example, we can see how using Monte Carlo simulation, rather than just a point estimate, gives much more insight into risk. In the real world, though, we simulate a million times using computers before drawing the final conclusion. Monte Carlo itself is a very useful tool to estimate and illustrate risk; for that reason, it is a popular method to use in the financial world.

Monte Carlo does not build models. People build models.

However, Monte Carlo does not build models. People build models. Statistical model-building is both a science and an art. The person who builds the model needs to design the underlying input variables the model is going to use—the relationships between these variables and the statistical distribution he/she is going to use for the model. Most of the time, making these choices means taking an educated guess: by observing real-world data, you try your best to match it.  

While Monte Carlo is useful for recognizing risks and variation involved in the already built model, it certainly cannot tell you which variables to put into the model and how to formulate the relationships—such as between actors, directors, and plot—in any given film. In the restaurant example, we already know that the underlying model is Profit = Revenue - Expense, with the randomness of the revenue and expense matching the randomness of the dice. However, for complex problems like movies, there are many more unknown variables. How do we formulize the revenue of a given film (Film’s Revenue = Actor, Plot, Production Budget, etc.)? Is it even possible? 

All of these are factors that can determine the performance of the final model, and many of these factors are assumptions. Without knowing all of the above information of Ryan Kavanaugh’s model (I don’t think he told anyone; these type of models are usually highly proprietary), it is very hard to say what he did wrong. Likely, he misused Monte Carlo without fully understanding the fundamental framework of statistical modeling. 

However, we do have data on the performance of most of his films, and we will analyze Relativity Media’s films’ revenue using Greenlight Essentials’ patent-pending software to draw useful insights from the results.

How to measure a movie's success using data

Each data point in the diagram represents a single movie, where its box office revenue is measured on the vertical axis and its marketing spend is measured on the horizontal axis. The green line in the middle of the graph is the benchmark that illustrates the relationship between marketing spend and revenue.

In the graph, 50% of all films are above the benchmark and the 50% are below. The movies above the benchmark are classified as out-performers, earning more revenue than the market benchmark. The movies below the benchmark are classified as under-performers.

When we see an imbalance in that percentage, we can draw useful insights on the underlying factor. In this case, films with actor Tom Hanks have a 95% chance of outperforming the benchmark. Thus, we can assume that Tom Hanks is very bankable in general.

On the other hand, Rance Howard has only 29% of his films above the benchmark, so he could be considered not bankable in general.

Looking at the spread of all the data below, we can see that film is a very risky business with high variation and that the risk exponentially goes up as marketing spend increases.   

 

Relativity Media films' box office performance

Looking at the above graph and comparing it to the previous ones, we can see that all of Relativity’s releases are clustering together with a much smaller variation. This means that their risks are reduced.

So, did we just show that Ryan Kavanaugh’s algorithm worked? Not so fast. These films have much less variation, even though most of them are below the benchmark. In fact, only 22% of all Relativity films are above the benchmark.

This means that their films have a much lower average return than the market. Relativity’s model worked in terms of shrinking the risk, but it has a negative return compared to the market. Given that 22% is a large margin away from the market average (50%), Relativity’s model is highly likely to have a negative absolute return. In other words, Relativity found a non-risky way to steadily lose investors' money.

Perhaps they were too concerned with the idea of shrinking down the variation/risk—so much so that they were not thinking about the return. Maybe their objective of shrinking down risk is not right, or maybe high risk (variation) is in the nature of the film business, and forcefully going against it would only damage the return.

Pixar films' box office performance

Now, let’s look at the contrasting example: Pixar. As we can see in the graph below, 90% of Pixar-animated films are above the benchmark. However, Pixar films have much higher risk compared to Relativity’s films. They are very successful because their average return is high enough to take care of this excessive risk/variance, which resulted in 90% of their films above the benchmark.

You are probably already wondering what the deal is with the software introduced above. In fact, this is how we answer the question:

Is there no way to use algorithms to make better decisions in the film industry?

We can still use math and technology to help us make much more informed decisions in filmmaking. All models are wrong, but some are useful, and that is the logic behind Greenlight Essentials’ software: to be useful. We do not make any assumptions to build models or simulate risks; rather, we only use real-world real data points to draw real conclusions.

For example, we can use the software to produce highly bankable cast suggestions based on genre and plot. For instance, if we look at the Romance genre and only focus on films that have outperformed the benchmark, we can see that Anne Hathaway and other names on the list are very bankable actors in the romance genre (almost the same score as Pixar!). We can also look at the graph and see the amount of risk (variation):

Plot elements like wedding, parents, and children are also very bankable in the romance genre. We can even use the software to find out where the target market for this specific subject matter is (i.e., people who love wedding in the romance genre) and their demographics:

On the map above, each circle represents a city. The size of the circle represents the market share of the city and the color of the circle represents how enthusiastic people are about the subject matter (in this case romance). The lighter the color, the more enthusiastic people are. We can also see that most of the people who are likely to watch this genre are females under the age of 35. This can help us picture the target market so that we can use our creative mind to match their taste.

Here's an example of how the location map’s information would be useful. Looking at the horror genre’s location map, we can see that most of the light-colored cities are in Texas:

By looking at the actual numbers on the detailed data chart, we can see that El Paso has the largest market share among the top four most enthusiastic cities about Horror genre:

Several days ago, I came across an article from The Wall Street Journal discussing the horror film Insidious.

Not so surprising! It's clear that by using the location map above, we can see where the demand is for the film.

With this tool, you'll get to know the people who have historically been interested in the subject matter of your film and come up with creative marketing strategies that feed the right content to the right people​​.

How can all of this help filmmakers succeed?

If you are an indie/low-budget filmmaker who is self-distributing your film or operating under a tight marketing budget, this information can be extremely useful. You can allocate marketing resources much more effectively according to a target market demand (more bang for the buck). With this tool, you'll get to know the people who have historically been interested in the subject matter of your film and come up with creative marketing strategies that feed the right content to the right people.

Math and science can never completely solve the mystery of predicting film outcomes. Without fully understanding model-building implications and blindly combining science and art, you can easily cause financial disasters such as Relativity has done. However, strategically using math and data in a useful way can help us find evidence-based information (such as plot attributes’ bankability, actors’ bankability, and target market descriptions).

All these goodies are not easily extractable without today’s technology. Used correctly, this information, as Greenlight Essentials’ software offers, can help filmmakers better match their audience’s taste and help investors make superior average return with known risk.       

Jack Zhang graduated from the mathematics faculty of University of Waterloo with a bachelor of mathematics, honours degree. He is the CEO and chief software architect of Greenlight Essentials, where he developed and implemented his patent-pending data analytic process for entertainment product formulation. Prior to Greenlight Essentials, he has had experience in the technology and finance industries.

Your Comment

24 Comments

Too much math for a little filmmaker like me.

April 13, 2016 at 5:17PM

4
Reply
avatar
Chris Foster
Writer/ Director
171

Sorry, did I just read an advert for software ;0)

It kind of lost me half way through when I realised that the house always wins and all those gamblers with formulas still lose, the house in this case being unpredictable human psychology.

April 13, 2016 at 7:00PM

0
Reply
avatar
Jonathon Sendall
Stories
1238

Yuck. I would hate dealing with this end of "the industry". I would rather make a film for nothing than play with someone else's money and have to deal with this...

April 13, 2016 at 7:11PM, Edited April 13, 7:15PM

0
Reply
avatar
Luke Neumann
Cinematographer/Composer/Editor
2541

You have heard "use the right tool for the job"? The Monty Carlo Algorithm is great for financial modeling and statistical analysis.....but the film industry is a complete and total abstract....with every single thing being a variable and can't be modeled accurately at a specific point in time. Think..budget, story, talent, direction, release timing, competition at the box office, economy, political climate, demographics, market mood, marketing, wow factor...yada yada yada.... This is not the first great tool that has been adopted or applied to an industry (unsuccessfully) that it was not designed for... Six Sigma comes to mind!

April 13, 2016 at 7:45PM, Edited April 13, 7:47PM

0
Reply

The average RT score of his films is something like 35% as apposed to Pixar's very high average. Inputting critical response would be useful, in this case his films most often had bad word of mouth.

http://www.rottentomatoes.com/celebrity/770675252/?sort_column=year&
sort_dir=asc#moviesFilmography

http://www.rottentomatoes.com/franchise/pixar/?search=pixar

April 13, 2016 at 8:04PM

0
Reply
eric wadsworth
Director, Storyboard artist, VFX artist.
105

Pixar is successful because they are focused on story not on some retarded market calculations.

April 13, 2016 at 9:05PM

8
Reply
avatar
Jan Becker
DP, Director, Producer
198

Fuckin A

April 14, 2016 at 3:53AM

4
Reply
avatar
Terma Louis
Photographer / Cinematographer / Editor
1244

I think that everyone makes these kind of calculations but subconsciously. For example if you see a Christopher Nolan film starring Christian Bale is coming out soon, you might be excited for that film. Why? Because you remember in the past films by the two of them have generally been pretty good. That's basically what this is trying to calculate. I think that perhaps something like IMDb ratings is a better determination, but that's me personally.

April 13, 2016 at 9:53PM

3
Reply
avatar
Zachary Will
Cinematographer
807

I've been reading No Film School for quite a long time now, so I say this with love. These kinds of posts need to be clearly identified by the author or admin as sponsored posts.

April 14, 2016 at 9:59AM

2
Reply
avatar
Andrew Lewis
Writer/Director/Producer
164

Admins: This post needs to be clearly marked as advertorial/paid marketing at the top of the page and with a tag on the front page.

April 14, 2016 at 11:25AM

1
Reply
Colin Herron
Videographer/Motion Designer
74

Two or more = dice.
One = die.

April 14, 2016 at 1:43PM, Edited April 14, 1:43PM

3
Reply

I actually found this very interesting but there should be an ethics statement or disclaimer at the top. It's definitely a biased viewpoint, and it's unethical not to fully disclose the author's involvement with Greenlight AT THE BEGINNING.

April 14, 2016 at 3:16PM

1
Reply

Well said but understated, I think the author IS Greenlight!

April 14, 2016 at 3:28PM

0
Reply
avatar
Andrew Lewis
Writer/Director/Producer
164

This is what happens when you mix MBA types and financial quant wizards with something that is, inherently -at its core- an emotional experience.

This type of software and these types of people are the reason why we're getting more and more sequels, more and more comic book junk food, and nauseating doses of Anne Hathaway in roles she doesn't remotely belong in (I'm thinking of a kickboxing astronaut right now).

Software like this is the indie filmmakers arch nemisis-- not a valuable financial partner. This is the most sickening post I've read yet on this site for the insidious way it's attempting to subvert the most sacred tenants of indie filmmaking's soul and its sources of inspiration. This article is a total disgrace to its audience.

April 14, 2016 at 4:04PM

0
Reply

I completely disagree with you! Indie film is what it is, outside of the studio system, and it is what the filmmaker makes out of it. Studios in the other hand, don't want to bet, they know what movies make money and which ones don't and the reason why they make sequels is plain and simple: repeat business (just the same reason why Apple releases a new model every year). They also do bundle productions so they are pumping original content in the regular basis, so take it easy on the studios amigo!

April 14, 2016 at 8:16PM

0
Reply

This is not a sponsored post, no money changed hands, etc. etc. We always label sponsored posts clearly (and in multiple places).

I get why you guys might think it is since clearly Jack has a dog in this fight. But we reached out to him, as an expert in the field, and asked him to write up an analysis of what happened with Relativity given we are not data scientists ourselves.

The point of reading No Film School, I hope, is not to agree with everything that's posted here. You can believe that trying to predict financial success (or failure) of art is trying to quantify the unquantifiable. However anyone who works in the industry is certainly being judged by algorithms, whether it is in the areas of genre, cast, foreign territories, audience size, demographics, video views, etc.

I read this and learned something. I hope you did too. As a director, on the creative side, I'd like to know more about what the gatekeepers and studios are thinking when they evaluate a project of mine. If you don't want to know that, you don't have to read.

It's not particularly different than someone running a Kickstarter campaign doing a guest post to help promote it, or a DP sharing a BTS article and promoting his work—absolutely everyone is promoting something. Just know that this is not sponsored content, and we have never run anything of the kind without labeling it clearly.

Despite all of this, I absolutely AGREE that we should've put Jack's involvement in an analytics company up top. That's something a site design refresh will help, so all authors have a mini-bio up top. Since that design is not yet live on the site, I'm adding that detail to the top. Hope that helps.

April 14, 2016 at 4:39PM, Edited April 14, 4:43PM

7
Reply
avatar
Ryan Koo
Founder
Writer/Director

Thanks Ryan, I feel a bit like a jack wagon for jumping the gun a bit. but i do agree with Tyler, mostly.

There is more than one way to mold your career, not everyone has the same goals. I work with filmmakers who are working everyday and built great careers looking at metrics such as these. For many, money is the name of the game, and there is nothing wrong with that. Kids gotta eat.

April 14, 2016 at 7:57PM

0
Reply
avatar
Andrew Lewis
Writer/Director/Producer
164

Thanks for the fix, Ryan!
Now... Could we do something about Alex Ferrari's incessant self-promotional spam in the boards, please?

April 15, 2016 at 12:06PM

0
Reply

Something here is not right. The example comparing Relativity's movies with Pixar's clearly shows that there's a huge gap in marketing cost which directly relates to the success of the movie, although, there are many distinct cases, in the first chart it is very clear how marketing spending directly affects the outcome.
I can be almost sure that Relativity was able to project their earnings spot on but didn't know how to balance their marketing costs versus production budget.

April 14, 2016 at 8:09PM

0
Reply

I think this is genuinely fascinating - I really appreciate that Zhang has acknowledged the realities of how volatile and unpredictable the film industry is. But this kind of data would still be useful in terms of focusing our limited distribution budgets on very specific markets that have a higher statistical likelihood to provide an audience.

Thanks to NFS for reaching out to Mr Zhang to provide this analysis, it's genuinely fascinating, and great to see people striving to find more reliable ways of analysing data for filmmakers' benefit.

April 14, 2016 at 8:54PM

0
Reply
Srini Madhavan
Writer/Director/DoP
81

Cool tool, but quants caused the housing bubble and obviously moneyballing film doesn't work. The biggest problem with MCMC (Markov Chain Monte Carlo) methods is assuming that you can really predict the future from the past. You can't totally, and this cannot be the basis of your optimization function. In truth, MCMC isn't supposed to be used for this purpose, but it is in practice. It should only be viewed as probabilities nothing more. As the industry seeks to reduce it's risk analytically, it will lead to creating a stale films which lack the kind of fresh creativity that attracts audiences to movies. Think about it, why would I want to see the same actors in the same movie over and over again? That's hollywood's problem in general. They don't get that audiences really respond to original and novel experiences because this is hard to quantify. That said, you can't just optimize for minimizing risk, profit must also be maximized. This means LP's should be employed. It's a min/max analysis on top of the MCMC. But before that you must identify the key attributes that contribute to higher revenues. This could be gleamed from ML -- as classifiers. This would minimize bad assumptions, maybe. But, when it comes down to it, films cost too much and so does marketing. Studios need to focus on using new media more effectively to reduce marketing costs, and they should stop bloating their budgets. But, quiet as it's kept, the real money in this biz is made on the way in. This is why Relativity failed. They made stale shitty movies that took no risk and then spent too much making and marketing them.

April 14, 2016 at 11:25PM, Edited April 14, 11:25PM

2
Reply
avatar
vishnu
EP / Writer / DIR
88

Interesting read.
I was never a fan of statistics, to only field in mathematics I really disliked, but this is a fun read.
There are 2 big lessons:
- trying to play it all safe will hurt revenue, because it will be middle of the road
- such statistics work in hindsight only: results in the past are no prediction for the future. It's like getting in a plane. An airplane is statisticly the safest means of transportation and still, it can be the plane your are in right now that crashes. Even if another one crashed 5 minutes ago...

April 15, 2016 at 1:33PM

4
Reply
avatar
WalterBrokx
Director, DOP, Writer, Editor, Producer
8473

Very interesting insight!

April 16, 2016 at 6:30PM

1
Reply

WTF, NoFilmSchool?

Why do you DISABLE ZOOMING on your mobile site? Why break crucial browser functionality that has been standard for EIGHT YEARS?

Moronic.

April 17, 2016 at 2:58PM

0
Reply
David Gurney
DP
1395

Analysis and projections are not able to protect profits or investors. Only a structure that doubles or triples tax credits, and creates cash collateral that meets or exceeds expenditures can make a film profitable from dollar one. Only one company has such a structure... Mine. Geneva Media Holdings. Everything else is simply hot air, crystal ball B.S., or someone's make-believe wish and hope for projections that amount to guesswork. Troy "Binky" Dyer

June 29, 2017 at 3:53AM

0
Reply
Troy Allen Dyer
Risk Mitigation for Media
1