When Analyzing Data on Film Incentives, You Have to Start with the Right Data

Summary: A recent study analyzing the impact of state film production incentive programs has a number of serious flaws including poor data selection that render its conclusions invalid.

In August 2016, the University of Southern California promoted a paper by Michael Thom, an assistant professor of public policy, which reached the odd and attention-grabbing conclusion that film production incentive programs have little to no impact on film industry employment and output.  This runs counter to the landscape of studies that have measured the direct employment impact of key state production incentive programs using Bureau of Labor Statistics data (some examples here and here). Unfortunately, the new Thom paper has fundamental scientific flaws that undermine the paper’s credibility and render its conclusions completely unfounded. (Note that a second paper co-authored by Thom examining the factors that lead to film production enactment and termination, also appears to be methodologically flawed but is not examined in this analysis.[1])

Here’s a quick look at a few key flaws in this research:

The use of imprecise data dilutes and biases the results

Although the author describes the analysis as focusing on “motion picture production,” the Bureau of Economic Analysis (BEA) data[2] being used to study the effect of film production incentives is aggregated data that includes movie theater and sound recording industry jobs and wages. A ticket taker at the local cinema or an engineer at a recording studio should not be affected by film production incentives. Their inclusion in the study adds noise that dilutes and biases the results of the model.

To adapt the proverbial apples and oranges analogy, in this case you are looking for some oranges buried within a large cart of apples. The presence of those apples are especially problematic in the models Thom reports. Aggregated data means that any effects for workers that could truly benefit from incentives can be buried by stagnant or falling employment of erroneously included segments.

In fact, movie theater jobs account for an average of 59% of all motion picture industry related jobs (as categorized by the Bureau of Labor Statistics (BLS), which is the source for granular employment data) outside California and New York. This percentage rises as high as 80 or 90% in certain states. Additionally, the measured sound recording industry jobs declined 45% nationally during the period covered by the research. The employment and wage trends for the more highly aggregated data series that Thom uses (NAICS 512) do not seem to be well correlated with trends in the more specified series more directly related to production employment (NAICS 51211), suggesting that using the more aggregated series will bias any potential results.  Moreover, the dataset does not adequately capture freelance workers and monthly workers that are prevalent in motion picture production.

The model suffers from data collection and design issues

The model used in the Thom paper focuses on whether or not a state has a certain type of incentive program in a given year: transferable or refundable, which refers to the technical way in which the recipient of the incentive can use it and is not a relevant factor for potential impact on employment. The analysis attempts to control for tax incentive size by state, which Thom includes as a measure of annual spending change. However, including this variable measure in millions of dollars allows for larger states (New York and California) to dwarf the effects of smaller states, and potentially dampens any measurable effects of differences in incentives. Additionally, it is unclear where Thom gathered his time series data of film tax incentives by state and how accurate it is, as the citations do not lead to data covering incentives for every year and every state.

Thom’s measures of the outcome of interest in the regression analysis are problematic in the opposite way to the incentive size variable. Thom uses annual percent change, which biases against larger and more stable states. Oxford Economics explains all of these issues in greater detail in a review they recently conducted of the Thom paper.

Properly accounting for the key components and difference in incentives is crucial for this type of model to assess the impact of production incentives.  More precise BLS data (NAICS 51211) indicates that film production employment grew in states with a credit rate of over 20% (excluding New York and California) at double the rate of non-incentive states between 1998 and 2014. Since higher credit rates are associated with double the growth in film industry employment, it seems reasonable to suspect correctly addressing the model design issues could have led to radically different results than what Thom reports.

There are flaws in the paper’s baseline assumptions

Overall, the Thom analysis reflects a lack of understanding of motion picture production and how incentives relate to those productions. The analysis strangely finds a different economic effect depending on how the credit is monetized by a production company.  The claims that transferable and refundable tax credits will have different impacts on employment, wages, and other metrics are not clearly explained in the Thom paper and are implausible based on how transferable film credits actually work, undermining the conclusions and interpretations of the results. It is also not explained why the metrics used would be more relevant to impact than indicators such as effective credit rates (the ratio of qualified expenditures and credit percentage as applied to production budgets).

In addition, the historical role of incentives from other countries such as Canada and the United Kingdom is not addressed in the Thom paper. Incentive programs outside the US are likely to spur program creation and impact spending levels by states that wish to attract productions, and the impact of international competition should not be ignored.

How do all these flaws affect the conclusions?

The author concludes that despite finding a statistically significant positive impact of transferable tax credits on changes in employment and of refundable tax credits on changes in wages of motion picture employees, that the results are minor. Given the major data and modeling flaws, the fact that the model nonetheless finds some significant effects on employment and wages likely indicates that if the data and model issues were corrected, measurable effects would be greater than shown by the current analysis.

In short, the data and methodological flaws of the study mean that the conclusions drawn by the Thom paper are not compelling evidence on this topic. Considering all of this, it’s safe to say that if this paper had been reviewed by economists familiar with labor data and with production incentive programs, it would not have stood up to scrutiny and would not have been recommended for publication.

[1] While we have not yet had the chance to conduct an in-depth review of Thom’s second paper, a quick read through suggests it may suffer from similar issues, including modeling flaws (unexplained, potentially unnecessary variables, questionable variable transformations, and failure to account for key factors that could affect the outcome variables, including the presence of incentive programs in other countries), misidentified causality, and limitations in the substantive interpretation of the results.

[2] Note that the other non-labor metrics (motion picture industry concentration and gross state product) used in the Thom paper are of unclear value to the analysis and the Thom paper’s own results show that the model used is a poor fit statistically when measuring effects on these metrics.

NOTE: This blog was updated on March 6, 2017, to incorporate revisions and additional context based on further examination of the Thom paper by the MPAA and other parties.