Learning in Microeconometrics

A review of some debates between the Structural and Experimental schools

Posted on Jun 08, 2019 By Nanjin Zeng

Zeng Nanjin & Li Zichao WISE IUEC 2016

1.Introduction

In the challenge “Causal Inference 2”, we are required to Summarize the arguments between the Structural and Experimental approachs in empirical works made by the various authors in the following series of papers in JEP Vol. 24 and views on the same topic by Rust (2014). We had spent two weeks in going through these papers and reached some common thoughts to their arguments. In section 2, we would give a brief summary for these five papers. In section 3, we would share some thoughts raised in our recent discussion about these paper.

2.Summary to these papers

The first four paper is a series of paper in JEP Vol. 24. The arugment is brought by Angrist and Pischke, chasing by papers from Keane, Sims, the two who are strongly disagree with their appeal, Nevo and Whinston holding a relatively netural opinion with their argument.

2.1 Taking the Con out of Econometrics (Angrist and Pischke, 2010)

In this paper, they begin with the review of some concerns on the quality and credibility in data analysis expressed by several economists in last century, for example, the concern by Leamer (1983) towards less robustness in contemporal works, like Ehrlich’s study (1983) about the deterrent effect of capital punishments, due to the variable result under different assumptions on functional form and addition controls. Moreover, Ehrlich used instrumental variables but did not attach some explainations, which makes his work less reliable. They also gives some other examples like The Education Production Function, Coleman et al. (1966). All of them are considered to be in a bad research design.

Then they claimed that these concerns have been address by these key improvements these days.

  1. Better and More data from a rapidly increasing amount of micro data and diministrative records.

  2. Fewer Distraction due to more understanding of regression analysis, with robust standard errors, automated clustering and so on, which lead to remarkably robust conclusions.

  3. Better Research Design and more transparent discussion in applied econometrics, like widely used RCT, DID, RD and IV. They claim that the application of better design could avoid much of complex mechanical frameworks. Then they give plenty of examples, like evaluating class size and school performent using IV (Angrist and Lavy,1999), which avoid omitted variable bias in the previous works.

After that, they criticize the less use of experiental method in macroeconomics and industrial organizations. At their view, the effort of computional researchers to build a dynamic model and caribrate the model is not design-based research and produces no evidence on the existence of causal effects. For the Industrial Organization, they also believe that a transparently identified design-based experimental estimates is better than a complex simulation-based estimates with too-strong assumptions.

Finally, they responds to the concern of experimental designs is result-oriented instead of trying to solve more meaningful problems. They think that the concern with external validity could be solved by accumulating empirical evidence. Moreover, the design-based approach has wide applicability to be complement with good economic questions. They give some examples to refute the claim that experimental works narrow policy effect.

2.2 A Structural Perspective on the Experimentalist School (Keane, 2010)

In this paper, Keane disagrees with Angrist and Pischke’s idea about experimental method, which regards it as a panecea.

He suspects that the econometrician fits many statistical model and choose the pleasing one to report. First, he states that the experimentalists also rely a series of assumptions using the example of class sizes. Second, he raises a concern whether the experimental method give robust answer ignoring other imput in the education production function. Then, he concludes that the experomental method also has its deficiencies thus far from a revolution which is significantly better than structural approach.

Then he give a opposite view to Angrist and Pischke’s appeal to utilizing experimental methods in more fields. In labor economics, ecomomists still have a number of controversies. Moreover, the marketing has more discoveries in this field using demand models and more data, and having a greater emphasis on external validation. And the failure in industrial organization is not due to the structural model itself but the misspecified models.

After that, he gives the examples how good data could benefit structural modeling and how could economic theory be useful in RCT. Finally, he emphasizes the external validation by sharing how he works with structural methods.

2.3 But Economics is not an Experimental Science (Sims, 2010)

In this paper, Sims citicizes with Angrist and Pischke’s idea. He thinks that these narrow, overly simplifed approaches to data analysis would lead to misunderstanding.

He concern that there is lack of technically tooled up ecnometricians who knows about the weakness and restriction of each experimental models. Lots of applied works imitates the procedures of prominents, which may lead to mispecifications. Therefore, econometricaians should be trained enough to confront the complexities and ambiguities in non experimental inference.

Then he raises some advance in macroeconometrics considered to be ignore by Angrist and Pischke, which contribute to broad concensus on policy.

At last, he goes back to the research of class size and give two concerns. The first one is whether the narrow constrainted result on the class size could be useful for policy makers. The second one is that the result could be variable if there is a nonlinear relationship between variables.

2.4 Taking the Dogma out of Econometrics: Structural Modeling and Credible Inference (Nevo and Whinston, 2010)

In this paper, Nevo and Whinston give their opinion that the best empirical work should be in combining careful design, credible inference, robust estimation methods and thoughtful modeling, no matter whether it is a experimental or a structural one.

First, they states that credible identification and structural analysis is complements, because the previously observed effects may not provide a good prediction to the current one. Structural analysis gives us the way to relate observations to different changes in the future. To emphasize, then they compare this two method in the background of merge analysis in industrial organizations. It may be difficult for statistic approach to find a proper control group in this field. Moreover, the treatment of merge is not completely exogenous, which relates to many market conditions. Another point is that the merge analysis usually require simulation without similar historical data. Due to these uncertainty, at least the statistical approach is not a simple solution.

Then, Nevo and Whinston evaluate the two approach by retrospective estimates and find that both of them do a good job in the past. Then they continue to carry some difference between labor and IO, including available data, policy maker’s need. They consider that we should choose the proper approach respecting the facts in the market, data, and questions in different fields.

2.5 The Limits of Inference with Theory: A Review of Wolpin(2013) (Rust, 2014)

It is a complement with Wolpin’s previous work. In that one, Wolpin emphasized the need of theory in the empirical study. To balance, Rust raises some limits of the structural works.

  1. The structural econometrics is based on the assumption that there are some parameters is stable when policy changes. It could help them to simulate the result due to policy variables. But due to technology accumulates and knowledge would alter the preference, these considered invariants might also change.

  2. As the number of variables in the dynamic model increase, the amount of computer time to solve the model would increase exponentially. However, it could be eased by developing algorithms and using simpler behaviour models.

  3. The underlying structures are based on rational expectations, utility maximization, separable perferences, conditonal independence, but the people sometimes are not rational. That means a mispecification problem.

  4. The equilibria outcome could be not unique, in which case they should choose some of them by posing extra restrictions. But whether these restriction is valid and constant is being asked.

  5. Limits to deductive inference.

But they have the same idea that experimentists and theorists should work together rather than attack towards each other’s method. Rust gives some examples in other fields to explain how could treat the relationship between theory and experiment. And he states how could experiment and theory could contribute to each other. For example, experiments can help structural economentricians develop better models and discard inappropriate assumptions while the structural models could guide the experiment design.

3.Our opinion

After discussion, we have some common thought to these paper.

  1. First, we disagree with Angrist and Pischke as they think that all the advance of credibility could be owed to experimental method. Though we think the experimental work is not fits many statistical model and choose the pleasing one to report, it needs careful model selection and experiment design. We think it could not take place of the theoretical analysis, which could guide our design. For example, in the paper we presented in other class, which is about Group Size and Incentives to Contribute (Zhang and Zhu, 2011). The authors give a structural model about utility and warm glow before going the regression on a natural experiment, and there finding could contribute to the setting of the structural model in the way discarding inappropriate assumptions in the model, just as Rust appeals in his paper. We think that it represnts a nice way for combining experimental and structural approach.

  2. We agree with Nevo and Whinston that we should choose the proper approach respecting the facts in the market, data, and questions in different fields. We should be tooled with different types of economic approach to choose a better one depending to the problem we want to study. We could not constraint ourselves in tools before we analyze the actural question.

Reference

[1]Angrist, J., & Pischke, J. (2010). The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics. The Journal of Economic Perspectives, 24(2), 3-30. Retrieved from http://www.jstor.org/stable/25703496

[2] Keane, M. (2010). A Structural Perspective on the Experimentalist School. The Journal of Economic Perspectives, 24(2), 47-58. Retrieved from http://www.jstor.org/stable/25703498

[3]Sims, C. (2010). But Economics Is Not an Experimental Science. The Journal of Economic Perspectives, 24(2), 59-68. Retrieved from http://www.jstor.org/stable/25703499

[4]Nevo, A., & Whinston, M. (2010). Taking the Dogma out of Econometrics: Structural Modeling and Credible Inference. The Journal of Economic Perspectives, 24(2), 69-81. Retrieved from http://www.jstor.org/stable/25703500

[5]Rust, J. (2014). The Limits of Inference with Theory: A Review of Wolpin (2013). Journal of Economic Literature, 52(3), 820-850. Retrieved from http://www.jstor.org/stable/24434112

[6]Xiaoquan (Michael) Zhang, & Zhu, F. (2011). Group Size and Incentives to Contribute: A Natural Experiment at Chinese Wikipedia. The American Economic Review, 101(4), 1601-1615. Retrieved from http://www.jstor.org/stable/23045913

[7]Jiaming Mao. “Foundations_of_Causal_Inference”[Z].2019-06-09.personal copy