Distortion or Surprise: Making Policy Evaluation Interesting

In short, rather than dwelling on what seems self-evident, economists should pay closer attention to the places where outcomes diverge from intentions. Unlike the natural sciences, social-science analysis is shaped by the fact that people adapt to policy.

Introduction: What Makes Policy Evaluation Interesting?

Once economics leans more toward explanation than prediction, it begins to value narrative in much the same way literature does. People often summarize their impression of a paper with a simple question: Is the story interesting enough to make me want to keep reading?

But what exactly counts as interesting? If we never make that explicit, then “interesting” and “research taste” become a black box controlled by discourse and authority. That is why, in Empirical Economics: Intuitive Does Not Equal Obvious, I pushed back against the tendency to mystify topic selection and tried to summarize the angles from which interesting papers are often built.

The more teachers and researchers I meet, the more I realize that good topics rarely come out of nowhere. Their topics are deeply tied to their life experience. Once you hear enough of their stories, you start to feel that the insight behind a research question really is an expression of lived wisdom. Many things behind a paper become much clearer only after hearing the author explain it in person.

To learn is also to understand oneself. If even those who study the social sciences never try to examine the source of their own insight, then the path of learning has already drifted away from the path of living.

Here I want to introduce three papers, all centered on policy evaluation.

For many students in economics and management, event-study designs and policy evaluation are their first encounter with thesis writing. Most papers now follow a familiar template: DID, regional heterogeneity, mediation analysis for mechanisms, spatial spillovers. The conclusion then becomes predictable: we see the policy effect, but not the policy cost. Naturally, that invites criticism.

But we should remember that genuinely good papers are rare. In reality, most people write papers in order to survive. When we criticize a paper for being uninteresting, one side of that criticism should reflect a commitment to truth, but another side should remain sympathetic to present constraints. Criticizing dull work should also prompt us to ask whether our own papers suffer from ordinary topics or thin exposition. At the same time, we should stay alert to the possibility that the other side is merely judging from within a private rulebook, where “interesting” becomes an empty black hole of rhetoric.

Real improvement comes from genuine understanding and two-way communication.

Cost-benefit analysis and rational expectations are two core habits of economic thinking. So how can we make the story of policy evaluation more engaging? The narrative keyword I want to emphasize here is surprise or, more concretely, policy distortion.

My advisor once said: any policy can make some splash if you throw enough money at it. But as researchers, are we really here just to measure how high the splash goes?

When studying a policy, instead of asking only whether it worked or failed, we might look more closely at a few deeper questions:

• Does a policy necessarily unfold in line with its original intention?
• Can a policy really be implemented without distortion?
• After large costs have been paid, is the policy still worth it?
• …

Rather than stopping at what feels obvious, economists should pay attention to where outcomes run against the original vision. Unlike the natural sciences, social science must reckon with how people adapt to policy. That is precisely what makes it interesting.

Judicial Reform and Local Strategic Interaction

Local protectionism and promotion incentives for officials are not new topics. Scholars have already documented them using tax data and GDP data.

Strategic Interaction Across Local Governments

Because local GDP and national GDP are compiled separately, comparisons with nighttime light data can reveal GDP manipulation. As the figure below shows, nighttime lights declined sharply around 2008, yet county-level GDP continued to rise.

In 2013, the central government reduced the weight of GDP in cadre evaluation, and mayors over age 58 had little chance of further promotion. These two features generated many clever quasi-natural experiments.¹

Strategic Interaction Between Central and Local Governments

Tax revenue is a crucial state resource. The central government’s basic policy stance has been to cut taxes² in order to stimulate business innovation.

One example is the VAT reduction shown below.

But if the government loses fiscal revenue, what happens next? It has to adapt by taking on more debt.

• Increase revenue? The center is cutting taxes.
• Cut spending? Public officials, as agents in a principal-agent framework, will naturally consider their own welfare.

Given that, it is not hard to understand why local governments may respond to national tax cuts by increasing debt.

Seeing Strategic Interaction Through Judicial Data

In judicial decisions, local protectionism often appears as judges favoring litigants from their own locality.

Officials also respond actively to higher-level policy signals in order to accumulate political achievements. For example, after a new party secretary takes office, bankruptcy-case filings rise significantly relative to the period under the previous secretary.

To weaken this kind of local judicial protection and improve judicial efficiency, China implemented a series of reforms. One of the best-known papers in this area is Shaoda Wang’s Judicial Independence, Local Protectionism, and Economic Integration: Evidence from China (NBER, 2022).

The establishment of circuit courts by the central government also significantly increased the win rates of small firms and private firms.

The Distortion Created by Judicial Crackdowns on Local Protection

Now comes the key question. If earlier papers focus on how judicial reform successfully reduced local protectionism, does that mean the crackdown came at no cost?

Tradeoff between local protection and public sector performance: Lessons from judicial fiscal centralization (JBOE, 2024) studies exactly this issue.

For a long time, local governments and courts formed a community of shared interests around local protection. Once that durable relationship is broken, both sides inevitably go through an adjustment process. This is where policy evaluation often becomes genuinely interesting: higher-level policy produces lower-level adaptation.

Judges favoring local parties can be understood as a kind of political contribution. Once that channel is interrupted, the regressions below show that court spending gradually shifts away from public welfare and toward private welfare, while local corruption cases rise significantly.

In other words, while the reform weakened local protectionism, it also destroyed the informational advantage that had grown out of long-term administrative-judicial collaboration. The result was a loss in administrative efficiency for both sides. That is one of the costs of reform.

Judicial justice is a future everyone longs for, but the interests entangled in reform are complex, and the traps within reform should never be ignored. That is the most brilliant insight of this paper, and it is also the best example of surprise or policy distortion in its narrative.

Environmental Regulation and Consumer Welfare

China places strong emphasis on sustainable development, so it has introduced environmental regulation.

Each firm faces emission limits and pollution fees. Environmentally friendly firms with unused quotas can sell their pollution permits, while heavy polluters that need more room can buy permits from others.

It sounds like everyone wins, doesn’t it?

The paper Passive Collusion Among Firms: Welfare Effects of Environmental Regulation Under the Dual-Carbon Goal (China Industrial Economics, 2022) uses theory to analyze how environmental regulation affects social welfare.

On Theoretical Modeling

The key tool is the Kuhn-Tucker conditions. Corner and interior solutions correspond to whether firms are effectively constrained by environmental regulation.

Methodology matters more than clever tricks.

Let me summarize the core argument briefly.

Take the oil industry as an example. It is easy for cartel behavior to emerge: large firms collude, reduce output, and raise prices to earn monopoly profits. But such alliances are unstable. They are not equilibrium outcomes in game theory, because each member has an incentive to secretly expand output and betray the cartel.³

In reality, taxing highly polluting firms is usually seen as a good thing. But environmental regulation can induce originally rival firms to cut output together and raise prices, creating a form of passive cartelization. Under environmental regulation, both clean firms and polluting firms can earn higher profits, while consumers end up worse off.

The relationship between external threats and social welfare has always made for rich historical stories. Why did the Three Kingdoms period last so long? Once one kingdom was destroyed, the fall of another was not far behind.

External threats can foster internal unity, but external benefits can also dissolve it. A local optimum is not necessarily a global optimum; individuals and the collective are always caught in strategic interaction. That is the paper’s central story of policy distortion or unexpected consequences.

When Agricultural Subsidies Accidentally Help in the Right Way

The policy surprises above were all negative. But can there also be positive surprises?

Evaluating the Effects of Agricultural Subsidy Policy: Incentive Effects and Wealth Effects (Chinese Rural Economy, 2016) studies whether agricultural subsidies expand labor supply in agriculture.

The paper finds that many households mainly use the subsidy to stop doing the farm work themselves and hire others instead. If that is the case, then labor supply has not really increased, has it?

Agricultural subsidies did not encourage family members to increase their own labor supply, but they significantly increased the likelihood of hiring labor, using agricultural machinery, and investing in inputs such as fertilizer, seeds, and pesticides. The main effect operated through greater use of production inputs such as fertilizer, seeds, and pesticides.

But under today’s agenda of rural revitalization, scale management and factor inputs matter a great deal. So even though labor supply did not rise, subsidies still helped mechanization and larger-scale agricultural operations.

That is why the paper argues that if output growth under labor-oriented subsidies does not come from greater labor supply, then policy should be designed more precisely. Agricultural subsidies should be paired with measures that promote larger-scale farming, reduce the opportunity cost of working in agriculture, or tilt support toward major grain-producing regions in order to improve policy effectiveness.

These are also central themes in today’s rural-revitalization strategy and agricultural development.⁴

There is far too much formulaic language surrounding rural issues. Much of the policy advice people offer follows a pattern of flawed reasoning paired with a correct conclusion, which makes the discussion stiff and conventional. By contrast, this paper explains subsidy policy from the angle of policy distortion leading to unexpectedly desirable outcomes. That too is a form of policy distortion or surprise.

Surprise and the Misuse of “Counterintuitive”

So please stop casually calling every good topic “counterintuitive.” Real science should first strike us as astonishing, but after repeated reflection it should come to seem almost inevitable.⁵ Do not stay intoxicated by the sense of astonishment alone. Turn back and appreciate why the result had to be that way.

Rational expectations are one major reason economics struggles with prediction. Put simply, it is like fortune-telling that includes the fortune-teller in the forecast: once the prediction enters the world, the future changes. Of course, sometimes recursive equilibrium solutions exist. But it is precisely because of this sliver of uncertainty that “surprise” in research can reveal the researcher’s insight.