The former USTR-Africa who designed the African Growth and Opportunity Act (AGOA) preferential trade scheme declares it a “phenomenal success“:

Rosa Whitaker: I think it’s been a phenomenal success. Has it been a panacea for everything in Africa? No, it wasn’t designed to do that. But if you look at the return on the investment, it’s been amazing. It costs the American taxpayer very little – about $2 million a year. In under a decade, exports from AGOA-eligible countries grew over 300% from $21.5 billion in 2000 to $86.1 billion in 2008…

AGOA helped develop an automobile industry in South Africa. In 2000, that industry was exporting about $148 million; it has increased to $1.9 billion in 2008. Car parts exported to the U.S. had an 18-25% tariff. When those tariffs came off for Africa, the assembly part of that manufacturing process moved to South Africa. There are plenty of other examples. Lesotho was exporting $139 million in apparel in 2000; now it’s over $340 million: a 143% increase. Kenya’s cut flower industry expanded from $34 million in 2001 and to exports over $240 million now. Swaziland was exporting $85,000 in jams and jellies in 2000; today it’s $1.6 million. For a small country like Swaziland, that’s important. Then you have Tanzanian coffee and other products. I could go on and on.

Policymakers frequently evaluate programs using this approach — they compare circumstances before and after legislation passed and judge the program based on the difference in outcomes over time. But of course, correlations aren’t very informative about causal relationships.

Economists are interested in the counterfactual — what impact did the program make relative to what would have happened without the program? The most obvious problem with a before-and-after comparison is that steady growth creates improvements over time, regardless of policy changes. For example, Singapore’s Business Times touted that US-Singapore trade had grown nearly 20% since the US-Singapore preferential trade agreement took effect, but US-Malaysia trade grew the very same amount during that period without any US-Malaysia PTA.

Similarly, telling us that African export volumes grew from 2000 to 2008 isn’t very informative, because we naturally expect exports to grow over time as economies grow. (If one wants to suggest that AGOA encouraged greater African openness, the appropriate measure would be the exports-to-GDP ratio.) Identifying the causal impact of AGOA requires a method that distinguishes the increase in exports due to the trade preferences from the counterfactual scenario. (A 300% increase in exports is big, so I’m not suggesting that AGOA necessarily had zero impact. The question is: what share of the increase was due to AGOA?)

In such circumstances, economists often turn to an identification strategy known as “differences in differences“. This involves comparing differences across countries in their differences across time. For example, only some African nations are AGOA-eligible. If African economies receiving preferential tariff treatment from the United States experienced export volume growth that was faster than export volume growth in ineligible economies, we might think that this suggests that AGOA spurred greater exports. However, such a comparison doesn’t constitute valid causal inference in the case of AGOA, because AGOA eligibility was determined according to governance and policy criteria that likely make a difference in economic and export growth. Countries with characteristics making them AGOA-eligible may grow faster than their neighbors due to those characteristics, even without any preferential market access.

Paul Collier and Tony Venables tackled this by taking what is akin to a differences-in-differences-in-differences approach: they looked at the value of a country’s apparel exports to the US relative to its apparel exports to the EU (World Economy, 2007). The thrust of their story is captured by their Figure 1:

Collier & Venables (2007) Figure 1.

African apparel exports to the US increased dramatically faster than such exports to the EU in the early 2000s (even though the EU’s Everything But Arms initiative, which is similar to AGOA, launched in 2001). Collier and Venables also present econometric results in which AGOA apparel eligibility is associated with significantly greater relative exports to the US. A glance at the data on South African automobile exports also suggests that Rosa Whitaker’s story is meaningful in comparative terms: auto exports to the US jumped while exports to the UK and Germany fell slightly.

Yet such evidence need not imply that AGOA caused a significant increase in exports by eligible countries. The AGOA trade preferences raised both the incentive to export and the relative incentive to export to the US. It is possible that AGOA-eligible countries would have experienced significant export increases even in the absence of the preferential program and the tariff advantages of AGOA only induced them to direct their sales to the US instead of the EU. Such a claim is compatible with the two pieces of evidence discussed thus far: (1) African exports to the US increased significantly after AGOA came into force and (2) AGOA-eligible economies export more to the US relative to the EU.

Collier and Venables (2007) and Frazer and Van Biesebroeck (2007) address such concerns to some degree. For example, the latter show that:

The impact of AGOA on E.U. imports is in column (6). The effects for most product categories are not significantly different from zero. Perhaps surprisingly, where the effect is significant, it is positive. For example, E.U. imports of GSP-Manufactured products, are found to increase by 4%. A potential explanation (among many) could involve spillover effects from the increased U.S. imports.

Note that though this evidence makes the alternative story about export diversion suggested in my previous paragraph rather unlikely, it cannot completely rule it out (perhaps the relative magnitudes aligned so that the size of the total export increase offset the change in relative shares, leaving exports to the EU constant). This demonstrates one of the difficulties of doing causal inference in a non-experimental setting. We have highly suggestive evidence, but, with enough effort, one can conceive of an alternative explanation.

So was AGOA a success? Probably. Economists have both theoretical reasons to expect it would improve African exports and empirical evidence that suggests that it did. Policymakers and other commentators would be more persuasive if they cited comparisons (in the spirit of Figure 1 from Collier and Venables) rather than just presenting the time series of US imports from Africa – say something like “AGOA-eligible countries’ exports to the US  grew 300% in the last eight years, substantially more than their exports to Europe”. Better (if imperfect) efforts at identifying the counterfactual distinguish the studies analyzing AGOA from meaningless statistics cited in support of other trade policies.

[I’ve tried to informally convey some ideas about empirical identification issues in the context of AGOA. For a proper introduction to the topic, start with a paper or book that mentions the Rubin causal model, such as Angrist and Pischke’s Mostly Harmless Econometrics or Imbens and Wooldridge (JEL, 2009).]