{"id":2484,"date":"2013-09-06T18:33:08","date_gmt":"2013-09-06T18:33:08","guid":{"rendered":"https:\/\/journal.c2er.org\/?p=2484"},"modified":"2018-04-30T22:11:20","modified_gmt":"2018-05-01T02:11:20","slug":"how-can-i-create-my-favorite-state-ranking","status":"publish","type":"post","link":"https:\/\/journal.c2er.org\/2013\/09\/how-can-i-create-my-favorite-state-ranking\/","title":{"rendered":"How Can I Create My Favorite State Ranking?"},"content":{"rendered":"

\"Yasimage002_generating-ranking\"<\/a>How Can I Create My Favorite State Ranking: The Hidden Pitfalls of Statistical Indexes<\/h3>\n

Yasuyuki Motoyama, Ph.D.<\/strong><\/p>\n

Jared Konczal<\/strong><\/p>\n

Ewing Marion Kauffman Foundation<\/strong><\/p>\n

The authors would like acknowledge Dane Stangler, Arnobio Morelix, Michelle St. Clair, Lacey Graverson, Melody Dellinger and Kate Maxwell<\/p>\n

\u00a0 State economic rankings are both pervasive and popular. The long list of well-known rankings includes the State Competitiveness Report by the Beacon Hill Institute, the State New Economy Index by the Information Technology and Innovation Foundation,[1]<\/a> the Cost-of-Doing-Business Index by the Milken Institute, the Economic Freedom Index by the Pacific Research Institute, and the State Business Tax Climate Index by the Tax Foundation, among many others. These rankings–and even many that are less familiar–receive substantial media attention on the scale of thousands of viewers each day. As a result, media outlets have started to create their own rankings: Forbes\u2019 Best States for Business and CNBC\u2019s America\u2019s Top States for Business. We know that there are more than one dozen out there, and the list seems to grow every year. [1]<\/a><\/p>\n

[[1]<\/a>\u00a0Disclosure: the Kauffman Foundation has given grant funding to support this report in the past. Also, Kauffman research has been used as the basis for ranking states, most notably on the rate of entrepreneurial activity, formalized in the Kauffman Index of Entrepreneurial Activity. This is an index-based data from the U.S. Census Bureau and Bureau of Labor Statistics, and has never been a normative attempt to claim that one state or another is “better” for entrepreneurs.]<\/p>\n

So business climate indexes have become a cottage industry within the economic development profession. Reacting to, and explaining why and how their state-community is ranked\u00a0by\u00a0business climate indexes has accordingly become a cottage industry among state and sub-state economic developers. The combination of publishing indexes and states and community’s reacting to them has not been a pretty picture–in fact, it can be quite challenging and sometimes demeaning.<\/p>\n

Frankly, a serious measure of cynicism, opportunism, confusion,\u00a0and manipulation has also entered this dismal business climate index picture. A large part of the business climate index “problem” is that, at heart, business climate indexes are a statistical and methodological concoction drawn from a database that is unavailable to the economic developer, the media, or the governance of your agency-community. We, as state and sub-state economic developers, are pretty much at the mercy of the entity and researcher compiling and presenting the index. Understanding statistical findings derived from poorly explained methodologies is usually not the forte of a state and sub-state economic developer. Hence this article. Statistics and methodologies are a magical land that badly needs some explanation.<\/p>\n

What You Should Know About the Methodology and Statistics of Indexes<\/b><\/p>\n

Despite their popularity and seemingly daunting sophistication, these rankings come with major limitations. First, it is not clear what exactly each ranking is measuring, and if they accurately assess what they claim to measure. This criticism may not apply to the simplest rankings; for example, the Tax Climate Index scores states only by their tax rates. If you assume that lower tax rates in any form (personal, corporate, sales, and property) are uniformly better for operating businesses, this ranking offers accurate information.<\/p>\n

Other rankings, however, may have significant internal validity problems. For instance, the Small Business and Entrepreneurship Council\u2019s Small Business Survival Index purportedly measures \u201cmajor government-imposed or government-related costs affecting investment, entrepreneurship, and business\u201d (SBEC, 2011, p. 5). They employ twenty-three indicators to create the index, including the state\u2019s taxes, right-to-work status, minimum wage level (relative to the federal level), health care and electricity costs, crime rate, and number of government employees. While it is easy to measure these costs of government, this system fails to factor in the benefits generated by government, such as investment in infrastructure, education systems, small business development centers, and business extension services.<\/p>\n

As the\u00a0number of factors\u00a0or variables on which\u00a0the rankings are constructed\u00a0becomes more numerous, the validity problem becomes even more pronounced. The Pacific Research Institute\u2019s Economic Freedom Index, for example, claims to measure free enterprise and consumer choice. One component of the index is a judicial score that is based on a creative combination of 1) straightforward quantifiable indicators, such as the compensation and terms of judges and the number of active attorneys (though it is not clear if higher judicial compensation is better or worse for the state–assumptions creep in almost without our realizing it), and 2) qualitative indicators, such as whether the state has had liability reform, class action reform, and jury service reform. Unfortunately, they do not provide a breakdown or discussion of the method for compiling these measures. And while the Pacific Research Institute admits that \u201cit is not easy to interpret these indicators\u201d (PRI 2008, 26), this judicial score is one of the five major components of its index. Perhaps, the methods have become so complex that the rankings\u2019 creators themselves\u00a0can feel a bit wary.<\/p>\n

The second limitation of these rankings is that research has found few correlations between rankings and actual economic growth-related indicators at the state level. Fisher (2005) tried hard to find connections by examining firm formation rate, job creation by the state economy, jobs created by fast-growing Inc. firms, the number of initial public offerings, and issued patents. None of the eight rankings he examined had statistically significant relationships to these indicators. Steinness (1984) and Skoro (1988) came to the same conclusion. Plaut and Pluta (1983) and Kolko et al. (2011) conducted more sophisticated, multivariate analyses, but had no luck, either. The rankings we have mentioned thus far are created by consultants and think tanks with specific ideologies.<\/p>\n

Do academic rankings provide any insights? While we do not find academic rankings of state business per se<\/i>, there has been work on some metropolitan rankings. Chapple et al. (2004) ranked high-tech areas based on occupational data instead of traditional industry-based data, and Gottlieb (2004) created the Metropolitan New Economy Index. However, Cortright and Mayer (2004) cautioned that rankings, as they are often based on a uni-dimensional scale, almost inevitably are vastly too simple to measure a complex, multi-faceted\u00a0phenomenon such as job creation, business formation\u00a0or innovation. Also as claimed by Gottlieb (2004), each individual ranking may have a different purpose and\u00a0use different measures.\u00a0Comparability between rankings is almost impossible, so academic debates about\u00a0rankings do not seem to offer us any solution.<\/p>\n

An even worse problem is the effect these indexes exert on state and local economic developers. \u00a0The current abundance of methodologically fragile rankings leads to misuse of them at the policy level. Abuse can be observed primarily in two ways. In one case, policymakers choose a conveniently positive ranking for their state and celebrate their performance (or political-ideological opponents use them to attack).\u00a0In another case, state government officials or economic development consultants choose rankings in which a state performs poorly to instigate efforts to remedy that problem, and justify a reform action or a sales pitch.<\/p>\n

Can\u00a0a Business Climate Index\u00a0usefully measure a real-life\u00a0business climate?<\/b><\/p>\n

Frankly, we reject the argument that because it purports to measure a very specific\u00a0aspect of the state or metropolitan economy, a\u00a0business climate\u00a0index-ranking is helpful to economic development. Unless the researcher can truly identify and use a specific \u00a0output indicator or measurement which mirrors the factor (job creation, business formation or innovation, for instance)\u00a0 the ranking, bluntly,\u00a0is meaningless.<\/p>\n

Moreover, our forthcoming research (Motoyama and Hui,2014) demonstrates that many rankings do not even correlate with business owners\u2019 business climate perceptions. Without any connection to macroeconomic indicators or perceptions of business owners, the primary target of rankings, what validity\u00a0can these indexes and their rankings possess? We will demonstrate in the sections below\u00a0that the limitations of every methodology, given the complexity of what they seek to measure (innovation, business formation or job creation, for instance), \u00a0can easily–sometimes\u00a0unconsciously, sometimes consciously–result in misleading or “manipulated” rankings. Even with good data and sound, transparent methodology, the process of creating rankings is extremely subjective.<\/p>\n

Subjectivity is a Relative Matter<\/b><\/p>\n

In the remainder of this\u00a0article, we conduct a series of hypothetical decisions that any researcher constructing a business climate index could likely encounter\/make and then demonstrate how unintentionally or not these simple decisions can lead to significant fluctuations in the index rankings. These decision points are seldom acknowledged in the index and are often unknown to consumers. The numbers may\u00a0reflect the state-level data, but they do not\u00a0necessarily mean what the reader\u00a0and consumer of the index think they mean, and thus we show how easily the rankings can be manipulated or misused.<\/p>\n

Next, we\u00a0will employ Pearson correlation analysis\u00a0to demonstrate to the reader\u00a0 which indicators\u00a0 are correlated with innovations and entrepreneurship. \u00a0\u00a0 [For statistical geeks who demand even more sophisticated methodology we will, at the end of this article provide a principal components analysis.] Finally in the last section, we will\u00a0“play around with” or simulate\u00a0these statistics to show how easy it is to affect the state rankings by making different,\u00a0perfectly reasonable, research decisions and using different\u00a0assumptions. Each\u00a0decision will lead to\u00a0creating personal rankings from various combinations of the measures. Maybe statistics really can lie? Here’s how business indexes are ever so easily subjective and usually misleading. [<\/p>\n

[Reader please understand that it is not necessary to follow the statistical methodology–feel free to leave that to the geeks in the crowd. What you should do is see how one can start out with seemingly fine indicators, then employ appropriate and sound statistical operations, and wind up with clusters of states that make no theoretical or apparent sense. Then, in the search for some meaning, the researcher “plays around with” some of the indicators, adjusts the statistics in some fashion (watch out for us throwing out the outliers for instance), and then comes up with new rankings. While a good magician should never disclose how his tricks work, a good researcher ought to believe that an educated consumer is preferable to a deceived or confused one]<\/b><\/p>\n

We must consider some indicators to compare. First, we\u00a0select three reasonable, even commonsensical, \u00a0indicators to measure levels of entrepreneurship<\/span> in any state:<\/p>\n