January 08, 2014

Introduction

In many, if not all, employment discrimination lawsuits both the employer-defendant and the plaintiffs realize that there is a statistical disparity between the protected and non-protected employee groups at issue in the case. It is the meaning of the statistical disparity that is most frequently the area of contention and subject to differing interpretation by the employer and plaintiffs in the discrimination lawsuit. Since the 1970s employment attorneys and the courts have relied heavily on the mathematical and probability concept of statistical significance in assessing the underlying importance of an observed statistical disparity in employment conditions between different groups of employees. The concept of statistical significance, which in essence is a method to assess the likelihood that random chance, and therefore not discrimination, produced the given employment disparity is also routinely used by courts to assist them with determining if the underlying statistical employment analysis meets the reliability requirements of Daubert v. Merrell Dow Pharmaceuticals , 509 US 579 (1993).

While the concept of statistical significance can be helpful for the courts and helps labor economists and statisticians assist juries with understanding the underlying context of a statistical employment disparity, an over reliance on the statistical significance paradigm, especially when it is done at the expense of economic significance, can potentially produce misleading inferences concerning employment discrimination. Economic significance is a well established concept that takes into account the magnitude and implications of the alleged disparity in employment conditions. When used in conjunction with the traditional concept of statistical significance the inferences from a given statistical analysis can be quite probative and compelling. In recent years, courts have in fact begun implicitly utilizing both significance paradigms when dealing with statistical evidence in employment cases. This article examines the concept of economic significance and its relevance to statistical evidence in employment discrimination cases.

Background

In recent years, the importance of statistical analysis in employment discrimination cases has grown from an analytical tool that was used almost exclusively in class action employment discrimination lawsuits to being utilized in many different types of employment causes of actions including pattern and practice lawsuits as well as cases involving single plaintiffs. Previously opaque statistical notions, such as the standardized measurement of statistical significance, are now becoming common concepts that attorneys and courts routinely contend with when handling statistical evidence in employment discrimination cases.

There is a substantial amount of case law that has both directly and indirectly contributed to the definition of statistical significance and how experts present statistical employment evidence in employment cases. Generally, when determining if an employment disparity is statistically significant the courts tend to reduce the assessment to a standard deviation. The number of standard deviations associated with a particular employment disparity tells the court, in terms that are uniform across different types of employment analyses, the statistical likelihood or probability that random chance would have produced a particular employment outcome in the absence of discrimination. Conceptually, the higher the number of standard deviations associated with a particular employment discrimination outcome the less likely that a random and non-biased employment process would have generated the outcome in the absence of discrimination. In practice, the calculation of the number of standard deviations is performed using generally accepted mathematical formulas.

Statistical significance testing in employment discrimination cases

Employment case law is replete with examples and discussions dealing with issues of statistical significance testing in employment discrimination cases. For many statistical experts in the employment area, the Supreme Court case Castenda v. Partida, 430 U.S. 482 (1977) provides one of the more definitive statements on statistical significance testing. In this case, which dealt with jury discrimination, the court stated that “as a general rule for such large samples, if the difference between the expected value and the observed number is greater than two or three standard deviations, then the hypothesis [concerning randomness] would be suspect to a social scientist, 430 U.S. at 496 n.17”.[1] Although the case specifically dealt only with a specific statistic test (the binomial test) labor economists and statisticians have adopted and applied this rule to other types of statistical tests in the employment area. Other courts have made clear statements concerning statistical significance testing in employment discrimination cases. For example, in Lopez v. Laborers, 987 F.2d 1210, 1214 (5th Cir. 1993), the Fifth Circuit held that the court should not reject chance as the underlying explanatory factor unless the differences are greater than 3 standard deviations.

Statistically significant employment disparities are not always economically significant

The reliance on the statistical significance paradigm in the courts is in some respects at odds with the professional and academic literature in economics, statistics and social sciences. Many studies in the social sciences suggest that the importance of a particular statistical or empirical result should be viewed not only in terms of statistical significance but also in terms of economic significance.[2] While there is no universal definition, economic significance is a well established concept that suggests that the researcher take into account issues such as magnitude and the overall implications of the reported correlation or effects. In the context of an employment discrimination case, determining the economic significance of a particular analysis would generally involve assessing the magnitude and scope of the alleged disparity in employment conditions between the protected and non-protected employee groups as well as determining the congruence of the statistical analysis with the case facts.

In essence, the fields of economics and statistics suggest that the most compelling and reliable statistical evidence in an employment discrimination case would utilize both statistical and economic significance paradigms when assessing the importance of the reported statistical disparities. Take, for example, alleged discrimination in a reduction-in-force suit. Typically the analysis would begin by examining whether the disparity in job terminations among groups was statistically significant. As discussed above, statistical evidence of discrimination would arise where the observed number of terminations is statistically significantly greater than the expected number of terminations. If the statistical analysis shows disparity, then the courts will examine whether this disparity is economically significant. This would entail looking at such factors as the magnitude of the statistically significant disparity, whether the disparity reflects permissible business practice, and whether ambient economic conditions may have created the disparity. In recent years, courts have found this two-pronged approach particularly suitable to the evidentiary framework established by McDonnell Douglas. See, e.g., Smith v. Xerox Corp., 196 F. 3d 358, 364-65 (2d Cir. 1999).

Using both the statistical and economic significance paradigms allows courts to further check the reliability of a statistical employment analysis and provides juries with a better way to understand complicated and often confounding statistical evidence. While the statistical significance model in its strictest form is silent on the effects and trade offs of relevant, but unmeasured employment factors the concept of economic significance allows the courts and the jury to at least consider their potential effect on the alleged statistical disparity. A number of recent cases have shown that courts are increasing looking at more than statistical significance. Generally the courts in these cases evaluate the statistical significance (or lack of) as well as related factors such as the magnitude and scope of the employment related disparity and the congruence of the expert’s research findings with the undisputed facts in the case.

In Meacham v. Knolls Atomic Power Lab. 381F. 3d56 (2d Cir. 2004) the court weighed the magnitude of the statistical disparity found by the plaintiffs versus the explanatory factors offered by the employer in an age discrimination case that involved an employee reduction in force (RIF) action. The plaintiff’s expert found that there was a highly statistically significant difference between the termination rates of employees 40 and over and those under 40. The data showed that about 98% of the individuals terminated were in the protected age class where as the protected class comprised only about 60% of the population that was eligible for the RIF. According to the plaintiffs’ expert the difference was highly significant and would only occur by random chance 1 in 1,260 times. The court found that the size of the disparity overwhelmed the explanatory economic factors offered by the defendant-employer.

In Morgan v. United Parcel Service of American. 8^th Cir. No. 02-2545, the court again weighed the existence of an alleged employment disparity versus the explanatory factors offered by the employer. In Morgan, the court found that the economic factors that the plaintiff did not include in the statistical model, which the court found to be flawed but admissible, outweighed the statistical disparity in pay and promotions the plaintiffs expert purported to find. These cases, and other recent employment cases, strongly suggest that courts are taking a more rigorous approach when evaluating statistical evidence in employment cases.

Implications for use of statistics in employment discrimination cases in the future

As courts become increasingly sophisticated and knowledgeable about the statistical methods in employment discrimination cases, the more stringent two prong approach of assessing the importance of statistical results, favored by professional and academic economists and empirical statisticians will become more and more common. While the process of determining economic significance does not lend itself to the discussion of a single metric such as a standard deviation, there are a number of issues that employment attorneys should consider when using statistical evidence in employment cases in the future. First, the implementation of the more rigorous approach of evaluating statistical evidence will necessarily increase the level of scientific rigor embodied in statistical analyses of employment discrimination. Second, while it may not be necessary from the model standpoint to have all the intimate details of an employment process, future analysis will require an even more careful dissection of the underlying employment process. Third, the nature of determining economic significance will require the skills of experts who possess a good understanding of employment processes as well as discrimination concepts in general and may involve a more multi disciplinary approach to produce the most reliable and compelling statistical evidence. Finally, the two-step paradigm of evaluating significance fits well into, and may actually reflect, the evidentiary framework of discrimination cases.

Dwight Steward, Ph.D. is an economist and statistician at Econ One Research Inc. Dr. Steward specializes in the statistical analysis of employment discrimination allegations.

Sean O’Donnell, Esq. is an attorney at the U.S. Department of Justice.