This article first appeared in TESOL Quarterly, Volume 45, Number 4, pgs. 628–660. Subscribers can access all issues for free here. TESOL members can subscribe to TESOL Quarterly here. To become a member of TESOL, please click here, and to purchase articles, please visit Wiley-Blackwell. © TESOL International Association.

The study discussed in this article explored the concept of test validity (the overall quality and acceptability of a test; Chapelle, 1999) and how teachers’ expert judgments and opinions can be viewed as part of a test’s validity argument. It did so in the context of a statewide battery of tests administered in Michigan, a large Midwestern state in the United States, to English language learners (ELLs) from kindergarten through 12th grade. Data were gathered in the weeks after the testing by surveying educators who had been involved in administering the tests.

Using these data, in this article I examine the validity of the tests by analyzing what the teachers believed the tests measure and what they believed the tests’ impacts are. In essence, this paper investigates the social consequences of a large-scale testing program and scrutinizes ‘‘not only the intended outcome but also the unintended side effects’’ of the English language tests (Messick, 1989, p. 16). Because the testing in Michigan was required by the federal government’s No Child Left Behind (NCLB) Act, I begin by summarizing that law.

THE NO CHILD LEFT BEHIND ACT AND LANGUAGE POLICY TESTS

Although the United States has no official national language policy (Crawford, 2000), NCLB is sometimes viewed as an ad hoc federal language policy that promotes an English-only approach to education (Evans & Hornberger, 2005; Menken, 2008;Wiley &Wright, 2004). NCLB has created stringent education requirements for schools and states. Title I of NCLB,1 which provides federal funding to schools with low-income students, requires those schools to meet state-established Annual Yearly Progress (AYP) goals and to achieve 100% proficiency relative to those goals by 2014. (For details about how states define proficiency, see Choi, Seltzer, Herman, & Yamashiro, 2007; Lane, 2004; Porter, Linn, & Trimble, 2005.) If a school fails to meet the AYP goals, it suffers increasingly serious consequences, eventually including state takeover.

The law has special requirements for seven identified subgroups: African Americans, Latinos, Asian/Pacific Islanders, American Indians, students with low socioeconomic status, special education students, and ELLs. Before NCLB, achievement gaps between subgroups and the overall student population were often overlooked (Lazarı´n, 2006; Stansfield & Rivera, 2001). The law addresses this problem in three ways. First, there are special testing requirements for the subgroups. Now 95% of students within each subgroup—including ELLs who have been in the United States less than 1 year—must be tested for a school or district to meet its AYP (U.S. Department of Education, 2004b). Second, since each subgroup must achieve the same school and statewide AYP goals that apply to the general population, ELLsmust alsomeet English language proficiency benchmarks through additional tests. Finally, schools must report the scores of these subgroups separately, enabling stakeholders such as funders, test developers, teachers, test takers, and parents (McNamara, 2000) to hold the educational system accountable for discrepancies in scores.