ALIS Newsletter - March 2014 (Plain Text Version)
|
||||||||||||||||||||||
|
In this issue: |
Articles STANDARDIZED ASSESSMENTS AND ITA CERTIFICATION PROGRAMS
This brief article discusses several oral assessments used for ITA screening. Although they have some similarities, they each differ in important respects. Table 1 gives some information on the assessments mentioned in this article. Table 1: Some Oral Assessments used for ITA Certification
In order to investigate the current state of practice regarding TOEFL iBT Speaking use in ITA programs, Farnsworth (2012) conducted an online survey upon which this article is based. Coordinators of ITA assessment programs were asked about the makeup of their institutions and the size of their ITA population. They were then asked to describe their ITA certification policies and specifically their policies regarding the TOEFL Speaking test. Seventeen participants responded to the survey. Participants responded from overwhelmingly research-oriented institutions, with only one participant reporting his or her institution as more teaching oriented. Most respondents were from large research institutions. The institutions very often used the TOEFL Speaking score as a prescreening measure to exempt high-scoring students from an in-house performance test. Nine institutions (of seventeen) implemented the TOEFL Speaking test in this way, with cut scores ranging from 23 to 28 points. An example of a typical response was the report on the policies of Purdue University. Purdue accepts scores of 27 or higher on TOEFL Speaking as evidence of ITA language competence. Students with lower scores must take an in-house teaching performance exam to be certified. Oklahoma State University, another large public research university, reports the same policy (a high TOEFL Speaking score exempts students from the local performance test) but with a cut-off score of 26 instead of 27. Both participants reported the cut-off score “working well” as an initial measure. Cornell University has an identical policy but requires a 28, based on a perception that lower scores are “all over the place” but that a very high TOEFL score is a reliable indicator of ITA proficiency. Only five survey participants reported SPEAK test use, and only two of these relied exclusively on SPEAK to make these decisions, with the other three SPEAK users also accepting TOEFL Speaking scores. Since the SPEAK test has been the primary tool used for ITA assessment over the past two decades, this may represent a fairly major change. One institution reported interest in moving from SPEAK to the Versant English Test, a fully automated computer-scored oral assessment that has been the subject of much debate in the testing literature during the past decade. Respondents to the survey reported mixed impressions of TOEFL Speaking use in practice. Some respondents, who utilized the scores, reported that “it seems to work” or “as an initial measure, it works well,” whereas others reported that the TOEFL did not measure the appropriate skills. For example, one participant reported: The iBT cannot replace our test since the iBT does not look at teaching skills, awareness of U.S. classroom, ability to use their language skills to successfully convey information to learners. However, based on our own analysis of past ITA tests, we now allow students with iBT ≥ 26 to be tested by just one rater rather than 4. Other respondents reported varying degrees of satisfaction with and confidence in TOEFL Speaking scores, saying that scores in the middle range are less useful as predictions of ITA communicative success. For example, one participant said, “We see that there are correlations at the higher levels of the iBT with oral proficiency tools such as the OPI. Anything below a 24 is all over the place.” Overall, the survey results indicate that TOEFL Speaking is in fact widely used to make these decisions. Of course, the practical advantages of using TOEFL iBT Speaking scores for ITA certification will be obvious to any ESL or testing program coordinator; TOEFL scores will in most cases be already available from the institution’s admissions department, and elimination of in-house performance testing could save substantial resources. One major advantage of using TOEFL iBT Speaking would be that incoming students and departments could make ITA decisions in advance of student intake. Clearly, though, not enough is known about how using these scores instead of an in-house measure may impact programs, students, and departments. All but one (Lim et al., 2012) of the TOEFL-specific studies described in this paper rely on experimental testing, and operational TOEFL Speaking scores may well have properties quite different from those derived from experimental studies, due to practice effects or other issues. In terms of practical recommendations to ITA program coordinators, the following may be tentatively concluded. The TOEFL Speaking test seems to measure the language needed by ITAs to a certain extent, enough so that very high iBT scores may be useful for ITA certification or that very low scores might be sufficient evidence to prevent candidates from teaching. There is no definitive answer as to the ideal cut score, however, and different institutions may decide on more lenient or more stringent cut scores depending on demand for ITAs, local resources, and other factors. Available research in addition to the anecdotal evidence gathered from the survey indicates that cut scores of 26 or higher seem to minimize the danger of false positive classifications (candidates who pass but are not truly qualified). A cut score of 28 would probably result in very few false positives (Xi, 2007), but relatively few candidates are likely to reach this high bar. The limited available research and anecdotal evidence suggest that scores in the range between 22 and 25 do not predict ITA success with sufficient accuracy (Lim et al., 2012) and local performance measures should be used. References
|
|||||||||||||||||||||