ITAIS Newsletter - July 2016 (Plain Text Version)
|
||
|
In this issue: |
CAN TOEFL IBT SCORES PREDICT ITA TEACHING PERFORMANCE?
Many university ITA programs use the speaking section scores of the Test of English as a Foreign Language (TOEFL iBT) as part of the process of determining if ITAs have the oral proficiency necessary to be classroom instructors, yet there are actually very few studies that have investigated the validity of using TOEFL iBT scores for ITA screening purposes. This is problematic, especially considering that the TOEFL iBT was designed (and validated) as a measure of a student’s academic language proficiency. The types of language ability that a university instructor needs might be quite different from the types of language ability that a student needs. This lack of research was the impetus for a 3-year long, Educational Testing Service grant-funded study (Educational Testing Service developed and administers the TOEFL) that I recently completed, and that was published in May, 2016: “A Study of the Use of the TOEFL iBT® Test Speaking and Listening Scores for International Teaching Assistant Screening” (Wagner, 2016). There were three main parts to the study. Part 1 examined whether ITAs’ TOEFL iBT Speaking and Listening scores correlated with and predicted their teaching competence, as measured by official end-of-semester evaluations. Part 2 examined whether TOEFL iBT Speaking and Listening scores correlated with and predicted the ITAs’ teaching competence, as measured by two additional measures of teaching competence. Part 3 examined the extent to which ITAs’ oral proficiency (as measured by TOEFL iBT Speaking and Listening scores) improved over their first semester as ITAs. Part 1 For part 1 of the study, TOEFL iBT Speaking and Listening scores were correlated with and used as predictors of ITA teaching competence, as measured by student feedback forms (SFFs). SFFs are the end-of-semester official university student evaluations of their instructors, used at Temple University and similar to the teaching evaluations used at most North American universities. The results indicated that TOEFL Speaking and Listening scores do not correlate with the SFFs, and are not good predictors of teaching competence (as measured by the SFFs). This was not particularly surprising, because oral English proficiency is a necessary but not sufficient condition for teaching competence, and thus TOEFL scores by themselves cannot be expected to be strong predictors of teaching competence. Part 2 While end-of-semester student evaluations (the SFFs) are widely used and accepted at North American universities, they have a number of serious shortcomings. Numerous researchers have documented student biases and other problems associated with teaching evaluations like SFFs, and so it was decided to use two alternative measures of teaching competence, as well: · students’ assessment of ITA teaching competence, and · observers’ assessment of ITA teaching competence. My assistants and I went to 33 different ITA classrooms to observe and assess each ITA’s teaching. The research team observed each ITA two times during the semester. It also gave the students in the class an eight-item questionnaire to evaluate their ITAs’ teaching competence. This questionnaire was designed to avoid some of the shortcomings of the standard SFFs. For part 2 of the study, TOEFL iBT Speaking and Listening scores were again correlated with and used as predictors of teaching competence, as assessed by these two measures. The results showed that, similar to the results of the first part of the study, TOEFL iBT Speaking scores do not correlate with or predict teaching competence. However, the results for the TOEFL iBT Listening scores were markedly different. TOEFL iBT Listening scores did correlate with and predict ITA teaching competence, as measured by the students’ assessment of teaching competence, and the observers’ assessment of ITA teaching competence. In other words, those ITAs who scored higher on the TOEFL iBT Listening tended to get higher ratings on these alternative measures of teaching competence. While these results might seem somewhat surprising, they probably should not be. Listening ability is obviously an important component of teaching ability (e.g., Elder, 1993; Plough, Briggs, & Van Bonn, 2010). A good instructor needs to be able to speak English well, but he or she also needs to be able to understand spoken English in order to lead class discussions, respond appropriately to student questions, and interact with students. In addition, ITA studies have found that undergraduates report being most frustrated by ITAs’ (perceived) inability to understand student questions (Myers, 1994; Plakans, 1997). A clear implication for ITA programs that use TOEFL scores for ITA screening is to use both TOEFL iBT Speaking and iBT Listening scores. This is an important finding, considering that that most ITA programs that use TOEFL iBT scores for ITA screening appear to use iBT Speaking scores only (Farnsworth & Wagner, 2013). From a theoretical perspective, it is logical to include TOEFL Listening scores as well as TOEFL Speaking scores, because the oral ability required for good teaching includes both speaking and listening ability, and this study provides empirical evidence supporting the use of TOEFL iBT Listening scores for ITA screening. Part 3 The third part of the study examined the extent to which ITAs’ oral proficiency developed over a semester in an English-speaking environment. There seems to be an expectation that being immersed in an English-speaking environment, and taking (and teaching) classes in English will lead to marked gains in ITAs’ oral proficiency. That is, many ITA programs require prospective ITAs who do not meet the required English proficiency cut scores to take a semester-long ITA training class, with the expectation that the instruction in the class, as well as the immersion setting, will lead to improved oral proficiency. Yet there do not appear to be any studies that have investigated how much ITAs’ oral proficiency really does improve after a semester, and thus it was decided to assess this expectation. A total of 84 ITA participants took a research version of the TOEFL iBT Speaking and Listening both at the beginning and at the end of their first semesters as ITAs. The results showed that the group scored slightly higher (.65 points on a 30-point scale) on the end of semester TOEFL iBT Listening scores than they did on the beginning-of-semester test, although this difference was not statistically significant. In contrast, the group scored .99 points (again, TOEFL iBT is on a 30-point scale) higher on the end-of-semester TOEFL iBT Speaking scores than they did on the beginning-of-semester test, and this difference was statistically significant. While these gains are relatively small, these results are similar to other studies (e.g., Derwing, Munro, & Thomson, 2008; Freed, 1995; Elder & O’Loughlin, 2003; Segalowitz & Freed, 2004) that found that development in oral proficiency is surprisingly small, even in an immersion setting. Language learning is a long process, and the time between the two tests was relatively short (3 months). In addition, it seems probable that some of the language abilities that the ITAs are learning in their ITA training class and their other courses, and in their oral interactions with English speakers on and off campus, are language skills that are not measured by the TOEFL iBT, and thus their scores on the TOEFL iBT might not completely measure the improvements in communicative competence they made over the 3-month period. “A Study of the Use of the TOEFL iBT® Test Speaking and Listening Scores for International Teaching Assistant Screening” is freely available on the Wiley Online Library. References Derwing, T., Munro, M., & Thomson, R. (2008). A longitudinal study of ESL learners’ fluency and comprehensibility development. Applied Linguistics, 29, 359–380. Elder, C. (1993). Language proficiency as predictor of performance in teacher education. Melbourne Papers in Language Testing, 2(1), 1–17. Elder, C., & O’Loughlin, K. (2003). Investigating the relationship between intensive English language study and band score gain on IELTS. In R. Tulloh (Ed.), IELTS research reports (Vol. 4, pp. 207–254). Canberra: IELTS Australia. Farnsworth, T., & Wagner, E. (2013, March). Using TOEFL iBT Speaking for ITA screening: Promises and perils. Paper presented at the annual meeting of TESOL International Association, Dallas, TX. Freed, B. (1995). What makes us think that students who study abroad become fluent? In B. Freed (Ed.), Second language acquisition in a study abroad context (pp. 123–148). Amsterdam, The Netherlands: John Benjamins. Myers, C. (1994).Question-based discourse in science labs: Issues for ITAs. In C. Madden & C. Myers (Eds.), Discourse and performance of international teaching assistants (pp. 83–102). Alexandria, VA: Teachers of English to Speakers of Other Languages. Plakans, B. (1997). Undergraduates’ experiences with and attitudes toward international teaching assistants. TESOL Quarterly, 31, 95–119. Plough, I., Briggs, S., & Van Bonn, S. (2010). A multi-method analysis of evaluation criteria used to assess the speaking proficiency of graduate student instructors. Language Testing, 27, 235–260. Segalowitz, N., & Freed, B. (2004). Context, contact and cognition in oral fluency acquisition: Learning Spanish in at home and study abroad contexts. Studies in Second Language Acquisition, 26, 172–199. Wagner, E. (2016). A study of the use of TOEFL iBT Speaking and Listening scores for international teaching assistant screening (TOEFL iBT Research Report No. 27). Princeton, NJ: Educational Testing Service. http://dx.doi.org/10.1002/ets2.12104 Elvis Wagner is an associate professor of TESOL at Temple University. His research focuses on the teaching and testing of oral language ability. |
|