Chi Duc Nguyen

Hanh Thi Hoang

VNU University of Languages and International Studies

Vietnam National University – Hanoi, Vietnam

Abstract

This study aimed to assess the construct validity of Pecorari, Shaw and Malmström’s (2019) Academic Vocabulary Test – Form 1 (AVT1). To this end, it first employed Rasch-based statistical evidence generated from the test responses of 989 high-school and university students in Vietnam to inspect five major aspects of Messickian construct validity: Content, Substantive, Structural, Generalizability and External (Messick,1995). It then moved on to thematically analyze the data collected from the follow-up focus-group interviews with 50 students randomly selected from the test-takers to detect any emerged patterns in their actual engagement with the test. Results from Rasch-based analyses showed that AVT1 could sufficiently measure the target ability – receptive academic vocabulary breadth – of an overwhelming 889 out of the 989 test-takers (90%) and there were seven statistically distinct groups of item difficulty in the empirical item hierarchy. In general, test items and test-takers performed as predicted by a priori hypotheses and displayed good fit to the Rasch model. Principal Component Analysis indicated that the test items formed a fundamentally unidimensional construct, suggesting that the test only measured one meaningful dimension, presumably the receptive academic vocabulary breadth. The evidence for the invariance in item calibration and person measure as well as the test’s external reliability enabled the generalization of score properties and interpretations across populations, settings, and tasks. Results from the interview data analyses revealed that AVT1 did allow some room for guessing effects, which was predicted by the test developers. These findings altogether suggest that this test can be a useful tool to gauge receptive academic vocabulary breadth.

Keywords

Academic Vocabulary Test, breadth of academic vocabulary knowledge, test validation, Messickian construct validity, Rasch-based analysis