Test Cohort

The pilot tested a very small cohort of 24 out of 41 evening students enrolled in the above classes. Students had ample advance notice of the time and date of the test. The teacher informed them the week before and earlier in the test week of the test date.

Therefore, it is possible that the students who actually showed up for the test may have self-selected. Therefore, their scores may be higher that those of the class overall since they might have been more highly motivated and higher achieving students to begin with.

Estimated Time on Task

The teacher provided me with an estimate for each class of the average amount of time students spent on task out of the 40 minutes maximum allocated for the test. The teacher instructed students to stop as soon as they believed the questions were getting too difficult to answer. Nonetheless, some students continued past this point as their later answers indicate significant errors and random guessing.

The teacher estimated that ESL 1, 2, and 3 students took only 25-30 minutes before stopping. ESL 5 students took 35-40 minutes, attempting all 75 test questions.

Implications for Admissions Placement Testing—Recommended Testing Times at Admissions.

Despite the small test cohort, however, the time students actually needed to complete the test suggests that it is very possible to reduce testing time upon admissions.

In the time allocations recommended below, I have shaved off a third of the time from that CELSA recommends for the test. I believe that the CELSA time allocation is far too long, encouraging excessive guessing.

For students estimated to be ESL 1s or 2s

: The test has three distinct sections each with 25 questions. Based on initial interviews, admissions staff could administer the first section of 25 questions with a 10 minute time limit to students whom they estimate fall into the ESL 1-2 range. If they get all or most of the answers correct, they could then be given the next section of 25 questions also with a 10 minute time limit.

For students estimated to be ESL 3s or 4s

: These students could be given the first 2 sections of 50 questions for a total for 20 minutes. Only if they get all or most correct would they be given the last section for another 10 minutes.

For students estimated to be ESL 5s or 6s or TOEFL ready:

These students would get all three sections for a total of 30 minutes.

Pilot Test Scores May Be Inflated

For three reasons, the scores in this pilot may be inflated because of.

cohort self-selection,
cheating, and
guessing.

It is not unreasonable to assume that students who showed up for the test were more highly motivated and therefore higher achieving than those who did not. Also, there is some suggestion of cheating. For example, some students got few if any correct answers in the beginning and middle parts of the test and then suddenly got a cluster of “correct” answers deeper into the test especially on horizontal tops of answer columns that would be more visibly exposed on the answer sheets. Finally, there is evidence of guessing as the test questions became harder, a factor that could inflate scores by as much as 25%.

Test Validation

Despite the small test cohort and observations above about the possibility of score inflation, the pilot indicated that the CELSA does test progressive points of difficulty indicated by evidence of where students:

· started to get mostly wrong answers, ,

· started to guess at answers, and/or

· stopped when unable to go further.

More than actual student scores, this helps confirm that the test measures a progression of skills at different ESL levels.

Placement Implications

CELSA leaves it to the institutions to set cut scores. Based on the very small numbers of students tested, however, the results suggest some existing placement disparities.

For example, the scores of 50% or more of ESL 1, 2, and 3 students and some ESL 5 students greatly overlapped. Their scores fell into a similar range of 30-38 out of 75 correct answers, as follows:

ESL 1-66%

ESL 2-50%

ESL 3-50%

ESL 5-20%

In addition, the test scores of another 20% of ESL 5 students suggest they may have tested at an ESL 4 level.

	Teacher’s estimate of average student time on task	Scores	Estimated point student started guessing	Student voluntarily stopped at or after this question
ESL 1—6 of 10 students enrolled	25 min.	12, 19, 30	guessing throughout	46
		33	48 forward	74
		35	27 forward
		38		61

ESL 2—6 of 10 students enrolled	25-30 min.	4		7
		18	heavy guessing, possible cheating	51
		23		43
		34		43
		35		66
		38		51

ESL 3—2 of 6 students enrolled	30 min.	32	guessing after 40	60
		55	guessing after 59

ESL 5—10 of 15 students enrolled	35-40 min.	0-30 (wrote in answers that did not usually correspond to questions)
		38
		52
		57
		59
		61, 62, 63, 65, 69