The Edmond Sun

September 9, 2013

State receives student test score study

Patty Miller
The Edmond Sun

OKLA. CITY — Recently the Oklahoma State Department of Education announced an independent study is complete to determine if student test scores had been compromised by the two days of disruption this past spring.

The HumRRO Statistical Investigation of Oklahoma Disruptions indicated that students appeared to do as well on the test as students not involved in the disruption. Based on the result of this study, State Superintendent Janet Barresi determined the state will retain all scores of impacted students who scored proficient or advanced.

However, many school districts, including Edmond, are expressing dissatisfaction with the testing company involved in the spring testing. Computer server issues caused most of the disruptions in April, which impacted 1,400 students in Edmond alone.

“We continue to be disappointed with the quality of services provided by the State Department of Education-approved testing service contractor, CTB/McGraw-Hill,” said Edmond Superintendent David Goin.

Goin stated a three-pronged reason for the basis of his dissatisfaction including:

• The timeline for receipt of results from last spring’s testing cycle has again been postponed,

• Even though the districts have preliminary results for these students’ performance, the vendor cannot now account for testing booklets used by some 258 Edmond students (apparently a circumstance affecting school districts around the state), and

• Numerous test invalidations were approved by the State Board of Education due to the vendor server issues last spring that affected about 1,400 Edmond students and could impact “growth” calculations for the 2014 testing cycle.  

“These circumstances are of great concern to us in that state-mandated tests come with high stakes for students, schools and communities,” Goin said. “They will affect students’ ability to be promoted and even to graduate; they will affect teachers’ performance evaluations, and they will determine in large measure grades assigned to schools and school districts, which in a broader context, will reflect upon the school community.    

“Valid and reliable testing results, timely reported, are essential in establishing the credibility of such a significant component of Oklahoma’s adopted education reform package. Anything less is a disservice to the state and its citizens.”   

While students as a whole did not experience depressions in scores, it is possible that some individual students did not perform to their highest potential during the period of disruption. Therefore, Barresi will not report the scores of impacted students who scored limited knowledge or unsatisfactory.

“Last year’s testing period was very difficult for our students, teachers and testing coordinators,” Barresi said. “The difficulties they experienced were unacceptable. It was a high priority for everyone to commission an independent study to determine what effect the disruptions had on the student test scores. Now that the study has concluded, we will work with the districts to take action on behalf of students and schools where the disruptions merit such action.

“Even though this study suggests no systematic impact on test scores, not reporting the scores of students who scored limited knowledge or unsatisfactory will ensure there is no lasting impact on student performance. This is the right thing to do for students and for schools.”

Server capacity problems by testing vendor CTB/McGraw-Hill on April 29 and April 30 caused a number of students to be interrupted during their testing experience, making it necessary to determine the extent to which the disruptions impacted students’ test scores. The State Department of Education asked HumRRO to conduct a third-party independent study to investigate the impact.

HumRRO was selected by the Department of Education because the organization is recognized for its objectivity and independent approach to evaluating the performance of testing vendors. They also have conducted similar work for the State Department of Education during a previous challenge with a different testing coordinator.

A state Technical Advisory Committee of researchers and assessment experts who meet twice per year to advise the State Department of Education on accountability and assessment issues recommended the design for HumRRO’s disruption study. HumRRO conducted the study independently of the Department of Education and CTB/McGraw-Hill.

Studies conducted for Indiana by another organization found similar results, as did a study conducted for Minnesota by HumRRO.

“The findings of the report in relation to the discrepancies did not merit action in HumRRO’s opinion,” Barresi said. “Because the two-day incident resulted in so much stress and chaos of the school day I wanted to assure students, teachers and schools were not held accountable for circumstances beyond their control.”

About 1,400 students received an immediate raw score that differed from the score they received on the two-week preliminary test score report.

The discrepancy occurred for two reasons: There appeared to be a short time period on the second day of the interruptions when some student responses to test questions were not saved and therefore not included on the two-week report.

CTB inadvertently left the “winter test system” as an option for download, which caused incorrect data to be saved.

Corrective action will include:

Students will receive the higher score between the raw score and the two-week report score. Students with higher raw scores will receive a letter from CTB indicating their proficiency level. They will be included in the report card, but not in the school or district summary report.

Students with a higher two-week score will receive a traditional score report that includes performance level and performance by content standard.

Additional testing

of second graders

raises questions

The State Department of Education accepted, as part of a settlement agreement with the vendor due to last spring’s testing snafu, a provision that will further expand tests administered by the vendor to now include second grade students, even though at this time the state does not test second graders, Goin added.

“This could mean more money for testing if, after the first year of testing second graders, the state decides to continue that policy,” he said.

Goin added administrators were informed during an Department of Education webinar that the department has engaged a five-year contract with CTB/ McGraw-Hill for testing in Oklahoma.