National Standards: Monitoring & Evaluation 2011

National Standards: School Sample Monitoring & Evaluation Project 2011

Publication Details

This is the fourth report from the National Standards School Sample Monitoring and Evaluation Project 2010-2013, a three year project on National Standards implementation in a representative sample of schools.

Looking at the extent to which National Standards were operating as intended in 2011, the project focused on Overall Teacher Judgments (OTJs), reporting to parents, reporting to the board of trustees through student achievement targets and identifying students for intervention. In mid-2011 the project team interviewed principals, and collected copies of schools’ student achievement targets and analysis of variance reports. At the end of 2011 they collected OTJs for students from the sample schools, as well as a sample of students’ end-of-year reports. There were also surveys of groups of teachers, principals, and board of trustees' chairpersons. The teacher survey used assessment scenarios to collect information about teachers’ judgments using the National Standards.

Author(s): Jenny Ward and Gill Thomas, Maths Technology Limited

Date Published: September 2012

NationalStandardsSchoolSampleMonitoringandEvaluationProject2011.pdf

Executive Summary

Information was collected from a stratified sample of 100 schools, representative of the population of schools in terms of school decile, school type and geographic region. Six main types of data were collected at two time points. In the middle of the year principal interviews were conducted and copies of schools’ student achievement targets and analysis of variance reports were collected. At the end of the year, OTJs were collected for all students, and copies of students’ end-of-year reports were obtained. Online surveys of teachers, principals, and Boards of Trustees Chairpersons were also conducted and information about teachers’ judgments in relation to the National Standards was collected using assessment scenarios.

Analysis focused on describing and evaluating the extent to which National Standards was operating as intended, and was based around specific monitoring and evaluation questions and performance criteria.

Overall Teacher Judgments

• Teachers used a range of information sources to make OTJs in reading, writing and mathematics. Most of the information sources identified by teachers as important in making OTJs were considered to be relevant to the National Standards.

• The sources of assessment information rated as most important by teachers included specific class observations in reading, writing, and mathematics, instructional text levels in reading, the collection of samples in writing, and GloSS and IKAN assessment results in mathematics.

• In an increase from 2010 results, approximately two-thirds of teachers can be considered to have used current assessment evidence to inform reading (68%) and writing (61%) OTJs, while just under half (49%) used current evidence to make mathematics OTJs. The remainder used evidence than was more than 12 weeks old.

• Approximately one-third of teachers took up to ten minutes to make one reading (39%) or writing (33%) OTJ, while just less than two-thirds (59%) were making mathematics OTJs in this time. This was considered to be efficient.

• Teachers and principals reported high confidence levels in both the accuracy and consistency of their school’s OTJs.

A variety of processes were used to moderate OTJs.

• Most schools used school wide moderation processes in writing (83%) and mathematics (90%), while about two-thirds of schools (67%) moderated reading OTJs. This is an increase from 2010, especially in mathematics, and results suggest schools tended to carry out formal moderation in writing in 2010, and extend this to mathematics in 2011.

• Approximately a third of schools used an efficient method of selecting OTJs for moderation by focusing on the judgments near the boundaries between the levels of the standards in reading (36%), writing (35%) and mathematics (30%).

• Thirty-six percent of principals indicated they had engaged in moderation practices with other schools. Writing was the area of focus for most between-school moderation.

The study collected information about teachers’ ability to rate individual pieces of student work in relation to the National Standards, and to collate several pieces of assessment evidence that had already been rated against the standards to make an OTJ. Student OTJ data was also used to provide information about the dependability of teachers’ OTJs.

• There was considerable variability in the accuracy of teachers’ ratings against the National Standards for individual work or assessment samples. In writing, accuracy ranged from 3% to 89% over the samples, while accuracy in mathematics ranged from 18% to 90%. This is a cause for concern as it is these individual judgements that are the basis of OTJs.

• Most teachers were able to collate four pieces of assessment evidence, each of which had been previously rated by experts, against the standards to make an accurate OTJ.

• Large positive shifts were observed for those students rated ‘below’ or ‘well below’ the standards in 2010. For example, approximately 60% of students rated ‘well below’ in 2010 received an improved rating in 2011. Given evidence from the assessment scenarios, and the magnitude of the changes observed, it is most likely the shifts in the data are attributable to teacher inconsistency in making OTJs.

• Aggregated reading, writing, and mathematics OTJs for 16,111 students were consistent with results from 2010. Demographic patterns in these data were in line with other evidence of student achievement in New Zealand, due to the large sample size that tends to cancel out random error in individual OTJs.

Reporting to parents

• Evidence suggests that nearly 90% of parents received an end-of-year report for their child that referred directly to the National Standards. Sixty percent of these reports were rated as sufficiently describing the child’s achievement in relation to the National Standards.

• Approximately 10% of reports that referred directly to the National Standards described children’s progress over time in relation to the reading (12%), writing (9%) and mathematics standards (9%).

• Fifty percent of the reports that described achievement in relation to the National Standards were rated as clear, that is, able to be easily understood by parents, families, and whānau.

• Sixty-eight percent of the National Standards reports identified the child’s next learning steps, while 55% included ways families can support learning at home.

Student achievement targets

• Seventy-five percent of schools included targets in their 2011 charter that addressed student achievement in relation to the National Standards.

• In terms of the nature of the students targeted, 94% of schools with National Standards targets focused on students who were below or well below the standards, while 6% included progress goals for all students.

• Fifty-seven percent of schools with National Standards achievement targets differentiated these to accelerate progress for specific groups of students. Thirty-three percent of schools included a focus on Māori students and 9% included a focus on Pasifika students. Other groups of students differentiated in National Standards targets included students with special needs (1%), boys (16%), and girls (1%).

• Most of the targets that addressed student achievement against the National Standards in reading (92%), writing (89%) and mathematics (88%) were specific and measurable.

• Of the targets that addressed the National Standards, approximately two-thirds addressed students at all year levels (59% reading, 67% writing, 60% mathematics), while over half were considered appropriate, i.e. both challenging and achievable (55% reading, 65% writing, 53% mathematics).

Identifying students for intervention

• Approximately three-quarters of principals collated school-wide National Standards data to describe student achievement in reading (78%), writing (77%), and mathematics (76%). In terms of using National Standards data to describe progress, around two-thirds had collated school-wide progress data (66% reading, 65% writing, 65% mathematics), and approximately 15% had collated progress data for some students (12% reading, 15% writing, 15% mathematics).

• About 85% of teachers reported tracking student progress in relation to the National Standards in reading (84%), writing (88%), and mathematics (86%) from the end of 2010 to the end of 2011 using OTJs.

• Just under two-thirds of principals indicated that they had used National Standards data to identify students for additional teaching support in reading (63%), writing (58%), and mathematics (63%). The interventions listed by principals included the provision of additional qualified teaching support, teacher aides, focused in-class teacher support, and the provision of additional learning programmes.

Perspectives of principals and Boards of Trustees

• Principals’ levels of understanding about the nature and intended consequences of National Standards had generally improved from the end of 2010 to the end of 2011.

• In general, principals felt more supported by the Ministry of Education in 2011 than in 2010, although more than half still described themselves as minimally supported or unsupported in nearly all aspects.

• Principals’ views over the usefulness of National Standards data were varied. Comments indicated both principals and Boards of Trustees felt they were already using data purposefully before the introduction of National Standards.

• Principals remain concerned over the unintended consequences of National Standards. Boards of Trustees share their concerns.

• Most Boards of Trustees feel they have a good understanding of the National Standards and what their school is doing to implement them. Most Boards are also confident their school is effectively implementing the standards.

ENDS

© Scoop Media