Qualifications : Meeting the Challenges of Change

Leadership in Qualifications : Meeting the Challenges of Change.

Tim McMahon Project Manager, Qualifications Development Group Ministry of Education The introduction of the NCEA was announced in late 1998.

It was seen then, as now, as enabling significant improvements in the senior secondary school qualifications system.

The improvements included the: * Introduction of a single, coherent, and inclusive qualification (on the National Qualifications Framework) to replace a confusion of types and styles of qualifications (most of which were not on the National Qualifications Framework).

* Recognition of the results of written examinations, along with internally assessed unit standards, within one comprehensive qualification.

* Provision of a wider range of learning pathways and subject choices for students, all leading to one qualification.

* Strengthening of linkages between the NZ Curriculum learning outcomes and standards to be met for the award of school qualifications.

* Delivery of more useful, accurate, and meaningful information about student achievement to whomever needed that information.

* Likelihood that the new system of assessment and qualification would, in the medium-term, decrease the high workloads of secondary teachers that have been generated by the current qualifications mix.

Two factors are seen as overriding in the design of any and all education reforms.

And they have been influential in our design thinking.

First is the nature and change of the economic and social pressures that face New Zealand as a small country caught up in what is at once a shrinking (due to influences like ICT) and expanding (due to the strengthening of second and third-wave economies) global society.

If we are to continue to make our way as a nation w are going to have to be highly skilled, extraordinarily innovative, and intellectually and practically adaptable.

The need to prepare for life long learning is not a cliché: it's a hard reality.

No less important is the alarming and persistent gap in education outcomes between identifiable groups in our society.

In particular Mäori and Pacific Nations peoples have been disproportionately over-represented among those for whom our system has not catered for very many years.

All education reforms at this point in our history need to be aligned in ways that aim to eliminate those gaps.

Continual systemic educational failure is a recipe for social disaster.

Further to that, the equality of esteem guaranteed by the Treaty of Waitangi has been seriously undermined by the differential outcomes of our system.

Adjusting the system to ensure that these 'gaps' are addressed does not mean shifting the goal posts or lowering our expectations.

Quite the opposite.

It means being very clear about the standards that all will have to meet, and ensuring that there's nothing inherent in our education and systems and process that create barriers to everyone's achieving them.

As the development phase of the NCEA draws toward a close, and the inevitability of implementation becomes more apparent, most in the sector seem to understand and welcome the changes.

Some are still worried and have raised issues that need to be addressed.

I want to deal with some of those issues.

Nature of qualifications Lack of understanding of the nature and purpose of qualifications goes to the heart of many of the debates and discussions that have surfaced recently.

The development of the National Certificate of Educational Achievement is producing a qualification that all senior secondary students can work towards.

The purpose of the NCEA is to provide opportunities for the diverse range of students in the increasingly wide variety of courses in our schools to have their achievements recognised and reported.

A qualification is not designed to reform teaching practices, change curriculum or to constrain schools in offering programmes that best meet the needs and interest of all their students.

Rather it is designed to recognise the achievements resulting from excellent teaching and to provide successful students with a record of their achievement.

Of course what we choose to value in qualifications will influence our teaching - that's an inevitability of the status that qualifications are accorded.

In this sense it is very important to ensure that the nature of our qualifications assessment is closely aligned to the outcomes of our best teaching practices.

The fundamental obligation on the government is to ensure that there is a qualification for New Zealand students that recognises the diversity of study and achievements across our entire secondary student population.

In developing this qualification we need to ensure that it motivates students of all abilities and interests to achieve their best.

It is important that our qualifications are both inclusive and encourage excellence.

* Inclusiveness is reflected in the fact that the qualification will allow a wide range of learning to be reflected in the credits that build up to the overall qualification.

* Excellence is encouraged by stimulating students to extend their learning to reach the highest levels of achievement.

The NCEA with its three levels and three grades of achievement for every standard will provide learning challenges for all students, from those who will find level 1 demanding, to the potential scholarship winners.

This may be analogous to university degrees where levels of achievement can be recognised in the "standards" demonstrated by those who aim to get a bachelor degree to those who strive to gain first class honours.

Reporting The issue of reporting is at the core of the purposes of a qualification.

Qualifications not only attest to learning but also provide useful information about what has been learned.

One of the fundamental things we are doing in the NCEA development is to enable the students or school-leavers of 2002 and beyond to be able to show the world in some detail what they know, what they can do, and how well they know and can do it.

This is not radical and shouldn't be alarming.

To judge from some reactions you'd think this proposal was wizardry of Harry Potter proportions! Mathematics has never been a national pursuit for New Zealanders, so it's astounding how much in awe we seem to be of aggregate percentage reporting.

Maybe part of this awe is the suspicion that qualifications are not supposed to be understood.

But the whole purpose of a qualification is exactly that it should be understood.

The reality is that different people, be they employers or tertiary providers, will interpret a student's qualification with different objectives in mind.

* For example, different tertiary providers will want to look at a record of learning to see whether it indicates that the student has the grounding to enter a tertiary level course in medicine, engineering, or dance.

Some courses require only a general indication of aptitude, others a much more specific background.

The NCEA will provide this level of information.

* The overseas student would want to be able to show to a Malaysian University, for example, that he or she has achieved in the areas that are pre-requisites for entry at that institution.

* A potential employer might want to know how well a school leaver had done in, say, number and computer skills, and oral communication.

Principals at a recent secondary sector forum saw that the detail available in the proposed way of reporting would provide them with much better information to help guide students into the best learning pathways.

For example, at present they face difficult judgements to make when advising the student who got 44% in School Certificate science whether to undertake a year 12 course.

In 2002 they will be able to look at the record of learning and see that that same student had earned a merit grade in biology, a credit grade only in chemistry but nothing in physics standards.

They could then be much more confident about advising the student to advance in biology but not undertake physics at the higher level and to think very hard as to whether or not they could advance in chemistry.

An employer will be able to hire a retail assistant not having to worry that 53% in mathematics might well mean that the employee is wonderful at creating and working with geometric patterns, but can't calculate correct change for toffee! The NCEA report will expose those relative strengths and weaknesses.

At the risk of being accused of trying to hide behind a professional mystique ("we know what 65% means - trust us"!) some in the sector have suggested that the proposed NCEA reporting will provide too much information.

These people say "Users don't want that much information - they won't understand it.

" I'm aware of businesses that failed because they didn't have enough information.

I know of managers who have regretted hiring staff about whom they wish they'd known more.

I've never heard an employer complain that she hired the wrong person because she knew too much about them! The proposed method of reporting grades and credits by standards may be aggregated to a simpler number, by subject or group of standards where that is useful.

The reverse is not the case.

If you want to know whether a potential candidate is a competent formal writer, and the only information you have is English: 48%, there is no way of disaggregating that number to get the information you need - information which, ironically, had to be collected in order to be aggregated in the first place! Some people say you can't compare students in the new scheme.

That's false.

Indeed as the example in the appendix shows, comparisons are not only possible but much more meaningful.

As a result of our experience on the recent roadshows, I am convinced that when parents and employers know that this level of detail is possible and available, they will demand it.

Breaking up of subjects Related to the issue of reporting by outcome is the allegation that this somehow destroys the holistic integrity of subjects.

This argument is false in as much as there's no necessary connection between the reporting system and the way the subject is packaged and presented for teaching purposes.

Teaching should always be organised in ways that encourage the most effective learning of what needs to be learned and reported.

It would be disingenuous to ignore the influence that high stakes student assessments have on the behaviour of teachers.

Teachers do "teach to the test".

They do it now and it would be naïve to suggest that this behaviour would change in the NCEA environment.

But some questions are begged.

Isn't it true that all subjects have component parts whose relative importance is a question of priorities? How are the traditional subjects taught now? If I look at subjects like mathematics or physics, I'm aware that there are conventionally regarded topics that a course should cover.

Maths should cover number, measurement, algebra, trigonometry, statistics and calculus for example.

Or should it? In the United States they don't do calculus until 'college'.

In some countries statistics is not considered part of mathematics.

The same is true of physics.

In recent years in NZ schools advanced study of heat has given way to study of nuclear physics (without explosive consequences!) This doesn't diminish the integrity of the subject or the nature of the topics, it just suggests that we have different and changing priorities about which parts of a subject we believe youngsters should be exposed to.

When I taught, we organised our curriculum into topics.

While connections between topics were made when it was necessary to make sense of the current topic, we nevertheless taught, say, graphs and equations sometimes for a week or two on end.

We did some testing on that, and then moved on to, say, geometry.

The reality is that teachers do focus teaching on particular outcomes.

The fact that we chose to report by lumping all that detail into an aggregate mark did not, of itself, give a subject holistic integrity.

Whether students will understand and appreciate the nature and internal integrity of a discipline is a question of the depth of understanding that teachers themselves have and the connections that they are able to help the students make in their programmes.

The assessment and reporting of sub-outcomes cannot and does not destroy that.

An experiment? Some of the most vocal critics of this development have tried to argue that standards-based assessment has never been tried; that we are subjecting New Zealand students to an 'experiment'.

Sadly, to back that argument up, the critics are characterising 'standards-based-assessment' in ways that bear no relationship to the achievement standards of the NCEA, and ignoring the kind of classroom practice we all take as common-place.

In this country, and overseas, we have long set objectives for lessons or lesson sequences, then taught and tested the outcomes of those lessons.

We have long used the notion of 'impression marking'.

It is only in a very few subjects where teachers have had the luxury of being able to add up ticks that were assigned on a straight yes/no basis.

Ask two people who have marked School Certificate English and School Certificate mathematics to compare notes.

(They may have common ground on marking multi-choice questions!) Markers have always had to make judgements on the basis of agreed or learned criteria.

Examiners' meetings have identified the key things that should be in the answers to questions and teachers have marked against a schedule that laid out what points should be made and so on.

Marking of achievement standards will not change that.

Indeed, the purpose of the professional development programme that's currently underway is to engage regular classroom teachers in the kinds of sharing of best practice and expectations of student achievement that are routine for teachers who have been and who will be involved in marking external examinations.

Dramatic? The Education Forum describes the NCEA as "a dramatic and far-reaching reform".

I don't want to diminish the importance of what the government has set out to do by trying to suggest that the change is trivial.

It isn't.

But whether you see this change as being earth shattering or merely sensible evolution depends, I suggest, on whether you see any change at all as disruptive or whether you see improvements as worthwhile; whether that which is in front is described as a problem to be avoided or a challenge to be met and managed.

We currently have examinations.

In those examinations we assess topics of the curriculum.

We're not very clear to students (or teachers - watch the hue and cry when a new examiner reinterprets the prescription!) precisely what we're after in that examination, but nevertheless we examine the topics.

We aggregate the 'marks' that are counted or allocated to each of those topics.

In the new system we are attempting to explain to all concerned, ahead of time, what we'll assess and what expectations we have.

And we propose to report the topics individually.

In the current system we have internal assessment of some parts of most subjects.

Here too the change is that we will have pre-announced the standards we want demonstrated.

And we'll report them individually.

A bigger change, I suppose, is that we also want to assess a few things that are in the curriculum that we expect to be taught and learned, but which haven't figured (validly anyway) in what we've previously assessed for qualifications purposes: like oral language in English; like laboratory work in chemistry.

Nevertheless these are things that we routinely teach and assess for local reporting purposes.

Is it such a big challenge to assess and report these for qualification purposes? When viewed through these lenses the changes we're proposing do not seem threatening.

If we have the will, the challenge of assessing and reporting against the criteria in the standards will be easily met.

While these changes will have far-reaching implications, I suggest that the evidence suggests these are evolutionary (and necessary), not revolutionary.

Moderation Many in the sector are concerned about moderation.

That's understandable.

Any modern qualifications system is going to have to report on student performance and on the basis of tasks that require extended time or specialist equipment to carry out.

These reports are going to have to be based on local 'internal' assessments.

There are obvious professional implications for the classroom work of a teacher who is playing two very different roles.

In helping a student to do as well as possible in external examinations, the teacher is the student's critical friend.

In 'internally' assessing the student for external certification, the teacher is the student's judge.

Balancing these roles is a matter of fine professional judgment.

Some who oppose internal assessment in principle, regard the moderation issue as a fatal flaw since it involves a classic 'Catch-22'.

Without moderation, it is argued, the system must fail because "Nobody will trust this".

The argument then continues along the lines that the level of moderation required to gain and retain trust will be intrusive and will impose an intolerable compliance burden.

This argument raises a number of issues.

The first, and most obvious, is why the professionalism of teachers in making and reporting local judgements of students' performance is questioned in this way.

All through primary schools and the first years of secondary, teachers routinely assess for both formative and summative purposes.

Their reports are accepted by parents.

'Ordinary' secondary teachers mark the external examinations.

Almost all qualifications assessment at undergraduate level in tertiary institutions is done internally; and without question.

One would have thought the stakes are even higher at this level than at school.

But the issue of trust seldom arises and where aberrations are discovered they are usually quickly and professionally dealt with.

There is a simple issue of professional trust and accountability in this situation.

If this challenge is worth meeting in the secondary sector, as it has been in tertiary, it is a matter of determining a method of professional discipline that would deal with miscreants in a way that demonstrates and demands professional accountability.

The work of the proposed Education Council will be helpful here in the longer term.

The Education Forum, while accepting that some internal assessment is inevitable, argues that it will be acceptable only if its results are moderated against external examinations.

While in some places such an approach is used (we ourselves have used written 'reference tests' to provide mark ranges for internal assessment results) there are obvious and serious flaws.

First, the process lacks validity.

The skills required to perform a written examination are quite different from the skills required to carry out a precision experiment or deliver a speech in public.

No one would think of altering a decathlete's 1500 metre time on the basis of his ranking in the shot put! You can't validly moderate the quality of chalk by the taste of cheese.

Second, it assumes that the externally examined component is totally reliable.

Cedric Hall and many others have shown that the reliability factor for external written examinations is around 0.

85.

According to Cedric Hall the best that can be said of a student with a mark of 50% is that he would almost certainly score in the range 39% - 61% if tested again.

The idea of adjusting an internally assessed result where evidence may have been collected over a number of observations, and which may therefore be more reliable than the examination mark, seems questionable at best.

Third, if the internal assessment result is to be adjusted in proportion to the external assessment result the external result is, in effect, all that is reported.

That renders the act and intention of carrying out internal assessment useless.

Notwithstanding the tension involved in being simultaneously the student's friend and the student's judge, the best way to ensure satisfactory internal assessment is to ensure shared expectations of student achievement standards among teachers, and professional accountability for the judgments made.

As part of the NCEA development, modeled on numerous international examples, we have chosen to try and bring all teachers together in professional interest groups to compare their expectations of student achievement and to defend their interpretation of student work.

Looking beyond the stage of initial implementation, we want to see on-going regular meetings of local groups at which teachers can compare and discuss the work of their students.

Teachers who have worked with this method speak of its great value, both as on-going professional development, and for establishing that tacit or informal knowledge which is needed for fashioning shared standards of judgement.

With such a arrangement set up on a regular basis, the system will be more robust and better able to evolve through difficulties, to provide system feedback, and to deal with future change.

Many teachers who've been involved in external examination marking here will say that it's the best form of professional development they have.

Arguing about acceptable responses has been the greatest influence on their understanding of national standards.

One-size-fits-all?.

One of the strongest attacks mounted by the Education Forum is that the NCEA is a 'one-size-fits-all' qualification.

The NCEA, it is alleged, is trying to reduce all qualifications to an egalitarian mish-mash bringing down the rewards for the academically able by forcing them into the same qualification as those who are pursuing vocational pathways.

There are several flaws in this argument.

The philosophy behind the NCEA is that we should recognise and reward diversity of achievement; not to encourage sameness.

That's the opposite of one-size-fits-all.

By design it is inclusive of the complete range of options that are available to our secondary students.

The whole concept of knowledge-value-added in our modern economy means that almost all workers need to be skilled decision-makers.

They need to understand the processes that are operating.

New areas of skill, such as electronics and computing, are at once academic and vocational.

Their learning and assessment is both demandingly theoretical and intensely practical.

Increasingly it will be necessary for our qualifications to recognise the achievements of students who demand a much greater degree of 'mix and match'.

Not one-size-fits-all, but catering for every possible diversity of demand.

It's true that the learning that underpins traditional academic courses, and industry driven vocational courses may be different.

But to argue, as some do, that academic qualifications and vocational qualifications are of different intrinsic value is to risk ignoring the reality of the modern world.

The days of simple, clear-cut, divisions between academic education for professionals and guild training for tradespeople have gone with the change, even disappearance, of the occupations for which they were conceived.

The only things that a student who chooses to pursue a traditional academic course assessed against NCEA achievement standards and a student who pursues a course based entirely around industry driven unit standards need have in common, is the currency of the credits they may obtain, and the colour and signature on their certificate.

The courses will be different.

Their understandings, skill and knowledge will be different.

Their results notice and record of learning will be quite different.

To claim one-size-fits-all on the basis of the shape and size of the certificate is a sensible as arguing that all work written on A4 is indistinguishable.

Some have argued that the countries which perform most successfully in international tests such as TIMSS, are those countries that have a strict academic/vocational pathway split in their secondary education system.

As these international tests are applied to primary level education, before the split is enacted, it is clearly not the qualifications pathway divide that is responsible for differences in TIMSS scores.

As a learning society, we need to discover what it is about countries like Singapore, and their traditions, that mean their students are so well motivated to learn.

But reinventing a system that drafts students into exclusive pathways before many of them have matured intellectually or emotionally is not a way forward.

Workload Reducing the workload involved with administering the current dual system was an important driving force in bringing this reform forward.

There seems little doubt that by removing Sixth Form Certificate and replacing at least half of the internal assessment in conventional sixth form subjects with external assessments for the Level 2 NCEA, we will succeed in reducing the assessment load on students and teachers at that level.

The new processes proposed by NZQA streamline the administration of moderation to such an extent that it seems realistic to conclude that workload in that area will decrease too.

There will be transitional work associated with the change.

It would be unrealistic for anyone to claim otherwise.

What is important is that in the steady state we will have a system that allows teachers to concentrate more on teaching and less on qualifications assessment and its associated administration.

John Taylor of Kings College has often lamented to me the encroachment of assessment and administration time onto the time that teachers can spend with students in extra-curricular activities.

I agree with that.

This qualifications reform will provide for better, more comprehensive, and more thorough assessment and reporting of students' learning.

But bookwork is not the sole purpose of adolescence.

Youngsters have so much more to learn.

If there really are areas where the new system threatens to increase workload for teachers who are already doing a good job, we need to identify them and address them as soon as possible.

Workload for APs and DPs is an obvious case in point.

While we believe that once the change phase is through, the new system will be administratively easier than the current system, if evidence emerges that this is not the case we will need to do something about it immediately.

We cannot put at risk a system that will be better for students, for lack of necessary support for those who have to make it work.

Where to from here While the NCEA is being designed to encompass the increasingly wide range of subjects and programmes offered in schools it does not prevent a school from having students working towards another qualification such as the National Certificate in Tourism or National Certificate in Employment Skills, or sitting someone else's examinations: perhaps those of another country or an international organisation such as the Trinity College.

We have a history of schools offering British qualifications alongside our own - like the Pitmans typing certificate.

Proposals to prepare some students for the International Baccalaureate or the Cambridge 'A Levels' need to be seen in this light.

While it is our responsibility to ensure that a credible qualification exists for all New Zealand school leavers, we accept that for some students it may well be useful to work towards other qualifications too.

Students should not be prevented from doing this nor should schools be prevented from offering such opportunities.

The great advantage that the NCEA will have over such imported examination-based qualifications, is the extent of its coverage and obvious relevance to New Zealand.

I would hope that the students in those schools that are seeking to differentiate themselves by their choice of alternative qualifications do not become unwitting victims in someone else's battle.

If those students leave their school without their oral language ability reported in their qualification, without their computer skills or laboratory or research skills reported in their qualification, or with a better understanding of Britain's relationship to Europe than of New Zealand's to Asia, they may find themselves less well served than those who have opted for the NCEA.

Meeting the challenges In the later evolutionary stages there will be challenges that will not be as easily countered as those I've addressed today.

Some will be achieved only when our whole cultural view of the value and status of secondary school qualifications and how they're attested has evolved too.

For example, to improve both reliability and validity in assessment it may at some stage be necessary to investigate the degree to which an element of internal assessment can be added to each externally assessed standard.

In the immediate future it will be necessary to monitor the impact of high stakes processes carefully to ensure that perverse 'back wash' effects do not affect quality teaching or learning opportunities.

It will be necessary to continually monitor the use and impact of the achievement standards both one-by-one, and in 'subject groups', to ensure we pick up every possible improvement as soon as possible.

Some of the challenges this development faces are huge issues that countries much bigger and with longer histories than ours, have grappled with and made little headway.

Because in some places they have given up and gone backwards, as the Education Forum alleges of Victoria, Australia, is no reason for us to do the same.

It is critical that every legitimate challenge to a system as important as secondary qualifications is heard and addressed.

That means listening to, and trying to analyse, each and every challenge.

As Hargreaves and Fullan say, in dealing with change, it is important to "walk towards the danger".

Only by approaching the challenges head on can we be sure what motivates the critic; what are real and what are imagined challenges; what issues need to addressed and accommodated by the government as developers, and what challenges we need to help others to meet.

This development has been consistently fine-tuned over the past eighteen months in consultation with all key stakeholders and will continue to be.

The NCEA will be decidedly better than the incoherent and unmanageable system we currently have.

But none of us will be satisfied, nor should we be, until our school qualifications system is all the things we are aiming for: manageable; coherent; inclusive; meaningful; valid; reliable; in tune with the learning and employment needs of the 21st century.

We are almost there.

When we have achieved those goals New Zealand will have achieved what few if any countries in the world have been able to do for their school students.

Appendix Comparing students Standard Using Algebra Drawing and using graphs Measurement Geometry Data and statistics Probability Solving number problems Trigonometry Assessed eXternally or Internally X X I I I X X X Credit value if achieved 3 3 4 3 3 2 4 2 Jason E M M C M E E M Dean C C C C C M Jason and Dean easily distinguished - and would get different grades by any report

Mere E M E C C C C M Nan C C C E M M E C Mere and Nan easily distinguished - but would get very similar grades by aggregate report

© Scoop Media