Iterative Best Evidence Synthesis: Strengthening Research, Policy and Practice Links to Improve Outcomes

Published: Thursday, 18 May 2006
Presented at: 4th Annual Policy
Evolution Conference

 

The Iterative Best Evidence Synthesis (BES)
Programme is a collaborative knowledge building approach across policy,
research and practice. The Iterative BES Programme synthesises and
explains evidence about what works for diverse learners. The primary
purpose of the programme is to support sustainable educational
development whereby a whole education system and its communities
strengthen a range of desired outcomes for all learners through
iterative processes of shared knowledge building and use. The iterative
approach is designed to be a collaborative tool and catalyst to
intensify and embed the interplay of research and development (R &
D) as a systemic lever for sustainable development in education.

The
programme is an innovation that grew out of work in the Medium Term
Strategy Policy Division of the Ministry of Education and has been
formally established within Medium Term Strategy for two and a half
years.

The Iterative BES Programme has been one of a number of
case studies considered in a series of OECD meetings focussed on
evidence-based policy research. This paper builds upon two earlier
papers prepared for OECD meetings describing the rationale for the
programme and its brokerage role across policy, research and practice.
The first paper in the series was for the 2004 joint OECD/United States
Evidence Based Policy Research Conference . In that initial paper,
because of the marked differences between our approach and federal US
approaches, considerable attention was given to the rationale for our
realist and fit-for-purpose methodological approach to synthesising
bodies of evidence. That paper explained for BES development: the
importance afforded local context, the rigorous pluralist approach, the
search for theoretical coherence, and the use of a 'jigsaw methodology'
to synthesise research that provides credible evidence about influences
on a range of desired outcomes for diverse learners (the what, what
magnitude of impact, under what conditions, for whom, why, and how).

The
second paper prepared for the 2005 joint OECD/Netherlands Evidence
Based Policy Research [EBPR ] Conference "Linking Evidence to Practice ,
focussed on the model and role of the Iterative BES Programme as a
brokerage agency for evidence-based policy research across policy,
research and practice.

Overview
I begin this paper by
briefly focussing on current national and global educational challenges.
I also provide some background about R & D as a system lever to
foreground the role of BES as a tool to support sustainable development.

Then
I explain how the Iterative BES Programme is a collaborative approach
led from a national policy agency. I explain the nature of the
engagement with and amongst: researchers and teacher educators,
teachers, educational leaders, policy workers and policy makers, and the
brokerage role of the Iterative BES Programme. I highlight weaknesses
and strengths inherent in the work and lessons we are learning as we
engage in this work. I also signal that the significant gap in the
growing evidence base is systematic consideration of evidence about the
effectiveness of policy. The final section foreshadows the kind of
analytical and empirical work that might further our knowledge.

The
process of identifying weaknesses in the work has been a deliberate,
pro-active and ongoing tool to strengthen this cumulative knowledge
building approach and particular attention is given to the 'iterative
approach' in this paper. The emphasis of this paper is on the Iterative
Best Evidence Synthesis Programme as a collaborative strategy and
catalyst to stimulate and optimise the potential of R & D for
sustainable educational improvement in New Zealand.

Thinking
about use of evidence should be fundamentally informed by an
evidence-based approach to sustainable educational development. To date
our thinking about 'use' of evidence is that it has its seeds in the
synthesis development process rather than following in some linear way.
This paper explains how work-in-progress on educational change processes
and interaction amongst policy workers, researchers, educators and
educational leaders is informing 'use'.

New Challenges for
Strengthened Performance of Education Systems
There are new
challenges for education systems in knowledge societies. It is no longer
sufficient for education systems to sort learners into those who pass
and those who fail. Rather all learners need to be well-served by their
education to develop their capabilities, their sense of belonging, their
well-being and their abilities to succeed and contribute to wider
communities. Governments are looking to education systems to rise to the
challenge to be more responsive to the diversity of their learners and
to meet the higher expectations and future-focus required by knowledge
societies.

The PISA studies show marked differences amongst
education systems in how well 15-year-old students are able to apply
their learning in mathematics, science and reading literacy. They also
show marked differences in disparities between groups of students within
countries. New Zealand has high mean scores, performing in the second
highest band of countries across the PISA studies. But New Zealand's
results show relatively high disparities in achievement by comparison
with most OECD countries. Despite high achievement by many M?ori and
Pasifika learners, there is a pattern of poor outcomes, particularly for
M?ori from New Zealand schooling.

The high disparities, the
relatively high variance within schools in the New Zealand PISA results,
and our rapidly growing demographic profiles for those learners
traditionally underserved by New Zealand schooling, indicate a need for
community and system development to be more responsive to diverse
learners.

Accordingly our collaborative knowledge building work
has at its foundation the goal of being more simultaneously effective
with diverse learners. This goal recognises the day-to-day challenge for
educators from early childhood, schooling through tertiary and adult
learning. Educators need to be working effectively and simultaneously
with students with different prior knowledges and experiences, speakers
of different languages, high and low achievers, students with multiple,
fluid and complex ethnic, gendered and social class cultures, heritages
(including indigenous heritage) and identities, and students who bring
varied dis/abilities and cultural resources to their learning.

Because
the context for this work is New Zealand, all BES developments are
informed by, and inform educational practice in both M?ori and
English-medium education. M?ori have a treaty relationship with the
Crown that protects Te Reo (M?ori language) and tikanga M?ori (M?ori
culture) and guarantees M?ori the same educational opportunities as
non-M?ori. However, the published BESs provide substantial evidence over
some decades of inequitable teaching of M?ori learners (fewer
teacher-interactions, less positive feedback, under-assessment of
capability, mispronounced names and so on) . Although M?ori medium
education has only been a very recent system provision in New Zealand,
and despite resourcing challenges in a language revitalisation context,
early cohorts of students emerging from continuous M?ori medium
education are performing more highly than Maori students in English
medium contexts.

There is much evidence that reveals
difference to be salient in education, albeit in complex and
context-specific ways. Our approach is to put difference at the centre
of this work through a 'responsiveness to diversity' framework. Because
difference is a characteristic that all learners share, the approach
allows for a 'universalising discourse of difference' (Britzman, 1995 ;
Town, 1998) . This approach moves away from 'norm' and 'other' thinking
that has constrained mainstream educational thinking to focus on the
homogeneous and the 'mean' and seeks to strengthen our evidence base
about what works for all learners.

While the Iterative Best
Evidence Synthesis Programme is designed to work across early childhood,
schooling, tertiary, community, industry and adult learning our
progress to date has been in the family and community, early childhood
and schooling areas. For the purposes of this paper the context of
schooling is predominantly used to illustrate the rationale.

The
Role of R & D in Strengthening Educational Practice for Diverse
Learners
It is the goal of practice being more effective for diverse
learners that drives our approach to knowledge building and use. The
syntheses bring together and explain bodies of outcomes-linked evidence
about educational approaches that optimise learning for diverse learners
simultaneously. This evidence foregrounds partnership research and
development between researchers and teachers that demonstrates, for
example :
- how ongoing attention to evidence about the prior
knowledge, thinking, experiences and learning processes of the
particular learners in any context is critical to effectiveness;
-
how to increase student self-regulation and use of metacognitive
(learning to learn) strategies;
- how to enhance student social and
collaborative skills in ways that strengthen both social and academic
outcomes;
- how to develop productive learning communities at each
level of the education system whereby cognitive conflict is valued,
managed and used to enhance learning, and peer supports for learning are
intensified;
- how to provide challenging educational environments
wherein the sense of belonging, the well-being of learners and teachers,
and enjoyment of learning are nurtured; and
- how to optimise
effective linkages with families and communities that strengthen
learning outcomes.

An understated theme across these kinds of
outcomes-linked research and development findings is that there is
outcomes-linked pedagogical research that can strengthen sustainable
ways of working with learners that reduce the stress of teachers and
educational leaders.

However, while the examples of research
identified above arise out of substantial traditions of R & D much
of this research is unknown to many teachers, educational leaders,
policy workers, policy makers and even teacher educators. There has been
a strong tradition leaning towards craft practice within education and
teacher education.

'Craft practice' is used here to mean the
model of teaching where practice is based on teachers' experience, where
there is discussion about teaching matters but involvement in other
teachers' day-to-day practice in classrooms occurs normally only in the
context of pre-service practica. There is emphasis on management and
discipline, evaluation is based on judgement about how the teaching went
rather than consideration of the children's learning, and the
prevailing norms and practices of classrooms are maintained. A craft
practice approach does not involve engagement with R & D around
pedagogy.

Doyle (1990) provides a contrasting framework of a
'reflective professional' approach to teaching which focuses on
reflective capacities of observation, analysis, interpretation and
decision-making linked to data about children's learning. The knowledge
base for the reflective professional is not exclusive of, and values the
craft knowledge of skilled teachers but also includes pedagogical,
subject, socio-cultural and other knowledge from the social sciences and
the use of formative assessment and inquiry processes to inform
teaching. Evidence-based practices become embedded within everyday
educational practice.

Despite the fact that research about
pedagogy is a potentially invaluable record of the work of teachers,
there is stronger ownership of that knowledge by researchers than
teachers and teachers are rarely named as co-authors of research reports
about their work. Teachers have often had reason to find the
educational research they have encountered of little appeal or practical
help (Kennedy, 1997) .

We have had a strong positive response
from both New Zealand's teacher unions to the best evidence synthesis
work; however, the response from teachers nationally has varied. One
response from educators has been the realisation that there are
substantial traditions of potentially helpful research about their core
professional work of which they had been unaware, even through their
initial teacher education.

A consideration of the status of R
& D, and educational research in general, in New Zealand provides
insights into the barriers and potential for future development.

The
Quantum and Place of R & D in Education
A definition of
educational research and development commonly used by the OECD Centre
for Educational Research and Innovation is:
Educational research and
development is a systematic, original investigation or inquiry and
associated development activities concerning the social, cultural,
economic and political contexts within which the educational systems
operate and learning takes place; the purposes of education; the
processes of teaching, learning and personal development of children,
youth and adults; the work of educators; the resources and
organisational arrangements to support educational work; the policies
and strategies to achieve educational objectives; and the social,
cultural, political and economic outcomes of education. (OECD, 1995, p.
37; CERI, 2001)

An example of effective and systemic R & D in
New Zealand is the Numeracy Development Project/Te Poutama Tau. This
project involves communities of New Zealand mathematics education
researchers, teacher educators, teachers and policy workers in a
national professional development programme which has a research process
embedded within it. Teachers carry out diagnostic interviews with their
students in order to guide their pedagogical approaches and maintain
records of their students' progress on a national database.

The
professional development includes conceptual and resource development
and processes of teacher collaboration, observation and reflection. The
pedagogical emphasis has been focussed on the mathematical ideas,
students' metacognitive strategies and strengthening learning community
amongst students. There has been an iterative cycle of development and
evaluation in the Numeracy Development Project which has recently,
partly as a response to findings from BES work, focussed teachers on
better meeting the needs of diverse learners.

Conservative
analyses of the 2004 data for 70,000 NZ students in Years 1 to 8
(English and M?ori medium) showed progress to a higher stage was greater
for all ethnic groups than in 2003 and following an explicit national
focus on responsiveness to diverse learners, for the first time in five
years there was a decrease in disparities between groups. Although the
average effect size advantage for addition/subtraction was only modest
(0.19 which is comparable with the UK Numeracy initiative gains) the
average effect sizes for multiplication and proportion/ratio were more
than double these (0.40 and 0.43) reflecting the emphasis on more
advanced mental strategies in the New Zealand development. In 2003 the
Numeracy Development community attended to the pattern of failure to
reduce disparity and a range of strategies were introduced to strengthen
responsiveness to diverse learners. Early analyses of the 2005
achievement data in the Numeracy Development Project indicate the
significant reduction in disparities apparent in 2004 may not have been
repeated in 2005. The Numeracy Development community has given
particular weight to interrogating this concerning trend change in their
national Hui in March 2006.

Although there is increasing
evidence of the potential value of R & D, a 2003 OECD report
identified the relatively low proportion of funding afforded to R& D
in education and the challenges this raises for knowledge societies.

A
rough estimate of the level of educational R & D as a percentage of
total expenditure on education is on average less than 0.3% in six
countries for which data are available. This is a very small figure when
education is compared with other knowledge sectors, for example, the
health sector where between 5-10% of the total health expenditure in
public and private sectors are directed to R & D.' (p.11).

The
OECD Report includes an assessment of educational research in New
Zealand and estimated educational research funding to be even lower than
that for other OECD countries at between 0.17- 0.20%:
At the same
time New Zealand invests far less in research and development of any
kind than other developed countries, and has far lower R & D
personnel per million population than Australia or Western European
countries. New Zealand is successful educationally, but is, by R & D
standards, not becoming a knowledge economy. (p. 89).

Government
strategies in New Zealand have followed or been in train to lift
research activity. For example; the establishment of a $2 million annual
grant for researchers to work in partnership with institutions to do R
& D focussed on needs identified by researchers and educators: The
Teaching and Learning Research Initiative.

Tertiary policy has
also led to the establishment of additional funding to Centres of
Research Excellence in tertiary institutions. However, none of the new
Centres is focussed on education. In 2003 the first national assessment
of the quality and extent of research activity of the tertiary sector
was conducted: The Performance Based Research Funding (PBRF) Quality
Evaluation.

The results of the PBRF showed educational research
in New Zealand to have the third highest actual numbers of A-ranked
researchers (research of world-class standard) of any discipline. The
number of A-rated researchers in education was outranked only by
academics in engineering and technology.

However, the evaluations
indicated that the submitted research portfolios for almost three
quarters of tertiary academics working specifically in education were
evaluated as either research inactive, emergent or not demonstrating
good-quality research; which meant that education was one of the poorest
performing subject areas across the board . Even if the poor
performance underestimates valuable unreported R & D activity such
activity is unavailable to inform development more widely through
publication.

The Education Peer Review Panel (2004) concluded
that the 'there is clear evidence of a critical mass of nationally and
internationally excellent researchers in education in New Zealand and
this augurs well for the future of our discipline' (p. 283) Despite this
optimism for the future, the need for R & D to improve practice for
diverse learners is pressing, and there are areas of concern.

Closer
links between research and practice are mitigated against because of at
least five factors. These are:
- the uneven distribution of the
excellent researchers across tertiary institutions;
- much of the
quality research may not be oriented towards R & D (only a
proportion of the available quality research focuses on improving
educational practice which is small subset of the wide-ranging interests
of academics in education);
- the relatively low prevalence of
quality research in some teacher education institutions,
- research
quality in New Zealand education was assessed as lowest in teacher
education, e-learning and curriculum (with the exception of mathematics
and science); and
- undermined social capital in the form of networks
and relationships fostering trust and reciprocity in New Zealand
educational research was identified as a national weakness in the OECD
Review (2001) .

This last factor raises concerns that many
educational researchers may be working in quite siloed ways in New
Zealand. This can mean researchers 'rediscovering the wheel' rather than
engaging with, and building on the work of others. This problem has
been highlighted by the authors of a recent review of New Zealand
research on initial teacher education .

Overcoming these barriers
to the escalation of R & D in New Zealand education is critical
because to understand and strengthen New Zealand education we need New
Zealand R & D. The international research provides a substantial
resource for public policy in a small economy. But, when using
international research, New Zealand educators and policy-developers need
to know if what the evidence indicates works in other countries would
apply in the New Zealand context, given regulatory, policy,
institutional, cultural, language, professional and other contextual
differences. For example, educational policy and practice needs to
understand the ways in which indigeneity is salient and the nature of
school-based self-management in New Zealand.

Further, to bring
about effective change we need the practice and benefits of R & D to
permeate New Zealand education. Critical to the potential role of R
& D as a lever for change is the degree of inter-relationship
between research and development. The potential of collaborative and
systematic action research for change exemplifies the power of an
ongoing and iterative cycle of feedback and improvement when R & D
inform each other.

This notion of an interdependent process
between R & D in education contrasts with traditional approaches
where educational resource development , innovation and research may be
occurring independently of each other. This siloed approach lacks the
leverage for realistic, effective, synergistic, cumulative and
sustainable development possible when R & D are interdependent. Such
fragmentation also carries risks for teacher burnout and
ineffectiveness for learners that bandwagonism and isolation can bring.

The
goal of the Iterative Best Evidence Synthesis Programme is to bring
together previously inaccessible research evidence about what works in
an iterative process of synthesis development that builds upon R &
D, informs R & D, is a capability building tool for R & D and
becomes a stimulus for intensified R & D activity across policy,
research and practice in ways that improve practice.

Attending to
a Range of Desired Outcomes
Our approach to the selection of
evidence within a synthesis recognises that within a democracy, desired
outcomes from an education system are part of an agreement within the
wider society and between educational institutions and their
communities. Desired outcomes are by nature subject to a contested and
evolving discourse about what parents and wider communities want for all
our learners.

Accordingly BES writers are required to seek out
research evidence relevant to a range of outcomes previously identified
in consultative processes with wider communities including academic
outcomes, skill development, social outcomes, cultural identity,
disposition as a learner, self-regulatory skills, enjoyment of learning,
preparation for local and global citizenship and success and
well-being, rather than just a narrow focus on particular measures of
academic achievement. Particular emphasis is placed on evidence about
approaches that strengthen a range of outcomes at the same time. For
example, research focused on outcomes-linked evidence about the
interdependence of the social and the academic in mathematics education
(Cohen, 1994 ; Stein, 2001 ).

While there is likely to be
consideration of evidence of impacts on a wide-range of student outcomes
in every BES, the focus on research that has illuminated impacts on
learner outcomes is incontrovertible in BES development and the
justification for the use of the term 'best'. That is 'best' does not
mean 'best available'; rather 'best' denotes evidence and explanation
about how educational or other processes impact positively on a range of
outcomes for diverse learners.

Part of the rationale for the
incontrovertible concern with impacts on learners is the compelling
evidence across studies that have linked teacher goals, learning
processes and student outcomes, that well-intentioned, caring and
experienced teachers can unknowingly teach in ways that have impacts
counter to their own goals (Doyle, 1983 ; Nuthall, 1999 ; Alton-Lee,
Nuthall & Patrick, 1995 ; Bossert, 1979 ). This finding is apparent,
for example, in research about unintended impacts of social studies in
exacerbating racism (Cole, 1998 ; Donn & Schick, 1995 ; Osler &
Starkey, 1999 ; Seixas, 2001 ; Shaver, 1999 ). The concern for impact on
outcomes is similarly critical for well-intentioned policy settings and
initiatives that can also have impacts counter to their goals, for
example, policy initiatives related to drug education (Biddulph,
Biddulph & Biddulph, 2003 ).

Policy advice has a legitimate
role in giving precedence to impacts on learners in education. However,
when evidence-to-practice policy work does not genuinely and positively
support educational development, then cynicism will result , and
opportunities for collaborative development can be in jeopardy.

The
Iterative Best Evidence Synthesis Programme has been from time to time
the subject of attacks from New Zealand academics arising from
discontent with the US 'gold standard' of large randomised controlled
trials or the UK use of systematic reviews. Such attacks have assumed
the Iterative BES Programme to be a clone of those approaches despite
its explicit commitment to the significance of context, critical
realism, rigorous eclecticism, use of outcomes-linked case study
examples, attention to evidence about change processes and to a
collaborative approach. While criticism continues to be a key resource
in the iterative development process for the programme, we are seeking
that criticism be relevant to BES.

Building Sustainability
through a Strategic Health-of-the-System Approach
Because of concern
in policy agencies to ensure accountability of government spending on
interventions, systematic reviews are sometimes commissioned to focus on
evaluations of interventions. The focus of the evidence-based
engagement can be more narrowly focussed on the links between policy
interventions and practice with research playing an evaluative but not a
cumulative developmental role in that link. Pawson (2001) warns of
impoverished knowledge building when there is insufficient attention to
explanation of the underlying causes or mechanisms mediating impacts of
interventions. A risk of an overly intervention-focussed approach is the
failure to advance sustainable development when expensive experimental
evaluations are insufficiently able to address or explain the problems
of poorly designed theoretically impoverished interventions that do not
optimise systemic or sustainable levers. There are instances in
education where randomised controlled experimental trials can provide
very powerful knowledge (for example, Biddulph, 1983 ) but there are
real constraints upon the value of experimental methodology in classroom
research (Alton-Lee, 2006 ).

This Iterative Best Evidence
Synthesis Programme uses a health-of-the-system framework in which there
is a broad concern with how infrastructure, wider policy settings, and
interactions amongst the multiple communities within an education system
contribute to a system that is functioning effectively for all its
learners. This means considering an evidence-based approach to larger
accountabilities for government expenditure and policy settings, not
just those focussed on new interventions.

This
health-of-the-system framework has many ramifications for our
evidence-based approach. For example:
- there is a need for attention
to inter-linkages across BESs and evidence-based theorising of
intermediary, and inter-linked influences on outcomes;
- there is the
need for attention to the educational impact of culture and cultural
mismatches between institutions and families and communities;
-
particular weight needs to be given to outcomes-linked research case
studies that illustrate and explain contextualised examples of effective
practice in New Zealand;
- the model calls for attention to changes
in system impact over time requiring a wider search than in many other
synthesis activities:
- there needs to be more weight given to
analysis and evaluations linked to outcomes that consider not only the
specific evaluations at the school or classroom level but also the ways
in which broader policy, infrastructure, regulatory and 'implementation'
influences can tribute to outcomes (for example, Annan & Robinson,
2005) .

Considerations of evidence-based use and sustainability
call for attention to the ways in which policy interfaces with wider
infrastructure for R & D in the tertiary sector. For example, if the
Ministry of Education were to use the BESs to inform the work of
contracted providers of teacher professional development without
linkages to the ongoing work of postgraduate courses, and thesis
supervision for educators in tertiary educations, then the impact could
be superficial and short-lived. The evidence-based development would, in
effect, be by-passing the engine of knowledge production, dissemination
and qualification gate-keeping that is integral to the role of tertiary
institutions, and is so influential in shaping the nature of
professional knowledge and practice in education nationally. The BES
approach seeks to achieve improvement and sustainability through
embedding the iterative process of development and use in
business-as-usual infrastructure.

What Makes a Bigger
Difference?
We have used the framework in Figure 1 below as a way of
pulling together our available evidence on the relative impact of
different influences on the variance in learner outcomes from schooling.

A
literature review commissioned by the Ministry of Education indicated
that about 40 to 65 percent of variance in outcomes is attributable to
the influences of family and communities, depending on the outcome of
focus. An analysis of multi-level studies of school and teacher/class
influences showed the impact on variance at the teacher/class level to
be variously 16 percent to 59 percent of the variance in learner
outcomes, depending on the subject area, level of schooling, and outcome
of interest. The largest teaching impacts on variance have been
identified in a recent Australian study across a wide range of subjects
at the senior secondary school level , , , . A limitation of this
framework to date is that most of the multi-level modelling studies on
school and class effects restrict their consideration to academic
outcomes. We have New Zealand evidence of teaching class impacts
accounting for 42% of the variance in student achievement in mathematics
at Year 9 level.

The impact on outcomes of school level
influences (from 0-20.9% of impact on variance) varied considerably
depending, for example, on the length of time the learner had spent in
the school, the subject area, and school level policies such as
allowing, or not, lower achievers to be assessed. But the school level
impact was consistently far smaller than that at the teacher/class level
both for primary and secondary education.
In summarising the
findings of the analysis of multi-level modelling studies of school and
between teacher/class effects, we framed teaching as the 'key system
influence' (compared with school-level effects). However, the
translation of evidence-based findings into easily communicable ideas by
speechwriters can rapidly transpose evidence-based messages into claims
that are clearly wrong. For example, the false claim that 'quality
teaching is the key influence on learner outcomes'. A rapid response to
this false claim surfacing came from the leadership of both teacher
unions. In an editorial to primary and early childhood teacher union
membership the union president urged teachers to go to the Ministry of
Education's best evidence synthesis website to check out the evidence,
concluding:

'Our members are committed to continuous improvement
and enhancement of quality teaching but, as the BESs, confirm, it is
when the myriad of influences come into focus, and are addressed,
holistically, that significant progress is made.'

This overview
of influences on variance in student outcomes provides both an indicator
that can direct shared knowledge building to the areas of potentially
larger direct influence such as families and teaching and to critical
mediating influences such as teacher education, leadership and
resourcing.

The implications of this overview analysis for
thinking about the policy importance of teaching have drawn the most
critique of any aspect of the Iterative Best Evidence Synthesis
Programme. So before addressing the policy importance of effective
teaching for diverse learners, it is important to emphasise the
importance afforded to family and community influences in this analysis
of impacts on variance in student outcomes. There are four key
implications that have arisen from our initial best evidence synthesis
on family and community influences on outcomes:
- The impact of wider
social and economic policies on families & community influences
should not be under-estimated.
- The implication of the wider impacts
of families and communities is that an evidence-based policy approach
requires strong interface between educational policy and wider social,
economic and health policy.
- Notwithstanding the implications above,
there is much New Zealand evidence that teacher deficit attributions to
families and students can pose a barrier to effective practice,
particularly for diverse students.
- An R & D approach to
school-family linkages/partnerships in which the school takes agency can
have particularly high impacts because they activate both the home and
the teaching influences and can forge greater coherence between these.

Because
of the importance of family and community influences on children, one
of the first tranche of first iteration BESs commissioned was focussed
on family and community influences on children's educational outcomes.
This BES has been influential in work with other policy agencies. For
example, it has strengthened the case for more priority to issues of
child poverty in government policy and for a higher priority for
children's untreated hearing loss.

This BES has also
stimulated interest in the wider community. For example: the Waitakere
City Council in New Zealand has been in touch with the Iterative BES
Programme about follow-up. They called a one day conference in October
2005 focused on the implications of the BES findings and invited the BES
writers to speak. The conference was described in the following way by
the Council:

Communities and Educators Working Together:
“Strengthening
Relationships�

Fostering a collaborative approach to
building and transforming relationships between communities and
educators

Examining models of effective practice locally,
nationally and internationally and looking at how these might be adapted
to the Waitakere environment.

There is a lack of
infrastructure for R & D in New Zealand in the critical area of
educator-community/family linkages and partnerships. While there are
examples of substantial and sustained gains for students, and busy
teachers and parents, when school-family partnerships are activated
effectively (for example, Biddulph, 1983 ; Biddulph and Tuck; 1983 ;
Biddulph, 1993 and Biddulph, 2004 , many evaluations of government
interventions in this area signal gaps in knowledge and effectiveness.
This is a national educational research weakness that impacts upon work
across government policy agencies.

The R & D
outcomes-linked collaborative work in this area led by Professor Joyce
Epstein at Johns Hopkins University Center on School, Family and
Community Partnerships in the U.S (linked to the Harvard Family Research
Project) is providing a model that could inform a collaborative
national research programme in New Zealand. The Johns Hopkins University
Center has established a new research and development initiative called
the National Network of Partnership Schools
(http://www.csos.jhu.edu/p2000). This network ensures an ongoing action
research approach to identifying what works in school-home partnerships
for specific contexts. The aims of the Center are to provide a research
foundation to guide schools, districts, and policymakers to create and
support positive, permanent programmes of partnership, to disseminate
information about what works, and to learn from a variety of approaches
implemented in diverse schooling environments.

If New Zealand
is to optimise linkages between evidence and practice in this area in
any sustainable way then there needs to be comparable infrastructure and
work carried out here in ways that are appropriate for educators,
families and whanau in the New Zealand context. A particular resource in
this work will be the substantial body of M?ori educational research
illuminating linkages between communities, iwi (tribes), families
(whanau) and schools (kura and wharekura). M?ori educational research
was noted to be an area of national strength in the Performance Based
Research Funding quality evaluation .

Rationale for the
Series of Best Evidence Syntheses
Walter and Davies' (2003) have
identified 'a strategy of creating evidence in priority areas, with
concomitant systematic efforts to accumulate evidence in the form of
robust bodies of knowledge (p. 126) as one of the four widely-agreed
underpinnings of any evidence-based practice agenda.

The
analysis of the relative impacts of different influences on outcomes has
been influential in the selection, prioritising and order of BES topics
for the first iteration BESs. At this stage it has not been possible to
gain agreement within the Ministry of Education about the criteria for
identifying and prioritising future BES topics. Tensions occur around
the BES focus on what can potentially make the most difference for
learners, perceived policy needs, methodological challenges around more
distant levers, and available resources internally and externally.

The
analysis of potential impacts on diverse learner outcomes influenced
the initial cohort of BESs focussed on:
- the influence of families
and communities on outcomes for children from birth across early
childhood education and schooling ,
- quality teaching for diverse
learners in early childhood ;
- quality teaching for diverse learners
in schooling , and
- professional development in early childhood
education .

The second cohort of BESs is now in progress focussed
on:
- effective pedagogy for diverse learners in
mathematics/p?ngarau
- effective pedagogy for diverse learners across
the social studies, social sciences, tikanga-?-iwi curricula areas
-
educational leadership " schooling
- teacher professional learning
and development " schooling

The two teaching-focussed BESs span
early childhood and schooling and all BESs are required to link to and
specifically build on the findings of the families and communities BES.
The focus on educational leadership was not initially planned so early
in the series because of the methodological challenges posed by indirect
links between leadership and outcomes. However, this BES was
commissioned in response to a strong case mounted from the sector and
internally within the Ministry of Education, given the policy importance
of site-based management as a feature of the New Zealand schooling
system.

Consultation with national advisory groups and within the
Ministry of Education indicates that the next four BESs are likely to
focus on tertiary teaching, industry and workplace training and
development, quality pedagogy for diverse M?ori learners and effective
teaching of literacy. The biggest constraint on initiating new BESs
arises out of the considerable resources required to mount, resource and
sustain the collaborative and national consultation processes involved
in a BES development at both the national policy level and across the
research institutions and stakeholders involved.

Collaborative
Knowledge Building as a Change Strategy
A key proposition in this
paper is that the project of bringing together rigorous and useful
bodies of evidence about what works in education needs to embed within
its approach ways of working that attend to the 'knowledge utilisation'
challenge as well as the knowledge building challenge. If such ways of
working are built into EBPR then the endeavour of itself can be a
transformational process that not only constructs a new kind of dialogue
and understandings amongst policy workers, practitioners and
researchers but also provides the foundation for using the knowledge to
make a bigger difference in education.

This means embedding
opportunities for dialogue into the knowledge building processes rather
than initiating them after a synthesis has been produced. This approach
is resource intensive because it is part of a national educational
development process of itself. Ginsburg and Gorostiaga (2003) explain
the underlying principle in the series editor's introduction to an
international consideration of the Limitations and possibilities of
dialogue among researchers, policy makers and practitioners':

Dialogue
isn't necessarily more efficient, but it's more democratic and,
therefore, more effective.

…Our preference is also based on
the belief that in the long run dialogue and participation by a wide
range of stakeholders produce better and more relevant educational
research, policy and practice. …Certainly, it may be easier " and, in
that sense, more efficient " for researchers, policy makers, and
practitioners in education to engage in action (or even in praxis) in
isolation of members of the other groups. However, the decisions that
are made and the actions that are pursued are likely to be less
effective. This is the case not only because the quality of judgements
may be lower but also because the activities of one group may detract
from or cancel out those of other groups.' (p. x)

The ways in
which we are forging the collaborative knowledge building strategy in
BES are being informed by our own outcomes-linked evidence within the
Ministry of Education. As indicated earlier in this paper we have an
outstanding national educational strategy for which there is clear
evidence of both a significant national shift in achievement outcomes
for all ethnic groups and a measurable reduction in disparities between
ethnic groups: The Numeracy Development Strategy . This policy programme
has exemplified a national research and development strategy in both
M?ori and English Medium Schooling.

Evaluators have described the
NDP as 'a dynamic and evolutionary approach to (policy) implementation'
. The senior officials involved, Malcolm Hyland and Ro Parsons have
in-depth educational knowledge and expertise and have worked from the
outset in an iterative R & D process with teacher unions,
researchers, teacher educators, leaders and teachers to successfully
create a national learning community. The project has used a deliberate
iterative process exemplifying Patton's (1997) utilization-focussed
approach to evaluation. This collaboration is evident in a 2003
publication co-authored by an academic and officials 'The Numeracy
Development Project: Policy to Practice' . In that paper the authors
explain the significance of shared leadership, open communication and
the valuing of conflict and challenge in the policy approach:

'The
learning community that has developed around the project is
characterised by high levels of trust, confidence and
ownership…Conflict and challenge are not viewed as threatening, but as
leading to new insights. (p. 167)'

Recent policy work has
provided a framework and strategy to achieve sustainability in this
national programme.

While developing a unique fit-for-purpose
approach to processes, BES has fundamentally modelled its processes on
the Numeracy Development Project strategies. A range of international
research and analysis including Fullan's (2005) analysis of what works
in educational leadership at a system level is also informing our
processes. Key elements are: emphasis on moral purpose, incentivizing
collaboration, engaging in continuous reflective action (that does not
remain at the level of just thinking), building capability at each stage
and underpinning growth with a press for growing financial investment
in R & D in education.

In the following section of this
paper BES development processes are explained with consideration of the
approach to collaborative knowledge building across research, policy and
practice as a change strategy.

Developing National
Guidelines for Best Evidence Synthesis Development
One of the key
challenges for EBPR is the contestation of what counts as rigorous
evidence amongst researchers. In order to gain the confidence of the
educational research community and their engagement in iterative
processes of BES development and use, the Ministry of Education drew
upon expertise across the country to strengthen the approach and to get a
high level of agreement about the methodology to be used.

In
2003 and 2004 the Ministry of Education brought together national
reference groups of researchers, methodological advisers, BES writers,
policy workers and teacher union representatives to develop formal and
agreed Guidelines for Generating a Best Evidence Synthesis Iteration .
The approach taken was to gain agreement about the purposes which then
informed a fit-for-purpose methodology. As part of the international and
national formative quality assurance for each BES, feedback is
commissioned also about the adequacy and short-comings of the Guidelines
for Generating a Best Evidence Synthesis Iteration to help strengthen
the Guidelines.

The national advisory groups particularly
emphasised the importance of interaction in the processes of BES
development. This advice has been operationalised in the Guidelines and
through the structuring of a series of six contractual milestone reports
on progress for each BES development. BES writers are required to
provide an update on a draft methodology chapter for each milestone, and
to successively provide scoping outlines and partial draft samples
through to a whole draft report. The draft is submitted for national and
international formative quality assurance some months before the final
report is due. This final stage enables the QA to feed into optimising
the quality of the final product.

Each milestone report is
considered by a BES management team that includes cross-Ministry and
sector representatives. BES writers also fill-in a quick self-audit
against the Guidelines at each of the preliminary milestones that
together with the developing draft provides the basis for dialogue,
critique and collaboration as the BES develops. This process assists
with transparency, capability building and wider ownership of the
knowledge building.

The full account of the methodological
approach, its underpinnings in a realist philosophy of social science,
its rationale for the privileging of outcomes-linked case studies, and
the weight given to theoretical coherence in BES can be found in the
Guidelines here. Dr Brian Haig, philosopher of science at the University
of Canterbury, advises BES on the realist underpinning.

The
formative quality assurance processes are providing valuable critical
feedback on the Guidelines for Generating a Best Evidence Synthesis
Iteration. In a March 2006 QA report Professor Paul Cobb advised that:
(a) the Guidelines need to give BES writers more concrete advice on ways
to manage the multiple audiences of BES, (b) there needs to be more
distinction made between two different kinds of causality claims in BES
(process oriented explanation and regularity type of causal
description), (c) there needs to be more concrete advice on using
health-of-the-system and other systemic perspectives in BES, and
revision should strengthen the consistency of the use of a
'responsiveness to diversity' framework for BES with particular
attention to applying the discussion of the complexity of individual
identity across the Guidelines.

Professor Cobb's criticism
will inform the iterative development of the Guidelines. His report has
also been useful in confirming strengths of the approach:
The
Formative Quality Assessment Seminar held in Auckland on February 13,
2006 exemplified transparency and provided an opportunity for members of
a wide range of constituencies to present and debate their views. The
BES Guidelines are outstanding and are clearly grounded in the hard-won
experience of synthesizing research findings to inform both policy and
practice. Strengths of the Guidelines include:
- The three detailed
checklists for developing a BES iteration (pp. 15 " 17).
- The mature
view of evidence apparent in the call for theoretical pluralism and
methodological pluralism.
- The consistent call for attention to
issues of language and culture.
- The consistent call for attention
to the worth of explanatory theories, including explanatory coherence.
-
The commitment to make the BES approach transparent.
- The use of a
grounded or bottom-up approach for interrogating studies.

Distinctiveness
of Iterative BES Methodology amongst Evidence-Based Policy Approaches
The
recognition of the distinctiveness of BES methodology by our
international quality assurers is increasingly critical because of the
growing international literature of discontent associated with overseas
approaches to evidence-based policy work in education.

There
has also been wider positive feedback for the fit-for-purpose,
innovative nature of the Ministry of Education's approach
internationally. For example, there has been a call to develop a
parallel approach in Australia . Notably, Professor Allan Luke and David
Hogan (2006) from the Centre for Research on Pedagogy and Practice in
Singapore have considered the Iterative BES Programme approach in depth
in World Yearbook of Education: Educational Research and Policy :

The
most comprehensive approach to evidence is the New Zealand Ministry of
Education's Iterative Best Evidence Synthesis Programme… What is
distinctive about the New Zealand approach is its willingness to
consider all forms of research evidence regardless of methodological
paradigms and ideological rectitude, and its concern in finding
contextually effective, appropriate and locally powerful examples of
“what works�. Its focus is on capturing and examining the impact of
local contextual variables (e.g., population, school, community,
linguistic and cultural variables). Indeed, 'what authentically works'
in educational interventions may be locally effective with particular
populations, in particular settings, to particular educational ends.
This stands against the base assumption of the US model: that there are
instructional treatments that can be shown to have generalisable and
universal efficacy across and in spite of contexts, that this efficacy
at the production of educational outcomes can be assessed solely through
standardised achievement test results, and that the matter of 'reform'
of systems requires the mandating, standardisation and implementation of
these approaches. Our position concurs with the New Zealand approach.
The
later sections of this paper endeavour to make transparent our approach
to a cost-effective, principled agenda for collaborative knowledge
building and process for collaboration.

Fit-for-Purpose Selection
of Best Evidence Synthesis Writers
The selection of BES writers is
also guided by fit-for-purpose considerations for both knowledge
development and use.
First, our approach requires the best evidence
synthesis lead writers to be New Zealand educational researchers who
have shown national research leadership in the BES area of focus, or
demonstrated capability to take on a professional and research
leadership role in the field in New Zealand.

Our purpose is
to ensure capability development within New Zealand. Strategies are used
to access international expertise including international quality
assurance but the writers must be local. Some funding is also provided
to writers to enable them to publish widely from the BES after
publication in order to continue the iterative process of peer scrutiny
and critique and to support the national leadership role sought from BES
writers.

Second, we aim to embed the iterative BES work
across the New Zealand tertiary infrastructure. This differs from an
approach that might develop expertise within a national policy centre or
located within one specialised external unit focussed on methodological
expertise in evidence-based synthesis. Accordingly, we seek to have BES
developments situated in, and spread across, Universities, Colleges and
Wananga so there is also institutional support for, and ownership of,
BES development.

Institutions are asked to demonstrate
commitment through contributing to the resourcing of the BES development
process, for example, contributing to teaching release time for BES
writers and funding writers to attend international conferences. Our
rationale is that there is value returned to the institution through the
collaborative BES opportunity because it helps to inform their core
practice and supports institutions in taking a national leadership role
in knowledge building and R & D.

Third, we are seeking
BES writers who are also teacher educators (or in the case of the
educational leadership BES involved in principal and leadership
development and so on). For the BESs to support an embedded R & D
approach to teacher education and teaching in New Zealand they need to
be used by teacher educators in ongoing and iterative processes of both
using and strengthening the evidence base. There is a need for New
Zealand teacher educators not only to be informed about the work but
also to take some ownership of the emerging knowledge.

Accordingly,
a substantial proportion of the contract funding for a BES development
is provided to the BES writers to include teacher educator/researcher
colleagues from across New Zealand as advisors to, and beneficiaries of
the development process. This strategy helps to overcome the
disincentives for collaboration of a nationally competitive tender
process.

The BES development process strengthens the work,
builds ownership and has other gains. For example, the process
strengthens the kinds of networks and social capital amongst researchers
and teacher educators nationally that are conducive to developing
enhanced infrastructure for R & D. Our experience shows that the BES
development processes can involve writers and their advisers taking the
time to read the work of colleagues that has been so siloed few are
aware of the work. This process is one of valuing and acknowledgement
that can have substantial rewards not only for the wider processes of
knowledge building but also for the individuals involved.

The
Role of BES as a Research-Policy Link Informing R & D Investment
There
are other strategic benefits for the evidence-practice agenda that
arise out of this link between policy and research. For example: BES
developments provide critical and systematic insight into major gaps and
areas of need for targeted R & D investment.

Linkages
Between Policy and Teaching?
The lesson we have taken from both the
importance of teaching and the risks of ineffective policy responses to
the need to strengthen teaching for diverse learners, is to work in
partnership with both New Zealand teacher unions in advancing the
Iterative Best Evidence Synthesis work. While the decision to work with
the teacher unions was a practical way of establishing a broad
partnership with New Zealand teachers, there is evidence for there being
a link between more intensive teacher union activity and higher student
outcomes in the US (Carr Steelman, Powell & Carini, 2000) . The
teacher unions have a history of professional leadership in New Zealand
particularly in the areas of curriculum and subject associations.

The
New Zealand Teachers Council (NZTC), a new professional body for all
registered teachers working in early childhood, primary, secondary,
tertiary and other teaching situations, was established in 2002. The
Iterative BES Programme has also established a partnership with the NZTC
and council policy workers including the CEO are also involved in the
collaborative approach we are jointly building. The NZTC has been
proactive in ensuring that the BESs also inform their work and are
assisting to fund the leadership BES development.

Through the
partnerships with the teacher unions and the New Zealand Teachers
Council, representatives are engaged with:
- the development of
requests for proposals;
- selecting the successful tenderer;
-
providing advice;
- engaging with an iterative and collaborative
process with BES writers through managing the milestones reports;
-
participating in national Think Tanks and seminars organised to support
BES development;
- contributing to the formative quality assurance of
new BESs; and
- continuing advice about approaches to use of BESs.

In
Wright & de Joux's (2003) case studies of successful innovation in
the public sector they find collaborative and not contractual
relationships to advance innovation. However, contractual relationships
are central to BES. The process of policy and stakeholder representative
engagement in contractual management is a strategy to maximise
collaboration, consultation, capability development, fairness and
transparent contract management. Ongoing advice from Ministry legal
advisers and strong, collaborative relationships with BES writers are
both critical to the process working optimally.

In New
Zealand we are watching with interest the innovative ways in which the
UK Centre for Use of Research and Evidence in Education (associated with
EPPI) is engaging policy workers and teacher union representatives in
synthesis development processes to optimise the ways in which content
and form serve function.

The partnership with teacher unions
is greatly strengthening the Iterative BES Programme. The programme has
also been instrumental in empowering teachers to reclaim the research on
educational practice as their own. As teacher union representatives
were talking to researchers within the context of a national Think Tank
for BES development, it was interesting to note the use of the term 'our
BESs'.

Recently the teacher unions have taken a pro-active
lead in strengthening the policy-research-practice dialogue about
evidence. In May 2005, the primary (NZEI) and secondary (PPTA) teacher
unions organised a cross-national forum with the Australian Curriculum
Studies Association entitled Quality teachers: Quality teaching
"Creating a new agenda for action by practitioners, researchers and
policy makers. They invited Keynote addresses (including one focussed on
BES) and 100 or so policy, teacher and research leaders from Australia
and New Zealand to engage in the forum. The secondary teachers' union
(PPTA) has initiated and organised a national professional conference
for April 2006: 'Quality Teaching: Leading the Way'. PPTA has invited a
keynote address and eight workshops from the Iterative BES Programme.
In
the most recent formative quality assurance forum held for the
Effective Pedagogy in P?ngarau/ Mathematics BES the NZEI prepared a
quality assurance report signed-off by their national president. In this
report the teachers involved exemplified the iterative process of R
& D by drawing upon their own experiences of the Numeracy
Development Project to highlight important aspects of project
effectiveness omitted or under-emphasised by the BES writers at the
draft stage.

In the case of syntheses focussed on teaching, we
seek each BES to be accessible to its multiple audiences; but it is
teachers and others closest to the teaching who are the primary
audience. The teacher unions have formally asked the Ministry of
Education to ensure in all communications about the use of BES that the
desire of teachers is to be first afforded time and opportunity to read
and discuss the syntheses. Through the quality assurance process the
teacher unions have now asked for distribution of BESs to all schools.

Accessibility
and Impact: Developing an Evidence based Approach to Presentation
The
issue of an evidence-based approach to what works for multiple
audiences, and in particular for a teacher audience is arguably where
evidence-based knowledge is most impoverished. What form of synthesis is
most useful for teachers? Despite the commonly espoused view that
simplifying the message will make them more accessible and more useful
to teachers, the evidence about educational change does not readily
support this view. There is a strong case for the use of research-based
vignettes of practice embedded in syntheses, to exemplify theoretical
tools that are of use to teachers adapting and using the findings in
their own contexts. Kennedy (1997) reviewed studies of research use by
teachers that showed research was more likely to be used conceptually
than instrumentally:

'practitioners did not take from research
tools that could be directly applied in their classrooms, but instead
took ideas: concepts that could, especially when combined with other
ideas, help them invent specific responses to local situations.' (p. 7)

We
seek from our BES writers, attention to embedded vignettes (not
anecdotes but actual data narratives) that explain the broader findings
with appropriate attention to the complexities of actual practice.
Frequently such vignettes not only bring the findings to life for
teachers, but also for policy workers and other researchers. Theoretical
tools derived from the research are seen as enabling teachers to use
research findings conceptually to inform their own work.

Current
BES writers Drs Graeme Aitken and Claire Sinnema have attended to
research on teacher learning not only as part of the focus of their BES
but also to inform the way in which they present the evidence . In their
words:

…in developing this format we have been conscious of
the need to respond to the particular needs of adult learners whose
approach to learning is characterised by (Edmunds, Lowe, Murray, &
Seymour, 1999):
1. deciding for themselves what is important
2.
the need to validate information based on their beliefs and values
3.
the expectation that what they are learning will be immediately useful,
and
4. fixed viewpoints based on considerable experience.
These
characteristics have been supported by research into resource design
that enhances teacher learning (for example: Aitken, 2005; Davis &
Krajcik, 2005; Spillane, Riser, & Reimer, 2002). This research
argues that it is important to:
1. acknowledge existing schema by
integrating new learning into structures and approaches that are
familiar (hence the case study format);
2. alert readers to possible
misconceptions about the ideas presented by explaining what the research
shows and, more importantly, what is does not (hence the marginal
comments to the left of each page);
3. minimise complexity through
transparent structures and clear expression(hence the use of white space
and bold headings);
4. integrate new learning at the level of detail
most likely to be attended to by teachers (hence the use of case
studies that have strong face validity);
5. support teacher
adaptation and transfer to their own familiar settings in ways that do
not compromise the validity of the research findings (hence the use of
marginal comments in the text and prompts within the case studies)' (p.
20 Draft BES December 2005)

The draft case study provided over
'Facilitative inclusion for Ian and his Peers' (see Appendix A)
exemplifies the ways in which the BES writers have drawn upon the
research cited above.

Like many examples from the quality
teaching BESs , the research focussing on Ian exemplifies the ways in
which effective pedagogy simultaneously addresses a range of outcomes
for diverse students at the same time " achievement, social skills,
cultural identity and potential 'behaviour problems'. The case study
illustrates also the significance of teacher agency in the ways students
learn and practice (or not) values such as respect as part of their
moment by moment experience of schooling. The responses to Ian
demonstrate compellingly how 'quality teaching for diverse students' is
not about adding more but about transforming business-as-usual.
Everybody benefits, including the teacher, because the teacher's actions
in strengthening the peer learning culture, lessen teacher stress. This
example is particularly important for the wider public sector because
it gives insight into ways in which a non-bullying culture can be
proactively forged within education institutions.

This case study
exemplifies recent research on overcoming the problem of
overassimilation when novices use the same language but without the
depth of understanding needed to engage in a way that changes practice:

One
method for overcoming this assimilation problem is to use carefully
calibrated sets of contrasting cases, grounded in practice, as well as
in theory, that help people progressively differentiate their
understanding rather than simply assimilate new information to
pre-existing ideas (p. 368)

In the case study example in
Appendix A, BES writers Claire Sinnema and Graeme Aitken have used
contrasting cases, drawn upon the research cited above, advice from
stakeholders and consultation with the researcher involved, Christine
Rietveld of the University of Canterbury. This case study is still work
in progress exemplifying both the case and its linkages to the wider
findings of the synthesis.

The international quality assurer
for this BES is Professor Jere Brophy, from Michigan State University.
Professor Brophy is an international expert in research on teaching in
social studies, editor of the Advances in Research on Teaching series
and author of many influential texts and Handbook chapters in the last
40 years of research on teaching. Professor Brophy's formative review of
the draft BES included the comment:

I believe that the
authors are well on their way toward developing an outstanding document
that will be of value not only for its intended purposes and audiences
in New Zealand but for an international audience of social educators.

The
status of this work at present is that the authors are responding to
the range of formative quality assurance reports from New Zealand
academics working in English and Maori medium education, the National
Education Monitoring Project, the Teacher Unions, and teacher educators
to strengthen the final synthesis to ensure its validity, rigour and
usefulness across early childhood education and schooling.

The
BES writers have also been invited to present work-in-progress to the
Invisible College of Research on Teaching in San Francisco in April 2006
and will be presenting their work at the American Educational Research
Association Conference (the largest international conference held
annually in education) to enable wide international critique and advice
to inform the final stage of BES development. The requirement for
presentation of work in progress at an international conference is a
requirement of the request for proposals for BES development. BES
writers' institutions are required to fund this aspect of BES
development to support capability building and in recognition of the
benefit accruing back to tertiary institutions from engagement in BES
developments. That is: engagement in a collaborative knowledge building
process bringing together the evidence about effectiveness in education
will deeply inform the practice of teacher education in the host
university or wananga.

Outcomes-linked Evidence about Teacher
Professional Learning Informing BES
The case study about the
teaching of Ian in Appendix A is of itself however, still just a
document. Effective use of this BES tool needs to be informed by
evidence about how this kind of outcomes-linked evidence can inform
business-as-usual practice in New Zealand education. Both the Teacher
Professional Learning and Development BESs and the Leadership BES will
inform policy thinking about 'use'.

The complexity of teacher
learning is exemplified in the framework for mapping studies Professor
Timperley of the University of Auckland has created out of early work on
the Teacher Professional Learning and Development BES (See Appendix B).
The diagram provides a framework to map the different studies and is
neutral with regard to particular outcomes. Once the studies retrieved
for the synthesis have been mapped, then the synthesis will reveal what
does or does not work across this body of evidence, and the framework
will evolve further in the light of later analyses.

Because
the Teacher Professional Learning and Development BES for Schooling will
be so influential in the Ministry of Education's value-for-money work
the collaborative processes across policy, research and practice have
been particularly critical for this BES. Because academics/ teacher
educators/ researchers in this field tend to be working within specific
curriculum areas or projects there has been very little networking
across this 'field'. In order to create a more strongly networked
community engaged in the BES development two national Think Tanks were
held. The Think Tanks created an intensive dialogue amongst leading
researchers working with outcomes-linked evidence and other
stakeholders.

After an extended iterative process Professor
Timperley received unanimous agreement to a mapping framework for this
BES (as shown in Appendix A) at the 2nd National Think Tank for BES
Development in September 2005.

This BES is due for completion
in September 2006. Dr Lorna Earl, who is the international quality
assurer for this BES has recommended that the final stage of BES
development involve two advisers from the Ministry funded school support
services in order to help inform the use of the BES findings to inform
in-service teacher education work carried out nationally.

As
the new cohort of BESs is completed we are seeking to bring together BES
writers and our national advisors to explore the possibilities for
evidence-based approaches to BES use. We are seeking advice on the
possibilities for setting up nationally networked postgraduate courses
for teachers offered across tertiary teacher education institutions. One
possibility is that such development may be linked to progress in
getting Centres of Research Excellence in Teaching and Teacher
Education. This strategy would only be one of a whole range of
multi-level system, and school-based strategies to strengthen R & D
models for teacher professional learning.

The goal for the
postgraduate courses would be for the pedagogy and practice of the
courses to be consistent with the findings from the forthcoming Teacher
Professional Learning and Development BES. Emerging evidence suggests
that the courses might be school-based, or teacher-education based (for
advisers to schools or mentors within schools), use a collaborative
action research approach, provide ongoing opportunities for
collaboration and networking amongst teachers, enable observation and
reflection linked to student learning data and lead to changes in
practice which become embedded. Outstanding examples of teacher or
teacher educator engagement in R & D through such courses, and the
findings from the evaluations of the impact of such courses on the
students of the teacher participants would also feed back into future
BES iterations.

Use of BES with/in Policy Work
A
consideration of the strengths and weaknesses of the policy dimensions
of this collaborative knowledge building work should first acknowledge
that having the leadership of a collaborative knowledge building
programme situated within a national policy agency is unusual
internationally. It is unusual because the focus of the policy advice
arising from the programme is on medium-term and long-term strategy
across election cycles. The possibility to lead such work within a
central policy environment exists because the New Zealand Ministry of
Education has, as do other New Zealand policy agencies such as the
Treasury and the Ministry for Economic Development, a Medium Term
Strategy Policy Division.

Perhaps the most substantial gap in the
available evidence-base is that which explains the links between policy
decisions, activity and outcomes for diverse learners, or explains the
communication, organisational learning and other processes that mediate
policy decisions and activities.

Nutley, Walter and Davies'
(2003) 'Framework for Understanding-the Evidence-into-Practice Agenda'
helpfully suggests six research fields that may advance knowledge about
'research utilisation'. These are research on: diffusion of innovations,
institutional theory; managing change in institutions, knowledge
management, individual learning and organisational learning. Work is in
progress internally within the Iterative Best Evidence Synthesis
Programme to draw upon these literatures to inform our thinking about
BES utilisation within the wider Ministry and inter-agency policy
context.

Significant challenges arise out of bringing
together bodies of outcome-linked evidence that can challenge current
practice. Such challenges require the kind of double loop learning that
promote continuous organisational improvement. Agyris and Schön (1996)
developed the theory of single-loop and double-loop learning to describe
organisational learning. Single-loop learning operates to correct
errors, whereas double-loop learning occurs when error is detected and
corrected in ways that involve the modification of an organization's
underlying norms, policies and objectives. Nutley et al. similarly
signal the importance of the distinction between adaptive learning and
generative learning in organisations:

'Adaptive learning
routines can be thought of as those mechanisms that help organisations
to follow pre-set pathways. Generative learning, in contrast, involves
forging new paths. Both sorts of learning are said to be essential for
organisational fitness but by far the most common are those associated
with adaptive learning.' (p. 136)

From a BES Guidelines
perspective research that focuses on 'what works' (but attends to what
doesn't work also), explains links between influences, mediating
processes and outcomes, and attends to context, particularly local
context will be most helpful in illuminating policy processes. We are
seeking through the Educational leadership BES, particularly, to
systematically interrogate this evidence for educational policy.

The
eleven case studies of innovation in the public service commissioned by
Treasury, the Department of Prime Minister and Cabinet and the State
Services Commission (Wright & de Joux, 2003) identified four key
themes and 14 attributes of successful innovations in the New Zealand
Public Service. Of particular resonance to BES are the following:
-
Forward-looking, externally focussed organisations
- Continuous
implementation process
- Monitor and evaluate to identify problems,
trigger ideas and assess success
- Develop diverse and diffuse
invisible colleges, partnerships, and collaborations across agencies,
individuals and organisations
- Exploit opportunities by consistent
forward planning and engagement with stakeholders
- Ensure senior
management support, mandate, commitment , faith and trust
- Provide
sufficient resources to support innovation
- Manage diverse
stakeholder interests, concerns and their tolerance for risk at
appropriate times during stages of innovation
- Manage projects and
risks tirelessly for each stage of innovation (p. 3)

The
Wright & de Joux (203) analysis is fruitful but also signals the
value of commissioning more rigorous secondary analysis of public sector
effectiveness. For example, the roles of University researchers and
research and development as critical levers in innovation are implicit
rather than explicit within the analysis of the Strengthening Education
in Mangere and Otara innovation. Ministry of Education Change Manager,
Brian Annan is completing a doctorate in which he is developing a
grounded theory of policy effectiveness, to explain change processes in
educational development in New Zealand. His particular focus is the
interface between local stakeholders and wider policy settings and
approaches in change processes that have led to demonstrable
improvement. A conference paper presented at the 2005 American
Educational Research Association received favourable comment from
discussant Michael Fullan; a leading academic in the area of educational
leadership and system-level change.

Cranefield's (2005) masters
research project : 'Inter-organisational knowledge transfer in the New
Zealand State Sector' is a further example of case study research that
can helpfully inform both within and inter- agency policy practice. The
research project identified key factors that impacted on knowledge
transfer within and across agencies in the Pathfinder Project, the two
year project to introduce the Managing for Outcomes framework into the
New Zealand public sector.

Key factors affecting knowledge
transfer for each stage of the model are outlined. These differed
according to the challenges of each stage, but three broad groups of
factors were identified: organisational factors, knowledge-related
factors, and gatekeeper-related factors.

Key organisational
factors were: the perceived degree of fit of new knowledge with an
organisation's discipline and experience, CEO support, the type of team
approach used, silo-based structures, subcultures, and a wide geographic
distribution of staff. Key knowledge-related factors were related to
(a) representations of knowledge: the nature of language used, the
transferability of case studies and examples, and the use of boundary
objects; and (b) knowledge processes: codification, translation,
interpretation, storytelling, a 'trickle-down' approach to disseminating
knowledge, and exposure of staff to small, regular, 'doses' of new
knowledge.

Gatekeepers were found to have played a critical part
in enabling knowledge transfer. Key gatekeeper-related factors were
based around a diversity of roles undertaken concurrently by gatekeepers
in the course of the project. These roles were flag-bearer, advocate,
translator and interpreter, scout, facilitator, storyteller, and expert.
The combined role of translator and interpreter is identified as
particularly important, involving the active conversion of knowledge to
meet different recipients' needs, and thereby increasing the
organisation's absorptive capacity.

Policy Progress and
Challenges

The Iterative Best Evidence Synthesis Programme
has been influential within the New Zealand Ministry of Education. For
example, in influencing the corporate Statement of Intent and in
informing the development of a five-year 'Schooling Strategy' agreed by
stakeholder and sector groups which has evidence-based development as a
central part of the strategy. A key role for BES is to show where
greater magnitude of impact and sustainability is already evident in
Ministry work. However, to avoid pitfalls such as misuse of evidence,
failures of empty rhetoric, and magical notions of change undermining
the potential of evidence-based approaches, there needs to be a
significant strengthening of knowledge in this area for the New Zealand
context. The need for strengthening the kind of evidence-base outlined
above and the need to enhance the potential for generative learning are
significant challenges. As it impacts on practice across our work, the
Ministry of Education's new Evaluation Strategy will play a significant
role in informing and supporting the effectiveness of BES as a catalyst
for change.

Dr Lorna Earl, co-evaluator of the UK Numeracy and
Literacy Strategy is advising BES on several aspects of the next phases
of our wider evaluation framework for BES. Dr Earl is developing a
protocol for the evaluation of sector-led innovation using BES with the
first case study commencing in September 2006.

There are emerging
examples of deep iterative and reciprocal processes of engagement with
BES across the Ministry of Education. For example, BES is being informed
by the evidence emerging from Te Kotahitanga. Te Kotahitanga is a New
Zealand secondary school improvement project developed by leading M?ori
researchers, professional developers, community elders and teachers.
Emerging evidence of substantial effect sizes in student achievement
increases in essential skills assessments suggests a powerful change
model has been developed. Professor Emeritus Christine Sleeter,
currently Vice-President of Division K (Teaching and Teacher Education)
for the American Educational Research Association recently visited the
schools and reviewed the programme. She concluded:

The project is
exceptionally well conceived, very comprehensive, very consistent with
the literature on culturally responsive teaching and professional
development, and backed by a sound research program. As you know, I've
worked on these same issues for about twenty-five years, as both a
higher education faculty member working with pre-service and practicing
teachers, and as a researcher/ theorizer. I see more potential to make
significant and sustained improvements in schools for students from
historically underserved communities in this project than in any other
project that I have had contact with. (March 28, 2005)

Deepening
evidence-based attention to issues of scaling and sustainability is
particularly critical given the vexed international policy history of
attempts to 'scale up' school improvement. Ministry colleagues
overseeing the contract for Te Kotahitanga over the past three years
have proactively engaged the project leaders in a reciprocal process of
informing and being informed by work in the Iterative Best Evidence
Synthesis Programme. For example, they have proactively used Coburn's
(2003) landmark overview of research Rethinking scale: Moving beyond
numbers to deep and lasting change and her framework which has been used
within the Ministry and in engagements with stakeholders. Coburn notes
that for the US context:

'the history of public schooling is
replete with evidence of reforms that barely scratched the surface of
schooling, failing to reach into the classroom to influence instruction'
(p. 4)

Coburn's analysis of what works identifies depth,
sustainability, spread and shift in reform ownership as critical and
inter-related dimensions for systemic and sustainable development.

Professor
Russell Bishop, who co-leads the Te Kotahitanga programme has been
granted funding by a New Zealand Centre of Research Excellence, Nga Pae o
te Maramatanga, to provide an analysis and interrogation of Te
Kotahitanga in the light of the study findings and the implications of
the wider evidence about scaling and sustainability. This iterative
process is, in turn, now informing the development of the new teacher
professional learning and development BES.

There are emerging
examples also of synergies between BES work and the work of other
agencies. Newlands Intermediate School implemented a two year whole
school development plan using the 'Quality Teaching for Diverse Students
in Schooling BES'. Along with systematic use of pre and post testing
around the teaching of units in each curriculum area, the school has
been monitoring student enjoyment of learning showing dramatic
improvements (see Appendix C). This work exemplifies a principal-led
research and development approach taken by a school using BES with no
external involvement of researchers.

Recently the principal,
Wendy Esera remarked that she has been using the Ministry of Youth
Development's (2005) Resource “Making it happen…'- Strengthening
youth development in schools� (Principles 2 and 4) in the School's
Charter (Strategic Goal No.5 2006-2008 Charter).

A consideration
of the “Making it Happen� resource reveals a policy framework
informed by both the Ministry of Education's Schooling Strategy and the
Iterative Best Evidence Synthesis Programme. The document does not just
reference the syntheses but also exemplifies key findings. Although I
was unaware of these linkages being forged, through business-as-usual
interagency consultation the Ministry of Youth Development has created
alignment and synergy in our shared work.

The iterative approach
is not only critical to the processes used in knowledge building and
use. As contexts change, and new evidence emerges, our understanding of
evidence about what works is, and should be, continually changing and
developing. Through embedding a sense of 'knowledge building' as
iteration BES is able to act as a catalyst to R & D. BES deeply
values what the past work of others can offer to educational development
and explains and guides how we might make a bigger and cumulative,
positive difference for all learners now and in the future.

Appendix
A: Aitken, G. & Sinnema, C. (work-in-progress). Facilitating
inclusion for Ian and his peers. Effective Pedagogy in
Tikanga-?-iwi/Social Studies/Social Sciences Best Evidence Synthesis
Iteration. BES prepared for the Ministry of Education's Iterative Best
Evidence Synthesis Programme.

Note: For formatting reasons this
Appendix has been provided with this paper rather than within it.