Educational research – How to navigate and use quality research
We investigate the pitfalls educators should be wary of when weighing up the merits of education research…
Navigating the vast sea of educational research can be daunting. It’s vital we can distinguish high-quality studies from flawed ones if we want to teach effectively. Here, experts from EEF explore how to go about it…
The quest to obtain the evidence teachers need
Christine Kelly and Faizaan Sami look at how the EEF’s new evaluations can give teachers better evidence to support their everyday practice…
Teachers regularly make decisions in their daily practice to try and improve pupil learning, with limited evidence available to support their choices.
To try and help with this, the Education Endowment Foundation introduced a new approach to evaluation in 2019 called Teacher Choices, which sought to generate relevant and actionable evidence that could inform the dilemmas that teachers routinely face.
After some disruption during the pandemic, we’re now launching a new series of Teacher Choices evaluations and taking stock of what we’ve learned so far.
In Autumn 2023 we published the first set of Teacher Choice feasibility studies, including ‘A Winning Start’, which compared two approaches to lesson starters in Y8 classes. This study, along with others at both primary and secondary level, aimed to assess whether real-world, practice-based questions could be successfully evaluated using a randomised controlled trial (RCT), and whether teachers could integrate a discrete set of approaches into routine teaching without disruption.
Prescription versus flexibility
The findings have left us highly encouraged. First, we found that teachers were highly motivated to take part. Participation in these trials aligned with schools’ professional development priorities to apply relevant, evidenced-based practices, and also supported a desire shared among teachers to directly contribute to the evidence base.
We also learned that short and simple teacher guides were sufficient for teachers to understand the trial, and apply the choice approaches with fidelity. Striking a good balance between strict prescription and flexibility within the guidance provided is no easy feat.
We therefore gathered input from EEF content specialists, as well as practitioners from our Research Schools Network, to distil the guidance down to the core elements of practice, and had teachers use their best judgement when ‘localising’ the approach to their contexts.
Finally, the pilots provided a useful test case for disseminating findings to the wider sector. We want to help teachers quickly digest which approaches are more impactful and find out about participants’ experiences of implementation, so that they can make more informed choices when replicating the chosen approaches in their own classroom.
What needs doing – and what’s to come…
Methodological challenges
The Teacher Choices feasibility trials have thus far underscored the importance of close collaboration with teachers at every stage of the educational research process.
Despite some exciting initial findings, however, there remain some methodological challenges associated with the Teacher Choices trials. For example, running causal evaluations of teacher practices means having to identify specific outcomes that we would expect the practice to influence, and evaluating these with sensitive assessment measures.
When standardised test measures lack the necessary precision, evaluators like NFER have opted to create and validate bespoke assessments, or collect teacher-developed topic tests in order to evaluate the impact of Teacher Choices.
Multidisciplinary expertise
Integrating RCTs into real-world practice entails navigating some level of variation. Not all teachers instruct the same topics in the same sequence, and pupils in classrooms may not be grouped in the same way across settings.
Such differences in contexts challenge our ability to test pupils across settings and produce a standard impact – but they also present unique opportunities for working alongside teachers to try and identify innovative testing solutions.
Every new Teacher Choice project will therefore be guided by a Study Advisory Board comprising methodological, subject and practice experts, who will be able to provide fresh perspectives in order to help inform the evaluation.
Next steps
Three more Teacher Choices evaluations have just completed their scoping phase – including ‘Using Examples to Teach Grammar to Year 7 (Cognitive Science Teacher Choices)’. This evaluation will seek to support teachers in establishing which modelling technique that uses examples is most effective for teaching English grammar.
We’ve also partnered with the Hg Foundation to fund a Teacher Choices trial examining whether ChatGPT-informed lesson and resource preparation is more effective in reducing teacher workload than traditional preparation methods.
Christine Kelly is the EFF’s methodological innovation lead and Faizaan Sami is its evaluation manager; for more information, visit educationendowmentfoundation.org.uk
How to tell the good from the bad
Harry Madgwick and Kirstin Mulholland highlight the pitfalls educators should be wary of when weighing up the merits of educational research…
Educators and leaders are increasingly turning to educational research evidence to help identify promising approaches and practices that have the potential to make a difference to pupils and communities.
However, we know from our experiences of working in schools and teacher professional development that while many products and resources are now badged as being developed based on ‘evidence’, finding the time to access, engage with and actively question said educational research evidence can be extremely difficult.
Leaders are liable to be bombarded with information regarding programmes and resources that make all manner of impressive claims. This is often courtesy of companies and organisations with a vested interest in selling those very same products or services.
So, when faced with the vast quantities of evidence available, how do we know which sources are trustworthy and relevant to the problems we are trying to solve?
The Education Endowment Foundation’s concise guide to using research evidence includes information on what educational research evidence is – and is not – as well as the advantages and potential limitations inherent to different types of research.
We hope the guide is able to help those involved in school improvement, and in the design and delivery of teacher development, to better reflect upon the role educational research evidence can play in educational outcomes and CPD activities.
Above all, we hope it will prompt educators to consider what kind of evidence they’re using, as well as why they’re using it, and the ways in which they ought to apply it. This way, we can harness the collective knowledge of what has worked in the past to make a meaningful difference to teaching and learning in future.
Educational research – a user’s guide
Build up a rich evidence picture
When looking at any area of educational research, it’s important to always consider multiple studies from a range of sources so that you can identify any common themes and trends. Try to avoid ‘cherry picking’ educational research that confirms your existing beliefs, and instead take a broader view of the evidence base.
Drawing from systematic reviews and meta-analyses that combine multiple previously published studies can help with this. For an example, see the EEF’s guidance reports, evidence reviews and Teaching and Learning Toolkit.
Look for variation in findings
Remember that the devil really is in the detail. When engaging with educational research evidence, be watchful for any variations in findings across different studies and what the reasons behind those variations might be. For example, are there any groups of pupils for whom a given approach seems more, or indeed less successful? If so, why?
Focus on the ‘how’ – not just the ‘what’
Look carefully at how someone has implemented a specific approach before deciding on whether you could apply the elements that make that approach effective in your context. The EEF’s implementation guidance report provides some practical recommendations that may be useful when engaging with such questions.
Maintain criticality
Be open to new ideas, but adopt a critical approach to the claims people make. Look out for warning signs, such as claims which seem too good to be true, or vaguely referenced, non-cited mentions of ‘impact’.
The EEF concise guide contains a list of red flags to be mindful of when it comes to the reliability of evidence. You can remember this using the mnemonic CLAIMS:
- Conclusions
- Limitations
- Applicability
- Independence
- Methods
- Sample size
Harry Madgwick is the EFF’s research and policy manager. Kirstin Mulholland is its senior associate for school engagement and evidence use. For more information, visit educationendowmentfoundation.org.uk
