Research as an intervention: A case study on violence against children in Peru

CRFR co-director Dr Sarah Morton, and colleague Tabitha Casey, share some of their findings from the recently published report, which examined the impact of a research programme aiming to prevent violence against children in Peru.

---

Preventing violence against children is becoming more of a focus for policy makers, with the United Nation’s Sustainable Development Goals including targets that aim to eliminate violence everywhere. How governments can achieve this is not so straightforward, however.

In 2014, UNICEF Office of Research- Innocenti and partners launched the Multi-Country Study on the Drivers of Violence Affecting Children in order to help governments understand and address this issue. The Study focused on four countries – Italy, Zimbabwe, Viet Nam and Peru – and partnered with Dr Debi Fry, Moray House School of Education, and Young Lives, based at the University of Oxford. Two years later we undertook a joint impact assessment with UNICEF to examine the impact of this research programme in Peru.

Few impact evaluations are carried out in low-and-middle-income countries (LMIC), especially on violence prevention research. Our aim was to assess the outcomes of the Multi-Country Study in Peru and also to test the Research Contribution Framework (RCF) (Morton, 2015), developed to examine how research uptake and use ‘contributes’ to policy and practice change, in a LMIC.

The RCF requires key partners to agree the main mechanisms through which the research might impact on policy or practice, which are then tested, along with key risks and assumptions. We collated all existing evidence (e.g. media, downloads, email testimonial), assessed the gaps in evidence and then took a field trip to Peru to gather more material. In Peru we conducted a range of stakeholder interviews with government, academic and UNICEF partners to assess whether the Study contributed to changes in awareness, knowledge, skills, behaviour, policy and practice among key stakeholders.

The impact assessment found that the multi-partner, relationships-driven approach of the Multi-Country Study helped to maximise impact in Peru. It was also important that the Study was nationally-led: this made sure that the findings were specific to Peru, which was key, considering the country’s geographic diversity and multi-culturalism, and also helped ensure national ownership and data sovereignty.

By using this approach, the Multi-Country Study contributed to a number of policy and programme changes in Peru, including the passage of a law banning corporal punishment in all settings. By focusing on capacity-building and awareness raising activities, the Study also helped to give violence against children a higher political priority, and study partners are committed to keeping it on the political agenda.

Recommendations from the impact assessment include:

• Plan sufficient timing – as well as flexibility - into the research design in order to build effective, trusting partnerships.
• Define and support key actors to take knowledge-brokering roles in order to ensure clear communication and engagement among study partners in complex projects.
• Plan an impact strategy from the start. Strategies should address complexity from the beginning, and also identify key monitoring criteria as well as risks and assumptions.

The RCF used here could be used both as an evaluation method and a planning tool to develop an impact strategy. Through this impact assessment, we found that the framework is adaptable and effective in a LMIC, and could be used to assess research impact in other contexts.

---

For more on the Multi-Country Study on the Drivers of Violence Affecting Children, visit the study site at: https://www.unicef-irc.org/research/274/

More information about the Peru study can be found here (in Spanish): https://www.unicef.org/peru/spanish/resources_33614.html

Sarah Morton (2015) Progressing research impact assessment: A ‘contributions’ approach. Res Eval 2015; 24 (4): 405-419. doi: 10.1093/reseval/rvv016

Getting evidence into action: how can we understand what we already know?

Our Co-director, Dr Sarah Morton writes about the processes involved in setting up an Evidence Bank to support public and voluntary sector partners in accessing existing research evidence to help to inform decision making.

At the Centre for Research on Families and Relationships we have always valued working closely with non-academic partners. At the end of one particular research partnership with national family organisations (Cool with Change) we started a conversation about what other research the partnership might do together. Many of the topics our partners were interested in were areas where there was already a lot of research, but it was very difficult for these non-academic agencies to get a sense of what that research was. So we set out together to develop ways to make existing research more useful.

We explain the reviewing process we developed to address that issue in an article just published by the journal Evidence and Policy (Evidence synthesis for knowledge exchange: balancing responsiveness and quality in providing evidence for policy and practice). The process focussed on making sure that evidence was useful, timely and relevant, and could be put into action by people working in children and family services.

Why is evidence reviewing important?

Often people need to understand what we already know from research in order to develop services. Traditional systematic reviewing is time consuming and often doesn’t offer very accessible summaries. Our evidence bank process sought to address these issues.

What kinds of topics were reviewed?

We developed a range of ways of interrogating what partners wanted to know from research in order to define review topics. This involved visiting and revisiting questions like:

• What is the general topic area you are interested in? 
• What is the issue or problem you are trying to address? 
• What do you want to know about this issue? 

But we also explored the potential uses of the evidence through questions like:

• What do you plan to do with the data/report? 
• Who will the data/report be relevant to within and out with your organisation? 
• What difference do you hope that using the data/report will make?

In addition to spending time exploring what people wanted, we also created a scoping stage, where we would look at the literature in general terms, get a sense of its size and scope, and return back to a discussion of what the specific focus should be.

Through the process we looked at: Parenting Teenagers; Transitions to Primary school; How families have changed; and other topics.

What was different about the reviews?

As well as being tailored very specifically to real world practice issues, the reviews have a number of distinctive features:

• They explain how the topic review fits in with the wider literature and which subjects and disciplines it comes from, to give a sense of the ‘evidence landscape’ in which it sits.
• The scoping and final reports are peer reviewed by an academic and a user reviewer who has practical experience and knowledge of the field.
• The reports are edited by a knowledge exchange expert, written in plain language with a short summary. 
• Reviews contain ‘talking points’ that focus on the practical implications of the evidence presented, and aim to move thinking away from the idea that research can tell you what to do, towards the idea that research provides a framework to help decision-making. 
• They are conducted in a shorter time frame with involvement from partners throughout.

How have the reviews been used?

We worked closely with partners to explore how to make the reviews useful. We tried and tested a number of approaches, from large forums, communities of practice, and linking with improvement approaches. They have led to some specific actions, for example:

• In West Lothian a review of transitions to primary school was used to test and embed new approaches to supporting children and families.
• One of the partners from the About Families programme used the evidence to develop new ways of including parents in the design and delivery of services. 
• In What Works Scotland a review of partnership working has been used by local authorities to reflect on and improve the effectiveness of partnerships for public service delivery.

What next?

The Evidence Bank review process has been taken into What Works Scotland and is being used there for a number of reviews. 

Action! Young people as researchers and decision-makers

Research Fellow and City of Edinburgh Council youth worker Christina McMellon reports on exploring the Young Edinburgh Action approach to effective participation.

Young Edinburgh Action (YEA) is an innovative approach to implementing the City of Edinburgh Council’s Young People’s Participation Strategy, the aim of which is to facilitate young people’s meaningful participation in partnership working and decision-making. YEA supports young people to identify and research issues that matter to them in Edinburgh. The findings from action research carried out by young people are shared with senior decision makers at ‘conversations for action’ where plans are developed based on the young people’s recommendations for improving services.

The YEA approach is informed by the views of young people and professionals, academic theory and the need for local government to evidence the work that they do. YEA has developed a partnership with the Centre for Research on Families and Relationships to share the learning from this approach. This partnership, headed up by Professor Kay Tisdall from University of Edinburgh, was awarded a Knowledge Exchange and Impact Grant by The University of Edinburgh’s College of Humanities and Social Science to create a short film about young people’s experiences of being a young researcher. The film will be used to stimulate discussion among academics and practitioners about the role of young people as researchers and about young people’s participation more broadly.

“YEA: It’s a Film” takes viewers through the YEA action research process, including a focus upon impact, and gives a range of practical examples. One such example is of a group currently working on gender inequality: They held a research event for 60 young people who agreed that it is important to challenge gender stereotypes at an early age. The group decided to write a storybook for children aged 4-6 and worked with an illustrator to develop a story called ‘Alex and Charlie’ that looks at a day in the life of two young friends. After a conversation for action where the group members presented their work, City of Edinburgh Council agreed to underwrite the costs of publishing the book and gifting two copies to every nursery and primary school in Edinburgh. The process is not, however, always straightforward and the film does not shy away from stating the challenges (and benefits) for young people of being involved with the project.

The film is accompanied by a CRFR briefing paper written by Council staff that outlines the YEA approach: Young Edinburgh Action: Reinvigorating young people's participation in Edinburgh.

Young people are currently working on their own briefing paper which will complete a trio of resources exploring the YEA approach and stimulating meaningful discussion about young people being researchers.

===================

Links:
Watch 'YEA:It's a Film' online here
Download the CRFR Briefing 'Young Edinburgh Action: Reinvigorating young people's participation in Edinburgh' here

Five tips for getting knowledge into action

This post, written by CRFR co-director Sarah Morton, was originally published on the Knowledge Network for Applied Education Research, (KNAER)/Réseau d’échange des connaissances pour la recherche appliquée en éducation (RECRAE) blog.

I think we are at a good time in the knowledge mobilisation field. We have built a body of research that explains a lot about what helps and hinders knowledge getting used to aid decision making in policy and practice. (see Oliver 2014). We are developing rewards and incentives to help academics get research out of the academy (e.g http://www.rcuk.ac.uk/innovation/policies/), and we are at least paying lip service to the idea that policy and practice should be informed by the latest evidence (KNAER is a good example).

Despite this, there are still many pitfalls along the way, and it is often easier to identify where things went wrong rather than success stories. The five tips I have identified below will not surprise many readers of this blog, and yet they are often the pitfalls that I see in practice. I’d really welcome any comments from readers about whether these are issues that you see in your practice, if I have missed something major, or if you disagree completely!

Five tips for getting knowledge into action:

1. Plan ahead

Any good project or process involves careful planning, but how often is evidence-use included in the plan? If researchers want their research to have impact, a well-planned user engagement and KMb strategies have been shown to be effective (http://www.esrc.ac.uk/files/research/evaluation-and-impact/taking-stock-a-summary-of-esrc-s-work-to-evaluate-the-impact-of-research-on-policy-and-practice/). On the policy or practice side, valuing evidence, showing leadership and embedding evidence into organisational practices are all key.

So what would a planned evidence use process look like? For those from policy or practice it might consider how evidence will frame any project or development (http://www.crfr.ac.uk/assets/CRFR_ESS_IS_Evidence_base_briefing.pdf) , how it will be considered and built on, what will be done when people don’t agree on what the evidence says, and how evidence will be accessed, analysed and interpreted. For research teams and partners it would consider who will be engaged and involved, what methods are best for engaging stakeholders and how the research might contribute to change. This needs to move beyond simple ideas of making research accessible, into more complex and process focussed projects. (I have written about this much more extensively here http://rev.oxfordjournals.org/content/24/4/405.abstract)

2. Get the right people round the table

In our evidence to action projects About Families (http://www.crfr.ac.uk/projects/current-projects/about-families/) and the Evidence Request Bank (http://www.crfr.ac.uk/projects/current-projects/evidencebank) we learnt a lot about who is involved in evidence-use processes. Like others taking a systems-thinking approach (e.g. Best and Holmes 2010) we believe that it is essential to include a range of key actors in any knowledge mobilisation process. This would include considering the skills mix of any team in accessing, interpreting and animating evidence of different types. Any systems change also needs to include the perspectives of all key players within the system. Depending on the size of the change project these views might be represented in person, or through consultation of various kinds.

3. Have the conversation

Often the starting point for evidence use projects is the evidence itself, but there are a variety of discussions and framings that are essential for evidence to action. What is evidence needed for? What kinds of evidence might be useful? How will they be interpreted? How will evidence inform change processes? Who needs to be involved? Research doesn’t speak for itself so relationships are key to evidence to action. Effective facilitation of knowledge mobilisation needs an ongoing sense of open dialogue, regular revisiting of planned aims, interrogation of context, and keeping the conversation going about the usefulness and relevance of evidence. We worked with Research Impact (http://researchimpact.ca/) and NCCPE (https://www.publicengagement.ac.uk/) to develop a manifesto for partnership research that can help frame some of this conversation http://www.crfr.ac.uk/manifesto-for-partnership-research/.

4. Focus on the process

Using evidence is not a one-off event, but an ongoing process. If people feel they have ‘done’ knowledge mobilisation then they are missing a trick. Using and reusing evidence, checking as programmes develop, and building up more evidence as events unfold are all essential parts of successful knowledge mobilisation. An ongoing focus on the processes can open up new opportunities, ensure ground is not lost, help address conflict and tension, and assess changing contexts and their implications for KMb. Overall a focus on processes helps to ensure knowledge mobilisation continues to be as effective and relevant as it can be.

5. Learn, evaluate, review

I said in the opening of this blog that we are in a good place in terms of understanding barriers and facilitators to knowledge mobilisation (although a recent review is opening up this conversation http://www.nesta.org.uk/publications/using-evidence-what-works). We are in a less clear place about what strategies and methods are most effective in which circumstances (http://www.ncbi.nlm.nih.gov/books/NBK299400/). As a community of knowledge mobilisers we need to develop evaluation methods, reflect more deeply and write up what we find out. My own approach to this has been published here http://rev.oxfordjournals.org/content/24/4/405.abstract. Every project needs a learning, review and evaluation process, even if on a simple team scale. As the field matures this will be essential in honing the craft, creating training programmes and developing the most effective strategies.


So those are my five tips for getting knowledge into action. How do they resonate with your own experience? What might you add? What resources do you use? I look forward to continuing the conversation: leave a comment below or find us on Twitter @CRFRtweets.

----
Best, A. and B. Holmes (2010). "Systems thinking, knowledge and action: towards better models and methods." Evidence & Policy: A Journal of Research, Debate and Practice 6: 145-159.

Evaluating the Young Mothers’ Service using contribution analysis: lessons learned

From 2014 to 2015, CRFR's co-director Sarah Morton and PhD student Cara Blaisdell worked with West Lothian Council’s Early Intervention Team to develop a framework for evaluating their suite of services, using contribution analysis. In this blog post, Cara reflects on their work with one of these services - the Young Mothers’ Service. This is a keyworking programme in West Lothian that supports pregnant and parenting women under the age of 25.

What is contribution analysis?

In a contribution analysis approach, evaluators identify the outcome that a service hopes to improve, and develop a ‘theory of change’ about how the service will bring about that improvement (Better Evaluation 2015). This theory is expressed visually, in the form of a results chain showing how each stage of the project links to the next. Our CRFR briefing on the project goes into more detail about how we used contribution analysis in this evaluation. Other resources on contribution analysis are linked to at the end of this post.

 

What were the results of the evaluation?

The evaluation showed positive results about how the keyworkers supported the young women. Feedback from young mothers indicated that they found their keyworkers supportive and had improved confidence:

‘They took everything I was going through into consideration.’

‘I’m very much more confident by myself and all the support I needed was met’.

‘Don’t think I’d be where I am today without my keyworker’s help. Thanks a lot!’

Other results from the evaluation showed:

• A high level of take-up of the service, with only one young mother declining when the service was offered 
• Young mothers experienced reduced depression and anxiety 
• 10 children removed from the Child Protection Register 
• Mothers were more aware of support that was available to them—for example, help with accessing benefit entitlements, help with finding a secure place to live, or attending local parenting groups for peer support 
• Eight mothers were supported to enter full-time employment, with 14 others working toward this goal
• All young mothers were approached about experiences of domestic and sexual abuse—revealing a 75% prevalence rate. Support was offered in partnership with the Domestic and Sexual Assault Team.

 

What did we learn about using contribution analysis?

• Working collaboratively with keyworkers and the service manager put the emphasis on learning through evaluation, rather than external judgement.  
• The service manager and keyworkers said that the evaluation prompted useful reflections about their work.  
• Training in contribution analysis and ongoing support were important. 
• Collecting quantitative data was straightforward, but it was more challenging to compile qualitative data in a systematic way. 
• We needed to work within the constraints of high workloads. 
• We needed to offer multiple routes for service users to give feedback, including finding ways to take into account the informal feedback that arose from relationships between young mothers and keyworkers.

For more about this evaluation, and further reflections on what we learned about using contribution analysis, see CRFR Briefing 79: Evaluating the Young Mothers’ Service using contribution analysis: lessons learned.

Additional (open access) resources about contribution analysis: John Mayne, who developed the approach, has written a short briefing about contribution analysis. Sarah Morton, who has developed a specialism in contribution analysis at CRFR, has spoken and blogged about her experience with the approach. There are also useful literature reviews available from Better Evaluation and IRISS.

Progressing research impact – a ‘contributions’ approach

Sarah Morton has been working in knowledge exchange since 2001 and has recently published a framework for assessing research impact based on contribution analysis. Here she talks about how her approach not only provide impact analysis, but helps improve knowledge exchange with a focus on making a difference for people and communities. http://rev.oxfordjournals.org/content/24/4/405

My interest in assessing research impact rose from working on getting evidence into action. With colleagues here at the Centre for Research on Families and Relationships (www.crfr.ac.uk) over the last 15 years I have explored how to make research accessible, useable and meaningful to those working with children and families, or making policy on their behalf. Knowing whether or not we were making any difference was a logical extension from that work.

We have a small team of KE professionals working closely with our research team at CRFR. We have worked around the knowledge exchange agenda through an artist in residence (Levehulme), a three year project working with the voluntary sector to try to close the knowledge to action gap, a six year contract working closely with analysts in the Scottish Government, a dissemination post attached to the Growing up in Scotland (http://growingupinscotland.org.uk/) study, as well as well-planned programmes of engagement and dissemination for each of our research projects. I became interested in impact assessment because I wanted to know more about if we were making a difference, and if so, in what ways.

But assessing the impact of research on policy and practice is not easy. Our approach at CRFR recognises that research users are not passive recipients of knowledge, but that they engage with research from their own perspective, and that complex relationships and networks of researchers and research users are often channels through which research is communicated, debated, utilised, reworked, reused and developed. To understand impact we need to consider changes in policy and practice, changes in people’s knowledge and understanding of an issue, as well as the broad range of potential areas of influence, all highly context dependent. All of these factors make understanding and assessing research impact difficult, as I discuss in more detail in my paper.

I was delighted to be introduced to Contribution Analysis by Erica Wimbush, then Head of Evaluation at NHS Health Scotland (http://www.healthscotland.com/), on a visiting scholarship to the University. Many things about it seemed to make sense for research impact:

• A focus on who is engaged and involved, in what ways, and the learning needed to underpin change.
• The concept of ‘contribution’ is more appropriate for research which cannot cause change on its own, but need to be facilitated by a receptive environment and the actions of change agents
• A way of linking activities to wider outcomes, with a focus on processes, to create a strong story of how impact occurs.

I used this approach to develop the Research Contribution Framework, within a study that looked at what research impact was and how it happened. (http://www.ingentaconnect.com/content/tpp/ep/2015/00000011/00000001/art00004) Since developing it we used it in various ways, including with CRFR’s research, ESRC genomics investments, a participatory research programme at the University of Leicester (http://www.crfr.ac.uk/briefing-assessing-research-impact/), as the basis of a four star REF impact case study (http://impact.ref.ac.uk/CaseStudies/CaseStudy.aspx?Id=23984) and to plan impact for the ESRC/MOD Future Reserves Programmes (http://www.future-reserves-research.ac.uk/).

At is core the framework sets out a pathway to impact, assessing for risks and assumptions and gathering evidence to show that it is robust. It can be used to plan and assess knowledge exchange as you go, retrospectively to assess impact, or in combination. It is highly adaptable to different timescales and contexts. It works best as a tool for teams, ideally including research users, to plan and align activities that have the most chance of really making a difference to communities and people. At its most simple it breaks down impact into three processes:

• Research uptake: research users have engaged with research: they have read a briefing, attended a conference or seminar, were research partners, were involved in advising and shaping the research project in some way, or engaged in some other kind of activity which means they know the research exists.
• Research use: research users act upon research, discuss it, pass it on to others, adapt it to context, present findings, use it to inform policy, or practice developments.
• Research impact: changes in awareness, knowledge and understanding, ideas, attitudes and perceptions, and policy and practice as a result of research

By thinking through these processes, a pathway with potential measures can be set out:

So at CRFR we now work with research teams or advisory groups to set out this kind of pathway for all of our new projects. The more perspectives, views and knowledge about the context for impact the easier it is to explore. We use this framework to help plan our KE activities, and we track as we go, allowing us to assess effectiveness early on, and adapt when approaches don’t seem to be working. We can see the impact story unfold in different ways, and if unexpected impacts occur, incorporate these into our pathway. Some of the tools we use for doing that are on our website (http://www.crfr.ac.uk/using-research/knowledge-exchange-training-and-resources/).

This approach sets out a linear pathway to impact, but this does not imply that research uptake, use and impact happens in a linear way. There may be several threads running through an impact story at different timescales, and looping back and forth between the categories. This is discussed and set out in the paper in more detail.

What’s great about working in this way, is that it helps to focus on why carry out KE activities, who they will engage and how, what they need to do differently for change to occur. It can help identify partners and steer resources towards where they can be most effective. Most of all it is usually an enjoyable way of getting teams together to explore what difference they might make through research and knowledge exchange.