The value of evaluation

Sarah Morton, Co-Director (Knowledge Exchange) at CRFR and Director for Evidence to Action for What Works Scotland, was keen to support Evaluation Support Scotland in their 10th Anniversary year by writing this blog on the value of evaluation.

We are moving to a world where ideas about evaluation are embedded into the way we think about the delivery of services. There are a number of reasons for this. Most of the public sector in Scotland is committed to delivering on outcomes, mainly derived from the National Performance Framework. Working towards outcomes shifts thinking from ‘what are we doing?’ to ‘are we making a difference?’. In addition, the squeeze on public spending means that there is much more reflection on ‘are we doing the right things?’

People managing and delivering services are having to get smarter about how they use their own data to measure performance, with an eye on outcomes. But on its own this doesn’t answer the bigger questions about what difference a services or programme makes, for who, how and why. These are the realm of evaluation.

Traditionally evaluation is often seen as an expensive add-on to programme delivery. It is often conducted by external teams and with ‘independence’ from those delivering the programme seen as an essential component of being able to judge its effectiveness. This type of evaluation has become less common with shrinking budgets. The mood is shifting much more to self-evaluation, to monitoring progress through constant feedback, and to using evaluative data to help guide programmes to their outcomes, rather than waiting for evaluation feedback once delivery has occurred. At Centre for Research for Families and Relationships we have been working with organisations to help them do this.

This shift in mood means that everyone, from frontline workers, managers, planners, and commissioners of services need to be thinking in evaluative ways on a day to day basis. Having ways of consulting with communities and service users is a core component of that thinking. There is a real need for more support, training, tools, approaches, theories to support this across the sector. Evaluation Support Scotland is an essential part of that picture, and we are delighted to be working with them through What Works Scotland to develop this essential capacity.

Contact Sarah by email.

The value of evaluation - looking forward

Sarah Morton is Co-Director (Knowledge Exchange) for the Centre for Research on Families and Relationships and the Director for Evidence to Action for What Works Scotland. She has conducted many evaluations, and also worked with organisations to support self-evaluation.

We are moving to a world where ideas about evaluation are embedded into the way we think about the delivery of services. There are a number of reasons for this. Most of the public sector in Scotland is committed to delivering on outcomes, mainly derived from the National Performance Framework. Working towards outcomes shifts thinking from ‘what are we doing?’ to ‘are we making a difference?’. In addition, the squeeze on public spending means that there is much more reflection on ‘are we doing the right things?’

People managing and delivering services are having to get smarter about how they use their own data to measure performance, with an eye on outcomes. But on its own this doesn’t answer the bigger questions about what difference a services or programme makes, for who, how and why. These are the realm of evaluation.

Traditionally evaluation is often seen as an expensive add-on to programme delivery. It is often conducted by external teams and with ‘independence’ from those delivering the programme seen as an essential component of being able to judge its effectiveness. This type of evaluation has become less common with shrinking budgets. The mood is shifting much more to self-evaluation, to monitoring progress through constant feedback, and to using evaluative data to help guide programmes to their outcomes, rather than waiting for evaluation feedback once delivery has occurred. At Centre for Research for Families and Relationships we have been working with organisations to help them do this.

This shift in mood means that everyone, from frontline workers, managers, planners, and commissioners of services need to be thinking in evaluative ways on a day to day basis. Having ways of consulting with communities and service users is a core component of that thinking. There is a real need for more support, training, tools, approaches, theories to support this across the sector. ESS is an essential part of that picture, and we are delighted to be working with them through What Works Scotland to develop this essential capacity.

ESS facilitates networks to share evidence and improve practice.

Safe and Sound

CRFR has completed an evaluation of a project delivered by Shelter and Relationships Scotland aimed at supporting young runaways. The results of the evaluation of Safe and Sound are being launched at an event in Dundee today by Aileen Campbell, Minister for children and young people 

The evaluation was conducted by Sarah Morton, Linda McKie and Christina McMellon. 

Here Sarah Morton highlights some of the key findings:

Working on this evaluation of a service that supports young people at risk of homelessness has been an interesting and rewarding experience. I have rarely been involved in evaluating a project which is so warmly endorsed by those using it, both young people and parents, as well as referral agencies and partners. In the report being launched today we set out the successes and challenges that Safe and Sound faces. In this blog I wanted to highlight a particular part of the evaluation involving young people.

We were using an evaluation approach called contribution analysis to understand how and why the project worked and for whom. As well as interviewing young people, their parents (separately), their support worker, mediator and referral agencies (31 interviews in total), we asked a small group of young people to work as ‘consultants’ to the evaluation. Their role was to meet with us near the start of the evaluation to help develop a picture of how the project worked, and associated risks and assumptions. We had already conducted a workshop with staff to explore these issues  - we wanted to sense-check this with young service-users as this would form the basis of our interview questions.

One of the highlights of the work for me was when Christina and I met with these young consultants to get their views on how the project worked, and why.

We used a sticky wall, coloured pens and paper, kept a regular supply of snacks and drinks, and through a mixture of asking the consultants to write their own comments, and scribing for them we explored these questions:

·       Is “Safe and Sound” good at making sure that people who need the project know about it?

·       How does “Safe and Sound” treat young people and families when they come to the project?

·       What does “Safe and Sound” want young people and families to do differently?

·       What does “Safe and Sound” want young people and families to learn?

·       What difference does “Safe and Sound” make for young people and families?

The framework facilitated a fascinating and wide ranging discussion, where our young consultants could talk about the project, and have control over the extent to which they revealed anything about their own personal circumstances. We made it clear that we were not interviewing them, but consulting them on how the project worked, so we wanted their opinions – if they wanted to share any personal details we agreed these would stay within the room.

I think the overwhelming message I took away from the discussion was that if you are a teenager who has ended up on the streets, or feeling you need to get away from home, it is because everyone has failed you – you are totally alone without a single person to stand up for you. The young people describe how hurt they felt when services that they had turned to for support sent them on their way because of their age. Some of them described some of the highly risky situations they had encountered on the streets, or in their own homes. They all described how the Safe and Sound project workers had filled that void and been someone to stand up for them when no-one else would. That act of getting alongside young people when they were so down and vulnerable was the thing that had started to turn their lives around.

So the Safe and sound project is demonstrating success at helping young people reach safer destinations. But for me it is the softer outcomes that are more profound. Young people identified the following positive outcomes:

·       Project workers had stood up for them, restored their trust in people, had cared when they had stopped caring about themselves

·       They had gained confidence about travelling alone, clothes, appearance and being happier “in their own skin”.

·       They had made friends with other young people in the project, felt less shy and less depressed.

The young consultants also wanted many more people to know about the support on offer through Safe and Sound. Some wished it had been around when they were younger to prevent the worst crises they had faced. Others thought that there were many more young people, at college and in schools, who should know about the project so they could access it when needed. One young service user gave us a specific instruction about reporting on this evaluation:

“You know, this project is, like, literally so amazing. You know they could have saved my life to be honest. So I don't think they get the recognition and the, I don't know, the respect or the rewards that they should. So that's what you people writing the report need to heed.” (Rebecca – a young service user)

We made a short film to provide feedback to all of the young people who had helped with the evaluation, either as interviewees or consultants.
Read the full evaluation report here

Parenting and Support

Given the recent launch of  the Scottish Government's National Parenting Strategy, we asked About Families project manager, Karen Mountney to highlight some of the key findings from the latest report looking at how families seek and experience support from family, friends and formal services.

"We know that parents manage better when they have sufficient and effective support. Social support can buffer the effect of stresses and difficulties which can arise from being a parent or other areas of life such as relationship conflict or financial difficulties, and help to bolster parents’ self-esteem and sense of efficacy. In terms of formal support, effectively engaging parents with support services and interventions is recognised as a key factor in successfully resolving problems and effecting positive change in families’ lives. However, those parents who are most in need of services, including those who lack support from family and friends, are often the least likely to access them.

Key findings from the report include:

• What support is for: how families define their own problems or needs can be different from service providers’ definitions. People weigh up the costs and benefits of seeking support, and using professional family support is sometimes felt or seen as failing by some parents.
• Family and friends: parents see support from family and friends as the natural first port of call. However this can be variable and can bring its own problems. Not wanting to impose, considering grandparents too infirm to care for children, lack of money, negative attitudes from families, having complex support requirements, and life events such as separation are all important factors in the level of support available.
• Barriers to using services: families can face a range of practical, material, social and cultural barriers to accessing and engaging with services. Families affected by disability, those from minority ethnic communities, and fathers, can face particular barriers.
• Family context: parents generally want to receive help if it is appropriate to their needs. Most barriers to engaging with services are not of parents’ making; numerous factors, such as stress, poverty, ill-health, and social isolation, can combine to undermine parents’ involvement in services.
• What do we know about supporting parents and families? Much of the existing research on engaging parents in formal services is from the perspective of service providers rather than those using services. Evaluations of services often assess attendance and completion rates rather than outcomes for parents and families. There is little research which explores the characteristics or perspectives of non-service users. It is not clear how social support can be best enhanced for those parents who need it. Most research relating to parenting and support focuses on white, non-disabled, women. Generally, different family forms are not referred to."

You can download About Families briefings and reports and see updates on development work at: http://aboutfamilies.org.uk/.