Case study: implementing outcomes in evaluation design

• The pilot integrated a financial capability (FinCap) workshop into an existing rugby‑based programme for vulnerable 13-14‑year‑olds across three schools. 
• It aimed to improve pupils' financial capability along with their self‑esteem, self‑control, self‑efficacy and social bonds. 
• The evaluation combined process and outcome elements, using a Theory of Change and outcomes framework to shape measures, data collection and reporting. 
• Mixed methods were used, combining surveys, interviews and analysis of programme data.
• A comparison between three groups enabled the evaluation of the programme's impact.
• The process evaluation showed that the programme was well received, with strong engagement from coaches and facilitators.
• The outcome evaluation showed improvements in young people's financial capabilty, and impacts on self-control, self-esteem, self-efficacy and social bonds.
• Findings were reported against the outcomes framework, supporting clear interpretation and actionable learning for future programme design.

The School of Hard Knocks (SOHK) integrated financial education into its existing sports‑based personal development programme. This programme was for vulnerable 13–14 year olds in three schools across Cardiff and the Valleys.

The financial education content was delivered in partnership with Principality Building Society.

The programme aimed to help young people:

• improve their financial capability
• strengthen self‑esteem, self‑control and self‑efficacy
• build stronger social bonds.

SOHK’s existing rugby‑based intervention involved regular physical activity, mentoring and coaching focused on behaviour, discipline, teamwork and confidence. 

The FinCap intervention was a single, structured financial capability workshop embedded into the SOHK timetable. It was delivered in a fun, engaging and interactive way by trained facilitators from Principality and covered saving, budgeting, keeping track, ways to pay, price comparison and online financial safety.

• Assess the design, development and delivery of the SOHK with FinCap programme to show what worked well and what could be improved.
• Understand what difference the SOHK with FinCap made for young people, SOHK coaches, Principality trainers and schools.
• Explore whether embedding FinCap within SOHK improved confidence, self‑control and financial capability.

• Develop a logic model and Theory of Change to set out expected activities, mechanisms and intended outcomes across financial capability and personal and social domains.
• Carry out process and outcome evaluation using mixed‑methods of data collection: 
  • baseline and follow‑up surveys with young people
  • interviews with SOHK coaches, school staff and Principality facilitators
  • delivery records.
• Three-group comparison design (SOHK with FinCap, FinCap‑only, and a comparison group with neither) to show impact.

• Strong relationships with SOHK coaches, interactive delivery from Principality trainers, and flexible adaptation to school contexts were key to successful engagement and learning.
• Young people showed improvements in financial knowledge, self‑control, self‑esteem and social bonds as a result of FinCap. The strongest gains were in the SOHK with FinCap group.

This study involved process and outcome evaluation. The table below focuses on the outcome element of the evaluation. The outcomes framework was set out in the logic model for the programme.

There is no evaluation plan in this report, something which our evaluation toolkitOpens in a new window recommends to support how outcomes are implemented.

However, the logic model and theory of change identified the intended outcomes, which shaped the evaluation design and reporting structure, addressing the outcome evaluation's core research question.

The outcomes framework set out in the logic model clearly showed the programme’s aims. So, the evaluation was designed to quantify these outcomes.

To implement the outcomes framework, the team:

• developed meaningful outcome measures and 
• designed a data‑collection strategy capable of capturing impacts.

The evaluation team based the outcome measures directly on the outcomes framework in the logic model.

Translating the outcomes into measurable indicators

The evaluators refined the logic model and the related theory of change. The logic model described how the programme ran in practice, while the theory of change explained why and how change was expected to occur.

Together, they clarified the connections between activities (rugby, mentoring, FinCap workshops) and the intended short and medium‑term outcomes. This mapping allowed the team to see exactly what needed to be measured to assess whether the programme was achieving its intended effects.

Developing age-appropriate and accessible survey questions

For the financial capability outcomes, the evaluation drew directly on the Money Advice Service’s Outcomes Framework for Children and Young People (ages 11–14). The team ensured it aligned with recognised financial capability domains.

Content from Principality’s training materials was also incorporated so that questions reflected what was actually taught in the FinCap sessions.

For the personal and social outcomes, the team developed questions capturing attitudes, behaviours and mindsets linked to the theory of change. These included items on emotional regulation, confidence, goal‑setting, relationships and perceived influence from peers and adults.

The evaluation team also adapted the survey tools to suit the needs of the age group. Feedback from the Welsh delivery team highlighted the need for simpler language and more engaging formats, so multiple‑choice questions, open questions and rating scales were added.

This ensured accessibility for young people with behavioural challenges, learning difficulties and low literacy levels. 

Because the programme identified its outcomes early, the evaluation team could design a data‑collection approach that was tightly aligned with the programme’s aims.

The evaluation used a before‑and‑after design

It used identical survey questions at baseline and follow‑up to track outcomes over time. This approach was chosen because the outcomes were about behaviours and attitudes, meaning they were best measured through repeated data collection.

This allowed changes in the outcomes set out in the framework to be measured, and avoided the biases associated with asking participants to self‑report how their behaviours or attitudes had changed - aligning with the recommended approach in our evaluation toolkitOpens in a new window.

The evaluation used a three-group comparison design

This was shaped by the outcomes framework and the core research question. To understand the effects of the FinCap workshop alone, and the combined effects of FinCap and SOHK, the evaluation compared:

• SOHK with FinCap
• FinCap‑only and
• a comparison group receiving neither intervention at the time.

These groups were already available within each school based on pupils' behaviour, eligibility for SOHK and decisions made by the schools on who was included. The FinCap‑only group was expected to be similar to the SOHK group and could potentially join SOHK later.

The comparison group provided a benchmark for typical levels of financial capability and personal and social skills in the same schools. However, they did not show the vulnerabilities associated with the other two groups and included the only female pupils in the study.

Using a comparison group tested whether changes could be attributed to the FinCap intervention. Comparing the two groups who received FinCap – one with and one without SOHK – allowed the evaluation to test whether embedding FinCap within SOHK had added impact.

We discuss the use of comparison groups further in our evaluation toolkitOpens in a new window.

The findings from the outcome evaluation were structured around the outcomes in the original framework. They showed a mixed picture of impacts of the FinCap intervention on financial capability, but stronger and more consistent improvements in the personal and social outcomes when linked to the SOHK model.

This evaluation highlights how implementing an outcomes framework:

• required defining clear outcomes early, which provided a sound footing for designing measures and aligning data collection with programme aims
• mapped the outcome measures needed, as survey questions were developed to inform financial capability and personal and social outcomes, reducing measurement drift
• used recognised frameworks, drawing on the MAS Outcomes Framework and FinCap content, which strengthened its validity
• informed data collection, supporting the need for a before‑and‑after design, and a three-group comparison design in combination with the core research question
• structured the report findings, with results presented clearly for each of the key outcomes.

You can view the full report: Evaluation of the School of Hard Knocks (SOHK) financial capability reportOpens in a new window

You can also read a summary of the report on our evidence hub.