January 2013

Report: Legal Capability for Everyday Life Evaluation Report
Report commissioned by: Lisa Wintersteiger, Law for Life
Report author: Liz Mackie, The Gilfillan Partnership


The evaluation indicators

It is apparent from observing the evaluation questionnaires being completed, and from the completed questionnaires themselves, that some questions were clearer for respondents to understand and could be answered more readily. Fewer people responded to the open questions (those which asked for a written response) than to the closed questions (those with tick box response options).

There was very little movement on the before and after responses to some indicators. With hindsight, it is apparent that the wording of these questions did not link them closely enough to the content of the PLE course, and so they were poorly suited to measure any changes as a result of participating in the course.

An assessment of how well each indicator worked is presented in Table IV.
Table IV: How well did the evaluation indicators work?


Cognitive bias

It seems likely that there are strong elements of cognitive bias in the responses on some indicators which distort the evaluation findings. Respondents appear to have rated themselves more positively on the before questionnaire than they would if they had the knowledge of legal issues which they then acquired through the PLE course. The before questionnaires were completed at a time when most respondents had no knowledge of the subject they were being asked about, but did not yet recognise this lack of knowledge. Taking the PLE course helped participants to better understand what constitutes knowledge of law-related issues.

An example of this is the evaluation indicator ‘Do you understand the difference between civil and criminal law?’. In the before questionnaires, 56 per cent of respondents said ‘yes’. But it was clear from observing the discussions of this topic during the first PLE sessions that the great majority of participants did not understand the difference between civil and criminal law. So the 56 per cent positive response on this indicator before the course took place is likely to be positively biased, as some respondents thought that they understood the difference when in fact they did not.

This pattern of positive bias may have occurred on most indicators. The results of the before questionnaires, therefore, probably over-estimate the legal capability of people who have not yet participated in PLE. The after questionnaires, completed when people have greater knowledge and understanding of law-related issues, probably more accurately reflect the legal capability of respondents.

For the current evaluation, this cognitive bias means that the achievements of the PLE course in improving legal capability have not been fully captured, as the improvement from the before to the after position is highly likely to be greater than measured.

Managing the evaluation process

The evaluation aimed to use a robust methodology but also to be straightforward for partner agencies to manage. Two separate sets of guidance notes were produced for project partners, one giving information about the external evaluation process and timetable and the other giving detailed instructions for administering the participant and control group questionnaires.

All three project partners reported that the evaluation process was straightforward and easy for them to implement. However, there were a number of issues with the evaluation questionnaires which suggest that the process was somewhat more problematic. These issues include:

  • Printing the questionnaires. The questionnaires were designed to print on to four sides of A4 paper. Several questionnaires were wrongly collated, so that page 4 followed page 2, which made the question order confusing and more difficult for respondents to answer. In many cases the pages were not stapled together and some completed pages were lost 
  • Coding the questionnaires. The instructions asked for each questionnaire to be coded in a way that would enable the before and after responses from the same individual to be matched but for individual respondents to remain anonymous. The simplest way to do this would be to put the individuals’ initials in the appropriate box at the top of the form. However, a large number of questionnaires were returned with no identifying code on them. Although 32 participants completed the before questionnaire and 32 participants completed the after questionnaire, only 21 before and after pairs could be matched. It is likely that there are more matching pairs within the completed questionnaires, but there is no way to identify these. The final number of matched pairs was too small to enable any meaningful analysis at this level.
  • Control group questionnaires. Although all three project partners reported that there were no difficulties in identifying control groups and administering before and after questionnaires with them, only one partner managed this aspect of the evaluation. It is not clear why other partners found this difficult to achieve. It seems likely that they were just too busy to spend time doing this.

Assessing the potential wider socio-economic impact

The legal capability matrix is focused on measures of legal capability and does not include any socio-economic measures. The PLE evaluation framework document discusses what the wider impacts of increased legal capability might be and how these could be captured through evaluation. The possible wider impacts from PLE which are identified include:

  • Preventing law-related issues arising or escalating;
  • Improving access to justice;
  • Improving individual health and well-being by reducing stress and anxiety;
  • Improving productivity, e.g. through reducing absenteeism due to stress and anxiety.

The legal capability indicators developed for this evaluation can be linked to some of these socio-economic impacts. Preventing law-related issues arising or escalating, for example, can be linked to the indicator ‘How confident are you that you understand your legal rights and obligations?’; improving access to justice can be linked to the indicator ‘How confident are you that you know when you need to get expert legal help to deal with a situation?’.

To identify whether there is any link between participation in PLE and these socioeconomic outcomes, some longer term follow up of PLE participants would be required. A longitudinal approach would revisit PLE participants after perhaps one year or two years to ask them again about their legal capability. Some additional indicators would be required to assess the impacts from PLE related to reduced stress and anxiety, as no indicators for stress and anxiety were included in the current evaluation. It would be valuable to explore methods of measuring these outcomes and for identifying any causal relationship with participating in PLE. This would be an ambitious research project requiring a longitudinal approach to track PLE participants (and non-participants) over a number of years to assess their outcomes on these or other socio-economic indicators.