January 2013
| Report: | Legal Capability for Everyday Life Evaluation Report |
| Report commissioned by: | Lisa Wintersteiger, Law for Life |
| Report author: | Liz Mackie, The Gilfillan Partnership |
2.2 Testing the conceptual model of legal capability The project aimed to test the conceptual model of legal capability set out in the legal capability matrix. The intended outcome was that partners would have a better understanding of the domains and elements of legal capability and how they apply to different groups of learners.
Law for Life selected and worked with three advice agencies which provide information and advice to three different groups of people. This approach enabled the project to test the legal capability model set out in the legal capability matrix and to see how it would work with different groups of learners. The project team developed the basic structure for a course to be delivered in six two-hour sessions. The broad course content and some materials were produced before the training course delivery began. But much of the detail and the resources for each session were developed week by week, to be responsive to the interests and level of knowledge for each group of learners.
The project team felt that it was not possible to condense the entire legal capability matrix into a 12 hour programme but aimed to ensure that all four domains were addressed, to a greater or lesser degree. The team consider that this was achieved and that the programme they developed matched the legal capability matrix well. From discussions with the project partners, it is not clear that all of them are familiar with the PLE evaluation framework document, the PLE evaluation framework, or the legal capability matrix, or have had time to consider the value of these in relation to the legal capability of their service users. Only one interviewee from the three partner agencies seemed familiar with the framework and was able to discuss which of the domains were most relevant to their service users.
It is clear from discussions with all three project partners that the domains that they regard as most relevant to their learner groups, whether expressed with reference to the legal capability matrix or not, are Domain 1 (recognising and framing the legal dimension of issues and situations) and Domain 3 (dealing with law-related issued).
2.3 Developing delivery of public legal education within the advice sector
The project aimed to implement and test the PLE evaluation framework in the context of advice agencies, and to improve advice agencies ability to undertake PLE within their local communities. The intended outcomes were that PLE evaluation framework would be tailored to adapt to advice agency delivery of PLE, and that advice agencies would better understand the skills and methodologies that are suited to delivering PLE in their communities.
The project partners had many different reasons for wanting to take part in this project, but a common motivation was the need to respond to cuts in advice services by enabling users to deal with some relatively uncomplicated issues by themselves, without drawing on the increasingly stretched resources of advice agencies. As an example of the need for this, one participant of the course at Paiwand had, prior to the course, come to Paiwand for advice every day. Since taking the course he had sought advice from Paiwand only twice in two months. Paiwand believe that PLE can help many of their service users to become more independent and less reliant on Paiwand’s services. This will help to free up Paiwand staff time to deal with more complex advice cases. And if Paiwand were no longer able to provide advice services, it would lessen the risk that their service users would be unable to cope without their support.
Taking part in the project confirmed to all three partners that PLE is a much needed and valuable approach for their service users. All three partners would like to do more PLE with their clients. DLS has very clear plans for further PLE provision and is actively seeking funding to implement these. DLS had already had some experience of PLE through a previous project in which Law for Life was involved. This previous experience may explain why DLS’ plans for further PLE are more clearly articulated than those of other project partners, and perhaps more likely to go ahead independently of any future programme that Law for Life may co-ordinate.
2.4 Measuring the impact of public legal education
The project aimed to begin to assess the potential wider socio-economic impact, the social benefit, of PLE projects. The intended outcome was that the pilot project would lead to more rigorous data collection to evaluate the socio-economic impact of legal capability. The external evaluation aimed to contribute to this aim and outcome by developing and delivering evaluation methods within the PLE evaluation framework. This section of the report describes the process of conducting the evaluation within this framework and the lessons learned from this.
Evaluation methods
The evaluation used a paper-based questionnaire to collect information about the benefits to individuals of participating in the PLE training courses. This method was adopted because it is the cheapest way to collect information and so potentially has the greatest transferability to other PLE courses, where evaluation budgets are always likely to be very limited.
The indicators used in the questionnaire were derived from the PLE evaluation framework evaluation measures (as shown in Table II). It became apparent at an early stage that it would not be useful to develop an evaluation indicator for each of the 21 measures in the evaluation framework. Not only would this have made the questionnaire very long, but it also proved difficult to sufficiently differentiate the wording for each question so that respondents could easily recognise that each question was in fact different from the previous one. In the end, 13 evaluation indicators were established. These reflect 10 of the 21 evaluation measures and all four of the evaluation domains in the PLE evaluation framework.
The 13 indicators provide a measure of legal capability across the four PLE evaluation domains. Most indicators were presented as questions with tick box response options. A variety of response options were tested, including Yes/No responses and rating scales using four, five and ten point scales. Participants were asked to complete a questionnaire at the beginning of the first session of the course (the before questionnaire) and to complete a similar questionnaire at the end of the final session (the after questionnaire). The before and after questionnaires were identical apart from some additional questions in the after questionnaire which asked participants
(a) if the course had helped them to understand more about legal issues,
(b) what was the most useful thing they learned on the course,
(c) if there was anything about the course that could have been improved.
To ensure that any changes in the before and after responses from project participants could be attributed to their participation in the training, a control group was established. Each project partner identified a group of 10 people who were similar to the course participants but who did not take part in the course. The control groups were asked to complete the before and after questionnaires at around the same time as the participants.
Using paper-based questionnaires
It was recognised from the outset that paper-based questionnaires might be difficult for some respondents to complete, for example those with poor literacy or English language skills. This did prove to be the case, as some of the participants at Paiwand struggled to complete the before questionnaire. This was probably less about English language skill and more to do with being unfamiliar with this type of questionnaire, as the Paiwand participants had no problems completing the after questionnaire, having learned from the previous experience and feeling more confident with the format. Because many of the Paiwand participants had difficulties completing the before questionnaire, they received a lot of help from the Law for Life team and Paiwand staff, and several questions were discussed as a group. The responses to some questions may perhaps reflect the group response, rather than individual views. This may explain why some of the Paiwand questionnaire responses are unexpected, with participants giving themselves very high ratings for confidence and knowledge on some indicators.