Evaluation

Evaluation of the work of the Core Skills Development Partnership over the period 1995-2004 has been undertaken in a number of stages.

  1. 1996-97: Internal review of the partnership’s systems and structures undertaken by PIEDA on behalf of Birmingham City Council; Government Office West Midlands and the (then) government Department of Environment, Transport and the Regions. This checked our fitness for purpose as a scheme undertaking development using the government’s Single Regeneration Budget funding stream.

    The evaluation established that we had all the necessary systems and structures in place to be able to:
    • account for spend
    • report against milestones
    • count beneficiaries, outputs and other measures used to establish the scale of impact being made
    • identify the extent to which benefits were accruing to specific communities
    • know the baseline from which we were changing things; and held a planned set of long term outcomes aimed for

  2. 1997-98: Birmingham City Council commissioned an interim evaluation of our processes, and of the progress being made. This established that we had:
    • procedures in place to commission change
    • annual ways of reporting the quantitative and qualitative progress being made, against annual targets, in shifting from the baseline positions in 1995 to our longer term (2003/4) planned goals
    • adequate ways of ensuring there was no double counting of benefits and beneficiaries
    • ways of establishing the extent to which the initial changes were due to our investments/interventions (as opposed to other activities/interventions)
    • ways of analysing ‘what works’ rather than simply responding to bids for activity support
    • the competences expected as managers of a change/regeneration programme.

  3. 1999: Inclusion in a national review to establish the extent to which the learning policy objectives of the Department for Education and Skills were being met via application of regeneration/renewal budgets. This national work (undertaken by GfA Consulting) was reported in the DfES document ‘Learning Elements of the Single Regeneration Budget’.

    In addition to writing up the Core Skills Development Partnership as a case study worthy of close consideration, the review established that:
    • activities are rooted in a strategy
    • the right partners are engaged
    • robust baseline research can ensure that solutions are addressing the right problems
    • and recommended that the DfES should look at the Core Skills Partnership (and any other similar models) to identify things that can be built into wider processes.

  4. 2000-01: Part 1 of an external evaluation of the Partnership’s work that we commissioned from GfA Consulting, to build on the work already described above in a cost effective way. This part of the longer term evaluation looked at the range of interventions/activities for change being funded or supported by the Partnership. Each of these activities had its own evaluation (internal or external) as a condition of support. The GfA Part 1 Evaluation established the extent to which these were:
    1. rather self-serving, internal, low level collection of views from satisfied participants
    2. contributions from wider scale, national evaluations
    3. external reviews seeking to pull out the lessons to be learned.

      Overall, this GfA ‘evaluation of the evaluations’ established that, at the activity level, the Partnership:
      • relied on a range of mechanisms and thus got a range of perspectives which could be triangulated with each other
      • gained sufficient detail from evaluations to be able to assess which activities were succeeding against longer-term objectives
      • could draw ‘lessons learned’ from the evaluations

  5. 1997-2001: Internal annual reviews.

    Each year one aspect of the Partnership’s operation was selected for an internal review, in the form of a formal structured internal discussion with partners against a set of criteria relevant to the aspect under review. In this way the following aspects were reviewed:
    • the structure and effectiveness of the Board
    • the structure and effectiveness of the Strategic managers/wider partners group
    • the mechanisms of attaching development expertise from partners to the work of the partnership
    • the evaluation framework
    • the communications strategy
    • the vision and development framework

  6. 2002-03: Part 2 of the GfA external evaluation. This focused on issues such as:
    • value for money
    • effectiveness of the ‘Core Skills approach’ i.e. the specific (and relatively unique) approach to system-wide change that was being adopted
    • deadweight; displacement; wastage of resources.

      The headline findings from this evaluation were:

      Achievements
      “Uniquely in Birmingham in my experience, the partnership actually set its own agenda. The partnership was therefore able to maintain ownership of its own strategy and priorities, in spite of its reliance on SRB funding.”
Headline Finding: The CSDP has represented very good value for money
Evidence summary
  • Scale of outputs
  • Net cost per unit of output
  • Process benefits, and related development of capacity
  • Strategic fit
  • Contribution to understanding the baseline, and related strategic issues
  • Minimal waste in the business processes for the partnership
  • Relative costs of administration and management, both low

“The decision to work through strategic partners was genuinely developmental. At the inception of the partnership, there were counter pressures – to spend, to view the CSDP as “non core business”, to carve up the cake.”

Headline Finding: The CSDP has had a lasting impact on the systems for the provision of core and basic skills training & education
Evidence summary - Contribution to the improvements in the baseline
- The scale of change in the baseline and the pace of change
- The increasingly cross-sectoral and inter agency focus
- Investment directly in the voluntary and community sector so that it could develop systems around core and basic skills
- Shaping the use of the standards’ fund, and complementing its priorities
- Enabling not delivering

“We avoided the institutional politics of “traditional” SRB schemes. Rather than that we developed an in built leverage model, under which the priorities for the investment of the CSDP funds were overwhelmingly developmental. Not only did this mean that there were few disputes about additionality – we only wanted to add value. It also meant that the involvement of partners and activity managers was incentivised. The CSDP resources meant that they were better equipped to get the best out of their own resources.”

Headline Finding: The CSDP has been a working model for the development and demonstration of how to plan and deliver lasting change in public services
Evidence summary

 - Start with a plan and then find the money
- scale of investment in infrastructure
- scale of investment in professional practice development
- embedding of core and basic skills in partners’ strategies and service plans
- establishing and maintaining the focus on development investment, rather than simply funding more activity
- development of the business planning and activity design processes
- embedded policy priority of core & basic skills in all of the partners’ delivery systems
- Augmenting the professional practices of front line staff

“In every Area Based Initiative in the City so far, funding has become the story, even if it was not the story at the outset. In the CSDP the opportunity for system-wide change was the story that prevailed.”

Headline Finding: The CSDP has raised the game and set the pace of change
Evidence summary - Commitment to rational but stretching goals in achieving outcomes
- The process of setting targets and reviewing progress in themselves
- Self belief in the importance and urgency of achieving goals
- Credible capacity to influence national policy developments
- Only added value

Lessons
“There was structural facilitation early on in the process….there were no competing visions at the outset…mainly because the board members provided perspective and functional leadership, rather than represented established interests. The fact that core and basic skills was no one’s statutory responsibility helped enormously with this process.”

Headline Finding: Strategic leadership and strategic facilitation have been critically important
Evidence summary - Clout
- Consistency and seniority of leadership
- Commitment to lead, rather than simply manage the programme
- Recognition of the importance of facilitating strategic progress (getting rid of blockages, showing that things can be done, picturing the benefits, simplifying things where they can be)
- Insistence on 4 strategic projects, approved once

“In the early stages there simply was not enough awareness of the rationale and policy justification for the strategies. Developing strategic awareness was therefore the first task in engaging with the voluntary sector …. Perhaps with all of the sectors.”

Headline Finding: Clear priorities and flexible activities
Evidence summary - Long term drive, which is self evaluative
- Clarity about what needed to change
- Openness to different approaches if they were plausible ways of making changes
- Reading big picture and joining up the story locally
- Lots of interventions: no magic bullet

“There was a strong focus on transfer of practice, working across institutional boundaries, working across phases.”

Headline Finding: Plausible relationship between scale of resources and nature of the problems being addressed
Evidence summary - Scale of resources invested over the 7 years
- Ease with which the resources could be accessed
- Scale of outputs
- Knowing problems locally (both supply and demand side) and
- Knowing what national government wanted

“We wanted to avoid the dreaded project appraisal. It took 9-12 months for the four original projects to get appraised and approved by the regional office. The leading members of the Board insisted that this was done in this way. The impact of this one technical detail on the subsequent efficiency and effectiveness of the scheme overall is hard to over estimate.”

Headline Finding: High levels of performance and sustainable processes
Evidence summary - Year on year delivery targets achieved
- Escalating volumes over the first 5 years
- 3rd party assessments and audits all squeaky clean
- resilient organization: both performs and is sustainable

“Not simply project managers…Credibility in their fields….Staff development and facilitation skills….Strong regional and national networks – both were inter dependent….There was a strong set of practitioner networks and policy networks…strategic and practitioner facilitation…”

Headline Finding: Systems met standards, but with a light touch
Evidence summary - Light touch business planning and project appraisal
- Decisive, low fuss (if not no fuss) project development practices
- Capacity to accommodate informality and rigour
- Reliable SRB administrative managements

“Fertile ground and framework….based on wholesale adoption of new initiatives…not piecemeal, not prepared to be piecemeal (but willing to experiment)….working with whole services”

Headline Finding: Working through partners in order to change the mainstream
Evidence summary - Governance was independent of the partners, but activity planning and development was not
- The development functions were distributed throughout the partner organisations

“Evident understanding of the issues…not delivering, enabling…commitment to change – willingness to lead actions at all levels in the partnership”

Headline Finding: Credible project managers who are more than project managers
Evidence summary - Exemplary project management practices by majority of activity managers
- Engagement of wide range of parties from different professional communities
- Kept partnership at the ‘cutting edge’ of policy
- Widespread learning as a by product of involvement

“Respected, and respectful – determined to “challenge”….the best SRB scheme I have ever encountered”

(The full report can be found in the Documents section of this web site)

7. 2003-04: Part 3 of the GfA external evaluation. This work was commissioned, at the midpoint of the Partnership’s long-term drive for change across the city. It was designed to:

  • highlight insights/lessons learnt re effective renewal/regeneration
  • produce some ‘how to’ guidelines and possible tools that may have wider applicability
  • to identify the features of the Core Skills approach that was making it successful in the context of a large complex city.

    Headline findings from this evaluation were:

    You need to get your thinking straight
Headline finding & lesson:
You need to have thought through (and shared) theories of change underpinning the strategic decisions made by the partnership.
Evidence summary - CSDP used several different theories of change to help model their core purpose
- Several theories of change helped, because of the range of interests that needed to be engaged, and the changes in the context for the partnership over 7+ years
- Having several theories (not at the outset) but over the 7+ years, helped to keep the strategic focus relevant and credible to all the partners

Headline finding & lesson: Baseline 2 things – under lying conditions and the state of health in partners’ planning & delivery systems.
Evidence summary - Baselining the under lying conditions for core and basic skills was vitally important.
- Equally significant was the explicit assessment (by the Board and Senior Managers group) of the changes needed in partners’ service delivery systems in order to shift the under lying conditions.
- The CSDP experience suggests that you need to match these up with one another so that change in systems drives changes in baseline conditions (outcomes)

Headline finding & lesson: Develop the partnership with delivering change and making a difference to outcomes in mind
Evidence summary - The CSDP was not afraid to challenge itself and its members. ‘Storming‘ – albeit agreeable debate - was a feature of the annual cycle of needs’ assessment and planning.
- The Executive Team saw that part of its role was to ‘raise the game’ of the whole delivery system.
- Membership of the mechanisms which the CSDP set up was by invitation, based on contribution to the overall strategy.
- The Executive personnel involved in the partnership were all in developmental roles, and regarded by informed observers as ‘more than project managers’; combining high quality project management skills and credible grasp of the relevant policy agenda.

You need to get your tactics consistent with your thinking

Headline finding & lesson: You increase the likelihood of adding value if you concentrate your activities on changing professional practice for the better.
Evidence summary - Delivery chain analysis and improvement was one of the core business processes that the partnership used in its operations.
- You need to absolutely recognise the fact that ‘buy in’ from the leading professionals working within the delivery systems you aim to change is a precondition for success.

Headline finding & lesson: You need to recognise that there is a continuous process of interpretation and translation…
Evidence summary - Change was ‘part of the message’; change and improvement was the message
- There was extensive use made of evidence and policy digests
- Value of very large scale investment in professional development and updating
- You need to take account of the need to change thinking & behaviour, and recognise that this is a continuous process: there is never an end to the need to interpret and communicate change and service improvement.

Headline finding & lesson: Create your infrastructure within the planning and delivery systems of your partners, whose practice you intend to change
Evidence summary - Strategic investment in supply side (e.g. voluntary sector, business advice organisations, Connexions’ Service)
- Use of virtual teams within partner organisations, who collaborated both on delivery activities, and development of plans
- Investment by the partnership in staff who worked on the CSDP agenda in partner organisations as ‘development attachments’
- Your plan is to re-shape their (your partners’) plans, rather than have them join in with yours
- Join ‘their’ networks (i.e. the networks of those people you want to influence)
- Use your know how and influence to help them achieve their objectives

Headline finding & lesson: Innovate and be flexible; maintain a space within in which experimentation can be done
Evidence summary - The approach to project development was based on the view that there were no ‘magic bullet’ solutions to core and basic skills improvement.
- Principles of rapid prototyping were used with many many interventions taking place within any one year, within the framework of an annual business plan, to which partners were commited in the expectation of delivering measurable change
- Very high quality back office systems meant that there was confidence in the integrity of the partnership’s project management and control systems. This created the space within which experimentation was feasible.
- Very high level champions and advocates of the partnership helped to maintain the focus on development of services.
- There was scrupulous use of evidence as the basis for either testing hypotheses, or rolling out practice (on a larger scale) which had been demonstrated as effective (on a smaller scale)

You need to follow through and keep the momentum going

Headline finding & lesson: Make change the message, or at least a significant part of the message
Evidence summary - Help people by interpreting national (sometimes international) trends and developments in the light of local needs and priorities
- Plan with outcomes in mind
- Have a message about change in order to be ‘on message’ getting your language right, hearts and minds
- Be sure that you can describe the change you have in mind in a form which is easily understood by, and engaging to partners
- Sometimes it helps to be different (if plausible) in your actions, if not radical: If the evidence fits this stance, don’t hesitate to stress the fact that you are working at the ‘leading edge’

Headline finding & lesson: Actively support implementation of activity to deliver outcomes
Evidence summary - Treat appraisal of activity as part of the process of project development
- Regular re-profiling of resources against projects in the light of an overview of how they are delivering against other projects
- Have a networking strategy in order to develop and maintain key relationships within which you influence the practice of partners
- The scale of resources that could be invested were proportionate to the changes in the baseline

Headline finding & lesson: Keep on stretching
Evidence summary

 - Report on performance: part of the habitual communication was around ‘how are we doing?’
- Up the ante, raise the game: focus investment on developing the standards to which systems are delivering. The partnership mixed a ‘hard nosed’ interest in what was being achieved with an aspiration to see standards of delivery raised.
- Telling the story about progress against strategy, linking what is happening now to the long term goals you have in mind.

The full text of the evaluation, and its pointers to ‘tools’ and 'insights’, can be found at www.coreskills.co.uk