Regeneration Project Evaluation Framework V3

ENSURE USE & SHARE LESSONS LEARNT
ENGAGE STAKE HOLDERS
JUSTIFY CONCLUSIONS
Regeneration Project Evaluation Framework

GATHER CREDIBLE EVIDENCE FOCUS THE EVALUATION DESIGN
DESCRIBE THE PROGRAM
Marie Broadhurst Evaluation Co-ordinator Regeneration North Staffordshire Evaluation Framework for Regeneration Practitioners July 2004 - V3
(Standards adapted from the Joint Committee on Educational Evaluation 1994; Approved Standards by the American National Standards Institute and endorsed by the American Evaluation Association)
PURPOSE OF THIS DOCUMENT The purpose of this document is to provide plain English support to project leaders and co-ordinators working in regeneration, to enable them to plan for, conduct and use Evaluation in an effective and more common approach. The Evaluation Framework is built from existing mandatory and suggested guidance that has published from both government bodies; public, private and voluntary sector services throughout the country, and therefore is sound and robust in its content. Advantage West Midlands expects that ALL projects must have built an evaluation process and procedure into their project, based on the suggested framework. The overall framework (figure 1, page 7) displays the main issues that should be considered when evaluating a project. Behind each of these points lies a definition of the issue, details of why this issue is important, step by step guidance for project leaders to follow in addressing the issue, followed by examples and case studies in relation to this issue. Not all points will be relevant for all projects, but it is recommended that all projects should consider each point, from initiating a project idea, planning, delivering, and closing a project.
CONTENTS
1. INITIATING Barriers to doing evaluation Creating an Evaluation Strategy The Evaluation Framework figure 1 Whos Who in Regeneration Neighbourhood maps identifying the need Costs Raising Awareness/Communication Roles and Responsibilities Linkage Nationally Linkage Regionally Ethical Considerations Internal/External Evaluation Terms of Reference Bid Writing 2. PLANNING/DELIVERING Aims Objectives Plausibility Baseline Setting/Updating Data Collection Data Protection Analysing Information Outputs Milestones/Targets Monitoring Outcomes Evaluating Processes Risk Assessment Assessing Impact Benchmarking Interim/Rolling/Final Evaluations 3. CLOSURE/FOLLOWUP Strategic Alignment Sustainability Mainstreaming Forward Strategy Project closure Disseminating Results Influencing Policy and Practice Action Learning
4. INVOLVING OTHERS Linkage with other Projects/Initiatives Linkage with Private Sector Linkage with Public Sector Involving the Wider Community Involving Young People Involving the BME Community Involving the Elderly Community Evaluating Partnerships Capacity Building Creating an Evaluation Culture! 5. SUPPORT Support and Guidance Available Learning/Training Opportunities Relevant Contacts
Section 1. INITIATING
BARRIERS TO DOING EVALUATION FROM THE PERSPECTIVE OF .. Below are some of the possible reasons for why organisations, services and individuals do not use evaluation to its full potential. These have been collated from members views at a North West Evaluation Society network meeting, views from individual regeneration projects throughout North Staffordshire and views from the Programme Co-ordination Unit based at Trentham, Stoke-on-Trent. The Participant
Awareness of how to become involved
The Evaluation Funders

Monitoring vs evaluation distinction between needs to be clear Level of detail / ethics Timing of submissions not in line with monitoring systems Worth / importance of evaluation is low Guidance is lacking and common guidance is even more lacking! Co-ordination in joint projects is required for outcomes Financial constraints Timing for appropriate feedback That results need to be disseminated Funders have different views of participation Control an issue funders trying to control results of research
The Evaluator
Multi-funded work: complex evaluation requirements Anecdotal discredited Unachievable targets lack SMART objectives etc. Needs more understanding by others Involving the wider community is hard Involving private businesses is hard Identifying experienced people is difficult Linkage with partner organisations is difficult Monitoring & reporting systems are required and are resource intensive Lack common guidance and understanding Plain English language is never used! Too much jargon language!
Practical barriers eg transport Power differential
Whats in it for the participants? Lack of feedback to participants Confidence to become involved Are known as the worker in the evaluation Ethics Design not good enough to engage participants effectively Fear - both services users and staff Cynicism raising expectations Can merely tell what they think you want Methods can be barrier forming Language & Literacy barriers Access difficulties, lack of support to become involved Equality in voice (confident articulate are more vocal that yourself) Disabilities less able not included Over-consultation of a problem
CREATING AN EVALUATION STRATEGY The 5WH principle What Who When Where Why How So what is evaluation? You Own Evaluation! It Is Not Done To You! Is a project review to gather evidence on whether a project is meeting its outputs, outcomes & milestone targets? Project evaluations look at the basic aims & objectives of the project and assess whether they are being met effectively by the project, thereby ensuring that the overall programme will achieve its strategic aims and objectives. (Villages Initiative project evaluation guidance notes) The primary purpose of evaluation is to learn lessons, both positive and negative. (New Deal for Communities Monitoring, Review & Evaluation 2000) It is a way of assessing whether objectives have been achieved. It is a way of assessing the outcomes of projects for people who use them. It is a way of learning from successes and mistakes. Key questions that evaluation aims to address: 1) Initial condition of the area: baseline info and other evidence. 2) The scheme: objectives & strategy targets adopted and their rationale, activities & expenditure, explaining significant amounts. 3) Outputs: recorded outputs by strategic objective, especially most important, compare outputs with targets, explaining significant differences. 4) Outcomes: changes in area in respect of local conditions, how far they reflect the scheme, other internal / external factors, improved conditions permanently, initiated improvement that will continue after the scheme. 5) Process: eg. Introducing better forms of partnerships working, community involvement, likely to be sustainable. 6) Conclusions: Scheme achieved, what cost, if failed why, especially successful, why, performance against initial key objectives. 7) Lessons & recommendations: methods & approaches especially effective effective models, anything done differently, anything still needs doing. (ODPM guidance 1999)
Why evaluate?
Specific objectives vary between schemes. Evaluation shows how far the scheme is succeeding. Monitoring = ensure the approved schemes are making satisfactory progress towards the targets set out in partnership delivery plans. Evaluation goes further than this = by assessing impact against a schemes objectives and testing that those objectives remain valid. Evaluation may suggest ways to manage the scheme more effectively, it may identify noteworthy practice for wider dissemination, reporting back accurately to partners, local people and Government Offices & Regional Development Agencys. (ODPM Local Evaluation for Regeneration Partnerships Good Practice Guide, March 1999) How much evaluation? Depends on 1) Size of scheme 2) Duration of scheme 3) Complexity of scheme 4) Innovation of scheme How much evaluation is appropriate for the scheme? Depends of the following: 1) Info needed to account to the partners and the people who the scheme is intended to help community sufficiently informed? 2) Info needed to manage the scheme effectively identify projects where change is required? 3) Info needed to identify any important good practice lessons? Why some projects are more successful that others? (ODPM Local Evaluation for Regeneration Partnerships Good Practice Guide (March 1999) When evaluate? The accountable body, in conjunction with the partnership should, from the outset of a scheme, draw up an evaluation plan ensuring that evaluation throughout the lifetime of a scheme is built into their structures and processes. In addition a specific project should be set up for post scheme evaluation studies. (SRB Guidance manual, revised March 2003)
3 timings for evaluation:

Final, interim and rolling Final: At or near the end of scheme. The main purpose is to discharge the duty of accountability, showing how far achievements of the programme justify the public expenditure and other resources it has absorbed. Interim: In the mid-point of the scheme. The main function is to help partnership boards, executive teams etc to improve performance. This also contributes to a forward strategy. Rolling eval: These are performance assessments carried out quarterly through key indicators and annually through periodic review. This leaves less to the discretion of partnerships than final and interim evaluations. Procedures are prescribed by government guidance and involved Government Office /Regional Development Agency supervisory because it is directly linked to continuation of funding and rollover of delivery plans. Minimum evaluation requirements Some partnerships require each project leader to provide a brief annual, six-monthly or even quarterly evaluation report, covering evidence of outcomes as well as the output statistics that are always required. Agreement must be made within the partnership. (ODPM Local Evaluation for Regeneration Partnerships Good Practice Guide March 1999) Who evaluates? These are the possible options for who will undertake your evaluation: Local Universities Local private consultants Joint evaluations with other local services or organisations (piggy-back research) National Universities and consultants Build a team of community researchers Build a team of employees as a resource in your organisation or service How to evaluate.introduction to the framework
Figure 1 The Evaluation Framework for Project Leaders

SCHEME/PROGRAMME DEVELOPMENT
ON-GOING SUPPORT & GUIDANCE, ACTION LEARNING, INVOLVEMENT OF OTHER PROJECTS, THE WIDER COMMUNITY, PUBLIC & PRIVATE SECTOR - MAINSTREAMING
RAISING AWARENESS
Roles & Responsibilities
project level
strategic level
partnership level
CLEAR BID WRITING

Linkage to National / Regional initiatives
INTERNAL /EXTERNAL EVALUATION AIMS / OBJECTIVES BASELINE / TARGETS MILESTONES/TARGETS UPDATED BASELINE OUTPUTS OUTCOMES ASSESS IMPACT
Measurement & Data collection Benchmarking Added value Deadweight Double counting Displacement Sustainable benefits Meaningful Info out B e n c h m a r k i n g (with self and others) Sources of baseline data existing / original research
Feedback Into Project Strategy
TIME, PEOPLE, RESOURCES, , RISK ASSESSMENT
Interim Eval
Rolling Eval
Final Eval
STRATEGIC ALIGNMENT (to related schemes) MAINSTREAMING / SUSTAINABILITY FORWARD STRATEGY
NOTEWORTHY PRACTICE
DISSEMINATE RESULTS CHANGING POLICY & PRACTICE
Partnership Partners GO/RDA Community Other schemes
10
WHOS WHO IN REGENERATION National level

English Partnerships ODPM NRU Housing Corporation
Regional level
GOWM
AWM
REGENWM
Regional Coordination Unit
Local level
Housing Market Renewal / RENEW North Staffs
S-o-T LSP (Including NRF and SRB)
North Staffordshire Regeneration Zone
Programme Co-ordination Unit
Area Implementation Teams / Community Facilitation Service
11
TERMS OF REFERENCE FOR BODIES IN REGENERATION National level Office of the Deputy Prime Minister (ODPM) The aim of the ODPM is: Thriving, inclusive and sustainable communities in all regions The objectives of the ODPM are: Work with the full range of Government Departments and policies to raise levels of social inclusion, neighbourhood renewal and regional prosperity Provide for effective devolved decision making within a framework of national targets and policies Deliver effective programmes to help raise the quality of life for all in urban areas and other communities (www.odpm.gov.uk) English Partnerships English Partnerships is the national regeneration agency, helping the Government to support high quality sustainable growth in England. They have 5 core business areas and deliver objectives through: Developing a portfolio of strategic sites Acting as the Governments specialist advisor on Brownfield land Ensuring that surplus Government land is used to support Government objectives, especially the Sustainable Community agenda Helping to create communities where people can afford to live Supporting the urban renaissance by improving the quality of towns and cities Their work is characterised by: The early and active involvement of local communities Close and productive relationships with the private sector and through joint adventures The innovative use of our own sites to promote new standards of sustainable development and advance best practice The highest standards of design, environmental sustainability and construction The recognition that lasting development unites the local, regional and national dimensions (www.englishpartnerships.co.uk)
12
Neighbourhood Renewal Unit (NRU) A National Strategy was launched by the Prime Minister on 15th January 2001. The Strategy sets out the Governments vision for narrowing the gap between deprived neighbourhoods and the rest of the country, so that: Within 10-20 years, no one should be seriously disadvantaged by where they live The aim is to: Deliver economic prosperity Safe communities High quality education Decent housing Better health to the poorest parts of the country At the regional level, neighbourhood renewal teams have been set up in the nine government offices to provide a direct channel of communication from neighbourhood/community groups to the neighbourhood renewal unit and will act as: Facilitators to support the development of LSPs Mediators to resolve difficulties which may arise over the participation of Government agencies in LSPs and in the negotiation of partnership and plan rationalisations proposals Creditors to assess whether neighbourhood renewal fun grant conditions are being met and that LSPs are effective and involve genuine community participation (www.neighbourhood.gov.uk) Housing Corporation The Housing Corporation is responsible for investing public money in housing associations and for protecting that investment and ensuring it provides decent homes and services for residents. They invest in housing associations to provide homes that meet local needs. Through regulation they seek to ensure that people will want and be able to live in these homes, now and in the future. There key aims are: To regulate to promote a viable, properly governed and properly managed housing association sector To invest for the creation and maintenance of safe and sustainable communities To champion a resident focus in the housing association sector To be modern, customer-centred, forward-looking organisations encouraging change in the sector (www.housingcorp.gov.uk)
13
Regional level Government Office of the West Midlands (GOWM) Will be working closely with partners in the region and with colleagues in central Government, on a wide range of policies and programmes in 2003/04, making a strong contribution to: Improving the economic performance of the region and promoting a sustainable approach to economic development Promoting the development of a skills and adaptable workforce in the region and providing opportunities for people to develop their potential Improving quality of life through supporting the development of partnerships and providing support to communities Providing good quality intelligence to central governmental departments and communicating government policies effectively in the region (www.go-wm.gov.uk) Advantage West Midlands (AWM) AWM is one of nine Regional Development Agencies established by the Government in 1999 to transform Englands regions through sustainable economic development. Their role is to help transform the West Midlands regional economy by connecting need and opportunity to create a better place in which to invest, work, learn, visit and live. They will do this primarily by leading on the development and delivery of the West Midlands Economic Strategy that is the framework for all of this partnerships activity up to 2010. The four objectives/pillars that will drive the transformation are as follows: 1. Developing a diverse and dynamic business base supporting enterprise and innovation and securing inward investment to establish, retain and grow more businesses in the West Midlands 2. Promoting a learning and skilful region raising skill levels to ensure our workforce meets the needs of present and potential employers in the region. 3. Creating the conditions for growth securing improvement to the regions transport, communication and property infrastructure to support the development of a diverse and dynamic business base. 4. Regenerating communities in the West Midlands focusing resources in places of greatest need to ensure that all our communities enjoy a better quality of life. These are supported by the need for a: Powerful voice for the region communicating the regions strengths, opportunities and needs, ensuring that all these are clearly articulated and fully understood both within and outside of the region. (www.advantagewm.co.uk)
Regen West Midlands (RegenWM)

14
RegenWM is one of the regional centres being set up to promote good practice and better skills in regeneration. They respond to local need and to the Governments Sustainable Communities agenda. The specific role and remit of RegenWM is still under debate but most members of the regeneration community have said that they want RegenWM as a networking, events and learning body. (www.regenwm.org) Regional Co-ordination Unit (RCU) The RCU acts as the corporate centre of the Government Offices throughout the regions, representing them in Whitehall, raising their profile and ensuring that their experience and expertise is taken into account in policy formulation. The Government Offices and RCU are at the heart of the delivery of policies and programme that reduce social exclusion. The RCUs 4 key themes of work are: 1. Acting as the corporate centre of the Government Offices 2. Better co-ordination of Area Based Initiatives 3. Involving Government Offices in the policy making process 4. Joining-up in the regions (www.rcu.gov.uk)
15
Local level Housing Market Renewal / RENEW North Staffs RENEW is one of nine HMR Pathfinders throughout England, funded by ODPM, aimed to: Create better future for communities hit by poor demand for property and poor housing quality Objectives are as follows: 1. Balance the supply and demand for housing by removing surplus properties and providing a better choice of homes in appropriate locations 2. Provide sustainable neighbourhoods through better management and increasing population by building at higher densities 3. Reduce outward migration from Stoke-on-Trent and retain and attract new population to the pathfinder area 4. Improve the environment radically by removing housing from heavily polluted roads, from areas of polluting industry and from areas with no long-term residential future. Quality open space will be provided on former housing land that is no longer needed. 5. Promote social cohesion by ensuring ethnic minorities can access new build housing and be reducing overcrowding 6. Link to wealth creation by supporting the new commercial core, town centres and by providing construction training to local people 7. Reduce crime and fear of crime in order to promote safe neighbourhoods (www.renewnorthstaffs.gov.uk) Stoke-on-Trent Local Strategic Partnership (LSP) This Partnership reflects the merging of the Single Regeneration Budget 2, 5 and 6 in the City, better alignment of Partnerships, more coherent and co-ordinated programmes under the emerging Community Strategy, North Staffordshire Regeneration Zone and Advantage West Midlands agenda. There are five pillars of activity under the LSP: 1. Housing and Development 2. Enterprise and Innovation 3. Education and Skills 4. Health and Wellbeing 5. Community Safety There are six cross cutting groups throughout the LSP activity: Image Community Development Learning Young People Environment and Sustainable Development Community Cohesion
16
North Staffordshire Regeneration Zone (NSRZ) The NSRZ aims to ensure that: North Staffordshire is a successful and rewarding place in which to work and invest, am exciting place to live with an increased quality of life for all. The core enabling activities are: The development of an integrated economic and spatial strategy to identify the forward vision for the region and to clearly articulate how opportunities can be secured and threats mitigated. The establishment of an integrated transport strategy and advocacy case to ensure major transport infrastructure can be made. Support for the further development of the planning framework for the subregion to ensure that the needs of both the major activities proposed within the Regeneration Zone area and the Housing Market are met. Contributing to the Housing Market Renewal Programme, led by OPDM, creating a dynamic new housing market that helps people into jobs, tackles social exclusion, and helps to create stronger, diverse communities. Activity will be delivered through a priority portfolio on activity grouped under four pillars linking closely with the West Midlands Economic Strategy: 1. Creating a diverse and dynamic business base 2. Promoting a learning and skilful region 3. Creating the conditions for growth 4. Regenerating communities in the West Midlands (www.nsrz.org) Programme Co-ordination Unit (PCU) To be agreed Area Implementation Teams (AITs) / Community Facilitation Service To be agreed
17
NEIGHBOURHOOD MAPS Identifying the need?! As part of the Neighbourhood Renewal Strategy, the Knowledge Management Team at Stoke-on-Trent City Council has divided the six key wards of the City into 49 smaller neighbourhood areas. This has gone through thorough consultation with key partners of the City and all have signed up to agree to participate in developing research for these neighbourhood areas. The Knowledge Management Section is headed by Ray Garside and includes a small team dedicated to the mapping of neighbourhood data. The Team has produced neighbourhood maps for Stoke-on-Trent, showing where the deprivation hot spots or priority areas are in relation to themes such as crime, health, education and skills and also where there is generic low deprivation. It is expected that regeneration initiatives will focus on these key areas and if your initiative does not include these key areas then reasons must be given for choosing the location of your initiative. The maps can be accessed through either Ray Garside or Steve Johnson (01782 232695) or via the City Council website in the near future. All regeneration research activity in the near future must contribute to the wealth of information that the Knowledge Management Team has. In return the Team can assist regeneration initiatives in providing both qualitative and quantitative evidence.
18
COSTS The project lead should cost out how much money will be required to conduct the whole of the initiative, including service delivery and also, the evaluation element, which is forgot in the majority of cases. However, it may be that you will be allocated a certain amount of money and you must cost the project around that amount. The Childrens Fund and Sure Start require that roughly 3% of your total running costs be allocated and used to conduct local evaluation. This amount is fluid depending on whether the initiative requires tailor-made methods and primary data collection, such as group consultation and surveys, and therefore more money should be reserved for evaluation. Initiatives must consider pooling resources together with other stakeholders or similar initiatives to undertake co-ordinated research that will ultimately costs less than single ventures (see also the section that follows on raising awareness, regarding the Aplaws system).
19
RAISING AWARENESS / COMMUNICATION What do we mean by Raising Awareness Telling everyone possible within your target audience that your initiative exists and informing them of what is can do in relation to them on an ongoing basis. Why is it important It is vital that once the project has a solid structure that it is publicised to all potential partners and service users. This will enable partners to gain interest in working with the initiative and will assist in attracting the right service users. This will ultimately inform the roles and responsibilities between partners. How to Stoke-on-Trent City Council has an internal system for raising the awareness and coordinating activity throughout the City. The system is termed Aplaws which merely means the process of getting information onto the system. When projects are planning specific research activity and want to raise awareness of this to others, also possibly inviting stakeholders to contribute to costs etc, they should make contact with the relevant person(s) who will put the brief details onto the City Council Community section of the website. The relevant contact for project leaders to initially pass details of research activity onto is initially the Evaluation Co-ordinator for Regeneration North Staffordshire Marie Broadhurst, contactable through the Programme Co-ordination Unit. Information of research activity being undertaken can be accessed via the Regeneration Zone and LSP websites, alternatively contact the PCU.
20
ROLES AND RESPONSIBILITIES Why is it important It is vital to identify ALL stakeholders. To ensure that all roles and responsibilities within the initiative are defined for all stakeholders involved in the initiative, and that these are allocated to relevant individuals both within the project, throughout the partnership and the community, that these individuals understand what their role is and what will be expected from them by when. This must be agreed between all stakeholders involved in the initiative. If roles and responsibilities are not defined and assigned to people then there is no certainty of the actions being undertaken at all or to what quality these will be. How to The latter chapters on Involving Others will give further information and advice on how to involve and agree roles and responsibilities. The following checklist will also help. To identify all stakeholders that will benefit from the initiative (primary stakeholders) and all stakeholders that are involved in the delivery (secondary stakeholders). To ascertain how you will add value to each group. Plan the evaluation down to detailed actions of what needs to be done. For example, questionnaire development, distribution, analysis and feedback to partners. Plan timescales of when the evaluation needs to be done by. For example, the questionnaire must be developed by March, will be distributed after every session, results will be analysed every quarter and results distributed every quarter. Allocate evaluation to the relevant individuals who will be responsible for conducting evaluation at all of the stages planned. Ensure that the individuals allocated actually understand what is expected of them, by when, to whom and in what format. Ensuring that there is clear agreement between all parties of expectations. Ensure that all roles and responsibilities are documented and circulated to all relevant parties involved.
21
Ensure that all communication that takes place which makes amendments to previous agreements is documented and circulated to all parties involved, to ensure that understanding is maintained. Have roles and responsibilities been identified? Including roles of partner representatives. Have roles and responsibilities been clearly specified? Is there a shared understanding? Documenting agreed roles and responsibilities might help to raise awareness. Are there structures in place to remind partners of their responsibilities? Or to induct new partners? Is there ownership and commitment amongst partner representatives and their organisations to fulfil the roles and responsibilities set out? Have management responsibilities been agreed and defined?
Types of stakeholders The community whose situation the programme seeks to change Project Field Staff who implement activities Programme Managers who oversee programme implementation Funders and other decision-makers who decide the course of action related to the programme Supporters, critics and other stakeholders who influence the programme environment (Monitoring and Evaluation Toolkit Stakeholder Participation May 2001)
22
LINKAGE NATIONALLY What do we mean All major regeneration programmes throughout the country will have some form of national evaluation being undertaken, many are contracted through a major University. The national evaluation will involve making contact with a sample of locally delivered projects under the programme and collecting evidence of what is working and what is not. Why is it important Any national evaluation should work alongside and support what regional and local partnerships are already doing and what they plan to do in the future about their own monitoring and evaluation work. A key aspect of national evaluation should be action research this aims to provide useful feedback to the partnerships and others within the lifetime of their scheme. Much can also be learnt from observing the results and conclusions from national evaluation undertaken, for example, research methods used, what worked, what did not work, types of regeneration initiatives. How to link with National Evaluations For this to happen, the national evaluation should fully engage with partnerships, communities and the Government Offices. It is essential that the main evaluation is a genuine partnership between all those who have an interest in the success of the programme. Data available will be used to show how projects stand in relation to district and national averages. Details of & guidance from the national evaluation should be circulated as soon as possible to partnerships and also project level. Examples of National Regeneration schemes to learn from Information and reports regarding the following national schemes can be found via the internet: National Evaluation of Single Regeneration Budget DTLR Research report Urban Regeneration Companies Learning the lessons DTLR Research report Neighbourhood Regeneration lessons and evaluation evidence from SRB case studies DTLR Research reports Partnerships in Community Safety, an Evaluation of Phase 2 of the Safer Cities Programme DTLR Research report New Deal for Communities National Evaluation Neighbourhood Management National Evaluation Neighbourhood Renewal National Evaluation Housing Pathfinder National Evaluation
23
LINKAGE REGIONALLY What do we mean Ensuring that all regeneration initiatives occurring within a region are aware of each others activity and link activity undertaken where applicable. That regions are aware of regeneration initiatives occurring within other regions are link where appropriately and learn from noteworthy practice. How to link with regional initiatives RegenWM RegenWM has an initiative termed Strengthening Regeneration Networks. The purpose of the project is to encourage and support the development of high quality networks within the region that enable regeneration practitioners (professionals, volunteers and residents within communities) to access: Good quality information including information about learning and funding opportunities, policy developments, good practice etc. To lobby and to advocate up the system as well as down it. Counties that the initiative will cover include West Midlands, Warwickshire, Worcestershire, Herefordshire, Shropshire and Staffordshire. The work is being carried out by Fran Fahy who has been seconded to GOWM and AWM, the RDA to work on this development (0121 202 3260).
24
ETHICAL CONSIDERATIONS What do we mean Anyone who carries out research needs to make sure that activities are ethnical. Why is it important to consider Depending on the nature of your project, its participants and the sensitivity of the issues being dealt with, ethnical consideration may lead you to revise your research questions. How to be ethnical The main factors to consider are as follows: 1. What sort of information will you be collecting and from whom? 2. What will you need to do to make sure that you are getting fully informed consent? 3. How will you make sure that information is collected and used sensitively? 4. How will you fully inform service users about this evaluation? 5. How will you make sure that no harm is done through this evaluation? The majority of people who are asked to provide information for an evaluation do so willingly. However, the responsibility lies with the evaluator to ensure that: 1. There is no coercion 2. That people are treated with respect when information is collected and used 3. That people know how the information is to be used and they are happy for it to be used in that way 4. That the report does not identify individuals
25
INTERNAL / EXTERNAL EVALUATION What is internal / external evaluation Internal evaluation is where evaluation skills and expertise is planned and undertaken from persons within the initiative. External evaluation is where evaluation skills and expertise is sought from persons outside of the initiative and there is usually a large cost implication. Why it is important to consider this issue It is important to state how the evaluation of your initiative will be planned, undertaken and managed at the development stage. Careful consideration needs to be made to what the initiative aims to achieve, how evidence can be collected throughout the delivery phase to prove that a difference has been made to the wider community, how this evidence will be collected, considering timescales and other resources and who will be responsible for undertaking this. Resources and expertise available to undertake evaluation will be a major influence in whether internal or external evaluation will be undertaken. How to Local evaluators should be independent of both partnership organisations and service providers (for example, independent consultants, members of a local research organisation group). To maintain this independence, partnerships should agree relevant and ethical protocols, for example, to access to partners, staff, users and other stakeholders, and on confidentiality, reporting and publication. Conversely, partnerships and providers of services must also have confidence in the expertise and knowledge of the local evaluations themselves. (Childrens Fund Guidance on objectives, target setting, local monitoring and evaluation Nov 2001) Internal evaluation If evaluation is to be undertaken internally, for example as self-evaluation, then the principles throughout the Evaluation Framework and the remainder of this guidance document, especially regarding undertaking primary research, should be followed where applicable. External evaluation Where evaluation is to be undertaken externally, then the principles throughout the Evaluation Framework should be considered as minimum. Possibilities of people who can undertake evaluation follows: Local Universities Staffordshire University & Keele University Local private consultants Other public agencies (for example Audit Commission and IDeA) Peer evaluation Local Research Groups
26
Tender Where an invitation to tender must be written to invite possible external evaluators, the following important points should be covered in the invitation: Background Purpose Objectives Outline of possible methods Timescales Budget Deliverables Obtaining further information Supporting documents
Allow at least one month for submissions from tenders. A decision must be made as to whether the invitation will be: 1. An open tender to all public 2. A closed tender to a selected list of reputable consultants 3. A single action tender to one consultant (Evaluation UK A Guide to Programme Evaluation December 1997)
27
TERMS of REFERENCE (ToR) What do we mean This lists the evaluation background as well as services and products that the evaluator is expected to deliver and should be drafted. Why it is important There is a mandatory requirement from the Single Programme Appraisal Guidance (Department for Work and Pensions 2003) to: Summarise the proposed arrangements for evaluating the results of the project during and after its delivery and show that the arrangements will be in place to ensure relevant data is made available. Explain how any lessons learnt will be disseminated and fed into future projects. How to develop ToR The ToR should: Describe the purpose of the evaluation and who the stakeholders of the evaluation are, specifying why the evaluation is being requested and what the information will be used for. In consultation with stakeholders focus on the key issues and concerns to be addressed and questions to be answered, for example handling sensitive issues. Specifying the methodology, describing data gathering instruments and methods of analysis. May be developed with the assistance of an expert. List background documentation that will be provided to be the evaluator. Specify the composition of the evaluation team profile etc. The evaluation methods as well as availability of funds will determine the evaluation team composition. Describe the evaluation work plan. Specify the responsibilities of the evaluator and the team leader, detail specific tasks to be undertaken as well as the time lines involved, Indicate the audiences are to receive which information at what times, what nature and schedule of written reports and oral briefings will be, how results will be disseminated. (Monitoring and Evaluation toolkit Planning and Managing an Evaluation May 2001)
28
BID WRITING Why bid submissions are important It is important to state as clearly and concisely as possible what your initiative is, how this will be delivered, what money is required to deliver this, how this will be evaluated and how it will contribute to strategic links and other strategies. How to submit a bid The Programme Co-ordination Unit has produced an updated version of the Neighbourhood Renewal Programme initiative submission form, which asks questions more specifically and clearly to assist in the planning, delivery and evaluation of initiatives. This common submission form will form the basis of all bids that are written and cover the same wide-ranging issues that the Evaluation Framework for Regeneration North Staffordshire specifies. A standard application form similar to this will be produced in time for all other programmes such as Regeneration Zone and Housing Market Renewal. The Neighbourhood Renewal Fund Application Fund
NRF APPLICATION FORM.doc
29
Section 2. PLANNING / DELIVERING
30
AIMS What is an aim The purpose of the initiative and what the initiative is striving for, described in a holistic way. An aim of a National Programme or scheme will be the result of accumulated issues and problems that need tackling throughout the country. Why aims are important The purpose of the scheme must be clearly planned out and must be linked to strategic aims. It is vital that there is clarity of understanding of what the purpose of a scheme is.
31
OBJECTIVES What are objectives This is the first most important step in evaluation that if not done correctly then evaluation will be impossible. Objectives are specific actions that describe how the aim(s) will be achieved in a Specific, Measurable, Agree, Realistic and Time bound nature (SMART). Why are objectives important To understand exactly what will be done, by whom and by when to meet the stated aim(s). If there are not SMART objectives then the initiative will not meet the aim(s) and evaluation cannot be done. How to set objectives Strategic objectives must be drawn from the objectives set out by the partnership or programme. The partnerships may include other aims that can be seen as vital to the delivery of the approved bid. Projects can include objectives to see whether outcomes are being achieved (impact evaluation) or objectives to see how and why things are working well for which groups of people (process evaluation). Objectives must be SMART! Clear Unambiguous
SPECIFIC
MEASURABLE
Quality Quantity Cost Time
AGREED
REALISTIC TIMEBOUND
Achievable Challenging Deadline
32
PLAUSIBILITY What is it Based on the methodology stated, initiatives must consider how possible their actions are in achieving their aim(s) and objectives. Why is it important If actions to achieve objectives are not possible then objectives will not be achieved. How to In essence, activities must be: 1. Relevant to the baseline position 2. Supported by evidence from elsewhere (case studies or general evidence that the intervention can be applied elsewhere) 3. Have plausible theory of change some logic which links them to the outcome (eg. Improving locks on doors will help decrease burglaries) 4. Are not necessarily funded projects - They might be activities which involve mainstreaming 5. Are not simply off the shelf unless they pass all of the above tests (LSP Toolkit renewal.net December 2003) Initiatives must describe the methodology behind achieving the aims and objectives stated, how will they be achieved in detail of what will be done, by whom, by when, how. Initiatives should then score how plausible their actions would be by using the following table. This score will be recorded at the Programme Co-ordination Unit along with other evaluation detail to support programme and project evaluation:
33
A plausible approach The actions that are being taken to achieve the desired outcomes. There are clear links between partnership actions and outcomes, what is being done is the best possible approach, it balances risk with safer options, will impact on Black and Minority Ethnic (BME) and other minority and excluded groups, and above all it is being delivered well.
Consider which of the indicators best describes the outcomes of your discussions
5
o There is a clear link between what the LSP is doing, and the outcomes it wants to achieve. o
3
There is some link between what the LSP is doing and the outcomes it wants to achieve, but it could be more specific. o
1
There are no obvious links between what the partnership / LSP is doing, and the specific outcomes it wants to achieve (even if what is being done will for example improve health, education etc., in other ways.) Ideas of what the problems are locally are based on anecdotal rather than clear evidence. The LSP has not made the link to what is causing problems. There is little evidence that the actions being taken are proven to be the best way of tackling the problems. There is some doubt that they will directly achieve the desired outcomes because they are untried, or the links to the outcomes are weak. How local problems affect black minority ethnic, and other disadvantaged groups specifically has not been considered. What is being done to tackle problems cannot be assessed according to the impact on minority groups. Specific examples of partnership working are not going well.
The LSP has a very clear idea, based on evidence, of what the problems are locally, and what is causing them. The actions being taken are known to be the best way of tackling the problems. They are highly likely to work because they have worked before in a similar situation(they are best practice). It is clear that the effect of local problems on black minority ethnic, and other disadvantaged groups has been assessed based on good quality evidence. What is being done to tackle problems can be assessed according to the groups affected, and will have a positive impact on them. As well as being the right thing to do, partnership working is going very well.
The LSP has some evidence of what the problems are locally, and what is causing them. Some of the actions being taken are known to be the best way of tackling the problems. Some things are highly likely to work because they have worked before (they are best practice). For others, there is more doubt. It is clear that the effect of local problems on black minority ethnic, and other disadvantaged groups has been considered. What is being done to tackle problems can be assessed according to the groups affected, and will have a positive impact on them. As well as being the right thing to do, partnership working is effective.
INDICATORS
34
Evaluation Methodology It is vital that thought must be given at this stage of setting aims and objectives that consideration should be given to how evaluation will occur and in what ways it should occur. This is where project developers can be innovative in the methods that they choose to conduct evaluation and this will make it easier when the time comes to collect evidence of whether aims and objectives have been achieved. The list below gives some examples of methods to choose from: Types of evaluation Evaluate project interventions Evaluate overall project achievement of outcomes Evaluate overall project processes Evaluate partner / stakeholder perspective Evaluate community perspective Evaluate overall programme Methodology both quantitative and qualitative Questionnaires Comparisons, focus groups, observations Comparisons, focus groups, observations Interviews, Questionnaires, focus groups, interviews Questionnaires, focus groups, interviews Other methods: Evaluate in tandem with other projects Video diaries Having students to evaluate the effect of something on something else eg. The effect of bonding and attachment Tracking of beneficiaries Analysis of professionals understanding of the project Satisfaction surveys
This information is not exhaustive and innovative tried and tested methods are contained throughout this guidance, for further information see the data collection section.
35
BASELINE SETTING / BASELINE UPDATING What is a baseline and why is it important A baseline is a bulk of information, both quantitative and/or qualitative, that describes a certain geographical location and/or its characteristics, which can be used to describe change over time. The purpose of requiring partnerships to describe progress against baselines is to identify problems and priorities in certain neighbourhoods, to then use this information to decide on outcomes that are required to be delivered and also choose indicators to help to measure progress towards those outcomes. Indicators will show how we have achieved an outcome some examples of indicators showing achievement of outcomes are given in the table below. Baselining is related to benchmarking (future section) where initiatives compare information they have obtained over time. Benchmarking can also be done where information obtained is compared to similar organisations, to compare both outputs and outcomes achieved and processes undertaken to achieve them. How to set a baseline For many strategic objectives baseline data is already available and should be used. Where numerical data and/or targets are available, they should be used to describe the intended changes. Where such data are not available partnerships should be as precise as possible in describing their starting point. partnerships should take steps to fill gaps in baseline data. Some data that is available may only be so in the form of national data, so projects will need to establish whether it is feasible to ask a relevant agency to perform a specific computer run on the data to obtain local data. (SRB Guidance manual, revised May 2003) The baseline variables need to measure the essence of the local economic and social conditions prevailing in the location both at the start of the programme and at the end of the programme, relative to the national average. The baseline also needs to throw light on whether any improvements in baseline conditions recorded is likely to be sustainable when the programme ends, or whether any improvements made could only be sustained if the regeneration effort continued. (Dept of Land Economy Discussion Paper An examination of baseline issues Essential Features of a Baseline 1999) Projects need to identify at least 1 baseline indicator per key strategic objective OR Select baseline indicators when drawing up the delivery plan, providing the opportunity to ensure output measures among key indicators are as closely linked as possible to baseline indicators. Baseline Indicators can be modified after the interim evaluation process this is why it is important to review against them on an ongoing basis!
36
Projects need to ensure that baselines are: Appropriate (how do they relate to the strategic objective?) Measurable In geographical boundaries of the scheme area (Never easy to meet data always regional or national never less than Local Authority level) The data be easily obtained The data be easily updated It is important to remember that the purpose of evaluation is to assess the improvements that the scheme brings to the lives of people living and working in the area. Where to get a baseline from
Contact Ray Garside at the City Council for information on existing data that is readily available regarding your target population and area. Find out what information your own organisation has in relation to your target population and area. Use the following web links and contacts to find what information is readily available regarding your target population and area. You will need to collect new information specifically for your purposes especially if the information you require is so specific that is has not been collected (For example, data regarding local transport facilities, shops, local services & community facilities) see sections on data collection and coordinating research activity, as referred to in the initiating section.
37
TIPS: Make a start in making a baseline as early as possible. Split it into small and manageable tasks. Take advice from other with expertise. Try and present as much information as is possible, quantitative & qualitative, which build up a complete picture of the area. But do not collect information for the sake of it. You should use the most recent information that is available to show the current position of the neighbourhood. You must take into account the time gap between any data sets you compare bearing this in mind when interpreting the information. It would be easier to get comparable data from the similar time period. If there is more than one source for the same data please present all! This will be a useful cross-reference. Use at least the local district and England whenever possible, however, in addition you may wish to show County or regional comparisons, to make more sense of the conditions. You should use the smallest geographical boundaries possible for your area depending on the information you are seeking and the boundaries you have drawn up. Eg. If your neighbourhood is made up of a number of wards, you will need to present each separately & calculate and average. Make a note of post codes for the areas and if boundaries for a particular indicator do not match the area note which post codes are covered to get hold of certain data sets at a later date to allow comparison. For each indicator you consider, check if there is information available from any previous years to get a reasonable time series (eg. Five years if available annually) and see what the trend is. Explore if any other data exists to plug any gaps in data you have collected paint as fuller picture as possible. Make a note of when the information you are collecting will be updated, to update your baseline. When benchmarking, take care when interpreting the information if different sources are used because the information will be collected in different ways.
38
Data that is available First point of contact for gathering baseline information Stoke-on-Trent City Council Knowledge Management Team Stoke-on-Trent City Council has a small resource that is collating baseline data for the City, but is also beginning to encompass data for the whole of Stoke-on-Trent and potentially North Staffordshire. The data can be supplied on a range of geographical levels, including residential areas, wards, forum areas, the City level, Regeneration Zone boundaries & can provide combined authority data. The data consists of the 2001 Census with additional supporting data for areas such as health, education, demographics, social & economic. The resource has the capacity to supply baseline data to individual projects on an initial basis, the projects then having the potential to update their own baseline data, which the City Council would like returned to them in order to update their baselines. This service can assist in identifying where data gaps exist and where projects must focus further research activity. The initial point of contact is Ray Garside, City Council. All projects must make initial contact with the Knowledge Management Team when obtaining baseline data or collecting baseline data before they begin All projects must have their project submissions agreed by the Knowledge Management Team before they are submitted to the Programme Co-ordination Unit
39
Other Points Of Contact CLIP Central and Local Government Information Partnership This is the main method of consultation between central government and local government on non-financial data and statistical issues. The purpose of CLIP is to enable central and local government to work together to develop an efficient and effective information infrastructure for policy development, implementation, monitoring and reporting. CLIP operates a number of topic-based groups which meet or three times a year to discuss statistical issues. The subgroups include: Census, deprivation statistics, education, housing, labour market, population, planning, social services and transport. These groups offer opportunities for people to get involved in the work on specific themes. Further information can be found on www.clip.gov.uk Key indicators for Economic Regeneration In response to the Spending Review 2002, an inter-agency working group was established for roughly one year to develop a set of indicators for economic regeneration. These indicators link to Public Service Agreements, floor targets and RDA targets and other government targets to try and rationalise the number of indicators that local authorities have to collect for different purposes, they are also a part of the Local Library of Performance Indicators, Quality of Life Indicators, which can be used in a wider regeneration context social and environmental as well as economic. The contextual indicators are of more general use in understanding the local socio-economy and setting a baseline, subject to the data reliability and availability. To ensure the indicators meet local need, firstly select the indicators that reflect the local economic regeneration priorities and then decide on further breakdowns and additional local indicators.
40
West Midlands Data & Intelligence Observatory Formally launched in April 2002. The aim of the Observatory is to provide accurate & comprehensive intelligence to improve the quality of strategy formulation and policy making in the West Midlands, by enhancing and sharing understanding amongst partners of the opportunities and threats facing the region and its sub-regions. The objectives include reviewing the state of the region, engaging partners in determining information and intelligence priorities and mobilising relevant research work, provide effective access to information and intelligence global through to local, ensuring consistency and best practice. The usefulness of this service requires further research with relevant others. Boundaries and Statistics for Town Centres The DTLR report Producing Boundaries and Statistics for Town Centres (2002) produces interactive maps and tables giving turnover, employment and floor space statistics for town centres across London. This is released free of charge on the internet and it is planned to extend across all Town Centres across the whole of England during the next twelve months. (Business Improvement Districts Guidance working draft August 2003) Data trends online There are a number of data sets online at the IDeA website which shows key socioeconomic measures and trends up dated to show trend for 2001 & 2002. The trends include data for education & training, including foundation & advanced modern apprenticeship data, crime data, health data, etc.
41
When data is NOT available DATA COLLECTION Local audit / primary research collection It may be necessary to conduct a local audit, to go out with a number of local residents and measure directly what the situation is on the ground, recording carefully what you find. Afterwards you might want to provide some assessment of the situation and discuss more widely before putting this information into a baseline. You will need expert assistance when conducting your own research. If you are able to access a Geographical Information System (GIS) you may find this helps to show what is actually contained in your neighbourhood. Examples of local research Interviews, surveys, factual surveys (into employment and economic activity levels etc) Opinion surveys (ask residents/local businesses for their views on the area they live in) Focus groups (exploring residents perception more thoroughly, also exploring process issues) Observation (looking through the community centres bookings and observing the state the place is kept in) All projects must make contact with Ray Garside and inform him of the data you are collecting and the area you are collecting data on both before you begin collecting and after you have collected this is to ensure co-ordinated research activity throughout the City Need for more local comparative area data! There has been over reliance on Census of Population data and comparative area data. There needs to be more co-ordinated local attitudinal surveys, data on ethnic communities, measurement of baseline conditions needs to be defined and linked to community development. The baselines variables selected should be desirable on conceptual grounds but also need, from a practical perspective, to be measurable at different spatial levels, including small local areas. Achieving this balance is a tall order. (Dept of Land Economy Discussion Paper An examination of baseline issues Essential Features of a Baseline 1999)
42
General TIPS for collecting own data: The people interviewed reflect the overall characteristics of the population. The questions asked match equivalent questions from national surveys to enable comparisons to be drawn. The questionnaire is set up and conducted in a way that minimises bias. A member of your partnership has specific responsibility to liaise with contractors, community groups and residents. Trawl for what other surveys have been done in your area to ensure different residents are used etc. Agree with contractors the questions to be asked need a wide range of questions to address the wide-ranging issues. That you test/pilot the questionnaire out with a number of residents (20 or more) to see how well it works in practice (these residents should not be asked to participate in the main survey). Make alterations as necessary. That the survey is publicised well, to ensure a high response rate, for example, a letter from the partnership telling residents about the planned survey, an article in the newsletter and talking to local groups. Below 65% there may be problems around the reliability of the information you obtain. Analyse results obtained from the questionnaire pilot and get a good feel for what the results will tell you. Then suggest any further information that you would like out of the data and reorganise questions with the contractors accordingly. That the results you obtain from the contractors are in a readable and understandable format, to enable you to do further analysis in the future if required. Feedback the results of you baseline survey to residents, for example through pamphlets, newspaper articles, presentations etc. This will promote ownership of the results to the community.
43
Using residents as interviewers You could use local residents as interviewers ensuring they have been trained in interviewing techniques: Advantages It increases employment opportunities for local people through training It is a good way to build the capacity of people to go on and do other things in the community and develop personally It is useful in letting people know about NDC Residents may possibly open up more or be more honest when answering questions from a local resident Factors that need considering Selecting residents for training can be difficult and lead to disappointment There is no guarantee that the training and employment can be applied to other local opportunities once the survey is completed The survey process will take longer and you will need to check the quality more The training will need to come from people who are both technical experts and good at working with inexperienced residents In communities will a diverse, multi0racial population it is essential that the interview team includes people from different sections of the community who speak the languages required to get a good response rate Be careful about ethical problems some people do not want neighbours to know about income, health and other sensitive matters
44
Focus groups Focus groups are not scientific or robust and so cannot be easily measured or revisited. However, they provide insight and depth into issues and are a good way to get residents to talk flexibly and share and build ideas, to inform the structure and content of a survey. TIPS: Explore existing focus groups that you could utilise attendees and expertise from. Keep the group small 8 to 10 is ideal, allowing everyone to have a say. Use a trained facilitator, who will draw up a topic guide with you. Tape the session & transcribe later. Be careful how you invite people to take part identify certain groups eg. Young unemployed. Hold the group in a neutral community facility, eg. Pub or community centre. Avoid sensitivities, eg. Inviting a group of Muslims to a Christian church hall. (New Deal for Communities Gathering Baseline Information 2000) Piggy-back Research Investigate if other organisations, services or partners are conducting research in a similar area that you require. It would be more resource effective and efficient to pour resources together to complete the research. You could have the data collected separately, analysed separately or add different questions. Joint working / partnership protocols should be put in place before commencing on this venture. Training Community Researchers Link in with the chapter on capacity building and involving others. Utilising congregation of people at large events
Combined research efforts with other services / organisations
45
Methods of data collection The research question or questions will determine the most appropriate methodology to be employed. Ask yourself the types of questions that you may want to ask about a particular issue/problem, for example: Who / what / where / why / when / how.for example How many people are in this situation.. are affected by this problem.. have been helped by this initiative? How prevalent is this problem? Which groups are most affected by these issues .are most at risk? How much of a difference does the initiative/programme make to the prevalence of these problems? Why are people affected by the problem? How do people end up in this situation? How does the initiative/programme work? Why does it work/not work? What do people think about the intervention? To answer the above, then a combination of both quantitative and qualitative information would be required, to gather information of both the topic and an understanding of it. The strengths and weaknesses related to both methods can be seen below: QUALitative Flexible Enables exploration of the meaning of concepts and events Produces valid data if issues are explored in sufficient depth to provide clear understanding Enables study of motivations and patterns of association Provides a detailed understanding of how individuals interact with their environment, etc Sample sizes are often small as the research is so time consuming QUANTitative Produces exact statistical data Where random probability samples are used, survey estimates can be defined within specific bounds of precision Can measure the extent prevalence, size and strength of observed characteristic Can determine the importance of factors in influencing outcomes Uses standardised procedures and questioning, enabling reproducible results Can be costly and time consuming, particularly if population rare or hard to reach
46
Strengths
Limitations
QUALitative Need to be able to anticipate factors associated with issues to be studied to design good sampling strategy Interviewing methods rely on respondents being reasonably articulate Analysis of data to generate findings is not always transparent or replicable
QUANTitative Sampling frame may not be available
Generalisation of findings can be an issue
Structured interview hinders detailed exploration of reasons underpinning decisions or views Standardised questionnaire design and administration means there is little flexibility to be able to deal with respondents misunderstandings of the questions Requires key concepts to be clearly defined and translated into meaningful survey questions. Fuzzy concepts are difficult to measure
Different methods of data collection include the following: Method Face to face interviews Strengths Best for control of field sampling and data collection Usually gets the highest rating for being the best method Weaknesses Most expensive usually Specialise skills required to conduct May require several months to capture decent sample size Used to long and complicated questionnaires May incur interviewer effects Can miss personal expressions and body language information Specialise skills required to conduct Depends on respondent availability Respondent tolerance may be limited
47
Telephone interviews
Can do from afar Usually around 50% of faces-to-face cost Usually fastest mode of data collection Good for respondent supervision
Method Postal
Strengths Relatively cheap (but questionnaires must be kept short and simple) Can target large sample
Web-based
Easily target a large amount of the population (but they must be signedposted where to go)
Weaknesses Reply rate is usually quite low Few means of controlling how the questionnaires are completed Weak quality response from those with poor literacy and motivation Worst for missing out data, having errors etc Usually lowest response rate Only a certain percentage of the population are techy
Written questionnaire Focus groups Observation Video diaries Personal written diaries Students to evaluate Tracking beneficiaries
Combining a number of methods used Advantages Save money Improve geographical coverage Overcome sample frame bias Improve data quality, such as response rate or item non-response Speed up data collection Overcome resource problems, such as lack of face-to-face interviewers Disadvantages Development time may be extended Costs will be increased Keeping track of the outcomes for each case will be required Need ensure that data is not lost and ensure data can be stitched together to make sense Impact on reliability and validity of data obtained can be hindered
48
Sampling Sampling is used to make inferences about a population from a relatively small number of observations, that are assumed to be representative of the population. In a social researchers ideal world, all data would be collected by census. However, in practice this approach is not always practical because of time and cost constraints. As an alternative, sampling methods are used to provide estimates based on data from a very small percentage of the population. Sampling works because: It is not necessary to collect data on ALL people in order to generate statistics about that population. After a certain sample size, there is no need to collect more data. The extra data does not improve the accuracy of the estimate to any great extent.
Sample bias: Population estimates based on survey data will be inaccurate if the sample is biased. Sample bias will occur because of the following: Sample bias occurs when the selected sample is systematically different to the population. The sample must be a fair representation of the population we are interested in. The sample size may be too small to produce a reliable estimate. There may be variability in the population. If you want your sample to give an estimate that is close to the population value then you need to take into account how much variability there is in trying to measure. All else being equal, the greater the variability the larger the sample size needed.
Sampling frames: The ideal sampling frame is a straight forward list of the elements you are trying to sample. So for a population sample, a comprehensive list of the population would be ideal. In practice, the ideal sampling frame hardly ever exists. The following are examples of possible sampling frames that social researchers can use: The Post-Code address file: This is the Post Office list of all addresses in the UK, which receive less than 25 items of post per day. The list is primarily residential addresses (about 94%) and the list covers approx 99% of all residential addresses. The list closely approximates to a sampling frame of households which makes it ideal for surveys of households. Random Digit Dialling: This involves selecting 11 digit numbers at random, the first 7 digits being randomly selected from the published list of prefix numbers and the last 4 being generated at random.
Sample size:
49
The main decision needed in deciding on a sample design is sample size. The following questions will help in deciding on this: What are the key estimates for the study? How precise do those estimates need to be? Are there key sub-groups fro which separate estimates will be needed? Does the survey need to be large enough to detect change over time between surveys, or differences between key sub-groups?
Participatory evaluation Participatory methods can be used in place of, or alongside, written feedback. They can be a rich source of feedback that can provide detailed information, for example on how the quality of the group event can be improved. The disadvantages are that participatory methods take longer to carry out than written evaluation methods and the results are more difficult to analyse and compare across different group events. Interviews Interviews can be conducted over the telephone or face-to-face. You will need to decide which method is most suitable for you and your service users. Decide how many interviewers to carry out and with whom. Decide who will carry out the interviews, how many each interviewer will carry out and over what period of time. Review the information you already have and decide what specific information you need from interviewers. Decide how to phrase questions to get the information you want. Focus on questions on the information you need. You may need alternative ways of asking the same question in case respondents ask you to re-phrase it. This is particularly important if several different interviewers carry out the interviews. It will ensure that interviewers continue to ask questions in a similar way, thereby ensuring that the responses are comparable. Construct an interview schedule detailing the question and the order on which they should be asked. The schedule should also detail the information given to the respondent. For example, it will tell them what the interview is for, how the information will be used, assure them of confidentiality and ask for explicit permission to carry out the interview. You may want to record responses in writing or you can use a tape recorder. Get explicit permission from respondents to use the tape recorder. Bear in mind that although this records all of the information, it means extra work in terms of transcribing and extracting the information you need. Keep your interviews fairly short. Most interviews can be conducted relatively easily in 15-30 minutes. Ask potential respondents whether they are willing to participate and whether it is convenient to carry out the interview or not.
50
Do a few pilot interviews, or treat the first few interviews as a pilot run, to endure that the timing is right, the questions work well and that you are getting useful responses. Make any modifications that are necessary before continuing. (Measuring Effectiveness NACVS June 2002)
Case Studies The case study method is a good way to discover and illustrate the process by which something was achieved over time. There are no rules about the number of case studies you should carry out. It depends on how many you need. A single case study might be sufficient or you might need more. The basic structure of a case study examining development would include the following: Description of service user(s) and their profile, how they knew about the project etc. Description of the service users development support needs Description of how the project and the service user worked together to address these needs. Include priorities set and nature of interventions Description of the progress made by the service user over time, achievement and milestones met over time Summary of how the service user has benefited and changes that have occurred. This will include a comparison from when the service user first came to the project Wherever possible, use direct quotes from the service users to illustrate what it feels like to have been benefited by the project etc Comments from the project perspective on which aspects of the case study are applicable to other service users and what worked particularly well for the project involved (Measuring Effectiveness NACVS June 2002)
Criteria for selecting data collection methods: Determine which data-collection methods best answer key evaluation questions. Tie method selection to available resources. This may mean revising the evaluation design and methods or determining other options to stay within budget. It may also mean funding additional resources to fund the most effective and useful evaluation design. Choose methods that will facilitate the participation of key programme stakeholders in the evaluation. Strengthen the credibility and usefulness of evaluation results by missing evaluation methods where appropriate. (Monitoring and Evaluation Toolkit Planning and Managing an Evaluation May 2001)
51
Characteristics of High Quality and Effective Data for Policy Making Data content Cover one or more major policy or program concerns with sufficient detail to clarify the implications of alternative policy choices. Data completeness Achieve sufficiently high submissions, reporting, or response rates and item completion, to limit biases leading to distorted conclusions. Data quality When using new data collection systems, be sure to check for standardisation of data collection and recording, data management and analysis, and structure and content of questions. Limitations of self-reported data When relying on self-reported data such as income level, exercise frequency or health screening behaviours, be aware of self-reporting bias. Measures will vary based on the type of data collection alone. Data validity & Reliability Revision of survey questions and the development of new data collection systems will require careful validity and reliability testing. In monitoring efforts, the validity of responses over time may also become an issue. Provide classification & coding consistency to enhance interpretability and reduce confusion. Periodicity of data availability Data collection efforts are not always performed on a regular basis. Take this into consideration when planning you dissemination and communication efforts. Timeliness of data availability A previously stated, this is not always possible, but still important. It helps to be able to regularly identify progress and areas that may need additional efforts. Representativeness of data Special considerations need to be made when collecting data for specific population groups or local communities. Do responses collected represent those individuals of interest? Analytical flexibility Support both routine and special analyses, particularly on an interactive or realt-time basis. Cross system flexibility Allows users to merge, compare or jointly use data from complementary systems; include compatible and consistent variable definitions, coding categories and a linkage mechanism.
52
Adaptability Allow data content and/or reporting to be readily modified to address changing needs. Small-area analysis Take into account the limitations of applying national data to the state, local and community levels. This pertains to using small numbers in ones statistics. Is the data simple to understand?
Accessibility Provide clear reports to a non-technical audience; make available diverse reports to information tailored to different needs or users, and provide access to public-use data sets at a reasonable cost so they can be independently analysed. Translation & policy applicability Effectively translate technical data to policy-relevant information. Disseminations Accurately and fully inform potential users or decision-makers about the resources and how to access it effectively.
53
DATA PROTECTION What is it and why is it important This is a summary of the definitive Data Protection Act 1998. These guidelines apply to anyone involved in the collection, processing and use of market research data and all methodologies quantitative and qualitative and sample sources which are potentially covered by the 1998 Act. How to comply with data protection The guiding principles of the Act are: Transparency ensuring individuals have a very clear and unambiguous understanding of the purpose(s) for collecting the data and how it will be used. Consent at the time that the data is collected, individuals must give their consent to their data being collected and also at this time have the opportunity to opt out of any subsequent uses of the data. Key data protection principles: 1. Personal data shall be processed fairly and lawfully. 2. Personal data shall be obtained only for one or more specified and lawful purposes and shall not be further processed in any manner incompatible with that purpose or other purposes. 3. Personal data shall be adequate, relevant and not excessive in relation to the purpose or purposes for which they are processed. 4. Personal data shall be accurate and where necessary kept up to date (with every reasonable step being taken place to ensure that data that is inaccurate or incomplete, having regard to the purpose(s) for which they were collected or for which they are being further processed, are erased or rectified). 5. Personal data processed for any other purpose or purposes shall not be kept longer than is necessary for the purpose or those purposes. 6. Personal data shall be processed in accordance with the rights of data subjects under this Act. 7. Appropriate technical and organisational measures shall be taken against unauthorised or unlawful processing of personal data and against accidental loss or destruction of or damage to personal data.
54
Sensitive data: Explicit consent is required for processing sensitive data this means that the consent must be absolutely clear and based on a detailed explanation of how the data will be used. This is defined as personal information covering: Race or ethnicity Political opinions Religious beliefs Trade union membership Physical or mental health Sexual life The commission or alleged commission of an offence or any proceedings for an offence committed and the outcome (Market Research Society Basic guide to the data protection Act 1998)
55
ANALYSING INFORMATION What do we mean and why is it important Interpreting the information that you have collected into conclusions and findings, drawing out the key messages from all information and making recommendations. How to analyse information Interpreting evidence can be a difficult task and enough time should be allowed for this phase. For example, reading 20 questionnaires and comparing and summarising the answers to 6 questions can take several hours. Quantitative information This type of information can be summarised in numerical forms. For example, How many families use a service per day/week How many children under 5 attend a play group The average number of people using a drop in advice service/week Qualitative information This type of information can be difficult to unpick. You need to sift through the findings and explore implications. The following points may help you: In order to get the best use out of qualitative data, it may be useful to sort the data into some order, positive/negative and advantages/disadvantages etc Summarise the material into a series of points Look for reoccurring points and note how many times they appear and with what qualification and differences if any Aggregate the information by noting the most frequently occurring points and any other points which seem to be of particular importance, bearing in mind the context Look back at your original measure of success and outcomes to help pick out what you need Avoid being unduly influenced by a tiny number of either very positive or very critical comments Try to establish the majority view Check that your judgements are based on evidence from difference categories of people involved with the project (eg. Participants, carers, group leaders) Look for unexpected outcomes Using quotes can give a flavour of the project and can often sum up unexpected outcomes indicate the type of person who gave the response but do not give the actual source Some quantitative analysis of the information that you have found can support your findings
56
Keep it simple! Evaluations are neither scientific experiments nor social science surveys and analysis should not be complex. It is for example, rarely necessary to conduct sophisticated multivariate analyses or modelling. The rule is Keep it simple! Most evaluations will only require frequency counts, averages and totals, illustrated by quotes and qualitative data from questionnaires and interviews. (Evaluation UK- A guide to programme evaluation - Dec 1997) Analysing questionnaires There is a range of database packages that you can use for storing and analysing data, the most commonly used are Microsoft Excel and SPSS. These packages enable you to generate statistics, tables and charts readily. For more sophisticated analysis SPSS should be suitable. Preparing questionnaires for entry into a database Familiarise yourself with the database package Collect all questionnaires and number them in numerical order. Ensure that each record on the database matched the questionnaire number. This makes it easy to check that a questionnaire has been entered correctly and can be tracked back to the original paper copy. Using a blank questionnaire, decide which answers need to be entered as numerical and which need to be entered as text. This will design a database structure. Assign numerical code to answers and keep a record of them to refer to if needed. For example 1 = yes, 2 = no. If a respondent has been asked whether they agree or disagree with something then you could assign the following codes to answers: Strongly disagree = 1 Disagree = 2 Undecided = 3 Agree = 4 Strongly Agree = 5 Where respondents can tick more than one answer, you can design your database so that it lists each of the answers. Use 1 = yes if they ticked that answer, and 2 = no if they did not tick it. You will also need a numerical code to indicate missing data. Missing data is where the respondents have not answered a question or where you cannot accurately read the response. It is useful to give missing data the code -9. The negative value means that most databases will automatically ignore these when generating total counts and frequencies. When you have entered all your questionnaires, you will need to do some data cleaning to check for errors. Randomly select every tenth questionnaire and
57
check that it has been entered correctly. Look for common errors and decide how bets to rectify them. Analysing questionnaire manually A team approach can be used to make analysing more efficient. Each team member can take responsibility for analysing a few questions and they can come together to pool the results: Count all questionnaire returns to give a total number of respondents Sort the questionnaires into different piles according to the answer categories, for example, one pile for question 1 and one for question 2 etc Count and keep a record of the totals for each answer on a blank questionnaire Always count a record the number of respondents who have answered a question, because this may not be the same as the total number of respondents For numerical reasons you may want to find out the average for the whole sample for example, number of full time staff. Sort all answers into ascending order, starting with lowest level of reporting staffing. Pick out the middle answer, this gives the median number of full time staff for you sample. If you have two different answers in the middle, add them together and divide by two to give you the median. If you asked respondents to give a rating out of ten, it is best to calculate the median score and present this as the average score for the sample. The median score is usually more accurate average than the mean score (you calculate the mean by adding up all numbers then dividing the total by the number of people in the sample). The mean is usually much more sensitive to any unusually low or high scores in the sample and therefore may give a skewed average score. Sometimes it is useful to present the mode as the average. Mode is the most common response to a question. It is particularly useful for analysing data from open-ended questions. For example, you might have a question such as what aspect of the workshop did you find more useful?, which respondents have to answer in their own words. To work out the mode, look at all the responses and sort them, putting similar responses together. This will allow you to identify which aspects were mentioned most frequently. (NACVS Measuring Effectiveness June 2002)
58
OUTPUTS What is an output Outputs are the apparent direct result of scheme activity. You can measure them or count them. Example of outputs Jobs created Number of pupils benefiting from projects designed to enhance/improve attainment. Why outputs are important Achieving outputs is a necessary condition of achieving outcomes. If outputs are nonexistent or insufficient, outcomes will be too. (ODPM Local Evaluation for Regeneration Partnerships Good Practice Guide, March 1999) How to measure outputs One of the main difficulties with some output indicators is that it is necessary to get an idea of the duration of the output being generated. For example, a drug awareness programme could help one person for five minutes on a help-line, or one for a whole year through a drug rehabilitation course. One option here is for outputs to be classified to a three banding system, where categories would seek to reflect the duration of the effect being measured. For example: Small = 0-10 hours Medium = 11-50 hours High = over 50 hours It is recommended that partnerships establish a banding system that is most relevant for the streams of output they intend to measure.
59
TIPS: Projects are encouraged to consider other factors relating to output streams which help to explain their relative quality. For example, in the case of jobs created, it would be relevant to provide some indication of the level of associated wage/salary or an indication of degree of skill, type of contract etc. Projects need to ensure that output definitions are in place and that they are mutually understandable and adequately explained so that individual projects understand them. When calculating outputs, partnerships should calculate the net additional outputs where possible. These are estimated from gross project outputs after making downward adjustments to allow for deadweight, displacement and labour market substitution (see section Assessing Impact).
60
MILESTONES / TARGETS What are milestones / targets The term milestone covers intermediate totals for quantifiable outputs and the key events or stages towards the achievement of non-quantifiable outputs. They can be both quantitative and qualitative in nature. It tells you what you hope to achieve on the way to delivering your chosen outcome and when it will happen. They help you identify where your strategy is failing, giving you time to make the necessary adjustments. If, alternatively, you are making faster progress than anticipated then you can raise your outcome target or devote extra time and resources to other priority areas. Why are they important Ministers need to know about these milestones because continued funding through the lifetime of the scheme will be dependent on their achievement and on satisfactory progress towards your long-term outcomes. Setting milestones and targets As stated previously, milestones and targets can be both quantitative and qualitative. The table below shows an example of how baseline, outcomes and milestones relate: Baseline data Crime 47% of residents consider the areas les safe than 2 years ago Area suffers 8% of burglaries, 8.7% criminal damage, 14.6% arson Recorded offences 1.5 times City average House burglary 3 times City rates Drug trafficking 2 times City rate House contents insurance unaffordable to 53% residents Prostitution considered major problem by 36% residents No. residents afraid leave Outcome 50% decrease in those classing the area as unsafe and less safe than 2 years ago Milestones
By year 3, reverse increase in fear of crime. By the end of year 6, 40% decrease in those classing area as unsafe and less safe than 3 years ago Crime rates reduced to or By end year 3 reduce incidents of below City average arson, criminal damage, youths causing a disturbance and burglary by 30%. By end of year 6 reduce all crime by 35% Numbers of recorded offences Within 12 months reduced to City average Secure extra police on beat, for at risk hours, for targeted House burglaries reduced to population. City average Target police operations on drug trafficking to increase arrests by Fear of crime reduced to 20%. national levels Within 3 years CCTV operation. Recorded offences gap closed by 20%. Burglaries gap closed by 20%. Prostitution reduced in residential
61
Baseline data home after dark is twice national average
Outcome
Milestones areas. Affordable house contents insurance widely available to reduce fear of crime gap. Within 7 years Recorded offences gap closed by 70%. Burglaries gap closed by 80%. Prostitution reduced in residential areas. Affordable house contents insurance widely available. Reduce fear of crime gap by 80%. Year 3 reduce fear of crime and increase residents sense of personal security (validate through retake survey inf.)
17% residents feel unsafe outdoors during day. 41% at night (compared to 5% and 21% respectively for the District). 52% of areas residents concerned about being victims of crime during the next 12 months. Baseline data Worklessness 66& of men & 32% women are economically active (compared to London average is 84% & 71%) 51% of households have an income below 10,000 pa (national average 34%). 25% owe money. 33% have no bank/building society account. 76% receive housing benefit. Unemployment for black and ethnic minority 12% higher than Borough average 6.8% school leavers going to college (District average 14.5%) 5.4% school leavers unemployed (District
Reduce fear of crime of District level
Outcome Reduce no. unemployed & economically inactive people by 50% Reduce no. households with income below 10,000 pa to inner London average
Milestones Unemployed residents reduced to 1200 by year 3, 900 by year 6 Reduce no. households with income below 10,000 to 45% by year 3, 39% by year 6.
Unemployment rate for black and ethnic minority groups within 5% of Borough average Proportion of working population engaged in professional occupations, numbers leaving school & going to college, university or
Reduce gap by 2.5% every 3 years Year 3 gap in numbers going to college, university or training reduced by 25%
62
Baseline data 9.2%) 10% school leavers unemployed (District 8.6%) Baseline data Health Admission rates for circulatory conditions (3055 years) 25.3/1000 for respiratory conditions (5565 years) 43.6/1000 Standardised ill health rates 39.8% higher than England & Wales average Premature death rate 50% higher than City average (twice national rate). Households with smokers 50% above City rate. 75% above national rate. Up to a quarter of adults consuming alcohol above NHS guidelines. 11% of 12-13 year olds suffering from asthma. Tranquilliser prescriptions 5% above City average. Hospital emergency admissions 40% above City average. Teenage pregnancies almost twice City average.
Outcome training raised to the District level Outcome Reduce admissions for circulatory and respiratory conditions to the London average Standardised illness rates to be reduced to within 10% of England & Waless average Reduce SMR to within 25% England & Wales average. Standardise illness rate to be reduced to within 10% of England & Wales average. Reduce premature death rate towards City rate. Reduce smoking and alcohol consumption towards national levels. Teenage pregnancy reduced to below City level. Hospital emergency admissions reduced to below the City rate.
Milestones
Milestones Reduce indifference by 30% by year 3, 60% by year 6 25% reduction in differential by year 3, 50% reduction by year 6. Reduce SILs by at least 10% every 3 years Narrow the gap by 10% after 3 years, 20% after 6 years and 38% after 9 years. Reduce SILs by at least 10% every 3 years. Within 12 months Health Impact Assessment underway, teenage pregnancy service operational. Within 3 years Neighbourhood health worker in post, health living centre open, new leisure facility open, teenage pregnancy gap reduced by 20%. Within 7 years, levels of smoking reduced towards City rates, adults consuming high levels of alcohol reduced by 20%, teenage pregnancy gap reduced by 60%, hospital emergency admissions rates reduced to below City rate. Milestones Reduce those who feel they can not get involved in their community because they dont know how, because the area is unfriendly or because of fear of harassment by 30% by year 3 and 50% by year 6.
63
Baseline data Other 12% residents cite not knowing as reason for lack of community involvement, 16% cite area unfriendly as reason, 8% cite fear of harassment, 77% residents report knowing little about
Outcome Community feels empowered to demand high quality services and facilities and play an active role in determining and managing its own future.
Baseline data or nothing about NDC.
Outcome
Milestones
Only 16% of residents use local sports facilities Dissatisfaction levels with local services: Repairs (home) 38% Police 31% Roads 29% Street cleaning 19% Street lighting 16%
Increase no. resident groups from 31 to 35 by year 3 and then sustain this to year 10 and beyond. Increase participation in Year 3 milestone to be agreed. leisure arts and community By year 6, 25% increase in use of activities by all sections of the sports facilities by local residents. community by 32%. Increase service delivery Within 12 months new service satisfaction levels to City group established, SLAs with key averages. service providers, Best Value Achieve Best Value services review underway. and develop joint commissioning. Within 3 years Civic Centre and One Stop Shop operational, Neighbourhood Manage and wardens teams undergo first local management boards set up, complete Best Value Review. Within 7 years, Neighbourhood Wardens 5 year review, One Stop Shop upgrade, undertake customer satisfaction benchmarking.
Establishing targets Set against which progress can be measured. Either: 1) Directional (increase the indicator) 2) Increase the indicator by units (% or no.) 3) Compare to other data (eg. Regional/national) Example of targets: Reduced truancy in inner London Reduced fear of crime Improved health services and facilities
64
MONITORING What is monitoring Monitoring will help your project to track the implementation of your scheme. It will also act as a key strand of information for reviews. Project monitoring will help your partnership manage projects at the individual level, and by bringing this information together with other information, inform discussions on the performance of individual projects and the scheme as a whole. Why monitor Agreed outcomes, milestones and outputs are being delivered to the agreed forecast budget and time. Those who are intended to benefit from the scheme do so. At scheme level, agreed outcomes, milestones and outputs are being delivered to the agreed forecast budget and time impact on grant payment. To identify at a very early stage where problems are occurring and projects are not delivering the anticipated outcomes and outputs.
How to monitor Projects should establish effective systems for local monitoring at the bid writing stage. These systems should cover funding forecast and actual, progress towards achieving milestones, outputs and outcomes of the scheme and individual projects. It is for each partnership to decide the exact arrangements for collecting monitoring information. The frequency of collection and the level of detail collected should be commensurate with the scale and nature of the project. Roles & Responsibilities for Monitoring Project Officers The Partnership There must be a nominated officer within Monitor progress of the overall scheme the project to carry out the following roles and all projects regularly Progress implementation of the project Keep records of projects, activities, expenditure, outputs and outcomes (forecast & actual) Keep records and be able to validate their Aggregate outcome and output data from activities, expenditure, outcomes and all projects (as set out in the delivery outputs (forecast and actual) plan/funding agreement) and supply such information periodically as required by the Government Office Report project information to the partnership accurately and on time Take corrective action
65
What should each monitoring system cover
Name, location, purpose of project and implementing agency Overall strategic outcome Outputs of each project against those identified in the delivery plan; Key longterm outcomes to which the project is expected to contribute; clear and agreed milestones (planned and actual). Target group include information on the overall beneficiaries and particularly whether they come from the local neighbourhood (both planned and actual achievements should be recorded over time). In particular, you will need to undertake ethnic monitoring to look at the extent to which members of ethnic minority groups are involved in and benefit from project activities. Start date & completion date. Key stages in implementation process. Expenditure profile by grant regime revenue, capital, actual, forecast, other resources funding the project, specifying exact funding sources.
Difference between monitoring and evaluation Monitoring Regular and periodic Keeping track and watching progress Improving efficiency Adjusting the work plan Managing the current programme Tracking inputs and outputs Evaluation Episodic undertaken from time to time Longer term assessment Improving effectiveness Assisting future programme Future programming Assessing outcomes and effectiveness (Tracking Progress renewal.net)
The partnership themselves must be certain that: Project appraisal and monitoring systems are in place Proper financial records are kept Financial systems are set up to ensure that money is properly handled
66
Checklist Are sound systems for monitoring activities and expenditure in place? Are there regular processes of reporting monitoring information to management and to the Board? Are there processes for identifying and explaining departures from planned levels of activity/expenditure? Has the Board received and accepted an Evaluation Plan? Do all partners understand and accept the procedures for tracking progress through monitoring and evaluation? Have you considered the relationship between yourselves and any national evaluation? Have you commissioned local evaluation to answer the questions no one else is going to answer? (Tracking Progress renewal.net)
67
OUTCOMES What is an outcome Outcomes are the wider effects of a scheme in a community, which are more difficult to measure than outputs. There has been little reference to evaluation of outcomes in Regeneration in previous years. Outcomes can either be defined as hard outcomes definite such as qualifications. Or soft outcomes that aim to understand distance travelled or progress towards for example, employability. Why are outcomes important Outputs and milestones have been the emphasis from the beginning of programmes such as Single Regeneration Budget, However, the importance of evidencing the wider effects of a scheme in a community has recently been emphasised, to show that money has been well spent and the initiative has made a change to peoples lives and the surrounding region. Soft outcomes can measure distance travelled towards more hard outcomes, for example, motivation and confidence can be evidenced rather than qualifications or job obtained. This shows that improvement/not has been made towards a harder outcome. Most importantly, there is a mandatory requirement from the recent Single Programme Appraisal Guidance (Department of Trade and Industry 2003 )to: Forecast the project outcomes (contribution to Tier 2 regional outcomes or other outcomes relevant to the project) and target delivery dates. These dates may be quantitative or qualitative and should show the links between objectives and outputs. Key points to consider One of the most common mistakes when setting out outcomes is to express then in terms of change to the way services are delivered or organised. Although these are important changes they are only a means by which better outcomes for people may be achieved. Long-term outcomes need to be people focused. Setting a realistic timescale is very important. If you believe that achieving meaningful change will take 10 years then that needs to be made explicit even if your project only has 2 years funding. Pretending you can achieve in 2 years what will realistically take 10 is doing nobody any favours. Outcomes should be clearly linked to the current position analysis and the strategic goal. They should be a measurable expression of the changes which would be visible if the goal was being achieved.
68
How to set outcomes The Evaluation Plan should ensure that the reader of the Plan is left in no doubt how the proposed outputs link to the schemes outcomes. This chapter thus is closely linked with the previous chapter regarding outputs. The table below show the differences between outputs and outcomes Theme Crime Output immediate change No of locks and entry phone systems installed, Hours spent by a community police officer in schools etc No visits by health outreach worker, the provision of newly built estate health centre Jobs created, training places provided & occupied, no. using job placement service, childcare places provided No. taking part in parent/school initiative, improvements to school facilities, no. contacts with truants Homes refurbished, demolished and newly built No. people trained in racial harassment issues Increased involvement of businesses in the partnership and the neighbourhood No. shopping units newly built or refurbished New bus routes through an estate, provision of community mini-bus Outcome long-term change Reduction in volume of crime, reduced fear of crime, relative to other areas Improved mortality rates, lower illness levels, relative to other areas Greater proportion of working aged population in work, reduced claimant unemployment rate, increase average incomes Improved school attainment levels, improved destinations of school leavers into employment and higher education Improved housing conditions, changes in tenure, reduction in turnover, satisfaction with accommodation Reduction of racial harassment Thriving business and commercial sector, growth of social/community enterprise sector Satisfaction with shopping facilities, increased no. business start-ups and their survival rate Improved mobility for residents, more journeys into and out of neighbourhood
Health Worklessness
Education
Physical environment Race Involving Business Encouraging enterprise Transport
69
Examples of Outcomes Community Management Interagency networks Open city council meetings Planning economic development, social planning council Policy environment Readiness fire escape plans, CPR training, retirement preparation Representation in the community Responsiveness in community groups Volunteerism level Voter turnout Growth & Nutrition Breastfeeding prevalence Developmentally delayed children Fruit and vegetable consumption Disability prevalence Enrolment in entitlement groups Enders who participate in fitness programmes Life expectancy Self-reported health status Health Behaviours Exercise levels Overweight prevalence Tobacco use Alcohol use/abuse prevalence Substance abuse treatment needed Health care resources Insurance status Medicare providers Managed care penetration Health care utilisation Hospital use rate Preventable hospitalization rate Health of mothers & children Sick days used Workmens compensation claims Worksite injury /deaths
70
Morbidity Dental caries among children Communicable disease rates Vaccine preventable disease/deaths Mental illness prevalence Mortality Infant mortality Major killers CHD, cancer, stroke, homicide, suicide, vehicle injuries, AIDS Preventative measures Blood pressure checks Childhood immunisation Cholesterol checks Colon cancer screening prevalence Diabetic eye and foot exam Flu vaccine among the elderly Mammography Pap test prevalence Social support measures Bike path mileage Recreation centre use Child abuse investigations Domestic violence Family & friend support networks Law enforcement Neighbourhood watch Transportation services Physical environment Environmental conditions / hazards Epidemics Household smoke detectors Household fuel efficiency Industrial waste recycling Natural disasters
71
TIPS: Consider the themes that were concluded from the assessment of baseline information and look at each theme in turn. Think about how problems in each theme manifest themselves in your neighbourhood. For example, if you are looking at crime, what is the recorded level of crime, what can you establish about unrecorded crime in the area, what types of crime predominate, where are they concentrated, is fear of crime as bog an issue as the reality, etc. Compare this condition in the surrounding district and to the national average. For example, you may see that they levels of domestic burglary are 10% higher then the district average, street crime is 15% higher, car theft 20% higher. .. You can see the gaps that distinguish your neighbourhood from others. Outcomes need to be selected that will close the gaps within each theme. Projects must take into account any significant changes that may impact on programme including population changes, changes in management structures and new demands being generation by cohorts and organisations etc. It is far more effective to set a core set of high level outcomes, perhaps one for each of the core themes (eg. Crime, health, education & worklessness) and one for any other priority theme you have identified. For example If crime is the key theme of your neighbourhood then your high level target could be To reduce levels of recorded crime in the neighbourhoods to the average for the district Which could then be supported by a number of more detailed outcome targets. For example Reducing fear of crime in the neighbourhood to the district average Reducing the level of drug dealing in the neighbourhood by x % Reducing recorded domestic burglaries in the area by x% The outcomes can be challenging, but must be realistic & achievable. For example You could set an outcome of reducing recorded crime levels per person to the district average, but have an aspirational target that would reduce recorded crime levels per person to the national average. Some outcomes can not be delivered alone and must have partners who know there roles and responsibilities.
72
Keep outcomes under review and be prepared to adjust outcomes as your thinking develops.
Initiatives should then score how they feel they are progressing towards the achievement of outcomes by using the following table. This score will be recorded at the Programme Co-ordination Unit along with other evaluation detail to support programme and project evaluation:
73
Outcomes achieved so far The LSP has set itself outcome targets, milestones and a yearly profile of performance goals. It is regularly monitoring these outcomes with accurate data. It is on track towards its targets in all areas. Consider which of the indicators best describes your position.
5
o All the outcomes have been achieved. Some outcomes have exceeded what was targeted. The targets set were challenging, yet realistic and achievable.
o
3
One or two of the outcomes may have been achieved, but others have been missed One or two of the outcomes may have worsened slightly Targets might have been overly ambitious, or not stretching enough. o
1
Outcomes may have got significantly worse in one or two areas. The partnership may not have data to monitor several of the outcomes. Outcomes may have got worse across all outcomes The partnership may not have set targets for individual years, or targets may not be based on information about what outcomes were likely. o Local people are concerned that things are not improving or getting worse.
o o
INDICATORS
o o
o o
Local people say that things have improved.
Local people do not think things have got much, if any better.
74
Softer outcomes As stated above, it is important to incorporate soft outcomes in addition to hard outcomes. They give a truer picture and rounder picture of the real success of the project as a whole and can demonstrate this to funders. Many target groups face multiple barriers and are a long way off being able to secure a hard outcome, for example, getting a job, so it is important to be able to measure the progress of these clients. The right kind of indicators must be used to be able to measure progress towards softer outcomes, for example, motivation and confidence etc. Employers and funders etc value soft skills and ways of demonstrating achievement in these areas can be an advantage to clients. Demonstrating to clients their progress and distance travelled can itself boost confidence and the process of working with them to measure and monitor achievements can also be empowering.
Examples of soft outcomes: Personal Development 1. Motivation and feelings of responsibility 2. Self-awareness 3. Confidence and self-esteem 4. Recognition of prior skills 5. Personal or career aspirations 6. Concentration and engagement 7. Fitness and health 8. Rates of sickness-related absence 9. Recognition of rights and responsibilities Basic work skills 1. Time keeping and reliability 2. Completion of work placements 3. Personal appearance and presentation 4. Team working 5. Relationship with peers and authority 6. Basic literacy and numeracy 7. Ability to complete forms and a CV
75
Core work skills 1. Communication 2. Numeracy 3. ICT 4. Problem solving 5. Verbal and numerical reasoning 6. Interpersonal skills 7. Planning and prioritising How to measure softer outcomes: The method that you choose will be dependent on your project, your client group, the timescales for your project and resources available. You should focus your attention on a couple of the indicators stated above, but it is a good idea to focus on very different indicators and different methods of capturing this information. The range of methods available is as follows: Questionnaires Individual action plans and goal setting Daily diaries and personal journals Discussion with personal advisors Recorded observation of individual or group activities Portfolios of material Diagnostic tests and assessments, including psychometric In-depth reflection on, during or after course Reviews with trainers or assessors Tests
Some of the above may only be available computer based. Where clients have learning difficulties or problems with numeracy or literacy then different formats will be have to be arranged. For example, using pictures, charts or facilitating discussion may be possibilities. Some formats may conclude a score for the soft outcome, which can be compared and therefore conclude whether improvement has taken place or not. A five point scale is most common to use: ranging from strongly disagree strongly agree. You can develop your own set of indicators and tools for measuring these, such as simple questionnaires. Guidance must be sought from the Evaluation Co-ordinator / Programme Co-ordination Unit. If distance travelled is to be measured, then a baseline must be set see baseline information chapter. A baseline must still be set for measuring soft outcomes, mostly through an initial assessment of client needs.
76
Examples of measuring outcomes ACET - Assessment & Progression through Soft Outcomes project (2002) This project was undertaken by the Adult & Continuing Education & Training Service is part of the Education Departments Lifelong Learning Group. The project has produced a simple set of documents designed to allow tutors, teachers and other support workers to measure progression in learners who require evidence of progression in so called soft skills. The toolkit was based on recommendations from a working group of similar organisations, most importantly with staff from the working level. The framework is being piloted with colleges, IAG etc, during 2003 and will be reviewed at the beginning of 2004 for effectiveness. It has only been used with Adults at present but can be applicable for use with young people the Youth Service has taken an interest. The soft skills are grouped into four assessment areas which can be seen in the appendix. The framework provides a starting point for asking questions about what do you know rather than what dont you know. The framework is being piloted until summer 2003 and then a 2nd edition will follow with amendments made from the consultation and pilot process. Foundation Stage Profile for reception pupils This toolkit has been produced by the QCA, DfES & Sure Start and is in the first full year of running 2003/4. The aim of this toolkit is to raise standards in education, by setting standards and assisting teachers to collect evidence to support the achievement of the standards, via assessment and observations. Teachers are expected to complete a Foundation Stage Profile for each pupil by the end of Reception years. The standards cover six areas: Personal, social & emotional development, Communication, language & literacy, Mathematical development, Knowledge & Understanding of the world, Physical development, & Creative development. Each of these standard areas has a number of criteria which the teacher must shown evidence against: For example, Knowledge & understanding of the world: Finds out about past & present events in own life and in those of the family and other people known. Begins to know about own culture & beliefs & those of other people. There is supporting guidance on how to use the assessment scales, who can contribute to the evidence & agreeing assessment judgements. The QCA & DfES plan to supply training to teachers in need regarding identifying & collecting evidence, this will cover watching video footage, identifying evidence and will be followed up by observations in the work place. Both the QCA & DfES will be conducting vigorous inspections of evidence profiles to ensure the standards are met effectively and efficiently. The Foundation Stage Profiles for each pupil will be sent to the LEA & DfES to be used for national data, being sent back to school & also to inform parents. Both the complete Profile & Standards can be seen in the appendix).
Dream Scheme
77
A Dream Scheme is a community-based project that engages young people into community activities and for every hour of work they do they receive points that go towards trips of their choice. By involving young people in solving the problems within their communities a sense of pride and belonging will be generated. It provides them with a sense of control and achievement that in turn instils community pride. There are Dream Schemes in operation across the country, the local contact is LisaJones whose contact details can be seen in the contacts section below. The Scheme has produced an interactive booklet designed for young people, called Have You Got the Bottle. It helps young people to discover what kind of person they want to be and how to be it, developing self-confidence and understanding assertiveness. Details of the manual can be seen in the appendix. The Institute for Employment Studies has produced guidance called, Guide to Measuring Soft Outcomes and Distance Travelled (2000). There are a number of definitions that need to be understood and distinguished: 1) Soft outcomes 2) Soft indicators 3) Distance Travelled The measurement of soft outcomes provides a truer, rounder picture of success. The measure used depends on the target group being measured. The soft outcomes chosen must be linked to project objectives a) Is the achievement of soft outcomes explicit in the project objectives and if not should it be? b) What is the ultimate aim of the intervention or project and what soft outcomes will be relevant? The process to measure soft outcomes: 1) Establish a baseline, which can normally be done during the initial assessment phase. 2) Decide the type of collection method, to capture a full picture of clients progress 3) Consider different assessment formats other than written paper-based 4) Utilising a recording system to evidence distance travelled 5) Understand a clients external situation in order to gain a better understanding of the context in which progress is being made (personal factors) 6) It is vital to measure before and after your intervention/project, ideally at a mid interval also
78
The tools themselves must be reliable and valid; the following checklist will help in ensuring this: a) Is it the most direct way of asking the question? b) Is this the most concrete way of asking the question? c) Is the question really an element of employability etc? d) Is the question measuring something that will change as a result of your work? e) Are you measuring each element of your project work? f) Is the question taking too much for granted? g) Is a positive answer necessarily good? h) Does the question duplicate information asked elsewhere in the tool, or elsewhere in the client records? i) Is the question unambiguous? j) Is the question in plain English? Scoring questions There is a need to add a score next to a question/response in order to avoid subjectivity, all comparisons and give precision to descriptions. The methods that can be used to score questions/responses include, Scales: for example, rating from one to five or one to seven Symbols or pictures, for example, smiley faces Physical artefacts, such as sliding counters on boards The accumulation of evidence To score the questions to measure distance travelled, the following guidelines may help: a) Is the context clear? b) Is the time period clear? c) Are the scales anchored? d) Do the scores need weighting? e) Is an aggregate score meaningful? f) Is the scoring positive? g) Consider using a norm or standard h) Is the scoring too obvious to the respondent? i) Minimise arithmetic Timescales In order to measure distance travelled it is essential that assessment takes place at the very least at the beginning and the end of your project. If time and resources permit, it is advisable to carry out assessment at regular intervals during the project this will provide more thorough and reliable evaluation of whether true progress is being made.
Inferences
79
Needles to say, the achievements of soft outcomes and progress towards employability cannot always be attributable to the intervention. Other external factors can exert strong influences on beneficiarys attitudes and achievement. A beneficiary may be more confident because of positive changes in their personal life because of support they are receiving from family, rather than the intervention. It is important to gain a better understanding of the context in which progress is being made and the factors that are most likely to be attributable to the project. (Guide to Measuring Soft Outcomes and Distance Travelled Institute of Employment Studies 2000) European Social Fund (ESF) Monitoring Committee tasks The ESF Monitoring Committee has the following tasks to pursue: To undertake a longitudinal case study evaluation which explores soft outcomes and distance travelled. Consult more widely on developing national monitoring indicators for soft outcomes and distance travelled.
Connexions Assessment Plan Implementation Review (APIR) Connexions recommend the use of an APIR framework for developing an assessment profile for young people. This is a visual representation of a young persons situation at a particular point in time, covering the broad range of issues that might be having an impact on their lives. It provides a summary showing where a young person has positive strengths or protective factors on which the young person can build and areas that present barriers to the young person. It also sets out what actions, if any, is being taken to address them. It prompts all concerned to view the young person and their circumstances as a whole, to consider the links between different issues and to evaluate their needs context. The framework highlights the breadth of issues that a personal advisor may need to consider to reflect the position of a young person. The profile will typically be built up from various sources primarily the young person, as well as information from their school, other agencies and personal advisor observations and judgements. This then translates into a position along a generic 5 point scale which allows the personal advisor to build up an overall picture and identify the critical issues for a young person. The outer edge of the wheel indicates areas in which the young person has positive strengths on which the intervention can build or where there are no immediate presenting issues. The general headings given to factors in the assessment profile relate closely to the terminology existing assessment models. One of the benefits of this should be to ease the transferability of information between different agencies working with young people. It is intended that as all young people should receive a copy of their assessment profile, it will be important for the personal advisor to interpret the profile with them in a meaningful way. The wheel framework can be seen below. The results from the assessment are backed up by general information about the young person, held on a separate sheet and then a personal action plan is drawn up which includes target setting and outcome achievement.
80
Black Country Connexions Centre This Connexions Centre is investigating the development of the APIR to assess soft outcomes of young people, including the possibility of recording this information onto computer and awarding young people as recognition for achieving targets and outcomes for them to build a portfolio. Sure Start language measure Sure Start Language Measure (SSLM) is a parental report tool to measure change in the language skills of 2-year-old children in Sure Start communities. It was developed in 2001 for the Sure Start Unit by City University in conjunction with Sure Start local Programmes. It has been designed to measure the Sure Start Public Service Agreement target on speech and language skills at both the national and local programme level. The SSLM is an interview in which parents report on their childs language skills and general development. Parents report which words on a 100 word list their child can say (the word list is from the MCDI, the McArthur Communicative Development Inventory). Use of words and word combining are also noted. Furthermore, parents answer ten questions about their childs development (from the PEDS, the Parents Evaluation of Developmental Status) including whether or not they have concerns about
81
their childs speaking ad listening. The SSLM Information Pack 1 (2001) and Information Pack 2 (2002) give detail on how programmes can use the SSLM. Contact, Sure Start Speech and Language Project, City University Francis Harris fharris@citysurestart.freeserve.co.uk James Law - j.c.law@city.ac.uk Sticky ladder rating scale Due to a lack of suitable rating scales for use with children and ICT and toys a novel rating scale has been devised. This scale is based on the idea of a tangible object that children can physically manipulate to express their views. Rather than pointing on a scale, children respond by sticking a range of items onto a small fabric ladder with Velcro.
82
EVALUATING PROCESSES What do we mean Processes are what make it possible to deliver towards achieving outcomes. Processes are the behind the scene activity. For example, having a monitoring system, having administration systems, having a system for personal development training etc. Why is it important The processes by which regeneration initiatives carry out their work can be just as important as what they achieve. Indeed, process and outcome and frequently interrelated. How do we evaluate processes Evaluating processes is usually best done by an independent evaluator rather than by those involved in the process, especially if the link between process and outcome is being examined. The methodology for evaluating processes will be the same as for other outcomes. The majority of chapters in this guidance would also be applicable for evaluating processes. A number of process indicators can be seen in the chapters on Involving Others. There may be overlap here. Examples of process indicators are as follows: Quality of administration Number of invitations sent and levels of attendance Range of organisations invited and attending Degree of shared understanding of members individual roles Degree to which members have an opportunity to contribute Clarity of the agenda Effectiveness of chairing/facilitation Degree to which members share responsibility for decisions and actions of the forum
Indicator: Quality of administration Questions to ask: Did you receive enough notice of the meeting? Were the pre-fixed papers you received useful? Have you experienced any difficulties in making contact with/working with the administrator? (NACVS Measuring Effectiveness June 2002) Questions to ask when examining the processes in your program:
83
On what basis do employees and/or customers decide that products or services are needed? What is required of employees trained about how to deliver the product or services? How are employed trained about how to deliver the product or service? How do clients/customers come into the program? What is required of clients/customers? How do employees select which products or services will be provided to the customer or client? What is the general process that clients/customers go through with the product or program? What do clients/customers consider to be the strengths and weaknesses of the program? What typical complaints are heard from employees/customers? On what basis do employees/customers decide that the product or service is no longer needed? (Basic guide to program evaluation www.mapnp.org/library/evaluatn)
84
RISK ASSESSMENT What do we mean by risk assessment Why is it important to assess risks Things are often overlooked and sometimes we fail to anticipate problems that could upset the achievement of our desire outputs. How to measure the risks involved Check against the assumptions that you have made about the initiative and identify the potential risk involved. Projects need to individually assess the extent of risk that could be attributable to the project in question. Example of Risk Assessment Staffordshire e-Government Partnership Staffordshire e-Government Partnership have developed a system based on the Office of Government Commerce (OGC), which is designed to have no end, thereby facilitating continuous improvement in the electronic service provision to citizens. The Risk Profile diagram can be seen below. The thick coloured line is the line that divides the risks that can be tolerated and those which can not. Table 1 below shows the individual risks identified and rated, showing proposed actions. These are plotted into the risk profile to specifically display those risks that are not tolerated. Do not forget stakeholder/partner views. The ratings for impact & consequences are: 2 = Low 3 = Moderate 4 = High The ratings for probability are: 2 = Unlikely 3 = Probable 4 = Highly likely For each critical areas the total risk score is calculated by multiplying column 3 by column 5. The banding of risk is identified by either 1*, 2* or 3* which are allocated according to total risk score: 1-3 = 1* 4 & 6 = 2* 9 = 3*
85
After each risk assessment undertaken of the critical areas, Management are advised to urgently address those that attract a score of 6 or 9 as identified by the risk profile. The critical areas examined in the risk assessment include the following: Strategic / Commercial risks Economic / financial / market risks Legal & regulatory risks Organisational / management / human resources factor risks Political risks Environmental risks Technical / operational / infrastructure risks However, it must be known that risks can occur at any level, but more so at the strategic & programme level or else get swamped with risks. Ensure that the project board owns the risks and build in regular review points. OGC recommends that between 1-3% of budget should be spent on initial risk analysis and management effort, plus 2% throughout the project lifecycle. A project risk policy should be established.
86
87
88
Other questions to consider Are partners committed to the project is the project a priority for them? Are the funding sources identified realistic? Has support from funding bodies been sought? What happens if match funding is not secured? Is the project deliverable in the timescales envisaged? What delays might occur and how can these be allayed? Is the management team in place and has it the relevant experience in the successful delivery of this type of project? Does the project depend on the success of other related projects or factors outside your control? What happens if the costs overshoot? What is the demand/need for the project? How can you evidence this? Are there statutory consents that need to be obtained? Is there a realistic timetable for securing these? Are their any potential problems for securing these? Are there potential lease or ownership problems? (Stoke-on-Trent City Council 2003)
89
ASSESSING IMPACT What is assessing impact and why is it important There is a need to consider whether the local regeneration scheme has played any role in local change. The terms that follow explain methods of assessing impact that projects can use. Added Value This is the benefit accrued over what was planned for and expected. Deadweight Activities / changes that would have occurred regardless of the intervention would not have occurred without the scheme must also account for any incremental change in combination with other schemes Those which there is uncertainty compare with neighbourhood renewals, tracking system beneficiaries, surveys/discussions asking clients what would have happened without the intervention Reasonable broad conclusions about the role of the scheme, not detailed calculations Double-counting Where spending has occurred more than once. Where an output or outcome is attributable to more than one local regeneration programme. Avoided by claiming outputs in proportion to the financial contribution of each programme and acknowledge contribution of other programmes. Displacement Where a regeneration scheme or project displaces activity or the problem from elsewhere can occur at both local and nation level eg. May encourage firms locate in one place rather than another, or may support start-ups or business expansion which effects employment growth. Can occur if regeneration programmes provide training to improve the employment chances of local residents gaining employment. Projects need to assess How their project outputs represent displacement Which social groups are at the losing end of displacement How many displacements are justified in terms of final policy objectives. Projects can gather this data by Previous location of companies, location of customers Interview companies on what they would be doing if it were not for the regeneration project Compare wider geog area to target area (projects need be conversant with these techniques and capable for providing an explanation to these results)
90
BENCHMARKING What is benchmarking Benchmarking is the process of comparing results. Projects can compare results that they have collected over time, or projects can compare results that they have collected with results that other similar organisations or services have this makes benchmarking very similar to updating your baseline data over time. However, projects can only meaningfully benchmark with the same measures, so when comparing with other organisations and services meaningful comparisons can be made between similar or identical organisations and services. How to benchmark In theory, all of the baseline data sites stated in the previous section have benchmarking capability, comparing data from the same indicator. The following services allow data from the same indicators to be compared. Data sharing protocol This draft protocol was formed by the National Project for Best Value in Police Training (2002), for use by all Police Forces throughout England and Wales as an agreement for sharing data: Users will be truthful and open regarding any information shared, including clarification and avoiding misunderstandings. Users are permitted with the prior permission of an evaluation projects sponsor(s) to share the results of the project and any data collected with other evaluators. Users will not disclose anything arising from a benchmarking study to another organisation or individuals without first obtaining permission from the other parties involved in the benchmarking. Users will be willing to provide the same type and level of information, in return, that they request from a benchmarking partner. Users will treat all information obtained through benchmarking only for improving organisational performance. The use or communication of a benchmarking partners identity with any data obtained or practices observes requires the prior permission of that partner. Users will not use any benchmarking information to promote their own services. Users will understand how their benchmarking partner would like to have their information handled, and used and handle and use it in that manner.
91
Library of Local Performance Indicators pilot The Idea & Audit Commission through the Library of Local Performance Indicators (PIs) have conducted a pilot to test out an electronic facility for submitting data on local PIs and to provide a facility for sharing & comparing this information outside of your own authority. The data collection for the pilot will cease on July 31st 2003, but is successful then a permanent facility will be provided (www.local-pilibrary.gov.uk/datacollection) Key indicators for Economic Regeneration Stoke-on-Trent City Council West Midlands Data & Intelligence Observatory Boundaries and Statistics for Town Centres Data trends online As stated in the baseline chapter.
92
INTERIM / ROLLING / FINAL EVALUATIONS What are these three types of evaluation Interim evaluation Evaluation conducted at roughly half way point throughout the delivery of an initiative which will inform delivery of the initiative. Rolling evaluation Evaluation conducted on an ongoing basis by the initiative which continually informs the delivery. Final evaluation Evaluation that it conducted only at the closing stages of an initiative, and therefore can only inform delivery if future delivery period is given. Why interim and rolling evaluations are important This is arguably the most important part of any planning and evaluation process, yet probably the most frequently overlooked. There is no point in collecting performance management and evaluation information unless you are willing to use it to inform future planning. Assess the progress of individual projects and take stock of the scheme as a whole against the delivery plan. Give an opportunity to the partnerships to present their achievements. Identify, discuss and seek solutions to areas of concern or difficulty. Discuss and assess how the programme is being managed and delivered. Discuss future activity/rolled forward delivery plan.
Initiative need to agree what type of evaluation will be adopted, who will be involved in these evaluations, how often they will be held, where they will be held, agree to above purpose and any additionalities. There must be a feedback loop at every stage of our planning, performance management and evaluation processes this will conclude what has been learnt and what can be used to inform our future planning.
93
The following key questions will conclude whether you have sufficient process here: 1. Have you built into the planning, a way of having regular reviews with all partners? 2. Are you open to learning from your mistakes as well as successes? 3. Are you willing to spell out the assumptions implicit in your plan and test them against evidence? 4. If you are successful, can you identify the factors that made it work in the particular context? 5. If you were less successful than you would have hoped for, then what were the factors which inhibited your progress? 6. Can you identify what you need to do differently now and are you flexible enough to change your plans? 7. Are your lessons transferable to other situations? 8. How will you share what you have learnt with others? How to Ask partners from the outset and agree a process for feeding back information into the planning process. Examples of these evaluation activities Health checks This is a new idea where an expert on particular aspects of Regeneration, for example, crime, education or jobs, will visit the partnership to look at what the project is doing. The health check is intended to help the project check your progress and make changes where necessary. Mid-year reviews This review should take place six months on from when the funding began. It should be partnerships led and informal. The project will assess their position against planned outputs, milestones and long-term outcomes and provide a report to the Government Office. This report must include the latest monitoring and finance data. Government Office will give the project feedback about performance and provide help if necessary.
94
Annual reviews This review should take place in February, a year from when funding began. It should be a more formal event, centred on a meeting between the project, your Government Office and other officials. There should be a greater focus on your outcomes and an assessment of whether all projects and the wider strategy remain relevant. It should also look closely at the partnership itself, its systems and the delivery of the scheme. The meeting should look at performance over the previous year, proposals for amending/rolling forward the 3 year plan and the 12 months delivery plan. Ministers can then inform the making of a decision regarding funding required by the partnership over the next year. A performance banding may then be given.
95
Section 3. CLOSURE / FOLLOWUP
96
STRATEGIC ALIGNMENT What is strategic alignment and why is it important Initiatives must show how they link in with strategic priorities of their overall programme, of which they will be a part of, to co-ordinate all related activities, ensuring money is best spent and prevent duplication. Initiatives must show how they link in with other related programmes and strategies, to co-ordinate all related activities, ensuring money is best spent and prevent duplication. Initiatives must show linkage to at least the following: Neighbourhood hot spots or priority areas See Neighbourhood Knowledge mappings available from Ray Garside at the City Council. Corporate Plan and Local Transport Plan (www.stoke.gov.uk) Regeneration Zone Consideration should be given to where your initiative links in with each of the pillars. Initiatives should contact pillars individually for this information as follows. Enterprise and Innovation jon.baldwin@swift.stoke.gove.uk Learning and Skills jeff.jervis@swift.stoke.gov.uk Creating Conditions for Growth peter.cooke@swift.stoke.gov.uk Regeneration Communities jeff.jervis@swift.stoke.gov.uk The Zone Implementation Plan can be accessed via Adrian.Bond@civic1.stoke.gov.uk Housing Pathfinder Early Action Plan Can be accessed through the Renew website or via Will Nixon or Brendan Nevin at the Pathfinder team. Consideration should be given to where your initiative links in with these areas and most importantly whether current or proposed activity will cause any duplication, displacement or add to the process in place. Community Strategy Initiatives should show how they link in with the five themes of the Community Strategy. Part of the plausibility of project proposals will include the matrix based on the thematic approach and the activity approach. Information on the themes can be found at www.stoke.gov.uk
97
Neighbourhood Renewal Strategy Initiatives will still need to show how they relate to the original Neighbourhood Renewal floor targets . Consideration must be given to how this will impact of collection of baseline data. Useful information can be found at www.stoke.gov.uk/culturalstrategy/regenerationstrategy Page 15 Interaction between Neighbourhood Renewal and other Strategies. Cross Cutting themes (such as young people, BME community, elderly, access to learning, priority themes) There is a Cross Cutting Community Group in place the contact is ????? for more information. Initiatives should state how their actions will impact on vulnerable groups. By solving a problem for one group this may create a problem for another group. Relevant links are the Community Facilitation Service patricia.read@stoke.gov.uk And liz.reynolds@civi2.stoke.gov.uk Area Action Plans These are still in development and will continually change to meet changing need as regeneration in the area moves forward. The Community Facilitation Service are currently working within ten Neighbourhood Renewal areas and have identified three key priorities in each of those areas. Currently there is no link to the action plans but contact patricia.read@stoke.gov.uk or liz.reynolds@civi2.stoke.gov.uk for more information. Health Impact Assessment This is a process that will measure all of the likely impacts on health of an initiative within an area or community. It will then recommend ways of reducing negative health impacts and increasing positive impacts. Depending on the stage that your initiative is at, this will impact on how you develop your initiative. Initiative should consider conducting a Health Assessment Impact but must take account that there will be a cost implication. Useful information can be found at www.meighbourhood.gov.uk Health and Neighbourhood Renewal Chapter 4 - HIA checklist. Any other relevant plans specific to their initiative Initiatives should provide evidence of how they will link with other strategies or programme that they link with in addition to the above.
98
SUSTAINABILITY What is sustainability and why is it important It is vital to assess whether the benefits of the scheme are sustainable, if they will survive after the end of the scheme. This will inform how future Regeneration money is spent and will ensure that mainstream services play a part in continuing the provision of initiatives that have proven to work. This is an important assessment that must be done during the project design stage and throughout to monitor progress. How to evidence sustainability Initiatives must consider which partners they have involved in with them which will be crucial and ideal to continue the delivery of the initiative (or part of it depending on what is proven to work). This will inform any other partners that you feel you need to become involved with from the beginning. Once partners involved are decided, it must be agreed with individual partners what contribution they each will make in real terms () to the delivery of the initiative. The commitment that each partner will make must be agreed. Initiatives will be asked to provide evidence of how they will sustain / mainstream delivery, for example, by providing evidence in minutes or contracts etc.
99
Examples of how to assess sustainability Sustainability Project Appraisal Routine Advantage West Midlands plan to develop a framework for Regional Sustainability for early 2004. However, until then they recommend the following technique is used to assess the West Midlands Economic Strategy, called SpeAR.
SPeAR is a project appraisal methodology used as a tool to rapidly review the sustainability of projects, plans, products and organisations. The centre point represents noteworthy practice progress towards sustainable improvements (the ultimate aim of the project). Projects are scored using a set of indicators based on the DETR background indicators detailed in A Better Quality Life: A Strategy for Sustainable Development for the United Kingdom (May 1999), which have been tailored to suit the task of appraising the sustainability of individual projects. Each indicator is scored on a scale ranging from optimum case to worse case scenario. The scoring of a project is also fundamentally affected by the comparisons that can be made between the value of an existing site and that of a proposed scheme. SPeAR: Allows the sustainability of a project to be assessed and illustrated graphically at all project stages, demonstrating continual improvement and evolution of projects over time. Allows the various aspects of sustainability to be balanced and the interrelationship of these assessed. Demonstrates the evolution of a project over time, as the design develops or as alternative strategies are adopted.
100
Identifies where they may be room for improvement and so achieve optimum benefit. The logical & transparent methodology is fully adaptable. Demonstrates the interaction between the various social, environmental, economic and natural resource indicators of sustainability. The spreadsheet behind the production of the SPeAR diagram ensures that all assessments are fully audit traceable. Prompts innovative thinking to include sustainability into projects design
101
MAINSTREAMING What do we mean by mainstreaming Mainstreaming involves getting these services the police, education, health, social housing and transport to re-focus more of their resources in favour towards failing neighbourhoods, helping them to recover from years of decline. (Mainstreaming and Neighbourhood Renewal Fact sheet 18 Neighbourhood Renewal Unit Feb 2004) There are 2 types of mainstreaming 1) Strategic mainstreaming refocusing of mainstream programmes and funding on to targets which are agreed or shared by local partners, reflecting local need. Sometimes known as bending the mainstream. 2) Bottom-up mainstreaming to spread approaches and learning from localised, short-term pilots on the periphery of mainstream services, to mainstream programmes and achieve sustainable funding. This is the process of changing policy/practice on the basis of the results of an experiment. (Evaluation of LSPs Mainstreaming, a briefing note - ODPM May 2004) Why it is important to mainstream Mainstreaming is part of the Governments long term strategy, recognising that the most disadvantaged parts of the country often have to rely on below average public or mainstream services, even though they are more urgently needed than in other areas. Mainstreaming reflects the need to reshape services to address local circumstances. One influential pressure for change is the Comprehensive Performance Assessment of local authorities (Audit Commission 2002) which requires local partnerships to work in partnership to achieve community outcomes. Other incentives include the LPSA rewards and freedoms from specific planning requirements. (Mainstreaming and Neighbourhood Renewal Fact sheet 18 Neighbourhood Renewal Unit Feb 2004) Benefits of mainstreaming 1) Achieving better outcomes for users in general or for specific client group 2) Solving intractable problems 3) Addressing issues which come together at a small area level 4) Supporting innovation in service delivery 5) Increasing the efficiency of resource use (Evaluation of LSPs Mainstreaming, a briefing note - ODPM May 2004)
102
How to mainstream The National Evaluation of NDC (New Deal for Communities) undertaken during 2002, concluded that there was considerable confusion in relation to what mainstreaming means for Partnerships and agencies. It added that there is little to support the view that agencies are systematically providing additional resources for, or providing better services towards NDC localities. There is evidence to suggest that some agencies have been seeking to use NDC resources to reduce existing commitments into the neighbourhood. (NDC National Evaluation 2003) Initial questions to ask about your mainstreaming activity: 1) 2) 3) 4) Is there a shared vision? Are there shared objectives? Are there shared targets? Is performance management in place to monitor activities towards agreed performance indicators and achieve agreed targets? 5) Are resources deployed to meet targets? Barriers to mainstreaming Uncertainty of what mainstreaming might mean Demands of centrally delivered performance targets Priority being achieving own spend and therefore mainstreaming coming second priority Short supply of evidence about will it work Challenge/threat to professional/middle managers How to overcome barriers
Neighbourhood managers can be important through their role in coordinating, communicating and promoting The role of individual leaders is critical enthusiastic! Alignment of planning processes to enable resource to be more effectively allocated to community planning priorities Having a performance management system in place for monitoring and reviewing resources and actions there needs to be agreement of outcomes between partners, short terms and long term and clarity on how different agencies will contribute to these indicators monitor achievement
103
Barriers to mainstreaming
How to overcome barriers Having creative combinations of joint resources, not formal budgets but through partnership working Agreeing joint local objectives of deprived areas and inter-related performance indicators therefore having a joint budget to prompt more efficient and flexible use of resources Evaluation of Short - term funding initiatives needs to focus not just on the impact of the specific project, but more importantly on the impact it has on mainstream policy, practice and resourcing
Neighbourhoods are understandably anxious for results in the sort term, which can mitigate against the processes and time needed to engage with service managers and to influence the mainstream, and can result in some dissonance between the programmes of different partners (Evaluation of LSPs Mainstreaming, a briefing note - ODPM May 2004) Five different types of mainstream activity have been identified: 1. Bending/reallocating mainstream resources changing spending patterns to target the most deprived areas 2. Focusing policy on deprived areas For example through legislative change 3. Reshaping services to reflect local need For example, by changing organisational design which prevent deprived areas receiving support 4. Joining-up of services, programmes and targets For example, through interdepartmental action and multi-agency delivery 5. Learning good practice, including from pilot or pathfinder including making stronger links between Area Based initiatives Resource mapping: A good starting point for mainstreaming is to identify how much money is being directed to the most deprived neighbourhoods before an attempt to reallocate funds from services to these areas. However, this is a very difficult exercise as many partners will be wary about saying how much goes in to which areas, or they may simply not know. A tool to assist in this task is currently being developed and details will be published through NRU which it is available. (LSP Delivery toolkit renewal .net Dec 2003) Examples
104
Doncaster New Deal in Communities investment if 250,000 in the local police force has reaped an impressive return. The area once suffered from 60% of central Doncasters crime even though it is home to just 8 per cent of residents. Basically, Doncaster came up with a radical solution, to establish a police training unit in the heart of the NDC area. Part of the training includes 25 weeks of on the streets accompanied by a tutor constable. By shifting police resources, Doncaster has gained 1 million of police funding a year, refocused on an area with high crime rates. Targeted police spending towards crime hot spots works alongside other NDC funded initiatives, such as CCTV camera and better street lighting. (Mainstreaming and Neighbourhood Renewal Fact sheet 18 Neighbourhood Renewal Unit Feb 2004)
105
FORWARD STRATEGY This section also links with the previous section explaining issues regarding sustainability and mainstreaming that should be considered when planning for after the funding period. What is a forward strategy This is a plan setting out what will happen after the lifetime of the funding period. Why is a forward strategy important It is important that the regeneration process brought about during the project/programme funding period should continue well beyond that time. Partnerships can ensure this through a forward strategy (or exit/succession strategy), setting out how the regeneration process started by the partnership will be sustained beyond the funding period. What it will include A Forward strategy should be built into the project delivery plan from the start being an essential element in the planning and implementation from the beginning. It will include projects plans to mainstream and what has been mainstreamed to date. It should not be very detailed and it will change and develop during the lifetime of the project. Helping projects with a potential longer shelf life to find continued funding. Finding & assisting any successor bodies to take over the projects. Ensuring improvements in main service supported are maintained. Finding ways to tie in the interest of external agencies after funding has ceased. Building the capacity and structures the community will need to continue to influence and lead services and projects. Merging or joining up with other projects within the community to share resources and spread costs. Consider whether some projects require specialised forms of assistance after funding stops.
106
Exploring the future role for the Management Board and identifying its future responsibilities and structure. Developing the most effective way to disseminate good practice and lessons learnt. Appropriate dates must be set for first and subsequent redrafts of the strategy, the timetable needs to tie in with the review process where project progress (outputs, outcomes & milestones) will be reviewed. It may be useful for setting up a working group, including a range of partners, with responsibility for drawing up the forward strategy. A final report must be produced by the project which will include the most up to date forward strategy for the project. Toward the end of the scheme the Government Office must be provided with the name, address and telephone number of the person who will act as a contact point after the partnership has wound down.
Issues to consider Projects must continually consider the long term needs of the project during design and development. For example, the design of a building should take into account the continuing costs of maintenance and how these will be met. Consider, investigate & utilise different sources of income and explore how these sources could be used as a source of revenue after Regeneration funding ceases. Consider how to improve and reshape mainstream funding spent in the area, after your project has ceased. Consider if it is possible to raise project running costs from fees and charges. For example, subletting space in a community building, or an agency paying the partnership to provide a service it may otherwise provide. Some projects are able to become self-financing because of the income generated over time by property purchased with the help of Regeneration resources. Where income is in excess of the level estimated in determining the amount of grant, the Government Office must be informed. Other sources of funding should be considered, for example, European Funds and National Lottery Boards. You are advised to discuss any such proposals with the relevant bodies in good time so that any agreements can be included in these bodies corporate plans and budgets. You should begin to discuss at least a year before any funding might be required.
107
The project must build in an end date from the start and not permit to slip unreasonably from this. The end dates of a number of projects must also be coordinated and fit into the wind down of the whole scheme. Even after funding has ceased, your partnership still has responsibilities to the Government in respect of income accrued from Regeneration money. The accountable body must keep an inventory of capital assets, loans, forecast receipts & plans for recycling income from a project funded by Regeneration money. Projects need to be innovative and imaginative but also practical & attainable when considering future funding. Some issues will become important after funding ends records must be retained, statutory requirements adhered to, papers may be needed for a National Audit Office enquiry and GO may request records to become public records. The accountable body must keep the bulk of records. Legal requirements include; Keeping building records for a minimum of sixteen years, finance records for seven years after the end of funding, personnel records for five years, legal documents must be retained with the asset register. Other records that must be kept include the follow; Consultants records, valuations, appraisals and approval any documents that are personally or commercially sensitive. Electronic documents must be treated as equivalent to an original, but seek your own advice at the time of relevant legislation. Remember, the overall aim of a forward strategy is to inform partners of projects which are working and which can go beyond the funding period and which projects have been unsuccessful and should be wound down. Remember that an integral part of a successful forward strategy is that the community has been involved in the project, has owned the project and been integrated into the planning, monitoring and delivery of the project. The forward strategy should ensure that there is a structure in place for residents and other sectors of the community to retain ownership over the project and assistance from agencies to help sustain it. This should be made well in advance of when funding will cease. (New Deal for Communities Forward Strategies 2000)
Examples to help Stoke-on-Trent City Council have a service called Grantfinder which provides a database search engine of over 2000 funding streams throughout the Country that could assist projects in identifying further funding. This database is updated on an ongoing basis. A relevant contact is Suzanne Clarke (01782 232147). Information regarding further funding streams can also be found through the Voluntary Action service and through the PCU.
108
PROJECT CLOSURE What do we mean and why is it important There is a mandatory requirement from the Single Programme Appraisal Guidance (Department for Work and Pensions 2003) to: Summarise the proposed exit strategy which will ensure confirmation of 1. the satisfactory completion of the project 2. the subsequent closure of the contractual relationships at the point where there are no outstanding financial claims in either direction. Summarise the proposed Forward Strategy which will protect the RDAs investment and maintain the projects impact after it is completed or post the RDAs funding where appropriate. All Partners must agree that their involvement in the initiative is finished in its current format and that a line is drawn in the sand. All Partners must be clear about what the initiative has achieved and conclusions made in order to plan for further progress. This is a part of managing performance and ensuring lessons are learnt and progress is made. Active follow-up is also vital to implement recommendations made to programme managers and incorporate lessons learnt in future decision-making processes such as the development of a new programme or projects. How to close a project It is good practice for the evaluation manager to organise a meeting with the appropriate persons and institutions to establish an implementation plan based on the recommendations with a timetable and identification of parties responsible for followup actions. The more stakeholders involved in the planning of the next steps, the more likely they are to follow through on implementing the evaluation recommendations. The evaluation manager should monitor the status of the implementation and by doing so advocate the use of evaluation results. (Monitoring and Evaluation toolkit Planning and Managing an Evaluation May 2001) All Regeneration projects under the responsibility of the Programme Co-ordination Unit are required to agree with, undertake and produce a quality evaluation, concluding in a report which will detail tracking of specific beneficiaries, statistical change throughout the areas of North Staffordshire on the issues that they have worked in, for example crime, and qualitative evidence to support that a difference has been made. This process and end report must be agreed and signed-off by all Partners, including the project in question, the tracking body involved, the Evaluation Co-ordinator and the PCU Programme Appraisal and Monitoring Manager. Any other individual who has contributed to this report must also sign the agreement, for example the Knowledge Management Team. An example of the sign-off sheet is given below: EXAMPLE SIGN-OFF SHEET:
109
Project lead: I ..(print name) agree that all information in this document is correct Signed - .. Evaluation Co-ordinator PCU: I ..(print name) agree that all information in this document is correct Signed - .. Programme Appraisal and Monitoring Manager PCU: I ..(print name) agree that all information in this document is correct Signed - .. Other - .(state organisation) I ..(print name) agree that all information in this document is correct Signed - .. Other - .(state organisation) I ..(print name) agree that all information in this document is correct Signed - ..
110
DISSEMINATING RESULTS Why is disseminating results important It is important to inform all relevant others about the progress of your project. Relevant others include project staff, the Partnership, stakeholders, the community, and specific cohorts like parents, young people or ethnic minorities. Improvements gained will include improve future funding and partnership working, and improve community interest and involvement in the project, ultimately improving service delivery to the public. A National Evaluation of NDC (New Deal for Communities) Partnerships during 2002 concluded that only a fifth of NDC Partnerships had a programme in place for disseminating results (NDC National Evaluation 2003). This highlights that disseminating results is not an important issue on the agenda for initiatives but should be. It is very important that local evaluators encourage the discussion and design of information sharing protocols among the Partnership, as pressures on peoples time can lead to lost opportunities in terms of dissemination opportunities. It is also important not to overburden front line staff within the services and the users with excessive demands on in terms of data collection. This again is why information-sharing protocols need to be developed, whereby surveys and fieldwork results can be shared and resources pooled. Initiatives that are implemented through local partnerships such as Sure Start, OnTrack, Quality projects, Connexions and NDC should be included in any specific information sharing protocol that is developed Early consultation with these key players is encouraged. (Childrens Fund Guidance on objectives, target setting, local monitoring and evaluation Nov 2001) How to disseminate results Firstly it is important to establish how the information you have gathered will be used: To gain publicity? To persuade funders? To influence policy? To influence practice?
111
Factors in effective dissemination 1. Target the right audience Use a variety of different formats for your findings 2. Balance speed with usable results Ensuring that evaluation results can be gathered to inform further activity or funding 3. Do not overload with information Busy people will not read a lot of information give summaries and only provide information people need 4. Clarity of message Information needs to be clear and understandable to have an impact 5. Relevant to current issues the right information at the right time 6. User ownership Involve stakeholders in the evaluation, especially in drawing out the key recommendations and they will be more likely to take on board the results. Link recommendations with policy makers 7. Accessibility Consider publishing in a professional journal etc that you know your target audience reads or use a local newsletter, hold a seminar, utilise an existing event to capture an audience, or visit organisations in person 8. Make material attractive 9. Clear strategy from the outset Plan ahead, think about the end product that you want and how you will disseminate it and plan for time and money. Example of dissemination methods Quarterly newsletter - with information on free contacts and good practice Training & events publicised some which are free Website to exchange good practice Good practice guides Setting up networks and forums Informing Partnership staff of strategic planning through to day-to-day management through articles, briefing meetings etc Informing Partners through reports, significant occasion updates and away-days Informing the community through articles in the press, scheme reports to interested parties, scheme newsletter, organisations forum and local ethnic minority organisations Informing other schemes and projects through handbook of good practice Presenting findings on tape
112
Large print Braille Summaries for different audiences young people, carers, older people or funders Conferences for large numbers Seminars for small numbers Videos
TIPS Appropriate media for the right audiences Written reports can be useful but they are not the only means of telling people about a project. Other means may be more appropriate and capture more closely what the project is all about Some methods can be more expensive and time consuming than others Start planning well before your evaluation is complete Select what aspects of the project will be particularly news worthy Possible format for writing a report Summary Background / context Aims / objectives Project delivery Evaluation methods (commenting on the main points in the Evaluation Framework) Results Possible subheadings Case studies Successes Lessons learnt Challenges Recommendations / conclusions Appendices
113
INFLUENCING POLICY AND PRACTICE Changes at the national level for example, changes in the Green Paper This may only be evident at the final stages of an evaluation This will increase the chances of services being mainstreamed Put in Health Assessment Impact work details of what it is, why, etc What strategies does you project link to? Regen Zone, Community Strategy, Pathfinder? (use grid plotting format?) The National Evaluation of NDC (New Deal for Communities) undertaken during 2002 reported that only 45% of Partnerships had changed and informed projects with the result of an evaluation, and only 25% of Partnerships had changed aspects of the Programme as a result of an evaluation. Local evaluation should have a direct bearing on the delivery of better services at local level. It therefore has to be formative in nature, allowing partnerships to appraise their progress at critical points through planning, set-up and service delivery process. This includes asking questions at key stages in the programme about what works and how it works, and about what works less well and what needs to be changed and improved. Partnerships therefore need to be committed to the evaluation process itself and how it leads to self-reflection and future decisions about changes and adjustments to services. (Childrens Fund - Guidance on objectives, target setting, local monitoring and evaluation Nov 2001)
114
ACTION LEARNING
What is Action Learning and why is it important
Learning is a key building block of effective neighbourhood renewal. Integrated monitoring and evaluation processes provide important opportunities for learning and reflection during the implementation of a project, enabling continuous review and development of the projects objectives and effectiveness. Ensuring that learning is not lost when projects end is also an important element of sustainability. Significant benefits can be gained by applying lessons learned from projects to future mainstream activities, including learning that has come from what did not work, as well as from what did. In this section of the form information regarding how learning in captured in sort. How to evidence Action Learning Initiatives should consider the following: 1. What mechanisms you intend to adopt to facilitate learning during the implementation stage of your initiative? How will you evidence these? 2. What mechanisms will you put in place to ensure that lessons learnt during the progress of your project are used to inform its on-going development? How will you evidence this? Alternatively, the following checklist may be of use when gathering evidence to assess whether you are a learning project/organisation/City (Birmingham Lifelong Learning Partnership Core Skills Development Partnership 2003): 1. It is easy for all people to learn and continue learning in a variety of ways Evidenced by Encourages the existence of an environment that is rich in stimulating resources Sees learning as being of different kinds, through various mechanisms Ensures that all ages, all communities, all sectors are able to engage in learning Makes learning opportunities readily known about, good information and guidance widely available Has loosely linked learning pathways/routes that are clearly understood Recognises that anyone can learn from a variety of others (and therefore help others to learn), allows for some self-direction in learning Recognises that learning happens in different ways for different people Recognises that learning can happen anywhere at anytime 2. Learning is seen as valuable contributing to individual organisational and social benefits Evidenced by Promotes learning, sees learning as being for all people Promotes high aspirations for all people Celebrates learning for a range of purposes Recognises learning component in most social processes Values experience, wisdom and critical thinking
115
Expects that each learning opportunity will foster a continuing motivation to learn
3. There is a broad interest in reflecting on things, pulling out insights and exploiting creativity Evidenced by Anticipates change and puts learning in place proactively to assist in managing changes, made more adaptable through learning Seeks out challenges and issues and commits to using learning to help resolve these Is able to be creative and to use this creatively for change Knows what is going on around it and uses this knowledge Wants to really know how it is doing, to get feedback and to improve on previous best Wants to work collaboratively with others, to exchange information and knowledge, adding to the common store of knowledge Goes out of the way to seek the views and voices of groups, individuals and communities Looks beyond the immediate, tries to get a sense of the bigger picture, uses frameworks as thinking tools 4. There are active commitments to supporting and developing learning Evidenced by Exceeds recognised quality standards, promotes guarantees, entitlements and minimum thresholds Makes active use of new technologies Fosters commitment to longer term visions and strategies, offers clear leadership and direction Seeks improved outcomes for people Puts in adequate resources in equitable ways Has ways of sharing knowledge 5. Learning draws on diversity and contributes towards cohesion Evidenced by Draws on resources of all sectors, reflecting diversity of traditions and cultures Looks for deeper understandings Matches diversity with cohesion Creates high skills levels for all Uses learning to enable engagement with wider processes A score can be obtained by using the following scoring for each of the above 33 features: 0 = clearly missing in the project/organisation/City 1 = Starting to be true/present 2 = Quite well established 3 = Clearly a strong feature Examples of Action Learning
116
The following examples show how learning can be captured either during initiative implementation or throughout the progress of the initiative: Informal reflection by the staff involved Formal learning workshops Regular participatory appraisals Regular and systematic documentation of lessons learnt during the project Sharing learning via a website or email discussion group Exchanging views and feedback with stakeholders At service user / community level Put in learning logs initial and revisit to assess if learning has taken place in a project. Ways in which this has been done can be demonstrated in the section on outcomes. At project level Part of Action Learning will be when an initial assessment of Evaluation Activity is completed with projects to form a baseline position in relation to all points stated in the A4 framework, this is recorded and then a follow-up action plan is produced for action to be taken to improve a projects position in relation to relevant evaluation points. This action plan will be reviewed either by the Programme Co-ordination Unit on a roughly monthly basis or will be reviewed and tracked by the project themselves to show records that demonstrate action learning has taken place. The relevant contact for more information on Action Learning is Judith Robinson Judith.robinson@staffs.ac.uk
117
Section 4. INVOLVING OTHERS
118
INVOLVING OTHERS What do we mean by involving others This can range from merely seeking an opinion, to actively participating in collecting information and using it. This will be described in more detail in the how to section. What do we mean by others Basically, we mean a stakeholder which is anyone who is affected by or who can influence the impact of an initiative. Stakeholders can be individuals, groups, communities and organisations. When planning your initiative, it would be useful to carry out a stakeholder analysis to look at who your stakeholders are, how important they are and what influence that can have over it: 1. List all stakeholders in your initiative 2. Categorise your stakeholders in terms of their importance to the initiative and influence on the initiative a) Importance those who are directly affected by the initiative (either benefiting from it or adversely affected by it) b) Influence those who can affect of influence the initiative The diagram below may make it easier for you to identify stakeholders: importance
High importance/ High influence High importance/ Low influence
influence
Low importance/ High influence Low importance/ Low influence
Why is involving others important There is a mandatory requirement from the Single Programme Appraisal Guidance (Department for Work and Pensions 2003) to: Explain how the project will benefit various groups (consider gender, age, disability, minority, ethnic groups, socio-economic groups, SMEs, geographical areas, etc and to explain how the project will be designed and managed to reach the target group.
119
The process of community involvement and partnership working will produce further benefits, in building trust and strengthening the community. The National Evaluation of NDC (New Deal for Communities) carried out during 2002 concluded that only 15% of NDC Partnerships had carried out an evaluation of the impact on specific groups of individuals. Local evaluation should be committed to user involvement, in particular, the involvement of children and their families, including those from difficult to reach groups. This might include early consultation and involvement in the local evaluation, to ensure that the user views are taken seriously and reduce the risk of treating key users as subjects of evaluation rather than active participants. (Childrens Fund Guidance on objectives, target setting, local monitoring and evaluation Nov 2001) How to involve others The following checklist will assist in identifying who should be involved in the evaluation of the project: Checklist for involving stakeholders in evaluation: 1. Who are the stakeholders of your initiative? (think wide! Identify all organisations and services that are linked to what you are trying to achieve) 2. What are their interests in relation to the initiative and its objectives? 3. How important is each stakeholders? 4. How influential is each stakeholder? 5. Which stakeholders therefore need to be involved in the evaluation? 6. What level of involvement should they have? To be informed of the evaluation (i.e. provided with information about the evaluation process and the findings?) To be consulted about the evaluation (i.e. given the opportunity to have a say about what should be evaluated and how?) To be partners in the evaluation (i.e. given an active role in deciding what should be evaluated and how?) To control the evaluation (i.e. to have a final say in what should be evaluated and how?) 7. What role should they play in the evaluation? In the planning of the evaluation? In collecting the information for the evaluation? In interpreting the findings? In making use of the findings? In disseminating the findings?
120
LINKAGE WITH OTHER INITIATIVES Why it is important to involve other projects It is vital to ensure that the scheme fits into a bigger, wider, more holistic strategy for the area concerned. To consider how the projects identified can be co-ordinated to maximise synergy in the attainment of wider objectives. How to involve other initiatives In order to be able to link in with other initiatives, then it is vital to know what other initiatives are out there to link with. There are a number of routes to knowing who and what is out there: 1. Contact the Programme Co-ordination Unit who will be able to tell you about similar projects and services that link with yourself. 2. Visit www.nsrz.org - where details of other projects and initiatives will be available and particularly details of research activity that other initiatives or projects are planning/undertaking that you could link in with! 3. Visit www.rcu.org.uk/abi/results - where details of wider initiatives and support are available. RegenWM The new Centre for Excellence for the West Midlands Region is operational since August 2003. As part of its scoping work, the Centre is asking all Regeneration practitioners working within the region to register with them. These individuals will be able to contribute to our discussions, take learning opportunities and will receive regular updates about work being done by RegenWM. The centre is also scoping the existence of Regeneration networks in order to highlight the noteworthy practice and to highlight gaps which require attention. Central evaluation website Birmingham.gov.uk have a website called Regeninfoexchange which co-ordinates a range of information for the area of Birmingham; It houses the Regeneration for Birmingham, a Regeneration Network, Other Regeneration initiatives & Regeneration links. PCU will shortly be offering services such as the following A Regeneration Zone website which may contain a searchable database of local evaluations, maps of and contacts, ongoing and planned research activity, specific guidance for conducting local evaluations, regional and local meeting & training dates and agendas, standard questionnaires and associated technical information to be downloaded, frequently asked questions, links to other relevant websites, information about standard software for quantitative & qualitative data analysis, a discussion forum, information on how the local and
121
national activities interface and strategies for co-ordinating national & local information collection. Locally collated evaluation details. The questionnaire may ask details of the following points: 1. Determine the local programmes and their support needs 2. Obtain details of those undertaking the local evaluation 3. Gain an understanding of the focus strategy and scope of the local evaluation 4. Obtain details of the range of methods and tools which are proposed to be used locally 5. Ascertain what information local evaluators are willing to share with each other and the outside world about their evaluations The outcome of the mapping process will be a database of evaluators and evaluations and is intended to be placed on the website and shared with significant others. North Staffordshire Evaluation Support Groups will meet when project request to meet to offer a chance to clarify and discuss issues regarding local evaluations.
122
LINKAGE WITH PRIVATE BUSINESSES Why is it important The private sector is an important provider of jobs, investment and services in neighbourhood renewal areas. It is also a key partner in regeneration, just like local residents. There are numerous examples of companies leaving neighbourhood renewal areas because of their poor environment, crime levels and often limited transport links. The private sector can be valuable in terms of Resources financial and staff Expertise in the design of strategies, project management and understanding how to deliver those outcomes that will be vital to a community Investment To capitalise on the opportunities often available in deprived areas Leadership ensuring sufficient priority to economic regeneration and the needs of businesses
However, at present, involvement is uneven and often people do not know how to make the most of it when they secure it. In fact the past has seen private sector representatives withdrawing from regeneration partnerships as their time was wasted. How to involve the private sector In practice the private sector can be involved as: A partner Advisors Consultation forum Mentors to staff There are perhaps 2 key lessons that have been learnt from seeking involvement of the private sector: Be certain about the role you want the private sector to play and what you are asking for Ensure there are benefits for the private sector representatives as well as the regeneration partnership
Business Improvement Districts Guidance a working draft (August 2003) A Business Improvement District (BID) is a partnerships arrangement through which local authorities and the local business community can take forward schemes which will benefit the local community, subject to the agreement of ratepayers. Ratepayers will agree to pay an additional levy on their rate bill to finance a BID. The rate payers themselves will decide in advance on the way in which their money is spent and how much they are prepare to pay in addition to their rates bills. BIDs can be located in town centre, in streets or larger areas, thus BIDs will encompass a broad range of projects in terms of their size and the issues they address. Businesss may be prepared to pay to secure preventative measures, such as more frequent policing, installation of CCTV etc.
123
The important point is that all parties in the BID process agree on the improvement that needs to be made and on the necessary steps to be taken to achieve it. BIDs involve collecting baseline positions from private businesses in the BID area, to give a position from which improvement can be measured. This would need to be done when private business involvement is within Regeneration projects. (See contacts list, ODPM and reference list for more information regarding BIDs) Community Interest Companies The Government is proposing a new type of company, the Community Interest Company, who profits will be used for the public good. The CIC will present new opportunities for social enterprises to benefit the people they serve: It will be easy to set up with all the flexibility & certainty of the company form, and some special features for guarantee that it is working for the benefit of the community or the wider public. Easy for investors to understand with an assured lock on profits and assets, to attract social investors, it will hep create a strong new brand for social enterprise and a new market place for social investment. By doing so it will cut the costs of setting up and financing social enterprises. It will recognise the strong links between social enterprise and the wider community, encouraging stakeholders to engage in enterprising communities. This in turn will improve the delivery of local services. Tesco Regeneration Partnership Tesco Regeneration Partnerships involve the retail giant investing in new stores in regeneration areas providing value and quality for customers while at the same time creating jobs and services in local communities. The relevant contact is through Leeds City Council. Business Brokerage Project Stoke-on-Trent (June 2003) The Business Broker pilot project is operating in nine areas throughout England (Stokeon-Trent, Sheffield, Nottingham, Hastings, West Cornwall, Eastington, Burnley, Luton & a joint project with Newham Brent & Greenwich). Funded by the Governments Neighbourhood Renewal Unit, the Home Office Activity Community Unit, the DTI Pheonix Fund, plus Stoke Works SRB6, it will run initially for 2 years. The project is in conjunction with the LSP and in partnership with Business in the Community, being managed locally by the North Staffordshire Chamber of Commerce and Industry. The Business Brokerage project was conceived to address problems of imperfect knowledge within the private sector of opportunities for support and to engage business in regeneration type activities.
Its mission statement was set down as: To effectively signpost businesses so that they can get involved in their local communities and local strategic partnership, in order to help regenerate Stoke-on-Trent
124
both socially and economically. This will be provided through a focused menu of opportunities and by effectively brokering partnerships. The projects four main strategic objectives are: 1. Developing two-way communication between local businesses and regeneration agencies 2. Helping improve the performance of businesses in Stoke-on-Trent, particularly those based in the 7-priority investment areas 3. Increasing business support to the community and voluntary sectors 4. Monitoring and evaluating performance and impact to improve effectiveness of interventions Achievements to date: The proforma template used during the interviews with the Business Brokers has generated valuable market information and has helped to understand the business community in the area Over 500 business interventions have taken place with 48 proformas being completed which are used to help signpost companies to relevant source of advice/information The large network of contacts which is being continually updated, is allowing business to be matched with appropriate organisations. This is effectively speeding up the companys ability to access services/information Companies at the various events/fairs organised by the Business Brokers has raised the profile of the Local Strategic Partnership. Companies have been made aware of regeneration issues and several business have also become involved in community projects Businesses have been made aware of the benefits of using local suppliers The profile of Business Brokers project has generated high numbers of Voluntary & Community Group enquiries many of these have been matched to private sector companies to develop a project (For further details contact Alan Woolley, Business Support Services 01889 590242) Working towards Green Businesses Small and medium sized enterprises (SMEs) represent ninety-nine percent of the UKs businesses. Small businesses have impacts on the environment and quality of life of local communities. Going Green is wider than just being environmentally aware and it can make good commercial sense some sustainability measures translate directly as business success factors, for example, better market access, cost savings, increase in productivity and improve brand value reputation. Examples of
125
Green Angels is a Brighton initiative where a local authority trained 31 people as Green Angels to support SMEs in an area designated for regeneration. The project provided support to some 600 businesses through offering advice, setting up a telephone help line and providing newsletters and training. Stepping Hill hospital uses a database to encourage employees to share car rides to work. This is a cheap way to organise and encourage more contact between employees. Employees who arrive to work as part of the care share scheme can park in a preferential parking location. Checklist for businesses 1. Is there an accessible source of case studies and examples for local businesses to read? 2. Are the methods and tools being developed easy to apply? 3. Is there any new or expected legislation that will encourage engagement with the green agenda? 4. Have the direct bottom line benefits been communicated to the businesses community? 5. If there are local business associations are they on board? 6. If there is no local businesses association, is there a forum where businesses can get together? Schemes to support businesses Envirowise offers practical environmental advice for business. For companies with less than 250 employees they offer a free confidential on-site waste review. This can help identify opportunities for you to become more efficient with your resources and make cost savings.
Action Energy is an initiative that has been designed to help businesses save energy and reduce the cost of the climate change levy through free advice, practical tips and access to finance. Interest free energy loans of between 5,000 and 50,000 to invest in energy efficient equipment are available for small businesses. SME Key is a key project of the European Business Campaign for Corporate Social Responsibility. Their website contains downloadable software that takes you step by step through all areas of your company activity, from human resources to the environment to help you complete an internal audit or publish a social report. FSB Federation of Small Businesses provides up to date links and information for those wanting to comply with environmental legislation, save money through resource efficiency or find funding for projects that will benefit the environment and a business.
126
WRAP Waste and Resource Action Programme is a not-for-profit company supported by funding from DEFRA, DTI and the devolved administrations of Scotland, Wales and Northern Ireland. It is working to promote sustainable waste management by creating stable and efficient markets for recycled materials and products. (renewal.net Solving the problem Green Business)
127
LINKAGE WITH OTHER PUBLIC SERVICES / ORGANISATIONS Why is it important to involve the wider public sector All major public service players must be involved in the Regeneration initiatives in some form, as they are all big players in serving the community and therefore should be involved in re-shaping the community. Major players include local health authority, primary care trust, learning and skills council, police authority. How to involve the wider public sector Examples of Sure Start Sure Start has a Support module for local evaluations, to support local evaluations in tracking how well their services are doing and to keep track of progress in meeting targets. The aim of the support is to facilitate local evaluations in achieving the main Sure Start objectives and to act as a conduit for information between the local evaluations and National Evaluation of Sure Start as a whole, ensuring that the local and national evaluations do not overlap and put undue pressure on families in the local area. The support structure includes 5 regional support units based on the Making Research Count University network (plus Lancaster University): These units are staffed one fulltime equivalent post who will be employed half-time as a fieldwork supervisor for the impact study and half-time to support local evaluations. The role of the units will include: 1. Making contact with local evaluators once initial mapping is undertaken, to discuss issues arising from local and national interface. 2. Disseminate examples of good practice in evaluation and monitoring activity at an earlier rather than later stage. 3. Organise regional workshops of local evaluators to address issues such as research methods, instruments, costing of services and action research. 4. Maintain links with relevant regional services 5. Co-ordinate in depth evaluations which will be conducted in collaboration with local evaluators The local evaluations will have close links with all aspects of the National Evaluation and these will be co-ordinated through the Central Office. A National Coordinator for Local Support will co-ordinate the regional offices and provide a link into the administration of the national evaluation, and national and regional workshops which will provide information and exchange between the national and local evaluations. The National team will provide materials for the local programmes so there will be more likelihood that similar questionnaires and information gathering techniques are used in local evaluations.
128
Workshops The purpose of the workshops that Sure Start have organised were to: 1. Introduce the National Evaluation strategy to local programme and local evaluators and discuss the interface between the two. This will include what programme can expect in terms of support & advice, the regional structure and timetables. 2. Consult local programmes and evaluators about the nature of support they require and how that support should best be organised 3. Consult on the nature and focus of any in depth collaborative evaluations. Connexions Staffordshire database Connexions Staffordshire have a database of supporting agencies that Personal Advisors can contact direct when they are confronted with young people who have a particular problem. The database consists of the area covered (north, stoke, west, east, all divisions) name of organisation, details of the org and contact point. The various organisations are stored under the following potential problem themes: Ethnic minorities, homelessness and accommodation, emotional wellbeing, info/advice, rights and advocacy, personal development and community development, health, disabilities and special needs, sexual health, family issues, substance misuse, and crime and offending.
129
INVOLVEMENT OF THE WIDER COMMUNITY What do we mean by community involvement There must be provision to ensure that residents genuinely have the chance to influence neighbourhood plans and secure their involvement, going beyond the usual suspects. Why involve the community Community Involvement is a must! By involving the community, you can receive and address issues at the grass root level and also capacity build evaluation skills. Therefore building a more cohesive community see indicators for measuring community cohesion below. Community participation and user involvement can result in a more relevant evaluation and improve response rates. Key stakeholders in your project, local people and service users can be involved at every stage of the process; setting the outcome measures, developing data collection methods, carrying out evaluation, assisting in the analysis and dissemination of the final product. How to involve the community Community engagement is a central plank, but it comes at a cost; burn-out of key local community players; intra-community tensions; the demands placed on the time and resources of NDC employees and agencies and so on. (NDC National Evaluation 2003)
130
Community Cohesion Indicators The Home Office has published a guide for Local Authorities and their Partners in how to measure community cohesion. The indicators are as follows and the information to evidence some indicators can be found through information collected for the purpose of other performance measures, as stated in the table below. Regeneration initiatives should take advantage of the opportunity to combine data collection/collected for other purposes. Indicator The percentage of people who feel that their local area is a place where people from different backgrounds can get on well together Collected by Local residents survey Cross reference to BVPI 03/04 QOL indicator 25 Home Office citizenship survey, main survey and local area boost BVPI general survey 03/04 Home Office citizenship survey, main survey and local area boost Statutory BVPI Home Office citizenship survey, main survey and local area boost QOL indicator 23 Definition To what extent do you disagree or agree that this local area is a place (15-20 mins walking distance) where people from different backgrounds get on well together (1-7 definitely agree definitely disagree) How strongly do you feel you belong to each of the following: Neighbourhood, local area, Local auth, County, England, Britain (1-5 very strongly dont know) Thinking generally, which of the things below would you say are most important in making somewhere a good place to live and which do you think most need improving in your local area access to nature sports and leisure facilities transport etc etc Do you agree or disagree that you can influence decisions affecting your local area? 1 5 definitely agree dont know To what extent do you agree or disagree that this local area (within 15/20 mins walk) is a place where residents respect ethnic
131
The % respondents who feel that they belong to their neighbourhood/local area/county/England/Wales/Britain
Local residents survey
Key priorities for improving an area
% adults surveyed who feel they can influence decisions affecting their local area % people who feel that local ethnic difference area respected
BVPI general survey 03/04 Home Office citizenship survey, main
Indicator
Number of racial incidents recorded by police authorities per 100,000 Local concentration of deprivation
Cross reference to survey and local area boost Home Office / HMIC BV174 Available on Home Office website Index of multiple deprivation -
Collected by
Definition difference between people? 1-5 definitely agree dont know Racial incidents are any incidents regarded as such by the victim or anyone else Population weighted average of the ranks of a district most deprived wards that contain exactly 10% of a districts population % pupils achieving 5 or more GCSEs at grade a* -C or equivalent in an authority secondary school A monthly job count of job seekers allowance claimants analysed by their age and the duration of their claim. In which of the following situations if any would be say you regularly meet and talk with people of a difference ethnic origin from yourself? In which of the following situations? Local shop relatives home youth club work etc etc
% pupils achieving 5 or more GCSEs at grade a* -C or equivalent in an authority secondary school % unemployed people claiming benefit who have been out of work for more than a year % people from different background who mix with other people from different background in everyday situations
Pupil level attainment schools census (PLASC) DfES National Online Manpower Information System (NOMIS) Claimant count Local residents survey
BV38
QOL indicator 2
BVPI general survey 03/04 Home Office citizenship survey, main survey and local area boost
132
You must consider: 1. What will motivate people to take part 2. What their role and responsibilities will be 3. How they will receive feedback 4. Training and support that they require to become involved You must consider what structures already exist rather than create something new from scratch. For example, there may be a Tenant Management Organisation, a community group or a local employer or business with a presence already in the area. More unusual ways of involving hard-to-reach groups may need to be explored, for example, outreach workers, or youth parliaments. Examples of community involvement Active Community Unit Sits within the Home Office, aiming promote the development of the voluntary and community sector and encourage people to become activity involved in their communities, particularly in deprived areas. Community Empowerment Fund Connexions Staffordshire Research Training Connexions Staffordshire have completed a draft-training programme for young people, which could be adapted for any aged participant, to give an introduction to the skills required to undertake research and evaluation. Young people could thus be able to participate more so in evaluation. Imagine method A community participation method based on an approach used in the USA called Appreciative enquiry. It builds a vision for the future using questions to focus peoples attention on success. The questions are designed to encourage people to tell stories from their own experience of what works and exploring why, to imagine and construct further success, ensuring that the future is created with a firm basis in reality. The method is based on the understanding that responsibility for decisions about how road-building or local health & education services are to be delivered ultimately lies with the wider community, the wider community has to be involved at the very heart of the process. For example, Ryedale District Council needed to reassure that its citizens that decisions were being made in the context of a shared vision of what people had agreed as their desired future for the area. In the course of using the Imagine approach, six themes emerged around which the authority developed its vision. The vision was thus a
133
contract drawn up between the council and the community against which the decisions about health, transport, education and housing can be checked. For imagine to work best, community involvement is the key. One core group must be formed which will include activitists, youth, older people, voluntary organisations etc. The group leader must be trained to ask and use appreciative questions, to draw out examples of personal experiences that identified what was most important to people in terms of values, aspirations and hopes. The themes could then be drawn out of the transcripts and conclude the vision statements, incorporating the exact words that people had used and not jargon. (lifted from an article on www.idea-knowledge.gov.uk, for further information on Imagine projects contact Richard Murray as below) The Market Towns Health Check book This health check comprises a number of measures that allows people to appraise the needs and opportunities of the town centre and helps them to devise a strategy for revitalisation (www.countryside.gov.uk/market-towns/pdf/healthcheck2002). Place check Similar to health check, but establishes what improvements would be of most benefit to the area. This tool has been developed by the Urban Design Alliance with the support of the English Partnerships and the ODPM. A Place Check can be initiated by anyone and can cover small or large areas. It prompts you to ask questions about the locations you are working in and how it could be improved. (www.placecheck.co.uk). Beacon Councils These councils have been recognised as centres of excellence in specific fields which may be of benefit (www.idea.gov.uk/beacons). Gaining community support before presenting to stakeholders The governments White Paper Strong Local Leadership suggests that to prevent abortive costs and unnecessary votes, the promoters of a bid should demonstrate that they have the support of a proportion, say 5% or 10% of affected community in that area. Training the community to evaluate themselves There are some examples around the country of how projects/organisations have trained young people/the community in research and evaluation skills, so that they can conduct or be part of the evaluation themselves. This adds immense value in capacity building. One example is Sure Start Rotherham Central, who have trained local mothers of young children in research techniques to deliver a parent carer questionnaire to others. Another example is the curriculum for young researchers training sessions run by SRB5 & Staffordshire University in May 2001.
134
There are certain organisations available who can train young people, parents, carers etc in the evaluation process and methods. Those identified so far include: Consumers in NHS Research (can give advice on where to go for training) The Neighbourhood Initiatives Foundation (who have worked with a number of projects) (contacts for these can be found in the contacts section) A Year of Liveability Challenges The IDeA has published a document which contains 55 innovative case studies of what local authorities are doing to improve the quality of peoples everyday lives. Crossing environmental, street scene, town planning, rural and community safety boundaries, the case studies have something to offer for everyone. It is especially relevant to those involved in youth councils, LSPs, community & voluntary groups, neighbourhood forums, local businesses, and Business Improvement Districts. Eastserve.com Website gets communities online Local residents are helping to combat exclusion by joining forces online through Eastserve.com an interactive website that is helping to connect individuals and services in East Manchester. To concept was developed out of an identified need to address the high unemployment levels, poor access to information and low PC ownership in the community. The pilot began in 2002. Over thirty recycled PCs have been installed in homes, schools and community centres, over thirty editors contribute to the site. Eastserve is now considered to be the first point of call for information in the local community. The project in funded by New Deal for Communities. For more information contact Ruth Hope, New East Manchester Regeneration Company (0161 230 8160). Social inclusion through ICT Central Government is keen to see the UK become a world leader in electronic business, recognising that ICT can provide powerful opportunities for equality of access to information, services and jobs. The choice of social exclusion through ICT as a category in the fourth round of beacon awards reflects the Governments desire to see local authorities playing a central role in creating a digital inclusive society.
Auditing Community Participation (Burns & Taylor 2000)

135
This handbook emerged from research on community involvement in the City Challenge & SRB partnerships & was designed for regeneration partnerships. The audit explores mapping the context for participation, the quality of participation strategies, the capacity of partner organisations, the capacity of communities to participate effectively & the development of impact assessments. Community Participation: A self-assessment toolkit for Partnerships (Engage East Midlands 2001) Contains general information on community participation, including breakdown of local communities, roles they play, symptoms of possible participation problems, a self assessment section which supports users to map current partnerships and to develop a future vision of community participation.
136
INVOLVING YOUNG PEOPLE Why involve young people The importance of consulting young people from the start cannot be overemphasised. (Childrens Fund Guidance on objectives, target setting, local monitoring and evaluation Nov 2001) How to involve young people In consulting young people as users of particular services, partnerships may wish to consider a group event or a series of focus groups to elicit their views. In the longer term, children and families themselves should be provided with the tools to take forward some of the monitoring and evaluation tasks, which will nurture ownership of the evaluation. (Childrens Fund Guidance on objectives, target setting, local monitoring and evaluation Nov 2001) Below is the agreed approach to measuring involvement of young people, which is being implemented from the date of publication of this Framework V3 and will be continually revised hereon. Note, this approach is the same approach as to measuring involvement of young people and other specific interest groups.
137
Measuring Involvement Of Specific Interest Groups Below is a proposed methodology for how to measure the involvement of specific interest groups of individuals within Regeneration. These indicators are built from a combination of Community Involvement Indicators (IDeA and Audit Commission 2002) and Hear by Right Standards for Involvement of Young People (National Youth Agency and Local Government Association 2003). As these indicators are only a sample of the indicators that are stated in the Community Involvement and Hear by Right documents, it is proposed that this realistic number of appropriate indicators are used initially and additional indicators are added as measurement of involvement is improved. Note: These indicators could be used for either measuring involvement of the BME, Young Persons or elderly groups etc in the community and therefore, in the following indicators these groups are generically referenced as the specific group.
Marie Broadhurst Evaluation Co-ordinator Connexions Staffordshire / Regeneration North Staffordshire Evaluation Framework for Regeneration Practitioners July 2004 - V3
138
Indicator Current situation of involvement is mapped
Resource and expertise for participation is mapped (Hear by Rights indicator) There is a dedicated budget for allocation by children and young people (Hear by Rights indicator) Interest in involvement is mapped at the grass root level (IDeA and Audit Commission indicator) Links with relevant organisations and services are established and maintained (Hear by Rights indicator) Development of confidence and competence of specific group (IDeA and Audit Commission indicator) Confidence of the specific group to resolve issues locally in a positive and proactive manner (IDeA and Audit Commission indicator) Confidence in the specific group by the local community, agencies and services (IDeA and Audit Commission indicator) Opportunities to shadow senior staff or undertake work experience (Hear by Rights indicator) The specific group are involved in feedback of progress to others (Hear by Rights indicator)
Examples of how to measure Numbers Names / postcode Methodology Barriers Considerations Budget available Expertise available Budget available Expertise available Management structure Numbers of individuals Number of consultation exercises conducted Names / postcode Considerations Barriers Names and numbers Details of how established activity with Details of on-going activity with Views from the other organisations and services Training details Experience details Tracking individual ability to undertake activities Ability of individuals to undertake activities successfully Feedback from within the specific group on how successful they can be Issues raised and barriers identified Feedback from community / agencies / services in how local issues have been resolved Details of actual engagement of the local community, agencies and services with the specific group Details of opportunities on offer Details of development undertaken, numbers and views of the experience Details of how the specific group give feedback to others, to whom, in what manner, timescales Feedback from receivers on how the information has been fed back to them How feedback has been used Neighbourhood Statistics Qualitative views from the neighbourhood residents and businesses Comparison to other neighbourhoods with regard to change and involvement practice Qualitative views from the neighbourhood residents and businesses Numbers and nature of involvement of neighbourhoods residents in local community activities or work Methods available for community engagement investment in community engagement Jobs created through community enterprise
An indication of the change that has happened in the neighbourhood due to involvement of this specific group (IDeA and Audit Commission indicator) An indication of community cohesion, a sense of belonging to the community and pride in the neighbourhood (IDeA and Audit Commission indicator)
139
It is proposed that the methodology that the Hear by Rights standard uses to plan improvement is used, as below. By observing the detail in the above indicators then weaknesses can be identified and improvement can be built in for the future. We have Priority Who? Resources Potential Potential How are identified the 1-10 needed? barriers? solutions? we going following to involve areas for (BME/YP) improvement in this process? 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Update on baseline position (qualitative or quantitative)
140
Principles of Children and Young People Participation The following principles are stated as should be underpinning any involvement of children and young people in decision-making by the DfES (2003). Regeneration initiatives could show evidence that they have achieved these principles: 1. Clear and visible commitment to involving children and young people with a route map of how to make it happen A public commitment to the principle and practice by Chief Officers, Governing Bodies and Leads, understood by everyone Setting out what will happen, for example at the strategic or community plan, reflecting the organisations needs and budget Setting out expectations, for example, consulting children and young people about their perceptions of priorities for improvement 2. Children and young peoples involvement is valued Respecting and trusting children and young people Considering their age, maturity and understanding Making appropriate information available, timely and relevant Managing expectations, being clear about what is out-of-bounds, for practical, legal or political reasons Providing clear and timely feedback on outcomes 3. Children and young people have equality of opportunity to be involved No-one is excluded or prevented from getting involved on grounds of race, religion, culture, disability, age, ethnic origin, language or the area in which they live Special support is provided to help hard-to-reach groups and those facing the greatest barriers The level of confidentiality and privacy that applies to the decision-making process is made clear to all participants 4. Children and young peoples participation and involvement is continually evaluated and reviewed Measure how well the plan is meeting the rationale and success criteria set out from the start Involve children and young people in the evaluation process and reviewing lessons learned 5. Quality Standards Abide by agreed quality standards and code of conduct for working with children and young people Handle child protection issues in line with established standards (Working together: Giving children and young people a say - July 2003)
141
Connexions Staffordshire Tracking Young Persons Connexions Staffordshire has a database called Aspire, which contains continually updated information regarding all 13-19 year olds throughout North Staffordshire in compulsory education. This data is collected directly from the LEA information on all year 9s, through interviews conducted by 200 delivery staff in schools, through group sessions in schools and working with key partners for example the Youth Service. Information held on the database includes, personal details, gender/ethnicity, part of vulnerable groups, learning disability or health difficulties, future plans/career route and any other notes. The database also holds information regarding 16-19 year olds throughout North Staffordshire in post-compulsory education and special needs clients up to 25 years. Data is collected in the same way as with 13-19 year olds and regarding the same details, but also including achievements and qualifications, history of situations (work/unemployment etc) and work with other agencies. The central contact for using this service is through the Knowledge Management Team based at the City Council. This database most importantly has the ability to track individual young people over time, tracking where they are now, employment situation, learning difficulties, other services involved with etc. All projects under the Programme Co-ordination Unit will be required to forward personal client details for tracking purposes to Connexions Staffordshire, unless projects have extenuating circumstances. There must always be a data sharing agreement in place. The process for tracking young peoples involvement is as follows: 1. Data regarding young people is requested from projects by the PCU 2. Projects forward young persons details to Connexions Intelligence team 3. Connexions meet with the project in question 4. Connexions run data findings on Aspire database 5. Connexions meet with project to check initial results and accuracy of the data 6. Connexions make any amendments to the data 7. Both the project, Connexions and Evaluation Co-ordinator agree the additional qualitative evaluation information that should be added to produce the whole evaluation report of the project 8. All parties agree the report and sign off report, including Connexions Staffordshire, the project, the Evaluation Co-ordinator and the PCU Programme Appraisal and Monitoring Manager. The central contact for using this service is through David Byatt, Business Intelligence Team at Connexions Staffordshire (01785 255755). All initial equiries should be made through the PCU.
142
Examples of involvement Good practice in young peoples involvement in North Staffordshire (May 2001). This is a joint project between professor Ruth Chambers, Ms Sue Linnell (Staffordshire University) & Mr Yousiff Meah (SRB5 Stoke-on-Trent) The conclusion states that, In North Staffordshirethere were many examples of good practice. But in general, youth participation seemed to be at the level of information exchange that is mainly confined to eliciting young peoples views rather than involving them in decision making or priority setting (p.29) The project trained young people in research and evaluation techniques with a curriculum which covered such areas as interview skills, how to ask a question, sampling methods and focus group sessions. (see appendix for core principles & good practice for involving young people, suggested by the project) Working together: Giving children and young people a say (July 2003). This draft guidance has been circulated for consultation until November. The document suggests a checklist for ensuring effective participation of young people, with examples. This checklist can be seen in the appendix. Neighbourhood Renewal Unit The National Evaluation Team within the NRU is involved with a piece of work to look in more detail at the theme of community engagement within the NRU programme, including young people. This could include work to develop the NDC Performance Management Framework to help set standards for youth engagement for NDC Partnerships. Faith in Young People in Regeneration Is part of the LARCH project, run by the Childrens Society. The purpose of the project is to ensure that the decision makers within systems that affect the lives of young people listen and respond to their young service users. The project has produced a Faith in Young People in the Regeneration Resource Tool Kit that is ideal for anyone wanting to address the issues of regeneration within young people of all ages. Packed full of tried and tested ideas, it provides a background on childrens rights and also offers tips on how projects can be sustained throughout the regeneration process. National Community Fire Safety Centre Guidance and partnership information on working with children and young people can be found in its online Toolbox for fire brigades and stakeholders (www.firesafetytoolbox.org.uk). RegenWM
143
The Centre of Regeneration Excellence in the West Midlands is planning for the launch of a Graduate Placement programme aimed at attracting undergraduates & graduates into regeneration-related positions within the region. The programme is dependent on Regeneration-related organisations offering to take on volunteer(s). It is envisaged that successful candidates will be placed with relevant organisations within regeneration and will follow a structured programme with a personal mentor for support. DreamScheme A DreamScheme is a community based project which engages young people into community activities and for every hour of work they do they receive points which go towards trips of their choice. By involving young people in solving the problems within their communities a sense of pride and belonging will be generated. It provides them with a sense of control and achievement that in turn instils community pride. Community relations improve an young people are no longer seen as the cause of all problems but instead begin to be viewed in a more positive light. There are DreamSchemes in operation across the country, the local contact is LisaJones whose contact details can be seen in the contacts section below. Promoting that Planning is a positive force The ODPM is leading a working party comprising amongst others CABE, Landscape Institution, Royal Geographical Society and the Royal Town Planning Institute. It will make a case for planning, promoting its past achievements and potential to positively influence the built and natural environment for the benefit of the population at large. Specifically it will aim to capture young peoples imagination that planning is a positive force to achieve sustainable communities and improve the quality of life for all, something one should have an understanding of an potentially pursuer as a career. Promoting Planning in school & colleges The RGS is currently working with the QCA on a new GCSE course specifically for geography and citizenship. The ODPM is considering how it might help raise the profile of urban geography/planning in the new courses. Student Force.org.uk A registered Charity that has been operating since 1996, whose vision is to obtain more sustainable ways of living, working, learning and developing for young people in the future, who are actively boosting the local community, economy and environment. There mission is to be recognised as the most effective and innovative organisation that enables young people to work and learn with communities and employers to boost their opportunities to both promote sustainability and to become more employable. They are funded by Lottery bodies, charitable trusts, European Union, Large & local businesses, Government departments and agencies and landfill operators. The business of the charity is as follows:
Work with all young people, undergraduates, graduates and post graduates
144
Provide project work of 2 to 12 months on most aspects of sustainable development Build project workers who are keen, bright, semi-skilled, low-cost Give support as guidance, mentoring, training and networking Develop new partnership programmes Offer volunteering and placement opportunities Provide grants, especially for communities and small businesses Provide bursaries and awards for young people to be placed with hosts
Projects underway as from October 2003: Recycling project workers Two positions to undertake the day-to-day co-ordination and implementation of the dry recyclable waste collection scheme on behalf of Forest Heath District and St Edmundsbury Councils. 300 / week for a 23 weeks duration. Waste minimisation team leader To promote existing sustainable waste management services provided by the Essex Local authorities and other organisations to the residents of Essex through a direct faceto-face marketing campaign. The role of Team Leader involves line-managing 4 Projectworkers and delivering district reports for doorstepping and participation monitoring. Previous experience of leadership, doorstepping and participation monitoring of kerbside collection schemes would be an advantage but is not essential since training will be provided. 220 / week for 23 weeks duration. Wet Woodlands project officer To develop a strategy (based on GIS mapping and liaison work) for wet woodland creation, expansion and management in the Nene and Welland Valley areas of the Peterborough City Council area. As well as office-based collation of data and liaison with landowners and other partners, the officer will also carry out or implement additional survey work to help fill in gaps in data coverage for opportunity mapping 250 / week for 36 weeks duration. Connexions Staffordshire Research Training Connexions Staffordshire have completed a draft training programme for young people, which could be adapted for any aged participant, to give an introduction to the skills required to undertake research and evaluation. They evaluate involvement through a
145
variety of evaluation questionnaires and focus group methods, depending on the appropriateness of the situation. East Hampshire District Council Run a citizenship module for the local schools curriculum and have an annual social event with Councillors, Officers and young people. North Lincolnshire Council Have a series of annual and ongoing events. As an example, a Great Youth Debate is followed up by young people making presentations to Full Council and Cabinet meetings. Surrey County Council Have a website for young people ages 11-19 years that is used to host online consultations between young people and organisations across the county. There is also a discussion board and the facility for young people to put videos on the site.
146
CAPACITY BUILDING What is capacity building Capacity building is the method of ensuring that everyone who wants to be involved in Regeneration, can be. Capacity building is for both community representatives and professions alike. (How to Build Capacity renewal.net) Why is capacity building important Regeneration agencies need to recognise the pressures being places on community organisations through their participation in partnerships. Volunteers now put forward as representatives and delegates of their organisation or neighbourhood, are required to have a sophisticated level of management and communication skills. Participation in partnerships consequently needs resource provision to assist the growth of skill and confidence of community representatives. (How to Build Capacity renewal.net) How to capacity build As you start to plan capacity building activities within your neighbourhood, you should begin by establishing what sort of community you are working with and identifying the existing capacity of partner organisations to support community participation. You should consider: What is the range and level of local community activity? Do those involved have any experience of project management? What communities are there within the localities covered by the partnership How do communities work together? (Do they work together?) Has a particular section of the community dominated any previous partnership? What are the local barriers to engaging communities? Do people live in close proximity to one another, or must they travel to get together? How effective is participatory decision making? What are the outcomes of participation? Who benefits from participation? How clearly is the area of benefit defined and understood by everyone? Are community groups able to run in an effective and inclusive way? How do groups ensure that their representatives are accountable? How accessible are local meetings? Are mainstream service structures compatible with community participation? What investment is made in developing and sustaining community participation? How strong is the leadership within partnerships and partner organisations?
Various elements of training for all participants include:

147
Governance issues including roles and responsibilities of board members and house rules for their behaviour Equalities issues, covering race, gender, age and abilities Developing the vision to show the bigger picture Advice on risk management Board election dos and dont Structure of partnerships Monitoring of the regeneration programme by board members Team working Project appraisal Finance for community organisations Relationship management Community Development Influencing and negotiation skills Communication and presentation skills Project management Developing individual development plans and opportunities for individual mentoring
Communication opportunities include the following: 1. 2. 3. 4. Information stalls (giving and receiving information) Open days and surgeries where local people can drop in for information Involving local schools, breakfast clubs, faith group activities Publicising success to encourage further involvement (using notice boards, signage for projects, programme logo on all material etc)
Capacity building needs to be: 1. Systematic, taking a full account and detailed look at training and organisational development needs 2. Comprehensive, looking at the needs of individuals in the community organisation itself as well as the environment in which the organisation operates 3. Forward looking, taking stock of the existing situation, helping community groups to plan ahead in terms of the skills and type of organisation needed to achieve future aims. (How to Build Capacity renewal.net)
Examples of capacity building

148
An example of outcome capacity building measures can be seen below BASELINES Consultation % of local population responding to consultation % of people with disadvantages in the local population responding to consultation Partnership Number of partnership decisions modified by community views Extent of support and development available to community & voluntary sector Beneficiaries Number, range and condition of community group/voluntary organisation operating Proportion of local population actively involved in 1+ groups/organisations Sustainability Extent of support and development available to the community and voluntary sector % of local people feeling ownership of public developments and obligation to sustain them OUTCOMES High % of local population responding to consultation High % of people with disadvantages respond to consultation Partnership decisions frequently modified by community views Long term community/voluntary sector support and development strategy in place Wide range of community groups and voluntary organisations operating effectively High % of local population actively involved in 1+ groups/organisations MILESTONES Measures in place to create a climate of participation and dialogue Measures in place to identify people with disadvantages and target them for participation Partnership decisions take account increasingly of community views Assessment of long term infra structure, assets and mechanisms for support of C/V sector Information in place to assess number/range/condition of community/voluntary organisations operating Measures in place to stimulate greater involvement of local people in community/voluntary organisations
Long term Assessment of long-term community/voluntary infra-structure, assets and sector support and mechanisms for support of development strategy in c/v sector place High % of local people feel Measure in place to ownership of scheme stimulate greater public results and commitment to sense of ownership of the sustain them scheme and its results (SRB Guidance manual revised, May 2003)
Example of improving capacity building Aim: To improve the capacity and quality of voluntary organisations and community groups
149
Outcomes: -To increase knowledge -Increase in organisational good practice -Increased organisational ability to provide required services Outcome indicators: -% service users who say their knowledge has increased -No. service users applying learning in their organisation -% participants on accredited courses achieving qualifications -% organisations developing organisational policies and procedures -Organisational funding -Staffing levels -Proportion of organisations obtaining legal status Examples evaluation questions: -How effective are we in improving the capacity of organisations? -Do our activities result in organisations adopting good practice and policies? -Are we improving our service users skills and knowledge? (Measuring Effectiveness NACVS June 2002) Building a picture of community cohesion The above document was published in June 2003 as a guide for Local Authorities and their partners to help improve the understanding of what community cohesion look like on the ground. The guide states the indicators, sources for collecting the indicators, what central government is doing & benchmarking. For each indicator it is stated where to collect the evidence, cross reference with other indicators, definition & relevance. The indicators are set out below: Headline outcome: The percentage of people who feel that their local area is a place where people from different backgrounds can get on well together Common vision and sense of belonging: The percentage of respondents who feel that they belong to their neighbourhood/town/country/England/Wales/Britain. Key priorities for improving an area The percentage of adults surveyed who feel they can influence decisions affecting their local area.
The diversity of peoples backgrounds and circumstance are appreciated and positively valued: The percentage of people who feel that local ethnic differences are respected. Number of racial incidents recorded by the Police authorities per 1000,000.
150
Those from different backgrounds have similar life opportunities: Local concentration of deprivation. The percentage of pupils achieving 5 or more GCSEs at grades A*-C or equivalent. The percentage of unemployed people claiming benefit who have been out of work for more than a year. Strong and positive relationships are being developed between people from different backgrounds in the workplace, schools and neighbourhoods: The percentage of people from different backgrounds who mix with other people from different backgrounds in everyday situations. Capacity building in local government research on capacity building needs (August 2003) This report sets out the findings and recommendations from OPMs research, commissioned by the ODPM into the capacity-building needs of local government, published August 2003. The support needed for capacity building concluded as follows (note the similarity of these to what this whole document recommends for effective & efficient evaluation): 1. A more systematic approach to peer support, along the lines of a Peer Clearing House that would broker the arrangements and quality assure those offering peer support. IDeA was felt to be an appropriate organisation to deliver this type of service. 2. Developing models and blueprints: Providing practical guidance about implementation of successful pathfinders or pilots. General toolkits should be provided on issues such as project management. There should be standard documents such as contracts and forms. 3. Central database that gives examples of noteworthy practice & different models and approaches used for service delivery, based on information gathered during the CPA process. 4. Broad-based public service programmes, including leadership and management development programmes. 5. Commissioning external support: Need to identify needs clearly, have a central resource to point towards possible companies, which would include a basic quality assurance of those suppliers & consulting with suppliers before drawing up specifications. 6. Carrying out effective evaluation of external support, to evaluate the longterm effect of any capacity-building and development activity.
Specific recommendations that have been discussed & funding identified are below: Develop the Local Government Graduate Development Programme to build capacity in the longer term.
151
Bringing more graduate into local government will be an important mechanism for building long-term capacity in the sector, measures also being put into place to keep these graduates within the local government workforce and to talent spot and develop other high-performing officers early in their careers. IDeA has been commissioned to establish a brokerage service to match local authorities wanting peer support with officers and members willing to provide it. Helping local authorities to prioritise and deliver their improvement plans. Initially to be provided by IDeA and targeted at authorities in the poor and weak categories, but it may be subsequently available to any council needing help with improving delivery. Increased places on Leadership Management courses. Establish a procurement competence level possibly through procurement management distance learning or e-learning material. ODPM are to commission a programme geared specifically at supporting the needs of district councils in responding to CPA assessment.
Recommendations up for discussion: Have a central, searchable web-based best practice database, covering all service areas. Extending an existing electronic best practice service would probably be the least costly option. Encourage the Regional Bodies to establish & maintain networks with those leading organisation development & change in their region. Beacon Authorities The schemes value in identifying noteworthy practice and raising standards has been confirmed with a long term government funding commitment to rounds 5, 6 & 7. Mean while the ODPM has already increased funding available for the sharing of good practice in round 4: Beacons will receive a share of 3 million, equating to roughly 50,000 each and an 8,000 increase in round 3. To further wider service improvement options, an additional 2 million has been made available to allow beacons to work directly with local authorities that have been underperforming in beacon themes areas. Once Beacons have risen to the challenge of securing beacon status, beacons then have the challenge of putting their message across to the outside world. In previous rounds, they have bought as much innovation to dissemination as they did to their original achievements, using online discussion forums, case study material and good practice tool-kits. Contact details should be directed at the IDeA, facilitators of the scheme. UPDATE: Delivering Sustainable Communities magazine Issue one of the new 'Update: Delivering Sustainable Communities' magazine was published September 2003. The magazine is intended as a communications tool for
152
disseminating news and progress towards delivering sustainable communities. This edition includes information on the Sustainable Communities Delivery Unit, the affordable homes initiative and market renewal pathfinders. Beacon open days Beacon Councils for 2003 will be sharing their excellence and innovation through a series of open days, under the following themes: Child & adult mental health services Community cohesion Quality of the built environment Removing barriers to work Rethinking construction Social inclusion through ICT Street highway works Supporting the rural economy Tackling homelessness Transforming secondary education With workshops, presentation, site visits and study tours, these interactive sessions are a fantastic opportunity to meet the officers, members and partners behind beacon excellence and gain detailed insight to the structures and practices that have laid the foundations for success. For enquiries call the IDeA beacon team on 020 7296 6587. Publication of innovative case studies A Year of Liveability Challenges is a new publication from the IDeA containing 55 innovative case studies of what local authorities are doing to improve the quality of peoples everyday life. The idea is that each week there is a challenge why dont we do that? Regional Seminars on Sustainable Communities Following February 2003 launch of the Communities Plan, a series of seminars are being held across the country to focus on the ways and means of delivery. Key players form each region are being invited to discuss the role of government and stakeholders in achieving Sustainable Communities. Seminars consider key regional issues and the potential roles of regional players in delivery plans. The first seminar took place in July in Essex, the second seminar on September 11th in Portsmouth. Further information on this programme will appear in future issues of the publication magazine, Update. Making Things Happen capacity building training and small grant programme The Making Things Happen Capacity Building Training and Small Grants Programme is run residentially at Trafford Hall, home of the National Tenants Resource Centre. The programme is funded by the ODPM as part of the National Tenant Empowerment
153
Grant Programme, with additional funding streams coming from the Housing Corporation and Welsh Assembly Government to enable tenants to attend and apply for grants. The Capacity Building programme provides tenants with: Skills, tools and confidence to improve local conditions and make a positive contribution to their communities and their lives. An understanding of Government initiatives and policy. An understanding of the impact of Government action. Example: Lymebourne and Arcot Park Tenants Association, Devon had a lack of energy awareness within the community, leading to unnecessary costs for the tenants and landlords. The group held a one-day conference for residents and local authority staff to highlight energy issues. A pilot scheme was set up to install light sensitive switches for lighting in a sheltered housing scheme to encourage energy saving across the LA area. For further information contact www.traffordhall.com 01244 300246 b.Watson@traffordhall.com
154
INVOLVING THE BME COMMUNITY Why involve BME communities The National Evaluation of NDC (New Deal for Communities) undertaken during 2002 concluded that, Partnerships have generally attempted to engage with BME communities, but not all have done so and the links between BME needs and aspirations on the one hand and appropriate projects and outcomes on the other are not always evident. flesh out moreso. How to measure involvement Below is the agreed approach to measuring involvement of BME groups, which is being implemented from the date of publication of this Framework V3 and will be continually revised hereon. Note, this approach is the same approach as to measuring involvement of young people and other specific interest groups.
155
Measuring Involvement Of Specific Interest Groups Below is a proposed methodology for how to measure the involvement of specific interest groups of individuals within Regeneration. These indicators are built from a combination of Community Involvement Indicators (IDeA and Audit Commission 2002) and Hear by Right Standards for Involvement of Young People (National Youth Agency and Local Government Association 2003). As these indicators are only a sample of the indicators that are stated in the Community Involvement and Hear by Right documents, it is proposed that this realistic number of appropriate indicators are used initially and additional indicators are added as measurement of involvement is improved. Note: These indicators could be used for either measuring involvement of the BME, Young Persons or elderly groups etc in the community and therefore, in the following indicators these groups are generically referenced as the specific group.
156
Indicator Current situation of involvement is mapped
Resource and expertise for participation is mapped (Hear by Rights indicator) There is a dedicated budget for allocation by children and young people (Hear by Rights indicator) Interest in involvement is mapped at the grass root level (IDeA and Audit Commission indicator) Links with relevant organisations and services are established and maintained (Hear by Rights indicator) Development of confidence and competence of specific group (IDeA and Audit Commission indicator) Confidence of the specific group to resolve issues locally in a positive and proactive manner (IDeA and Audit Commission indicator) Confidence in the specific group by the local community, agencies and services (IDeA and Audit Commission indicator) Opportunities to shadow senior staff or undertake work experience (Hear by Rights indicator) The specific group are involved in feedback of progress to others (Hear by Rights indicator)
Examples of how to measure Numbers Names / postcode Methodology Barriers Considerations Budget available Expertise available Budget available Expertise available Management structure Numbers of individuals Number of consultation exercises conducted Names / postcode Considerations Barriers Names and numbers Details of how established activity with Details of on-going activity with Views from the other organisations and services Training details Experience details Tracking individual ability to undertake activities Ability of individuals to undertake activities successfully Feedback from within the specific group on how successful they can be Issues raised and barriers identified Feedback from community / agencies / services in how local issues have been resolved Details of actual engagement of the local community, agencies and services with the specific group Details of opportunities on offer Details of development undertaken, numbers and views of the experience Details of how the specific group give feedback to others, to whom, in what manner, timescales Feedback from receivers on how the information has been fed back to them How feedback has been used Neighbourhood Statistics Qualitative views from the neighbourhood residents and businesses Comparison to other neighbourhoods with regard to change and involvement practice Qualitative views from the neighbourhood residents and businesses Numbers and nature of involvement of neighbourhoods residents in local community activities or work Methods available for community engagement investment in community engagement Jobs created through community enterprise
An indication of the change that has happened in the neighbourhood due to involvement of this specific group (IDeA and Audit Commission indicator) An indication of community cohesion, a sense of belonging to the community and pride in the neighbourhood (IDeA and Audit Commission indicator)
157
It is proposed that the methodology that the Hear by Rights standard uses to plan improvement is used, as below. By observing the detail in the above indicators then weaknesses can be identified and improvement can be built in for the future. We have Priority Who? Resources Potential Potential How are identified the 1-10 needed? barriers? solutions? we going following to involve areas for (BME/YP) improvement in this process? 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Examples of involvement Update on baseline position (qualitative or quantitative)
158
INVOLVING THE ELDERLY Add proposed indicators as previous sections. To be researched
159
EVALUATING PARTNERSHIPS What do we mean by Partnership working Working with related others to get a job done evaluating partnerships usually involved stating who you are working with, in what capacity, timescales, where, why and how. This information can then be fed back into the process to improve Partnership working. Why work in Partnership The best regeneration partnerships under SRB achieved good partnership working by avoiding missing-out partners if it was at all possible, or at least ensuring that the partnership scheme had access to the missing partners through a link to a wider more strategic local partnership on which they were represented. (The impact of SRB on Final Outcomes National Evaluation SRB Team 2002) Partnership working is an essential ingredient to achieving longer-term outcomes in the five main theme areas. (NDC National Evaluation 2002/03) How to evaluate Partnerships The method below is recommended as the best approach to evaluate and monitor Partnership working in Regeneration, answers which can then be recorded at the Programme Co-ordination Unit to report to Regional bodies. (Whole process modified by Marie Broadhurst, Regeneration North Staffordshire). Gaining & sharing data with partner agencies/data protection A tool for Evaluating Economic Development & Regeneration Partnerships This toolkit aims to assist Economic Regeneration to build key partnerships, avoid partnership overload, ensure partnerships are effective and efficient, are action and outcome focused and are able to have their progress measured effectively, to demonstrate the value of partnerships to stakeholders and Government, for example in the context of Best Value and Comprehensive Performance Assessment. The toolkit takes you through all stages of the evaluation of partnerships process: 1. Mapping the partnership landscape identifying the partnerships that help you deliver economic development & regeneration. 2. Prioritising partnerships where you have a large number. 3. Evaluating individual partnerships identifying strengths & weaknesses, assessing whether they are action focused, efficient & inclusive, identifying ways to improve performance and actions you want to take in light of the evaluation. 4. Taking action and planning the next stages.
160
Projects are advised to work through the following questions: 1. List the key aims/objectives/priorities for your Regeneration initiative: a). b) c). d).. e).. 2. List all Partnerships relevant to and affecting your Regeneration initiatives and make an initial assessment of them in terms of a) Relative importance to key priorities b) Performance and effectiveness Partner(s) a) Key Partnership? Yes No b) Performance? Good Adequate Poor
3. Map existing Partners to the following landscape of economic development: Mainstream activity Transport Education Community Development Tourism Highways Housing Land Use Environmental / Property Services Procurement Personnel 4. Map existing Partners by geographical area: Geographical area Local/District Sub-regional / County
Partner(s)
Partner(s)
161
Regional National European and International The following questions are for evaluating individual Partners: 5. At which stage is the Partnership at at the present time: Stage of Partnership Forming Frustration Functioning Flying Non-productive Productive Partner(s)
6. The Regeneration initiative must ask questions about the Partner regarding the following areas:
7. Summary assessment of the Partnership: Is the Partnership contributing to the Regeneration initiative priorities? Partner(s) Initiative priorities Yes Contributes? No
8. On balance, is the Partnership: Partner(s) Assessment Action Focused Efficient Inclusive Learning and developing Performance managed A key partnership Performing Is the Partnership. Yes No
162
9. Should the Partnership continue? Partner(s) Yes Continue? No
10. Translate the most important actions identified in the Partnership evaluation criteria into an overall action plan, relevant to the partnerships future: We have Priority Who? Resources Potential Potential How are identified the 1-10 needed? barriers? solutions? we going following to involve areas for (Partners) improvement in this process? 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. Partnership Toolkit Health Action Zone North Staffordshire (2002) This method is recommended as the best approach to for individual projects to evaluate and monitor Partnership working in Regeneration, providing qualitative answers that can then be used for internal purposes. This Partnerships toolkit contains a checklist of the fundamental principles for partnership working. The toolkit then takes the reader through steps for defining the need for a Partnership, starting the process, setting and maintaining a Partnership and evaluating and monitoring a Partnership. Workbook questions take the reader through this process, recording answers in a qualitative format, ideal for internal detailed use. The workbook gives an example of a Partnership agreement, terms of reference and a business plan. Contact junestokes@nsha.wmids.nhs.uk - Health Action Zone Stoke
Update on baseline position (qualitative or quantitative)
163
Example of a Partnership Questionnaire
Partnership Evaluation Questionnaire
1. List the key aims/objectives/priorities from your perspective for working in this
Partnership: a). b) c). d).. e).. 2. List all Partnerships that you work with to provide the service you provide and make an initial assessment of them in terms of a) Relative importance to key priorities b) Performance and effectiveness Partner(s) a) Key Partnership? Yes No b) Performance? Good Adequate Poor
2. How would you rate your contribution to this partnership? (please circle)
Vgood____________good____________average____________poor_________vpoor
3. What has worked well by being in this partnership?
______________________________________________________________________ ______________________________________________________________________ ___________________________________________________________________

4. What has not worked well by being in this partnership?
______________________________________________________________________ ______________________________________________________________________ ___________________________________________________________________

5. How can we change so that we can both work better in the future?
______________________________________________________________________ ______________________________________________________________________ ___________________________________________________________________

164
6. Are there any organisations/service that are not included in our partnership
working that should be? Give reasons why ______________________________________________________________________ ______________________________________________________________________ ___________________________________________________________________
7. What do you think would happen if this service and our partnership did not
continue after March 2006? How will this impact on young people? ______________________________________________________________________ ______________________________________________________________________ ___________________________________________________________________
8. How can we ensure that the service we provide does continue after March 2005?
______________________________________________________________________ ______________________________________________________________________ ___________________________________________________________________ 10. Do you have any other comments that could improve the way in which we work in partnership together and improve the impact that we have on young people? ______________________________________________________________________ ______________________________________________________________________ ___________________________________________________________________
165
Connexions Partnership Working toolkit Checklists of noteworthy practice This toolkit is based on research from the Building Links Project, which was commissioned by Government Office for the South East in September 2002. The aim of the project was to find out how the outcomes of childrens and young peoples service can be improved. The research produced a number of ideas and suggestions for increasing the effectiveness of partnership working in childrens and young peoples services. The toolkit contains a number of checklists stating noteworthy practice that may be useful for projects when setting up, maintaining or evaluating Partnerships: 1. Managing meetings 2. Service planning 3. Service delivery (including having common operational policies, joint monitoring & evaluation, common identification and tracking systems, generic workers, posts seconded between partnerships, joint management etc) 4. Research (pulling together existing research, combine efforts for service mapping etc) 5. Involvement of children & young people & parents 6. Involvement of the voluntary and community sector 7. Involvement of the private sector 8. Working with different age groups 9. Transition 10. Communication 11. Information sharing (including a central database and a single assessment framework) 12. Links beyond childrens & young peoples service partnerships 13. Training (joint training in essential skills, local & regional meetings etc) 14. Working relationships 15. Managing time & priorities The toolkit also contains an assessment checklist designed for use by all young peoples services partnerships. The exercise enables services to assess the quality of your partnership working, concluding in a score. It should be done by the whole partnership or by a small but representative group, rather than by an individual. All checklists can be found at the following links: www.go-se.gov.uk
166
Examples of other toolkits available to evaluate partnership working Below are examples of other toolkits that may assist you in setting up, maintaining and evaluating Partnerships some of the elements which you may find useful. IDeA Knowledge Partnership toolkits This includes guidance on: Structured partnership planning The evaluation process beginning with identifying partnerships to help deliver economic development & regeneration Evaluating individual partnerships to help identify strengths and weaknesses and assess whether partnerships are action focussed, efficient and inclusive Action & planning help in taking action and planning the next stages in building effective partnerships (www.idea.gov.uk) Integrating Action Locally: A Toolkit for Partnerships (Keech & Wilks 2001) Describes the three main areas of organisational shift required to create the optimum environment for successful partnerships organisational development as major mind shifts, strategic change and the development of front line staff, exercise, checklists, guidance and local experience. Partnerships and Networks: An Evaluation and Development Manual (McCabe, Lowndes & Skelcher (1997) Draws on the authors recognition of the multi-agency nature of many community programmes, including urban regeneration and community safety. Tis toolkit provides practical activities for partnerships to review, assess, check, evaluate and plan activities across both partnerships and networks. It provides clear approaches towards understanding the complexities about partnerships and networks through attitudes, accountability, leadership and tensions between them. Partnership Self Assessment toolkit (Frearson 2002) The author highlights the potential for duplication of effort and dilution of resources and expertise as partnerships continue to emerge. Helps readers to understand the requirements of forming a partnership. Key areas for review are described and include maintaining successful partnerships, the need to understand partners, their cultures and their skills, arrangements for communication and involvement, the process for measuring the impact of a partnerships work and reviewing its success.
167
Local Strategic Partnerships: Lessons from New Commitment to Regeneration (Russell 2001) Proposes a new approach to tackling regeneration through partnership at the strategic level. Significant ingredients of change include shared values, strategy, style, staff & skills. Making Partnerships Work: A Practical Guide for the Public, Private, Voluntary & Community Sectors (Wilson & Charlton 1997) Highlights the wide range of variables that have an impact on approaches to partnership: Stakeholder mapping, community involvement, strategies for informed decision making, clear communication and promotion, co-ordination of management boards and teams, the importance of personal skills and conflict management, measure of inputs and outputs and positive publicity and reflection. Partnership Working in Rural Regeneration: Governance & Empowerment (Edwards, Goodwin, Pemberton & Woods 2000) This guidance is strongly supported by six case studies which describe issues of good practice, selection of partners, community representation, committee structures, key factors associated with working in partnership, funding roles, ineffectiveness & constraints, commitment, competence, empowerment and accountability. Quality Standards in Inter-agency Work (Hughes & Settle 2001) Offers and quality standards framework for inter-agency work. Issues explained include aims, commitment, management, communication, evaluation, accountability, levels of strategic leadership, executive management & operational implementation. The standard offers statements on these issues under four headings leadership, operational management, communication and information exchange, monitoring and evaluation.
168
CAPACITY BUILDING
169
Section 5. SUPPORT
170
SUPPORT AND GUIDANCE AVAILABLE Programme Co-ordination Unit This team is based at Trentham, Stoke-on-Trent, and consists of monitoring, evaluation and development officers for Regeneration Projects throughout North Staffordshire. This should be projects first point of contact if they are in doubt about any issues relating to project development, monitoring and evaluation. The contact details are 01782 Evaluation Co-ordinator for Regeneration North Staffordshire There is one officer in post to assist individual projects with the evaluation of their projects. This officer is a full time post available to discuss and give guidance on evaluation issues either on the telephone, through personal visits and via email. This post will be regularly visiting projects throughout their delivery period to provide ongoing evaluation support. It is vital that if projects are in any doubt about evaluating then projects make this first point of call for clarification. Practitioner Evaluation Support Groups Once the decision of what projects will receive funding is completed, the PCU will be able to collate projects into groups either on the basis of regional, theme clusters or cross cultural groups, which can then meet on a regular basis as the projects need to discuss issues in relation to evaluation and to enhance the networking that happens at project level. More consultation is required with individual projects to agree what the need is at this level. North Staffordshire Regeneration Zone website This website holds the latest information regarding evaluation of Regeneration projects In particular it holds a message board that projects can use to post queries and noteworthy practice for other members and relevant bodies to answer. There will also be projects own evaluation strategies for others to see and learn from, these will be evaluation done both in previous years that have been recovered and recent evaluations. The site will also hold worked examples of the completed submission and monitoring forms that are required by the PCU. Programme Co-ordination Unit newsletter This will contain latest evaluation and other related information that will assist Regeneration project leaders. This will be maintained by the Evaluation Co-ordinator. Central Evaluation Database All evaluation information will be collected at the PCU. This will include the details of evaluation strategies of individual projects, evaluation monitoring reports. It will be the responsibility of the PCU to co-ordinate the evaluation activity being undertaken by individual projects, to lesson the burden of over-surveying populations
171
and to reduce duplication of effort and resources. This information will also be used to feed-upwards to inform the evaluation of overall programmes and schemes. The database will also hold information of which projects are where throughout North Staffordshire, which will assist in identifying how programme outcomes have been achieved and to what extent. Programme Co-ordination Unit Learning Programme This will be based on the need request of issues to cover from projects. Evaluation of the service you receive from the PCU City-wide Data Facility Ginger Group advisors A set of Advisors that already exists. They are there to inspire, to pass on their knowledge and to give you a head-start if you are stuck on something. They are not there to help you with the detail or to work through the technicalities of a subject or the nitty-gritty. Projects can use the Advisors for up to 8 hours free of charge, paid for by Regeneration Zone money. Projects can use the Advisors after this 8 hours, but must pay for this themselves. How to guides The advantage of written reports and guidance is that it is always available 24 hours a day, to refer to it whenever it is needed. Guidance that is available includes, Developing Delivery plans, race equality guidance, finance & grant claim guidance, delivering strategies, project appraisal, monitoring and evaluation. Other guides available offer advice on regeneration themes, for example, Crime Reducing neighbourhood crime manual Health Guidance from the Department of Health Education Raising Standards, Opening doors Drugs Drug Prevention Advisory Service Compendium Race Equality NDC guidance Other guidance on partnerships & involving young people & the community is available in the relevant sections of this guidance. Visits to other communities involved in regeneration Visits can give people new ideas, example of what has actually worked in practice, a chance to talk to people facing similar challenges and inspiration about what is possible. Visits can be made to a wide range of initiatives, other partnerships, Housing Action Trusts, community-based projects all over the country. Once the baseline data of existing projects has been collected, the Programme Coordination Unit can give you relevant contacts for individual visits, or could potentially organise a group visit at the regional or national level.
172
Grant Finders! Stoke-on-Trent City Council and Voluntary Action Stoke have a Grant Finder service that has the capacity to sign-post initiatives to funding streams. Contact the Programme Co-ordination Unit for further information. On a National Basis, the Regeneration and Renewal magazine publish an updated Regeneration Funding Directory that contains details of all funding streams available and contact details. The Programme Co-ordination Unit will hold details of this. Governmentfunding.org.uk This is the online route to grants for the voluntary and community sector from the following funders Department for Education, Department for Health, Home Office and Office of the Deputy Prime Minister. Evaluation Support Unit The Health Action Zones Evaluation project (Boyd, Geerling, Gregory, Midgley, Murray, Walsh & Kagan 2001) has endorsed an agenda for action to support the development of an Evaluation Support Unit. This agenda has been publicised during July 2003 and will be actioned once financial support has been sought. The two main objectives of the Unit are as follows: 1) To support individuals & organisations in developing knowledge about and skills of conducting meaningful evaluations in relation to project impact, delivery and commissioning 2) To continue to develop an evaluation culture by raising awareness of the importance of evaluation and about ways in which it can contribute to processes of change, project progress and improvement There may be a possibility that support for Regeneration Partnerships in North Staffordshire may be able to benefit from this Unit. This will be investigated. Making Research Count Initiative This is an initiative that seeks to develop the links between research & practice in social work, social care and the interface with health. The addressed the growing interest in the topic of evidenced-based practice across a range of initiatives. Member agencies wishing to become involved pay an annual fee, in return for a range of services which include, individual workshops and seminars, shared regional events, free places at national conferences, regular newsletters, research briefings, access to nationally recognised experts. Economic Development & Regeneration self-assessment tool As part of the Local Government Improvement Programme, IDeA Knowledge has produced a self-assessment toolkit for Economic Development & Regeneration initiatives to comprehensively review the way in which you develop, deliver & improve your Economic Development & Regeneration function.
173
The toolkit includes, A High Level Diagnostic Matrix of the Ideal Economic Development Service, which is organised under three main themes: 1. Leadership & Strategic Planning 2. Democratic & Community Engagement 3. Performance Management Within these themes there are fifteen further sub-themes, ranging from, alliance building, change management, innovation & creativity, motivation, vision & strategy, communication (with customers and citizens), consultation & participation, customer & citizen focus, democratic representation, scrutiny, financial management, people management, planning & review, project management, & systems & processes management. Neighbourhood Management Partnerships self-assessment The Government has recently invested in the development of a new management tool called a Performance Assessment Framework, which is intended to help Regeneration schemes and Government Offices assess how well Partnerships are performing and to suggest ways in which they should plan how to do even better in the future. The process starts with self-assessment. The Regeneration Partnership must fill in questions about a number of key qualities regarding delivery and the internal structures and processes. The questions each ask for a rating and for examples of evidence. The self-assessment answers are sent to Government Office where it is compared to their own assessment of performance, eventually leading to an agreed set of priorities for improving performance. This is an opportunity that should not be missed. More information can be found on the Neighbourhood Management Pathfinder website or NRU website.
174
LEARNING OPPORTUNITIES / TRAINING COURSES There are a number of postgraduate courses in regeneration available throughout the United Kingdom, some will be more suitable that others. All courses require either full time or part time attendance at the University Site. Below are the only two distance learning courses available in the United Kingdom. Sheffield Hallam University This University offers an MSc in Local & Regional Economic Development, distance learning, fulltime for four years. The modules cover: Theory dynamics and processes of local economic development structure organisation and strategy of local economic development operational practice in local economic development independent study module dissertation. University West England This is the only University that offers a distance learning course in Urban & Rural Regeneration which has an Evaluation module and an Appraisal module. These modules can either be studied as part of a MSc / PGDip / PGCert or can be studied as accredited stand alone modules. The evaluation module covers, baselining, indicators, inputs, outputs, outcomes, additionality & displacement. The appraisal module covers, principles, roles & responsibilities, strategic context, options analysis, inputs, outputs, risk assessment, deliverability, value for money and integrated appraisal. Other training/learning opportunities RegenWM The Centre of Regeneration Excellence in the West Midlands is planning for the launch of a Graduate Placement programme aimed at attracting undergraduates & graduates into regeneration-related positions within the region. The programme is dependent on Regeneration-related organisations offering to take on volunteer(s). It is envisaged that successful candidates will be placed with relevant organisations within regeneration and will follow a structured programme with a personal mentor for support. RegenWM New databases have been launched by the RegenWM organisation, to provide to provide the most comprehensive list of training courses and organisational information yet produced for the region. The databases are accessible and easy to use, with a search engine in place to identify appropriate training courses from 300 carefully researched entries. There is an equally exhaustive directory of regeneration organisations and community groups, which could be used to share noteworthy practice and find progression routes into new careers (www.regenwm.org). A Functional Map for the Management & Delivery of Neighbourhood Renewal draft
175
A draft Functional Map has been produced by the Neighbourhood Renewal Fund during September 2003. This is the first phases of a project to ensure that there are National Occupational Standards and a qualifications framework available to meet the development needs of all those involved in Neighbourhood Renewal. The standards designed as a result of this work will offer opportunities for giving credit and recognition to people already involved in neighbourhood renewal. The standards will be designed so as to offer qualifications at a number of levels. In order to do this accurately & objectively, it is important to identify the functions that people carry out on a daily basis as part of their involvement in Neighbourhood Renewal. The Functional map is a structured way of describing the functions of neighbourhood renewal, staring from mission statement and then breaking this down into smaller functional areas, each a subset of the last. For example, from strategic management to tackling vehicle crime. The map is derived through an analysis of a range of job descriptions for national & local posts, organisation charts and key documents. The Functional Map will inform the analysis of existing National Occupational Standards which may be appropriate for use by Neighbourhood Renewal workers, to identify any gaps and finally to develop the qualifications framework. The structure of the map is as follows: Overall objectives of neighbourhood renewal and the broad functions that are carried out to achieve that objective Management functions of planning, resourcing & evaluation Structures and systems that need to be developed Supporting & developing local communities Developing learning & education Improving economic development & reducing worklessness Improving health & wellbeing Crime, safety & neighbourhood security Housing & the physical environment
The map is available for consultation during September 2003. National Centre for Sustainable Communities Skills On Monday 19 April, the Deputy Prime Minister welcomed Sir John Egan's 'Skills for Sustainable Communities' and announced he would be taking forward its key recommendation - to establish a new National Centre for Sustainable Communities Skills. The overarching aim for the National Centre will be to work with others to promote and deliver world class skills sets for everyone engaged in the sustainable communities agenda. Planning for the National Center for Sustainable Community Skills is being taken forward by a newly-formed Task, with members drawn from practitioner and policy backgrounds in the built environment, planning, heritage, regeneration and
176
neighbourhood renewal, regional and local government, private sector development, and skills. The Task Group first met on the 13 -14 May 2004 and discussed a range of options for the Center. These will be worked up in more detail over the coming months and evaluated against various criteria. An announcement about the Center is expected early next year. All other postgraduate courses RegenWM has a section on its website which signposts to current Regeneration Training courses (www.regenwm.org) A full list of postgraduate course in regeneration available throughout the UK can be found on the Regeneration and Renewal website (www.regenerationmagazine.com/courses)
177
RELEVANT CONTACTS NAME Andy Golding Eileen Lapeene ROLE SRB National Guidance ODPM University West England Distance Learning Regeneration/Evaluation course coordinator New Deal National Evaluation Sheffield Halam University University of Cambridge, Department of Land Economy: National Evaluation SRB Stockton-on-Tees Borough Council: Evaluation into the effects of the end of SRB funding the impact on the surroundings initial report end of October 03 Logical Frameworks contact CONTACT DETAILS 020 7944 4400
Paul Lawless Professor Peter Tyler Tom Holvey
p.l.lawless@shu.ac.uk 01223 337138 pt23@camb.ac.uk
Centre for International Development & Training, University of Wolverhampton Ian Butler Keele University Paula Mason NESS Sharon Richards & Danny Flynn (Chief Executive) Richard Murray (New Economics Foundation) www.audit commission.gov.uk Local Research Ethics Committee (LREC) Multi-Centre Research Ethics Committee (MREC) Consumers in NHS Research, The Help for Health Trust
01902 323219 www.wlv.ac.uk/CIDT
Making Research Count initiative Keele University National Evaluation of Sure Start (NESS) Senior Regional Evaluation Support Officer YMCA funding contact to involve young people For information of Imagine projects Co-ordinating the Working Group for Economic Regeneration Performance Indicators Established to monitor research carried out For guidance if projects are involved with several other programmes in an evaluation, if more than 5 LRECs are involved Can give advice on where to go for training
I.butler@keele.ac.uk 0781 388 7039
Sharon@ymcafunding.fsnet.co.uk Richard.murray@neweconomics.org
www.corec.org.uk
Highcroft Ronsey Road Winchester

178
NAME
ROLE
CONTACT DETAILS Hampshire SO22 5PH 01982 872242 www.conres.co.uk The Poplars Lightmoor Telford TF4 3QN 0870 770 0339 or 01952 590777 www.nif.co.uk
The Neighbourhood Initiatives Foundation
Have worked with a number of programmes
Knowledge Management Team Lisa Jones
City Council Stoke-on-Trent DreamScheme Development worker 01782 525572 or 07909634470
179
DICTIONARY OF TERMS TERM DEFINITION
180

Regeneration Project Evaluation Framework V3

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Regeneration Project Evaluation Framework V3

Uploaded by

Copyright:

Available Formats

ENSURE USE & SHARE LESSONS LEARNT

ENGAGE STAKE HOLDERS

Regeneration Project Evaluation Framework

DESCRIBE THE PROGRAM

The Evaluation Funders

Practical barriers eg transport Power differential

3 timings for evaluation:

Figure 1 The Evaluation Framework for Project Leaders

CLEAR BID WRITING

Feedback Into Project Strategy

TIME, PEOPLE, RESOURCES, , RISK ASSESSMENT

STRATEGIC ALIGNMENT (to related schemes) MAINSTREAMING / SUSTAINABILITY FORWARD STRATEGY

DISSEMINATE RESULTS CHANGING POLICY & PRACTICE

Partnership Partners GO/RDA Community Other schemes

WHOS WHO IN REGENERATION National level

Regional Coordination Unit

Housing Market Renewal / RENEW North Staffs

S-o-T LSP (Including NRF and SRB)

North Staffordshire Regeneration Zone

Programme Co-ordination Unit

Area Implementation Teams / Community Facilitation Service

Regen West Midlands (RegenWM)

NRF APPLICATION FORM.doc

Section 2. PLANNING / DELIVERING

Quality Quantity Cost Time

Achievable Challenging Deadline

Combined research efforts with other services / organisations

QUANTitative Sampling frame may not be available

Generalisation of findings can be an issue

Baseline data home after dark is twice national average

Reduce fear of crime of District level

Baseline data or nothing about NDC.

What should each monitoring system cover

Physical environment Race Involving Business Encouraging enterprise Transport

Local people say that things have improved.

Section 3. CLOSURE / FOLLOWUP

Section 4. INVOLVING OTHERS

Local residents survey

Key priorities for improving an area

Local residents survey

Local residents survey

Local residents survey

BVPI general survey 03/04 Home Office citizenship survey, main

Auditing Community Participation (Burns & Taylor 2000)

Indicator Current situation of involvement is mapped

Various elements of training for all participants include:

Examples of capacity building

Indicator Current situation of involvement is mapped

INVOLVING THE ELDERLY Add proposed indicators as previous sections. To be researched

9. Should the Partnership continue? Partner(s) Yes Continue? No

Update on baseline position (qualitative or quantitative)

Example of a Partnership Questionnaire

Partnership Evaluation Questionnaire

______________________________________________________________________ ______________________________________________________________________ ___________________________________________________________________

______________________________________________________________________ ______________________________________________________________________ ___________________________________________________________________

______________________________________________________________________ ______________________________________________________________________ ___________________________________________________________________

Paul Lawless Professor Peter Tyler Tom Holvey

p.l.lawless@shu.ac.uk 01223 337138 pt23@camb.ac.uk

01902 323219 www.wlv.ac.uk/CIDT

I.butler@keele.ac.uk 0781 388 7039

Highcroft Ronsey Road Winchester

The Neighbourhood Initiatives Foundation

Have worked with a number of programmes

Knowledge Management Team Lisa Jones

__ _

__ _

__ _