Preventing Repeat Residential Burglary:

A meta-evaluation of two Australian demonstration projects

Contents | Acknowledgements | Executive Summary | Chapter 1: About repeat victimisation | Chapter 2: The demonstration projects | Chapter 3: Meta-Evaluation | Chapter 4: Programme guide and resource tools | Appendix 1: Summary of research study outcomes | Appendix 2: Process evatuation framework | References

APPENDIX 2

Process evaluation framework

The following framework provides issues, questions and potential indicators to assist in collecting information relevant to process evaluation objectives such as:

• identifying ways of enhancing the programme to achieve improved outcomes
  
• explaining outcomes from the impact evaluation (for example, whether failure to achieve the predicted effect is because the programme was not implemented as intended, rather than because the intervention is not an effective crime prevention technique)
  
• contributing to conclusions about the continuing appropriateness and relevance of the programme (for example, with the benefit of hindsight, are the objectives as defined the most appropriate ones, are there wider crime prevention outcomes or community benefits than those assessed by the outcome evaluation?)
  
• identifying issues important for replicating or expanding the programme, such as strategies for overcoming implementation problems or unique features of the programme location critical for success that might not be routinely available in other locations.

Consistency of implementation with original objectives

Consider:



Has the project infrastructure been implemented as intended?

Categories include:


• project staff and resources


• roles and responsibilities of officers and agencies involved


• training and skills development


• information systems


• interagency liaison processes


• project management


• performance standards/monitoring systems


• policy and procedures development, etc


• project materials

If not:


• What is different?


• Why?


• What are the implications?


• Is the reason for the change a response to an implementation problem that needed to be addressed? (If yes, how effective was the alternative in fully resolving the issue of concern?)



Potential indicators:

• project staff and resources in place at expected time/stage of project (to take into account the effects of any delays on implementation and outcomes)


• actual roles and responsibilities as evolved during the project align with roles and responsibilities established at project design stage


• whether interagency liaison processes critical to support the project's objectives have operated effectively throughout the project


• any changes to project management or coordination structures occurring during the time course of the project (either to address implementation difficulties or to further enhance project objectives).



Consider:



Have service delivery processes been implemented as intended?

Key issues include:


• whether the processes to identify target households have operated as intended


• whether each component of intervention procedures (for example security audit/assessment, information/advice, contact with neighbours, etc) has been implemented as intended


• whether the intervention components targeted specifically at repeat victimisation have been implemented as designed in the original project specifications


• whether predetermined timelines have been met


• whether predetermined service standards have been met.



Potential indicators:

• proportion of cases of reported incidents in the intervention area visited by programme staff/volunteers as the first stage of the programme; percentage of these occurring within the timeframe established by the project


• proportion of cases where increased police patrol activity occurred; percentage of these occurring within x-hours/days of identifying vulnerable area/household (if increased police activity is a core component)


• proportion of relevant households where security equipment provided to the household was fitted; percentage of these where the response occurred within the timeframe established by the project


• proportion of participating households receiving different components or different combinations of interventions (as an indicator of the extent to which a problem-oriented solution versus a standardised response was applied)


• audit of volunteers/police personnel activity by project staff


• review of any complaints by victims.



Consider:



Has the intended target population been reached?

Key issues include:


• whether all households falling within the scope of the project intervention had the same opportunity to participate


• whether all households participating in the project had the same opportunity to receive the intervention appropriate to that target location in accordance with the project's intentions (for example, was every household judged as benefiting from the loan of a portable alarm given the opportunity to receive one; or was the number of alarm units limited relative to assessed need?)


• whether all households agreeing to par ticipa te h ad equa l opportunity to be included in the outcome measurement process


• whether there are any characteristics unique to participating/non-participating households that may affect outcome results in the impact evaluation


• whether there was any heightened vulnerability among refusals or non-contacts or other non-participants that warrants greater attention to encouraging such households to participate when implementing similar projects in the future.



Potential indicators:

• proportion of reported incidents in the intervention area(s) contacted by the project for participation consent (to assess the extent to which the opportunity for intervention was available to the whole target population)


• refusal rate i.e. proportion of victim households approached that did not agreed to participate at initial project contact stage


• loss rate per stage i.e. proportion of households initially agreeing to participate that was lost at each relevant project follow-up stage for whatever reason, for example non-contact, later refusal, etc


• comparison of available information on households in the study with non-participating households coming within the scope of the project (because of refusal, non-contact, missed referral/notifications, etc)


• overall loss rate, i.e. number of households where full project intervention and follow-up occurred as a proportion of all victimised households in the project area reporting an incident to police during the project time period


• number of households receiving specific interventions provided directly by the project (for example a loaned alarm, locks fitted) as a proportion of households assessed as warranting such intervention in the security audit/assessment).



Consider:



How has the target population responded to the intervention?

Key issues include:


• response to security advice by victims


• barriers to acting upon security advice identified by victims


• willingness to report incidents to police by victims


• satisfaction with processes/advice.



Potential indicators:

• take up rate of security advice by households assessed at follow-up interview


• content analysis of issues identified by victims as encouraging changes in their security practices at follow-up interview


• proportion of unreported offences identified at follow-up survey compared to non-reporting rates identified at initial interview


• reported satisfaction by victims.

Return to the top



Other issues potentially affecting evaluation scope

Consider:

Have measurement issues limited the capacity of the evaluation to meet its objectives?

Possible issues include:


• definition of repeat victimisation


• limited capacity to assess the relative contribution of the individual crime prevention components if a multi-component problem-solving approach is adopted


• small number of repeat victimisations limiting impact evaluation analysis


• limited project timeframe precluding assessment of long-term outcome effects


• format of police information systems limiting availability of relevant indicators, especially in control areas


• difficulty in discriminating attempted and completed incidents from available police statistics


• no information on extent of non-reporting outside the survey follow-up period


• extraneous variables in intervention or control areas, (for example, the introduction of new crime prevention programme in control sites, police special operations against property offending in the programme area).



Potential indicators:

• project staff identification of nature and extent of the effects of relevant issues, based on project experience


• stakeholder identification of extraneous issues.

Continuing approp riateness of the objectives as originally defined

Consider:

• Are the project rationale and objectives as originally defined still considered appropriate after experience with the project's actual operation?


• Are the project goals and objectives internally consistent?


• Are they clearly defined and communicated (i.e. in a format that can be readily understood and adopted in wider application or replication)?


• Has the relevance of project procedures to programme goals and objectives been established to the satisfaction of key stakeholders?


• Are there any inconsistencies between the programme objectives and those of the wider organisation or system within which they operate that need to be addressed if extending or replicating the project?



Potential indicators:

• project staff judgements about ongoing appropriateness of objectives supported by project experience


• stakeholder assessment of relevance of objectives to current circumstances.

Critical implementation issues

Consider:

1. What were the critical implementation issues to overcome?


Key issues may include:


• information systems' capacity to identify repeat victimisations


• notification systems for identifying target households


• volunteer safety concerns


• audit of police/volunteer practices and quality of service


• fostering wider agency cooperation/involvement


• programme ownership (centralised control versus local control)


• relationship to other crime prevention initiatives/structures.
2. What strategies were adopted to overcome critical implementation issues and how effective were they in resolving the issue?



3. Are there any other critical success factors, regardless of whether they were associated with any difficulties during the pilot programme implementation?



4. What aspects could be improved upon in future projects of this kind?



Potential indicators:

• critical success factors agreed by project staff


• implementation issues consistently identified by key stakeholders.

Return to the top

Special features not necessarily available in wider application

Consider:

What features have promoted successful outcomes that may not be routinely available in implementing the intervention in other locations or in expanding the intervention from a pilot project?

Key issues may include:


• availability of dedicated project staff for coordination, support, or evaluation


• establishment or special project funds


• extent of wider organisational support, for example, the extent of structured support for volunteers


• interest and cooperation of key third parties because it is a pilot/new programme that may not be sustained


• publicity or awareness raising because of trial nature of programme (for example, possible associated deterrence effects for offenders operating in the area).



Potential indicators:

• features identified and agreed by project staff.

Wider consequences/outcomes

Consider:

What wider outcomes and consequences (positive or negative) should be considered in evaluating the intervention beyond changes in victimisation rates?

These may include:


• community capacity-building with a volunteer model


• greater community satisfaction with enhanced service delivery models


• opportunity for police to apply wider crime prevention approach rather than limited to traditional reactive police practices, and associated professional development and morale effects on individual officers involved, as well as diffusion benefits to the overall organisation


• expectation of ongoing service by victims


• fear of the re sul t s of crime.



Potent i al indicators:

• content analysis of stakeholder responses to structured interview question asking about wider project benefits and unintended consequences


• outcomes of project staff meetings/workshops relevant to this issue.

Cost effectiveness

Consider:

• Is the approach considered to be a cost-effective one and why?


• Could the project be replicated more cost effectively, and if so, how?


• What opportunities exist for resource sharing that were not considered or taken up?



Potential indicators:

• project cost per intervention relative to the cost of traditional policing/crime prevention models for break and enter


• estimated savings associated with prevented break and enters over the project's duration.

Return to the top