Guidelines for Evaluating Community Crime Prevention Projects

Contents

STAGE 5 - SELECTING AN EVALUATOR

What you will do in Stage 5

• Prepare the evaluation brief
• Decide on internal and external evaluators
• Identify the skills the Evaluator must have
• Select evaluators
• Draft a contract between the project and the Evaluator, if the evaluator selected is external to the project

DEVELOPING THE EVALUATION BRIEF

The Evaluation Brief sets out what the project needs from the evaluation. External Evaluators preparing tenders will use it to develop realistic proposals. Internal Evaluators will use it to plan their work. The evaluation brief summarises:

• the crimes the project aims to prevent or reduce
• the activities of the project
• how these activities are meant to prevent or reduce crime
• the purpose of the evaluation
• the type of evaluation required
• the timeframe for the evaluation
• the main questions the evaluation needs to answer, and the information required by stakeholders
• who will use the information generated in the evaluation, and what they will use it for
• how the project needs the Evaluator to report the findings
• the financial limitations on the evaluation, if appropriate.

You have collected most of this information in Stages 2-4.

INFORMATION ABOUT THE PROJECT

1. Make copies of:

• PLANNING SHEET 2A
  This lists the stakeholders and their involvement in the project.
• PLANNING SHEET 2C
  This lists what information stakeholders need and want and how they will use it.
• PLANNING SHEET 3
  This summarises the problems the project was set up to prevent or reduce, and what the project does to achieve this.
• PLANNING SHEET 4B
  This shows the timetable for the evaluation.

2. Make a copy of PLANNING SHEET 5A and write on it the reasons for evaluating the project.

3. Make a copy of PLANNING SHEET 5B. Write on it any ways information needs to be gathered that may affect how the evaluation is done.

For instance, write down whether the Evaluator needs to:

• observe people taking part in the project
• monitor criminal activity
• review specific documents or information
• access information the project does not have
• interview community members, project staff or members of the stakeholder groups
• involve people with special needs.

While this needs to be done, care must be taken to not overly constrain the evaluation consultant.

4. On PLANNING SHEET 5B note any ethical issues the Evaluator must consider when planning and conducting the evaluation.

For instance, note whether the Evaluator must gain the informed consent of all people contributing to the evaluation. This may be difficult in the case of people with special needs, such as people:

• with poor literacy
• with intellectual disabilities
• experiencing mental illness
• affected by age
• affected by trauma
• under 18 years of age, especially young children.

Note whether the Evaluator needs access to confidential information, and how they are to keep it secure. The Evaluator will also need to consider the ethics of using confidential information for a purpose other than that for which it was first provided.

Note whether the Evaluator is expected to forward confidential information to authorities where the safety or health of people overrides the confidentiality in which the information was given.

Note whether the evaluation plans need to be submitted to an appropriate 'ethics committee'.

5. On PLANNING SHEET 5B, detail the budget the Evaluator will have to evaluate the project.

If the project is going to hire an external Evaluator, you may not want to tell applicants the funds available. Consider setting an upper limit on funds, as this will ensure all proposals are within the funds available. If you do not wish to give potential Evaluators a figure, write 'Evaluator to supply'.

6. Make a copy of PLANNING SHEET 5C once you have decided if the project needs the Evaluator to provide other updates, briefings or reports during the evaluation or before the final report. For each, write down:

• the date it is to be delivered
• what is to be covered in the report, briefing or presentation
• who will use this information
• how they will use this information
• what format the information should be in.

You will be able to copy some of this from PLANNING SHEET 4A.

It is a good idea to see a draft of the final report so you can refine the Action Plan and ensure it will be suitable for all stakeholders.

7. Draft a cover letter:

• introducing the project
• explaining the reasons it needs to be evaluated
• listing all of the planning sheets you prepared and summarising what each of them contains.

These will be PLANNING SHEETS 2A, 2C, 3, 4B, 5A, 5B and 5C. Together, this letter and these planning sheets are the Evaluation Brief.

DECIDING ON INTERNAL AND EXTERNAL EVALUATORS

8. List all the reasons for and against using an Evaluator.

Reasons may include:

• finances-does the project have funds to hire a consultant?
• internal resources-does the project already have evaluation skills?
• independence-does the evaluation need to be independent?
• staffing-could staff be taken off-line to do an evaluation?

9. Discuss the reasons for and against each type of Evaluator with the Project Committee.

10. Ask the Project Committee to decide whether to use internal or external Evaluators or a combination.

IDENTIFYING THE SKILLS NEEDED

11. Make a copy of PLANNING SHEET 5D and write down the skills an Evaluator will need.

These are the skills, training and experience you will assess potential Evaluators against.

Qualities you will be looking for in an Evaluator include:

• training appropriate to the needs of the project
• relevant qualifications
• planning and project management skills
• negotiating skills
• experience designing evaluations and evaluation tools, such as survey forms or interview protocols
• experience interviewing people or conducting surveys
• experience evaluating similar projects
• experience analysing the type of data to be gathered
• demonstrated ability to make practical recommendations for action
• knowledge of the relevant ethical and professional standards
• any special requirements-for instance, whether they will need to work with town planning or development committees, very young children or people with disabilities
• evaluation knowledge and competencies as well as substantive area knowledge.

Note if they will need to undergo a security or police check before they can evaluate the project or have access to confidential information.

List any professional or quality standards the Evaluator should observe or work to, such as:

• the Programme Evaluation Standards developed by the Joint Committee on Standards for Educational Evaluation
• the Australasian Evaluation Society's Ethical Guidelines
• Australian Standards for service organisations, AS 3901 and AS 3902, which provide assurance that Evaluators operate according to accepted management standards. (There is no Australian Standard for project evaluation.)

Consultants tendering for an evaluation study could be expected to state that they would conduct their work according to at least one set of recognised standards.

SELECTING AN INTERNAL EVALUATOR

12. Follow your organisation's usual internal recruitment procedures to advertise the position and get applications.

13. Use PLANNING SHEET 5E to draft a timetable for selecting an Evaluator.

This timetable needs to include the dates that:

• advertisements will be published
• applications will close
• you will make an initial assessment of applications
• you will draft the interview questions
• you will send the selection panel interview questions
• the selection panel will interview applicants
• the selection panel will choose an Evaluator.

14. Check the timetable with the selection panel. Make any changes needed.

15. Follow your organisation's selection procedure to interview and select an applicant.

16. Make a copy of PLANNING SHEET 5F and write on it:

• the name of the person or organisation selected to evaluate the project
• the names of the people who applied
• the reasons the successful applicant was selected.

Attach a copy of:

• the advertisement
• each person's application
• the selection criteria
• interview questions
• interview reports for each applicant, if you prepared them.

TENDERING FOR AN EXTERNAL CONSULTANT

Most organisations that host community crime prevention projects are large enough to have an established procedure for tendering contractors. If the project is hosted by a small organisation, you may be able to take a much more informal process than outlined here.

Keep a record of each interview. Even if you are not required to make a record, it is often a good idea, in case you need to defend your choice later. Unsuccessful applicants may also contact you after the interview for an interview report.

Normally, tendering for an Evaluator will involve:

• setting up any administration needed in the tendering process
• preparing tender documents
• advertising a request for tender
• sending tender documents to potential tenderers
• receiving applications from tenderers
• assessing the applications against selection criteria
• interviewing tenderers
• selecting a tenderer to evaluate the project
• negotiating a contract with them.

17. Use PLANNING SHEET 5E to draft a timetable for selecting an Evaluator.

This timetable needs to include the dates that:

• you will finalise the advertisement and tender documents
• the project will approve the tender documents and advertisement
• the request for tenders will be advertised
• applications will close (allow at least three weeks for applicants to prepare proposals, plus three business days for postal deliveries)
• you will make an initial assessment of applications
• you will draft interview questions
• you will send the selection panel the interview questions
• the selection panel will interview tenderers
• the Evaluator will be selected
• contract negotiations will be completed.

18. Follow your organisation's selection procedure to interview and select an applicant.

19. Make a copy of PLANNING SHEET 5G and fill out:

• the name of the person or organisation selected to do the evaluation
• the names of the people and organisations that applied
• the reasons the successful applicant was selected.

Attach a copy of:

• the advertisement
• each application
• the selection criteria
• interview questions
• interview reports for each applicant, if you prepared them.

DRAFTING THE CONTRACT

You only need to draft a contract if an external contractor will evaluate the crime prevention project.

20. Ask your Human Resources or Procurements section for a copy of your organisation's standard contract form.

Most organisations that host community crime prevention projects are large enough to have a standard contract form for similar services.

21. Use the information in the evaluation brief to draft a contract between the project and the Evaluator.

A contract with an Evaluator will normally contain:

• the purposes of the evaluation
• outcomes expected from the evaluation
• reporting requirements, including methods and dates
• responsibilities of the Evaluator, including responsibilities after they have submitted their final report
• ethical obligations or standards they must observe
• responsibilities of the project, such as providing access to staff, stakeholders and information
• confidentiality considerations
• intellectual property ownership, particularly ownership of data, reports and presentations developed by the Evaluator
• conditions under which existing information may be used
• budget
• terms of payment
• a procedure for altering the contract
• a procedure for resolving any differences that arise between the project and the Evaluator.