Framework for
Demonstration or Bilateral Evaluations

INTRODUCTION

Governments¹ recognize the importance of evaluating programs and services supported under the Multilateral Framework for Labour Market Agreements for Persons with Disabilities to help determine impacts and outcomes.² This evaluation framework will guide governments in undertaking demonstration or bilateral evaluation activities, which, along with program and societal indicators, will help assess program and service effectiveness in improving the employment situation of persons with disabilities. The framework may also be used for jurisdictional evaluations, where appropriate. The framework identifies federal funding availability, scope, methodologies, evaluation administration, information dissemination, evaluation issues and possible questions.

FUNDING

Each province or territory may choose to engage in bilateral evaluations with the Government of Canada or undertake evaluations on its own.

To demonstrate the benefits of joint evaluations and comprehensive evaluation results, and to promote efficient joint evaluation processes, the Government of Canada will fully fund demonstration evaluations in selected jurisdictions in 2003-04 and 2004-05. In addition to the demonstration evaluations, where provinces and territories choose to undertake bilateral evaluations, the Government of Canada will contribute 80% of approved costs and provinces and territories will contribute 20%. Provinces or territories can undertake their own evaluations, which can be cost-shared up to 50% within their existing cost sharing envelope.

THE SCOPE OF EVALUATION ACTIVITIES

Demonstration and bilateral evaluations will examine how well programs and services have been designed and implemented to achieve their stated objectives. They will also assess the context and relevance of programs and services.

In addition, demonstration and bilateral evaluations will address the question of how effective programs and services are in achieving their stated objectives by assessing the degree to which the program or services was responsible for the attainment of the desired outcomes.

Demonstration evaluations are intended to be broad and comprehensive in scope. The Joint Evaluation Committees described below will determine the exact scope of demonstration and bilateral evaluations.

Evaluation issues and possible questions are outlined in Appendix 1.

EVALUATION METHODOLOGIES

Governments agree to use commonly accepted research designs and methodologies to ensure that demonstration or bilateral evaluation activities are reliable and credible, and can be used with confidence to determine program or service outcomes.

For the purposes of demonstration evaluations, governments may agree to share relevant program information, including information about individuals seeking assistance from those programs funded under this agreement. Specific information-sharing agreements will be developed between the Government of Canada and each provincial and territorial government.

The review of written reports, interviews with program stakeholders including sponsoring agencies, managers, service-delivery staff and participants, the collection of program statistics related to inputs and outputs and on-site observation are examples of research methods that could be undertaken to assess how well programs have been designed and implemented.

To determine how effective the programs and services are in achieving their objectives, research tools such as baseline intake forms, program participation forms, follow-up surveys and administrative data bases could be used.

To assist in ensuring valid, credible and unbiased conclusions, individuals performing the evaluation work must be independent from the management of the programs they evaluate.

ADMINISTRATION

Where a provincial or territorial government and the Government of Canada agree to participate in demonstration or bilateral evaluations, as described above, they will establish a Joint Evaluation Committee composed of federal representatives and representatives of that province or territory, and co-chaired by a federal representative and a provincial or territorial representative. Additional representatives should have the agreement of both co-chairs.

The Joint Evaluation Committee will be responsible for directing the development and approval of the evaluation of programs and services supported under the Multilateral Framework. The committee will also recommend third party evaluation contracts, and review and recommend evaluation reports.

To ensure timely evaluation reports, each party will take all reasonable steps to streamline evaluation processes while ensuring professional standards.

INFORMATION DISSEMINATION

Governments agree to share information on effective evaluations, lessons learned from evaluation, and effective practices for program and service delivery.

Upon agreement of both governments, demonstration and bilateral evaluation reports will be made available to the public.

Appendix 1
Evaluation Issues and Possible Questions

A. Program Relevance

• To what extent are the objectives and mandate of the Strategy/program still relevant?
• Are the activities and outputs of the program consistent with its stated objectives?
• Are the activities and outputs plausibly linked to the desired outcomes?
• What are the factors that explain any variance between the intended and actual implementation of the program?
• What is the reaction of participants to the design and delivery of the program or service?

B. Program Success

• Who has been helped by the program, and what are the profiles of the program participants?
• What were the nature of the interventions, in terms of types, costs and duration?
• To what extent do program participants achieve the desired outcomes?
• To what extent is the program responsible for the outcomes achieved by its participants?

C. What impacts and effects, both intended and unintended, have resulted from carrying out the program in the following areas (measurable examples are given within the brackets)?

• Employability and Integration (employment found and kept, increased earnings and duration of employment, and improved job readiness)?
• Economic inclusion and Well-Being (income increase and more sustainable employment)?
• Quality of Life (satisfaction and motivation about employment or services received and outlook on future labour market activities)?
• Investment in Human Capital (Skills and Literacy training received and level of competency acquired)?
• Employment Equity/Employment Barriers (results of assistive devices, attitudinal changes)?
• Self-Reliance/Independence for Government Support (the increase of earned income as a result of program intervention)?
• Labour Market Adjustment (aggregated supply side increase as a result of the program)?
• Community Impacts (improved partnership and collaboration with various stakeholders in the communities)?
• Cost-Effectiveness (participants’ earnings, employability and reliance on government transfers affected by the program compare to the associated costs of the program)?

1.  While the Quebec government subscribes to the general principles of this document, it did not participate in its elaboration. However, it does contribute by sharing information and best practices. The Quebec government intends to continue treating this question with the federal government in a bilateral way. All references to joint positions of the federal, provincial and territorial governments in this document do not include the Quebec government.

2. Given the proposed duration of the Multilateral Framework, and recognizing that the outcomes of the demonstration evaluations will be used to inform future evaluation activities and practices, governments acknowledge that not all jurisdictions will be able to evaluate the full range of funded programs and services prior to March 31, 2006. However, under the Multilateral Framework and any subsequent extension, governments commit to undertake evaluation of the range of funded programs and services, within a reasonable evaluation cycle (five to seven years).