Executive Summary

Advanced algorithms (i.e. artificial intelligence, or AI) are increasingly being used to support and improve the efficiency, consistency and transparency of decision-making within governments, and to enable whole new categories of public decision-making. Used in these public-sector applications, algorithms raise a host of social and ethical concerns. Although most automated decision systems are largely in the conceptual, exploratory, or early pilot phases, Government of Canada (GOC) has seized the opportunity to be pro-active in critically assessing these technologies and implementing a policy response through its Directive on Automated Decision-Making (the Directive). The Directive sets out rules for how federal departments and agencies must develop and implement algorithms to inform (or make) service decisions.

The Directive specifies a peer review mechanism intended as an additional check to ensure that any risks associated with new algorithmic decision-making tools are properly identified and mitigated. To date, GOC has not received external guidance on what would constitute best practices for conducting such peer review.

This report details the results of a collaborative study between the authors, Treasury Board Secretariat and Canadian School of Public Service, and is aimed at providing additional guidance on how to conduct robust peer review under the Directive. Ultimately, the recommendations and a practical toolkit arising from this work will enable and promote the continued responsible development and implementation of AI within GOC.

Using an evidence-based approach, the report delivers a toolkit consisting of:

·       14 Key Recommendations for conducting peer review;

·       A Model Peer Review Process;

·       A Model Evaluation Grid for use in procurement of automated decision tools, completion of the online algorithmic impact assessment (AIA), and in peer review; and

·       a Supporting Documentation Framework for use in completing AIAs and peer review processes.

The toolkit is designed to be practical and useful for:

·       Policy officials to realize and even accelerate their goals for responsible AI in GOC;

·       Algorithm developers (either external or internal) to better understand the core expectations of GOC and bring their projects into alignment with emerging standards;

·       Policy officials overseeing the AIA process from online impact scoring to peer review; and

·       Peer reviewers in their task of evaluating both the social and ethical risks of algorithms, and the various measures that were put in place during the development of the algorithm to mitigate those risks.

Moreover, the toolkit is designed to accomplish several specific goals common to analogous peer reviewing activities in other domains (e.g. Research Ethics Board reviews), including:

·       Clarifying accountability in peer review (e.g. roles and responsibilities);

·       Establishing consistent peer review practices (e.g. defining standard documents for assessment, identifying key decision points); and

·       Establishing transparency in peer review to assist in procurement (i.e. with vendors), development, and engagement with external stakeholders or affected communities.