AI Frameworks,
Guidelines,
Toolkits
ACLU, Algorithmic Equity Toolkit
​​
ACT-IAC Emerging Technology COI, White Paper: Ethical Application of AI Framework, (2020)
ACT-IAC Emerging Technology COI, AI Playbook for the US Federal Government, (2020)
​
Ada Lovelace Institute, AI Now Institute and Open Government Partnership. Algorithmic Accountability for the Public Sector, (2021)
AI4People (Atomium - European Institute for Science, Media and Democracy), Ethical Framework for a Good AI Society: Opportunities, Risks, Principles, and Recommendations
AI4People (Atomium - European Institute for Science, Media and Democracy), On Good AI Governance 14 Priority Actions, a S.M.A.R.T. Model of Governance, and a Regulatory Toolbox
AI4People (Atomium - European Institute for Science, Media and Democracy), 7 AI Global Frameworks
​
AI Global: Responsible Design Assistant
​​
AI Now Institute, Algorithmic Impact Assessments: A Practical Framework for Public Agency Accountability
​AI Now Institute, Algorithmic Accountability Policy Toolkit
​
AISP, A Toolkit for Centering Racial Equity Throughout Data Integration
​
Alan F. Winfield ; Katina Michael ; Jeremy Pitt ; Vanessa Evers: Machine Ethics: The Design and Governance of Ethical AI and Autonomous Systems, Published in: Proceedings of the IEEE ( Volume: 107 , Issue: 3)
​
Alan Turing Institute and the Information Commissioner’s Office (ICO), Project ExplAIn
​
Algofairness, BlackBox Auditing Tool
​
AlgoritmWatch, Ethics and algorithmic processes for decision making and decision support
​
Allied Media: People’s Guide to AI
​
All Tech is Human: Guide to Responsible Tech: How to Get Involved & Build a Better Tech Future
​
Amazon - AWS Conditional Demographic Disparity (CDD)
Amazon - AWS Responsible AI training modules
Amazon - AWS AI Service Cards
Amazon - AWS SageMaker Model Cards
​​
Ambacher, B. u. a. , Trustworthy Repositories Audit & Certification: Criteria and Checklist (TRAC), CRL Center for Research Libraries, Chicago, (2007)
​
Arogyaswamy, B. Big tech and societal sustainability: an ethical framework. AI & Soc (2020)
​
Australian Human Rights Commission: Using artificial intelligenceto make decisions:Addressing the problemof algorithmic bias
​
ATARC: Artificial Intelligence and Data Analytics (AIDA) Guidebook
​
B. d’Alessandro, C. O’Neil, T. LaGatta, Conscientious classification: A data scientist’s guide todiscrimination-aware classification, Big data 5 (2) (2017)
​
Barcelona City Council Open Digitisation Plan
​
Barocas, Solon and Hood, Sophie and Ziewitz, Malte, Governing Algorithms: A Provocation Piece, (2013)
​
BASIC - A Toolkit and Ethical guidelines for Applying Behavioural Insights in Public Policy, OECD, (2018)
​​
Ben Shneiderman (2020) Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy, International Journal of Human–Computer Interaction, 36:6
​
Bethge Lab: AI Foolbox
​
Berkley Haas: Mitigating Bias in Artificial Intelligence Playbook
​
Brown University, A Framework for Making Ethical Decisions
​
BSA: Confronting Bias: BSA’s Framework to Build Trust in AI
​
Brundage, Miles et al. “Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims.” ArXiv abs/2004.07213 (2020)
​
Canada Algorithmic Impact Assessment
​
Centre for Applied Data Ethics, Ethical Self-assessment Tool
​​
​Center for Democracy & Technology (CDT), DD Tool
​
Christian Sandvig, Kevin Hamilton, Karrie Karahalios and Cedric Langbort. When the Algorithm Itself Is a Racist: Diagnosing Ethical Harm in the Basic Components of Software, Int’l. J. Comm. 10: (2016)
​
Christian Sandvig, Kevin Hamilton, Karrie Karahalios and Cedric Langbort. “Auditing Algorithms: Research Methods for Detecting Discrimination on Internet Platforms.” In Data and Discrimination: Converting Critical Concerns into Productive: A preconference at the 64th Annual Meeting of the International Communication Association. Seattle, WA, (2014)
​
CIGREF, Digital Ethics
​
Council of Europe: Human Rights, Democracy, and the Rule of Law Assurance Framework for AI Systems
​
​
Dafoe, Allan. "AI Governance: A Research Agenda", 2018
​
Databaseline, ML Cards for D/MLOps Governance
​
​
DataEthics, White Paper on Data Ethics in Public Procurement of AI-based Services and Solutions, 2020
DataEthics, Impact Assessment, 2021
​
Data for Children Collaborative with UNICEF, Ethical Assessment
​
​
Data Society, Algorithmic Impact Assessment for the Public Interest
​
David Freeman Engstrom, Stanford University Daniel E. Ho, Stanford University, Catherine M. Sharkey, New York University, Mariano-Florentino Cuéllar. "Government by Algorithm: Artificial Intelligence in Federal Administrative Agencies"
​
Dawson D and Schleiger E*, Horton J, McLaughlin J, Robinson C∞, Quezada G, Scowcroft J,and Hajkowicz S†(2019) Artificial Intelligence: Australia’s Ethics Framework. Data61 CSIRO, Australia.
​
Deepcheck, Test Suites for Validating ML Models & Data
​
Dent, Kyle and Dumond, Richelle and Kuniavsky, Mike, A Framework for Systematically Applying Humanistic Ethics when Using AI as a Design Material (July 1, 2019). Temes de Disseny 35
​
​
Diakopoulos, N., “Algorithmic Accountability Reporting: On the Investigation of Black Boxes”, (2014)
​
Digital Catapult: Ethics Framework
​
doteveryone, Consequence Scanning Kit
​
D. Peters, K. Vold, D. Robinson and R. A. Calvo, "Responsible AI—Two Frameworks for Ethical Design Practice," in IEEE Transactions on Technology and Society, vol. 1, no. 1, pp. 34-47, March 2020
​
DQ Institute: DQ Framework Global Standards on Digital Literacy and Skills
​
Driven Data, Deon ethics checklist
​
​​Ethics Canvas
Expert Group Data Ethics, Ethical Codex for Data- Based Value Creation, (2019)
​
European Commission High Level Expert Group on AI, Ethics Guidelines for Trustworthy AI, 2019
European Commission High-Level Expert Group on AI, Trustworthy AI Assessment List
European Commission High-Level Expert Group on AI, Assessment List for Trustworthy Artificial Intelligence (ALTAI) for self-assessment
European Commission High-Level Expert Group on AI, Towards a European strategy on business-to-governmentdata sharing for the public interest
European Commission, White Paper on Artificial Intelligence -A European approach to excellence and trust
​
European Commission & ADAPT, EnTIRE (Mapping Normative Frameworks for EThics and Integrity of REsearch)
​
European Council Committee of Ministers, Recommendation CM/Rec(2020)1 of the Committee of Ministers to member States
on the human rights impacts of algorithmic systems, 2020
​
European Parliament, A governance framework for algorithmic accountability and transparency, (2019)
​
​​
Facets, Analyzing machine learning datasets by visualization
​
FactSheets: Increasing Trust in AI Services through Supplier’s Declarations of Conformity
​
FairLearn, Open source Python toolkit
​​
FairLens, Open source Python toolkit
​
FairSight, Fair DecisionMaking Tool
​
FBPML Wiki, Best Practices in ML
​
F. Kamiran, T. Calders, Data preprocessing techniques for classification without discrimination, Knowledge and Information Systems 33 (1) (2012)
​
Floridi, L. (2020). Ethical Foresight Analysis: What it is and Why it is Needed. Minds and Machines
Floridi, L. (2020). How to Design AI for Social Good: Seven Essential Factors
​
Fujitsu, AI Ethics Impact Assessment & Toolkit
​
GAO (Government Acountability Office) AI Accountability Framework for Federal Agencies and Other Entities
GitHub, CodeSearchNet
​
Global AI Standards Repository
​
Google, Playing with AI Fairness: What-if Tool
Google, Perspectives on Issues in AI Governance
Google, ML-fairness-gym: A Tool for Exploring Long-Term Impacts of Machine Learning Systems
​
GovEx, the City and County of San Francisco, Harvard DataSmart, and Data Community DC, Ethics and Algorithms Toolkit
​
GovLab, Re-imagining Governance in PracticeBenchmarking British Columbia’s Citizen Engagement Efforts, 2013
GovLab, The Open Policy Making Playbook
GovLab, Mapping and Comparing Responsible Data Approaches, 2016
​
Government of the Netherlands, Fundamental Rights and Algorithms Impact Assessment (FRAIA)
​
Hallensleben, Sebastian and Hustedt, Carla. From Principles to Practice: An interdisciplinary framework to operationalise AI ethics, Bertelsmann Stiftung 2020
​
Harvard University - Berkman Klein Center for Internet and Society, Ethics and Governance of AI
​
​
Humanitarian Data Science and Ethics Group (DSEG), A Framework for the Ethical Use of Advanced Data Science Methods in the Humanitarian Sector
​
Information Accountability Foundation (IAF), Comprehensive Publications About Information Accountability
​
IAPP, Building Ethics into Privacy Frameworks for Big Data and AI
​​
IBM, AI Fairness 360 Open Source Toolkit
IBM, AI Explainability 360 Toolkit
IBM, AI FactSheets 360 Toolkit
IBM, IBM Watson OpenScale
IBM, Everyday Ethics for Artificial Intelligence
IBM, Advancing AI ethics beyond complianceFrom principles to practice
IBM, Adversarial Robustness Toolbox
IBM, inFairness​
​
ICAEW: New technologies, ethics and accountability
ICO, Guidance on the AI auditing framework
ICO, Guidance on AI and data protection
​
IDEO, AI Ethics Cards
​
IEEE, Ethically Aligned Design, Version 1 (EADv1)
IEEE, Global Initiative on Ethics of Autonomous and Intelligent Systems. Ethically Aligned Design: A Vision for Prioritizing Human
Well-being with Autonomous and Intelligent Systems, Version 2 - Request for Input. IEEE, 2017.
IEEE, CertifAIEdTM certification program
​​
IFTF & Omidyar Network, Ethical OS Framework
​
Inioluwa Deborah Raji, Andrew Smart, Rebecca N. White, Margaret Mitchell,Timnit Gebru, Ben Hutchinson, Jamila Smith-Loud, Daniel Theron, and Parker Barnes. Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing. In Conference on Fairness, Accountability, and Transparency (FAT* ’20), January 27–30, 2020, Barcelona,Spain .ACM, New York
Institute for Ethical AI & ML, AI-RFX Procurement Framework
​
Institute and Faculty of Actuaries (IFoA) and the Royal Statistical Society (RSS), A Guide for Ethical Data Science
Institute for the Future of Work, Policy Briefing on algorithmic impact assessments
​​
​​interpretML
​
ISO/IEC JTC 1/SC Standardization on Artificial Intelligence
​
Japanese Society for AI (JSAI), Ethical Guidelines
​
Jessica Morley, Luciano Floridi, Libby Kinsey, Anat Elhalal. From What to How: An Overview of AI Ethics Tools, Methods and Research to Translate Principles into Practices and Supporting Material
​​
Jobin, A., Ienca, M. & Vayena, E. ,The Global Landscape of AI Ethics Guidelines, Nat Mach Intell 1, (2019)​
​
Judy Goldsmith, Emanuelle Burton, Why Teaching Ethics to AI Practitioners Is Important, AAAI Conference on Artificial IntelligenceThirty-First AAAI Conference on Artificial Intelligence, 2017
​
Kaminski, Margot E. and Malgieri, Gianclaudio, Algorithmic Impact Assessments under the GDPR: Producing Multi-layered Explanations. U of Colorado Law Legal Studies Research Paper No. 19-28. (2019)
​
Kat Zhou, Design Ethically Toolkit
​
​Kathy Baxter (Salesforce Research), How to Build Ethics into AI — Part I Research-based recommendations to keep humanity in AI
Kathy Baxter (Salesforce Research): Building Ethics into AI: Lessons Learned from Pioneers in the Trenches
​
Kiritchenko, Svetlana and Mohammad, Saif M., Equity Evaluation Corpus (EEC)
​
Koshiyama, Adriano and Kazim, Emre and Treleaven, Philip and Rai, Pete and Szpruch, Lukasz and Pavey, Giles and Ahamat, Ghazi and Leutner, Franziska and Goebel, Randy and Knight, Andrew and Adams, Janet and Hitrova, Christina and Barnett, Jeremy and Nachev, Parashkev and Barber, David and Chamorro-Premuzic, Tomas and Klemmer, Konstantin and Gregorovic, Miro and Khan, Shakeel and Lomas, Elizabeth, Towards Algorithm Auditing: A Survey on Managing Legal, Ethical and Technological Risks of AI, ML and Associated Algorithms (February 4, 2021)
​
Kriebitz, A., & Lutge, C. Artificial Intelligence and Human Rights: A Business Ethical Assessment. Business and Human Rights Journal, 5(1), 84–104. Cambridge University Press (2020)
Leidner, Jochen L. and Vassilis Plachouras. Ethical by Design: Ethics Best Practices for Natural Language Processing
​
Leighton Andrews, Bilel Benbouzid, Jeremy Brice, Lee A. Bygrave, David Demortain, Alex Griffiths, Martin Lodge, Andrea Mennicken, Karen Yeung. Algorithmic Regulation. The London School of Economics and Political Science, (2017)
​
Lepri, Bruno et al. “Fair, Transparent, and Accountable Algorithmic Decision-Making Processes.” Philosophy & Technology 31, 4(December 2018): 611–627 © 2017 Springer Science+BusinessMedia
​
Leslie, D. Understanding artificial intelligence ethics and safety: A guide for the responsible design and implementation of AI systems in the public sector, The Alan Turing Institute, (2019)
​
LinkedIn Fairness Toolkit (LiFT)
​
Linux Foundation, Apache NiFi ‹› AI Fairness 360 (AIF360) Integration – Trusted AI Architecture Development Report 1
Linux Foundation, AI Explainability 360
​
Lütge, Christoph. AI Ethics and Governance “Building a Connected, Intelligent and Ethical World”, 2020
​
Madaio, Michael A. and Jennifer Wortman Vaughan. “Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI.” (2020).
​
Markkula Center for Applied Ethics, A Framework for Ethical Decision Making
​
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. 2019. Model Cards for Model Reporting. In Proceedings of the Conference on Fairness, Accountability, and Transparency (FAT* ’19). Association for Computing Machinery, New York
​
MetaAI, Fairness Flow
​
Microsoft, AI Fairness Checklist
Microsoft, Judgment Call
Microsoft, Error Analysis
Microsoft, Responsible AI Toolbox
Microsoft, Human-AI eXperience (HAX) Toolkit
​
Ministry of Economy, Trade and Industry (METI) of Japaan: Contract Guidelines on Utilization of AI and Data
​
Moral Framework for Understanding of Fair ML through Economic Models of Equality of Opportunity
​
NetHope Solutions Center, Artificial Intelligence (AI) Suitability Toolkit for Nonprofits
​
NIST, Artificial Intelligence and User Trust (NISTIR 8332)
NIST, A Plan for Federal Engagement in Developing Technical Standards and Related Tools
NIST, AI Risk Management Framework (AI RMF)
NIST, AI Risk Management Playbook - draft
NIST, Generative AI Profile - draft
​
OECD, Framework for the Classification of AI Systems
​
Office of the Privacy Commissioner for Personal Data, Ethical Accountability Framework for Hong Kong
​
Open Data Institute, The Data Ethics Canvas
​
​​
Open Robo Ethics, AI Ethics Assessment Toolkit
​
Oracle, Skater
​​
Oxford Handbook of Ethics of AI: Online Supplement
​
PARC, AI Ethics Review
​
Partnership on AI, About ML
​
Platform for the Information Society, Artificial Intelligence Impact Assessment
​
PrepareCenter.org: Data Playbook
​
Princeton, REVISE: REvealing VIsual biaSEs
​
ProPublica, Data Store
​
pymetrics, audit AI
​
​
R. Benjamins, A. Barbado, D. Sierra, Responsible AI by design
​
Responsible Research and Innovation, Self-Reflection Tool
Responsible Innovation Compass, Self-Check
​
Rolls Royce, AIetheia Framework
​
Ruf, B., Boutharouite, C., & Detyniecki, M. Getting Fairness Right: Towards a Toolbox for Practitioners.
​
Sabou, Marta & Bontcheva, Kalina & Derczynski, Leon & Scharl, Arno. (2014). Corpus Annotation through Crowdsourcing: Towards Best Practice Guidelines. Proceedings of the 9th International Conference on Language Resources and Evaluation (LREC'14).
​
Salesforce, AI Economist, Simulation Card Foundation Economic Simulation Framework
​
​​Santa Clara University, Markkula Center, An Ethical Toolkit for Engineering/Design Practice
​
Sasha Costanza-Chock, Maya Wagoner, Berhan Taye, Caroline Rivas, Chris Schweidler, Georgia Bullen, & the T4SJ Project, 2018. #MoreThanCode: Practitioners reimagine the landscape of technology for justice and equity. Research Action Design & Open Technology Institute.
​
​
SciPy 2021 Tutorial Fairness in AI systems: From social context to practice using Fairlearn
​
Sendak, M.P., Gao, M., Brajer, N. et al. Presenting machine learning model information to clinical end users with model facts labels. npj Digit. Med. 3, 41 (2020).
​
SHAP (SHapley Additive exPlanations)
​
SHERPA (Shaping the ethical dimensions of smart information systems–a European perspective): Guidelines for the Ethical Development of AI and Big Data Systems: An Ethics by Design approach
​
Singapore, Infocomm Media Development Authority & Personal Data Protection Commission: Model Artificial Intelligence Governance Framework, 2nd Edition
Singapore, Infocomm Media Development Authority & Personal Data Protection Commission: Implementation and Self-Assessment Guide for Organizations
Singapore, Infocomm Media Development Authority & Personal Data Protection Commission: AIverify - AI Governance Testing Framework & Toolkit
​​
Smart Dubai, AI System Ethics Self-Assessment Tool
​
Stanford Center on Philanthropy and Civil Society, Digital Impact Toolkit
​
Stark, Jeannette and Nicholas Diakopoulos. “Algorithm Tips : A Resource for Algorithmic Accountability in Government.” (2017)
​
TensorFlow, Fairness Indicators
TensorFlow, Model Analyzer
TensorFlow, Responsible AI - Model Remediation
​
Thilo Hagendorff, The Ethics of AI Ethics -- An Evaluation of Guidelines
​
Tilburg University, Handbook on Non-discrimination by Design
​
Timnit Gebru, Jamie Morgenstern, Briana Vecchione, Jennifer Wortman Vaughan, Hanna Wallach, Hal Daumé III, Kate Crawford, "Datasheets for Datasets," (2018)
​
Timnit Gebru, Emily Denton, FATE/CV Tutorial (Fairness, Accountability, Transparency, Ethics / Computer Vision)
​
UC Berkeley Center for Long-Term Cybersecurity (CLTC), Decision Points AI Governance
​
UK Government: UK Data Ethics Framework
UK Government: Ethics, Transparency and Accountability Framework for Automated Decision-Making
UK Government: Artificial Intelligence and Public Standards Report
UK Government: Algorithms: How they can reduce competition and harm consumers
UK Government: Algorithmic Transparency Standard
UK Government: Al Assurance Guide & Roadmap
​
​
UNDP, UN Global Pulse, ‘A Guide to Data Innovation for Development: From Idea to Proof of Concept,’ 2016
​
​University of Chicago Center for Data Science and Public Policy, Aequitas Bias & Fairness Audit
​​
University of Washington, LIME
Urban Institute: Spatial Equity Data Tool
​
US Health & Human Services - Trustworthy AI Playbook
​
Utrecht University, Data Ethics Decision Aid (DEDA)
​
Value Sensitive Design and Information Systems
​
Vidgen, R., Hindle, G., and Randolph, I., (2020). Exploring the ethical implications of business analytics with a business ethics canvas. European Journal of Operational Research, 281(3)
​
Vollmer, Sebastian & Mateen, Bilal & Bohner, Gergo & Király, Franz & Ghani, Rayid & Jonsson, Pall & Cumbers, Sarah & Jonas, Adrian & McAllister, Katherine & Myles, Puja & Granger, David & Birse, Mark & Branson, Richard & Moons, Karel & Collins, Gary & Ioannidis, John & Holmes, Chris & Hemingway, Harry. (2018). Machine learning and AI research for Patient Benefit: 20 Critical Questions on Transparency, Replicability, Ethics and Effectiveness.
​
World Economic Forum (WEF) with Centre for the Fourth Industrial Revolution Fellows from Accenture, BBVA, IBM, Suntory Holdings, Australian Institute of Company Directors, Best Practice AI, Latham & Watkins, and Splunk, with contributions from AI4All, AI Board Toolkit
​
World Economic Forum (WEF) Unlocking Public Sector AI Toolkit: AI Procurement in a Box
World Economic Forum (WEF) AI Procurement in a Box: AI Government Procurement Guidelines
World Economic Forum (WEF) AI Procurement in a Box: AI Procurement in a Box: Workbook
World Economic Forum (WEF) Responsible Use of Technology
World Economic Forum (WEF) AI Governance - A Holistic Approach to Implement Ethics into AI
World Economic Forum (WEF) C-Suite Toolkit
​
Zuiderwijk, Anneke and Chen, Yu-Che and Salem, Fadi (2021), Implications of the Use of Artificial Intelligence in Public Governance: A Systematic Literature Review and a Research Agenda, Government Information Quarterly
10 Simple Rules for Responsible Big Data Research
​
510, F.A.C.T Score for Responsible AI
​​