Building confidence in clinical use of AI

AI accountability toolkits are being used to encourage trustworthiness in AI by enabling users to confront and address potential risks, such as algorithmic bias and opacity. For example, the algorithmic impact assessment (AIA) we are developing with the Ada Lovelace Institute is a type of ‘accountability toolkit’ intended to support AI developers with auditing their technology at an early stage and to ultimately increase trust in the use and governance of AI systems. Other accountability toolkits include commercial tools, such as Google’s ‘What-If’ Interface and IBM’s ‘Fairness 360’, which support users with making technical fixes to improve the interpretability of a model or to measure bias.

We are collaborating with the Wellcome Trust and Sloan Foundation to support the Oxford Internet Institute (OII) with developing the necessary evidence base and tools to assess and enhance the efficacy of AI accountability toolkits used in health and care. This project will complement our work with the Ada Lovelace Institute to trial an AIA, helping us to ensure that we have the necessary policies and standards in place to support the cultural and organisational adoption of such accountability toolkits.

The OII research team will ultimately produce a 'meta-toolkit' for trustworthy and accountable AI that comprises technical methods, best practice standards, and guidelines designed to encourage sustainable development, use, and governance of trustworthy and accountable AI systems. The meta-toolkit will help health and care practitioners, administrators, and policy-makers determine which accountability tools and practices are best suited to their particular use cases, and will ultimately be most effective at identifying and mitigating risks of AI systems at a local level.

Research published by the team as part of this project has already shown that all state of the art ‘bias preserving’ fairness methods in computer vision, used for example in medical imaging AI systems, make things fairer in practice by decreasing performance for the most disadvantaged groups. The team has recommended simple alternative best practices for improving performance without the need to ‘level down’ in the interest of fairness.

We have partnered with Health Education England to research factors influencing healthcare workers’ confidence in AI-driven technologies and how their confidence can be developed through education and training. We have published two reports in relation to this research.