The Mitigating Hidden AI Risks Framework

• Do people/teams have the relevant skills (e.g. AI literacy), expertise and knowledge to quality assure outputs? For example, do they have sufficient subject matter knowledge to know if the AI output is accurate or high quality?
• Are there likely to be time pressures which hinder people’s ability for them to quality assure outputs?
• Could the quality of work delivered by teams be negatively impacted by people using the tool?
• Could there be pressure to reduce team size or change team expertise based on assumptions about the capability of the tool (the quality and/or efficiency of outputs)?

• If people perceive the outputs as low quality they may be less inclined to use tools
• If people dislike quality assuring outputs (e.g. it doesn’t interest or excite them), this may deter them from quality assuring or even from using AI
• If poor quality outputs go into the public domain, this could result in reputational damage for government and loss of trust in use of AI

• Test the tool to assess how effective it is at performing specific tasks compared to humans
• Let teams know what types of tasks are likely to require the most quality assurance
• Implement comprehensive AI literacy training programmes for all staff that allow staff to learn by doing and learn from others
• Provide comprehensive briefings and demonstrations for leaders, ensuring that these cover both strengths and limitations 
• Encourage hands-on experience with the tools for decision-makers
• Establish systems for monitoring tool performance and impact on work quality
• Identify areas where human expertise remains crucial
• Foster a culture of continuous learning and adaptation to emerging AI technologies

• Do people know, or do you anticipate people will know, what the purposes or goals are of your AI tool? 
• Have you appropriately informed your users about the tasks your tool is appropriate for, or do you have a communications plan in place to appropriately inform them
• Could people use the tool for purposes/goals which it wasn’t designed for and/or isn’t good at? While experimentation is a core part of being innovative, if the tool is used in ways it is not optimised for, this could create risks.
• What positive or negative consequences could arise from these uses?

• If people perceive the outputs as low quality (because the tool isn’t optimised for that use case) they may be less inclined to use tools
• If poor quality outputs go into the public domain, this could result in a loss of trust in use of AI, making it harder to roll out AI for purposes for which it is genuinely useful
• Poor use cases could limit the positive impact of AI and reduce the potential to build early successful case studies to demonstrate AI potential

• Define problems clearly before developing solutions, ensuring that any implemented tools directly address identified needs rather than being adopted for their own sake
• Make the tool better at the tasks that people are using it for
• If your tool is not good for a task people are using it for and you cannot make it better for that task (at least immediately), ensure you direct them to other tools which are available and suitable for them to use for that use case (i.e. as a substitution, so they use your tool less for that need)
• Take steps to systematically test what the tool is and isn’t good at (learning from users as much as possible)
• Let people know what tasks the tool is and isn’t good at
• Ban and monitor malicious uses 

• What emotional response might we observe from staff from rolling out the tool and what behavioural consequences could this cause? 
• Could staff perceive the tool as presenting a threat to their job, the type of work they do or skills/expertise? 
• Does the tool replace/augment tasks that people enjoy doing rather than dislike doing? Could this reduce uptake of the tool and therefore limit the positive impact AI could have?
• How might staff with a particularly positive or negative attitude towards AI respond?
• What might an AI roll out signal about the value or importance your organisation places on its staff?
• What might the roll out of the tool imply about your organisation’s priorities? (for instance, could it signal that efficiency and speed is more important than quality?)

• What expectations, attitudes or perceptions might the public form as an outcome of the government using the public-facing tool? 
  • How might it impact the public’s expectations or attitudes towards your organisation’s services? 
  • How could it impact trust in your organisation, or perceptions of your organisation’s competency?
• What emotional response might we observe from members of the public using the tool and what behavioural consequences could this cause? 

• How might wider narratives about the use of AI in other sectors (e.g. articles written by companies saying that they have built tools to replace specific jobs) influence people’s views? How might changes in political or social context shape these views?

• If people are concerned tools could replace their jobs, they may be reluctant to adopt and use them
• If the public do not have trust or confidence in the use of AI tools then this could make it harder for organisations to realise the potential benefits

• Take a human-centred approach to AI implementation. For example, conduct thorough user testing to identify and address potential negative reactions, create a bottom-up as opposed to top-down approach to tool development whereby tools are designed according to the preferences and priorities of staff (research from Wharton also suggests that staff will be the best group of people to identify the most valuable use cases), design tools to perform the tasks that staff least want to do themselves (or tasks which staff actively say they would like support with)
• Ensure that communications provides a balanced perspective of AI, drawing on AI experts to build trust in the objectivity of your communications
• Provide clear pathways for users to report issues or concerns, creating a safe environment for people to feel comfortable raising these concerns
• Provide clear information about how AI will impact roles and responsibilities
• Offer upskilling opportunities to help staff work alongside AI tools
• Clearly articulate how AI adoption aligns with broader government objectives
• Demonstrate how AI tools can improve both efficiency and quality of work
• Regularly report on the outcomes and benefits of AI use, beyond just efficiency metrics
• Anticipate, monitor and mitigate risks to reduce the likelihood that tools will cause harms that could undermine employee and/or public trust

• See also section on “Ethics”
• Develop user-friendly interfaces and clear explanations of AI tool capabilities
• Provide options for human interaction alongside AI-driven services
• Regularly gather and act on public feedback about AI-powered services
• Offer alternative service options for those uncomfortable with AI-driven solutions (not just those with accessibility needs)
• Ensure transparency about when and how AI is being used in your services
• Implement robust safeguards and communicate these to the public
• Showcase successful AI implementations and their benefits to the public
• Show how your organisation is working with industry experts to ensure AI is being used where it can have the most positive effects

• What are the barriers to AI adoption in your organisation? Do you know what these are? Do you have ways of identifying them? 
• Do you have the resources and plans in place to support staff to adopt and sustain use of AI tools?
  • Do you have plans in place to motivate people to use tools, and reassure them of any concerns they may have? 
  • Do you have plans in place to build organisational and staff capability to use AI tools?
  • What practical barriers might hinder people from using tools and how will you mitigate / remove these?
• Are there parts of your organisation that may struggle to adopt AI or specific groups of people who may struggle to adopt?
• To what extent could low or patchy adoption of AI negatively impact your organisation?
• For example, could low adoption hinder your ability to deliver efficiently relative to others that do adopt AI? 
• Will infrastructure, systems and teams be able to cope with the uptake of the tool? (for example, if adoption happens at pace)
• Are there any ways in which objectives (e.g. to drive efficiency) could be undermined, for instance, due to additional behaviours that people may have to undertake in order to embed the tool in their workflow or teams?  
  • What additional tasks might teams need to undertake in order to make best use of the tools? 
  • For example, could the tool add to the time required for people to complete tasks? (for example, if training and QA is required)
• Are there any ways in which the tool could reduce job satisfaction and motivation?
  • For example, could the tool replace easy tasks and leave people with a high volume of cognitively demanding tasks that exceeds people’s cognitive loads? Or could the tool replace tasks that people most enjoy doing?
• Do organisational leaders understand the strengths, limitations and appropriate applications of the tool?
• Could use of the tool create dependence and/or erosion of skills that might need to be retained by humans? For example, if AI tools become expensive to access or unavailable due to malicious attacks?
• How might introducing the tool reduce incentives or introduce barriers to collaboration across the organisation?
  • For example, might it reduce engagement with subject matter experts (e.g. research teams) internally or externally in ways that reduce quality of outputs (e.g. if the tools are imperfect)?

• If staff are unable to build the skills needed to maximise the impact of AI use (e.g. prompt engineering skills) then this could limit the positive impact of AI in government
• If staff feel that AI is adding to their workloads rather than reducing it, they may be reluctant to adopt and use tools
• If user feedback shows that staff report AI replaces the tasks they most enjoy doing, this could reduce job satisfaction and result in a decline in performance and productivity/efficiency 
• If leaders do not understand the strengths/limitations of the tool, this could result in AI being deployed for poor use cases that could limit the positive impact of AI and reduce the potential to build early successful case studies to demonstrate AI potential

• Give teams secure access to tools
• Provide training and support in prompt engineering
• Identify and address training needs
• Identify and address concerns pro-actively
• Measure and track so adoption challenges can be identified and tackled as appropriate
• Put in place a plan for testing ways to boost adoption and sustained use of your tools
• Develop a phased rollout plan to manage adoption pace
• Gather user feedback to identify and address inefficiencies
• Map out what steps will be required for the organisation, teams and individuals to make the most of the new tools and account for these in roll-out plans e.g. by ensuring staff are given time to undertake training
• Design tools to perform the tasks that staff least want to do themselves (or tasks which staff actively say they would like support with)
• Involve staff in tool development to ensure it enhances rather than replaces satisfying work
• Encourage hands-on experience with the tools for decision-makers
• Establish a panel of trusted experts to advise leadership on AI capabilities and limitations
• Develop contingency plans for scenarios without AI tool access 
• Ensure critical skills are documented and regularly updated

• What perverse incentives might be created through use of your tool? Specifically, an incentive which produces unintended and undesirable results, often contradicting the goals it was designed to achieve.
• How might use of the tool impact public trust in your organisation?
• How could the use of the tool reinforce or exacerbate discriminatory beliefs or outcomes or existing inequalities?
  • For instance, could the tool affect the nature or quality of work of some groups of people in society more than others?
  • In cases where biased decision making already affects humans, could the tool increase the frequency, speed or extent of these biased decisions or outcomes?
  • How easy or quickly could you identify any harms arising from the tool(s)? Could it be spotted instantly or would it only come to light after it had been in operation/use for an extended period of time?

• Discriminatory outputs could cause harm to potentially affected individuals or groups.
• Discriminatory outputs or unethically sourced inputs could cause public uproar that could make it harder to realise the benefits of AI.

• To help build public trust, demonstrate accountability and transparency by regularly publishing accessible reports on AI tool effectiveness, issues encountered and how they have/will be rectified. This includes ensuring the public is informed that these reports have been published
• To help mitigate unequal and/or discriminatory outcomes, establish systems to monitor for these outcomes and implement strategies to mitigate them. For example, ensuring there is diverse representation in AI development and decision-making teams. Another mitigation is identifying stakeholder groups who may be affected by, or may affect, the design, development and deployment of an AI tool, as a means to ensure meaningful inclusion of those who may be disproportionally at risk from the use of the tool (or its outputs) so that they can be engaged through the process of adoption and use.
• Measure equality of uptake (e.g. are people with certain protected and vulnerable characteristics adopting in lower numbers) so you can take steps to remedy discrepancies
• Identify what barriers some groups may face so that you can incorporate this into your awareness raising, strategic engagement and onboarding processes 
• Regularly compare AI-assisted decisions with human-only decisions to identify discrepancies
• Create a safe environment so people feel comfortable raising concerns or risks 
• Ensure critical skills are documented and regularly updated

• Could a technological solution to the problem the tool aims to address, undermine support for non-technological solutions that may be more effective or better accepted by end users  (for example, the public)?
• Could use of the tool result in a loss of skills/specialist expertise that could be considered particularly important or meaningful to the public or specific communities? (For example, the industrial revolution led to a decline in heritage skills such as stonemasonry and thatching which are needed to preserve the UK’s history and heritage)
• Could removing or reducing access to humans result in negative unintended consequences? 
  • For example, are there tasks, insight, expertise or interpersonal engagement that only a human can provide or fulfil?
  • If public-facing, what impact could removal of humans have on vulnerable people or those who have accessibility needs? Have you done research with these populations to explore and de-risk this?
  • How might the public feel and respond to the reliance on AI for previously human-facing tasks? See Perceptions, Emotions and Signalling risk category.
• Could use of the tool reduce incentives or create barriers to teams or people working with one another in ways that could reduce the quality of people’s experiences and/or joy experienced from human connection? 

• Poor AI use cases could undermine support for AI use in your organisation, reducing the positive impacts AI could have
• Declines in staff motivation could result in losses in productivity and performance
• Poor staff feedback about utility of tools could also undermine projects to roll out AI, making it harder to realise the potential benefits
• Overreliance on AI may reduce collaboration in your organisation 

• Identify areas where human expertise remains crucial (in the broad sense - crucial for doing the task effectively or to ensure there is trust and buy-in) 
• Keep a human in the loop where it is assessed as being crucial
• Define problems clearly before developing solutions, ensuring that any implemented tools directly address identified needs rather than being adopted for their own sake
• Engage end users or those impacted by the potential use of AI for a use case in decisions that could involve the removal/replacement of a social solution for a technological one to ensure their views and preferences are at the heart of all decisions (e.g. replacing a call centre with a chat bot or providing a therapy app instead of an in-person counselling).