In October 2023, Daron Acemoglu joined GovAI’s Economics of AI Lead – Anton Korinek – for a discussion on Daron’s book: Power and Progress: Our 1000-Year Struggle Over Technology & Prosperity. The Webinar was introduced by Ben Garfinkel, and facilitated by Emma Bluemke.
This summarises a recent multi-author white paper on frontier AI regulation. We are organizing a webinar about the paper on July 20th. Sign up here.
GovAI research blog posts represent the views of their authors, rather than the views of the organisation.
AI models are already having large social impacts, both positive and negative. These impacts will only grow as models become more capable and more deeply integrated into society.
Governments have their work cut out for them in steering these impacts for the better. They have a number of challenges they need to address, including AI being used for critical decision-making, without assurance that its judgments will be fair and accurate; AI being integrated into safety-critical domains, with accompanying accident risks; and AI being used to produce and spread disinformation.
In a recent white paper, we focus on one such challenge: the increasingly broad and significant capabilities of frontier AI models. We define “frontier AI models” as highly capable foundation models,1 which could have dangerous capabilities that are sufficient to severely threaten public safety and global security. Examples of capabilities that would meet this standard include designing chemical weapons, exploiting vulnerabilities in safety-critical software systems, synthesising persuasive disinformation at scale, or evading human control.
We think the next generation of foundation models – in particular, those trained using substantially greater computational resources than any model trained to date – could have these kinds of dangerous capabilities. Although the probability that next-generation models will have these capabilities is ambiguous, we think it is high enough to warrant targeted regulation. The appropriate regulatory regime may even include licensing requirements.
Effective frontier AI regulation would require that developers put substantial effort into understanding the risks their systems might pose, in particular by evaluating whether they have dangerous capabilities or are insufficiently controllable. These risk assessments would receive thorough external scrutiny and inform decisions about how and whether new models are deployed. After deployment, the extent to which the models are causing harm would need to be continually evaluated. Other requirements, such as high cybersecurity standards, would likely also be appropriate. Overall, regulatory requirements need to evolve over time.
Based on current trends, creating frontier AI models is likely to cost upwards of hundreds of millions of dollars in compute and also require other scarce resources like relevant talent. The described regulatory approach would therefore likely only target the handful of well-resourced companies developing these models, while posing few or no burdens on other developers.
There are three core challenges for regulating frontier AI models:
Self-regulation is unlikely to provide sufficient protection against the risks of frontier AI models: we think government intervention will be needed. The white paper explores the building blocks such regulation would need. These include:
The white paper also suggests some preliminary, minimum safety standards for frontier AI development and release:
Other standards would also likely be appropriate. For example, frontier AI developers could be required to uphold high cybersecurity standards to ward off attempts at theft. In addition, these standards should likely change substantially over time as we learn more about the risks from the most capable AI systems and the means of mitigating those risks.
While we feel confident that there is a need for frontier AI regulation, we are unsure about many aspects of how an appropriate regulatory regime should be designed. Relevant open questions include:
We are planning to spend a significant amount of time and effort exploring these questions. We strongly encourage others to do the same. Figuring out the answers to these questions will be extremely difficult, but deserves all of our best efforts.
The white paper was written with co-authors from a number of institutions, including authors employed by industry actors that are actively developing state-of-the-art foundation models (Google DeepMind, OpenAI, and Microsoft). Although authors based in labs can often contribute special expertise, their involvement also naturally raises concerns that the content of the white paper will be biased toward the interests of companies. This suspicion is healthy. We hope that readers will be motivated to closely engage with the paper’s arguments, take little for granted, and publicly raise disagreements and alternative ideas.
The authors of this blog post can be contacted at
ma***************@go********.ai
,
jo***********@go********.ai
, and
ro***********@go********.ai
.
Research Assistants help members of the GovAI research community with their ongoing research projects. These roles provide an opportunity to contribute to an AI governance research project, learn abou
Source link
This post explores possible outcomes of the upcoming UK-hosted global summit on AI safety. It draws on ideas raised in a recent expert workshop hosted by GovAI, but is not intended as a consensus statement.
GovAI research blog posts represent the views of their authors, rather than the views of the organisation.
Following growing concern about possible extreme risks from AI, the United Kingdom recently announced plans to host the first major global summit on AI safety. The announcement sketched a high-level vision for the summit:
The summit, which will be hosted in the UK this autumn, will consider the risks of AI, including frontier systems, and discuss how they can be mitigated through internationally coordinated action. It will also provide a platform for countries to work together on further developing a shared approach to mitigate these risks.
However, more detailed aims have not yet been shared. GovAI recently convened a small expert workshop – organised without government involvement and held under the Chatham House rule – to explore possible objectives for the summit.
Attendees suggested a wide variety of valuable direct outcomes of the summit. These can could mainly be grouped into six categories:
Several attendees suggested that the summit may be a unique and fleeting opportunity to ensure that global AI governance includes China. China is likely to be excluded from important discussion venues, such as the OECD and G7. China may also have only a temporary period of AI policy “plasticity” and will be more inclined to reject any global governance principles that Western states begin crafting without its input. However, one attendee disagreed that China should be invited, emphasising that Chinese participation could make the summit less productive.
Potential valuable outcomes from the summit include:
A number of participants suggested that this summit may be the only opportunity to include China in productive global AI governance discussions. Other important venues – including the OECD, the G7, or meetings initiated by the US – will likely exclude China by default. More inclusive institutions such as the UN may also struggle to make forward progress due to structural constraints.
This opportunity may be critical, since global AI governance will likely fail if it does not ultimately include China. If China – the world’s third major AI power – does not adopt responsible AI safety policies, then this could threaten the security of all other states. External constraints, such as export controls on advanced chips and relevant manufacturing equipment, will probably only delay China’s ability to create high-risk models.
Summit outcomes (such as those listed above) could therefore be more valuable if they include China. One participant whose research focuses on Chinese AI policy argued that China can be positively influenced by the summit.1 There is interest in AI safety in China – for instance, a number of prominent Chinese academics signed the recent CAIS statement on AI risk – and Chinese actors have been signalling an interest in participating in global AI governance efforts.
Trying to involve China later – for instance, in future iterations of the summit – may also be much less impactful than involving China now. China is more likely to adopt global frameworks if it can legitimately view itself as a co-creator of these frameworks; this will be less likely if China is not invited from the start. As China develops its own regulatory frameworks for advanced AI and makes its own independent policy statements, Chinese AI policy may also quickly lose some of the “plasticity” it has now
However, within the group, there was not universal consensus that China should be invited to the summit. One potential cost of inviting China is that it may reduce the enthusiasm of other states and non-governmental actors to participate in the summit. Inviting China may also make the summit less productive by increasing the level of disagreement and potential for discord among participants. There may also be some important discussion topics that would not be as freely explored with Chinese representatives in the room.
These concerns are partly mitigated by the likely existence of side meetings – and discussions in venues beyond the summit – which could include smaller subsets of participants. However, involving China in the summit would inevitably involve trade-offs. These trade-offs could be evaluated in part by seeking the views of other likely participants.
The first global summit on AI safety is an important opportunity to make progress toward managing global risks from advanced AI systems. It could directly produce several outcomes, including: (1) shared commitments and consensus statements from states, (2) the seeds of new international institutions, (3) the advertisement and diffusion of UK AI policy initiatives, (4) shared commitments from labs, (5) increased understanding of AI risks and governance options, and (6) commitments to further productive discussions.
It may also be a unique and fleeting opportunity to involve China in global AI governance. Ultimately, however, making the correct decision regarding China’s participation will require weighing the value of Chinese participation with the frictions and stakeholder management challenges issues it may bring.
The authors of this piece can be contacted at
be***********@go*******.ai
and
le**********@go********.ai
.
This post summarises a new report, “International Governance of Civilian AI: A Jurisdictional Certification Approach.” You can read the full report here.
GovAI research blog posts represent the views of their authors, rather than the views of the organisation.
Many jurisdictions have begun to develop their own approaches to regulating AI, in response to concerns that range from bias to misinformation to the possibility of existential risks from advanced AI.
However, without international coordination, these purely national and regional approaches will not be enough. Many of the risks posed by AI are inherently international, since they can produce harms that spill across borders. AI supply chains and product networks also span many countries. We therefore believe that a unified international response to the risks posed by AI will be necessary.
In recognition of this need, we have developed a proposal – outlined in a new report – for an international governance body that can certify compliance with international standards on civilian AI.1 We call this proposed body the International AI Organization (IAIO).
Our proposal for the IAIO follows a jurisdictional certification approach, modelled on the approach taken by other international bodies such as the International Civilian Aviation Organization (ICAO), the International Maritime Organization (IMO) and the Financial Action Task Force (FATF). Under this approach, the international body certifies that the regulatory regimes adopted within individual jurisdictions meet international standards. Jurisdictions that fail to receive certifications (e.g. because their regulations are too lax or they fail to enforce them) are excluded from valuable trade relationships – or otherwise suffer negative consequences.
Our report outlines the proposal in more detail and explains how it could be put into practice. We suggest that even a very partial international consensus on minimum regulatory standards – perhaps beginning with just a few major players – could be channeled into an institutional framework designed to produce increasingly widespread compliance. An initial set of participating states could establish the IAIO and – through the IAIO – arrive at a set of shared standards and a process for certifying that a state’s regulatory regime meets these standards. One of the standards would be a commitment to ban the import of goods that integrate AI systems from uncertified jurisdictions. Another standard could be a commitment to ban the export of AI inputs (such as specialised chips) to uncertified jurisdictions. The participating states’ trade policies would thereby incentivise other states to join the IAIO themselves and receive certifications.
We believe that the IAIO could help to mitigate many of AI’s potential harms, from algorithmic bias to the longer-term security threats. Our hope is that it can balance the need to prevent harmful forms of proliferation against the imperatives to spread the benefits of the technology and give voice to affected communities around the globe.
This post summarises a recent paper, “Protecting Society from AI Misuse: When are Restrictions on Capabilities Warranted?”
GovAI research blog posts represent the views of their authors, rather than the views of the organisation.
Recent advancements in AI have enabled impressive new capabilities, as well as new avenues to cause harm. Though experts have long warned about risks from the misuse of AI, the development of increasingly powerful systems has made the potential for such harms more widely felt. For instance, in the United States, Senator Blumenthal recently began a Senate hearing by showing how existing AI systems can be used to impersonate him. Last week, the AI company Anthropic reported that they have begun evaluating whether their systems could be used to design biological weapons.
To address growing risks from misuse, we will likely need policies and best practices that shape what AI systems are developed, who can access them, and how they can be used. We call these “capabilities interventions.”
Unfortunately, capabilities interventions almost always have the unintended effect of hindering some beneficial uses. Nonetheless, we still argue that — at least for certain particularly high-risk systems — these sorts of interventions will be increasingly warranted as potential harms from AI systems increase in severity.
Some existing AI systems – including large language models (LLMs), image and audio generation systems, and drug discovery systems – already have clear potential for misuse. For instance, LLMs can speed up and scale up spear phishing cyber attacks to target individuals more effectively. Image and audio generation models can create harmful content like revenge porn or misleading “deepfakes” of politicians. Drug discovery models can be used to design dangerous novel toxins: for example, researchers used one of these models to discover over 40,000 potential toxins, with some predicted to be deadlier than the nerve agent VX.
As AI systems grow more capable across a wide range of tasks, we should expect some of them to become more capable of performing harmful tasks as well. Although we cannot predict precisely what capabilities will emerge, experts have raised a number of concerning near- and medium-term possibilities. For instance, LLMs might be used to develop sophisticated malware. Video-generation systems could allow users to generate deepfake videos of politicians that are practically indistinguishable from reality. AI systems intended for scientific research could be used to design biological weapons with more destructive potential than known pathogens. Other novel risks will likely emerge as well. For instance, a recent paper highlights the potential for extreme risks from general-purpose AI systems capable of deception and manipulation.
Developers and governments will need to adopt practices and policies (“interventions”) that reduce risks from misuse. At the same time, they will also need to avoid interfering too much with beneficial uses of AI.
To help decision-makers navigate this dilemma, we propose a conceptual framework we refer to as the “Misuse Chain”. The Misuse Chain breaks the misuse of AI down into three stages, which can be targeted by different interventions.
Capabilities interventions target the first stage of the Misuse Chain, in which malicious actors gain access to the capabilities needed to cause harm. For instance, if an AI company wishes to make it harder for bad actors to create non-consensual pornographic deepfakes, it can introduce safeguards that block its image generation systems from generating sexual imagery. It can also take additional measures to prevent users from removing these safeguards, for instance by deciding not to open-source the systems (i.e. make them freely available for others to download and modify).
Harm mitigation interventions target the next stage in the Misuse Chain, when capabilities are misused and produce harm. For example, in the case of pornographic deepfakes, harm can be mitigated by identifying the images and preventing them from being shared on social media.
Harm response interventions target the final stage of the Misuse Chain, in which bad actors experience consequences of their actions and victims receive remedies. For example, people caught using AI systems to generate explicit images of minors could not only have their access revoked but also face punishment through the legal system. Here, the aim is to deter future attempts at misuse or rectify the harm it has already caused.
Interventions that limit risks from misuse can also hinder beneficial uses; they are subject to what we call the “Misuse-Use Tradeoff”. As interventions move “upstream” along the Misuse Chain — and more directly affect the core capabilities that enable malicious actors to cause harm — they tend to become less targeted and exhibit more significant trade-offs. Interventions near the top of the chain can limit entire classes of use.
Take, for example, content policies for large language models that prohibit the model from outputting any text related to violence. Such a blunt policy tool may prevent harmful content from being generated, but it also prevents the model from being used to analyse or generate fictional works that reference violence. This is an undesirable side effect.
However, if an AI system can be used to cause severe harm, then the benefits of even fairly blunt capability interventions may outweigh the costs.
Where possible, developers and policymakers should prefer harm mitigation and harm response interventions. These interventions are often more targeted, meaning that they are more effective at reducing misuse without substantially limiting beneficial use.
However, these sorts of interventions are not always sufficient. There are some situations where capabilities interventions will be warranted, even if other interventions are already in place. In broad terms, we believe that capabilities interventions are most warranted when some or all of the following conditions hold:
Ultimately, as risks from the misuse of AI continue to grow, the overall case for adopting some capability interventions will likely become stronger. Progress in developing structured access techniques, which limit capability access in finely targeted ways, could further strengthen the case.
In practice, to reduce misuse risk, there are several varieties of capability interventions that developers and policymakers could pursue. Here, we give three examples.
As AI systems become more advanced, the potential for misuse will grow. Balancing trade-offs between reducing misuse risks and enabling beneficial uses will be crucial. Although harm mitigation and harm response interventions often have smaller side effects, interventions that limit access to certain AI capabilities will probably become increasingly warranted.
AI labs, governments, and other decision makers should be prepared to increasingly limit access to AI capabilities that lend themselves to misuse. Certain risks from AI will likely be too great to ignore, even if that means we must impede on legitimate use. These actors should also work on building up processes for determining when capabilities interventions are warranted. Furthermore, they should invest in developing interventions that further reduce the Misuse-Use Tradeoff. Continued research could yield better tools for preventing dangerous capabilities from ending up in the wrong hands, without also preventing beneficial capabilities from ending up in the right hands.
This post aims to give policymakers an overview of available methods for preventing the misuse of general-purpose AI systems, as well as the limitations of these methods.
GovAI research blog posts represent the views of their authors, rather than the views of the organisation.
As general-purpose AI models have become more capable, their potential for misuse has grown as well. Without safety measures in place, for example, many models can now be used to create realistic fake media. Some experts worry that future models could facilitate more extreme harm, including cyberattacks, voter manipulation, or even attacks with biological weapons.
Governments and companies increasingly recognise the need to evaluate models for dangerous capabilities. However, identifying dangerous capabilities is only a first step toward preventing misuse.
Developers must ultimately find ways to remove dangerous capabilities or otherwise limit the ability of users to apply them.1 To this end, developers currently implement a number of misuse-prevention techniques. These include fine-tuning, content filters, rejection sampling, system prompts, dataset filtering, and monitoring-based restrictions.
However, existing misuse-prevention techniques are far from wholly reliable. It is particularly easy to circumvent them when models are “open-sourced”. Even when models are accessed through online interfaces, however, the techniques can still be fairly easily circumvented through “jailbreaking” techniques. External researchers have demonstrated, for example, that jailbroken systems can perform tasks such as crafting targeted phishing emails.
If future models do eventually develop sufficiently dangerous capabilities (such as assisting in the planning and execution of a biological attack), then it may not be responsible to release them until misuse-prevention techniques are more reliable. Therefore, improving misuse-prevention techniques could reduce both social harms and future delays to innovation.
In this section, I will briefly survey available techniques for preventing misuse. I will then turn to discussing their limitations in more depth.
Companies developing general-purpose AI systems — like OpenAI and Anthropic — primarily reduce harmful outputs through a process known as “fine-tuning”. This involves training an existing AI model further, using additional data, in order to refine its capabilities or tendencies.
This additional data could include curated datasets of question/answer pairs, such as examples of how the model should respond to irresponsible requests. Fine-tuning data may also be generated through “reinforcement learning from human feedback” (RLHF), which involves humans scoring the appropriateness of the models’ responses, or through “reinforcement learning from AI feedback” (RLAIF). Through fine-tuning, a model can be conditioned to avoid dangerous behaviours or decline dangerous requests. For instance, GPT-4 was fine-tuned to reject requests for prohibited content, such as instructions for cyberattacks.
As with other misuse-prevention techniques, however, there are trade-offs with this approach: fine-tuning can also make models less helpful, by leading them to also refuse some benign requests.
Filters can be applied to both user inputs and model outputs. For inputs, filters are designed to detect and block users’ requests to produce dangerous content. For outputs, filters are designed to detect and block dangerous content itself.
Developers have various methods to train these filters to recognise harmful content. One option is using labelled datasets to categorise content as either harmful or benign. Another option is creating a new dataset by having humans score the harmfulness of the outputs a model produces or the outputs users send to it. Language models themselves can also be used to evaluate both inputs and outputs.
Rejection sampling involves generating multiple outputs from the same AI model, automatically scoring them based on a metric like “potential harm”, and then only presenting the highest scoring output to the user. OpenAI used this approach with WebGPT, the precursor to ChatGPT, to make the model more helpful and accurate.
System prompts are natural language instructions to the model that are pre-loaded into user interactions and are usually hidden from the user. Research indicates incorporating directives like “ignore harmful requests” into prompts can reduce harmful outputs.
Before training an AI model, developers can remove harmful data to ensure the model doesn’t learn from it. For example, OpenAI removed sexual and violent content from the dataset they trained DALLE-3 on. They make use of the same ‘classifiers’ developed for filters to identify training data that should be removed. A distinct downside of this approach, however, is that it can only be applied before a model is trained.
The techniques above take a purely preventative approach. Monitoring-based restrictions instead allow developers to respond to initial cases of misuse – or misuse attempts – by limiting users’ abilities to engage in further misuse.
AI service providers can leverage automated tools like input and output classifiers to screen for misuse. For instance, if a user repeatedly makes requests that are blocked by an input filter, the user may be flagged as high-risk. They might then be sent a warning, have their access temporarily reduced, or even receive a ban. Monitoring can also help reveal new vulnerabilities and patterns of misuse, allowing policies and other safeguards to be iteratively improved.
One potential problem with monitoring is that it can compromise privacy. That said, there are ways to preserve privacy. For instance, reports about high-level patterns of misuse can be generated without company staff needing to examine individuals’ data.
Unfortunately, these mitigations are currently ineffective for open-source models (i.e. models that users are allowed to download and run on their own hardware). Safeguards like filters, system prompts, and rejection sampling mechanisms can often be disabled by changing a few lines of code. Fine-tuning is also relatively trivial to reverse; one study also showed that fine-tuning for various Llama models could be undone for under $200. Additionally, once a model is open-sourced, there is no way to reliably oversee how it is being modified or used post-release. Although open-source models have a range of socially valuable advantages, they are also particularly prone to be misused.2
Putting behind an “application programming interface” (API) can help prevent the removal of safeguards. However, even when developers put their models behind an API, the safeguards can reduce, but not eliminate, misuse risks. OpenAI successfully reduced rule-breaking responses by 82% in GPT-4, but risks still remain. Anthropic similarly states that, despite interventions, their systems are not “perfectly safe”.
One challenge in preventing misuse is the phenomenon of “jailbreaks”, where users bypass AI system’s safety measures to generate prohibited content. Successful jailbreaks have employed tactics such as switching to a different language to evade content filters or framing requests for dangerous information within hypothetical scenarios like a script for a play.
Despite efforts from AI developers, it is still relatively trivial to jailbreak models. For instance, GPT-4 was jailbroken within 24 hours of release, despite the fact that OpenAI spent 6 months investing in safety measures. Developers are actively working to patch identified vulnerabilities, but users have continuedtofind ways around restrictions, as documented on sites like JailbreakChat.
It seems likely that the benefits of current models outweigh their harms. This is in part thanks to the mitigation techniques discussed here, but also in large part due to the model’s limited capabilities. For instance, the propensity of GPT-4 to make factual mistakes puts a cap on its ability to perform cyberattacks. In some cases, existing safeguards may not even be worth the costs they create for users (e.g. for instance by accidentally filtering benign content).
However, the unreliability of current misuse-prevention techniques may pose significant challenges in the future. If future models become sufficiently powerful, then even a single case of misuse, like a user building a bioweapon, could eventually be catastrophic. One possible implication is that models with sufficiently dangerous capabilities should not be released until better misuse-prevention techniques are developed.
If this implication is right, then better misuse-prevention techniques could prevent future roadblocks to innovation in addition to lowering societal risks.
There are a range of techniques developers can use to limit the misuse of their models. However, these are not consistently effective. If future models do develop sufficiently dangerous capabilities, then the reliability of misuse-prevention techniques could ultimately be a bottleneck to releasing them.
The author of this piece can be contacted at
be**********@go********.ai
. Thanks to the following people for their feedback: Alan Chan, Ben Garfinkel, Cullen O’Keefe, Emma Bluemke, Julian Hazell, Markus Anderljung, Matt van der Merwe, Patrick Levermore, Toby Shevlane.
GovAI blog posts represent the views of their authors, rather than the views of the organisation.
The UK AI Safety Summit was largely a success. To build on it, the international community has committed to hold further AI Safety Summits. The first follow-on summit will be held in mid-2024, with the Republic of Korea serving as host.
One underlying goal, for the organisers of the next summit, should be to reinforce the world’s commitment to an ambitious summit series. Even participants who are committed to further summits will need to decide, in future years, just how ambitious or pro forma their participation will be. The next summit is an opportunity to ensure ambitious participation, by demonstrating the promise of the series.
To this end, three concrete goals for the Korean summit could be to:
Create shared clarity on the need for progress in AI safety, for instance by:
Showcase progress since the last summit, for instance by:
Begin to build a long-term vision and roadmap for the series, for instance by:
Accomplishing these goals would be challenging, but would make the promise of the series clearer and lay a strong foundation for future summits. Participants would leave with a firmer sense of the problems the series is addressing, greater confidence that progress on these problems is possible, and a vision for how future summits will ultimately bring the world closer to shared solutions.
The first AI Safety Summit generated significant momentum. Remarkable outcomes included twenty-nine governments — including the US, EU, and China — signing a joint declaration on risks and opportunities from AI, seven leading AI companies releasing public reports on their own safety practices, and the UK government launching an AI Safety Institute.
However, there is no guarantee that this momentum will be sustained. It is common for international forums to become less productive over time. If participants lose clarity on a forum’s purpose or lose faith that the forum can achieve its purpose, then their participation will tend to become increasingly pro forma.
When future AI Safety Summits are held, both governments and companies will need to decide how ambitiously to participate. For example, governments will need to decide whether to send their most senior representatives, whether to dedicate substantial energy to pre-summit negotiations, and whether to sign onto bold shared commitments. If they do not participate ambitiously enough, the series will not achieve its potential.
Since participants notice and respond to each other’s participation levels, rising passivity would be self-reinforcing. Fortunately, in the same way, rising ambition would be self-reinforcing too.
The next AI Safety Summit is an important opportunity not only to sustain, but to reinforce the momentum generated by the first summit.
A valuable goal for the next summit would be to create shared clarity on the need for progress in AI safety. Although the first summit has helped to build this clarity, the subject is still new for most participants and substantial ambiguity remains. Greater clarity would allow participants to more fully appreciate the importance of the summit series. It would also facilitate more productive discussions.
More specifically, the summit could aim to create shared clarity about both (a) the risks AI might pose and (b) the ways in which the world is and is not prepared to handle these risks. Understanding these factors together makes it possible to see how much progress urgently needs to be made.
To increase clarity about risks, the summit organisers can:
To increase clarity about preparedness, the summit organisers can:
If the next summit can showcase progress since the first AI Safety Summit — held only around six months prior — then it can reinforce the ambition of participants. Meaningful progress would allow participants to see that, with enough effort, core AI safety challenges are tractable.
To this end, the summit organisers can:
The first summit produced a consensus statement — the Bletchley Declaration — that suggested high-level goals for future summits. The statement notes that the “agenda for our cooperation” will focus on “identifying AI safety risks of shared concern” and “building respective risk-based policies across our countries to ensure safety.”
However, the statement only briefly elaborates on these goals. It also does not say much about how future summits will work toward achieving them.
It could be valuable, then, to develop a fuller vision of what the series aims to achieve, along with a roadmap that describes how the series can fulfill this vision. This kind of vision and roadmap could increase the ambition of participants, by helping them understand what their continued participation can produce. A vision and roadmap would also support more effective summit planning in future years.
To take steps in this direction, the summit organisers could:
The AI Safety Summit series will likely run for many years. It could produce tremendous value. However, like many international forums, it could also sputter out.
The Korean AI Safety Summit — the first follow-up to the initial summit — is a key opportunity to ensure that the series lives up to its potential. It can reinforce the momentum produced by the initial summit and convince participants that they should approach the series ambitiously.
This underlying goal can be furthered by achieving three more concrete goals. In particular, it would be valuable for the next summit to: (1) create shared clarity on the need for progress in AI safety, (2) showcase progress since the last summit, and (3) begin to build a long-term vision and roadmap for the series.
This post outlines a concrete proposal for a pilot information-sharing regime, to give the UK government foresight into emerging risks and opportunities from AI.
GovAI research blog posts represent the views of their authors, rather than the views of the organisation.
The United Kingdom aims to become a global AI superpower, while also safely navigating the risks AI poses. However, there is an important gap in the government’s ability to achieve these goals: it cannot see advances in AI capabilities coming. As a result, it cannot easily prepare for new opportunities and risks.
This week, in recognition of this problem, the UK government secured pledges from three leading AI labs to provide “early access” to their models. However, this pledge has yet to be operationalised. It also is not clear that early access to new AI systems will, on its own, grant the government sufficiently thorough and longrun foresight.
To more systematically address the “foresight gap,” we propose that the UK government create an information-sharing regime between AI labs and the Office for Artificial Intelligence. This post outlines a concrete pilot proposal, based on interviews with think tank researchers, UK government staff working on AI policy issues, and staff at leading AI labs.1
We recommend that the Office for Artificial Intelligence set up a voluntary information-sharing pilot program with frontier AI labs, centered on model capability evaluations and compute usage. Such information-sharing would only be encouraged for a limited subset of new foundation models, which are especially compute-intensive or have especially general capabilities. This information could be shared both before and throughout model training and deployment processes. Shortly before release, in line with the recently announced commitments, labs could also grant the Office direct access to their models.
Subject-matter experts employed by the Office – or seconded from partner organisations – could analyse this information for its implications about current risks and forthcoming developments. Then, these experts could communicate actionable policy recommendations to relevant stakeholders throughout the UK government. For instance, recommendations could inform government efforts to develop technical standards and regulatory frameworks for foundation model development, or help the government to plan for national security incidents related to AI safety failures and misuse.
Foundation models are AI systems that can be applied or adapted to perform a wide range of tasks. As is noted in the recent UK whitepaper on AI regulation, the general-purpose capabilities of foundation models could have a transformative impact on society and should be a primary focus of government monitoring and horizon-scanning efforts for AI.
The degree of risk posed by current foundation models is contentious. However, continued progress could produce models that pose considerable risks across political, cyber, biotech, and critical infrastructure domains.2 Some experts believe that increasingly dangerous capabilities could emerge rapidly. At the same time, these models promise to enable significant efficiencies in areas such as healthcare and education.
The capabilities of new foundation models are growing more quickly than the government can design and implement AI policy. The natural result of this gap is a hasty and panicked approach to policymaking, as was evident in the wake of the ChatGPT launch.3
AI policymaking will have reduced effectiveness if it continues to be driven reactively after the launches of increasingly powerful AI systems. Policymakers need to be able to foresee potential harms from forthcoming systems, in order to prepare and take proactive policy actions. It is also important for policymakers to understand how development and risk mitigation practices are evolving within labs.
Unfortunately, public information about foundation model development is actually becoming scarcer. Up until recently, AI labs disclosed compute requirements and model design details alongside new large model releases. With the release of GPT-4, OpenAI declined to share compute and model details due to competition and safety concerns. Secrecy around development practices may continue to increase as AI capabilities advance. At the same time, as labs adopt new pre-release risk management practices, the lag between new capabilities being developed and being announced to the world may grow as well.
However, in recognition of these issues, lab leaders have stated a willingness to share sensitive details about their work with policymakers so that the public interest can be protected.4 The UK government should take lab leaders up on their offer and work to build an information-sharing regime.
We propose a voluntary pilot that centers on two types of information: model capability evaluations, which are helpful for assessing the potential impacts of new models,5 and compute requirements, which provide insight into the trajectory of artificial intelligence development.6 Model capability evaluations and compute requirements would each be voluntarily reported to the Office for Artificial Intelligence by frontier labs both ahead of and during deployment for a limited subset of new foundation models that surpass some level of compute usage or general-purpose capability. Using this information, the Office for Artificial Intelligence can monitor progress in capabilities and communicate suggested policy actions to relevant bodies throughout the government, to reduce risks and ensure the impact of new foundation models is beneficial.
Model capability and compute requirement information could be provided in the form of a training run report, which labs share at a few different milestones: before they begin the training process for new foundation models, periodically throughout the training process, and before and following model deployment.7
Training run reports7 could contain the following elements:
Capability evaluations: Knowledge of new foundation model capabilities will allow policymakers to identify likely risks and opportunities that deployment will create and to prepare for these proactively. Industry labs are already generating capability information by using benchmarks to quantify the model’s performance in its intended use cases. Labs are also performing both quantitative and qualitative evaluations which seek to identify harmful tendencies (such as bias or deceptiveness) and unintended dangerous capabilities (such as the ability to support cyberattacks). Details provided to the Office for Artificial Intelligence team could include:
Compute requirements: Knowledge of the compute used to train foundation models will provide policymakers with the context necessary to forecast future model capabilities and to monitor how underlying technical factors that influence model research and development, such as compute efficiency, are progressing. Information about compute usage is also helpful for assessing the climate impacts of AI.9 Ultimately, compute trends will allow policymakers to estimate the pace at which they should implement policies to prepare for forthcoming risks and opportunities. Industry labs already possess compute infrastructure information, since they must plan ahead and budget for compute requirements when designing a new foundation model. Details provided to the Office for Artificial Intelligence team could include:
Increasingly complete versions of the training report could be shared across the different stages of the development pipeline: before model training, during the training process, before model deployment, and after model deployment. If possible, in order to give sufficient foresight, it would be desirable for at least one month before training and one month before deployment.
In the pre-deployment stage, as a complement to the training report, companies may also be encouraged to offer early access to their fully trained models. Direct model access is a not a substitute for training reports, as training reports: (a) are likely to contain results from some evaluations that the government does not currently have the capacity or expertise to run, (b) contain some information that one cannot learn simply by using a model, and (c) allow the government to learn certain valuable information before a model is fully trained. However, the government may also be able to learn some additional useful information by interacting directly with the model.
This information-sharing program requires the capacity to effectively engage with AI lab staff, perform analysis, and disseminate findings across relevant policy bodies. To handle these tasks, the program could consist of a policy manager and two subject-matter experts. These subject-matter experts could potentially be identified and hired using a secondment program such as the DSIT Expert Exchange or also brought on through existing partnerships with publicly-funded technical research groups such as the CDEI. As an expert organisation working on relevant research, the Centre for the Governance of AI would also be willing to participate in or help facilitate secondment efforts for this program.
We suggest that a natural home for the project would be the Office for Artificial Intelligence. Alternatively, however, the project could be structured as an initiative within the Foundation Models Taskforce or as a more independent body akin to the Cyber Security Information Sharing Partnership within the National Cyber Security Centre.
In any case, there should also be a role with this team’s processes for third-party and academic organisations to participate in a consultation period for its resulting policy recommendations.10 This will help to ensure a diverse and representative set of viewpoints is included in its final guidance to policymakers.
This program should include robust information security measures to reduce the risks of sensitive AI development details leaking and ending up in the hands of other labs (who might gain a competitive advantage) or bad actors (who might find ways to exploit the information). Technical security measures should be in place for ensuring that unredacted training run reports are only accessible to a limited set of high-trust team members within the Office for Artificial Intelligence. Program administrators and lab partners could also collaboratively define an information-sharing protocol and require that the protocol be adhered to whenever new information is disclosed from one party to another.11 This helps ensure all program participants are aligned on how a specific piece of information can be used or whether it can be disseminated to other government stakeholders. Over time, such protocol rules could also be enforced through the technical architecture of a data portal and APIs by using structured transparency techniques.
A voluntary information-sharing regime between the UK government and AI labs would build frontier AI expertise within the Office for Artificial Intelligence and create working relationships with UK-based industry labs. Key government recipients of policy recommendations could include:
Over time, we imagine that the information-sharing program could scale to further information types and policy recommendation areas. Additionally, it could begin interfacing with a wider set of international AI labs and coordinating with relevant foreign government and multilateral offices focused on AI monitoring and evaluation (such as the NIST AI Measurement and Evaluation program in the United States or the Security and Technology Programme at the United Nations). Eventually, the program’s findings could help to inform an overall policy regime that steers AI development in a direction that is beneficial for the United Kingdom’s future.
Importantly, implementing an information-sharing program would not be without risks or costs. One challenge, as noted above, would be to ensure that sensitive information reported by labs does not leak to competitors or bad actors. We believe that a small government unit with strong information-security requirements would be unlikely to produce such leaks. Nonetheless, in order to secure lab buy-in, it would be important for this risk to be taken quite seriously.
Second, it would be important to ensure that the program does not disincentivize labs from performing evaluations that might reveal risks. To reduce the chance of any disincentive effect, it may be important to provide specific assurances about how information can be used. Assurances concerning liability protection could also conceivably be worthwhile.12
The authors of this piece can be contacted at ni****************@wh*****.edu and je**@lo****************.org .
This post summarises a new report, “Computing Power and the Governance of Artificial Intelligence.” The full report is a collaboration between nineteen researchers from academia, civil society, and industry. It can be read here.
GovAI research blog posts represent the views of their authors, rather than the views of the organisation.
Computing power – compute for short – is a key driver of AI progress. Over the past thirteen years, the amount of compute used to train leading AI systems has increased by a factor of 350 million. This has enabled the major AI advances that have recently gained global attention.
Governments have taken notice. They are increasingly engaged in compute governance: using compute as a lever to pursue AI policy goals, such as limiting misuse risks, supporting domestic industries, or engaging in geopolitical competition.
There are at least three ways compute can be used to govern AI. Governments can:
Compute governance is a particularly important approach to AI governance because it is feasible. Compute is detectable: training advanced AI systems requires tens of thousands of highly advanced AI chips, which cannot be acquired or used inconspicuously. It is excludable: AI chips, being physical goods, can be given to or taken away from specific actors and in cases of specific uses. And it is quantifiable: chips, their features, and their usage can be measured. Compute’s detectability and excludability are further enhanced by the highly concentrated structure of the AI supply chain: very few companies are capable of producing the tools needed to design advanced chips, the machines needed to make them, or the data centers that house them.
However, just because compute can be used as a tool to govern AI doesn’t mean that it should be used in all cases. Compute governance is a double-edged sword, with both potential benefits and the risk of negative consequences: it can support widely shared goals like safety, but it can also be used to infringe on civil liberties, perpetuate existing power structures, and entrench authoritarian regimes. Indeed, some things are better ungoverned.
In our paper we argue that compute is a particularly promising node for AI governance. We also highlight the risks of compute governance and offer suggestions for how to mitigate them. This post summarises our findings and key takeaways, while also offering some of our own commentary.
A note on authorship: The paper includes 19 co-authors, several of whom work at OpenAI, a company developing state-of-the-art foundation models. This naturally raises concerns that the content of the paper will be biased toward the interests of AI developers. This suspicion is healthy. Further, many of the mechanisms explored in the paper have only seen cursory investigation and given the size of the author group, authorship does not imply endorsement of all the paper’s statements from any author or their respective organisation. We hope that readers will be motivated to closely engage with the paper’s arguments, take little for granted, publicly raise disagreements, and offer alternative ideas. We intend for this paper to provide a basis for continued research and thoughtful examination of the role of compute in AI governance.
Much AI progress over the past decade has resulted from significant increases in the amount of computing power (“compute”) used to train and run AI models. Across large-language models, Go, protein folding, and autonomous vehicles, the greatest breakthroughs have involved developers successfully leveraging huge amounts of computing power to train models on vast datasets to independently learn how to solve a problem, rather than hard-coding such knowledge. In many AI domains, researchers have found scaling laws: performance on the training objective (e.g. “predict the next word”) predictably increases as the amount of compute – typically measured in the number of operations (e.g. FLOP) involved – used to train a model increases.
Hardware improvements and massively increased investments have resulted in the amount of compute used to train notable AI systems increasing by a factor of 350 million in the past thirteen years. Currently the compute used to train notable AI systems doubles every six months. In the last year alone, Nvidia’s data center revenue nearly quadrupled.
Compute is easier to govern than other inputs to AI. As such, compute can be used as a tool for AI governance.
Four features contribute to compute’s governability:
The importance of compute to AI capabilities and the feasibility of governing it make it a key intervention point for AI governance efforts. In particular, compute governance can support three kinds of AI governance goals: it can help increase visibility into AI development and deployment, allocate AI inputs towards more desirable purposes, and enforce rules around AI development and deployment.
Visibility is the ability to understand which actors use, develop, and deploy compute-intensive AI, and how they do so. The detectability of compute allows for better visibility in several ways. For example, cloud compute providers could be required to monitor large-scale compute usage. By applying processes such as know-your-customer requirements to the cloud computing industry, governments could better identify potentially problematic or sudden advances in AI capabilities. This would, in turn, allow for faster regulatory response.
Visibility also raises important privacy concerns. Fortunately, some methods may offer noninvasive insights into compute usage. Data center operators have minimal access to information about their customers’ compute usage, such as the number and types of chips used, when those chips are used, and how much internet traffic is processed through the relevant computing cluster. However, that information can be used to glean certain insights. For example, the computational signatures of training and running inference on AI systems tend to differ. Clusters used for inference require constant internet traffic to serve customers, whereas clusters used for training typically access training data hosted locally.
Allocation is the ability to shift AI capabilities among different actors and projects to achieve some end. Once again, features of compute such as excludability and quantifiability offer promising opportunities to govern AI through allocation.
For example, policymakers may seek to differentially advance beneficial AI development by increasing the resources available to certain kinds of beneficial AI research and development. This could include subsidising research into AI applications for climate change mitigation, agricultural efficiency, clean energy, public health, education, or even defence against AI misuse. Compute can also be allocated to actors lacking compute resources, such as academics, startups, or low and middle-income countries.
Perhaps compute could also be allocated to adjust the pace of AI progress. A large reserve of compute could be procured by a government or an alliance of governments. The reserve could be used to modulate the amount of compute in the economy, influencing the overall pace of AI progress.
Finally, enforcement is the ability to respond to violations of norms and laws related to AI, such as reckless development and deployment or certain deliberate misuse.
One enforcement mechanism discussed in the paper is physically limiting chip-to-chip networking to make it harder to train and deploy large AI systems. For example, the US government’s export controls on high-end AI-relevant chip sales to China aim to hamper Chinese actors’ ability to develop frontier AI models, where (tens of) thousands of chips are orchestrated for one training run. That goal could be met in a more targeted way by exporting chips that can only have high-bandwidth connections with a sufficiently small number of other chips. Such chips do not exist today, but could potentially be developed.
A more speculative enforcement mechanism would be preventing risky training runs via multiparty controls. Certain decisions about which AI systems to develop or deploy may be too high-stakes to leave to a single actor or individual. Instead, such decisions could be made jointly by a number of actors or a governing third party. Multisignature cryptographic protocols could be used to share control of a metaphorical “start switch” between many actors.
The power to decide how large amounts of compute are used could be allocated via digital “votes” and “vetoes,” with the aim of ensuring that the most risky training runs and inference jobs are subject to increased scrutiny. While this may appear drastic relative to the current state of largely unregulated AI research, there is precedent in the case of other high-risk technologies: nuclear weapons use similar mechanisms, called permissive action links.
Although compute governance can be an effective regulatory tool, it may not always be the right one to use. It is one option among many for policymakers.
For example, compute governance may become less effective as algorithms and hardware improve. Scientific progress continually decreases the amount of computing power needed to reach any level of AI capability, as well as the cost to perform any number of computations. As the power and cost necessary to achieve any given AI capability decreases, these metrics will become less detectable and excludable.
The extent to which this effect undermines compute governance largely depends on the importance of relative versus absolute capabilities. Increases in compute efficiency make it easier and cheaper to access a certain level of capability, but as long as scaling continues to pay dividends, the highest-capability models are likely to be developed by a small number of actors, whose behavior can be governed via compute.
On a related note, compute may be an inappropriate tool to govern low-compute specialised models with dangerous capabilities. For example, AlphaFold 2 achieved superhuman performance on protein folding prediction using fewer than 1023 operations – two orders of magnitude less compute than models like GPT-4. Compute governance measures to limit the development models risk also limiting the development of similarly-sized, but harmless, models. In other words, compute governance measures seem most appropriate for risks originating from a small number of compute-intensive models.
Perhaps more importantly, compute governance can also cause harm. Intrusive compute governance measures risk infringing on civil liberties, propping up the powerful, and entrenching authoritarian regimes. Indeed, some things are better ungoverned.
Certain compute governance efforts, especially those aimed at increasing visibility into AI, may increase the chance that private or sensitive personal or commercial information is leaked. AI companies, users of AI systems, and compute providers all go to great lengths to preserve the integrity of their and their customers’ data. Giving more actors access to such information raises the chance of data leakage and privacy infringement.
Large concentrations of compute are also an increasingly crucial economic and political resources. Centralising the control of this resource could pose significant risks of abuse of power by regulators, governments, and companies. Companies might engage in attempts at regulatory capture, and government officials could see increased opportunities for corrupt or power-seeking behaviour.
Fortunately, there are a number of ways to increase the chance that compute governance remains effective while reducing unintended harm. Compute governance is one tool among many available to policymakers and it should be wielded carefully and deliberately.
Exclude small-scale AI compute and non-AI compute from governance regimes. Many of the above concerns can be addressed by applying compute governance measures in a more targeted manner; for example, by focusing on the large-scale computing resources needed to develop and deploy frontier AI systems.
Implement privacy-preserving practices and technologies. Where compute governance touches large-scale computing that contains personal information, care must be taken to minimise privacy intrusions. Take, for example, know your customer (KYC) regimes for cloud AI training: applying them only to direct purchasers of large amounts of cloud AI compute capacity would impose almost no privacy burdens on consumers. KYC could also feasibly draw on indicators that are already available – such as chip hours, types of chips, and how GPUs are networked – preserving existing privacy controls for compute providers and consumers.
Focus compute-based controls where ex ante measures are justified. Compute governance (especially in its “allocation” and “enforcement” forms) is often a blunt tool, and generally functions upstream of the risks it aims to mitigate and the benefits it seeks to promote. Regulatory and governance efforts typically focus on ex post mechanisms, imposing penalties after some undesirable behaviour has occured. Such measures are likely appropriate in dealing with many governance issues arising from AI, especially stemming from inappropriate use of AI systems.
However, some harms from AI may justify ex ante intervention. For example, where the harm is so large that no actor would be able to compensate for it after the fact, such as catastrophic or national security risks, ex ante measures would be appropriate.
Periodically revisit controlled compute related policies. Regulatory thresholds – such as a training compute threshold of 1026 operations – or list-based controls on technologies – such as those used in export controls – can become outdated fairly quickly as technology evolves. Compute governance efforts should therefore have built-in mechanisms for reviews and updates.
Ensure substantive and procedural safeguards. Like many government interventions, compute governance efforts could be abused. Measures to control compute should therefore include substantive and procedural safeguards. Substantively, such controls could prevent downsides of compute governance by, for example, limiting the types of controls that can be implemented, the type of information that regulators can request, and the entities subject to such regulations. Procedural safeguards could include such measures as notice and comment rule making, whistleblower protections, internal inspectors general and advocates for consumers within the regulator, opportunities for judicial review, advisory boards, and public reports on activities.
Compute governance efforts are already underway and compute will likely continue to play a central role in the AI ecosystem, making it an increasingly important node for AI governance. Compute governance can support AI policy goals in multiple ways: by increasing visibility into AI progress, shaping the allocation of AI inputs, and enforcing rules regarding AI development and deployment. At the same time, designing effective compute governance measures is a challenging task. The risks of privacy violations and power concentration must be managed carefully. We hope this paper will help policymakers and scholars scrutinise, rethink, and ultimately improve compute governance proposals.
* – Corresponding author: Lennart Heim
le**********@go********.ai
>
Protected by Security by CleanTalk
