Category: AI Careers

A Note from Ben Garfinkel, Acting Director

2021 was a year of big changes for GovAI.

Most significantly, we spun out from the University of Oxford and adopted a new non-profit structure. At the same time, after five years within the organisation, I stepped into the role of Acting Director. Our founder, Allan Dafoe, shifted into the role of President. We also created a new advisory board consisting of several leaders in AI governance: Allan, Ajeya Cotra, Tasha McCauley, Toby Ord, and Helen Toner.

Successfully managing these transitions was our key priority for the past year. Over a period of several months — through the hard work, particularly, of Markus Anderljung (Head of Policy), Alexis Carlier (Head of Strategy), and Anne le Roux (Head of Operations) — we completed our departure from Oxford, arranged a new fiscal sponsorship relationship with the Centre for Effective Altruism, revised our organisation chart and management structures, developed a new website and new branding, and fundraised $3.8 million to support our activities in the coming years.

Our fundamental mission has not changed: we are still building a global research community, dedicated to helping humanity navigate the transition to a world with advanced AI. However, these transitions have served as a prompt to reflect on our activities, ambitions, and internal structure. One particularly significant implication of our exit from Oxford has been an increase in the level of flexibility we have in hiring, fundraising, operations management, and program development. In the coming year, as a result, we plan to explore expansions in our field-building and policy development work. We believe we may be well-placed to organise an annual conference for the field, to offer research prizes, and to help bridge the gap between AI governance research and policy.

Although 2021 was a transitional year, this did not prevent members of the GovAI team from producing a stream of new research. The complete list of our research output – given below – contains a large volume of research targeting different aspects of AI governance. One piece that I would like to draw particular attention to is Dafoe et al.’s paper Open Problems in Cooperative AI, which resulted in the creation of a $15 million foundation for the study of cooperative intelligence.

Our key priority for the coming year will be to grow our team and refine our management structures, so that we can comfortably sustain a high volume of high-quality research while also exploring new ways to support the field. Hiring a Chief of Staff will be a central part of this project. We are also currently hiring for Research Fellows and are likely to open a research management role in the future. Please reach out to

co*****@go********.ai











if you think you might be interested in working with us.

I’d like to close with a few thank-yous. First, I would like to thank our funders for their generous support of our work: Open Philanthropy, Jaan Tallinn (through the Survival and Flourishing Fund), a donor through Effective Giving, the Long-Term Future Fund, and the Center for Emerging Risk Research. Second, I would like to thank the Centre for Effective Altruism for its enormously helpful operational support. Third, I would like to thank the Future of Humanity Institute and the University of Oxford for having provided an excellent initial home for GovAI.

Finally, I would like to express gratitude for everyone who’s decided to focus their career on AI governance. It has been incredible watching the field’s growth over the past five years. From many conversations, it is clear to me that many of the people now working in the field have needed to pay significant upfront costs to enter it: retraining, moving across the world, dropping promising projects, or stepping away from more lucrative career paths. This is, by-and-large, a community united by a genuine impulse to do good – alongside a belief that the most important challenges of this century are likely to be deeply connected to AI. I’m proud to be part of this community, and I hope that GovAI will find ways to support it for many years to come.

‍

Publications

Publications from GovAI-led projects or publications with first authors from the GovAI Team are listed with summaries. Publications with contributions from the GovAI team are listed without summaries. You can find all our publications on our research page and our past annual reports on the GovAI blog.

‍

Political Science and International Relations

The Logic of Strategic Assets: From Oil to AI: Jeffrey Ding and Allan Dafoe (June 2021) Security Studies (available here) and associated The Washington Post article (available here)

Summary: This paper offers a framework for distinguishing between strategic assets (defined as assets that require attention from the highest levels of the state) and non-strategic assets.

China’s Growing Influence over the Rules of the Digital Road: Jeffrey Ding (April 2021) Asia Policy (available here)

Summary: This essay examines the growing role of China in international standards-setting organizations, as a window into its efforts to influence global digital governance institutions, and highlights areas where the U.S. can preserve its interests in cyberspace.

Engines of Power: Electricity, AI, and General-Purpose Military Transformations: Jeffrey Ding and Allan Dafoe (June 2021) Preprint (available here)

Summary: Drawing from the economics literature on GPTs, the authors distill several propositions on how and when GPTs affect military affairs. They argue that the impacts of GMTs on military effectiveness are broad, delayed, and shaped by indirect productivity spillovers.

The Rise and Fall of Great Technologies and Powers: Jeffrey Ding (November 2021) Preprint (available here)

Summary: In this paper, Jeffrey Ding proposes an alternative mechanism based on the diffusion of general-purpose technologies (GPTs), which presents a different trajectory for countries to leapfrog the industrial leader.

Reputations for Resolve and Higher-Order Beliefs in Crisis Bargaining: Allan Dafoe, Remco Zwetsloot, and Matthew Cebul (March 2021) Journal of Conflict Resolution (available here)

Summary: Reputations for resolve are said to be one of the few things worth fighting for, yet they remain inadequately understood. This paper offers a framework for estimating higher-order effects of behaviour on beliefs about resolve, and finds evidence of higher-order reasoning in a survey experiment on quasi-elites.

The Offense-Defense Balance and The Costs of Anarchy: When Welfare Improves Under Offensive Advantage: R. Daniel Bressler, Robert F. Trager, and Allan Dafoe (May 2021) Preprint (available here)

Coercion and the Credibility of Assurances: Matthew D. Cebul, Allan Dafoe, and Nuno P. Monteiro (July 2021) The Journal of Politics. (available here)

‍

Survey Work

Ethics and Governance of Artificial Intelligence: Evidence from a Survey of Machine Learning Researchers: Baobao Zhang, Markus Anderljung, Lauren Kahn, Noemi Dreksler, Michael C. Horowitz, and Allan Dafoe (May 2021) Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (available here, associated Wired article here)

Summary: To examine the views of machine learning researchers, the authors conducted a survey of those who published in the top AI/ML conferences (N = 524). They compare these results with those from a 2016 survey of AI/ML researchers and a 2018 survey of the US public.

Skilled and Mobile: Survey Evidence of AI Researchers’ Immigration Preferences: Remco Zwetsloot, Baobao Zhang, Noemi Dreksler, Lauren Kahn, Markus Anderljung, Allan Dafoe, and Michael C. Horowitz (April 2021) Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (available here) and associated Perry World House Report (available here)

Summary: The authors conducted a survey (n=524) of the immigration preferences and motivations of researchers that had papers accepted at one of two prestigious AI conferences: the Conference on Neural Information Processing Systems (NeurIPS) and the International Conference on Machine Learning (ICML).

‍

Economics of AI

Artificial Intelligence, Globalization, and Strategies for Economic Development: Anton Korniek and Joseph Stiglitz (February 2021) National Bureau of Economic Research Working Paper (available here) and associated IMF Working Paper (available here)

Summary: The authors analyze the economic forces behind developments caused by new technologies and describe economic policies that would mitigate the adverse effects on developing and emerging economies while leveraging the potential gains from technological advanc

Covid-19 Driven Advances in Automation and Artificial Intelligence Risk Exacerbating Economic Inequality: Anton Korinek and Joseph Stiglitz (March 2021) The British Medical Journal (available here)

Summary: In this paper, the authors make the case for a deliberate effort to steer technological advances in a direction that enhances the role of human workers.

AI and Shared Prosperity: Katya Klinova and Anton Korinek (May 2021) Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society (available here)

Economic Growth Under Transformative AI: A Guide to the Vast Range of Possibilities for Output Growth, Wages, and the Laborshare: Philip Trammell and Anton Korinek (October 2021) Global Priorities Institute Report (available here)

‍

Law and Policy

AI Policy Levers: A Review of the U.S. Government’s Tools to Shape AI Research, Development, and Deployment: Sophie-Charlotte Fischer, Jade Leung, Markus Anderljung, Cullen O’Keefe, Stefan Torges, Saif M. Khan, Ben Garfinkel, and Allan Dafoe (October 2021) GovAI Report (available here)

Summary: The authors provide an accessible overview of some of the USG’s policy levers within the current legal framework. For each lever, they describe its origin and legislative basis as well as its past and current uses; they then assess the plausibility of its future application to AI technologies.

AI Ethics Statements: Analysis and lessons learnt from NeurIPS Broader Impact Statements: Carolyn Ashurst, Emmie Hine, Paul Sedille, and Alexis Carlier (November 2021) Preprint (available here)

Summary: In this paper, the authors provide a dataset containing the impact statements from all NeurIPS 2020 papers, along with additional information such as affiliation type, location and subject area, and a simple visualisation tool for exploration. They also provide an initial quantitative analysis of the dataset.

Emerging Institutions for AI Governance: Allan Dafoe and Alexis Carlier (June 2020) AI Governance in 2020: A Year in Review (available here)

Summary: The authors describe some noteworthy emerging AI governance institutions, and comment on the relevance of Cooperative AI research to institutional design.

Institutionalizing Ethics in AI Through Broader Impact Requirements: Carina Prunkl, Carolyn Ashurst, Markus Anderljung, Helena Webb, Jan Leike, and Allan Dafoe (February 2021). Nature Machine Intelligence (preprint available here)

Summary: In this article, the authors reflect on a novel governance initiative by one of the world’s largest AI conferences. Drawing insights from similar governance initiatives, they investigate the risks, challenges and potential benefits of such an initiative.

Future Proof: The Opportunity to Transform the UK’s Resilience to Extreme Risks (Artificial Intelligence Chapter): Toby Ord, Angus Mercer, Sophie Dannreuther, Jess Whittlestone, Jade Leung, and Markus Anderljung (June 2021) The Centre for Long-term Resilience Report (available here)

Filling gaps in trustworthy development of AI: Shahar Avin, Haydn Belfield, Miles Brundage, Gretchen Krueger, Jasmine Wang, Adrian Weller, Markus Anderljung, Igor Krawczuk, David Krueger, Jonathan Lebensold, Tegan Maharaj, and Noa Zilberman (December 2021) Science (preprint available here)

‍

Cooperative AI

Cooperative AI: Machines Must Learn to Find Common Ground: Allan Dafoe, Yoram Bachrach, Gillian Hadfield, Eric Horvitz, Kate Larson, and Thore Graepel (2021) Nature Comment (available here) and associated DeepMind Report (available here)

Summary: This Nature Comment piece argues that to help humanity solve fundamental problems of cooperation, scientists need to reconceive artificial intelligence as deeply social. The piece builds on Open Problems in Cooperative AI (Allan Dafoe et al. 2020).

Normative Disagreement as a Challenge for Cooperative AI: Julian Stastny, Maxime Riché, Alexander Lyzhov, Johannes Treutlein, Allan Dafoe, and Jesse Clifton (November 2021) Preprint (available here)

‍

History

International Control of Powerful Technology: Lessons from the Baruch Plan for Nuclear Weapons: Waqar Zaidi and Allan Dafoe (March 2021) GovAI Report (available here) and associated Financial Times article (available here)

Summary: The authors focus on the years 1944 to 1951 and review this period for lessons for the governance of powerful technologies. They find that radical schemes for international control can get broad support when confronted by existentially dangerous technologies, but this support can be tenuous and cynical.  

‍

Computer Science and Other Technologies

A Tour of Emerging Cryptographic Technologies: Ben Garfinkel (May 2021) GovAI Report (available here)

Summary: The author explores recent developments in the field of cryptography and considers several speculative predictions that some researchers and engineers have made about the technologies’ long-term political significance.

RAFT: A Real-World Few-Shot Text Classification Benchmark: Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, Michael Noetel, and Andreas Stuhlmüller (October 2021) NeurIPS 2021 (available here)

‍

Select Publications from the GovAI Affiliate Network

Legal Priorities Research: A Research Agenda: Christoph Winter, Jonas Schuett, Eric Martínez, Suzanne Van Arsdal, Renan Araújo, Nick Hollman, Jeff Sebo, Andrew Stawasz, Cullen O’Keefe, and Giuliana Rotola (January 2021)

Legal Priorities Project (available here)

Assessing the Scope of U.S. Visa Restrictions on Chinese Students: Remco Zwetsloot, Emily Weinstein, and Ryan Fedasiuk (February 2021)

Center for Security and Emerging Technology (available here)

Understanding the Capabilities, Limitations, and Societal Impact of Large Language Models: Alex Tamkin, Miles Brundage, Jack Clark, and Deep Ganguli (February 2021)

Preprint (available here)

Artificial Intelligence, Forward-Looking Governance and the Future of Security: Sophie-Charlotte Fischer and Andreas Wenger (March 2021)

Swiss Political Science Review (available here)

Technological Internationalism and World Order: Waqar H. Zaidi (June 2021)

Cambridge University Press (available here)

Public Opinion Toward Artificial Intelligence: Baobao Zhang (May 2021)

Forthcoming in the Oxford Handbook of AI Governance (preprint available here)

Towards General-Purpose Infrastructure for Protecting Scientific Data Under Study: Andrew Trask and Kritika Prakash (October 2021)

Preprint (available here)

‍

Events

You can register interest or sign up to updates about our 2022 events on our events page. We will host the inaugural Governance of AI Conference in Autumn, and seminars with Sam Altman, Paul Scharre, and Holden Karnofsky in Spring.

David Autor, Katya Klinova, and Ioana Marinescu on The Work of the Future: Building Better Jobs in an Age of Intelligent Machines

GovAI Seminar Series (recording and transcript here)

In the spring of 2018, MIT President L. Rafael Reif commissioned the MIT Task Force on the Work of the Future. He tasked them with understanding the relationships between emerging technologies and work, to help shape public discourse around realistic expectations of technology, and to explore strategies to enable a future of shared prosperity. In this webinar David Autor, co-chair of the Task Force, discussed their latest report: The Work of the Future: Building Better Jobs in an Age of Intelligent Machines. The report documents that the labour market impacts of technologies like AI and robotics are taking years to unfold, however, we have no time to spare in preparing for them.

Joseph Stiglitz and Anton Korinek on AI and Inequality
GovAI Seminar Series (recording and transcript here)

Over the next decades, AI will dramatically change the economic landscape. It may also magnify inequality, both within and across countries. Joseph E. Stiglitz, Nobel Laureate in Economics, joined us for a conversation with Anton Korinek on the economic consequences of increased AI capabilities. They discussed the relationship between technology and inequality, the potential impact of AI on the global economy, and the economic policy and governance challenges that may arise in an age of transformative AI. Korinek and Stiglitz have co-authored several papers on the economic effects of AI.

Stephanie Bell and Katya Klinova on Redesigning AI for Shared Prosperity

GovAI Seminar Series (recording and transcript here)

AI poses a risk of automating and degrading jobs around the world, creating harmful effects to vulnerable workers’ livelihoods and well-being. How can we deliberately account for the impacts on workers when designing and commercializing AI products in order to benefit workers’ prospects while simultaneously boosting companies’ bottom lines and increasing overall productivity? The Partnership on AI’s recently released report Redesigning AI for Shared Prosperity: An Agenda puts forward a proposal for such accounting. The Agenda outlines a blueprint for how industry and government can contribute to AI that advances shared prosperity.

Audrey Tang and Hélène Landemore on Taiwan’s Digital Democracy, Collaborative Civic Technologies, and Beneficial Information Flows

GovAI Seminar Series (recording and transcript here)

Following the 2014 Sunflower Movement protests, Audrey Tang—a prominent member of the civic social movement g0v—was headhunted by Taiwanese President Tsai Ing-wen’s administration to become the country’s first Digital Minister. In a recent GovAI webinar, Audrey discussed collaborative civic technologies in Taiwan and their potential to improve governance and beneficial information flows.

‍

Essays, Opinion Articles, and Other Public Work

The Economics of AI: Anton Korinek

Coursera Course (available here)

Survey on AI existential risk scenarios: Sam Clarke, Alexis Carlier, and Jonas Schuett

EA Forum Post (available here)

Is Democracy a Fad?: Ben Garfinkel

EA Forum Post (available here)

Some AI Governance Research Ideas: Markus Anderljung and Alexis Carlier

EA Forum Post (available here)

Open Problems in Cooperative AI: Allan Dafoe and Edward Hughes

Alan Turing Institute Talk (available here)

Why Governing AI is Crucial to Human Survival: Allan Dafoe

Big Think Video (available here)

Human Compatible AI and the Problem of Control: ​​Stuart Russell, Allan Dafoe, Rob Reich, and Marietje Schaake

Center for Human-Compatible AI Discussion (available here)

Artificial Intelligence, Globalization, and Strategies for Economic Development: Anton Korinek, Joseph Stiglitz, and Erik Brynjolfsson

Stanford Digital Economy Lab talk (available here)

China’s AI ambitions and Why They Matter: Jeffrey Ding

Towards Data Science Podcast (available here)

Disruptive Technologies and China: Jeffrey Ding, Matt Sheehan, Helen Toner, & Graham Webster

Rethinking Economics NL Discussion (available here)

‍

As our 2022 Summer Fellowship comes to a close, we’re proud to highlight what our Summer Fellows have been up to! 

Summer and Winter Fellowships provide an opportunity for early-career individuals to spend three months working on an AI governance research project, learning about the field, and making connections with other researchers and practitioners. 

Summer Fellows come from a variety of disciplines and a range of prior experience – some fellows ventured into entirely new intellectual territory for their projects, and some fellows used the time to extend their previous work. 

We gratefully thank all of the supervisors for their mentorship and guidance this summer, and for dedicating time to training the next generation of researchers. 

If you’re interested in applying for future fellowships, check out our Opportunities page. You can register your expression of interest here.

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

‍

Read the paper “Open Problems in Cooperation” here. You can also read about the work of the Cooperative AI Foundation, whose work was inspired by the paper, on their website. The following extended abstract summarizes the paper’s key message:
‍

Problems of cooperation—in which agents have opportunities to improve their joint welfare but are not easily able to do so—are ubiquitous and important. They can be found at all scales ranging from our daily routines—such as driving on highways, scheduling meetings, and working collaboratively—to our global challenges—such as peace, commerce, and pandemic preparedness. Human civilization and the success of the human species depends on our ability to cooperate.

Advances in artificial intelligence pose increasing opportunity for AI research to promote human cooperation. AI research enables new tools for facilitating cooperation, such as language translation, human-computer interfaces, social and political platforms, reputation systems, algorithms for group decision-making, and other deployed social mechanisms; it will be valuable to have explicit attention to what tools are needed, and what pitfalls should be avoided, to best promote cooperation. AI agents will play an increasingly important role in our lives, such as in self-driving vehicles, customer assistants, and personal assistants; it is important to equip AI agents with the requisite competencies to cooperate with others (humans and machines). Beyond the creation of machine tools and agents, the rapid growth ofAI research presents other opportunities for advancing cooperation, such as from research insights into social choice theory or the modeling of social systems.

The field of artificial intelligence has an opportunity to increase its attention to this class of problems, which we refer to collectively as problems in Cooperative AI. The goal would be to study problems of cooperation through the lens of artificial intelligence and to innovate in artificial intelligence to help solve these problems. Whereas much AI research to date has focused on improving the individual intelligence of agents and algorithms, the time is right to also focus on improving social intelligence: the ability of groups to effectively cooperate to solve the problems they face.

AI research relevant to cooperation has been taking place in many different areas, including in multi-agent systems, game theory and social choice, human-machine interaction and alignment, natural language processing, and the construction of social tools and platforms. Our recommendation is not merely to construct an umbrella term for these areas, but rather to encourage focused research conversations, spanning these areas, focused on cooperation. We see opportunity to construct more unified theory and vocabulary related to problems of cooperation. Having done so, we think AI research will be in a better position to learn from and contribute to the broader research program on cooperation spanning the natural sciences, social sciences, and behavioural sciences.

Our overview comes from the perspective of authors who are especially impressed by and immersed in the achievements of deep learning(1) and reinforcement learning (2). From that perspective, it will be important to develop training environments, tasks, and domains that can provide suitable feedback for learning and in which cooperative capabilities are crucial to success, non-trivial, learnable, and measurable. Much research in multi-agent systems and human-machine interaction will focus on cooperation problems in contexts of pure common interest. This will need to be complemented by research in mixed-motives contexts, where problems of trust, deception, and commitment arise. Machine agents will often act on behalf of particular humans and will impact other humans; as a consequence, this research will need to consider how machines can adequately understand human preferences, and how best to integrate human norms and ethics into cooperative arrangements. Researchers building social tools and platforms will have other perspectives on how best to make progress on problems of cooperation, including being especially informed by real-world complexities. Areas such as trusted hardware design and cryptography may be relevant for addressing commitment problems and cryptography. Other aspects of the problem will benefit from expertise from other sciences, such as political science, law, economics, sociology, psychology, and neuroscience. We anticipate much value in explicitly connecting AI research to the broader scientific enterprise studying the problem of cooperation and to the broader effort to solve societal cooperation problems.

We recommend that “Cooperative AI” be given a technically precise, problem-defined scope; otherwise, there is a risk that it acquires an amorphous cloud of meaning, incorporating adjacent (clusters of) concepts such as aligned AI, trustworthy AI, and beneficial AI. Cooperative AI, as scoped here, refers to AI research trying to help individuals, humans and machines, to find ways to improve their joint welfare.For any given situation and set of agents, this problem is relatively well defined and unambiguous. The Scope section elaborates on the relationship to adjacent areas. Conversations on Cooperative AI can be organized in part in terms of the dimensions of cooperative opportunities. These include the strategic context, the extent of common versus conflicting interest, the kinds of entities who are cooperating, and whether the researchers are focusing on the cooperative competence of individuals or taking the perspective of a social planner. Conversations can also be focused on key capabilities necessary for cooperation, including:

• Understanding of other agents, their beliefs, incentives, and capabilities.
• Communication between agents, including building a shared language and overcoming mistrustand deception.
• Constructing cooperative commitments, so as to overcome incentives to renege on a cooperative arrangement.
• Institutions, which can provide social structure to promote cooperation, be they decentralizedand informal, such as norms, or centralized and formal, such as legal systems.

Just as any area of research can have downsides, so is it prudent to investigate the potential downsides of research on Cooperative AI. Cooperative competence can be used to exclude others, some cooperative capabilities are closely related to coercive capabilities, and learning cooperative competence can be hard to disentangle from coercion and competition. An important aspect of this research, then, will be investigating potential downsides and studying how best to anticipate and mitigate them. The paper is structured as follows. We offer more motivation in the section “Why Cooperative AI?” We then discuss several important dimensions of “Cooperative Opportunities.” The bulk of our discussion is contained in the “Cooperative Capabilities” section, which we organize in terms of Understanding, Communication, Commitment, and Institutions. We then reflect on “The Potential Downsides of Cooperative AI” and how to mitigate them. Finally, we conclude.

‍

To read the full “Open Problems in Cooperative AI” report PDF, click here.

‍

This post, authored by Markus Anderljung, Lennart Heim, and Toby Shevlane, argues that a newly proposed US government institution has an opportunity to support “structured access” to large AI models. GovAI research blog posts represent the views of their authors, rather than the views of the organization.

Compute funds and pre-trained models

One of the key trends in AI research over the last decade is its growing need for computational resources. Since 2012, the compute required to train state-of-the-art (SOTA) AI models has been doubling roughly every six months. Private AI labs are producing an increasing share of these high-compute SOTA AI models(1), leading many to worry about a growing compute divide between academia and the private sector. Partly in response to these concerns, there have been calls for the creation of a National AI Research Resource (NAIRR). The NAIRR would help provide academic researchers with access to compute, by either operating its own compute clusters or distributing credits that can be used to buy compute from other providers. It would also further support academic researchers by granting them access to data, including certain government-held datasets. Congress has now tasked the National Science Foundation with setting up a National AI Research Resource Task Force, which is due to deliver an interim report on the potential design of the NAIRR in May 2022.

We argue that for the NAIRR to meet its goal of supporting non-commercial AI research, its design must take into account what we predict will be another closely related trend in AI R&D: an increasing reliance on large pre-trained models, accessed through application programming interfaces (APIs). Large pre-trained models are AI models that require vast amounts of compute to create and that can often be adapted for a wide array of applications. The most widely applicable of these pre-trained models have recently been called foundation models, because they can serve as a “foundation” for the development of many other models. Due to commercial considerations and concerns about misuse, we predict that private actors will become increasingly hesitant to allow others to download copies of these models. We instead expect these models to be accessible primarily through APIs, which allow people to use or study models that are hosted by other actors. While academic researchers need access to compute and large datasets, we argue that they will also increasingly require API access to large pre-trained models. (Others have made similar claims.) The NAIRR could facilitate such access by setting up infrastructure for hosting and accessing large pre-trained models and inviting developers of large pre-trained models (across academia, industry, and government) to make their models available through the system. At the same time, they could allow academics to use NAIRR compute resources or credits to work with these models. 

The NAIRR has an opportunity, here, to ensure that academic researchers will be able to learn from and build upon some of the world’s most advanced AI models. Importantly, by introducing an API, the NAIRR could provide structured access to the pre-trained models so as to reduce any risks they might pose, while still ensuring easy access for research use. API access can allow outside researchers to understand and audit these models, for instance identifying security vulnerabilities or biases, without also making it easy for others to repurpose and misuse them.  

Concretely, we recommend that the NAIRR:
‍

1. provides infrastructure that enables API-based research on large pre-trained models and guards against misuse;
2. allows researchers to use their NAIRR compute budget to do research on models accessed through an API; and
3. explores ways to incentivize technology companies, academic researchers, and government agencies to provide structured access to large pre-trained models through the API. 

‍

Signs of a trend

We predict that an increasing portion of important AI research and development will make use of large pre-trained models that are accessible only through APIs. In this paradigm, pre-trained models would play a central role in the AI ecosystem. A large portion of SOTA models would be developed by fine-tuning(2) and otherwise adapting these models to particular tasks. Commercial considerations and misuse concerns would also frequently prevent developers from granting others access to their pre-trained models, except through APIs. Though we are still far from being in this paradigm, there are some early indications of a trend.

Particularly in the domain of natural language processing, academic research is beginning to build upon pre-trained models such as T5, BERT, and GPT-3. At one of the leading natural language processing conferences in 2021, EMNLP, a number of papers were published that investigated and evaluated existing pre-trained models. Some of the most relevant models are accessible only or primarily through APIs. The OpenAI API for GPT-3, announced in June 2020, has been used in dozens of research papers, for example investigating the model’s bias, its capabilities, and its potential to accelerate AI research by automating data annotation. Furthermore, Hugging Face’s API interface has been used to investigate COVID-19 misinformation and to design a Turing test benchmark for language models.

At the same time, in the commercial domain, applications of AI increasingly rely on pre-trained models that are accessed through APIs. Amazon Web Services, Microsoft Azure, Google Cloud, and other cloud providers now offer their customers access to pre-trained AI systems for visual recognition, natural language processing (NLP), speech-to-text, and more. OpenAI reported that its API for its pre-trained language model GPT-3 generated an average of 4.5 billion words per day as of March 2021, primarily for commercial applications.

Five underlying factors in the AI field explain why we might expect a trend towards academic research that relies on large pre-trained models that are only accessible through APIs:

‍

• Training SOTA models from scratch requires large amounts of compute, precluding access for actors with smaller budgets. For instance, PaLM – a new SOTA NLP model from Google Research – is estimated to have cost between $9 and $17M to train. The training compute cost of developing the next SOTA NLP model will likely be even greater.
• In comparison, conducting research on pre-trained models typically requires small compute budgets. For instance, we estimate that a recent paper investigating anti-muslim bias in GPT-3 likely required less than $100 of compute(3). Developing new SOTA models by fine-tuning or otherwise adapting “foundation models” will also typically be dramatically cheaper than developing these models from scratch.
• The developers of large pre-trained models are likely to have strong incentives not to distribute these models to others, as this would make it both more difficult to monetize the models and more difficult to prevent misuse.
• Given the right infrastructure, it is significantly easier for researchers to use a pre-trained model that is accessed through an API than it is for them to implement the model themselves. Implementing large models, even for research purposes, can require significant engineering talent, expertise, and computing infrastructure. Academics and students often lack these resources.
• Academics may increasingly aim their research at understanding and scrutinizing models, as this is important scientific work and plays to academia’s comparative advantage. 

‍

We discuss these factors in detail below, in an appendix to this post.

‍

How the NAIRR could provide access to pre-trained models

We offer a sketch of how the NAIRR could provide access to pre-trained models in addition to data and compute, illustrated in the figure below. First, it would create a platform for hosting and accessing pre-trained models via an API. The platform should be flexible enough to allow researchers to run a wide range of experiments on a range of models. It should be capable of supporting fine-tuning, interpretability research, and easy comparison of outputs from multiple models. The API should allow researchers to interface with both models hosted by the NAIRR itself and models hosted by other developers, who may often prefer to retain greater control over their models. 

Second, researchers would be allowed to use their NAIRR compute budgets to run inferences on the models. We recommend that researchers be allowed to use their budgets for this purpose even if the model is hosted by an organization other than the NAIRR. 

The biggest challenge will likely be securing access to pre-trained models from developers across industry, academia, and government. In some cases, developers might be motivated to provide access by a desire to contribute to scientific progress, the prospect of external actors finding issues and ways to improve the model, or a belief that it might improve the organization’s reputation. The NAIRR could also create an expectation that models trained using NAIRR compute should be accessible through the platform. Access to particularly high-stakes government models in need of outside scrutiny could also potentially be mandated. Additionally, the NAIRR could consider incentivizing government agencies to provide API access to some of their more impactful models in exchange for access to compute resources or data (similar to a Stanford HAI proposal regarding data access).

Encouraging private actors to make their models accessible through the platform may be especially difficult. In some cases, companies may provide model access as a means to build trust with their consumers. They may recognize that the public will be far more trusting of claims concerning the safety, fairness, or positive impacts of their AI systems if these claims are vetted by outside researchers. For example, Facebook and Twitter have recently created APIs that allow outside researchers to scrutinize company data in a privacy-preserving manner. Further, the NAIRR could consider offering compensation to developers for making their models available via the API. Developers may also be particularly concerned about risks to intellectual property, something that can be assuaged by the NAIRR upholding high cybersecurity standards.

Crucially, the API should also be designed to thwart model misuse, while still ensuring easy access for research use. Multi-purpose models trained with NAIRR resources could be used maliciously, for instance by criminals, propagators of misinformation, or autocratic governments around the world. Large language models could, for example, significantly reduce the cost of large-scale misinformation campaigns. The NAIRR should take measures to avoid models trained with publicly funded compute being put to such uses. Misuse could be reduced by introducing a tiered access approach, as suggested in the Stanford HAI report for datasets hosted on the NAIRR. For instance, researchers might get easy access to most models but need to apply for access to models with high misuse potential. Further restrictions could then be placed on the queries or modifications that researchers are allowed to make to certain models. In addition, API usage should be monitored for suspicious activity (e.g. the generation of large amounts of political content). 

‍

Helping academic researchers share their models

An appropriately designed API could also solve a challenge the NAIRR will face as it provides compute and data for the training of large-scale models: academic researchers will likely want to share and build on models developed with NAIRR resources. At the same time, open-sourcing the models may come with the risk of misuse in some cases. By building an API and agreeing to host models itself, the NAIRR can address this problem: it can make it easy for researchers to share their models in a way that is responsive to misuse concerns. 

Academics are significantly more likely to voluntarily make their models available via the API than private developers of SOTA models with a profit motive. As such, the NAIRR could start by focusing on providing infrastructure for academic researchers to share their models with each other, thereby building a proof-of-concept, and later introducing additional measures to secure access to models produced in industry and across government.

‍

Conclusion

By building API infrastructure to support access to large pre-trained models, the NAIRR could produce a number of benefits. First, it could help academics to scrutinize and understand the most capable and socially impactful AI models. Second, it could cost-effectively grant researchers and students the ability to work on frontier models. Third, it could help researchers to share and build upon each other’s models while also avoiding risks of misuse. Concretely, we recommend that the NAIRR:

‍

1. provides infrastructure that enables API-based research on large pre-trained models and guards against misuse;
2. allows researchers to use their NAIRR compute budget to do research on models accessed through an API; and
3. explores ways to incentivize technology companies, academic researchers, and government agencies to provide structured access to large pre-trained models through the API. 

‍

Appendix: Five factors underlying our prediction that pre-trained models accessed via APIs will become increasingly central to academic AI research 

‍Training SOTA models requires large amounts of compute

Training a SOTA model often requires large amounts of computational resources. Since 2012, the computational resources for training SOTA models have been doubling every 5.7 months. 

The final training run of AlphaGo Zero in 2017 is estimated to have cost $35M(4). GPT-3, a SOTA NLP model developed in 2020 that is accessible via an API, has been estimated to have cost around $4.6M to train.2 Gopher – a recent frontier NLP model developed in 2021 by DeepMind – already doubled the compute requirements, costing around $9.2M.(5) PaLM, a new SOTA NLP model from Google Research, is estimated to have cost between $9 and $17M to train. The training compute cost of developing the next SOTA NLP model will likely be even greater.

‍

Research on pre-trained models requires small compute budgets

Second, research on pre-trained models is much less compute-intensive in comparison. For example, if one would have spent the computational resources required to train GPT-3 on inference rather than training, one could have produced up to 56 billion words — that’s 14-times the number of words of the English Wikipedia.(6) We estimate that a recent paper investigating anti-muslim bias in GPT-3 likely required less than $100 of compute.(7) Fine-tuning of pre-trained models also appears very low cost, at least in natural language processing. OpenAI charges $120 for fine-tuning GPT-3 on 1 million tokens, which is more than the company used in a recent paper to fine-tune GPT-3 to avoid toxic outputs.

Through access to pre-trained models, many more people can cheaply access high-end AI capabilities, including people from domains outside AI and AI researchers who lack access to large amounts of compute and teams of engineers. To illustrate the low cost, we estimate that every US AI PhD student could be provided with five times the compute required to produce a paper on biases in large language models for the cost of training one Gopher-sized model (around $9.5M).(8) 

‍

Relative to open-sourcing, model access via API reduces the chance of misuse and supports model monetization

Third, the ability to provide structured access may incentivize producers to make their models available via API rather than open-sourcing them. Using an API, developers can allow access to their models while curtailing misuse and enabling monetization. AI models can be used for ill, for instance through disinformation, surveillance, or hacking. Large language models can also reveal privacy-infringing information, not intended by their developers. By only providing access to the model via an API, a developer can put in place checks, tripwires, and other forms of monitoring and enforce terms of service to avoid the model’s inappropriate use. They can also introduce restrictions on the inputs that can be sent to the models and update these restrictions over time, for instance to close loopholes or address newly discovered forms of misuse.

Although open-source models typically provide researchers with a greater deal of flexibility than API access, this discrepancy can be reduced. Access via API does not need to be limited to only running inference on a given input; API functionality can go further. Fundamental tasks, such as fine-tuning the model, can and should be enabled to offer a wide variety of research. For example, OpenAI recently allowed the fine-tuning of GPT-3 via API access, letting users customize this language model to a desired task. Google’s Vision API and Cohere’s language model also offer customization via fine-tuning. In the future, more functionality could be introduced to ensure that research via API is only minimally restrictive. For instance, it is possible to allow external researchers to remotely analyze a model’s inner workings without giving away its weights.

Companies are also likely to increasingly offer access to their most powerful models via API, rather than by open-sourcing them, as doing so provides them with the opportunity to monetize their models. Examples include OpenAI, Cohere, Amazon Web Services (AWS), and Google Cloud which allow access to their models solely via an API, for a fee.

‍

Given the right infrastructure, it is significantly easier for researchers to use a pre-trained model that is accessed through an API than it is for them to implement the model themselves

Doing AI research and building products using pre-trained models accessed via API has some advantages compared to implementing the model oneself. Implementing large models, even for research purposes, can require significant engineering talent, expertise, and computing infrastructure. Academics and students often lack these resources and so might benefit from API access. 

It is becoming increasingly inefficient to do “full-stack” AI research. AI research is seeing an increasing division of labor between machine learning engineers and researchers, with the former group specializing in how to efficiently run and deploy large-scale models. This is a natural development: as a field matures, specialization tends to increase. For instance, AI researchers virtually always rely on software libraries developed by others, such as Tensorflow and Pytorch, rather than starting from scratch. Similarly, an increasing portion of tomorrow’s AI research could be done largely by building on top of pre-trained models created by others.

‍

Scrutinizing large-scale models may be increasingly important research and plays to the comparative advantage of academic researchers

Lastly, academic researchers may be increasingly drawn to research aimed at scrutinizing and understanding large-scale AI models, as this could constitute some of the most important and interesting research of the next decade and academics could be particularly well-suited to conduct it. 

As AI systems become more sophisticated and integrated into our economy, there’s a risk that these models become doubly opaque to society: Firstly, the workings of the models themselves may be opaque. Secondly, the model developer might not reveal what they know about its workings to the wider world. Such opacity undermines our ability to understand the impacts of these models and what can be done to improve their effects. As a result, research aimed at understanding and auditing large models could become increasingly valued and respected.

Academic researchers are also particularly well-suited to conducting this kind of research. Many are drawn to academia, rather than industry, because they are motivated by a desire for fundamental understanding (e.g. how and why AI systems work) and care relatively less about building systems that achieve impressive results. On average, researchers who decide to stay in academia (and forgo much higher salaries) are also more likely to be concerned about the profit incentives and possible negative social impacts of private labs. This suggests that academics could find scrutinizing private labs’ models appealing. On the other hand, in addition to access to large amounts of compute and the ability to implement large models, there are a number of factors that place private labs at an advantage with regard to developing large models. Private labs have the ability to offer higher salaries, vast datasets, the infrastructure necessary to deploy models in the real world, and strong financial incentives to develop models at the frontier, as these models can be integrated into billion-dollar products like search and news feeds. 

Some early examples are already beginning to emerge, which illustrate how this division of responsibilities could work in practice. For instance, Facebook and Twitter recently opened up APIs that give researchers and academics access to data of user interactions with their platforms in a safe, privacy-preserving environment.

The AI governance space needs more rigorous work on what influential actors (e.g. governments and AI labs) should do in the next few years to prepare the world for advanced AI.

We’re setting up a Policy Team at the Centre for the Governance of AI (GovAI) to help address this gap. The team will primarily focus on AI policy development from a long-run perspective. It will also spend some time on advising and advocating for recommendations, though we expect to lean heavily on other actors for that. Our work will be most relevant for the governments of the US, UK, and EU, as well as AI labs.

We plan to focus on a handful of bets at a time. Initially, we are likely to pursue:

• Compute governance: Is compute a particularly useful governance node for AI? If so, how can this tool be used to meet various AI governance goals? Potential goals for compute governance include monitoring capabilities, restricting access to capabilities, and identifying high-risk systems such that they can be put to significant scrutiny.
• Corporate governance: What kinds of corporate governance measures should frontier labs adopt? Questions include: What can we learn from other industries to improve risk management practices? How can the board of directors most effectively oversee management? How should ethics boards be designed?
• AI regulation: What present-day AI regulation would be most helpful for managing risks from advanced AI systems? Example questions include: Should foundation models be a regulatory target? What features of AI systems should be mandated by AI regulation? How can we help create more adaptive and expert regulatory ecosystems?

‍

We’ll try several approaches to AI policy development, such as:

• Back-chaining from desirable outcomes to concrete policy recommendations (e.g. how can we increase the chance there are effective international treaties on AI in the future?);
• Considering what should be done today to prepare for some particular event (e.g. the US government makes an Apollo Program-level investment in AI);
• Articulating and evaluating intermediate policy goals (e.g. “ensure the world’s most powerful AI models receive external scrutiny by experts without causing diffusion of capabilities”);
• Analyzing what can and should be done with specific governance levers (e.g. the three bets outlined above);
• Evaluating existing policy recommendations (e.g. increasing high-skilled immigration to the US and UK);
• Providing concrete advice to decision-makers (e.g. providing input on the design of the US National AI Research Resource).

‍

Over time, we plan to evaluate which bets and approaches are most fruitful and refine our focus accordingly.

The team currently consists of Jonas Schuett (specialisation: corporate governance), Lennart Heim (specialisation: compute governance), and myself (Markus Anderljung, team lead). We’ll also collaborate with the rest of GovAI and people at other organisations.

We’re looking to grow the team. We will be hiring Research Scholars on the Policy Track on a regular basis. We’re also planning to work with people in the GovAI 3-month Fellowship and are likely to open applications for Research Fellows in the near future (you can submit expressions of interest now). We’re happy for new staff to work out of Oxford (where most of GovAI is based), the Bay Area (where I am based), or remotely.

If you’d like to learn more, feel free to leave a comment below or reach out to me at

ma***************@go********.ai











.

This post, authored by Toby Shevlane, summarises the key claims and implications of his recent paper “Structured Access to AI Capabilities: An Emerging Paradigm for Safe AI Deployment.”

GovAI research blog posts represent the views of their authors, rather than the views of the organisation.

‍

Sharing powerful AI models

There is a trend within AI research towards building large models that have a broad range of capabilities. Labs building such models face a dilemma when deciding how to share them.

One option is to open-source the model. This means publishing it publicly and allowing anyone to download a copy. Open-sourcing models helps the research community to study them and helps users to access their beneficial applications. However, the open source option carries risks: large, multi-use models often have harmful uses too. Labs (especially industry labs) might also want to maintain a competitive edge by keeping their models private.

Therefore, many of the most capable models built in the past year have not been shared at all. Withholding models carries its own risks. If outside researchers cannot study a model, they cannot gain the deep understanding necessary to ensure its safety. In addition, the model’s potential beneficial applications are left on the table.

Structured access tries to get the best of both approaches. In a new paper, which will be published in the Oxford Handbook on AI Governance, I introduce “structured access” and explain its benefits.

‍

What is structured access?

Structured access is about allowing people to use and study an AI system, but only within a structure that prevents undesired information leaks and misuse. 

OpenAI’s GPT-3 model, which is capable of a wide range of natural language processing tasks, is a good example. Instead of allowing researchers to download their own copies of the model, OpenAI has allowed researchers to study copies that remain in its possession. Researchers can interact with GPT-3 through an “application programming interface” (API), submitting inputs and then seeing how the AI system responds. Moreover, subject to approval from OpenAI, companies can use the API to build GPT-3 into their software products.

This setup gives the AI developer much greater control over how people interact with the model. It is common for AI researchers to open-source a model and then have no way of knowing how people are using it, and no way of preventing risky or unethical applications.

With structured access, the developer can impose rules on how the model should be used. For example, OpenAI’s rules for GPT-3 state that the model cannot be used for certain applications, such as targeted political advertising. The AI developer can then enforce those rules, by monitoring how people are using the model and cutting off access to those who violate the rules.

The development of new technologies often runs into the “Collingridge dilemma”. The theory is that, by the time the impacts of a technology have become apparent, they are already irreversible. Structured access helps to fight against this. If the developer learns that their model is having serious negative impacts, they can withdraw access to the model or narrow its scope.

At the same time, structured access allows the research community to better understand the model – including its potential risks. There has been plenty of valuable research into GPT-3, relying simply on the API. For example, a recent paper analysed the “truthfulness” of the model’s outputs, testing GPT-3 on a new benchmark. Other research has explored GPT-3’s biases.

The hope is that we can accelerate the understanding of a model’s capabilities, limitations, and pathologies, before the proliferation of the model around the world has become irreversible.

‍

How can we go further?

Although there are existing examples of structured access to AI models, the new paradigm has not yet reached maturity. There are two dimensions along which structured access can be improved: (1) the depth of model access for external researchers, and (2) the broader governance framework.

GPT-3 has demonstrated how much researchers can accomplish with a simple input-output interface. OpenAI has also added deeper access to GPT-3 as time goes on. For example, instead of just getting GPT-3’s token predictions, users can now get embeddings too. Users can also modify the model by fine-tuning it on their own data. GPT-3 is becoming a very researchable artefact, even though it has not been open-sourced.

The AI community should go even further. An important question is: how much of a model’s internal functioning can we expose without allowing an attacker to steal the model? Reducing this tradeoff is an important area for research and policy. Is it possible, for example, to facilitate low-level interpretability research on a model, even without giving away its parameter values? Researchers could run remote analyses of the model, analogous to privacy-preserving analysis of health datasets. They submit their code and are sent back the results. Some people are already working on building the necessary infrastructure – see, for example, the work of OpenMined, a privacy-focussed research community.

Similarly, labs could offer not just the final model, but multiple model checkpoints corresponding to earlier stages in the training process. This would allow outside researchers to study how the model’s capabilities and behaviours evolved throughout training – as with, for example, DeepMind’s recent paper analysing the progression of AlphaZero. Finally, AI developers could give researchers special logins, which give them deeper model access than commercial users.

The other area for improvement is the broader governance framework. For example, with GPT-3, OpenAI makes its own decisions about what applications should be permitted. One option could be to delegate these decisions to a trusted and neutral third party. Eventually, government regulation could step in, making certain applications of AI off-limits. For models that are deployed at scale, governments could require that the model can be studied by outsiders.

Structured access is very complementary with other governance proposals, such as external audits, red teams, and bias bounties. In a promising new development, Twitter has launched a collaboration with OpenMined to allow its models (and datasets) to be audited by external groups in a structured way. This illustrates how structured access to AI models can provide a foundation for new forms of governance and accountability. 

‍

Industry and academia

I see structured access as part of a broader project to find the right relationship between AI academia and industry, when it comes to the development and study of large, multi-use models.

One possible arrangement is where certain academic research groups and industry research groups compete to build the most powerful models. Increasingly, this arrangement looks outdated. Academics do not have the same computational resources as industry researchers, and so are falling behind. Moreover, as the field matures, building stronger and stronger AI capabilities looks less like science and more like engineering.

Instead, industry labs should help academics to play to their strengths. There is still much science to be done, without academics needing to build large models themselves. Academics are well-placed, for example, to contribute to the growing model interpretability literature. As well as being scientific in nature, such work could be extremely safety-relevant and socially beneficial. As scientists, university-based researchers are well-placed to tackle the important challenge of understanding AI systems.

This benefits industry labs, who should try to cultivate thriving research ecosystems around their models. With the rise of very large, unwieldy models, no industry lab can, working alone, truly understand and address safety or bias issues that arise in them — or convince potential users that they can be trusted. These labs must work together with academia. Structured access is a scalable way of achieving this goal.

‍

Conclusion

This is an exciting time for AI governance. The AI community is moving beyond high-level principles and starting to actually implement new governance measures. I believe that structured access could be an important part of this broader effort to shift AI development onto a safer path. We are still in the early stages, and there is plenty of work to be done to work out exactly how structured access should be implemented.