Blog

GovAI’s Economics and AI Lead, Anton Korinek, recently released an open online course on the Economics of AI. The course introduces participants to cutting-edge research in the economics of transformative AI, including its implications for growth, labour markets, inequality, and AI control.

The course is free, supported by a grant from the Long-Term Future Fund. The structure involves six distinct modules, the first of which is accessible to anyone with a social science background, while the other modules are aimed at people with an economics background at the graduate or advanced undergraduate level. 

The course proceeds to analyse how modes of production and technological change are affected by AI, investigate how technological change drives aggregate economic growth, and examine how AI-driven technological change will impact labour markets and workers. The course closes by looking at several key questions for the future in AI governance: How can progress in AI be steered in a direction that benefits humanity, and what lessons does economics offer for how humans can control highly intelligent AI algorithms?

Joseph Stiglitz is University Professor at Columbia University. He is also the co-chair of the High-Level Expert Group on the Measurement of Economic Performance and Social Progress at the OECD, and the Chief Economist of the Roosevelt Institute. A recipient of the Nobel Memorial Prize in Economic Sciences (2001) and the John Bates Clark Medal (1979), he is a former senior vice president and chief economist of the World Bank and a former member and chairman of the US President’s Council of Economic Advisers. Known for his pioneering work on asymmetric information, Stiglitz’s research focuses on income distribution, risk, corporate governance, public policy, macroeconomics and globalization.

Anton Korinek is an Associate Professor at the University of Virginia, Department of Economics and Darden School of Business as well as a Research Associate at the NBER, a Research Fellow at the CEPR and a Research Affiliate at the AI Governance Research Group. His areas of expertise include macroeconomics, international finance, and inequality. His most recent research investigates the effects of progress in automation and artificial intelligence for macroeconomic dynamics and inequality.

‍

You can watch a recording of the event here or read the transcript below:

‍

Joslyn Barnhart [0:00]

Welcome, I’m Joslyn Barnhart, a Visiting Senior Research Fellow at the Centre for the Governance of AI (GovAI), which is organizing this series. We are part of the Future of Humanity Institute at the University of Oxford. We research the opportunities and challenges brought by advances in AI and related technologies, so as to advise policy to maximise the benefits and minimise the risks from advanced AI. Governance, this key term in our name, refers [not only] descriptively to the ways that decisions are made about the development and deployment of AI, but also to the normative aspiration that those decisions emerge from institutions that are effective, equitable, and legitimate. If you want to learn more about our work, you can go to http://www.governance.ai.

I’m delighted today to introduce our conversation featuring Joseph Stiglitz in discussion with Anton Korinek. Professor Joseph Stiglitz is University Professor at Columbia University. He’s also the co-chair of the high-level expert group on the measurement of economic performance and social progress at the OECD, and the chief economist of the Roosevelt Institute, a recipient of the Nobel Memorial Prize in Economic Sciences in 2001, and the John Bates Clark Medal in 1979. He is a former senior vice president and chief economist of the World Bank, and a former member and chairman of the US President’s Council of Economic Advisers, known for his pioneering work on asymmetric information. Professor Stiglitz’s research focuses on income distribution, risk, corporate governance, public policy, macroeconomics and globalisation.

Professor Korinek is an associate professor at the University of Virginia, Department of Economics and Darden School of Business, as well as a research associate at the NBER, research fellow at the CEPR, and a research affiliate at the Centre for the Governance of AI. His areas of expertise include macroeconomics, international finance, and inequality. His most recent research investigates the effects of progress in automation and artificial intelligence [on] macroeconomic dynamics and inequality.

Over the next decades, AI will dramatically change the economic landscape and may also magnify inequality both within and across countries. Anton and Joe will be discussing the relationship between technology and inequality, the potential impact of AI on the global economy, and the economic policy and governance challenges that may arise in an age of transformative AI. We will aim for a conversational format between Professor Korinek and Professor Stiglitz. I also want to encourage all audience members to type your questions using the box below. We can’t promise that [your questions] will be answered but we will see them and try to integrate them into the conversation. With that, Anton and Joe, we look forward to learning from you and the floor is yours.

Anton Korinek [3:09]

Thank you so much, Joslyn, for the kind introduction. Inequality has been growing for decades now and has been further exacerbated by the K-shaped recovery from COVID-19. In some ways, this has catapulted the question of how we can engineer a fairer economy and society to the top of the policy agenda all around the world. As Joslyn has emphasised, what is of particular concern for us at the Centre for the Governance of AI is that modern technologies, and to a growing extent artificial intelligence, are often said to play a central role in increasing inequality. There are concerns that future advances in AI may in fact further turbo-charge inequality.

I’m extremely pleased and honoured that Joe Stiglitz is joining us for today’s GovAI webinar to discuss AI and inequality with us. Joe has made some of the most pathbreaking contributions to economics in the 20th century. As we have already heard, his work was recognised by the Nobel Prize in Economics in 2001. I should say that he has also been the formative intellectual force behind my education as an economist. What I have always really admired in Joe — and I still admire every time we interact — is that he combines a razor-sharp intellect with a big heart, and that he is always optimistic about the ability of ideas to improve the world.

We will start this webinar with a broader conversation on emerging technologies and inequality. Over the course of the webinar, we will move more and more towards AI and ultimately the potential for transformative AI to reshape our economy and our society.

Let me welcome you again, Joe. Let’s start with the following question: Can you explain what we mean by inequality? What are the dimensions of inequality that we should be most concerned about?

Joseph Stiglitz [5:33]

[Inequality is the] disparities in the circumstances of individuals. One is always going to have some disparities, but not of the magnitude and not of the multiplicity of dimensions [that we see today]. When economists talk about inequality, they first talk about inequalities of income, wealth, labour income, and other sources of income. [These inequalities] have grown enormously over the last 40 years. In the mid-1950s, Simon Kuznets, a great economist who got a Nobel Prize, had thought that in the early stages of development, inequality would increase, but then [in later stages of development], inequality would decrease. And the historical record was not inconsistent with that [model] at the time he was writing. But then beginning in the mid-1970s and the beginning of the 1980s, [inequality] started to soar. [Inequality] has continued to increase until today, and the pandemic’s K-shaped recovery has exposed and exacerbated this inequality.

Now beyond that, there are many other dimensions of inequality, like access to health[care], especially in countries like the United States [without] a national health service. As a result, the US has the largest disparities in health among advanced countries, and even before 2019, had an average decline in life expectancy and overall health standards. There are disparities in access to justice and other dimensions that make for a decent life. One of the concerns that has been highlighted in the last year is the extent to which those disparities are associated with race and gender. That has given rise to the huge [movement], “Black Lives Matter.” [This movement] has reminded us of things that we knew, but were not always conscious of, [including] the tremendous inequalities across different groups in our society.

Anton Korinek [8:23]

Thank you. Can you tell us about what motivated you personally to dedicate so much of your work to inequality in recent decades? I’ve heard you speak of your experience growing up in Gary, Indiana. I have heard a lot about your role as a policymaker, as a chair of the President’s Council of Economic Advisors, and as a chief economist of the World Bank in the 1990s. How has all of this shaped your thinking on inequality?

Joseph Stiglitz [8:55]

I grew up, as you said, in Gary, Indiana, which was emblematic of industrial America, though of course I didn’t realise that as I was growing up. [In Gary], I looked at my surroundings, and I saw enormous inequalities in income and across races; [I saw] discrimination. That was really hard to reconcile with what I was being taught about the American Dream: that everybody has the same opportunity and that all people are created equal. All those things that we were told about America, which I believed on one level, seemed inconsistent with what [I saw].

That was why I had planned [to study economics]. Maybe it seems strange, but I had wanted to be a theoretical physicist. [But with all] the problems that I had seen growing up around inequality, suddenly, at the end of my third year in college, I wanted to devote my life to understanding and doing something about inequality. I entered economics with that very much on my mind, and I wrote my thesis on inequality. But life takes its turn, [so I spent] much of the time [from then until] about 10 years ago on issues of imperfect information and imperfect markets. This was related, in some sense, to inequalities because the inequalities in access to information were very much at the core of some of the inequalities in our society. [For example,] inequalities in education played a very important role in the perpetuation of inequalities. So, the two were not [part of] a totally disparate agenda.

From the very beginning, I also spent a lot of time thinking about development, which interacted with my other work on theoretical economics. It may seem strange, but I did go to Africa in 1969: [I went to] Kenya not long after it got its independence. I’m almost proud to say that some people in Africa claim me to be the first African Nobel Prize winner: [Africa] had such an important role in shaping my own research. That strand of thinking about inequality between the developing countries and the developed countries was also very important [to my understanding of inequality].

Finally, to answer your question, when I was in the Clinton administration, we had a lot of, you might say, fights about inequality. Everybody was concerned about inequality, but some were more concerned than others. Some wanted to put it at the top of the agenda, and [others] said, “We should worry about it, but we don’t have the money to deal with it.” It was a question of prioritisation. On one side Bob Reich, who was the Secretary of Labour, and I were very much concerned about this inequality. We were concerned about corporate welfare: giving benefits to rich corporations meant that we had less money to help those who really needed it. Our war against corporate welfare actually led to huge internal conflicts between us and some of the more corporatist or financial members of the Clinton team.

Anton Korinek [13:53]

That brings us perhaps directly to a more philosophical question. What would you say is the ethical case for [being concerned with] inequality? In particular, why should we care about inequality in itself and not just about absolute levels of income and wealth?

Joseph Stiglitz [14:17]

The latter [question] you can answer more easily from an economic point of view. There is now a considerable body of theory and empirical evidence that societies that are marked by large disparities and large inequalities behave differently and overall perform more poorly than societies with fewer inequalities. Your own work has highlighted the term “macroeconomic externalities,” which [describes when a system’s functioning] is adversely affected by the presence of inequality. An example, for instance, is that when there are a lot of inequalities, those at the bottom engage in “keeping up with the Joneses,” as we say, and that leads them to be more in debt. That higher level of debt introduces a kind of financial fragility to the economy which makes it more prone to economic downturns.

There are a number of other channels through which economic inequality adversely affects macroeconomic performance. The argument can be made that even those at the top can be worse off if there’s too much inequality. I reflected this view in my book, the Price of Inequality, where I said that our society and our economy pay a high price for inequality. This view has moved into the mainstream, which is why the IMF has put concerns about inequality [at] the fore of their agenda. And as Strauss-Kahn, who was the Managing Director of the IMF at the time said, [inequality] is an issue of concern to the IMF because the IMF is concerned about macroeconomic stability and growth, and the evidence is overwhelming that [inequality] does affect macroeconomic performance and growth.

[There is a] moral issue which economists are perhaps less well-qualified to talk about rigorously. Economists and philosophers have used utilitarian models and equality-preferring social welfare functions. [These models build on] a whole literature of [philosophy], of which Rawls is an example. [Rawls] provides a philosophical basis [for] why, behind the veil of ignorance, you would prefer to be born into a society with greater equality.

Anton Korinek [17:40]

So that means there is both a moral and an economic efficiency reason to engage in measures that mitigate inequality. Now, this brings us to a broader debate: what are the drivers of inequality? Is inequality driven by technology, or by institutions [and] policies, broadly defined? There is a neoclassical caricature of the free market as the natural state of the world. In this caricatured description of the world, everything is driven by technology, and technology may naturally give rise to inequality, and everything we would do [to mitigate inequality] would be bad for economic efficiency. Can you explain the interplay of technology and institutions more broadly and tell us what is wrong with this caricature?

Joseph Stiglitz [18:46]

[To put it in another way:] is inequality the result of the laws of nature, or the laws of man? And I’m very much of the view that [inequality] is a result, overwhelmingly, of the laws of men and our institutions. One way of thinking about this, which I think [provides] compelling evidence for my perspective, is that the laws of nature are universal: globalization and [technological advancement] apply to every country. Yet, in different countries, we see markedly different levels of inequality in market incomes and even more so in after-tax and transfer incomes.

It is clear that countries that should be relatively similar have been shaped in different ways by the laws. What are some of those laws? Well, some of them are pretty obvious. If you have labour laws that undermine the ability of workers to engage in collective bargaining, workers are going to get short stacked; they’re not going to be treated well. You see that in the United States: one of the main [reasons for] the weakening of the share of labour in the United States is, I believe, the weakening of labour laws and the power to unionise.

At the other extreme, more corporate market power [allows companies to raise prices, which] is equivalent to lowering wages, because [people] care about what [they] can purchase. The proceeds of [higher prices] go to those who own the monopolies, who are disproportionately those at the top. During Covid-19 we saw Jeff Bezos do a fantastic job of making billions of dollars while the bottom 40% of Americans suffered a great deal. The laws governing antitrust competition policy are critical.

But actually, a host of other details and institutional arrangements that we sometimes don’t notice [drive inequality]. United States [policy] illustrates that we do things so much worse than other countries. Bankruptcy laws, which deal with what happens if a debtor can’t pay [back] all of the money, give first priority [to banks]. In the United States, the first claimant is the banks, who sell derivatives – those risky products that led to the financial crisis of 2008. On the other hand, if you borrow money, to get ahead in life or to finance education, you cannot discharge your debt. So, students are at the bottom, and banks are at the top. So that’s another example [of how laws drive inequality].

Corporate governance laws, that give the CEOs enormous scope for setting their salaries in any way they want, result, in the United States, in the CEOs getting 300 times the compensation of average workers. That’s another example [of how laws create inequality].

But there are a whole host of things that we often don’t even think of as institutions, but [they] really are. When we make public investments in infrastructure, do we provide for public transportation systems, which are very important for poor people? When we have public transportation systems, do we connect poor people with jobs? In Washington D.C. they made a deliberate effort not to do that. When we’re running monetary policy, are we focusing on making sure that there’s [as] close to full employment as possible, which increases workers bargaining power? Or do we focus on inflation, which might be bad for bondholders?

Monetary policy, in the aftermath of the 2008 crisis, led to unprecedented wealth inequality, but didn’t succeed very well in creating jobs. 91% of the gains that occurred in the first three years of that recovery went to the top 1% in the United States. So, [inequality stems from] an amalgam of an enormous number of decisions.

Now, even when [considering] the issue of technology, we forget that [it is] man-made to a large extent — [it is] not like the laws of quantum mechanics! Technology [itself], and where we direct our attention [within technology], is man-made, and the extent to which we make access to technology available to all is our decision. Whether we steer technology to save the planet, or to save unskilled jobs, we can determine whether we’re going to have a high level of unemployment of low-skilled people or whether we’re going to have a healthier planet. [We witnessed] fantastic success in quickly developing COVID-19 vaccines. But now the big debate is, should those vaccines be available only to rich countries? Or should we waive the intellectual property rights in order to allow poor countries to produce these vaccines? That’s an issue being discussed right now at the WTO. Unfortunately, although a hundred countries want a waiver, the US and a few European countries say “no”. We put the profits of our drug companies over [peoples’] lives, not only over [lives] in developing countries, but possibly over the lives of people in our own country. As long as the disease rages [in developing countries], a mutation may come that is vaccine resistant, and our own lives are at risk. It’s very clear that this is a battle between institutions, and that right now, unfortunately, drug companies are winning.

Anton Korinek [26:04]

It’s a battle of institutions within the realm of a new technology.

If we now turn to another new technology, AI, you hear a lot of concern about AI increasing inequality. What are the potential channels that you see that we should be concerned about? To what extent could AI be different from other new technologies when it comes to [AI’s] impact on inequality?

Joseph Stiglitz [26:36]

AI is often lumped together with other kinds of innovations. People look historically, and they say “Look, innovations are always going to be disturbing, but over the long run, ordinary people gain.” [For example,] the makers of buggy whips lost out when automobiles came along, but the number of new jobs created in auto repair far exceeded the old jobs, and overall, workers were better off. In fact, [automobiles] created the wonderful middle class era of the mid-20th century.

I think this time may be different. There’s every reason to believe that it is different. First, these new technologies are labour replacing and labour saving, rather than increasing the productivity of labour. And so [these technologies are] substituting for labour, which drives down wages. There’s no a priori theory that says that an innovation [must] be of one form or the other. Historically, [innovations] were labour augmenting and labour enhancing; [historically, innovations] were intelligence-assisting innovations, rather than labour-replacing. But the evidence now is that [new innovations] may be more labour replacing. Secondly, the new technologies have a winner-take-all characteristic associated with them: [these new technologies] have augmented the potential of monopoly power. Both characteristics mean there will be a less competitive market and greater inequality resulting from this increased market power, and almost everybody may lose.

In the case of developing countries, the problems are even more severe for two reasons. The first [reason] is that the strategy that has worked so well to close the gap between developing and developed countries, which was manufacturing export-led growth, may be coming to an end. Globally, employment in manufacturing is declining. Even if all the jobs in manufacturing shifted, say, from China to Africa, [this shift] would [hardly] increase the labour force in Africa. I and some others have been trying to understand: why was manufacturing export-led growth so successful? And what [strategies] can African countries [employ] today if [manufacturing export-led growth] doesn’t work? Are there other strategies that will [be effective]? The conclusion is that there are other things that work, but they’re going to be much more difficult [to implement]. And there won’t likely be the kind of success that East Asia had beginning 50 years ago.

The second point [concerns] inequalities that occur within our country [as a result of AI]. [For example,] when Jeff Bezos [becomes] richer or Bill Gates [becomes] richer, we always have the potential to tax these gainers and redistribute some of their gains to the losers. The result, [which] you and I wrote about in one of our papers, shows that in a wide class of cases we can make sure that everybody could be better off [via redistributive taxation]. While [implementation is] a matter of politics, at least in principle, everybody could be made better off. However, [AI] innovations across countries [drive down] the value of unskilled labour and certain natural resources, which are the main assets of many developing countries. [Therefore, developing countries are] going to be worse off. Our international arrangements for redistribution are [very limited]. In fact, our trade agreements, our tax provisions, and our international [arrangements] work to the disadvantage of developing countries. We don’t have the instruments to engage in redistribution, and the current instruments actually disfavour developing countries.

Anton Korinek [32:44]

Let me turn to a longer-term question now. Many technologists predict that AI will have the potential to be really transformative if it reaches the ability to perform substantially everything that human workers can do. This [degree of capacity] is sometimes labelled as “transformative AI,” though people have also [described] closely-related concepts like Artificial General Intelligence and human-level machine intelligence. There are quite a few AI experts who predict that such transformative advances in AI may happen within the next few decades. This could lead to a revolution that is of similar magnitude to or greater magnitude than the agrarian or industrial revolution, which could make all human labour redundant. This would make human labour, in economic speak, a “dominated technology.”

[When we consider inequality,] the dilemma is that in our present world labour is the main source of income. Are you willing to speculate, as a social scientist, and not as a technologist, [about] the likelihood and timeframe of transformative AI happening? What do you see as the main reasons why it may not be happening soon? [Alternatively,] what would be the main arguments in favour of transformative AI happening soon? And how should we think about the potential impacts of transformative AI, from your perspective?

Joseph Stiglitz [34:36]

There is a famous quip by Yogi Berra, who is viewed as one of the great thinkers in America. I’m not sure everybody in the UK knows about him. He was a famous baseball player who had simple perspectives on life and one of them was “forecasting is really difficult, especially about the future.”

The point is that we don’t know. But we certainly could contemplate this happening, and we ought to think about that possibility. So as social scientists, we ought to be thinking about all the possible contingencies, but obviously devote more of our work to those [scenarios] that are going to be most stressful for our society. Now, you don’t think that people should train to be a doctor to deal just with colds. You want your doctor to be able to respond to serious maladies.  I don’t want to call [transformative AI] a malady – it could be a great thing. But it would certainly be a transformative moment that would put very large stresses on our economic, social [and] political system.

The important point is that […] these advances in technologies make our society as a whole wealthier. These [advances] move out what we could do, and in principle, everyone could be made better off. So the question is: can we undertake the social, economic, [and] political arrangements to ensure that everyone, or at least a vast majority, will be made better off [by advances in AI]? When we engage in this sort of speculative reasoning, one could also imagine [a world in which] a few people [are] controlling these technologies, and that our society [may be] entering into a new era of unprecedented inequality – with a few people having all the wealth, and everybody else just struggling to get along and [effectively] becoming serfs. This would be a new kind of serfdom, a 21st century or 22nd century serfdom that is different from that of 13th and 12th century [serfdom]. For the vast majority [of people, this serfdom would not be] a good thing.

Anton Korinek [37:59]

For the sake of argument, let’s take it as a given that this type of transformative AI will arrive by, say, 2100. What would you expect to be the effects of [transformative AI] on economic growth, on the labour share, and in particular, on inequality? What would be the [impact] on inequality in non-pecuniary, non-monetary terms?

Joseph Stiglitz [38:36]

The effect [of transformative AI] on inequality, income, wealth, and monetary aspects will depend critically on the institutions that we described earlier in two key [ways]. If we move beyond hoarding knowledge via patents and other means, and gain wide[spread] and meaningful access to intellectual property, then competition can lower prices and the benefits of [transformative AI] can be widely shared.

This was what we experienced in the 19th and 20th century [during the Industrial Revolution]. Eventually, when competition got ideas out into the marketplace, profits eroded. While the earlier years of the [Industrial] Revolution were not great for ordinary workers, eventually, [ordinary workers] did benefit and competition [served to ensure] that the benefit of the technological advances were widely shared. There is a concern about whether our legal and institutional framework can ensure that that will happen with artificial intelligence. That’s one aspect of our institutional structure.

Even if we fail to do the right thing in that area, we have another set of instruments, which are redistributive taxes. We could tax multibillionaires like Jeff Bezos or Bill Gates. From the point of view of incentives, most economists would agree that if multibillionaires were rewarded with 16 billion dollars, rather than 160 billion [dollars], they would probably still work hard. They probably wouldn’t say “I’m going to take my marbles and not play with you anymore.” They are creative people who want to be at the top, but you can be at the top with 16 [billion dollars], rather than 160 billion [dollars]. You take that [extra tax revenue] and use it [for] more shared prosperity. Then, obviously, the nature of our society would be markedly different.

If we think more broadly, right now, President Biden is talking a lot about the “caring economy.” Jobs are being created in education, health, care for the aged, [and] care for the sick. Wages in those jobs are relatively low, because of the legacy of discrimination against women and people of colour who have [worked] in these areas. Our society has been willing to take advantage of that history of discrimination and pay [these workers] low wages. Now, we might say, why do that? Why not let the wages reflect our value of how important it is to care for these parts of our society? [We can] tax the very top, and use that [tax revenue] to create new jobs that are decently paid, [which would create] a very different outcome [for the economy]. I think, optimistically, this new era could create shared prosperity. There would still be some inequality, but not the nightmare scenario of the new serfdom that I talked about before.

Anton Korinek [42:49]

Let’s turn to economic policy. You have already foreshadowed a number of interesting points on this theme. But let’s talk about economic policy to combat inequality more generally. People often refer to redistribution and pre-distribution as the main categories of economic policy to combat inequality. Can you explain what these two [policy categories] mean? What are the main instruments of redistribution and of pre-distribution? And how do [these policies] relate to our discussion on inequality?

Joseph Stiglitz [43:37]

Pre-distribution [looks at] the factors that determine the distribution of market income. If we create a more equal distribution of market income, then we have less burden on redistribution to create a fair society. There are two factors that go into the market distribution of income. [The first factor] is the distribution, or the ownership, of assets. [The second factor] is how much you pay each of those assets. For instance, if you have a lot of market power, and weak labour power, you [end] up with capital getting a high return relative to workers and [high] monopoly profits relative to workers’ [incomes] — that’s an example of the exercise of market power leading to greater inequality. The progressive agenda in the United States emphasises increasing the power of unions and curbing the power of big tech giants to create factor prices that are conducive to more market equality.

We can [also consider] the ownership of two types of assets: human capital and financial capital. The general issue here is: how do we prevent the intergenerational transmission of advantage and disadvantage? Throughout the ages, there have always been parents who want to help their children, which is not an issue. [Rather, the issue is] the magnitude of that [helping]. In the United States, for instance, we have an education system which is locally-based. We have more and more economic segregation, which means that rich people live with rich [people] and poor [people] with poor [people.] If schools in [rich] neighbourhoods give kids a really good education and conversely [in poor neighbourhoods, then even] public education perpetuates inequality.

The most important provision in the intergenerational transmission of financial wealth is inheritance tax and capital taxation. Under Trump, [Congress] eviscerated the inheritance taxes. So 1716328364 the question is how to [reinstate these taxes] to create a more equal market distribution, called pre-distribution.

Anton Korinek [47:34]

[You began to address] taxation in the context of estate taxation. For the non-economists in the room, I should emphasise that among the many contributions that Joe has made to economics is a 1976 textbook with Tony Atkinson that is frequently referred to as the “Bible of Public Finance” which lays out the basic theory of taxation and still underlies basically all theoretical economic work on taxes.

In recent decades, the main focus of this debate has been on taxing labour versus capital. A lot of economists argue that we should not tax capital, because it’s self-defeating: [taxation of capital] will just discourage the accumulation of capital and ultimately hurt workers. My question to you is: do you agree? [If not,] what is wrong with this standard argument?

Joseph Stiglitz [48:43]

It is an argument that one has to take seriously: that attacks on capital could lead to less capital accumulation which would in turn lead to lower wages, and even if the proceeds of the tax were redistributed to workers, workers could be worse off. You can write down theoretical models in which that happens. The problem is that this is not the world we live in. In fact, [there are] other instruments at [our] disposal. For instance, as the government taxes [private] capital, [the government] can invest in public capital, education, and infrastructure. [These investments lead to an increase in] wages. Workers can be doubly benefited: not only [do workers benefit] from direct distribution, but [they also benefit from a greater] equality of market income caused by capital allocation to education and infrastructure.

Many earlier theories were predicated on the assumption that we were able to tax away all rents and all pure profit. We know that’s not true: the corporate profit tax rate is now 21% in the United States, and the amount of wealth that the people at the top are accumulating [provides evidence that] we are not taxing away all pure profits. Taxing away [these pure profits] would not lead to less capital accumulation, [but instead] could lead to more capital accumulation.

[Let’s] look broadly at the nature of capitalism in the late 20th and early 21st century. We used to talk about the financial sector intermediating, which meant [connecting] households and firms by bringing [households’] savings into corporations. [This process] helped savings and helped capital accumulation. [However,] evidence is that over the last 30 or 40 years, the financial sector has been disintermediating. The financial sector, [rather than] investing monopoly profits, has been redistributing [these profits] to the very wealthy, [to facilitate] the wealthy’s consumption or increase the value of their assets, [including their international assets], and their land. [Ultimately], this simple model [of financial intermediation] doesn’t describe [late] 20th and [early] 21st century capitalism.

Anton Korinek [52:17]

Should we think of AI as [the same kind of] capital described in theories of capital taxation in economics, or is AI somehow inherently different? Should we impose what Bill Gates calls a “robot tax” [on AI]?

Joseph Stiglitz [52:36]

That’s a really good question. [If we had had more time, I would have] have distinguished between intangible capital, called R&D, and [tangible capital, like] buildings and equipment. 21st century capital is mostly intangible capital, which is the result of investment in R&D. [Intangible capital] is more productive in many ways than buildings, and so in that sense, it is real capital, and is [well-described by the word] “intangible.” [Intangible capital is also the] result of investment: people make decisions to hire workers to think about [certain] issues, or individuals decide themselves to think about these issues, when [employers or individuals otherwise] could have done something else. [In this way, intangible capital] is capital: it requires resources, which could have been put to other uses, [and these alternative uses are foregone] for future-oriented returns.

The question is: is this [intangible] capital getting excess returns? Are there social consequences of those investments, that [the investors] don’t take into account? We call [these social consequences] externalities. People who invest in coal-fired power plants may make a lot of money, but [their investment] destroys the planet. If we don’t tax carbon, then society — rather than the investor — bears these costs. Gates’s robot tax is based on the same [concept]. If we replace workers, and [these workers] go on the unemployment roll, then we as a society bear the cost of [these workers’] unemployment. [Gates argues that] we ought to think about those costs, [though] how we balance the tax and appropriate its excess returns is another matter. Clearly, [the robot tax] is an example of steering innovation. You and I, [in our research,] have [also argued that we must] steer innovation to save the planet [rather than] create more unemployment.

Anton Korinek [55:32]

How would you recommend that we should reform our present system of taxation to be ready for not only [our present time in the] 21st century but also for a future in which human labour plays less of a role? How should we tax to make sure that we can still support an equitable society?

Joseph Stiglitz [56:02]

Let me first emphasise that not just taxation, but also investment, is important. [Much of the economy’s direction is determined by] the basic research decisions of the National Science Foundation and science foundations in other countries. [These decisions inform which] technologies are accessible to those in the private sector. Monetary policy [is also important]. We don’t think the central bank [affects] innovation, but it actually does. [At a] zero interest rate, the cost of capital is going to be low relative to the cost of labour, [which will] encourage investors to think about saving labour rather than saving capital. So monetary policy is partly to blame for distortions in the direction of innovation. The most important thing is to be sensitive to how every aspect of policy, including tax policy, shapes our innovative efforts and [directs where we] devote our research. Are we devoting our research to saving unskilled labour or to augmenting the power of labour? We talked before about intelligence-assisting innovations like microscopes and telescopes which make us more productive as human beings. We can replace labour, or we can make labour more productive. [While this distinction can be] hard to specify, it’s very clear that we have tools to think about these various forms of innovation.

Anton Korinek [58:16]

On the expenditure side, one policy solution that a lot of technologists are big fans of is a universal basic income. What is your perspective on a UBI: do you advocate it or do you believe there are other types of expenditure policy that are more desirable? Do you think [UBI] may be a good solution if we arrive at a far-future – or perhaps near-future – [scenario] in which labour is displaced?

Joseph Stiglitz [58:53]

I am quite against the UBI [being implemented] in the next 30 or 40 years. The reason is very simple: for the next 30 years, the major challenge of our society is the Green Transition, which will take a lot of resources and a lot of labour. Some people ask if we can afford it, and [I argue that] if we redirect our resources, labour, and capital [toward the Green Transition] then we can afford it. Ben Bernanke [describes] a surplus of capital and a savings glut. However, if [we look] at the challenges facing the world, [we understand Bernanke’s assertion] is nonsense. Our financial system isn’t [developing] the [solutions] our society needs [like] the Green Transition.

I also see deficiencies in infrastructure and in education in so many parts of the world. I see a huge need for investments over the next 30 to 40 years such that everybody who wants a job will be fully employed. It is our responsibility [to ensure] that everybody who wants a job should be able to get one. We must have policies to make sure that [workers] are decently paid. This should be our objective now.

[If] in the far-future [we don’t need] labour, we have the infrastructure that we need, we’ve made the Green Transition, and we have wonderful robots that produce other robots and all of the goods, food, and services that we need, then we will have to consider the UBI. We would [then] be engaged in a discussion of what makes life meaningful. While work has been part of that story of meaningfulness, there are ways of serving other people that don’t have to be monetised and can be very meaningful. While I’m willing to speculate about [this scenario,] it’s a long way off, and [is] well after my time here on this earth.

Anton Korinek [1:01:46]

Would you be willing to revise your timelines if progress in AI occurs faster than what we are currently anticipating?

Joseph Stiglitz [1:01:59]

I cannot see [a scenario where we] have excess labour and capital [over] the next 30 or 40 years, even if [AI] proceeds very rapidly, given the needs that we have in public investment and the Green Transition. We could have miracles, but I think if that happens, we could face that emergency of this unintended manna from heaven and we would step up to that emergency.

Anton Korinek [1:02:51]

We are already [nearing] the end of our time. Let me ask you one more question, and then I would like to bring in a few questions posed by the audience. My question is: what are the other dimensions of AI that matter for inequality, independent of purely economic [considerations]? What is your perspective [on these dimensions of inequality] and how we can combat them?

We’ve talked about meaning in life and meaningful work. If AI takes away work, we will have to find meaning in other places. In the shorter term, AI will take away routine jobs, which will mean that we as a society will be able to devote more labour to non-routine jobs. This should open up possibilities [for people to be] more creative. Many people [have] thought the flourishing of our society is based on creativity. It would be great for our society if we could devote more of our talents to doing non-routine, creative things.

The audience had a question about workplace surveillance, which is one element of [AI] that could potentially greatly reduce the well-being of workers. What are your thoughts on [workplace surveillance]?

Joseph Stiglitz [1:05:06]

I agree [that AI could reduce the well-being of workers]. There are many [adverse effects of AI] we haven’t talked about. We are in an early stage [of AI policy], and our inadequate regulation allows for a whole set of societal harms from AI. Surveillance is one [example of these harms]. Economists talk about corporations’ ability to acquire information in order to appropriate consumer surplus for themselves, or in other words, to engage in discriminatory pricing. Anybody who wants to buy an airline ticket knows what I’m talking about: firms are able to judge whether you really want to [fly] or not. Companies are using AI now to charge different prices for different people by judging how much [each consumer] wants a good. The basis for market efficiency is that everybody faces the same price. In a new world, where Amazon — or the internet — uses AI, everybody [faces] a different price. This discrimination is very invidious: it has a racial, gender, and vocational component.

Information targeting has other adverse [implications], like manipulation. [AI] can sense if somebody has a predilection to be a gambler and can encourage those worst attributes by getting [the person] to gamble. [AI] can target misinformation at somebody who is more likely to be anti-vax and give [them] the information to reinforce that [belief]. [AI] has already been used for political manipulation, and political manipulation is really important because [it impacts] institutions. The institutions — the rules of the game — are set by a political process, so if you can manipulate that political process, you can manipulate our whole economic system. In the absence of guardrails, good rules, and regulations, AI can be extraordinarily dangerous for our society.

Anton Korinek [1:08:25]

That relates closely to another question from the audience: do you think there is a self-correcting force within democracy against high inequality and in particular against the inequality that AI may lead to?

Joseph Stiglitz [1:08:47]

I wish I felt convinced that there were a self-correcting force. [Instead], I see a force that [works] in the [opposite] direction. This [perception] may be [informed] by my experience as an American: [in the US], a high level of inequality [causes] distortions and [gives] money a role in the political system. This has changed the rules in the political and economic system. Money’s [increasing] power in both the political system and the economic system has reinforced the creation of that kind of plutocracy that I talked about [earlier].

[The changes] we’ve seen in the last few years in the United States are shocking, but in some ways are what I predicted in my 2010 book The Price of Inequality. The Republican Party has openly said, “We don’t believe in democracy. We want to suppress voters and their right to vote. [We want to] make it more difficult for them to vote.” [They’ve said this without] any evidence of voter fraud. It’s almost blatant voter suppression. In some sense, this [scenario] is what Nancy MacLean [described] in her book Democracy in Chains, though it has come faster [than she predicted].

I’ve become concerned that what many had hoped would be a self-correcting mechanism isn’t working. We hope we are at a moment when we can turn back the tide. As more and more Americans see the extremes of inequality, they will turn to vote before it’s too late, before they lose the right to vote. This will be a watershed moment in which we will go in a different direction. I feel we’re at the precipice, and while I’m willing to bet that we’re going to go the right way, I would give [this path] just over 50% odds.

Anton Korinek [1:11:37]

I think fortunately Joe and all his work on the topic is part of the self-correcting force.

The top question in terms of Q&A box votes is whether AI will be a driver for long run convergence or divergence in global inequalities. Do you believe that current laggards, or poor countries, will be able to catch up with the front runners more easily or less easily [because of AI]?

Joseph Stiglitz [1:12:12]

I’m afraid that we may be at the end of the era of convergence that we saw over the last 50 years. There was widespread convergence in China and India, and though some countries in Africa did not converge, we broadly saw a convergence [occurring]. I think [that now] there is a great risk of divergence: AI is going to decrease the value of unskilled labour and many natural resources, which are the main assets of poor countries. There will be [complexity]: oil countries will find that oil is not worth as much if we make the Green Transition. A few countries like Bolivia, that have large deposits of lithium, are going to be better off, but that will be more the exception than the rule. Access to [AI] technology may be more restricted. A larger fraction of the research is [occurring] inside corporations. The model of innovation [used to be] that universities were at the centre, and [innovators received a] patent with a disclosure, which means that the information was public and others built on that [information]. However, AI [innovation] so far has been within companies that have better hoarded information. [Companies can’t protect all information]: one non-obvious [path forward] is that [members of the public] could still access the underlying mathematical theorems that are in the public domain. While that’s an open possibility, I [still] worry that we will be seeing an era of divergence.

Anton Korinek [1:14:41]

Thank you so much, Joe for sharing your thoughts on AI and inequality with us. We are almost at time for our event. I am wondering if I may ask you a parting question that comes in two parts. What would be your message to, on the one hand, young AI engineers and, on the other hand, young social scientists and economists, who are beginning their careers and who are interested in contributing to make the world a better and more equitable place?

Joseph Stiglitz [1:15:30]

Engineers are working for companies, and a company consists of people. Talented people are the most important factors in the production of these companies. In the end, the voice of these workers is very important. We [must] conduct ourselves in ways that mitigate the extent to which we contribute to increases in inequality. There are many people, understandably, within Facebook and other tech giants that are using all their talents to ink the profits of, say, Facebook, regardless of the social consequences and regardless of whether it results in a genocide in Myanmar. These things do not just happen, but rather are a result of the decisions that people make.

To give another example, I often go to conferences out in Silicon Valley. When we discuss these issues, they say, “there is no way we can determine if our algorithms engage in discrimination.” [However], the evidence overwhelmingly is that we can. While the algorithms are always changing, taking in new information, and evolving, at any moment in time we can assess precisely whether [algorithms] are engaging in discrimination. Now, there are groups that are trying — at great cost — to see who is getting [certain] ads. You can create sampling spaces to see how [ads] are working.

I think it is nihilistic to say [that gauging discrimination is] beyond our ability and that we have created a monster out of our control. These companies’ workers need to take a sense of responsibility, because the companies’ actions are a consequence of their workers’ actions. When working for these companies, one has to take a moral position and a responsibility for what the companies do. One can’t just say, “Oh, that’s other people that are doing this.” One has to take some responsibility.

For social scientists, I think this is a very exciting time because AI and new technologies are changing our society. They may even be changing who we are as individuals. There is a lot of discussion about what [new technologies] are doing to attention span and how we spend our time. [These technologies] have profound effects on the way that individuals interact with each other.

Of course, social science is about society and how we interact with each other. [It is about] how we act as individuals. [It is about] market power [and] how we curb that market power. The basic business model of many tech giants [relies on] information about individuals. Policy [determines] what we allow those corporations to do with our [personal] information and whether [these corporations] can store [our information] and use it for other purposes. It is clear that AI has opened up a whole new set of policy issues that we had not even begun to think about 20 years ago. My Nobel Prize was in the economics of information, but when I did my work, I had not thought about the issue of disinformation and misinformation. [At the time], we thought we had laws dealing with [misinformation], which [are called] fraud laws and libel laws. We put [misinformation] aside because we thought it was not a problem. Today, [misinformation] is a problem. I mention that because we are going to have to deal with a whole new set of problems that AI is presenting to our society.

Anton Korinek [1:21:46]

Thank you, Joe. Thank you for this really inspiring call to action. Let me invite everybody to give a round of virtual applause. Have a good rest of the day.

‍

Molly Roberts is an Associate Professor in the Department of Political Science and the Halıcıoğlu Data Science Institute at the University of California, San Diego. She co-directs the China Data Lab at the 21st Century China Center. She is also part of the Omni-Methods Group. Her research interests lie in the intersection of political methodology and the politics of information, with a specific focus on methods of automated content analysis and the politics of censorship and propaganda in China.

Jeffrey Ding is the China lead for the AI Governance Research Group. Jeff researches China’s development of AI at the Future of Humanity Institute, University of Oxford. His work has been cited in the Washington Post, South China Morning Post, MIT Technology Review, Bloomberg News, Quartz, and other outlets. A fluent Mandarin speaker, he has worked at the U.S. Department of State and the Hong Kong Legislative Council. He is also reading for a D.Phil. in International Relations as a Rhodes Scholar at the University of Oxford.

‍

You can watch a recording of the event here or read the transcript below

‍

Allan Dafoe  00:00

Welcome, I’m Allan Dafoe, the director of the Center for the Governance of AI, which is organizing this series. We are based at the Future of Humanity Institute at the University of Oxford. For those of you who don’t know about our work, we study the opportunities and challenges brought by advances in AI, so as to advise policy to maximize the benefits and minimize the risks from advanced AI. It’s worth clarifying that governance, this key term, refers descriptively to the ways that the decisions are made about the development and deployment of AI, but also the normative aspiration: that those decisions emerge from institutions that are effective, equitable, and legitimate. If you want to learn more about our work, you can go to governance.ai. I’m pleased today to welcome our speaker Molly Roberts, and our discussant Jeffrey Ding. Molly is Associate Professor of Political Science at the University of California, San Diego. She’s a scholar of political methodology, the politics of information, and specifically the politics of censorship and propaganda in China. She has produced a number of fascinating papers, including some employing truly innovative experimental design, probing the logic of Chinese web censorship. Molly will present today some of her work co-authored with Eddie Yang, on the relationship between AI and Chinese censorship. I was delighted to learn that Molly was turning her research attention to some issues in AI politics. After Molly’s presentation we will be joined by Jeffrey Ding in the role of discussant. Jeff is a researcher at FHI and Oxford DPhil PhD student in a pre-doctoral fellow at CSAC, at Stanford. I’ve worked with Jeffrey for the past three years now, and during that time, have seen him really flourish into one of the premier scholars on China’s AI ecosystem and politics. So now Molly, the floor is yours.

Molly Roberts  01:57

Thanks, Allan. And thanks for so much for having me. And I’m really excited to hear Jeffrey’s thoughts on this since I’m a follower of his newsletter, and also his work on AI in China. So this is a new project to try to understand the relationship between censorship and artificial intelligence. And I see this as sort of the beginning of a larger work on this relationship between censorship and artificial intelligence. So I’m really looking forward to this discussion. This is joint work with Eddie Yang, who’s also at UC San Diego. So you might have heard, and probably on this webinar series, that a lot of people think that data is the new oil, data is the input to a lot of products. It can be used to predict financial, to make financial predictions that can be used to then trade stocks or to predict the future of investments. And and at the same time, that data might be the new oil, we also worry a little bit about the quality of this data. So how good is this data? How good is data that’s inputted into these products, applications that are that we’re using now a lot in our AI world. So we know there’s a really interesting new literature in AI about politics and bias within artificial intelligence. And this idea behind this is that this huge data that powers AI applications is affected by human biases that are then encoded in that training data, which then impacts the algorithms that are then used within user facing interfaces or products that encode, that replicate or enhance that bias. So there’s been a lot of great work looking at how racial and gender biases can be encoded within these training datasets that then are put into these algorithms and user facing platforms. For example, there’s been – I don’t know why my tex didn’t work here – but Latanya Sweeney has some great work on ad delivery, speech recognition, there’s been also some great work on word embeddings and image labeling.

Sweeney, Latanya. “Discrimination in online ad delivery.” Communications of the ACM 56.5 (2013): 44-54.

Koenecke, Allison, Andrew Nam, Emily Lake, Joe Nudell, Minnie Quartey, Zion Mengesha, Connor Toups, John R. Rickford, Dan Jurafsky, and Sharad Goel. “Racial disparities in automated speech recognition.” Proceedings of the National Academy of Sciences 117, no. 14 (2020): 7684-7689.

Davidson, Thomas, Debasmita Bhattacharya, and Ingmar Weber. “Racial Bias in Hate Speech and Abusive Language Detection Datasets.” Proceedings of the Third Workshop on Abusive Language Online. 2019.

Caliskan, Aylin, Joanna J. Bryson, and Arvind Narayanan. “Semantics derived automatically from language corpora contain human-like biases.” Science 356.6334 (2017): 183-186.

Zhao, Jieyu, Tianlu Wang, Mark Yatskar, Vicente Ordonez, and Kai-Wei Chang. “Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints.” In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017.

Li, Shen, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du. “Analogical Reasoning on Chinese Morphological and Semantic Relations.” In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 138-143. 2018.

So in this talk, we’re going to explore another institution that impacts AI, which is censorship. Censorship impacts the training data, which then impacts NLP models and applications that are used. So we’re going to look instead of at institutional or human biases that might impact training data, here, we’re going to look at how censorship policies on behalf of governments impact training data. But then how this might have a downstream impact on applications. So we know that large user-generated datasets are the building blocks for AI. So this could be anything from Wikipedia corpuses to social media data sets, government curated data, that more and more data is sort of being put online and this is being used in downstream AI applications. But we also know that governments around the world influence these datasets and have political incentives to influence these datasets which are then used downstream. And they can influence these datasets through fear, through threats or laws that create self censorship that make it so people won’t put things on social media or that they’re whatever their activities are not reflected in government curated data, they can influence these these data sets through friction, what I call friction, which is sort of deletion or blocking of social media posts, or preventing certain types of posts on Wikipedia or preventing some sort of data to be uploaded to a government website, for example. And they can also influence these datasets through flooding, or coordinated addition of information. So we think about coordinated sort of internet armies or other types of government organized groups trying to put information on Wikipedia or on social media, or to to influence the information environment.

Molly Roberts  06:05

So this data is then used in other user facing applications. So increasingly, AI is taking data available on the internet through common crawl through Wikipedia, through social media, and then using it as a base for algorithms in entertainment applications and productivity applications in algorithmic governance and a lot of different downstream applications. So our question is, how does censorship, how does this government influence on these data sets then affect the politics of downstream applications? And it could be that, it could be that even if some of these applications are not in themselves political, that because of this political censorship, they could have some political implications. Deciding which corpus to use, for example, could have political implications on downstream applications. So this paper looks particularly at censorship of Wikipedia corpuses. So we study censorship of Chinese online encyclopedias, and we look at how these different online encyclopedias have different implications for Chinese language, NLP (natural natural language processing). And I’m sorry that my citations aren’t working, but we use word embeddings, they’re trained on two Chinese online encyclopedia corpuses. These are trained by Lee et al., which are, are trained in the same way on Baidu Baike encyclopedia corpus and Chinese language Wikipedia. So we look at Chinese language Wikipedia, which is not blocked within China, and Baidu – sorry, which is blocked but uncensored and Baidu Baike which is not blocked within China but has pre-publication censorship restrictions on it, We look at  how using each of these different corpuses, which have different censorship controls can have different implications for downstream applications. We measure political word associations between these two corpus, and we find that word embeddings, which I’ll go over in a second, trained on Baidu Baike associate more negative adjectives with democracy in comparison to Chinese language Wikipedia, and more positive associations with CCP and social CCP and other types of social control words. And we find with a survey that Baidu Baike word embeddings are not actually more reflective of views of people within China. And therefore, we don’t think that this is coming from simply people’s contributions to Wikipedia, but we think it is coming from the censorship of Wikipedia. And then we also identify a tangible effect of the decision to use pre-trained word embeddings on Baidu Baike versus pre-trained word embeddings on Chinese language Wikipedia in downstream NLP applications. And we’ll talk a little bit at the end about what the strategic implications this might have for politics and AI. So pre-trained word embeddings, some of you may be familiar with what these are, but just by way of introduction in case you’re not: natural language processing, which are algorithms that are used on text, rely on sort of numerical representations of text. So, we have to figure out how to represent text numerically in order to use it then in downstream application. So anything that is doing AI on social media data, Wikipedia data, encyclopedia, on predictive text, this is all relying on a numerical representation of that text. So one way that is common within the social sciences, is to represent text is to simply give each word a number essentially, and say, is this word included within this document or not? This is called the bag of words representation, a one or zero whether or not you use this particular word or not within a text. But another way to represent this text that has become very, very popular in computer science and increasingly also in social sciences is to use word embeddings to represent text. So the idea behind this is that each word, word embeddings estimate a K-dimensional vector, sometimes a 200 300 length vector for any word within a within a huge dictionary of words. And this vector encodes the similarity between words. So words that are likely to be used as substitutes for each other words that are often used in the same context will be in more similar areas of this k dimensional space than other words, and this allows so using pre-trained word embeddings, which already have a K-dimensional vector trained on a large corpus allows an NLP application to know, at the start, how these words might be similar to each other. So often, these word embeddings are pre-trained on very large corpuses, and then they’re used as inputs in smaller NLP tasks. So I already know that two words are more similar to each other than another word, even before starting to train my data.

Molly Roberts  11:13

So often, pre-trained word embeddings are made available by companies, by academics. So this is just one example of a screenshot from fastText: Facebook makes available a lot of different pre-trained word vectors that are trained, these ones are trained on common crawl and Wikipedia. So really large corpuses, they’re using these in 157 languages, you can then go download them and use them as inputs into your NLP model. So here’s just an example,  sort of to fix your fix ideas of what word embeddings are doing. So say that I have two documents. This is from an IBM Research blog, they have two documents document x on the left here, says I gave a research talk in Boston and document y on the right, this is a data science lecture in Seattle, these actually don’t share any words. But if you have word embeddings as a representation of the text, you would know – so this is a very simple two dimensional word embeddings, but imagine in a 300 dimensional space, right? – you would know that actually, these two documents are quite similar in content to each other. Because [garbled] Boston are often reusing the same, often in the same context as each other, [garbled] similar research and science or similar talk and lecture are similar. So the place that these words are within space would be pre-trained on a large corpus, then you could use this word embedding as an input, which would give you more information about those documents. So here, we come to censorship of training data. So these are often pre-trained word embeddings are often trained on very large data sets, like Wikipedia: because they’re user generated, they cover lots and lots of different topics. So we think that they’re sort of representative of how people talk about many different things. In China, or in the Chinese language, however, this is complicated by the fact that the Chinese government has blocked Chinese language Wikipedia. And therefore there’s also been the development of another Wikipedia corpus Baidu Baike, which is unblocked within China but is censored. So both of these are online encyclopedias in China, they’re both commonly used as training data for NLP and if you look at CS literature, you’ll see both of them as used as training data. Chinese language Wikipedia is uncensored as I said, but it is blocked and Baidu Baike is censored in that there are a lot of regulations of what can be written on Baidu Baike, but it is unblocked in that it is available within mainland China. So for example, if you want to create a post or a entry on Baidu Baike about the June 4th movement, it automatically tells you you cannot create this post. Also, there are a lot of regulations about political topics have to follow Chinese official government, Chinese government official news sources, so there’s so there’s a lot of pre-censorship of these entries, unlike Chinese language Wikipedia, where you can contribute without pre-censorship. There’s been some great work by Zhang and Zhu in 2011, American Economic Review, censorship of Wikipedia has reduced contributions to it. So they show that when Chinese language Wikipedia was censored, that there are many, many fewer contributions to Chinese language Wikipedia, because there’s a decrease in that in the audience of the of the site. And because of this, we are apparently at least because of this, Baidu Baike is many, many times larger than Chinese language Wikipedia with 16 times more pages. And therefore it’s increasingly an attractive source of training data for Chinese language NLP.

Molly Roberts  14:49

So what we do in this paper is we compare word embeddings. Between, we compare essentially where word vectors sit in pre trained word embeddings, pre trained on Chinese language Wikipedia versus Baidu Baike. So I’ll just give you a really simple example of what this might look like. So we have word embedding a say this is Baidu Baike. These are a few different word vectors from this trained on this corpus, and we have word embedding b., say this is Chinese language Wikipedia What we’re interested in is how some target words, for example words like democracy, or other types of political words, where they sit in relation to adjectives, positive and negative adjectives. So in this case, is democracy closer in word embedding space to stability, or is it closer to chaos. And we could compare where democracy sits between these two positive and negative added adjectives on in Chinese language Wikipedia versus Baidu Baike. So what we do is we come up with groups of target words, and we have many different categories of target words, each category of target words, has about 100 different words associated with that category. So we use democratic concepts and ideas, we use categories such as democratic, democracy, freedom and election. So each of these categories that has about 100 different words that are sort of synonymous with this within it. And then we also use non targets of propaganda. So for example, social control, surveillance, collective action, political figures, CCP, or historical other historical events, we also find lots of lots of words associated with these in these categories. And then we look at how they are related to attribute words, like adjectives for example, or evaluative words, which are these words in blue. So we use a list of propaganda attribute words, words that we know from reading and studies of propaganda that are often associated with these concepts. And we also use general evaluative words from these big adjective evaluated word lists in Chinese that are often used in Chinese language NLP. So what we do is we take each target word vector, so this is xi, where xi is either that word that part of the vector of the target word from Baidu, or from Wikipedia. And then we take the attribute word vectors, where A is positive attribute that that vector in either Baidu or Wikipedia, and B has the negative word vector for other by doing Wikipedia. And then for each embedding for Baidu, or for Wikipedia, we examine the cosine similarity between the target word and the positive attribute words, minus the mean cosine similarity between the target word and the negative attribute words. And then we take the difference between these differences across all of the word target words within a category to get the relationship or how much closer positive words are overall to the target category in comparison of Baidu to Chinese language Wikipedia. So if Baidu is Category A, and Wikipedia is Category B, if this is very negative, it means that the target, the target category is more associated with negative words. If this is more positive, this means that this target category is more associated with positive words. And the difference between these would be negative which means that Baidu would be associated more negatively than Chinese language Wikipedia. To assess statistical significance, we do a permutation test where we permute the assignment of the word vectors to A B and then we see how extreme our result is in comparison to that permutation. So the theoretical expectation of this is that overall, freedom, democracy, election, collective action, negative figures, all of these categories will be more associated with negative attribute words in Baidu Baike as in comparison to Chinese language Wikipedia. On the other hand, categories like social control, surveillance, CCP, historical events, and positive figures should be more positively associated. And this is exactly what we find. So here is the effect size for propaganda words for each of these categories. And here’s the effect size for evaluative words for each of these categories along with the p value of statistical significance. And we find that overall, Baidu Baike target words in the categories of freedom, democracy, election, collective action, negative figures, are more associated with negative attribute words in Baidu Baike than they are in Chinese language Wikipedia, and the opposite for categories such as social control, surveillance, CCP, etc.

Molly Roberts  19:31

So this could be one possibility that you might think is that perhaps it’s just simply that mainland Chinese internet users view target categories differently than overseas internet users contributing to Chinese language Wikipedia. And therefore this just this difference in word associations between these two, these two sets of internet users is creating this difference in online encyclopedias. So to try to get at this, we did an online survey of about 1000 response in mainland China and we asked people if they thought that we asked them between the following options, which do you think better describes a particular target word. And we took the closest attribute word from Baidu Baike, in the word embedding space, and the closest attribute word for Wikipedia, and we asked people to evaluate that. And what we found is that overall, neither of neither of the Baidu Baike nor Chinese language Wikipedia seemed to better reflect the associations of our survey respondents. So for some words, Chinese so this on the x axis is the likelihood of choosing the Baidu word for some categories and for some lists of attribute words, Chinese language Wikipedia was preferred, and in some categories for some list of attribute words Baidu Baike was preferred. So we didn’t see one of these to necessarily dominate the other in terms of users evaluations of them. So we didn’t we that sort of rejected this sense that Baidu Baike is just better reflecting people’s work associations. So the third thing that we did was we evaluated the downstream effect of these word embeddings on a machine learning task. So the task that we set out to do is classify news headlines according to sentiment. And we use a big general Chinese news headlines data set as our training data. So this is saying, so say you were, you wanted to create a general sentiment news headline classifier, say to create a recommendation system, or to do content moderation on a social media website, for example, say you were creating this algorithm, you might use a general Chinese news headline data set as training data. And then we’re going to look at how the algorithm that that was trained, performs on news headlines that contain these target words, like words related to democracy and election and freedom and social control and surveillance and words with historical or figures that might be of interest to CCP. So we use three different models, Naive Bayes, SVM, and neural network. And we look at how using the same training data, the same models, but simply with different pre-trained word embeddings one that comes from Baidu Baike, one that comes from Chinese language Wikipedia, how just using different pre-trained word embeddings can influence that systematic classification error of this downstream of this downstream task. So do overall do models trained with pre-trained Baidu Baike word embeddings have a just a slightly more negative classification of headlines that contain democracy, than models that contain word embeddings that are models that were trained using pre-trained Chinese language Wikipedia word embeddings.

Molly Roberts  22:57

So this is an example of so for example, this is a headline “Tsai Ing-wen: Hope Hong Kong can enjoy democracy as Taiwan does” the Wikipedia label here comes out as positive when we train this, but the Baidu Baike label when we use the same classifier, same training data, just a different word embeddings comes out as negative, or “Who’s shamed by democratization in the kingdom of Bhutan”, the Baidu Baike label here is coming out as negative, the Wikipedia label is coming out as positive even though the human label here is negative. So, um, so, you know, what are the sort of systematic mistakes that these classifiers are making? Overall, we see that these classifiers actually have very similar accuracy. So it’s not that Baidu Baike models trained on Baidu Baike word embeddings have a higher accuracy than the models trained on Chinese language Wikipedia word embeddings, we see that the accuracy is quite similar between each each of these different word embeddings. But we see big effects on the classification error in each of these different categories. So me, LIJ is the human labeled score for a news headline for target word I in category J. So this would be a negative one if it was a negative sentiment and a positive one if it was a positive sentiment. And if we use the if we get the predicted scores from Baidu and from Wikipedia, our models trained on Baidu Baike word embeddings versus Wikipedia word embeddings. And then we create a dependent variable that is the difference between Baidu and the human label and Wikipedia and the human label for a category, then we can we can estimate how the difference between how the difference between the human label and the predicted label changes by category for the Baidu classifier versus the Wikipedia classifier. So our coefficient of interest here is beta J. How does, how is, is there systematic differences in the direction of classification for a certain category for the algorithm trained with Baidu word embeddings versus Wikipedia word embeddings. And what we find is that there are quite systematic differences across all different machine learning models in the direction that we would expect. So Baidu Baike overall is much are the classifiers trained with pre-trained word embeddings on Baidu Baike overall are much more likely to categorize headlines that contain target words in the categories of freedom, democracy, election, collective action, negative figures, as more negative than social control, surveillance, CCP, historical and positive figures. So just to sort of think about a little bit of the implications of the potential implications of this. So there are sort of strategic incentives. So given that what I hope I convinced you so far is that censorship of training data can have an impact, a downstream impact on NLP applications. And if that’s true, one thing that we might try to think about is, are there strategic incentives to manipulate training data? So we do know that there are lots of government-funded AI projects to create more training data, to gather more training data that then can be used in AI in order to sort of push AI along, might there be sort of strategic incentives to influence this part, the politics of this training data? And how could this play out sort of downstream. So you might think that there would be a strategic benefit to, for example, a government, for example, to manipulate the politics of the training data. And we might think that that could be that their censored training data could in some circumstances reinforce the state, right. So in applications like predictive text, where we’re creating predictive text algorithms, the state might want sort of associations that are reflective of its own propaganda, and not reflective of things that it would like to censor, to sort of replicate themselves within these predictive text algorithms, right. Or in cases like recommendation systems or search engines, we might think that a state might want these applications trained on on data that they themselves curate.

Molly Roberts  27:20

On the other hand, and this I think is maybe less, less obvious when you first start thinking about this, but became more obvious to us as we started thinking about this more: censored training data, it might actually make it more difficult for the state to see society in a ways that it might actually undermine some applications in in certain ways. So for example, content moderation, there are a lot of new AI algorithms to moderate content online, whether it’s to censor it, to remove content that that violates the terms of service of a website, etc. If content moderation is trained on data that has all of sensitive or objectionable topics removed, in fact, it might be worse actually distinguishing between these topics, from the state’s perspective, than if that initial training data were not censored, right, and so we can think about ways in which censorship of training data might actually undermine what the state is trying to achieve. The other way in which it could be problematic for the interests of the state is to see in public opinion monitoring. So if, for example, a lot of training data were censored, in that removed opinions or ideas that were in conflict with the states, it might also if that was used as training data to then understand what the public thinks on on in, for example, by looking at social media data, which we know a lot of states do. And this could bias the outcome of this data in ways that would make it harder for the state to sort of see society. So just to give a plug for another paper that I’ve been that it’s coming out in the Columbia Journal of Transnational Life, I work with some co-authors on the Chinese legal system. And we show that sort of legal automation, which is one of the objectives of the Supreme People’s Court in China is sort of undercut by a data missing this within this big legal data set that the the Supreme People’s Court has been trying to curate. So in summary, data reflects, we know, the institutional and political contexts in which it was created. And not only do human biases replicate themselves in AI, but also political policies impact training data, which then has downstream applications. We showed this in word embeddings and my downstream NLP applications as a result of Baidu Baike and Chinese language Wikipedia word embeddings. But of course, we think that this is a much more general phenomenon that is potentially worthy of future study. And this could have an effect In a wide range of areas, including public opinion monitoring, conversational agents, policing and surveillance and social media curation. So AI, is in some sense can can, in some sense, replicate or enhance sort of an automation of politics. And there have been some discussions about trying to de-bias AI, we also think that this is, would be might be difficult to do, especially in this context where we’re not really sure what a de-biased political algorithm would look like. And so, thanks to our sponsors, and really looking forward to your questions and comments.

Allan Dafoe  30:41

Thanks, Molly. 66 people are applauding right now. That was fantastic. I know I had troubles processing all of your contributions, and I did a few screenshots, but not enough. So I’m sure we’ll, or I think there’s a good chance we’ll have to have you flick back through some of the slides. A reminder to everyone, there’s a function at the bottom where you can ask questions, and then you can vote up and down on people’s questions. So that’d be great to have people engaging. So now, over to Jeffrey Ding for some reflections.

Jeffrey Ding  31:13

Great. Yeah, this was really cool, really cool presentation. Dr. Roberts sent along like a early version of the paper beforehand, so we can get into a little bit and unpack the paper a little bit more in the discussion. But I just wanted to say off the bat that it’s just a really cool paper and it’s a good example of kind of flipping the arrow because a lot of like related work in this area most people look at the effect of NLP and language models on censorship. And just flipping the arrow to look at the reverse effects is really cool. And it also speaks to like this broader issue in NLP research where the L matters in NLP. So much of the time, most of the time, we talk about like English language models, but we know there are differences in terms of languages, in terms of how NLP algorithms are applied. So we see that with like low resource languages, like Welsh and Punjabi, there’s still a lot of barriers to developing NLP algorithms. And your presentation shows that even for the two most resourced languages, English and Chinese, there are still significant differences, tied to the censorship. And finally, the thing that really stuck out to me from the papers, just in the presentation is just an understanding and the integration of the technical details about about how AI actually works, and tied to political implications. And then one line that really stuck out and then you emphasize in the presentation is that the differences that you’re seeing and the downstream effects don’t necessarily stem from the training data in the downstream applications, or even the model itself, but from the pre-trained word embeddings, that have been trained on another data set. So that’s just a really cool, detailed finding, and kind of a level of nuance that you just don’t really see in the space. So really excited to dig in. I just have kind of three buckets, and then a couple of just thoughts to throw at you at the end. So the first bucket is which words to choose, which target words to choose. And I find it really interesting just like thinking through this, because for example, you pick election and democracy as two of the examples. And for democracy, it actually brings up an interesting question in that like, the CCP has kind of co-opted the word democracy, mínzhǔ. And and it actually like ranks second on the party’s list of 12 core values that they published in December 2013. Elizabeth Perry has written on this sort of like, populist dream of Chinese Democracy. So I’d be curious if you thought about that of like, when you you know, when you’re in different Chinese cities, and you see like the huge banners with democracy plastered along all these banners? And just, I wonder, like, what if you picked a target word like representation, or something that might speak to this more kind of populist dream or populist co-option of what democracy means?

Jeffrey Ding  34:07

And then on the point about I think the second point is sort of, on the theoretical expectations kind of tied to this democracy component, whether we should expect kind of more negative connotations related to democracy in the first place, is this idea of the negative historical events and negative historical figures. And the question is, why should we expect a more negative portrayal if these events and figures have been erased from the corpus? So shouldn’t it be, shouldn’t it be just basically not positive or not negative, kind of like just a neutral take? And I think in the paper, you gestured, you kind of recognize this and say that there’s very little information about these historical figures, sSo so their word embeddings do not show strong relationships with the attribute words and I’m just curious if we should expect the same thing with the negative historical events as well, like Tiananmen Square is the most obvious example. And then on the results I just had a quick thing that surprised me a little bit was that you showed that Baidu Baike and Wikipedia perform at the same level of accuracy overall. And then kind of the setup of the initial question is that Baidu Baike has just become a much better corpus, and there’s much more time spent on the corpus, it’s 16 times larger. So I’m just curious why we didn’t see that Baidu Baike corpus perform better.

Jeffrey Ding  35:37

And then yeah, I had, I had some comments on kind of threats to inference, kind of like alternative causes other than censorship that are producing the results. And actually one of them was just a different population of editors. And it’s cool that you all have already done a survey experiment to kind of combat that. That kind of alternative cause I was just thinking, like, as you’re talking about the social media stuff, I wonder if the cleanest way to kind of show the censorship as the key driving factor would be to like, train, train a language model based off of a censored version of like Weibo posts, a sample Weibo posts versus like the population that includes all the Weibo posts from a certain time period. And some, no, that’s like something that other researchers have used to study censorship. And then my last thought, just to open it up to like, kind of bigger questions that I actually don’t know that much about, but it would be cool to know, there’s a lot of technical people on the webinar as well, they could chime in on this point. But the hard part about studying these things is the field moves so fast. So now people are saying that it’s only a matter of time before pre-trained word embeddings and methods like word2vec are just completely replaced by pre-trained language models like, OpenAPI’s work, Google’s work, ELMo GPT2, GPT3. And the idea is that pre-trained word embeddings, kind of they only incorporate previous knowledge into the first layer of the model. And then the rest of the network still needs to be trained from scratch. And recent advances have basically taken kind of what people have done with computer vision and just taken a, to pre-train the entire model with a bunch of hierarchical representations. So I guess like word2vec, would be just wanting the edge and then these pre-trained language models would be learning like the full hierarchy of all of the features from like edges to shapes. And it’d be interesting to explore whether, to what extent, these new language models would still fall into the same traps, or whether they will provide ways to kind of combat some of the problems that you’ve raised. But yeah, looking forward to the discussion.

Molly Roberts  37:53

For great, fantastic comments, and thank you so much, I really appreciate that. And just to sort of pick up on a few of them. And, yeah, we were actually we didn’t, we had certain priors about the category of democracy, we thought that overall, it would be more negative. But of course, we did discuss this issue of mínzhǔ and how and how it’s been used within propaganda within China. The way that we did it was we took, we used both sets of word embeddings to look at all of the closest words to democracy, and get all hundred of those. So it’s not just mínzhǔ, but it’s also all of the other things that are sort of subcategories of democracy. And so it could be that for one of these words, it might be different than others. Right. And so I think that we’re seeing sort of like the overall category, but I think it’s something we should look a little bit more into, because it could sort of piece out some of these mechanisms. Yeah, so one of the things we find with negative historical events and figures is we get less decisive results in these categories. And we think that this is because Baidu Baike just doesn’t have entries on these negative historical events and figures. I think this is one example of how censorship of training data can make can sort of undermine the training corpus, because even from the perspective of the state, if algorithms were using this, and for example, social media, or censorship down the road, you would expect the state would want the algorithm to be able to distinguish between these things, but in fact, because of censorship itself, the algorithm is maybe going to do less well, there might we haven’t shown this yet, but we wouldn’t expect it to do less well on the censorship task than it would have if the training data weren’t censored in the first place. So that’s sort of an interesting kind of catch 22 of this for the from the state’s perspective, right. So and it is interesting that Baidu Baike and Wikipedia at least in our case performed with about the same level of accuracy. And there are papers that show that for certain really more complicated models, the magnitude of the Baidu Baike corpus is better. But of course, I think it sort of depends on your application. In our case, there wasn’t really a difference between the performance or the level of accuracy.

Molly Roberts  40:21

And I really liked this idea of looking at censored versus uncensored corpuses of Weibo posts to try to understand how that could have a downstream effect on training, I think that’d be great way to kind of piece that out. And then this point that you have about pre-trained language models sort of superseding this sort of like pre-trained embeddings, and this transfer learning task, I think that that’s really, really interesting development. And I think that this only makes these questions of where what the trick with initial training data is in transfer learning become more and more important, right? Because these are just sort of the biggest data, whatever has the most data, is that data itself has been amplified by the algorithm downstream. And it’s hard to sort of think about how to delete those biases without actually just fixing the training data itself, or making it more representative of whoever gets, you know, of the population or the language, etc. So yes, I’m looking forward to more discussion. And thank you so much for this awesome comments.

Allan Dafoe  41:35

Great, Jeff, do you want to say any more?

Jeffrey Ding  41:38

Yeah, that last point is really interesting, because there’s some people that are saying that, like, basically, NLP is looking for it’s kind of ImageNet, and kind of, you know, this big, really representative really good data set that you can then just train, you know, your train the language models on and then you do you do transfer learning and learning to all these downstream tasks? And yeah, I think your paper really points to like, if Baidu Baike becomes the ImageNet of Chinese NLP, and you know, I don’t know enough of the technical details in terms of like, if there’s ways in transfer learning to like, do some of the debiasing from the original training set, but yeah, I think, yeah, I think, obviously, the paper will still be super relevant to kind of wherever the NLP models are going.

Allan Dafoe  42:30

Great. Well, I have some thoughts to throw in, and I see people asking questions and voting. So that’s good. We’ll probably start asking those two eventually. And also, you too, should continue. Yeah, saying whatever you want to say. So but just some brief thoughts from me. So I also really like this idea of doing this kind of analysis on a corpus of uncensored data, and then you have an indicator for whether the post was censored. And of course, Molly, I think was it your PhD work, in which you did this. Yeah. So Molly’s already been a pioneer in this research design. And it’s not to say that this, I think it would just be a nice complement to this project. Because this project, you know, you have two nice corpuses, but it’s not obvious what’s causing the differences. It’s like, it could be censorship, it could be fear, it could be different editorial, you know, editors, or just contributors. And whereas that would really, I mean, you’d get the result anyway, so I think that’d be really cool. Okay, so a question I have is maybe one way to phrase it is how, like, how deep are these biases in, in a model trained on these corpuses? And I mean, we, I’d say, currently, you know, we don’t know of a solution for an easy solution for how you can kind of remove notions of bias or, or meaning from, from a language model. You can often remove a kind of the, the connections that you think of, but then there’s still lots of, you know, hidden connections that you may not want. Now, here’s maybe an idea for how you can look at how robust these biases are. I think it was your study three. I don’t know if you’re able to flick to that slide. So you had this, you have a pre trained model. And then in study three, you gave it some additional, right, some additional training data. Okay, yeah.

Molly Roberts  44:36

I’ll go to the setup.

Allan Dafoe  44:39

Yeah, exactly. Yeah. So you have your pre-trained model, and then you give it some additional, right, these Chinese headlines, training data. And sort of the graph I think I want to see is your outcome, so how biased it is, as a function of how much training it’s done, and so on. Initially, it should be extremely biased. And then as you train, if I, well, I think this applies to study three, but if not study three, you could just do it for another corpus where you have sort of the intended associations in that corpus, and see how long it takes for these sort of inherited biases to diminish. You know, maybe they they barely diminished at all, maybe they very rapidly go down with, you know, the first. Well, anyhow, with not too large of a data set, maybe, you know, you never quite get to no bias, but you get quite low. So, yeah, that might be one interesting way of looking at how deep are these biases? How hard is it to extract them? And and I guess another question for you,a nd a question for anyone in the audience who is a natural language expert? Is, are there techniques today or likely on the horizon that could allow for unraveling or flipping or kind of, without requiring almost overwhelming the pre-trained data set, having some way of sort of doing surgery to change the biases in a way that’s not just superficial, but fairly deep? And so, you know, like, for example, maybe with if you have this censored, uncensored and censored data set, you could infer what are the biases being produced by censorship. And then, even if you have a small dataset of this uncensored sensor data, set the biases that you would learn from that. You could then, like, subtract that out of these larger corpuses. And I guess the question is, how effective would that be? And I don’t expect we know the answer, but might be worth reflecting on.

Molly Roberts  46:50

Those are really great points and really interesting points, and I am I, you know, we’re really standing on the shoulders of giants here. This, there’s this whole new literature, and I’m embarrassed that my tech didn’t work. And I’ll I’ll post links to these papers later, that are, is this whole literature on bias of within AI, with respect to race and gender. And, and certainly one of the things that this literature has started to focus on is what are the harms and the downstream applications? So when you talk about like, how deep are these biases, I think one of the things that we have to quantify sort of downstream is like, what are what are there? How are they being used? How is this data being used within applications? And then how does the application that affect people’s decision making or people’s opportunities, etc, or it’s up down down the road? And I think that’s a really hard thing to do. But, but it’s important. And I think that’s one of the ways that where we want to go sort of inspired by this literature. I think how bias is a function of the training data is really interesting. And I think we’ve done a little bit of a few experiments on that, but I think we should include that as a graph within the paper. And certainly, as you get more and more training data, the word embeddings will be less important. Right? And it’s, that would be at least my prior on that. And I think this idea of trying to devise, like, how would you sort of subtract out the bias? I like the idea of trying to figure out, so if you had a corpus, which included both uncensored and included an entire uncensored corpus, and then what information was censored, and then trying to reverse engineer, what are the things that are missing, right, and then adding that to back into the corpus, that would be sort of the way to go about it. It seems hard at one of the things that it doesn’t overcome is self censorship. Because of course, if people didn’t originally add that information to the corpus, even if it were, you never even see that within the data. And, and also sort of, with training data itself is affected by algorithms, because you know, what people talk about. So for example, on social media might be that a lot of people are talking about a lot of different political topics, but certain conversations are amplified, say, by moving them up the newsfeed and other applications are down the newsfeed. And so then you get sort of this feedback loop on what the training data is like. But then, if you then use that, again, into training data, amplifies that again. So I think that there’s so many complicated feedback, AI feedback loops within this space that they’re really difficult to piece out. But that doesn’t mean we shouldn’t try. Yeah, yeah.

Allan Dafoe  49:33

Yeah. A thought that occurred during your talk, is I can imagine the future of censorship is more like light editing. So I submit a post and then the language model says, let’s use these slight synonyms for the words you’re using that have a better connotation than that you can imagine just the whole social discourse being run through this filter of the right associations. And I guess a question for you then also on this is what is is there like an arms race with citizens? So if if citizens don’t entirely endorse the, the state pushed associations, how can what countermeasures can they take? So can they, you know, if one word is sort of appropriated, can they, you know, deploy other words? And I know, there’s kind of like symbolic, you know, games where you can kind of use a symbol as a substitute for a censored term. And, and, and so yeah, are we is there this kind of arms race dynamic happening, where the state wants to control associations and meanings, and people want to express meanings that are not approved of, and so then they change the meaning of words. And you know, maybe even in China, we would see, like a faster cycling or evolution of the meaning of words, because you have this cat and mouse game?

Molly Roberts  50:55

Yeah, I think that’s absolutely right. And I have definitely talked to people who have created applications for like suggesting words that would get around censorship, right. And, and, and that’s, you know, would be like an interesting technology, cat and mouse game around this with AI being used to censor and also adding news to evade censorship. I think one of the interesting implications of what we’re like if you think about the sort of the political structure of AI, as you think about, you know, maybe a set of developers who aren’t necessarily political in themselves, they’re creating applications that are, you know, productivity applications, entertainment applications that are being used in a wide from a wide variety of people. And they’re looking for the biggest data, right, and so and like the most data, the data that’s going to get them the highest accuracy. And because of that, I think the state has a lot of influence over what types of training data sets are developed. And, and has a lot of influence on these applications, even if the application developers themselves are not political. And I think that’s an interesting like, interaction. I’m not, you know, I think, I’m not sure how much states around the world have thought about the development, the politics within training data, and maybe, but I think it could be something that they start thinking about, and might be something to sort of try to understand that. You know, how they might, as training data begin somewhere more important, how they might try to influence it. Yeah.

Allan Dafoe  52:28

Good. Well, we’re at time. So yeah, the remaining questions, I’m afraid will will go unanswered. There was a request for your attention. What was the paper, automating fairness paper? And also, I think, I’m sure people are excited for this paper to come out. So yeah, we look forward to seeing this come out. And, you know, continuing to read, you’re really fascinating and creative. And I guess, yeah, just especially an empirical, like, your work is really thoughtful and effortful, and the extent to which you use sort of different quantitative designs and experimental designs to answer these, and almost kind of field experimental, I guess, designs where you’re, you’re really? Yeah, you can only deploy these experiments if you know the, the nature of the political phenomenon well enough, and I guess, have the resources to devise these experiments that you have been doing. So it’s very exciting work, and thanks for sharing the latest today.

Molly Roberts  53:39

Thanks. Thanks so much for having me. And yeah, thanks, Jeff, also for your fabulous comments.

Molly Roberts  53:47

Thanks, everybody, for coming.

‍

The governance of AI is in my view the most important global issue of the coming decades, and it remains highly neglected. It is heartening to see how rapidly this field is growing, and exciting to be part of that growth. This report provides a short summary of our work in 2018, with brief notes on our plans for 2019.

2018 has been an important year for GovAI. We are now a core research team of 5 full-time researchers and a network of research affiliates. Most importantly, we’ve had a productive year, producing over 10 research outputs, ranging from reports (such as the AI Governance Research Agenda and The Malicious Use of AI) to academic papers (e.g. When will AI exceed human performance? and Policy Desiderata for Superintelligent AI) and manuscripts (including How does the Offense-Defense Balance Scale? and Nick Bostrom’s Vulnerable World Hypothesis).

We have ambitious aspirations for growth going forward. Our recently added 1.5 FTE Project Manager capacity between Jade Leung and Markus Anderljung, will hopefully enable this growth. As such, we are always looking to help new talent get into the field of AI governance. If you’re interested, visit www.governance.ai for updates on our latest opportunities.

Thank you to the many people and institutions that have supported us, including our institutional home–the Future of Humanity Institute and the University of Oxford–our funders–including the Open Philanthropy Project, the Leverhulme Trust, and the Future of Life Institute–and the many excellent researchers who contribute to our conversation and work. We look forward to seeing what we can all achieve in 2019.

Allan Dafoe
Director, Centre for the Governance of AI
Future of Humanity Institute
University of Oxford

Below is a summary of our research, public engagement, in addition to our team and growth.

‍

Research

On the research front we have been pushing forward a number of individual and collaborative research projects. Below is a summary of some of the biggest pieces of research published over the past year.

AI Governance: A Research Agenda
GovAI/FHI Report.
Allan Dafoe

The AI Governance field is in its infancy and rapidly developing. Our research agenda is the most comprehensive attempt to date to introduce and orient researchers to the space of plausibly important problems in the field. The agenda offers a framing of the overall problem, an attempt to be comprehensive in posing questions that could be pivotal, and references to published articles relevant to these questions.

Malicious Use of Artificial Intelligence
GovAI/FHI Report.
Miles Brundage et al [incubated and largely prepared by GovAI/FHI]

Artificial intelligence and machine learning capabilities are growing at an unprecedented rate. These technologies have many widely beneficial applications, ranging from machine translation to medical image analysis. Countless more such applications are being developed and can be expected over the long term. Less attention has historically been paid to the ways in which artificial intelligence can be used maliciously. The report surveys the landscape of potential security threats from malicious uses of artificial intelligence technologies, and proposes ways to better forecast, prevent, and mitigate these threats.

The report was featured in over 50 outlets, including the BBC, The New York Times, The Telegraph, The Financial Times, Wired and Quartz.

Deciphering China’s AI Dream
GovAI/FHI Report
Jeffrey Ding

The Chinese government has made the development of AI a top-level strategic priority, and Chinese firms are investing heavily in AI research and development. This report contextualizes China’s AI strategy with respect to past science and technology plans, and it also links features of China’s technological policy with the drivers of AI development (e.g. hardware, data, and talented scientists). In addition, it benchmarks China’s current AI capabilities by developing a novel index to measure any country’s AI potential and highlights the potential implications of China’s AI dream for issues of AI safety, national security, economic development, and social governance.

Cited by dozens of outlets, including The Washington Post, Bloomberg, MIT Tech Review, and South China Morning Post, the report will form the basis for further research on China’s AI development.

The Vulnerable World Hypothesis
Manuscript.
Nick Bostrom

The paper introduces the concept of a vulnerable world: roughly, one in which there is some level of technological development at which civilization almost certainly gets devastated by default, i.e. unless it has exited the “semi-anarchic default condition”. Several counterfactual historical and speculative future vulnerabilities are analyzed and arranged into a typology.

Discussed in Financial Times.

How Does the Offense-Defense Balance Scale?
Manuscript.
Ben Garfinkel and Allan Dafoe

The offense-defense balance is a central concept for understanding the international security implications of new technologies. The paper asks how this balance scales, meaning how it changes as investments into a conflict increase. To do so it offers a novel formalization of the offense-defense balance and explores models of conflict in various domains. The paper also attempts to explore the security implications of several specific military applications of AI.

Policy Desiderata for Superintelligent AI: A Vector Field Approach
In S. Matthew Liao ed.  Ethics of Artificial Intelligence. Oxford University Press.
Nick Bostrom, Allan Dafoe, and Carrick Flynn

The paper considers the speculative prospect of superintelligent AI and its normative implications for governance and global policy. Machine superintelligence would be a transformative development that would present a host of political challenges and opportunities. The paper identifies a set of distinctive features of this hypothetical policy context, from which we derive a correlative set of policy desiderata — considerations that should be given extra weight in long-term AI policy compared to in other policy contexts.

When Will AI Exceed Human Performance? Evidence from AI Experts
Published in Journal of Artificial Intelligence Research.
Katja Grace (AI Impacts), John Salvatier (AI Impacts), Allan Dafoe, Baobao Zhang, Owain Evans (Future of Humanity Institute)

Our expert survey, we report the results from a large survey of machine learning researchers on their beliefs about progress in AI. The piece was the 16th most discussed article in 2017 according to Altmetric. It was reported on in e.g. the BBC, Newsweek, NewScientist, Tech Review, ZDNet, Slate Star Codex and The Economist.

Governing Boring Apocalypses: A New Typology of Existential Vulnerabilities and Exposures for Existential Risk Research
Published in Futures.

Hin-Yan Liu (University of Copenhagen), Kristian Cedervall Lauta (University of Copenhagen), and Matthijs Maas

This article argues that an emphasis on mitigating the hazards (discrete causes) of existential risks is an unnecessarily narrow framing of the challenge facing humanity, one which risks prematurely curtailing the spectrum of policy responses considered.  By focusing on vulnerability and exposure rather than simply existential hazards, the paper proposes a new taxonomy which captures factors contributing to these existential risks. The paper argues that these “boring apocalypses” may well prove to be the more endemic and problematic, than those commonly focused on.

Syllabus on AI and International Security
GovAI Syllabus.
Remco Zwetsloot

This syllabus covers material located at the intersection between artificial intelligence and international security. It is designed to be useful to (a) people new to both AI and international relations; (b) people coming from AI who are interested in an international relations angle on the problems; (c) people coming from international relations who are interested in working on AI.

Scaling Up Humanity: The Case for Conditional Optimism about Artificial Intelligence
In Should we fear artificial intelligence? an in-depth analysis for the European Parliament by the Scientific Foresight Unit.
Miles Brundage

This paper makes a case for conditional optimism about AI and to fleshes out the reasons one might anticipate AI being a transformative technology for humanity – possibly transformatively beneficial. If humanity successfully navigates the technical, ethical and political challenges of developing and diffusing powerful AI technologies, AI may have an enormous and potentially very positive impact on humanity’s wellbeing.

Public engagement

We have been active in various public fora – you can see a sample of the presentations, keynotes, panels and interviews that our team has engaged in here.

Allan Dafoe has been giving several talks each month, including at a hearing to the Security and Defense subcommittee of the European Parliament, Oxford’s Department for International Relations, and being featured in the documentary “Man in the Machine”, by VPRO Backlight (Video, at 33:00). He has done outreach via the Future of Life Institute, Futuremakers and 80,000Hours podcasts.

Nick Bostrom participated in several government, private and academic events, including DeepMind Ethics and Society Fellows event, Tech for Good Summit convened by French President Macron, Sam Altman’s AGI Weekend, Jeff Bezos’s MARS, World Government Summit – Dubai, Emerging Leaders in Biosecurity event in Oxford, among others. In 2018 his outreach included circa 50 media engagements, including BBC radio and television, podcasts for SYSK, WaitButWhy, print interviews, and multiple documentary filmings.

Our other researchers have also participated in many public fora. Jeffrey Ding has, on the back of his report on China’s AI Ambitions, interviewed with the likes of the BBC and has recently been invited to lecture at Georgetown University to DC policy-making circles. Additionally, he runs the ChinAI newsletter, weekly translations of writings on AI policy and strategy from Chinese thinkers, and has contributed to MarcoPolo’s ChinAI which presents interactive data on China’s AI development. Matthijs Maas presented work on “normal accidents” in AI at the AAAI/ACM conference on Artificial Intelligence, Ethics, and Society and presented at a Cass Sunstein masterclass on human error and AI (video here). Sophie Fischer  was recently invited to China as part of a German-Chinese Young Professional’s program on AI, and Jade Leung has presented on her research at conferences in San Francisco and London, notably at the latest Deep Learning Summit on AI regulation.

Moreover, we have participated in Partnership on AI working groups on Safety-Critical AI, Fair, Transparent, and Accountable AI in addition to AI, Labor and the Economy. The team has also interacted considerably with the effective altruism community, including a total of six talks at this year’s EA Global conferences.

Members of our team have also published in select media outlets. Remco Zwetsloot, Helen Toner and Jeffrey Ding published “Beyond the AI Arms Race: America, China, and the Dangers of Zero-Sum Thinking” in Foreign Affairs, a review of Kai-Fu Lee’s “AI Superpowers: China, Silicon Valley, and the New World Order.” In addition, Jade Leung and Sophie-Charlotte Fischer published a piece in the Bulletin of the Atomic Scientists on the US Defense Department’s Joint Artificial Intelligence Center.

Team and Growth

We have large ambitions and demands for growth. The Future of Humanity Institute has recently been awarded £13.3 million from the Open Philanthropy Project, we have received $276,000 from the Future of Life Institute, and we have collaborated with Baobao Zhang on a $250,000 grant from the Ethics and Governance of Artificial Intelligence Fund.

The team has grown substantially. We are now a core research team of 5 full-time researchers, with a network of research affiliates who are often in residence, coming to us from across the U.S. and Europe at institutions such as ETH Zurich and Yale University. As part of signalling our growth to date, as well as our planned growth trajectory, we are now the “Center for the Governance of AI”, housed at the Future of Humanity Institute.

We continue to receive a lot of applications and expressions of interest from researchers across the world who are eager to join our team. We are working hard with the operations team here at FHI to ensure that we can meet this demand by expanding our hiring pipeline capacity.

On the operations front, we now have 1.5 FTE Project Manager capacity between two recent hires, Jade Leung and Markus Anderljung, which has been an excellent boost to our bandwidth. FHI’s recently announced DPhil scholarship program as well as the Research Scholars Program are both initiatives that we are looking forward to growing in the coming years in order to bring in more research talent.