Module 5 - AI Risk

Introducing Bias in Automated Decision-Making Systems 

What is automated bias, and where does it come from? 

Automated decision-making systems, often driven by advanced AI technology, are now used to obtain predictions in a variety of domains: predictive policing technologies are used to determine bail and sentencing guidelines, based on an individual’s predicted recidivism risk (Lum and Isaac 2016; ProPublica 2016); mortgage and loan applications increasingly rely on machine learning to determine an individual’s loan-worthiness (Singh et al. 2022); medical decisions making is increasingly aided by black-box AI systems (London 2019); and high-stakes military decisions are often informed by automated decision-making systems. 

One common concern about this development is that automated decision-making systems may be biased in certain respects. But what does this mean? 

Definition: A system is biased when its behavior systematically deviates from some relevant standard. This standard can be moral, legal, epistemological, etc. (Epistemology is the branch of philosophy concerned with knowledge and justification: what can you know, and what is your justification for it?)

For example, a moral bias could exist when a system systematically deviates from some moral standard, while a statistical bias could exist when a system systematically deviates from statistical regularities observed in training data. Different kinds of bias can overlap, so that systems can be biased in some ways but not others, and can exhibit multiple different kinds of biases at once.

In general, automated decision-making systems have three main parts (diagram 1). They take some input (diagram 2), use sophisticated algorithms to process that data (diagram 3), and produce a decision about that input as an output (diagram 4). For example, an insurance company might use AI to determine whether to grant someone a loan. The system might take information about their salary, current job, and credit history, and their loan application, and output ‘yes’ or ‘no’ depending on whether they get the loan or not.

Why might an AI system undertake an action with such catastrophic consequences? One potential reason is that its values do not align with ours. By values we mean the ends or goals that something pursues. So, for example, common human values include things like physical comfort, safety, meaningful relationships, etc. Humans also value the means to these ends, things like money or status, which can be used to attain these valuable goods. Value alignment occurs when two or more agents share the same values. So, for example, human beings in general tend to share many values, even as they disagree about others. Most human beings value the continuation of the human species, though they may disagree about what direction that future should take. But in other cases, our values may fail to align. This can create conflicts. At a fundamental level, wars are waged because of failures of value alignment: the two countries value different things and see military conflict as the only means of resolving the disagreement.

An influential group of AI researchers is concerned with the catastrophic risks posed by AI due to the value alignment problem. The value alignment problem is the technical problem of ensuring that future AI systems act in accordance with human values, broadly understood. The concern is that in the near-to-mid-term future, we may develop very powerful AI systems that do not share human values. This includes the possibility of superintelligent AI systems that outperform human beings on essentially every task. The worry is that these advanced AI systems may act in ways that are contrary to human values, with potentially catastrophic results. In the following video, I will explain the contours of this problem by discussing a well-known thought experiment: the paperclip maximizer, a thought experiment discussed by Nick Bostrom and Eliezier Yudkowsky (Bostrom 2003)
 

I will consider these issues in order.

First, the term ‘human values’ is unclear, and there is no obvious way of making it precise. There is no consensus about many of the fundamental moral values that would be relevant for specifying the goals of an AI system. For instance, some moral philosophers endorse radical versions of consequentialism, or the ethical position that only the consequences of an action—and not other factors such as its motivation, or whether it violates intuitive principles of justice—are morally relevant. For a consequentialist, sacrificing the welfare of some to provide a greater benefit to others is not just morally acceptable, but is in fact morally obligatory. Many others reject this consequentialist viewpoint, holding instead that each person has certain unalienable rights that cannot be interfered with even for some ‘greater good.’

Second, even if we came to an agreement about the nature of value, we would still face the difficult practical challenge of instilling these values in AI systems. This is easier said than done. The current machine learning strategies for creating AI systems are infamously opaque. It is currently an unsolved problem knowing what rules or goals AI systems created by machine learning methods are in fact employing. A system trained to sort resumes based on the projected aptitude of the applicants may learn problematic rules such as favoring applicants with the types of names which have traditionally been hired for these positions in the past. Only a careful review of the output of such systems will reveal that they are using inappropriate procedures. Likewise, a system may appear to act in ways that conform to human values in training but later reveal itself to be acting in ways that are contrary to human values when it is released into the world.

Third, even if we determine the ‘correct’ values and figure out how to program them into an AI system, there are potential unintended consequences. Suppose the ultimate moral goal is to reduce suffering and we give this goal to an AI system. It might reason that the best way to achieve this goal is simply eliminate all conscious beings, since without consciousness there can be no suffering. Or perhaps the ultimate moral goal is to produce as much pleasure as possible, but this is most easily done by hooking human beings up to a device which artificially stimulates pleasure centres in the brain. These scenarios may seem outlandish, but arguably they are logical extensions of widely accepted ethical theories. We cannot easily know in advance whether a highly intelligent and capable AI system will share our ‘common sense’ aversion to these ideas.

For these reasons, the value alignment problem poses a serious—even potentially catastrophic—risk. Without some way of reliably instilling appropriate values in AI systems, we are at the mercy of whatever values they happen to develop. Given the potentially superhuman capabilities of these systems, this poses a significant risk for humanity.