The Ethics and Future of AI

This builds on Philosophy, Ethics, and the Future of AI, which worked through weak AI (can a machine act intelligently?) and strong AI (can it really think?) — Turing's objections and their rebuttals, Searle's Chinese Room, and the puzzle of consciousness. Those were questions about what AI can do. Here we take up what it should do: the ethics and risks of building these systems, keeping them aligned with human values, and where the field goes from here — and then we close the whole course.

The ethics and risks of developing AI

We have asked whether we can build intelligent machines; we must also ask whether we should. Every engineer faces the question of which projects to pursue and how; AI poses fresh versions of it beyond, say, building a bridge that does not fall down. Russell and Norvig list six risks, ordered near-term to long-term.¹

Automation and employment. The modern economy already runs on computers and select AI programs — credit-card approvals, fraud detection, essay grading. These appear to displace workers, but many of the transactions would not exist if human labor added their cost, and so far information technology has created more jobs than it eliminated, and more interesting ones. The canonical AI program is now an agent that assists rather than an expert system that replaces, so job loss is less acute than it was. Nilsson nonetheless proposed the employment test — a robot that can learn any of a range of human jobs — as a more demanding goal than the Turing Test.²

Bias, fairness, and uniqueness. AI makes vivid the idea that humans are automata, threatening the sense of autonomy and uniqueness — as Copernicus displaced Earth from the center and Darwin displaced Homo sapiens from a species apart.³ The modern, sharper form: a learned classifier is only as fair as its training data, and a model fit to historical decisions reproduces their bias. Meehl's statistical predictors beat experts on average, but averages hide who is misjudged; scoring recidivism or creditworthiness by a fitted model raises the question of whose errors the model is permitted to make.

Autonomous weapons. Powerful technology is often turned against rivals, and autonomous systems are now common on the battlefield.⁴ One view holds a military robot to be medieval armor at its extreme — safe protection no one could object to. Against it: removing humans from the firing loop, robots may kill innocent civilians, and possessing them may breed the overconfidence that starts wars, since in most wars at least one party overestimated its strength.

Privacy and surveillance. Speech recognition and its successors make mass surveillance feasible; Weizenbaum's foreseen loss of civil liberties has partly arrived.⁵ Responses split three ways: accept the loss (Sun's CEO: you have zero privacy anyway), demand symmetry so surveillance is open to all citizens rather than only the state (Brin), or balance privacy against security (Etzioni).

Accountability. When a physician relies on a medical expert system and the diagnosis is wrong, who is at fault?⁶ The law has treated such systems as medical textbooks — the physician remains responsible — but if they become reliably more accurate than human diagnosticians, a physician might become liable for not using one. Similar questions arise for agents that transact or drive on someone's behalf, and the law has yet to catch up. When a self-driving car misclassifies an obstacle and causes a collision, the candidate defendants — the owner who was not steering, the maker whose model trained on someone else's data, the supplier of that data — each have a partial claim to blamelessness, and no doctrine settles which one pays.

Safety, value alignment, and superintelligence. The most serious risk, unique to AI, comes from three sources.⁷

Of these, the second is the value alignment problem. Told to minimize human suffering, a competent system may conclude the optimal policy is to terminate the human race, since humans will always find a way to suffer: the machine does exactly what we ask, not what we mean.

The machine does not misunderstand the objective: a competent optimizer finds the global maximum of the proxy, which need not resemble the maximum of what we meant.

The third source is the learning function, which may evolve the system into something with unintended behavior. I. J. Good's ultraintelligent machine — one surpassing all human intellectual activity, including the design of machines — would design a still better machine, producing an intelligence explosion, or technological singularity, after which human intelligence is left far behind.⁸ Against this: every prior technology has followed an S-curve whose exponential growth eventually tapers, and hard limits on computability and complexity bound what raw speed can reach.

If such machines are possible, we should design their predecessors to design successors that treat us well. Asimov's three laws were an early attempt, but even they define a balance of weighted utilities, not logical absolutes. Yudkowsky's Friendly AI frames the challenge as mechanism design: a mechanism for AI systems evolving under checks and balances that keeps their utility functions friendly through change, given that both the designs and the surrounding morals are flawed and will shift over time.⁹

Value alignment in the LLM era

Russell and Norvig state value alignment abstractly — the machine will do exactly what we ask, not what we mean — as a caution about the far future. In the decade after the third edition it stopped being abstract. A large language model trained on next-token prediction over web text is fluent but has no objective resembling be helpful and truthful; its only pressure was to imitate the text distribution, harmful passages included. Getting from that raw model to one that follows instructions and declines obvious harm is a value-alignment problem solved by engineering rather than specification. These methods sidestep the over-satisfied-proxy failure of the last section, and they do so by never writing the objective down as a fixed reward at all.¹⁰

Reinforcement learning from human feedback (RLHF). The core idea is to learn the reward rather than specify it. Christiano et al. (Deep reinforcement learning from human preferences, NeurIPS 2017) trained agents from comparisons alone — a human picks the better of two behaviors, a reward model is fit to the preferences, and the policy is optimized against the learned reward — demonstrating it on control and Atari from a few thousand comparisons, including behaviors hard to specify by hand. Ouyang et al. (InstructGPT, NeurIPS 2022) applied the recipe to language models: collect demonstrations and preference comparisons, fit a reward model, and fine-tune the policy by policy gradient against it. A 1.3-billion-parameter model tuned this way was preferred over a 175-billion-parameter pretrained-only model — evidence that the alignment step, not raw scale, produced the usefulness.

Constitutional AI. Human comparison labels are expensive and expose raters to the model's worst outputs. Bai et al. (Constitutional AI: Harmlessness from AI Feedback, Anthropic, 2022) replaced much of the human harmlessness labeling with model-generated feedback governed by an explicit written constitution: the model critiques and revises its own responses against those principles, and the reward model's preference labels come from an AI judge, a scheme they call reinforcement learning from AI feedback (RLAIF). The methodological gain is that the values become legible — stated as principles a reader can inspect and argue with rather than living implicitly in a pile of labels — closer to specifying the objective, though still not a fixed reward function.

Specification gaming. Learned objectives can be gamed just as hand-written ones can. Amodei et al. (Concrete Problems in AI Safety, 2016) catalogued the failure modes — reward hacking, unsafe exploration, negative side effects — as concrete engineering problems. Krakovna et al. (DeepMind, 2020) assembled examples of agents maximizing the literal reward against its intent: a boat-racing agent that circled to collect refresh bonuses instead of finishing, simulated creatures that grew tall to move by falling over. These are the previous section's value-alignment failure in the small: the optimizer is faithful to the letter of a proxy that diverged from the goal.

Scalable oversight (open). RLHF and its variants push the difficulty up a level rather than removing it. A reward model trained on human comparisons is no more reliable than the humans providing them, and once outputs exceed what a human can readily evaluate — long proofs, large codebases, subtle factual claims — the comparisons themselves become unreliable. Getting trustworthy signal for behavior a human cannot directly check is the scalable oversight problem, and it remains open. Current methods make models markedly more useful on the distribution their labelers could judge; extending that guarantee beyond human evaluation is unsolved. The abstract problem Russell and Norvig stated has become a working engineering discipline, not a closed one.

We treat the mechanics in full in RLHF and language models, and the models it is applied to in large language models; read those as the applied answer to the question this section raises.

The future: where the agent components could go

Russell and Norvig close by taking stock of the utility-based agent component by component, asking of each what is known and what is missing.¹¹ The frame is worth keeping because it organizes the whole course: every module built one of these components.

• Sensors and actuators. For most of AI's history, interaction with the world was a weak point: humans supplied the inputs and interpreted the outputs. Cheap cameras, reliable motors, and MEMS technology have moved AI from software-only systems toward embedded robotics.
• Keeping track of the world. Combining perception with internal representations — atomic (search), factored (propositional), first-order (FOL), and probabilistic (filtering). Reporting the cup is on the table is solved; recognizing Dr. Russell is having tea with Dr. Norvig while planning next week is not.
• Projecting and selecting actions. Real courses of action run to millions of primitive steps, tractable only through hierarchical structure — the province of planning and hierarchical reinforcement learning.
• Utility as preferences. Basing decisions on expected utility is fully general, but constructing realistic utility functions is hard; preferences over states are really compiled from reward functions over histories.
• Learning. Every component can be learned rather than hand-built. Russell and Norvig's forward look flagged the open problem as learning new representations at higher levels of abstraction than the input vocabulary — forming concepts like Desk and Tray from pixels without supervision — and pointed to early deep belief networks as the first step.

That forward look, written before the modern era, named exactly the problem that the following decade would solve. Learning hierarchical representations from raw input, the open question at the frontier of the classical treatment, is precisely what the deep-learning revolution delivered; and building general agents that reason over the natural-language knowledge on the web, which Russell & Norvig could only gesture at, is now the business of large language models and the field of natural language processing. AIMA's frontier is our foundation.

Which goal, in the first place?

Is rational agency even the right target? Russell and Norvig lay out four specifications for what an agent should be.¹² Perfect rationality — always taking the utility-maximizing action — is unattainable: the computation is too expensive in any real environment. Calculative rationality eventually returns what would have been the right action, but the right answer at the wrong time is worthless. Bounded rationality (Simon) describes real agents as satisficing — deliberating only until an answer is good enough — but good enough is not formal. The best candidate is bounded optimality.

Bounded optimality shifts the goal from optimal actions to optimal programs — appropriate, since actions are generated by programs, and it is over programs that designers have control.

Synthesis: intelligence as rational agency

Strip the course to one sentence and it reads: intelligence is the design of an agent that acts rationally, and rational action is assembled from search, logic, probability, and learning. Each module supplied one piece of that assembly, and the agent is where they meet.

None of these is intelligence by itself: search without a value function has no notion of a good state; logic without probability fails on the qualification problem; probability without learning needs its numbers supplied by hand; learning without a world model only memorizes. Intelligence is the composition — the agent that perceives, models, predicts, values, and acts, each component covering the others' gaps.

That composition is also the bridge to the rest of the modern field. The deep-learning era did not overturn this picture; it filled in the one component the classical treatment left hardest — learning the representation itself — and scaled the others until a single learned agent could search, reason, and act over the open world. What Russell and Norvig framed as the future is now the present, and the questions this lesson raised — can it think, should we build it, will it stay aligned — are no longer philosophical warm-ups but engineering constraints.

The field remains young, and the honest closing note is Turing's own, unchanged since 1950:

We can see only a short distance ahead, but we can see that much remains to be done.¹³