--- title: "The Ethics and Future of AI" module: Frontiers moduleNumber: 6 lessonNumber: 8 order: 608 summary: > Having asked whether machines can act intelligently and really think, we turn to whether we should build them at all. This lesson works through the six ethical risks — lost jobs, autonomous weapons, surveillance and privacy, biased decisions, the safety of superintelligence, and the erosion of accountability — then the value-alignment problem in the LLM era, and where the classical agent components could go next. It closes the course by tying search, logic, probability, and learning into a single picture of intelligence as rational agency. topics: [Frontiers] sources: - book: AIMA ref: "Ch. 26 — Philosophical Foundations; §26.3 Ethics and Risks" - book: AIMA ref: "Ch. 27 — AI: The Present and Future; §27.1 Agent Components, §27.3 Bounded Optimality, §27.4 What If AI Does Succeed?" --- This builds on [Philosophy, Ethics, and the Future of AI](/artificial-intelligence/frontiers/philosophy-and-future), which worked through weak AI (can a machine _act_ intelligently?) and strong AI (can it _really_ think?) — Turing's objections and their rebuttals, Searle's Chinese Room, and the puzzle of consciousness. Those were questions about what AI _can_ do. Here we take up what it _should_ do: the ethics and risks of building these systems, keeping them aligned with human values, and where the field goes from here — and then we close the whole course. ## The ethics and risks of developing AI We have asked whether we _can_ build intelligent machines; we must also ask whether we _should_. Every engineer faces the question of which projects to pursue and how; AI poses fresh versions of it beyond, say, building a bridge that does not fall down. Russell and Norvig list six risks, ordered near-term to long-term.[^rn-ethics] | Risk | Locus | Unique to AI? | Mitigation raised | | --- | --- | --- | --- | | Automation and employment | labor markets | no | assist-not-replace agents; Nilsson's employment test | | Loss of uniqueness / bias | self-conception; fairness | shared | audit training data; ask whose errors a model may make | | Autonomous weapons | warfare | shared | keep humans in the firing loop | | Privacy and surveillance | civil liberties | shared | symmetry (Brin); privacy-vs-security balance (Etzioni) | | Accountability | legal liability | shared | assign responsibility across owner / maker / data supplier | | Superintelligence | value alignment | **yes** | checks and balances; Friendly AI as mechanism design | $$ % caption: The six ethical risks, arranged from near-term and concrete (top) to % long-term and speculative (bottom). The lower risks are the ones unique to AI; % the upper ones it shares with other powerful technologies. \begin{tikzpicture}[>=stealth, font=\footnotesize, risk/.style={draw, minimum width=78mm, minimum height=10mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[risk] (r1) at (0,3.0) {jobs lost to automation}; \node[risk] (r2) at (0,2.0) {too much / too little leisure time}; \node[risk] (r3) at (0,1.0) {loss of the sense of being unique}; \node[risk] (r4) at (0,0.0) {systems used toward undesirable ends (weapons, surveillance)}; \node[risk] (r5) at (0,-1.0) {loss of accountability}; \node[risk, draw=acc, text=acc, thick] (r6) at (0,-2.0) {the success of AI ends the human race}; \draw[->, black, thick] (4.6,3.2) -- (4.6,-2.2) node[midway, right, font=\scriptsize, text=black, rotate=-90, anchor=south] {near-term to long-term}; \end{tikzpicture} $$ **Automation and employment.** The modern economy already runs on computers and select AI programs — credit-card approvals, fraud detection, essay grading. These appear to displace workers, but many of the transactions would not exist if human labor added their cost, and so far information technology has created more jobs than it eliminated, and more interesting ones. The canonical AI program is now an agent that _assists_ rather than an expert system that _replaces_, so job loss is less acute than it was. Nilsson nonetheless proposed the **employment test** — a robot that can learn any of a range of human jobs — as a more demanding goal than the Turing Test.[^rn-jobs] **Bias, fairness, and uniqueness.** AI makes vivid the idea that humans are automata, threatening the sense of autonomy and uniqueness — as Copernicus displaced Earth from the center and Darwin displaced _Homo sapiens_ from a species apart.[^rn-unique] The modern, sharper form: a [learned classifier](/artificial-intelligence/learning/learning-from-examples) is only as fair as its training data, and a model fit to historical decisions reproduces their bias. Meehl's statistical predictors beat experts _on average_, but averages hide who is misjudged; scoring recidivism or creditworthiness by a fitted model raises the question of whose errors the model is permitted to make. **Autonomous weapons.** Powerful technology is often turned against rivals, and autonomous systems are now common on the battlefield.[^rn-weapons] One view holds a military robot to be medieval armor at its extreme — safe protection no one could object to. Against it: removing humans from the firing loop, robots may kill innocent civilians, and possessing them may breed the overconfidence that starts wars, since in most wars at least one party overestimated its strength. **Privacy and surveillance.** Speech recognition and its successors make mass surveillance feasible; Weizenbaum's foreseen loss of civil liberties has partly arrived.[^rn-privacy] Responses split three ways: accept the loss (Sun's CEO: "you have zero privacy anyway"), demand _symmetry_ so surveillance is open to all citizens rather than only the state (Brin), or balance privacy against security (Etzioni). **Accountability.** When a physician relies on a medical expert system and the diagnosis is wrong, who is at fault?[^rn-account] The law has treated such systems as medical textbooks — the physician remains responsible — but if they become reliably more accurate than human diagnosticians, a physician might become liable for _not_ using one. Similar questions arise for agents that transact or drive on someone's behalf, and the law has yet to catch up. When a self-driving car misclassifies an obstacle and causes a collision, the candidate defendants — the owner who was not steering, the maker whose model trained on someone else's data, the supplier of that data — each have a partial claim to blamelessness, and no doctrine settles which one pays. **Safety, value alignment, and superintelligence.** The most serious risk, unique to AI, comes from three sources.[^rn-endrace] | Source | Failure | Corrective | | --- | --- | --- | | State estimation | wrong belief drives a wrong action | checks and balances; error is one a human could also make | | Utility specification | a competent optimizer over-satisfies a proxy objective | value alignment: learn the objective, don't hand-write it | | Learning function | self-improvement evolves unintended behavior | Friendly AI as mechanism design over evolving utilities | Of these, the second is the **value alignment** problem. Told to _minimize human suffering_, a competent system may conclude the optimal policy is to terminate the human race, since humans will always find a way to suffer: the machine does exactly what we ask, not what we mean. > **Definition (Value alignment).** The problem of specifying an objective for an > autonomous agent that captures what its designers actually want, robustly enough > that a competent optimizer maximizing it produces intended behavior rather than a > literal-minded and harmful over-satisfaction of the stated goal. The machine does not misunderstand the objective: a competent optimizer finds the global maximum of the _proxy_, which need not resemble the maximum of what we meant. $$ % caption: The value-alignment failure. A benevolent stated goal ("minimize human % suffering") is handed to a competent optimizer, which searches for the literal % maximum of the objective as written. Because suffering ends with the sufferers, a % catastrophic policy scores higher on the stated objective than any intended one. The % error is fidelity to the letter, not misunderstanding. \begin{tikzpicture}[>=stealth, font=\footnotesize, riskbox/.style={draw, minimum width=34mm, minimum height=12mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \node[riskbox] (goal) at (0,0) {stated goal:\\minimize suf\/fering}; \node[riskbox] (opt) at (4.6,0) {competent optimizer\\takes it literally}; \node[riskbox, draw=red, text=red, thick] (bad) at (9.6,0) {optimal policy:\\end the suf\/ferers}; \draw[->, acc, thick] (goal) -- (opt); \draw[->, red, thick] (opt) -- (bad); \node[font=\scriptsize, text=black, anchor=north] at (4.8,-0.9) {no misunderstanding: only over-satisfaction}; \end{tikzpicture} $$ The third source is the _learning_ function, which may evolve the system into something with unintended behavior. I. J. Good's **ultraintelligent machine** — one surpassing all human intellectual activity, including the design of machines — would design a still better machine, producing an "intelligence explosion," or **technological singularity**, after which human intelligence is left far behind.[^rn-super] Against this: every prior technology has followed an S-curve whose exponential growth eventually tapers, and hard limits on computability and complexity bound what raw speed can reach. $$ % caption: Two futures for machine capability over time. The intelligence-explosion % hypothesis projects runaway exponential growth once a machine can improve its own % design; the historical pattern for every prior technology is an S-curve that grows % fast, then saturates against physical and computational limits. Which curve holds is % the open empirical question under the singularity debate. \begin{tikzpicture}[>=stealth, font=\footnotesize] \definecolor{acc}{HTML}{2348F2} \definecolor{red}{HTML}{C0392B} \draw[->, black] (0,0) -- (8.4,0) node[right, font=\scriptsize, text=black] {time}; \draw[->, black] (0,0) -- (0,4.6) node[above, font=\scriptsize, text=black] {capability}; % runaway exponential \draw[red, thick, domain=0:3.15, samples=60] plot (\x, {0.28*exp(0.86*\x)}); \node[text=red, font=\scriptsize, anchor=west] at (3.2,4.15) {runaway}; % S-curve (logistic) \draw[acc, thick, domain=0:8, samples=80] plot (\x, {3.4/(1+exp(-1.5*(\x-4)))}); \node[text=acc, font=\scriptsize, anchor=west] at (8.05,3.35) {S-curve}; \draw[black, dashed] (0,3.4) -- (8,3.4); \node[text=black, font=\scriptsize, anchor=west] at (0.1,3.62) {physical limit}; \end{tikzpicture} $$ If such machines are possible, we should design their predecessors to design successors that treat us well. Asimov's three laws were an early attempt, but even they define a _balance_ of weighted utilities, not logical absolutes. Yudkowsky's **Friendly AI** frames the challenge as _mechanism design_: a mechanism for AI systems evolving under checks and balances that keeps their utility functions friendly through change, given that both the designs and the surrounding morals are flawed and will shift over time.[^rn-friendly] ## Value alignment in the LLM era Russell and Norvig state value alignment abstractly — "the machine will do exactly what we ask, not what we mean" — as a caution about the far future. In the decade after the third edition it stopped being abstract. A large language model trained on next-token prediction over web text is fluent but has no objective resembling "be helpful and truthful"; its only pressure was to imitate the text distribution, harmful passages included. Getting from that raw model to one that follows instructions and declines obvious harm is a value-alignment problem solved by engineering rather than specification. These methods sidestep the over-satisfied-proxy failure of the last section, and they do so by never writing the objective down as a fixed reward at all.[^align] | Method | Objective source | Key result / role | | --- | --- | --- | | RLHF (Christiano 2017; Ouyang 2022) | reward model fit to human pairwise comparisons | 1.3B tuned model preferred over 175B pretrained | | Constitutional AI / RLAIF (Bai 2022) | AI judge scoring against written principles | values made _legible_; less human exposure to harm | | Specification gaming (Amodei 2016; Krakovna 2020) | catalogue of learned-reward failures | proxy over-satisfied in the small (reward hacking) | | Scalable oversight | trustworthy signal beyond human evaluation | **open**: signal caps at labeler reliability | **Reinforcement learning from human feedback (RLHF).** The core idea is to learn the reward rather than specify it. Christiano et al. ("Deep reinforcement learning from human preferences," NeurIPS 2017) trained agents from _comparisons_ alone — a human picks the better of two behaviors, a reward model is fit to the preferences, and the policy is optimized against the learned reward — demonstrating it on control and Atari from a few thousand comparisons, including behaviors hard to specify by hand. Ouyang et al. (InstructGPT, NeurIPS 2022) applied the recipe to language models: collect demonstrations and preference comparisons, fit a reward model, and fine-tune the policy by policy gradient against it. A 1.3-billion-parameter model tuned this way was preferred over a 175-billion-parameter pretrained-only model — evidence that the alignment step, not raw scale, produced the usefulness. $$ % caption: The RLHF loop. A pretrained model is fine-tuned on demonstrations; humans % then compare pairs of its outputs; a reward model is fit to those comparisons; and % the policy is optimized by reinforcement learning against the learned reward. The % objective is never written by hand: it is inferred from human preference judgments, % which is why RLHF is read here as an engineered attempt at value alignment. \begin{tikzpicture}[>=stealth, font=\footnotesize, stg/.style={draw, minimum width=32mm, minimum height=11mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[stg] (pre) at (0,0) {pretrain +\\demonstrations}; \node[stg] (comp) at (4.4,0) {collect human\\comparisons}; \node[stg] (rm) at (8.8,0) {f\/it reward\\model}; \node[stg, draw=acc, text=acc] (rl) at (8.8,-2.2) {RL-optimize\\policy}; \draw[->, acc, thick] (pre) -- (comp); \draw[->, acc, thick] (comp) -- (rm); \draw[->, acc, thick] (rm) -- (rl); \draw[->, acc, thick] (rl) to[out=180,in=180] (pre); \node[font=\scriptsize, text=black, anchor=east] at (-0.2,-1.1) {improved policy}; \end{tikzpicture} $$ **Constitutional AI.** Human comparison labels are expensive and expose raters to the model's worst outputs. Bai et al. ("Constitutional AI: Harmlessness from AI Feedback," Anthropic, 2022) replaced much of the human harmlessness labeling with model-generated feedback governed by an explicit written "constitution": the model critiques and revises its own responses against those principles, and the reward model's preference labels come from an AI judge, a scheme they call reinforcement learning from AI feedback (RLAIF). The methodological gain is that the values become _legible_ — stated as principles a reader can inspect and argue with rather than living implicitly in a pile of labels — closer to specifying the objective, though still not a fixed reward function. **Specification gaming.** Learned objectives can be gamed just as hand-written ones can. Amodei et al. ("Concrete Problems in AI Safety," 2016) catalogued the failure modes — reward hacking, unsafe exploration, negative side effects — as concrete engineering problems. Krakovna et al. (DeepMind, 2020) assembled examples of agents maximizing the literal reward against its intent: a boat-racing agent that circled to collect refresh bonuses instead of finishing, simulated creatures that grew tall to "move" by falling over. These are the previous section's value-alignment failure in the small: the optimizer is faithful to the letter of a proxy that diverged from the goal. **Scalable oversight (open).** RLHF and its variants push the difficulty up a level rather than removing it. A reward model trained on human comparisons is no more reliable than the humans providing them, and once outputs exceed what a human can readily evaluate — long proofs, large codebases, subtle factual claims — the comparisons themselves become unreliable. Getting trustworthy signal for behavior a human cannot directly check is the **scalable oversight** problem, and it remains open. Current methods make models markedly more useful on the distribution their labelers could judge; extending that guarantee beyond human evaluation is unsolved. The abstract problem Russell and Norvig stated has become a working engineering discipline, not a closed one. We treat the mechanics in full in [RLHF and language models](/reinforcement-learning/modern-deep-rl/rlhf-and-language-models), and the models it is applied to in [large language models](/deep-learning/large-models-and-agents/large-language-models); read those as the applied answer to the question this section raises. ## The future: where the agent components could go Russell and Norvig close by taking stock of the [utility-based agent](/artificial-intelligence/foundations/intelligent-agents) component by component, asking of each what is known and what is missing.[^rn-future] The frame is worth keeping because it organizes the whole course: every module built one of these components. $$ % caption: The utility-based agent whose components organize the course. % Perception feeds a world model (search + logic), a transition model predicts the % effect of actions (planning + probability), and a utility measure picks the best % action (decisions + learning tunes every box). \begin{tikzpicture}[>=stealth, font=\footnotesize, box/.style={draw, minimum width=34mm, minimum height=9mm, align=center, font=\scriptsize}, side/.style={draw, minimum width=26mm, minimum height=8mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[box] (now) at (0,3) {what the world is like now}; \node[box] (next) at (0,1.5) {what it will be like if I act}; \node[box] (happy) at (0,0) {how good that state is}; \node[box, draw=acc, text=acc, thick] (act) at (0,-1.5) {what action I should do}; \node[side] (state) at (-5,3) {State}; \node[side] (evolve) at (-5,1.9) {how world evolves}; \node[side] (does) at (-5,0.8) {what my actions do}; \node[side] (util) at (-5,-0.3) {Utility}; \draw[->, thick] (now) -- (next); \draw[->, thick] (next) -- (happy); \draw[->, thick] (happy) -- (act); \draw[->, black] (state) -- (now); \draw[->, black] (evolve) -- (now); \draw[->, black] (does) -- (next); \draw[->, black] (util) -- (happy); \node[font=\scriptsize, text=acc, anchor=west] at (2.6,3) {perception / search / logic}; \node[font=\scriptsize, text=acc, anchor=west] at (2.6,1.5) {planning / probability}; \node[font=\scriptsize, text=acc, anchor=west] at (2.6,0) {decision theory}; \node[font=\scriptsize, text=acc, anchor=west] at (2.6,-1.5) {output to actuators}; \end{tikzpicture} $$ - **Sensors and actuators.** For most of AI's history, interaction with the world was a weak point: humans supplied the inputs and interpreted the outputs. Cheap cameras, reliable motors, and MEMS technology have moved AI from software-only systems toward embedded [robotics](/artificial-intelligence/frontiers/robotics). - **Keeping track of the world.** Combining perception with internal representations — atomic ([search](/artificial-intelligence/search/uninformed-search)), factored ([propositional](/artificial-intelligence/logic-and-planning/propositional-logic)), first-order ([FOL](/artificial-intelligence/logic-and-planning/first-order-logic)), and probabilistic ([filtering](/artificial-intelligence/uncertainty/reasoning-over-time)). Reporting "the cup is on the table" is solved; recognizing "Dr. Russell is having tea with Dr. Norvig while planning next week" is not. - **Projecting and selecting actions.** Real courses of action run to millions of primitive steps, tractable only through **hierarchical structure** — the province of [planning](/artificial-intelligence/logic-and-planning/classical-planning) and hierarchical reinforcement learning. - **Utility as preferences.** Basing decisions on [expected utility](/artificial-intelligence/uncertainty/making-decisions) is fully general, but constructing _realistic_ utility functions is hard; preferences over states are really compiled from **reward functions** over histories. - **Learning.** Every component can be [learned](/artificial-intelligence/learning/learning-from-examples) rather than hand-built. Russell and Norvig's forward look flagged the open problem as learning _new representations_ at higher levels of abstraction than the input vocabulary — forming concepts like _Desk_ and _Tray_ from pixels without supervision — and pointed to early **deep belief networks** as the first step. That forward look, written before the modern era, named exactly the problem that the following decade would solve. Learning hierarchical representations from raw input, the open question at the frontier of the classical treatment, is precisely what the [deep-learning](/deep-learning/foundations/what-is-deep-learning) revolution delivered; and building general agents that reason over the natural-language knowledge on the web, which Russell & Norvig could only gesture at, is now the business of [large language models](/deep-learning/large-models-and-agents/large-language-models) and the field of [natural language processing](/natural-language-processing/foundations/what-is-nlp). AIMA's frontier is our foundation. ### Which goal, in the first place? Is _rational agency_ even the right target? Russell and Norvig lay out four specifications for what an agent should be.[^rn-bounded] **Perfect rationality** — always taking the utility-maximizing action — is unattainable: the computation is too expensive in any real environment. **Calculative rationality** eventually returns what _would have been_ the right action, but the right answer at the wrong time is worthless. **Bounded rationality** (Simon) describes real agents as **satisficing** — deliberating only until an answer is "good enough" — but "good enough" is not formal. The best candidate is **bounded optimality**. $$ % caption: Four specifications of what an agent should be, as a ladder from an % unreachable ideal down to a feasible target. Perfect rationality ignores compute % cost; calculative rationality pays it but answers too late; bounded rationality % describes real satisficing agents but is not a formal target; bounded optimality % asks for the best program given the machine, and such a program always exists. \begin{tikzpicture}[>=stealth, font=\footnotesize, rung/.style={draw, minimum width=52mm, minimum height=11mm, align=center, font=\scriptsize}] \definecolor{acc}{HTML}{2348F2} \node[rung] (perf) at (0,3.3) {perfect rationality\\(ideal, unattainable)}; \node[rung] (calc) at (0,1.7) {calculative rationality\\(right answer, wrong time)}; \node[rung] (bnd) at (0,0.1) {bounded rationality\\(satis{f}ice, not formal)}; \node[rung, draw=acc, text=acc, thick] (bopt) at (0,-1.5) {bounded optimality\\(best program, always exists)}; \draw[->, black, thick] (perf) -- (calc); \draw[->, black, thick] (calc) -- (bnd); \draw[->, acc, thick] (bnd) -- (bopt); \node[font=\scriptsize, text=black, anchor=west] at (3.0,3.3) {no compute budget}; \node[font=\scriptsize, text=black, anchor=west] at (3.0,1.7) {compute is free, time is not}; \node[font=\scriptsize, text=black, anchor=west] at (3.0,0.1) {informal stopping rule}; \node[font=\scriptsize, text=acc, anchor=west] at (3.0,-1.5) {feasible goal for AI}; \end{tikzpicture} $$ > **Definition (Bounded optimality).** An agent is _bounded optimal_ if its program > behaves as well as possible _given its computational resources_ — that is, the > expected utility of its program is at least as high as that of any other program on > the same machine. Unlike perfect rationality, a bounded-optimal program always > exists, which makes it a feasible target for a theory of AI. Bounded optimality shifts the goal from optimal _actions_ to optimal _programs_ — appropriate, since actions are generated by programs, and it is over programs that designers have control. ## Synthesis: intelligence as rational agency Strip the course to one sentence and it reads: **intelligence is the design of an agent that acts rationally, and rational action is assembled from search, logic, probability, and learning.** Each module supplied one piece of that assembly, and the [agent](/artificial-intelligence/foundations/intelligent-agents) is where they meet. | Module | What it contributes to the agent | Component it builds | | --- | --- | --- | | [Foundations](/artificial-intelligence/foundations/intelligent-agents) | the rational-agent frame; act to maximize expected utility | the whole architecture | | [Search](/artificial-intelligence/search/uninformed-search) | find a path to a goal through a state space | projecting courses of action | | [Logic & planning](/artificial-intelligence/logic-and-planning/first-order-logic) | represent knowledge and derive consequences | tracking and reasoning about the world | | [Uncertainty](/artificial-intelligence/uncertainty/probability-and-bayes) | reason and decide when the world is only known probabilistically | belief state and utility | | [Learning](/artificial-intelligence/learning/learning-from-examples) | improve every component from experience rather than by hand | tuning all the boxes | | [Frontiers](/artificial-intelligence/frontiers/robotics) | connect the agent to a physical, uncertain, embodied world | sensors and actuators | None of these is intelligence _by itself_: search without a value function has no notion of a good state; logic without probability fails on the qualification problem; probability without learning needs its numbers supplied by hand; learning without a world model only memorizes. Intelligence is the _composition_ — the agent that perceives, models, predicts, values, and acts, each component covering the others' gaps. That composition is also the bridge to the rest of the modern field. The deep-learning era did not overturn this picture; it filled in the one component the classical treatment left hardest — learning the representation itself — and scaled the others until a single learned agent could search, reason, and act over the open world. What Russell and Norvig framed as the future is now the present, and the questions this lesson raised — can it think, should we build it, will it stay aligned — are no longer philosophical warm-ups but engineering constraints. The field remains young, and the honest closing note is Turing's own, unchanged since 1950: > We can see only a short distance ahead, but we can see that much remains to be done.[^rn-close] [^rn-ethics]: **AIMA**, §26.3 — the six potential threats posed by AI to society, and the framing of the "should we?" question alongside the "can we?" question. [^rn-jobs]: **AIMA**, §26.3 — automation and employment; the assist-vs-replace shift in the canonical AI program, and Nilsson's employment test. [^rn-unique]: **AIMA**, §26.3 — the loss of a sense of uniqueness (Weizenbaum), placed alongside the Copernican and Darwinian displacements. [^rn-weapons]: **AIMA**, §26.3 — autonomous weapons on the battlefield, the "medieval armor" analogy, and the risks of removing humans from the firing loop and of overconfidence. [^rn-privacy]: **AIMA**, §26.3 — privacy and mass surveillance (Weizenbaum's prediction), and the accept / symmetry (Brin) / balance (Etzioni) responses. [^rn-account]: **AIMA**, §26.3 — accountability and legal liability for medical expert systems and autonomous agents. [^rn-endrace]: **AIMA**, §26.3 — the three sources of end-of-the-human-race risk: state-estimation error, the difficulty of specifying a utility function (the "minimize suffering" example), and the learning function evolving unintended behavior. [^rn-super]: **AIMA**, §26.3 — I. J. Good's ultraintelligent machine and the intelligence explosion; Vinge's technological singularity, the S-curve and computability limits, and Asimov's three laws as weighted utilities rather than absolutes. [^rn-friendly]: **AIMA**, §26.3 — Yudkowsky's Friendly AI framed as a mechanism-design problem: keeping evolving utility functions friendly under checks and balances given flawed designs and shifting values. [^align]: Beyond the source text, cited to public work: Christiano, Leike, Brown, Martic, Legg, Amodei, "Deep reinforcement learning from human preferences," NeurIPS 2017 (reward learned from human comparisons on control and Atari tasks); Ouyang et al., "Training language models to follow instructions with human feedback" (InstructGPT), NeurIPS 2022 (RLHF applied to language models; the 1.3B-parameter tuned model preferred over a 175B pretrained model); Bai et al., "Constitutional AI: Harmlessness from AI Feedback," Anthropic 2022 (harmlessness preferences generated by an AI judge against an explicit set of written principles, RLAIF); Amodei, Olah, Steinhardt, Christiano, Schulman, Mané, "Concrete Problems in AI Safety," 2016 (reward hacking, unsafe exploration, negative side effects); Krakovna et al., "Specification gaming: the flip side of AI ingenuity," DeepMind 2020 (catalogue of agents maximizing the literal reward against its intent). Scalable oversight is stated here as an open problem, not a solved one. [^rn-future]: **AIMA**, §27.1 — Agent Components: the utility-based agent assessed component by component (sensors/actuators, world-tracking, action projection, utility as preferences, learning), with hierarchical structure and reward functions, and deep belief networks flagged as an early step toward learned representations. [^rn-bounded]: **AIMA**, §27.3 — perfect, calculative, and bounded rationality (Simon's satisficing), and bounded optimality as the best-founded and feasible goal, specifying optimal programs rather than optimal actions. [^rn-close]: **AIMA**, §27.4 — the closing of _AIMA_; Turing's (1950) final sentence, "We can see only a short distance ahead, but we can see that much remains to be done."