Philosophy, Ethics, and the Future of AI

Philosophers had been asking how minds work for two millennia before there was a computer to make the question urgent. Now there is one, and the field it defines inherits two distinct questions that are easy to confuse. The first is whether a machine can act as if it were intelligent — pass whatever behavioral test we care to set. Philosophers call this claim weak AI. The second is whether a machine that acts intelligently is actually thinking, with a real, conscious mind, rather than merely simulating one; this is strong AI.¹ Most of the research in this course takes weak AI for granted and does not much care about strong AI: as long as the agent works, whether you call it a simulation of intelligence or the real thing changes nothing about the code. But every builder of such systems should care about their consequences, and that concern — ethics — is a third question that stands apart from both.

This lesson and its companion together close the course. They step back from algorithms to ask what they mean, what they risk, and where they might go. This part takes up the two questions of what a machine can be — whether it can act intelligently, and whether it can really think.

The debate spans decades. Turing framed the behavioral question in 1950; the field organized around it at Dartmouth in 1956; Searle mounted the sharpest attack on the machinery in 1980; and the empirical facts changed when deep networks began to pass hard perceptual tests around 2012, with conversational systems following a decade later. The philosophy did not resolve — it acquired working examples.

Weak AI: can machines act intelligently?

The 1956 workshop that founded the field asserted that every aspect of learning or any other feature of intelligence can be so precisely described that a machine can be made to simulate it.² The field thus rested from the start on the assumption that weak AI is possible. Whether it is possible depends on the definition. Under the definition used throughout this course — AI as the search for the best agent program on a given architecture — weak AI is trivially possible in principle: a digital architecture with $k$ bits of program storage admits exactly $2^k$ agent programs, and finding the best one is a finite (if astronomical) search.² Philosophers deal with the theoretical, not the practical, so the engineering answer is yes.

That answer is unsatisfying because the traditional question was posed differently: not can we maximize expected utility? but can machines think? Edsger Dijkstra remarked that the question of whether machines can think is about as relevant as the question of whether submarines can swim.³ Both are questions about the usage of English words, not about the design or capabilities of the artifact. Turing sidestepped the verbal dispute entirely: instead of asking whether machines can think, ask whether a machine can pass a behavioral test.

The setup removes every cue except the conversation itself. The interrogator sees no face, hears no voice, and knows only that one hidden respondent is a person and one is a program. The judgment is forced onto behavior alone, which is precisely Turing's point: replace the metaphysical question does it think? with an operational one anyone can score.

Turing anticipated nearly every objection to thinking machines that would be raised in the following half-century, and examined them himself. Three are worth working through, because each names a real intuition and each has a real rebuttal.

The argument from disability

The argument from disability claims a machine can never do $X$, where Turing's own list of $X$'s runs to: be kind, resourceful, beautiful, friendly, have initiative, have a sense of humor, tell right from wrong, make mistakes, fall in love, learn from experience, use words properly, be the subject of its own thought, do something really new.⁵ In retrospect many of these are easy. We are thoroughly familiar with computers that make mistakes. Programs play chess and checkers, inspect parts on assembly lines, steer cars, diagnose diseases, and make small but genuine discoveries in mathematics and the sciences — each at the level of a human expert.

The deeper point is that the objection confuses behavior with the mental processes behind it. As far back as 1955, Paul Meehl studied trained experts making subjective judgments — predicting a student's success, or a criminal's recidivism — and found that in 19 of 20 studies simple statistical procedures like linear regression predicted better than the experts.⁵ Tasks that seem to demand human insight often yield to a learning algorithm. That doesn't show the machine uses insight; it shows that one's first guess about which tasks require insight is often wrong.

The mathematical objection

Through the work of Turing (1936) and Gödel (1931), certain questions are known to be unanswerable by particular formal systems. Gödel's incompleteness theorem, which we met in the study of logical inference, says that for any formal axiomatic system $F$ powerful enough to do arithmetic, one can construct a Gödel sentence $G(F)$ with two properties: $G(F)$ is a sentence of $F$ but cannot be proved within $F$, and if $F$ is consistent, then $G(F)$ is true.⁶ Philosophers such as J. R. Lucas argued that this makes machines mentally inferior to humans: a machine is a formal system and so cannot establish the truth of its own Gödel sentence, whereas a human supposedly can.

There are three problems with the claim.⁶ First, Gödel's theorem applies only to formal systems powerful enough to do arithmetic, which requires infinite capacity; a physical computer is a finite system describable in propositional logic, to which the theorem does not apply. Second, an agent should not be ashamed of a sentence it cannot assert while others can. Consider:

J. R. Lucas cannot consistently assert that this sentence is true.

If Lucas asserted it he would contradict himself, so he cannot — and hence it is true. There is a true sentence Lucas cannot assert while other people (and machines) can, and this does not make us think less of Lucas. The construction is the same self-reference that generates the Gödel sentence: the argument that a human sees a truth a formal system cannot prove works just as well with the human as the system.

Third, and most important, there is no evidence that humans are immune to the very limitations they attribute to machines. It is impossible to prove that humans are not subject to Gödel's theorem, since any such proof would require formalizing the allegedly unformalizable human talent and so refute itself. The appeal that we must assume our own consistency fails on the facts: humans are known to be inconsistent, even in careful mathematical thought. Alfred Kempe's 1879 proof of the four-color theorem was accepted for eleven years before Percy Heawood found the flaw, and the theorem stood unproved until 1977.

The argument from informality

The most persistent criticism of AI as an enterprise is the argument from informality: human behavior is far too complex to be captured by any set of rules, and since computers do nothing but follow rules, they cannot behave as intelligently as humans. The inability to capture everything in logical rules is the qualification problem we met in classical planning.⁷ Its principal proponent, Hubert Dreyfus, aimed his critique at what John Haugeland named Good Old-Fashioned AI (GOFAI) — the view that all intelligent behavior can be captured by a system reasoning logically from facts and rules, corresponding to the simplest logical agent.

Dreyfus is right that a purely logical agent is vulnerable to the qualification problem. But the critique lands on one particular way of programming, not on computers as such. Probabilistic reasoning is far better suited to open-ended domains than rigid rule-lists, and machine learning generates behavior no one wrote down as a rule.⁷ Under Dreyfus's own view, human expertise is a holistic context or background — a grandmaster does not deliberate over each move but sees the board as demanding a certain response. That the process is not open to introspection does not mean it does not exist. Dreyfus and Dreyfus later proposed a five-stage model of expertise that ends in instantaneous, non-rule-based response — and in doing so proposed a neural-network architecture, moving from AI critics to AI theorists.

One of Dreyfus's strongest points survives as an insight rather than an objection: the case for situated agents over disembodied inference engines. An agent whose grasp of dog comes only from sentences like $\text{Dog}(x)\Rightarrow \text{Mammal}(x)$ is at a disadvantage to one that has watched dogs run, played fetch, and been licked by one. This is the embodied cognition thesis: cognition takes place within a body embedded in an environment, and the brain augments its reasoning by referring to the world rather than modeling it exhaustively. Under this view perception and vision and robotics become central to intelligence, not peripheral.

Strong AI: can machines really think?

Grant that a machine passes every behavioral test. Many philosophers hold it would still not be actually thinking — only simulating thought. Turing foresaw this too, citing Professor Geoffrey Jefferson's demand that a machine write a sonnet or compose a concerto because of thoughts and emotions felt, and not by the chance fall of symbols. This is the argument from consciousness: the machine must be aware of its own mental states.⁸ Related objections turn on phenomenology (the machine must actually feel) and intentionality (whether the machine's purported beliefs and desires are genuinely about things in the world).

Turing's reply was not to prove machines conscious but to note that the question is as ill-defined as can machines think? We never have direct evidence of the inner mental states of other humans either; in ordinary life we simply adopt the polite convention that everyone thinks. A future in which such conversations with machines are commonplace might dissolve the distinction between real and artificial thinking, just as the 1848 synthesis of artificial urea dissolved the supposed line between organic and inorganic chemistry once chemists agreed the synthetic product was urea, having all the same properties.⁸

Against dissolving the question, one may insist there is a genuine factual issue: humans have real minds, and machines might or might not. Settling it requires understanding how humans have minds at all — the mind–body problem.

The positions form a small tree. Dualism splits mind off as a separate substance and is largely abandoned in modern philosophy of mind. Physicalism keeps everything physical, and then divides on what makes a mental state the state it is. Functionalism says it is the causal role — the pattern of inputs, outputs, and internal transitions — which makes strong AI possible in principle, since a program can reproduce that role on any substrate. Biological naturalism says it is the specific physical stuff — the causal powers of actual neurons — which denies that reproducing the role on silicon is enough. The whole strong-AI dispute lives on this one branch.

The brain replacement experiment

If physicalism is right, the proper description of a mental state is determined by the brain state producing it. Functionalism sharpens this: a mental state is any intermediate causal condition between input and output, so any two systems with isomorphic causal processes have the same mental states — and a program could then have the mental states of a person.¹⁰ The claim is dramatized by the brain replacement experiment: replace the neurons in a brain one at a time with electronic devices that perfectly mimic their input–output behavior. By construction the external behavior is unchanged. What happens to the inner experience?

Intuitions clash sharply. The functionalist (Moravec) is convinced consciousness would persist; the biological naturalist (Searle) is convinced it would slowly vanish — that you would find yourself losing control, hearing your voice say I see a red object while your conscious experience shrank to nothing.¹⁰ But if awareness shrank while behavior stayed fixed, the shrinking would have to be removed from behavior instantaneously and totally, which is an odd thing to happen from gradual neuron-at-a-time replacement. And if we poke the electronic subject and it says ouch, that response must have an explanation appealing only to the functional properties of the neurons — an explanation that must apply equally to the original biological brain. Reducing consciousness to something with no causal connection to behavior makes it epiphenomenal: a shadow that the brain says 'ouch' is not caused by, which is deeply counterintuitive.

The dilemma is sharp and has only two horns. Either consciousness does affect behavior, in which case a functional duplicate that reproduces all the behavior must reproduce the consciousness that helps cause it — and functionalism wins. Or consciousness does not affect behavior, in which case it is epiphenomenal, a by-product that could be subtracted with no observable trace — and we are left unable to explain why we ever talk about it, since our talking is caused by brain states, not by the experience itself. The biological naturalist wants to deny both horns: to say the electronic brain lacks the experience and that this absence would eventually show. The gradual-replacement construction is designed to make that combination hard to hold, because it forces any behavioral difference to appear from a single neuron-sized swap.

Biological naturalism and the Chinese Room

Searle's biological naturalism holds that mental states are high-level features caused by low-level physical processes in the neurons, and that it is the unspecified causal properties of those neurons that matter. Running the right program on the wrong substrate is therefore not sufficient for a mind. His argument for this is the Chinese Room.¹¹

The most cited reply is the systems reply: asking whether the person understands Chinese is like asking whether a CPU can take cube roots. In both cases the answer is no, and in both cases the whole system has the capacity in question. The person is one component; understanding is a property of the person-plus-rulebook-plus-paper. Searle's rejoinder — let the person memorize the rule book, so there is no external system — trades on the intuition that a property of the whole must reside in one of its parts. Yet water is wet though neither hydrogen nor oxygen is.

Traced as a chain, the exchange is a sequence of moves and counter-moves, each answering the last, and the residue is an intuition standoff rather than a settled verdict.

The four axioms make the shape of the disagreement explicit. Axioms 1, 2, and 4 are widely granted; the whole argument rests on axiom 3, and axiom 3 is close to the conclusion it is used to reach.

Searle's real argument rests on four axioms: (1) programs are formal (syntactic); (2) minds have mental contents (semantics); (3) syntax by itself is neither constitutive of nor sufficient for semantics; (4) brains cause minds. From the first three he concludes programs are not sufficient for minds; from the fourth, that any artificial mind would need causal powers equivalent to those of brains.¹¹ Axiom 3 is the load-bearing one, and it is essentially a denial of functionalism — close to what it sets out to prove. Daniel Dennett called the Chinese Room an intuition pump: it amplifies prior intuitions, so biological naturalists leave more convinced and functionalists leave convinced only that axiom 3 is unsupported. It has stirred up combatants without changing many minds. And even granting the argument, it shows only that the room is not a mind by virtue of running the program — it says nothing about how to decide what is a mind, leaving everyone with only their intuitions.

Consciousness, qualia, and the explanatory gap

Running through every debate about strong AI is consciousness — and within it the part that resists all accounts, subjective experience: why it should feel like something to have certain brain states (tasting a strawberry) but presumably not others (being a rock). The technical term for the intrinsic character of an experience is qualia.¹²

Qualia challenge not just functionalism but all of science. Suppose brain research were complete — every neural process mapped, every molecule accounted for. There is no currently accepted form of reasoning that would carry us from such findings to any conclusion about a subject's inner experience. This explanatory gap leads some philosophers to hold that humans are simply incapable of understanding their own consciousness; Dennett in particular closes the gap by denying qualia exist at all. Turing conceded consciousness is a mystery but denied it need be solved before we answer the practical question: he wanted programs that behave intelligently, and so do we. Making them conscious is a further project, and not one we are equipped either to carry out or to verify.

Whether a machine can act intelligently, and whether it can really think, are the questions of what AI can do. They leave untouched the question of what it should do — and once systems act competently in the world, that becomes the pressing one. The ethics and risks of building these systems, the problem of keeping them aligned with human values, and where the agent components could go from here are taken up in the companion lesson, The Ethics and Future of AI, which also closes the course by tying its six modules into one account of intelligence as rational agency.