# Deep Learning

Deep learning is how computers learn patterns too subtle to program by
hand — turning pixels into objects, audio into words, and prompts into
prose.


_Figure 001 — Neural super-sampling: a network upscales a cheap low-res frame into a
crisp one — DLSS, every frame.
_

_Figure 002 — A neural network: layers of weighted connections.
_

The basic unit is simple: multiply inputs by weights, add a bias, pass
the result through a nonlinearity, and repeat. Stack enough of those and
the whole network can approximate essentially any function.


Stacking layers is what adds power: the early ones learn edges and
textures, the later ones objects and meaning — features composed from
features, with no hand engineering.


_Figure 003 — A transformer cell: each token attends to the others.
_

Learning reduces to calculus. Backpropagation carries the error from the
output back through every layer, computing for each weight the direction
to adjust — and gradient descent takes the step.


Inside each cell, **attention** lets every token draw on
every other; the mixed signal then passes through ordinary weighted units,
and gradient descent tunes millions of those weights until the output
matches the data.


_Figure 004 — Training: the loss falls as the model learns.
_

Architecture encodes assumptions about the data. Convolutions bake in the
structure of images, recurrence the order of sequences, attention the
long-range links in language — each a different prior about the world.


_Figure 005 — A convolution: a kernel slides across the image, building a feature map.
_

Scale changed what the method could do. The same gradients and the same
layers, given enough data and compute, went from research curiosity to
systems that write, draw, and reason.


And it _scales_: with more data, compute, and parameters, the same
methods keep improving — which is why they now underpin most of modern AI.


---

## Contents

### 1. Mathematical Background

1. [Linear Algebra](/deep-learning/mathematical-background/linear-algebra-for-deep-learning)
2. [Probability & Information Theory](/deep-learning/mathematical-background/probability-and-information-theory)
3. [Numerical Computation](/deep-learning/mathematical-background/numerical-computation)
4. [Calculus](/deep-learning/mathematical-background/calculus)

### 2. Foundations

1. [What Is Deep Learning?](/deep-learning/foundations/what-is-deep-learning)
2. [A Machine-Learning Refresher](/deep-learning/foundations/machine-learning-refresher)
3. [Linear Models & the Perceptron](/deep-learning/foundations/linear-models-and-the-perceptron)

### 3. Neural Networks

1. [The Multilayer Perceptron](/deep-learning/neural-networks/the-multilayer-perceptron)
2. [Activation Functions](/deep-learning/neural-networks/activation-functions)
3. [Universal Approximation](/deep-learning/neural-networks/universal-approximation)
4. [Backpropagation](/deep-learning/neural-networks/backpropagation)
5. [Loss Functions & Output Units](/deep-learning/neural-networks/loss-functions-and-output-units)

### 4. Optimization

1. [Gradient Descent & SGD](/deep-learning/optimization/gradient-descent-and-sgd)
2. [Momentum & Adaptive Methods](/deep-learning/optimization/momentum-and-adaptive-methods)
3. [Weight Initialization](/deep-learning/optimization/initialization)
4. [The Optimization Landscape](/deep-learning/optimization/the-optimization-landscape)
5. [Second-Order & Approximate Methods](/deep-learning/optimization/second-order-and-approximate-methods)

### 5. Regularization

1. [Regularization Overview](/deep-learning/regularization/regularization-overview)
2. [Dropout & Data Augmentation](/deep-learning/regularization/dropout-and-data-augmentation)
3. [Early Stopping & Parameter Sharing](/deep-learning/regularization/early-stopping-and-parameter-sharing)
4. [Normalization](/deep-learning/regularization/normalization)

### 6. Architectures

1. [Convolutional Networks](/deep-learning/architectures/convolutional-networks)
2. [CNN Architectures](/deep-learning/architectures/cnn-architectures)
3. [Recurrent Networks](/deep-learning/architectures/recurrent-networks)
4. [LSTM & GRU](/deep-learning/architectures/lstm-and-gru)
5. [Attention & Transformers](/deep-learning/architectures/attention-and-transformers)
6. [The Transformer Architecture](/deep-learning/architectures/the-transformer-architecture)
7. [Transformers in Practice](/deep-learning/architectures/transformers-in-practice)
8. [Graph Neural Networks](/deep-learning/architectures/graph-neural-networks)
9. [State-Space Models and Mamba](/deep-learning/architectures/state-space-models)

### 7. Theory & Frontiers

1. [Generalization Theory](/deep-learning/theory/generalization-theory)
2. [Adversarial Robustness](/deep-learning/theory/adversarial-robustness)
3. [Adversarial Defenses](/deep-learning/theory/adversarial-defenses)
4. [Bayesian & Ensemble Methods](/deep-learning/theory/bayesian-and-ensemble-methods)
5. [Deep Equilibrium Models](/deep-learning/theory/deep-equilibrium-models)

### 8. Generative Models

1. [Linear Factor Models](/deep-learning/generative-models/linear-factor-models)
2. [Autoencoders](/deep-learning/generative-models/autoencoders)
3. [Variational Autoencoders](/deep-learning/generative-models/variational-autoencoders)
4. [Generative Adversarial Networks](/deep-learning/generative-models/generative-adversarial-networks)
5. [Autoregressive Models & Normalizing Flows](/deep-learning/generative-models/autoregressive-and-normalizing-flows)
6. [Energy-Based & Boltzmann Machines](/deep-learning/generative-models/energy-based-and-boltzmann-machines)
7. [Diffusion and Score-Based Models](/deep-learning/generative-models/diffusion-and-score-based-models)

### 9. Probabilistic Methods

1. [Structured Probabilistic Models](/deep-learning/probabilistic-methods/structured-probabilistic-models)
2. [Monte Carlo & MCMC](/deep-learning/probabilistic-methods/monte-carlo-and-mcmc)
3. [Approximate Inference](/deep-learning/probabilistic-methods/approximate-inference)

### 10. Practical Deep Learning

1. [Practical Methodology](/deep-learning/practical/practical-methodology)
2. [Hyperparameters & Debugging](/deep-learning/practical/hyperparameters-and-debugging)
3. [Representation Learning](/deep-learning/practical/representation-learning)
4. [Transfer Learning](/deep-learning/practical/transfer-learning)
5. [Applications](/deep-learning/practical/applications)
6. [Model Compression and Distillation](/deep-learning/practical/model-compression-and-distillation)
7. [Meta-Learning and Few-Shot Learning](/deep-learning/practical/meta-learning-and-few-shot)

### 11. Large Models & Agents

1. [Large Language Models](/deep-learning/large-models-and-agents/large-language-models)
2. [Scaling, Inference, and Alignment of Language Models](/deep-learning/large-models-and-agents/scaling-inference-and-alignment)
3. [Denoising Sequence-to-Sequence Pretraining: BART](/deep-learning/large-models-and-agents/seq2seq-pretraining-and-bart)
4. [Text-to-Text Transfer and Conditional Generation](/deep-learning/large-models-and-agents/text-to-text-transfer-and-conditional-generation)
5. [Speech Recognition: Front-Ends and Alignment](/deep-learning/large-models-and-agents/speech-and-audio-models)
6. [Self-Supervised Speech Models and Synthesis](/deep-learning/large-models-and-agents/self-supervised-speech-and-synthesis)
7. [AI Agents: Tools and Reasoning](/deep-learning/large-models-and-agents/ai-agents)
8. [Agent Memory, Retrieval, and Orchestration](/deep-learning/large-models-and-agents/agent-memory-retrieval-and-orchestration)
9. [Mixture-of-Experts](/deep-learning/large-models-and-agents/mixture-of-experts)
10. [Multimodal Contrastive Learning](/deep-learning/large-models-and-agents/multimodal-models)
11. [Fusion and Vision-Language Models](/deep-learning/large-models-and-agents/fusion-and-vision-language-models)

### 12. Reinforcement Learning

1. [Foundations of Reinforcement Learning](/deep-learning/reinforcement-learning/foundations-of-reinforcement-learning)
2. [Model-Free Prediction and Control](/deep-learning/reinforcement-learning/model-free-prediction-and-control)
3. [Deep Q-Networks](/deep-learning/reinforcement-learning/deep-q-networks)
4. [Policy Gradients and Actor-Critic Methods](/deep-learning/reinforcement-learning/policy-gradients-and-actor-critic)
5. [Reinforcement Learning from Human Feedback](/deep-learning/reinforcement-learning/rl-from-human-feedback)