Backtracking & Search/Branch & Bound and Meet in the Middle

Lesson 9.34,214 words

Branch & Bound and Meet in the Middle

Plain backtracking prunes a search tree by feasibility; for optimization problems we can prune far more aggressively by value. Branch and bound keeps the best complete solution found so far and discards any partial solution whose optimistic bound cannot beat it.

╌╌╌╌

The previous lessons built backtracking: a depth-first walk over a tree of partial solutions that prunes a branch the moment it becomes infeasible, such as a queen attacking another or a graph coloring conflict. That kind of pruning asks a yes/no question: can this partial solution still be completed at all? For optimization problems, maximize this or minimize that, we can ask a sharper question: even in the best case, can completing this partial solution beat the best answer I already have? If not, the entire subtree is dead, feasible or not. Pruning by value rather than mere feasibility is the idea behind branch and bound, and it often cuts the running time by many orders of magnitude.

When even aggressive pruning is not enough, when the search tree is genuinely -shaped and is, say, , a second technique buys a square root of the running time outright. Meet in the middle splits the instance, enumerates each half independently, and stitches the halves back together with sorting and binary search. Both techniques attack the exponential running time, from different sides.

Branch and bound: pruning by value

Branch and bound is backtracking with two extra pieces of bookkeeping. The first is the incumbent: the value (and witness) of the best complete solution found anywhere in the search so far. The second is a bound computed at every node, an optimistic estimate of the best objective achievable by any completion of that partial solution. For a maximization problem the bound is an upper bound (no completion can do better than this); for minimization it is a lower bound.

A bound is sound exactly when this holds — when it never prunes a subtree that contains an optimum. An optimistic bound guarantees soundness, so branch and bound stays complete for the optimization problem: the optimal solution survives every cut and is eventually reported. A bound that could under-estimate (maximization) would be unsound, silently discarding the answer.

The method's effectiveness depends entirely on two design choices. A tighter bound prunes more nodes; a bound equal to the true optimum would prune everything but the answer. And a better search order, finding a strong incumbent early, raises sooner, which retroactively prunes more of the tree. The two interact: a good incumbent makes a mediocre bound effective.

Prune any node whose optimistic bound can't beat the incumbent

The right subtree carries bound , so it is pruned without ever being expanded; the blue path is the active best completion that established the incumbent.

Worked example: 0/1 knapsack by branch and bound

We have items with values and weights and a capacity ; choose a subset of maximum total value with total weight . The decision tree is binary, take item or skip it, so it has leaves. To bound a node, we use the LP relaxation: relax the integrality constraint and allow a fraction of the next item. First order all items by value density, descending. At a node that has fixed a prefix of decisions, with accumulated value and remaining capacity , greedily fill with the not-yet-decided items in density order, taking the last one fractionally:

where are the remaining items that fit wholly and is the first item that overflows (filled fractionally). This fractional fill is the optimal solution to the relaxed problem, so it can only over-estimate the integral optimum, which is the optimism a sound bound requires.2

The LP-relaxation bound: greedily fill remaining capacity in density order; the overflowing item is sliced fractionally (the part beyond is the relaxation's over-estimate)
Algorithm:KnapsackBnB\textsc{KnapsackBnB} — maximize value within capacity WW (items sorted by vi/wiv_i/w_i)
  1. 1
    z0z^\ast \gets 0
    incumbent value
  2. 2
    Expand(i=0, V=0, weight=0)\textsc{Expand}(i = 0,\ V = 0,\ \text{weight} = 0):
  3. 3
    if weight>W\text{weight} > W then return
    infeasible
  4. 4
    if V>zV > z^\ast then zVz^\ast \gets V
    new incumbent
  5. 5
    if i=ni = n then return
  6. 6
    if Bound(i,V,weight)z\textsc{Bound}(i, V, \text{weight}) \le z^\ast then return
    prune by value
  7. 7
    Expand(i+1, V+vi, weight+wi)\textsc{Expand}(i+1,\ V + v_i,\ \text{weight} + w_i)
    take item ii
  8. 8
    Expand(i+1, V, weight)\textsc{Expand}(i+1,\ V,\ \text{weight})
    skip item ii
  9. 9
  10. 10
    Bound(i,V,weight)\textsc{Bound}(i, V, \text{weight}):
  11. 11
    bV;cWweightb \gets V;\quad c \gets W - \text{weight}
  12. 12
    for jij \gets i to n1n-1 do
  13. 13
    if wjcw_j \le c then bb+vj; ccwjb \gets b + v_j;\ c \gets c - w_j
  14. 14
    else return b+cvj/wjb + c \cdot v_j / w_j
    fractional fill
  15. 15
    return bb

Branching on take before skip tends to find a heavy, valuable incumbent early, which makes the bound bite sooner. Compare this against the dynamic-programming solution. The DP runs in time, pseudo-polynomial, because enters as a magnitude, not a bit-length. When is enormous (say weights are -digit numbers) the DP table is hopeless, yet if is moderate the branch-and-bound tree, heavily pruned, finishes quickly. The two methods are complementary: DP wins when is small; branch and bound wins when is huge but is moderate.

Knapsack B&B (; items by density). The skip- node's LP bound , so that subtree is pruned (red)

Branching take-first dives straight to the incumbent with value . The skip- subtree's optimistic LP bound is only (take whole, then of : ), which cannot beat , so the entire right half is discarded before a single completion is built.

knapsack_branch_and_bound.pypython
from typing import NamedTuple, Sequence

class Item(NamedTuple):
  """
    One knapsack item: the value gained and the weight it costs.\n
  """
  value: float
  weight: float

def _density(item: Item) -> float:
  """
    Value per unit weight; a zero-weight item is infinitely dense.\n
  """
  return float("inf") if item.weight == 0 else item.value / item.weight

def _bound(
  ordered: Sequence[Item],
  index: int,
  value_so_far: float,
  weight_so_far: float,
  capacity: float,
) -> float:
  """
    Optimistic upper bound for the node fixing items `0..index-1`.\n
    Greedily fill the remaining capacity with the undecided items in\n
    density order, slicing the first overflowing item fractionally. That\n
    fractional fill solves the LP relaxation, so it can only over-estimate\n
    the best integral completion.\n
  """
  bound: float = value_so_far
  remaining: float = capacity - weight_so_far

  # take whole items in density order until one overflows.
  for position in range(index, len(ordered)):
    item = ordered[position]
    if item.weight <= remaining:
      bound += item.value
      remaining -= item.weight
    else:
      # slice the overflowing item to exactly fill what is left.
      bound += remaining * _density(item)
      break

  return bound

def knapsack_branch_and_bound(
  items: Sequence[Item],
  capacity: float,
) -> float:
  """
    Best total value of a subset of `items` whose weights fit within\n
    `capacity`, found by depth-first branch and bound.\n
  """
  if capacity < 0:
    return 0.0

  # density order makes the fractional bound exact for the relaxation.
  ordered: list[Item] = sorted(items, key=_density, reverse=True)
  count: int = len(ordered)
  incumbent: float = 0.0

  def expand(index: int, value_so_far: float, weight_so_far: float) -> None:
    nonlocal incumbent

    # abandon overweight nodes, then bank this node's value if it leads.
    if weight_so_far > capacity:
      return
    if value_so_far > incumbent:
      incumbent = value_so_far
    if index == count:
      return

    # prune by value: no completion here can beat what we already hold.
    if _bound(ordered, index, value_so_far, weight_so_far, capacity) <= incumbent:
      return

    item = ordered[index]

    # branch take-first: dives toward a heavy incumbent that prunes hard.
    expand(index + 1, value_so_far + item.value, weight_so_far + item.weight)
    expand(index + 1, value_so_far, weight_so_far)

  expand(0, 0.0, 0.0)
  return incumbent

Search order: depth-first vs best-first

The skeleton above is depth-first branch and bound: it recurses to a leaf fast, so it finds some complete solution, an incumbent, almost immediately, and it uses only stack. The cost is that the first incumbent may be poor, weakening early pruning. The alternative is best-first search: keep a priority queue of live nodes keyed by their bound, and always expand the node with the most promising bound. Best-first tends to drive toward the optimum with the fewest expansions and, for many problems, expands the optimal node first, but it can hold an exponential frontier of live nodes in the queue, so its memory is the liability. The practical compromise is to seed the incumbent with a quick greedy solution, then run depth-first with strong bounds: cheap memory, and a floor high enough that the bound prunes hard from the start.

Two search orders. Depth-first dives to a leaf for an early incumbent with stack; best-first expands the highest-bound live node — fewer expansions, but an exponential frontier

Depth-first (left) follows one accented path to a leaf, banking an incumbent fast while holding only the current root-to-node stack. Best-first (right) instead pops the live node of highest bound () from a priority queue, steering toward the optimum in fewer expansions at the cost of keeping the whole frontier in memory.

best_first_knapsack.pypython
import heapq
from itertools import count
from typing import NamedTuple, Sequence

class Item(NamedTuple):
  """
    One knapsack item: the value gained and the weight it costs.\n
  """
  value: float
  weight: float

class LiveNode(NamedTuple):
  """
    A node awaiting expansion: how far decisions are fixed, the value and\n
    weight accumulated, and the optimistic bound that orders the queue.\n
  """
  position: int
  value_so_far: float
  weight_so_far: float
  bound: float

def _density(item: Item) -> float:
  """
    Value per unit weight; a zero-weight item is infinitely dense.\n
  """
  return float("inf") if item.weight == 0 else item.value / item.weight

def _bound(
  ordered: Sequence[Item],
  index: int,
  value_so_far: float,
  weight_so_far: float,
  capacity: float,
) -> float:
  """
    Optimistic upper bound: greedily fill the remaining capacity with the\n
    undecided items in density order, slicing the first overflowing item\n
    fractionally (the LP relaxation, an over-estimate of the integer optimum).\n
  """
  bound: float = value_so_far
  remaining: float = capacity - weight_so_far

  # take whole items in density order, slicing the first overflow.
  for position in range(index, len(ordered)):
    item = ordered[position]
    if item.weight <= remaining:
      bound += item.value
      remaining -= item.weight
    else:
      bound += remaining * _density(item)
      break

  return bound

def best_first_knapsack(items: Sequence[Item], capacity: float) -> float:
  """
    Best total value of a subset of `items` fitting within `capacity`,\n
    found by best-first branch and bound (highest bound expanded first).\n
  """
  if capacity < 0:
    return 0.0

  # density order makes the fractional bound exact for the relaxation.
  ordered: list[Item] = sorted(items, key=_density, reverse=True)
  total: int = len(ordered)
  incumbent: float = 0.0

  # heapq is a min-heap; negate the bound to pop the largest first. The
  # monotonic tie-breaker keeps NamedTuples off the comparison path.
  tie_breaker = count()
  frontier: list[tuple[float, int, LiveNode]] = []

  def push(node: LiveNode) -> None:
    heapq.heappush(frontier, (-node.bound, next(tie_breaker), node))

  root_bound: float = _bound(ordered, 0, 0.0, 0.0, capacity)
  push(LiveNode(0, 0.0, 0.0, root_bound))

  while frontier:
    negated_bound, _, node = heapq.heappop(frontier)

    # every remaining live node is at least this good; if even the best
    # cannot beat the incumbent, the whole frontier is hopeless.
    if -negated_bound <= incumbent:
      break
    if node.position == total:
      continue

    item = ordered[node.position]

    # child that takes the current item, if it still fits.
    taken_weight: float = node.weight_so_far + item.weight
    if taken_weight <= capacity:
      taken_value: float = node.value_so_far + item.value
      incumbent = max(incumbent, taken_value)

      # queue the child only while its bound can still beat the incumbent.
      taken_bound: float = _bound(
        ordered, node.position + 1, taken_value, taken_weight, capacity
      )
      if taken_bound > incumbent:
        push(LiveNode(node.position + 1, taken_value, taken_weight, taken_bound))

    # child that skips the current item.
    skip_bound: float = _bound(
      ordered, node.position + 1, node.value_so_far, node.weight_so_far, capacity
    )
    if skip_bound > incumbent:
      push(
        LiveNode(
          node.position + 1, node.value_so_far, node.weight_so_far, skip_bound
        )
      )

  return incumbent

Meet in the middle

Some problems resist pruning entirely: the bound is weak, the structure symmetric, every branch genuinely live. If the instance is a subset problem over items and , meet in the middle sidesteps pruning and attacks the exponent directly. Split the items into two halves and of size . Enumerate all subset sums of into a list , and likewise all subset sums of into . Every subset of the whole is one choice from paired with one from , so the full answer is recovered by combining one element of with one of , but we perform that combination efficiently, not by trying all pairs.

For the canonical task, find a subset whose sum is closest to a target (this is the minimum-partition-difference problem with ), sort , then for each binary-search for the value nearest . Each query is , so the whole combine is .

Algorithm:MeetInTheMiddle\textsc{MeetInTheMiddle} — subset sum closest to target TT
  1. 1
    split items into halves AA (size n/2\lceil n/2\rceil) and BB
  2. 2
    SAS_A \gets all 2A2^{|A|} subset sums of AA
  3. 3
    SBS_B \gets all 2B2^{|B|} subset sums of BB
  4. 4
    sort SBS_B
  5. 5
    bestbest \gets \infty
  6. 6
    for each aa in SAS_A do
  7. 7
    rTar \gets T - a
    complement from BB
  8. 8
    ss \gets value in SBS_B nearest rr (binary search: floor and ceiling of rr)
  9. 9
    bestmin(best, a+sT)best \gets \min(best,\ |\,a + s - T\,|)
  10. 10
    return bestbest

The enumeration is per half, the sort is , and the combine is , so the whole algorithm is , a quadratic improvement over the brute force. Concretely, is about (out of reach) while is about (instant). The technique is exact, no approximation and no pruning luck, and it is the intended solution to every Hard subset problem in this lesson's practice set.

— enumerate each half, then binary-search the complement

For each sum on the top we binary-search the sorted bottom list for ; the blue pair hits the target exactly. To see the whole method end to end, take the eight numbers and target . Split into and . Enumerating every subset sum:

  • (the sums of ).
  • (sorted).

Now walk : for we seek in , and is present, so hits the target exactly — the subset from plus from . The combine did binary searches of a -element list instead of scanning all subsets, and it scales: at it is searches rather than subsets.

The same split-and-recombine idea is the graph analog bidirectional search: to find a shortest path, run BFS forward from the source and backward from the target simultaneously and stop when the two frontiers meet, exploring nodes instead of .

meet_in_the_middle.pypython
from bisect import bisect_left
from typing import Sequence

def _subset_sums(items: Sequence[float]) -> list[float]:
  """
    Every subset sum of `items`, including the empty subset (sum 0).\n
    Built by doubling: each new item either extends or skips every sum\n
    seen so far, giving 2^len(items) sums in total.\n
  """
  sums: list[float] = [0.0]
  for item in items:
    sums += [existing + item for existing in sums]
  return sums

def closest_subset_sum(items: Sequence[float], target: float) -> float:
  """
    The subset sum of `items` nearest to `target` (its absolute distance is\n
    minimized). The empty subset, with sum 0, is always a candidate.\n
  """
  # enumerate one half, sort the other so it can be binary-searched.
  midpoint: int = (len(items) + 1) // 2
  left_sums: list[float] = _subset_sums(items[:midpoint])
  right_sums: list[float] = sorted(_subset_sums(items[midpoint:]))

  best_distance: float = float("inf")
  for left_sum in left_sums:

    # the right sum nearest the complement sits at one of two neighbours.
    complement: float = target - left_sum
    position: int = bisect_left(right_sums, complement)

    # check both neighbours, keeping the closest total to the target.
    for neighbour in (position - 1, position):
      if 0 <= neighbour < len(right_sums):
        total: float = left_sum + right_sums[neighbour]
        best_distance = min(best_distance, abs(total - target))

  return best_distance

def min_partition_difference(items: Sequence[float]) -> float:
  """
    The smallest achievable difference between the sums of two groups that\n
    partition `items`. Reduces to a closest-subset-sum query with target\n
    half the total: a subset summing to `total/2` splits the items evenly,\n
    and a subset sum `s` leaves a difference of |total - 2*s|.\n
  """
  total: float = sum(items)
  best_distance: float = closest_subset_sum(items, total / 2)

  # distance from total/2 is half the resulting partition gap.
  return 2 * best_distance
Bidirectional search: two BFS frontiers of radius from and meet in the middle, touching nodes instead of one frontier of

When to reach for which

The three pruning disciplines line up neatly along one axis: what justifies discarding a branch.

  • Backtracking prunes by feasibility: a partial solution that violates a constraint can never be completed, so cut it.
  • Branch and bound prunes by value: a partial solution whose optimistic bound cannot beat the incumbent is pointless to complete, so cut it.
  • Meet in the middle prunes nothing; it instead trades exponential time for the square root of it, , paying with memory to store the enumerated half.

Branch and bound works best when a cheap, tight optimistic bound exists (knapsack's LP fill, a TSP node's spanning-tree lower bound). Meet in the middle works best when no such bound exists but is small enough that is affordable. Both are exact; neither changes the worst-case exponential complexity; both routinely turn an infeasible instance into a feasible one.

How the world actually solves hard optimization

Branch and bound solves the large integer programs behind logistics, scheduling, and network design every day.

Branch and cut. Modern integer-programming solvers — CPLEX, Gurobi, the open-source SCIP — run branch and cut: branch and bound whose LP-relaxation bound (exactly the knapsack bound of this lesson, generalized) is tightened at each node by adding cutting planes, linear inequalities valid for all integer solutions but violated by the current fractional optimum.3 Gomory's cuts (1958) and the Padberg–Rinaldi cuts for the traveling salesman turned instances once deemed hopeless into routine ones; a combination of branch and cut solved a TSP over all cities of a VLSI application to proven optimality.4 The engineering lesson matches this lesson's theory: a tighter bound (better cuts) and a stronger incumbent (better heuristics) each prune more of the tree.

A*: the same idea. Best-first branch and bound is, essentially, the A* search algorithm (Hart, Nilsson & Raphael, 1968): expand the live node minimizing , where is the cost so far and is an admissible heuristic — a bound that never overestimates the remaining cost.5 Admissibility supplies the optimistic-bound condition that makes pruning sound here; A* is branch and bound with the objective shortest path and the bound

heuristic-to-goal.
The bound-tightening spiral of branch and cut. A cutting plane shaves the fractional LP region toward the integer hull, lowering the optimistic bound at a node; a stronger incumbent raises the floor. Where the two meet, the subtree is pruned.

Meet in the middle in cryptanalysis. The meet-in-the-middle split is older than its algorithmic-puzzle use: Diffie and Hellman (1977) introduced it to attack double encryption, showing that encrypting twice with two keys gives far less than double the security because an attacker enumerates each key-half and matches in the middle — the same collapse, applied to key search.6 The subset trick and the cryptographic attack are the same idea.

Takeaways

  • Branch and bound is backtracking for optimization: maintain an incumbent (best complete solution so far) and a bound (optimistic estimate per node), and prune any node whose bound cannot beat the incumbent.
  • The method's power is all in the bound tightness and search order: a tighter bound and an earlier strong incumbent each prune more of the tree.
  • For 0/1 knapsack, order by density and bound by the LP-relaxation fractional fill; branch and bound beats the DP when is huge but is moderate.
  • Depth-first branch and bound finds an incumbent fast with memory; best-first (priority queue on bound) targets the optimum with fewer expansions but can hold an exponential frontier.
  • Meet in the middle enumerates each of two halves ( subset sums) and recombines by sorting + binary search, giving , exact search up to ; bidirectional search is the graph analog.
  • One axis: backtracking prunes by feasibility, branch and bound by value, meet in the middle trades exponential time for of it at the cost of memory.

Footnotes

  1. Erickson, Ch. — Backtracking: branch and bound as backtracking augmented with a value bound; the optimism of the bound is what makes pruning sound.
  2. Skiena, § — Combinatorial Search / Heuristics: pruning a combinatorial search by bounding the best achievable completion, illustrated on knapsack-style problems.
  3. Padberg, M. & Rinaldi, G. (1991), A branch-and-cut algorithm for the resolution of large-scale symmetric traveling salesman problems, SIAM Review 33(1), 60–100 — branch and bound tightened by cutting planes, the template of modern IP solvers; cutting planes trace to Gomory, R. E. (1958).
  4. Applegate, D. L., Bixby, R. E., Chvátal, V. & Cook, W. J. (2006), The Traveling Salesman Problem: A Computational Study, Princeton University Press — solving TSP instances with tens of thousands of cities to proven optimality by branch and cut.
  5. Hart, P. E., Nilsson, N. J. & Raphael, B. (1968), A formal basis for the heuristic determination of minimum cost paths, IEEE Transactions on Systems Science and Cybernetics 4(2), 100–107 — A* as best-first search with an admissible (optimistic) heuristic, i.e. branch and bound for shortest paths.
  6. Diffie, W. & Hellman, M. E. (1977), Exhaustive cryptanalysis of the NBS data encryption standard, Computer 10(6), 74–84 — the meet-in-the-middle attack on double encryption, the same split used for subset problems.
Practice

╌╌ END ╌╌