Section 4a: From the variational bound to a trainable min-max game

I am working through Mathematical Foundations of Generative AI, Prof. Prathosh AP's public lectures. The playlist is the spine. These notes are my deep dive on each section: a visual when the picture is the point, a detour when a prerequisite is doing real work, and every formula read aloud with a plain-words line right after it.

Previously: Section 3b: The convex conjugate and the variational bound.

Week 2 roadmap

Week 2 picks up right where Week 1 stopped. Here is how the four sections of Week 2 fit together:

Prerequisites for this note (detours below): Week 1's lower bound on $D_f$, "an average is approximately an expectation", and what a saddle point is.

Symbol cheat-sheet

Section 4a: From the variational bound to a trainable min-max game

1. Intuition first

In Week 1 we wanted to make our model distribution $p_\theta$ match the real data distribution $p_x$ by shrinking a distance between them (the f-divergence $D_f$). The problem: we cannot compute that distance directly, because we do not know either density formula. We only have samples.

Week 1 ended with a workaround: a lower bound on $D_f$ that only needs averages over samples, but it contains a "maximise over all functions $T$" step. This section answers: how do you actually maximise over "all functions" on a computer? Answer: use a neural network for $T$, and you get a two-player game.

2. Real-world analogy

Think of a fraud-detection team vs. a fraudster.

• The fraudster (the generator, $g_\theta$) produces fake transactions.
• The fraud team (the critic, $T_w$) builds a scoring model that gives real transactions high scores and fakes low scores.
• The quality of the fraud team's best scoring model is a measure of how different fakes still are from real ones. If even the best detector cannot separate them, the fakes are indistinguishable from real traffic.
• The fraudster improves the fakes to make even the best detector useless.

That is the exact mathematical structure: the critic maximises a separation score, and the generator minimises that best-achievable score.

3. Visual

The critic only ever sees samples (real ones and fake ones), and the generator only ever sees noise. All the learning signal flows through the single game score $J(\theta, w)$.

Things to watch in the widget:

• Move the critic slider $w$: the blue curve shows $J(\theta, w)$ as a hill over $w$. The coral dot is the current critic's score $J(\theta, w)$, which always sits at or below the dashed green ceiling $D_{KL}(p_x \,|\, p_\theta)$.
• Click "Snap critic to $w^*(\theta)$": the critic climbs to the top of the hill, where the lower bound touches the true KL divergence.
• Move the generator slider $\theta$ toward $0.50$: the entire hill sinks down until its peak hits 0 at the saddle point $(\theta^* = 0.50, w^* = 0)$.

4. The formulas, decoded

Before we start, here is a quick recap of the Week 1 ingredient this all rests on.

Detour 1: The Week 1 lower bound (what $f^*$ and $T$ are)

In Week 1, the f-divergence was defined as

$$D_f(p_x \,|\, p_\theta) = \int_{\mathcal{X}} p_\theta(x)\, f!\left(\frac{p_x(x)}{p_\theta(x)}\right) dx$$

Read aloud: "the f-divergence from $p_x$ to $p_\theta$ is: at every point $x$, take the ratio of real density to model density, push it through the function $f$, weight it by how often the model visits $x$, and add up over all $x$."

In plain words: it is a model-weighted measure of how far the ratio $\text{real}/\text{model}$ is from 1, on average.

The trouble is that this formula needs the density formulas for $p_x$ and $p_\theta$, which we do not have. Week 1's trick was the convex conjugate $f^*$: every convex $f$ can be rewritten as "the best of many straight lines",

$$f(u) = \max_t \,\big[\,t\,u - f^*(t)\,\big]$$

Read aloud: "$f$ of $u$ equals the maximum over $t$ of $t$ times $u$ minus $f$-star of $t$."

In plain words: any bowl-shaped curve equals the upper envelope of its tangent lines. Plugging that into $D_f$ and swapping the order gave an expression that only contains averages, which we can estimate from samples. The price: we have to choose a slope $t$ for every $x$ — a whole function $T(x)$ — and we only get a lower bound unless we pick $T$ perfectly.

Back to the main thread. That brings us to the first equation of Week 2.

Formula 1: The lower bound

$$D_f(p_x \,|\, p_\theta) \;\ge\; \max_{T \in \mathcal{T}}\; \Big[\, \mathbb{E}{x\sim p_x}[T(x)] \;-\; \mathbb{E}{x\sim p_\theta}\big[f^*(T(x))\big] \,\Big]$$

Read aloud: "The divergence between the real distribution and the model is at least as big as the best value you can get, over all functions $T$ in some family $\mathcal{T}$, of: the average of $T$ on real samples, minus the average of $f^*$-of-$T$ on model samples."

Piece by piece:

In plain words: reward $T$ for giving high scores to real data, penalise it (through $f^*$) for giving high scores to fake data; the best possible net score is a guaranteed underestimate of how different the two distributions are.

Why it is a lower bound and not an equality: equality only holds if the menu $\mathcal{T}$ contains the perfect scorer, which Section 3b showed is

$$T^*(x) = f'!\left(\frac{p_x(x)}{p_\theta(x)}\right)$$

Read aloud: "T-star at $x$ equals $f$-prime evaluated at the ratio of $p_x(x)$ to $p_\theta(x)$."

In plain words: the ideal critic outputs the slope of $f$ at the true density ratio. A smaller menu $\mathcal{T}$ can only do worse, so the max can only be smaller.

Formula 2: The training objective

We want $\theta^* = \arg\min_\theta D_f(p_x \,|\, p_\theta)$. Since we cannot compute $D_f$, we swap it for its lower bound:

$$\theta^* \;\approx\; \arg\min_{\theta}\; \max_{T\in\mathcal{T}}\Big[\,\mathbb{E}{p_x}[T(x)] - \mathbb{E}{p_\theta}\big[f^*(T(x))\big]\,\Big]$$

Read aloud: "The best generator parameters $\theta^*$ are approximately those that minimise, over $\theta$, the best score any critic $T$ in $\mathcal{T}$ can achieve."

• The outer min is over $\theta$, the weights of the generator $g_\theta(z)$.
• The inner max is over $T$, a class of critic functions $T(x) \in \mathcal{T}$.
• The $\approx$ matters: we are minimising a lower bound, not the true divergence. If the critic family $\mathcal{T}$ is weak, the bound is loose, and the generator may appear to have low loss even when its samples are poor.

In plain words: pick the generator such that even the best possible critic can barely tell real from fake.

Formula 3: Making "max over all functions" computable

"Max over a set of functions" is not something a computer can execute directly. The fix: let the menu $\mathcal{T}$ be every function a particular neural network architecture can represent. A network $T_w(x)$ with weights $w$ is a function; changing $w$ changes the function. So "search over functions" becomes "search over weights $w$", which gradient ascent can do:

$$\theta^,\, w^ \;=\; \arg\min_{\theta}\;\max_{w}\;\Big[\,\mathbb{E}{x\sim p_x}[T_w(x)] \;-\; \mathbb{E}{x\sim p_\theta}\big[f^*(T_w(x))\big]\,\Big]$$

Read aloud: "Find the generator weights $\theta^$ and critic weights $w^$ such that $\theta$ minimises, and $w$ maximises, the critic's net score."

In plain words: replace the abstract function search with a neural network $T_w$, so both players are just tuning network weights.

The lectures name the bracket $J(\theta, w)$:

$$J(\theta, w) = \mathbb{E}{x\sim p_x}[T_w(x)] - \mathbb{E}{x\sim p_\theta}\big[f^*(T_w(x))\big]$$

Read aloud: "J of theta and w equals the expected value of $T_w(x)$ under $p_x$ minus the expected value of $f^*(T_w(x))$ under $p_\theta$."

In plain words: $J(\theta, w)$ is one single number measuring how well critic $w$ separates real data from fake data produced by generator $\theta$.

One step the slides leave implicit — and the reason gradients can reach $\theta$ — is how $\theta$ enters the second average. Because fake samples are made as $\hat{x} = g_\theta(z)$ with $z \sim \mathcal{N}(0, I)$, we rewrite the second term as:

$$\mathbb{E}{x\sim p\theta}\big[f^(T_w(x))\big] = \mathbb{E}_{z\sim\mathcal{N}(0,I)}\big[f^\big(T_w(g_\theta(z))\big)\big]$$

Read aloud: "the expected value of $f^(T_w(x))$ for $x$ drawn from $p_\theta$ equals the expected value of $f^(T_w(g_\theta(z)))$ for noise $z$ drawn from a standard normal."

In plain words: averaging over model samples is the exact same thing as drawing random noise vectors $z$, running each one through the generator $g_\theta(z)$, and averaging the critic's penalty on the resulting fakes. We never need a density formula for $p_\theta$.

Saddle point and adversarial training

Detour 2: What is a saddle point?

Picture a horse saddle (or a Pringle crisp). Walking front-to-back, the centre is the lowest point; walking side-to-side, it is the highest point. A point that is a minimum in one direction and a maximum in another is a saddle point:

$$J(\theta^, w) \;\le\; J(\theta^, w^) \;\le\; J(\theta, w^)$$

Read aloud: "$J$ at $(\theta^, w)$ is at most $J$ at $(\theta^, w^)$, which in turn is at most $J$ at $(\theta, w^)$."

In plain words: at the saddle point $(\theta^, w^)$, if the critic changes $w$, the score $J$ cannot go up (the critic is already doing its best), and if the generator changes $\theta$, the score $J$ cannot go down (the generator is already doing its best). Neither player wants to move.

Back to the main thread. That is why the lectures call this saddle-point optimisation and an adversarial problem: two players pulling one single number $J(\theta, w)$ in opposite directions.

5. Tiny worked example

Use the forward KL divergence from Week 1: $f(u) = u\ln u$, whose conjugate is $f^*(t) = e^{t-1}$. Data lives on two points, $a$ and $b$:

First, the true KL divergence:

$$D_{KL}(p_x \,|\, p_\theta) = 0.5\ln!\left(\frac{0.5}{0.8}\right) + 0.5\ln!\left(\frac{0.5}{0.2}\right) = 0.5(-0.470) + 0.5(0.916) \approx 0.223$$

Read aloud: "0.5 times natural log of 0.5 over 0.8, plus 0.5 times natural log of 0.5 over 0.2, equals approximately 0.223."

In plain words: the true KL gap between these two coins is $0.223$.

Now let three different critics try. For each critic,

$$J(\theta, w) = \Big(0.5\,T(a) + 0.5\,T(b)\Big) - \Big(0.8\,e^{T(a)-1} + 0.2\,e^{T(b)-1}\Big)$$

Read aloud: "$J$ equals the real average of $T$ minus the model average of $e^{T-1}$."

In plain words: average score on real coin flips minus average exponential penalty on model coin flips.

What the table shows:

• Every critic gives $J \le 0.223$ (matching the "$\ge$" in Formula 1), and only the optimal critic hits $0.223$.
• The good critic scores $b$ higher, because $b$ is where real data is under-produced by the model ($0.5$ vs $0.2$). That is the critic pointing at what the generator gets wrong.
• Now the generator moves: if $\theta$ changes so $p_\theta = (0.5, 0.5)$, the true KL drops to $0$, and the best any critic can achieve is $J = 0$. The generator wins when no critic can score above zero.

In practice, those averages come from mini-batch samples: for instance, four real draws ${a, a, b, b}$ give sample average $(0.530 + 0.530 + 1.916 + 1.916)/4 = 1.223$, matching the exact expectation by the Law of Large Numbers.

6. Connect back

• Week 1 (Section 3b) gave the variational lower bound with an abstract $\sup_T$.
• This note (Section 4a) makes it concrete: $T$ becomes a neural network $T_w$, and training becomes the two-player saddle-point game $\min_\theta \max_w J(\theta, w)$. This general recipe is called f-GAN or Variational Divergence Minimisation (VDM).
• Next (Section 4b): $T_w$ must output numbers that lie inside $\operatorname{dom} f^$ (since some $f^$ only accept restricted ranges), which we enforce with an output activation $\sigma_f$.
• Section 4c: picking one specific $f$ turns $J(\theta, w)$ into the classic GAN loss $\mathbb{E}[\log D(x)] + \mathbb{E}[\log(1 - D(g_\theta(z)))]$.
• Week 6 (WGAN): replaces f-divergences with the Wasserstein distance while keeping this exact generator-vs-critic game shape.

Quick check (optional)

1. Why can minimising $J(\theta, w)$ over $\theta$ fool you into thinking the generator is good, even when it is not?
2. In $J(\theta, w)$, which of the two averages actually depends on $\theta$, and how does $\theta$ get in there?
3. If the critic network were tiny (say, it could only output constants $T_w(x) = c$), what would $\max_w J(\theta, w)$ always equal, and what would that mean for training?