Gradient descent updates are exactly viscous Hamilton‑Jacobi evolutions, a reformulation that collapses ResNets, Transformers, and recurrent models into a single mathematical object. By treating the weight vector as the initial datum of a PDE, each training step becomes a Hopf–Cole propagation that fits the observed loss surface. This perspective replaces ad‑hoc layer‑wise intuition with a unified dynamical system that can be analysed with the full machinery of PDE theory.
Historically, residual connections were justified by eased gradient flow, attention mechanisms were explained through information bottlenecks, and recurrence was motivated by sequential processing constraints. Each line of work built its own set of tricks—batch renormalization, positional encodings, gating functions—without a common analytical backbone. Consequently, advances in one family rarely translated into principled insights for the others.
Residual networks, transformers, and recurrent architectures (RNNs, LSTMs, SSMs) each discretize the same class of Hamilton–Jacobi equations, with architecture‑dependent Hamiltonian and viscosity. “Residual networks, transformers, and recurrent architectures (RNNs, LSTMs, SSMs) each discretize the same class of Hamilton–Jacobi equations, with architecture-dependent Hamiltonian and viscosity.” [1] This discretization view explains why depth, attention heads, and recurrence depth all behave like time‑steps in a numerical integrator.
A single deformation parameter ε indexes four perspectives on the same object (neural network, tropical algebra, PDE, convex optimization) and the resulting commutative diagram closes under Lipschitz and convex conditions. “A single parameter indexes four perspectives on the same object (neural network, tropical algebra, PDE, convex optimization) and the resulting commutative diagram (Theorem 7.1) closes under Lipschitz conditions.” [1] The authors suggest that the parameter ε may influence the trade‑off between smoothness of the PDE solution and the sparsity of the tropical limit, potentially serving as a knob for regularization and model expressivity.
The theory predicts the minimax optimal generalization rate O(n^{‑1/(d+2)}) for fixed diffusion time t, matching known lower bounds for non‑parametric regression in d‑dimensional intrinsic data manifolds. This rate emerges directly from the PDE quadrature analysis, linking sample complexity to the viscosity term. Practitioners can read the exponent as a quantitative target for architecture scaling in high‑dimensional regimes.
The correspondence is exact for log‑sum‑exp layers and only structural for broader architectures, leaving a gap between theory and the ReLU‑dominated models used in practice. “The correspondence is exact for log‑sum‑exp layers and structural for broader architectures.” [1] Moreover, the paper provides no empirical validation that the ε‑controlled robustness or the O(N) influence function improve real‑world performance. These limitations suggest that the framework is presently a powerful lens rather than an all‑purpose design tool.
If training truly follows a viscous Hamilton‑Jacobi flow, then architecture comparison should shift from layer‑type taxonomy to a classification by Hamiltonian form and viscosity coefficient. Benchmarks that measure numerical stability of PDE discretizations could replace current accuracy‑only suites, revealing hidden trade‑offs in robustness and generalization. Re‑examining existing models through this PDE lens may uncover a new generation of hybrid designs that deliberately tune ε for task‑specific smoothness.
