Gradient Descent in Two Variables

Gradient descent uses the slopes in both coordinate directions to search for a minimum of $f(x,y)$. A concrete step shows how the two coordinates change together.

Practical Example

Consider the illustrative room-temperature function

$$f(x,y)=85-\frac{1}{90}x^2(x-6)^2-y^2(y-6)^2, \qquad (x,y)\in[0,5]^2.$$

This is a different model from the preceding sauna example, and it is not a calibrated physical temperature model. The bounded room matters: without the domain restriction, the polynomial is unbounded below, so there is no coolest point to find.

First gradient step

Start at $(x_0,y_0)=(0.5,0.6)$ with learning rate $\alpha=0.05$. To calculate the partial derivatives, let $h(t)=t^2(t-6)^2$. Then $f=85-h(x)/90-h(y)$ and $h'(t)=4t(t-3)(t-6)$, giving the gradient

$$\nabla f(x,y)=\left(-\frac{4x(x-3)(x-6)}{90},\;-4y(y-3)(y-6)\right).$$

At the starting point, $\nabla f(0.5,0.6)\approx(-0.3055556,-31.104)$. Subtract $0.05$ times each component:

$$\begin{aligned} x_1&=0.5-0.05\left(-\frac{11}{36}\right)\approx0.5152778,\\ y_1&=0.6-0.05(-31.104)=2.1552. \end{aligned}$$

Both components must be computed from the old point before updating either variable. The new point is still inside the room, and $f$ falls from approximately $74.418372$ to $16.248271$.

Conceptual Overview

The gradient $\nabla f=\left(\frac{\partial f}{\partial x},\frac{\partial f}{\partial y}\right)$ collects the two partial derivatives. At a point where it is nonzero, it points in the direction of greatest instantaneous increase; its magnitude is that maximal rate per unit distance. Gradient descent moves in the opposite direction, the direction of steepest local decrease. This is a local direction, not a guarantee that any chosen finite step decreases the function or reaches a minimum.

Mathematical Formulation

For the current point $(x_k,y_k)$, the update is

$$\begin{aligned} (x_{k+1},y_{k+1}) &=(x_k,y_k)-\alpha\nabla f(x_k,y_k)\\ &=(x_k,y_k)-\alpha\left(\frac{\partial f}{\partial x},\frac{\partial f}{\partial y}\right)_{(x_k,y_k)}. \end{aligned}$$

The learning rate $\alpha$ scales both components of the displacement. A large step can overshoot a minimum; a very small one can make convergence slow.

Algorithm for Two Variables

Choose an initial point $(x_0,y_0)$, compute its gradient, and apply the update repeatedly. For a box-constrained room, keep every new point feasible by projecting the update:

$$q_{k+1}=\Pi_{[0,5]^2}(q_k-\alpha\nabla f(q_k)), \qquad q_k=(x_k,y_k),$$

where $\Pi$ clips each coordinate into $[0,5]$. Monitor the objective decrease and the projected-gradient residual

$$\frac{\|q-\Pi_{[0,5]^2}(q-\alpha\nabla f(q))\|_2}{\alpha}.$$

Set an iteration cap as well. Small changes between iterates alone do not prove a minimum. For the box constraint, checking only the raw gradient is insufficient: a boundary optimum need not have zero gradient. Even a small projected residual is only a stationarity test for this nonconvex problem.

Challenges and Considerations

For this particular model, the minimum can also be found exactly. On $[0,5]$, $h$ attains its unique maximum $81$ at $3$. Maximizing each subtracted term gives the global minimum $(3,3)$ with $f=3.1$. By contrast, starting at $(0,0)$ gives zero gradient and never reaches that minimum. In general, gradient descent may converge to a local rather than a global minimum; trying different starting points can reduce this risk but does not prove global optimality.

At $(3,3)$, the Hessian is $\operatorname{diag}(0.4,36)$. Linearizing the update gives the sufficient local contraction range $0<\alpha<1/18$. Thus $0.05$ is close to the stability limit in $y$ but moves slowly in $x$. From the stated start, 1000 steps give approximately $(3,3)$; this empirical run does not prove convergence from every start.

In Distill’s interactive explanation of momentum, compare plain gradient descent with momentum on elongated quadratic contours. Changing the step size helps explain why steep and shallow directions progress differently, as in the Hessian calculation here; the demo uses a separate objective.

My reading · Opened notes are remembered in this browser.