Skip to main content

Gradient and Multidimensional Optimization

Definition of Gradient​

For a function f(x,y)f(x, y) with two variables, the gradient, denoted as ∇f\nabla f, is defined as the vector of its partial derivatives. Formally,

∇f=(∂f∂x,∂f∂y)\nabla f = \left( \frac{\partial f}{\partial x}, \frac{\partial f}{\partial y} \right)

For f(x,y)=x2+y2f(x, y) = x^2 + y^2, the gradient is ∇f=(2x,2y)\nabla f = (2x, 2y). This vector represents the direction and rate at which the function increases most rapidly. The components of ∇f\nabla f correspond to the slopes of the tangent lines to the level curves of ff at any given point.

Gradient at a Specific Point​

To find the gradient of f(x,y)=x2+y2f(x, y) = x^2 + y^2 at a particular point, say (2,3)(2, 3), we substitute x=2x = 2 and y=3y = 3 into the gradient formula:

∇f=(2⋅2,2⋅3)=(4,6)\nabla f = (2 \cdot 2, 2 \cdot 3) = (4, 6)

This result signifies the gradient of ff at the point (2,3)(2, 3) and represents the vector pointing in the direction of greatest increase of ff from that point.

Application in Optimization​

The concept of the gradient extends naturally to the optimization of functions. For a function of a single variable, f(x)=x2f(x) = x^2, finding its minimum involves solving f′(x)=0f'(x) = 0. The solution indicates the point where the function has a horizontal tangent line, suggesting a local minimum or maximum.

For functions of two or more variables, such as f(x,y)=x2+y2f(x, y) = x^2 + y^2, the optimization process seeks points where the gradient is zero, i.e., ∇f=0⃗\nabla f = \vec{0}. This condition implies that both partial derivatives, ∂f∂x\frac{\partial f}{\partial x} and ∂f∂y\frac{\partial f}{\partial y}, are zero. Solving the system of equations

∂f∂x=2x=0∂f∂y=2y=0\begin{align*} \frac{\partial f}{\partial x} = 2x &= 0 \\ \frac{\partial f}{\partial y} = 2y &= 0 \end{align*}

yields the point (0,0)(0, 0) as the location of the minimum. This methodology generalizes to functions of any number of variables, where the gradient being zero indicates a local extremum.

In summary, the gradient is a powerful tool in calculus and analysis, facilitating the understanding and optimization of multivariable functions. Its computation and the conditions for optimization form the basis for many applications in mathematics, physics, engineering, and machine learning.

Optimization in a Two-Dimensional Sauna​

Considering a sauna as a two-dimensional space allows for movement in any direction within a 5x5 room, aiming to find the coldest point based on the temperature distribution.

  • Temperature Function: For a given position (x,y)(x, y), the temperature is a function T(x,y)T(x, y) represented by the height in a three-dimensional plot. Hot areas are indicated by red (higher values) and cold areas by blue (lower values).

  • Objective: To locate the coldest area (T(x,y)T(x, y) minimum) where moving in any direction increases the temperature.

  • Mathematical Approach: The process involves calculating the partial derivatives ∂T∂x\frac{\partial T}{\partial x} and ∂T∂y\frac{\partial T}{\partial y}, setting them to zero, and solving for xx and yy to find potential minimum points.

  • Example Function: T(x,y)=85−190x2(x−6)y2(y−6)T(x, y) = 85 - \frac{1}{90}x^2(x - 6)y^2(y - 6).

  • Partial Derivatives:

    ∂f∂x=−190x(3x−12)y2(y−6)\frac{\partial f}{\partial x} = -\frac{1}{90}x(3x - 12)y^2(y - 6) ∂f∂y=−190x2(x−6)y(3y−12)\frac{\partial f}{\partial y} = -\frac{1}{90}x^2(x - 6)y(3y - 12)

  • Finding the Minimum: Solving ∂f∂x=0\frac{\partial f}{\partial x} = 0 and ∂f∂y=0\frac{\partial f}{\partial y} = 0 yields several candidate points. The coldest point within the sauna's bounds is identified by evaluating these candidates.

Linear Regression Optimization​

Linear regression, a fundamental machine learning model, is optimized through a similar multidimensional calculus approach but involves finding the best fit line to a set of data points.

  • Problem Statement: With given coordinates of power lines, the task is to minimize the total cost of connecting these to a main fiber line. The cost is proportional to the square of the distances from the power lines to the fiber line.
  • Mathematical Formulation: The line equation y=mx+by = mx + b represents the fiber line, with mm and bb being the slope and y-intercept, respectively. The optimization goal is to minimize the total cost function E(m,b)E(m, b), which depends on mm and bb.
  • Cost Function: E(m,b)=14m2+3b2+38+12mb−42m−20bE(m, b) = 14m^2 + 3b^2 + 38 + 12mb - 42m - 20b.
  • Partial Derivatives and Optimization:
    • ∂E∂m=28m+12b−42\frac{\partial E}{\partial m} = 28m + 12b - 42
    • ∂E∂b=6b+12m−20\frac{\partial E}{\partial b} = 6b + 12m - 20
  • Solution: Setting ∂E∂m=0\frac{\partial E}{\partial m} = 0 and ∂E∂b=0\frac{\partial E}{\partial b} = 0 and solving for mm and bb yields the optimal line parameters minimizing the cost.
  • Result: The optimal values m=12m = \frac{1}{2} and b=73b = \frac{7}{3} are found, with a minimum cost of 4.167.

Gradient Descent: An Efficient Optimization Method​

The traditional approach of solving partial derivatives can be cumbersome, especially with an increasing number of variables. Gradient descent offers a more efficient way to find minimum values of multidimensional functions by iteratively moving towards the steepest descent direction. This method is pivotal in machine learning for optimizing complex models beyond linear regression.