14.7 - Multivariable Optimizations

Remember

Recall from Calculus 1 that if $f (x)$ has a local maximum/minimum at a point $x = c$ , and if $f^{'} (c)$ exists, then $f^{'} (c) = 0$ .

MatPlotLib Graph

import micropip
await micropip.install("matplotlib")
await micropip.install("numpy")
 
import matplotlib.pyplot as plt
import numpy as np
 
# Define the function f(x) and its derivative f'(x)
def f(x):
    return np.sin(x)  # Example of a function with a local max and min
 
def df(x):
    return np.cos(x)  # Derivative of sin(x)
 
# Create x values
x_vals = np.linspace(0, 2 * np.pi, 500)
 
# Points of local max and min (pi/2 and 3pi/2 for sin(x))
x_max = np.pi / 2
x_min = 3 * np.pi / 2
y_max = f(x_max)
y_min = f(x_min)
 
# Tangent lines at local max and min
tangent_max_x = np.linspace(x_max - 1, x_max + 1, 100)
tangent_max_y = df(x_max) * (tangent_max_x - x_max) + y_max
 
tangent_min_x = np.linspace(x_min - 1, x_min + 1, 100)
tangent_min_y = df(x_min) * (tangent_min_x - x_min) + y_min
 
# Create plot
plt.figure(figsize=(6, 6))
plt.plot(x_vals, f(x_vals), label=r'$f(x) = \sin(x)$', color='black')
 
# Plot points of local max and min
plt.scatter([x_max, x_min], [y_max, y_min], color='black', zorder=5)
 
# Plot tangent lines
plt.plot(tangent_max_x, tangent_max_y, color='red', linestyle='-', linewidth=2)
plt.plot(tangent_min_x, tangent_min_y, color='red', linestyle='-', linewidth=2)
 
# Add labels and title
plt.text(x_max + 0.1, y_max + 0.01, 'Local Max', fontsize=12)
plt.text(x_min + 0.1, y_min - 0.3, 'Local Min', fontsize=12)
plt.title(r'Demonstration of Local Max and Min of $f(x) = \sin(x)$')
 
# Add axes
plt.axhline(0, color='black',linewidth=0.5)
plt.axvline(0, color='black',linewidth=0.5)
 
# Show plot
plt.show()

Calculus 1: local extrema correspond to horizontal tangent lines

But what about multivariable functions in the 3D plane?

f(x)=4x^2

Extrema in 3D

Suppose a local extremum occurs at point P on the surface $z = f (x, y)$

This means that P must simultaneously be an extremum on every trace curve passing through P

MatPlotLib Graph

import micropip
await micropip.install("matplotlib")
await micropip.install("numpy")
 
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
 
from mpl_toolkits.mplot3d import Axes3D
 
# Define the 2-variable function f(x, y)
def f_xy(x, y):
    return -(x**2 + y**2) + 4  # Example of a function with a local maximum at (0, 0)
 
# Create x and y values
x_vals = np.linspace(-2, 2, 400)
y_vals = np.linspace(-2, 2, 400)
x_vals, y_vals = np.meshgrid(x_vals, y_vals)
 
# Calculate z values for the surface
z_vals = f_xy(x_vals, y_vals)
 
# Coordinates for local extremum P
P_x = 0
P_y = 0
P_z = f_xy(P_x, P_y)
 
# plot the surface
fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111, projection='3d')
 
# Plot surface
ax.plot_surface(x_vals, y_vals, z_vals, cmap='Blues', alpha=0.6, edgecolor='none')
 
# Plot point P
ax.scatter(P_x, P_y, P_z, color='purple', s=100, zorder=5)
 
# Highlight the axes and projections (a, b, and P)
ax.plot([P_x, P_x], [P_y, P_y], [0, P_z], color='black', linestyle='--')  # Vertical line to the surface
ax.plot([P_x, P_x], [P_y, 0], [0, 0], color='black', linestyle='--')  # Projection on x-z plane
ax.plot([P_x, 0], [P_y, P_y], [0, 0], color='black', linestyle='--')  # Projection on y-z plane
 
# Cross-sectional lines in red on x and y planes
ax.plot(x_vals[0], [P_y] * len(x_vals[0]), f_xy(x_vals[0], P_y), color='red', linestyle='-', linewidth=2)  # Along y=0
ax.plot([P_x] * len(y_vals[:, 0]), y_vals[:, 0], f_xy(P_x, y_vals[:, 0]), color='red', linestyle='-', linewidth=2)  # Along x=0
 
# Add labels
ax.text(0.1, 0, P_z, 'P', color='purple', fontsize=12)
ax.text(2, 0, 0, 'a', fontsize=12)
ax.text(0, 2, 0, 'b', fontsize=12)
ax.text(0, 0, 4.2, r'$z=f(x, y)$', fontsize=14)
 
# Set labels
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('Z')
 
# Set view angle for better perspective
ax.view_init(45, 235)
 
# Show plot
plt.show()

In Other Words:

The directional derivative at P must equal zero in every possible direction, as the tangent plane must be horizontal!

$D u f (a, b) = 0 \to \nabla f (a, b) \cdot u = 0$

For all $u$ , resulting in:

$\nabla f (a, b) = 0$

What this means:

This says that if $f (a, b)$ is a local maximum or minimum, and if the first order partial derivative exists, then the gradient vector at $(a, b)$ must equal zero.

$f (a, b)$ is a local maximum
- this means that $f (a, b) \geq f (x, y)$ for all $(x, y)$ in some neighborhood of $(a, b)$

At $(a, b)$ , $D u f (a, b) = 0$ for all $u$ in some neighborhood of $(a, b)$

Remarks

This does not say if $\nabla f (a, b) = 0$ , then $(a, b)$ corresponds to a local extremum. However, it does suggest that we can find the local extrema by looking for where $\nabla f = 0$
More specifically, " $\nabla f = 0$ " means $⟨ f_{x}, f_{y} ⟩ = ⟨ 0, 0 ⟩$ , ie. $f_{x} = 0$ AND $f_{y} = 0$ .
The points where $\nabla f = 0$ are called Critical Points. So, just like in Calculus 1, the local extrema “candidates” occur at Critical Points

Problems

Example

Find the local extrema of $f (x, y) = x^{2} + y^{2} - 2 x - 6 y + 14$

Solution

To approach this problem, we need to start by finding the Critical Points:

$\nabla f = 0 \to ⟨ 2 x - 2, 2 y - 6 ⟩ = ⟨ 0, 0 ⟩$

$\to 2 x - 2 = 0 an d 2 y - 6 = 0$

$\to x = 1, y = 3$

Observe that $f (1, 3) = 1 + 9 - 2 - 18 + 14 = 4$ , and by completing the square, we get that $f (x, y) = [(x - 1)^{2} - 1] + [(y - 3)^{2} - 9] + 14$ , resulting in:

$= (x - 1)^{2} + (y - 3)^{2} + 4$

Because $(x - 1)^{2}$ and $(y - 3)^{2}$ are always non-negative, $f (x, y) \geq 4$ for all $(x, y)$ . Therefore, $f (1, 3) = 4$ is a local minimum

Saddle Points

Saddle points are points where the gradient is zero, but the point is neither a maximum nor a minimum.

import micropip
await micropip.install("matplotlib")
await micropip.install("numpy")
import matplotlib.pyplot as plt
import numpy as np
from mpl_toolkits.mplot3d import Axes3D
 
# Create a grid of points
x = np.linspace(-5, 5, 100)
y = np.linspace(-5, 5, 100)
X, Y = np.meshgrid(x, y)
 
# Define the saddle function
Z = X**2 - Y**2  # Classic saddle point equation
 
# Create a 3D plot
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
 
# Plot the surface
ax.plot_surface(X, Y, Z, cmap='coolwarm')
 
# Mark the saddle point at (0, 0)
saddle_x, saddle_y = 0, 0
ax.scatter(saddle_x, saddle_y, saddle_x**2 - saddle_y**2, color='r', s=100, label='Saddle Point')
 
# Set labels
ax.set_xlabel('X axis')
ax.set_ylabel('Y axis')
ax.set_zlabel('Z axis')
ax.set_title('Saddle Point: $z = x^2 - y^2$')
 
# Show plot
plt.show()

Saddle Points present a unique challenge in optimization, as they are neither maxima nor minima. They are points where the gradient is zero, but the point is neither a maximum nor a minimum.

Second Derivative Test

Hessian Matrix

The Hessian Matrix is a square matrix of second-order partial derivatives of a scalar-valued function.

$H = [f * xx f * y x f * x y f * yy]$

We need to know the eigenvalues of the Hessian Matrix to determine the nature of the critical point.

$d e t (H - λ I) = 0$

Now that we know this, we can move on tho the Second Partial Derivative Test.

Second Partial Derivative Test

Suppose the second order partial derivative $f (x, y)$ are continuous, and that $(a, b)$ is a Critical Point.

If $d e t (H) > 0$ and $f_{xx} (a, b) > 0$ , then $f (a, b)$ is a local minimum.
If the Hessain Matrix is positive definite, then the point is a local minimum.
If the Hessina Matrix is negative definite, then the point is a local maximum.
- This becomes a saddle point if the determinant is negative.
If the determinant is zero, the test is inconclusive.

Graph View

14.7 - Multivariable Optimizations

MatPlotLib Graph

Extrema in 3D

MatPlotLib Graph

Remarks

Problems

Saddle Points

Second Derivative Test

Hessian Matrix

Second Partial Derivative Test

Table of Contents

Explore

Explore