Skip to content

Commit 67efe9e

Browse files
inducerinducer
committed
Add demo on convergence of steepest descent
1 parent b004482 commit 67efe9e

3 files changed

Lines changed: 663 additions & 6 deletions

File tree

‎cleared-demos/optimization/Convergence of Steepest Descent.ipynb‎

Lines changed: 247 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,247 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "e6e37e33-e07a-47ac-9cc7-62ba940f7bf3",
6+
"metadata": {},
7+
"source": [
8+
"# Convergence of Steepest Descent\n",
9+
"\n",
10+
"Copyright (C) 2026 Andreas Kloeckner\n",
11+
"\n",
12+
"<details>\n",
13+
"<summary>MIT License</summary>\n",
14+
"Permission is hereby granted, free of charge, to any person obtaining a copy\n",
15+
"of this software and associated documentation files (the \"Software\"), to deal\n",
16+
"in the Software without restriction, including without limitation the rights\n",
17+
"to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n",
18+
"copies of the Software, and to permit persons to whom the Software is\n",
19+
"furnished to do so, subject to the following conditions:\n",
20+
"\n",
21+
"The above copyright notice and this permission notice shall be included in\n",
22+
"all copies or substantial portions of the Software.\n",
23+
"\n",
24+
"THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n",
25+
"IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n",
26+
"FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n",
27+
"AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n",
28+
"LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n",
29+
"OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN\n",
30+
"THE SOFTWARE.\n",
31+
"</details>"
32+
]
33+
},
34+
{
35+
"cell_type": "code",
36+
"execution_count": 1,
37+
"id": "43ee5b85-2fab-4e97-9747-3723d3911082",
38+
"metadata": {},
39+
"outputs": [],
40+
"source": [
41+
"import sympy as sp"
42+
]
43+
},
44+
{
45+
"cell_type": "code",
46+
"execution_count": 115,
47+
"id": "a90e5f57-b44e-4f49-8432-1a280cebf422",
48+
"metadata": {},
49+
"outputs": [],
50+
"source": [
51+
"lam1, lam2 = sp.symbols(\"lambda_1, lambda_2\", positive=True)\n",
52+
"u, v = sp.symbols(\"u,v\", positive=True)\n",
53+
"\n",
54+
"def grad(expr, vec):\n",
55+
" return sp.Matrix([expr.diff(vec[0]), expr.diff(vec[1])]) "
56+
]
57+
},
58+
{
59+
"cell_type": "code",
60+
"execution_count": 116,
61+
"id": "31f507ac-2875-4982-b5de-f9e5b199b4df",
62+
"metadata": {},
63+
"outputs": [],
64+
"source": [
65+
"A = sp.Matrix([[lam1, 0], [0, lam2]])\n",
66+
"A"
67+
]
68+
},
69+
{
70+
"cell_type": "code",
71+
"execution_count": 117,
72+
"id": "1d93038c-90ea-4fe9-8b74-16a8e1f0cb9d",
73+
"metadata": {},
74+
"outputs": [],
75+
"source": [
76+
"x = sp.Matrix([u,v])\n",
77+
"x"
78+
]
79+
},
80+
{
81+
"cell_type": "code",
82+
"execution_count": 118,
83+
"id": "79115a9c-d9d0-4def-97c1-f4d6e939535e",
84+
"metadata": {},
85+
"outputs": [],
86+
"source": [
87+
"objective = (sp.Rational(1, 2) * x.T @ A @ x)[0]\n",
88+
"objective"
89+
]
90+
},
91+
{
92+
"cell_type": "markdown",
93+
"id": "1bb41284-6ebe-4a14-a3e9-5350a2d61a30",
94+
"metadata": {},
95+
"source": [
96+
"**Question:** What is the minimizer we're looking for?"
97+
]
98+
},
99+
{
100+
"cell_type": "code",
101+
"execution_count": 119,
102+
"id": "c7636016-95b6-4822-b8c4-467272741cd3",
103+
"metadata": {},
104+
"outputs": [],
105+
"source": [
106+
"grad_objective = grad(objective, x)\n",
107+
"grad_objective"
108+
]
109+
},
110+
{
111+
"cell_type": "markdown",
112+
"id": "fb0a76a9-9f64-4ec5-bd10-900a4a484b81",
113+
"metadata": {},
114+
"source": [
115+
"**Question:** What does this coincide with? Can you prove it?\n",
116+
"\n",
117+
"Set up the line search as `line` and `line_objective`:"
118+
]
119+
},
120+
{
121+
"cell_type": "code",
122+
"execution_count": 120,
123+
"id": "14b51b3d-3d97-4971-b076-ba3e0d7ffae1",
124+
"metadata": {},
125+
"outputs": [],
126+
"source": []
127+
},
128+
{
129+
"cell_type": "markdown",
130+
"id": "1e3b7b6c-bd0b-40af-8193-9046c3bbd855",
131+
"metadata": {},
132+
"source": [
133+
"And find the optimal $\\alpha$:"
134+
]
135+
},
136+
{
137+
"cell_type": "code",
138+
"execution_count": 121,
139+
"id": "f349e3a5-9283-4673-88f7-b57b73830f42",
140+
"metadata": {},
141+
"outputs": [],
142+
"source": []
143+
},
144+
{
145+
"cell_type": "markdown",
146+
"id": "815c073b-b571-47d6-b313-2fa671d436f0",
147+
"metadata": {},
148+
"source": [
149+
"**Question:** What is this in general? Can you prove it?\n",
150+
"\n",
151+
"Next, find the next iterate:"
152+
]
153+
},
154+
{
155+
"cell_type": "code",
156+
"execution_count": 122,
157+
"id": "9415b4f1-265a-47d0-b2ad-b3f587a2f71b",
158+
"metadata": {},
159+
"outputs": [],
160+
"source": [
161+
"x_next = line.subs(alpha, alpha_opt)\n",
162+
"x_next"
163+
]
164+
},
165+
{
166+
"cell_type": "markdown",
167+
"id": "d8c2cb68-894e-4ae6-a745-dc42bd9512e1",
168+
"metadata": {},
169+
"source": [
170+
"Next, consider the decrease in energy error:"
171+
]
172+
},
173+
{
174+
"cell_type": "code",
175+
"execution_count": 140,
176+
"id": "ef221751-72eb-49a5-a59a-7fd6888bf1e9",
177+
"metadata": {},
178+
"outputs": [],
179+
"source": [
180+
"def energy_error_squared(vec):\n",
181+
" return (vec.T @ A @ vec)[0]\n",
182+
"\n",
183+
"ratio = sp.factor(energy_error_squared(x_next)/energy_error_squared(x))\n",
184+
"ratio"
185+
]
186+
},
187+
{
188+
"cell_type": "markdown",
189+
"id": "8074511f-b3aa-496b-8965-e461421291a7",
190+
"metadata": {},
191+
"source": [
192+
"Take gradient, find critical point:"
193+
]
194+
},
195+
{
196+
"cell_type": "code",
197+
"execution_count": 145,
198+
"id": "2bf92061-339d-4620-a19a-30b4a4e4e488",
199+
"metadata": {},
200+
"outputs": [],
201+
"source": [
202+
"bad_soln, = sp.solve(grad(ratio, x), x)\n",
203+
"x_bad = sp.Matrix(bad_soln)\n",
204+
"x_bad"
205+
]
206+
},
207+
{
208+
"cell_type": "code",
209+
"execution_count": 147,
210+
"id": "c803a626-7c11-49c7-9cce-8e9117c096be",
211+
"metadata": {},
212+
"outputs": [],
213+
"source": [
214+
"sp.factor(ratio.subs(u, x_bad[0]).subs(v, x_bad[1]))"
215+
]
216+
},
217+
{
218+
"cell_type": "code",
219+
"execution_count": null,
220+
"id": "196f51cf-8e3d-4a38-bde5-fe6ab81e8b98",
221+
"metadata": {},
222+
"outputs": [],
223+
"source": []
224+
}
225+
],
226+
"metadata": {
227+
"kernelspec": {
228+
"display_name": "Python 3 (ipykernel)",
229+
"language": "python",
230+
"name": "python3"
231+
},
232+
"language_info": {
233+
"codemirror_mode": {
234+
"name": "ipython",
235+
"version": 3
236+
},
237+
"file_extension": ".py",
238+
"mimetype": "text/x-python",
239+
"name": "python",
240+
"nbconvert_exporter": "python",
241+
"pygments_lexer": "ipython3",
242+
"version": "3.14.3"
243+
}
244+
},
245+
"nbformat": 4,
246+
"nbformat_minor": 5
247+
}

0 commit comments

Comments
 (0)