Gradient Descent From First Principles

How Models Learn From Error Gradients

Gradient descent updates model parameters in the opposite direction of the loss gradient: $w \leftarrow w - \eta \frac{\partial L}{\partial w}$, where $\eta$ is the learning rate.

Important Choosing the Learning Rate $\eta$ If $\eta$ is too small, convergence is slow; if $\eta$ is too large, updates overshoot the minimum and diverge.

# Minimize L(w) = (w - 3.5)^2 using gradient descent
w = 0.0
learning_rate = 0.25

for step in range(1, 8):
    grad = 2.0 * (w - 3.5)
    w = w - learning_rate * grad
    loss = (w - 3.5) ** 2
    print(f"Step {step}: w = {w:.4f}, Loss = {loss:.5f}")