The Jacobian is a map’s local gearbox.

For one input and one output, a derivative answers: “if I move right a tiny bit, how much does the output move?” With two inputs and two outputs, there are four such questions. Put their answers in a table:

Here is a complete linear example. Let a map use the matrix

Applied to the point \(p=(2,1)\), matrix multiplication says

The first output coordinate is the first row’s weighted combination of the input coordinates; the second output coordinate is the second row’s combination. A linear map uses the same matrix everywhere. A nonlinear map needs a different best linear matrix at different points—that point-dependent matrix is the Jacobian.

The first column is what a tiny step in input-\(x\) becomes; the second is what a tiny step in input-\(y\) becomes. In the widget, begin at identity: nothing moves. Then try stretch, shear, and flip one at a time. Finally type your own four values and watch the right-hand grid respond. This makes the entries of the matrix feel like instructions rather than a table to memorize.

A matrix sends the unit square’s two edge vectors to its two columns. Those columns make a parallelogram. Its signed area is \(\det J\): absolute value tells how much a tiny patch expands or shrinks; the sign says whether its orientation was flipped.

This is why a determinant matters only for square Jacobians: only then are input and output patches the same dimension, so “volume multiplier” makes sense.

Suppose \(x\) first passes through \(g\), then \(f\). A small arrow is first reshaped by \(J_g(x)\), then by \(J_f(g(x))\). Doing those two matrix actions gives the chain rule:

Backpropagation is this principle applied efficiently to a long chain of neural-network layers.

An ordinary tangent arrow \(v\) says “move this way in the input.” The map carries it forward: \(v\mapsto Jv\). This is the pushforward.

A gradient is different: it is a question that eats a direction and returns a rate of change. If the output has gradient \(a\), the input gradient must give the same rate to every input arrow. The unique answer is \(J^\mathsf{T}a\). Carrying that gradient back is the pullback.

A model is a long composition: input \(\to\) layer 1 \(\to\) layer 2 \(\to\) prediction \(\to\) loss. Training asks one question at the end: which earlier changes would lower this loss? That is a pullback question. If a layer maps \(h\) to \(z\), and the downstream computation has already supplied the output gradient \(g_z=\nabla_z L\), then the gradient for the layer input is

Read this from right to left. \(g_z\) says which output changes matter to the loss; \(J_z^\mathsf{T}\) translates that concern into which input changes matter. Repeating this one layer at a time is backpropagation. It is the chain rule organized so shared intermediate computations are reused instead of rediscovered for every parameter.

For the common affine layer \(z=Wh+b\), the Jacobian with respect to \(h\) is \(W\), so \(\nabla_hL=W^\mathsf{T}g_z\). The same local bookkeeping gives parameter gradients:

This outer product has a useful interpretation. A weight receives a large update only when its input feature was active and its output unit mattered to the loss. With an activation \(z=\sigma(Wh+b)\), the derivative of \(\sigma\) scales each output gradient before this pullback. Repeated derivatives smaller than \(1\) explain vanishing gradients; repeated large derivatives can explain exploding gradients.

Frameworks almost never materialize the enormous Jacobian. A layer with \(m\) outputs and \(n\) inputs would have \(mn\) entries, while training only needs the single product \(J^\mathsf{T}g_z\). Reverse-mode automatic differentiation computes that product directly, at a cost comparable to another pass through the layer. Forward-mode differentiation instead computes \(Jv\), useful when one wants how a chosen input perturbation propagates—for example, sensitivity analysis or some scientific ML models.

Generative normalizing flows need the other half of the Jacobian story. They start with an easy density \(p_X\), often a Gaussian, and learn an invertible warp \(y=f(x)\). A small input patch with probability mass \(p_X(x)\,d x\) becomes an output patch with area \(d y=|\det J_f(x)|\,d x\). Probability mass cannot be created, so the density must compensate:

Taking logs turns a product into an addition, which is how flows train with likelihood:

For a chain of flow layers, their log-determinants simply add. A map must be invertible and have nonzero determinant everywhere used; otherwise distinct input patches collide and no single output density can be recovered. This requirement shapes flow architectures. They deliberately use transformations with cheap determinants—diagonal scalings, triangular Jacobians, or coupling layers—rather than an arbitrary neural network whose determinant would be expensive or whose inverse might not exist.