A companion to our thermoelectric SVD lesson

A gentle introduction to the math behind SVD and PCA

Let us build the ideas together, from two vectors to the geometry of SVD and PCA.

I will develop these ideas in two dimensions so that you can connect vector and matrix algebra with coordinate geometry. We can draw the arrows, change them, and see what our calculations mean. The same ideas extend to three dimensions and then to n dimensions, even when drawing the whole space becomes difficult.

This cannot be a detailed course in linear algebra. I will try to build the ideas as intuitively as possible. My only request is that you stay with me. Let us begin with two vectors and see where they take us.

← Back to our thermoelectric SVD walkthrough

Let us start with one arrow

We are familiar with vectors in one, two and three dimensions. In a plane, we can describe a vector using two components.

Take the vector

\[\mathbf v=\begin{bmatrix}2\\1\end{bmatrix}.\]

Starting at the origin, move two units horizontally and one unit vertically. The arrow from the origin to that point represents v.

I have written its components vertically, as a column vector. We could write the same components horizontally as the row vector \(\begin{bmatrix}2&1\end{bmatrix}\). For now, I will use column vectors so that our notation stays consistent.

Before adding anything else, look at this arrow. Its components describe both its length and its direction.

Find the endpoint (2, 1). Follow the horizontal and vertical movements that take you there.

Two components, one arrow

v011

The dashed guides connect the endpoint to its two coordinates. All the vectors in this opening are dimensionless teaching examples.

What happens when you multiply the vector by a number?

Let us multiply v by two:

\[2\mathbf v=2\begin{bmatrix}2\\1\end{bmatrix}=\begin{bmatrix}4\\2\end{bmatrix}.\]

Both components double. The new arrow points in the same direction and has twice the length.

What would multiplying by one-half do? You would get a shorter arrow along the same direction.

Now try a negative multiplier:

\[-\mathbf v=\begin{bmatrix}-2\\-1\end{bmatrix}.\]

The arrow reverses direction. Multiplying by zero gives the zero vector, whose endpoint is the origin.

Let us call the multiplier \(c\). As you change \(c\), the endpoint of \(c\mathbf v\) moves along a straight line through the origin.

You can make the arrow longer, shorter or point the other way. But scaling this one vector cannot take you away from its line.

Move the multiplier through positive, zero and negative values. Watch the endpoint trace the same line.

Explore positive, zero and negative scaling

v2v011

The pale arrow is the original v. The teal arrow is cv. The dotted line shows every possible multiple of v.

Can a second vector help us reach somewhere new?

I will now introduce another vector:

\[\mathbf w=\begin{bmatrix}1\\2\end{bmatrix}.\]

This arrow points in a different direction from v.

Let us add them:

\[\mathbf v+\mathbf w=\begin{bmatrix}2\\1\end{bmatrix}+\begin{bmatrix}1\\2\end{bmatrix}=\begin{bmatrix}3\\3\end{bmatrix}.\]

The arithmetic is familiar: add the corresponding components. But look at what happens geometrically.

First, follow v from the origin. Then place the start of w at the tip of v, keeping w’s length and direction unchanged. Following both arrows takes you to (3, 3).

The arrow directly from the origin to that endpoint is v + w.

You could also follow w first and then v. The route changes, but the endpoint stays the same.

Follow the arrows in both orders. Notice how their sum forms the diagonal of the parallelogram.

Two routes to the same endpoint

vv+ww, moved to the tip of v011

We move an arrow without changing its length or direction. Both routes end at (3, 3).

What if you could scale both arrows?

We have scaled one vector and added two vectors. Let us put those operations together:

\[c\mathbf v+d\mathbf w.\]

Here, \(c\) controls how much of v we use, and \(d\) controls how much of w we use. After scaling them, we add the resulting arrows.

For example,

\[2\mathbf v-\mathbf w=2\begin{bmatrix}2\\1\end{bmatrix}-\begin{bmatrix}1\\2\end{bmatrix}=\begin{bmatrix}3\\0\end{bmatrix}.\]

We have reached a point on the horizontal axis, even though neither original arrow points horizontally.

This expression is called a linear combination. The name describes the two operations you have just performed: multiply the vectors by numbers, then add them.

Now I would like you to explore a question: what points can you reach by changing both multipliers?

Change c and d independently. Watch the two scaled arrows and the endpoint of their sum.

Choose how much of each direction to use

vwv+w011

Teal is cv; orange is dw placed at its tip; the dark arrow is their sum. Try c = 2 and d = −1.

Can we reach every point in the plane?

With our chosen vectors, we can.

Let us try to reach the point (1, 0). We need numbers \(c\) and \(d\) such that

\[c\begin{bmatrix}2\\1\end{bmatrix}+d\begin{bmatrix}1\\2\end{bmatrix}=\begin{bmatrix}1\\0\end{bmatrix}.\]

Comparing the components gives us two equations:

\[2c+d=1,\qquad c+2d=0.\]

Their solution is \(c=2/3\) and \(d=-1/3\). So these two sloping arrows can combine to produce a horizontal unit vector.

We could similarly reach a vertical unit vector. Once we can make both coordinate directions, we can combine them to reach any point in the plane.

The set of all vectors we can make from our chosen vectors is called their span. For this pair, the span is the whole plane.

But would that be true for any two vectors?

Make a horizontal or vertical unit vector

vwv+w011

The coefficients are calculated exactly as fractions. The graph shows the resulting vector sum.

Let us choose a pair that cannot reach the whole plane

Keep v, but replace w with

\[\mathbf u=\begin{bmatrix}4\\2\end{bmatrix}.\]

You may recognize this arrow: u = 2v. It is longer than v, but it lies along the same line.

Now consider any combination of them:

\[c\mathbf v+d\mathbf u=c\mathbf v+2d\mathbf v=(c+2d)\mathbf v.\]

Whatever numbers you choose, the result is still a multiple of v. We remain on its line.

The second arrow has not given us a new direction.

We call this pair linearly dependent because one vector can be made from the other. Our earlier pair, v and w, is linearly independent: neither is a multiple of the other.

In two dimensions, two independent vectors span the plane. Two nonzero dependent vectors span only a line.

Switch between the two pairs. With one pair, your endpoint can explore the plane. With the other, it remains trapped on a line.

Does the second arrow add a new direction?

vu = 2v011

Faint dots show a finite sample of reachable combinations, using coefficients from −2 to 2. They illustrate a patch of the plane or part of the line; the complete span is infinite.

Now let us put the two vectors into a matrix

We have been working with the vectors individually. I will now place our independent vectors side by side:

\[A=\begin{bmatrix}2&1\\1&2\end{bmatrix}=\begin{bmatrix}\mathbf v&\mathbf w\end{bmatrix}.\]

The first column is v. The second column is w.

Let us also collect the two multipliers into a column vector:

\[\mathbf x=\begin{bmatrix}c\\d\end{bmatrix}.\]

What happens when we multiply A by x?

\[A\mathbf x=\begin{bmatrix}2&1\\1&2\end{bmatrix}\begin{bmatrix}c\\d\end{bmatrix}=\begin{bmatrix}2c+d\\c+2d\end{bmatrix}.\]

Compare that with the combination we have been drawing:

\[c\mathbf v+d\mathbf w=\begin{bmatrix}2c+d\\c+2d\end{bmatrix}.\]

They are the same.

Multiplying this matrix by a vector produces a combination of its columns. The entries of x tell us how much of each column to use.

We have arrived at matrix–vector multiplication through the arrows you have already explored.

The two columns of A, drawn as arrows

column vcolumn w011

The teal arrow contains the first column’s entries (2, 1); the orange arrow contains the second column’s entries (1, 2).

What does the matrix do to our coordinate directions?

The horizontal and vertical unit vectors are

\[\mathbf e_1=\begin{bmatrix}1\\0\end{bmatrix},\qquad\mathbf e_2=\begin{bmatrix}0\\1\end{bmatrix}.\]

Multiplying by e₁ selects the first column:

\[A\mathbf e_1=\mathbf v.\]

Multiplying by e₂ selects the second:

\[A\mathbf e_2=\mathbf w.\]

So the columns of A tell us where the matrix sends the original coordinate directions.

Any input vector can be written as

\[\mathbf x=c\mathbf e_1+d\mathbf e_2.\]

After applying A, it becomes

\[A\mathbf x=c\mathbf v+d\mathbf w.\]

Look at the coordinate square in the picture. Its horizontal and vertical edges become our two column vectors, turning the square into a parallelogram.

If the columns were dependent, that square would collapse onto a line.

Compare the independent and dependent matrices. Watch what happens to the coordinate square and to an arrow inside it.

\[A=\begin{bmatrix}2&1\\1&2\end{bmatrix}\]
\[D=\begin{bmatrix}2&4\\1&2\end{bmatrix}\]

Columns tell us where the coordinate directions go

Ae₁ = vAe₂ = w011

The left panel is the input plane; the right is its image under the selected matrix. Both panels use the same unit scale. The coordinate square becomes a parallelogram under A and a line segment under D.

Where will this take us?

We now have two connected ways to understand \(A\mathbf x\): calculate it from the entries, or see it as a combination of column arrows.

I would next like to ask a more revealing question:

Can we choose input directions that make the matrix’s action especially simple?

That question will lead us towards SVD. We will investigate how a matrix changes lengths and directions, and whether we can separate those changes into simpler operations. Then we will return to our thermoelectric measurements and give the factors in their decomposition a geometric meaning.

How long is our vector?

We have been drawing arrows and changing their lengths. Let us now put a number to that length.

Return to our first vector:

\[\mathbf v=\begin{bmatrix}2\\1\end{bmatrix}.\]

Its horizontal and vertical components form the two shorter sides of a right triangle. The arrow is the hypotenuse. Using Pythagoras, its length is

\[\|\mathbf v\|=\sqrt{2^2+1^2}=\sqrt5.\]

I will use \(\|\mathbf v\|\) to mean “the length of v.”

The same calculation works in three dimensions: square the three components, add them and take the square root. In n dimensions, we do the same with all n components.

Now suppose we want to keep the direction of v, but make its length equal to one. We divide every component by its length:

\[\widehat{\mathbf v}=\frac{\mathbf v}{\|\mathbf v\|}=\frac1{\sqrt5}\begin{bmatrix}2\\1\end{bmatrix}.\]

This is a unit vector. It points in the same direction as v, but its length is one.

Why will this help us? If we give a matrix input arrows of equal length, we can compare how strongly it stretches different directions.

Compare v with its unit vector. Notice that the direction stays the same while the length changes.

From a right triangle to a unit vector

v: length √5unit v011

The dashed horizontal and vertical legs make the right triangle. The teal unit arrow has the same direction and length 1.

How can we describe the angle between two vectors?

Let us bring back our second vector:

\[\mathbf w=\begin{bmatrix}1\\2\end{bmatrix}.\]

You can see that v and w point in different directions. How could we calculate the angle between them?

We multiply corresponding components and add:

\[\mathbf v^{\mathsf T}\mathbf w=2(1)+1(2)=4.\]

This is their dot product. You already recognize the row-by-column multiplication. Its geometric meaning comes from

\[\mathbf v^{\mathsf T}\mathbf w=\|\mathbf v\|\,\|\mathbf w\|\cos\theta,\]

where \(\theta\) is the angle between the vectors. To obtain the cosine, we must divide the dot product by both vector lengths:

\[\cos\theta=\frac{\mathbf v^{\mathsf T}\mathbf w}{\|\mathbf v\|\,\|\mathbf w\|}.\]

This ratio lies between −1 and 1. Only when both vectors have length one does their dot product equal the cosine itself.

Both vectors have length \(\sqrt5\), so

\[\cos\theta=\frac4{\sqrt5\sqrt5}=\frac45.\]

Their angle is approximately 36.9°.

Now consider another arrow:

\[\mathbf p=\begin{bmatrix}-1\\2\end{bmatrix}.\]

Its dot product with v is

\[\mathbf v^{\mathsf T}\mathbf p=2(-1)+1(2)=0.\]

The arrows are perpendicular.

This gives us a useful connection: we can recognize a right angle through a zero dot product, even in dimensions we cannot draw.

When two vectors are perpendicular and both have length one, we call them orthonormal. The word combines two properties you can now picture: a right angle and unit lengths.

Turn one arrow. Watch its dot product with the other change. Find the position where the angle is a right angle and the dot product becomes zero.

Connect the dot product with the angle

vwp: perpendicular to vw, moved to the tip of v011

The fixed arrow is v = (2, 1). The moving arrow keeps length √5. The arc shows the angle between them; the dot product is positive below 90°, zero at 90°, and negative above 90°.

Does a matrix preserve those right angles?

Our earlier matrix helped us understand multiplication as a combination of columns. I would now like to introduce a different matrix:

\[B=\begin{bmatrix}4&0\\3&5\end{bmatrix}.\]

I have chosen a nonsymmetric matrix: its entries on opposite sides of the diagonal differ.

Let us apply it to the two perpendicular coordinate directions:

\[\mathbf e_1=\begin{bmatrix}1\\0\end{bmatrix},\qquad\mathbf e_2=\begin{bmatrix}0\\1\end{bmatrix}.\]

You know what happens: we obtain the columns of B.

\[B\mathbf e_1=\begin{bmatrix}4\\3\end{bmatrix},\qquad B\mathbf e_2=\begin{bmatrix}0\\5\end{bmatrix}.\]

Are these output arrows perpendicular?

Their dot product is 4(0) + 3(5) = 15. Each output vector has length 5. Using the full dot-product formula,

\[15=5\times5\cos\theta,\qquad\cos\theta=\frac{15}{5\times5}=0.6.\]

The output angle is therefore approximately 53.13°, rather than 90°. The number 15 is the dot product; 0.6 is the cosine.

The matrix has changed the angle between our original coordinate directions.

This leads us to a question:

Could we choose another pair of perpendicular input directions whose outputs remain perpendicular?

Compare the original coordinate square with its transformed parallelogram. Notice how a right angle changes.

Perpendicular inputs need not remain perpendicular

Coordinate inputse₁e₂Outputs under BBe₁Be₂

Both panels use the same unit scale. Teal and orange identify the two input arrows and their corresponding outputs. The dotted square becomes a parallelogram.

Let us follow one useful input direction

I will first show you a useful direction, then we will see how to find it mathematically.

Take an input arrow along (1, 1). To give it length one, divide by \(\sqrt2\):

\[\mathbf v_1=\frac1{\sqrt2}\begin{bmatrix}1\\1\end{bmatrix}.\]

Apply B:

\[B\mathbf v_1=\frac1{\sqrt2}\begin{bmatrix}4\\8\end{bmatrix}.\]

Look at what has happened. The input pointed equally horizontally and vertically. The output points along (1, 2), so its direction has changed.

Its length is

\[\|B\mathbf v_1\|=\sqrt{\frac{4^2+8^2}{2}}=\sqrt{40}.\]

Our input had length one. Therefore, B stretches this input direction by a factor of \(\sqrt{40}\).

Let us call that stretch \(s_1\):

\[s_1=\sqrt{40}.\]

We can describe the output direction separately using a unit vector:

\[\mathbf u_1=\frac1{\sqrt5}\begin{bmatrix}1\\2\end{bmatrix}.\]

Now we can write the whole transformation of this input as

\[\boxed{B\mathbf v_1=s_1\mathbf u_1.}\]

Read this with me:

“B takes the unit input direction v₁, changes its direction to u₁, and gives it length \(s_1\).”

We have separated where the output points from how long it is.

Follow v₁ into B. Compare its direction with u₁, and its unit length with the output length s₁.

One input direction, one stretch, one output direction

Unit input v₁v₁Full output and unit outputBv₁u₁

The input v₁ has length 1. The full output Bv₁ has length √40. The shorter teal output arrow is u₁: the same output direction, reduced to length 1.

Can we find a second pair?

Choose the perpendicular unit input

\[\mathbf v_2=\frac1{\sqrt2}\begin{bmatrix}-1\\1\end{bmatrix}.\]

Applying B gives

\[B\mathbf v_2=\frac1{\sqrt2}\begin{bmatrix}-4\\2\end{bmatrix}.\]

This output has length \(\sqrt{10}\), and its unit direction is

\[\mathbf u_2=\frac1{\sqrt5}\begin{bmatrix}-2\\1\end{bmatrix}.\]

So our second relationship is

\[\boxed{B\mathbf v_2=s_2\mathbf u_2,\qquad s_2=\sqrt{10}.}\]

Now check the picture.

The input directions v₁ and v₂ are perpendicular. Their output directions u₁ and u₂ are also perpendicular.

But the two stretches differ: \(s_1\) is twice \(s_2\).

If we transform a whole circle of unit input arrows, these two pairs identify the long and short axes of the resulting ellipse.

The numbers \(s_1\) and \(s_2\) are called the singular values. We have reached their meaning through lengths.

Show both pairs together. Find the perpendicular input arrows, their perpendicular outputs, and the two different stretch factors.

The special perpendicular inputs and their outputs

Perpendicular unit inputsv₁v₂Perpendicular stretched outputss₁u₁s₂u₂

Teal follows v₁ to s₁u₁; orange follows v₂ to s₂u₂. The faint unit circle becomes an ellipse, with different long and short axis lengths.

Now let us collect those relationships into matrices

We have two statements:

\[B\mathbf v_1=s_1\mathbf u_1,\qquad B\mathbf v_2=s_2\mathbf u_2.\]

Before collecting them, let us give the two kinds of direction their names.

Each vᵢ is a special unit input direction. In \(B\mathbf v_i\), it multiplies B from the right. We call it a right singular vector. I put these input directions into the columns of V, in order of their corresponding stretches:

\[V=\begin{bmatrix}\mathbf v_1&\mathbf v_2\end{bmatrix}=\frac1{\sqrt2}\begin{bmatrix}1&-1\\1&1\end{bmatrix}.\]

Each uᵢ is the matching unit output direction. We call it a left singular vector. To see the other side of the pairing, we can also write

\[\mathbf u_i^{\mathsf T}B=s_i\mathbf v_i^{\mathsf T}.\]

Here the transposed output vector multiplies B from the left. The names “right” and “left” identify the input and output sides of the transformation.

I collect these output directions into the columns of U, keeping the same order as their matching input directions:

\[U=\begin{bmatrix}\mathbf u_1&\mathbf u_2\end{bmatrix}=\frac1{\sqrt5}\begin{bmatrix}1&-2\\2&1\end{bmatrix}.\]

Every column of U and V has length one, and different columns within either matrix are perpendicular. The first column of V is paired with the first column of U; the second is paired with the second. We can therefore follow each direction through the transformation without mixing up the pairs.

The matrix \(\Sigma\), read “capital sigma,” collects the singular values on its diagonal in that same order:

\[\Sigma=\begin{bmatrix}s_1&0\\0&s_2\end{bmatrix}=\begin{bmatrix}\sqrt{40}&0\\0&\sqrt{10}\end{bmatrix}.\]

Here I write the singular values as \(s_1,s_2\); they are also commonly written \(\sigma_1,\sigma_2\). Every off-diagonal entry of \(\Sigma\) is zero.

Our two vector relationships now become one matrix relationship:

\[BV=U\Sigma.\]

Because V contains a complete perpendicular unit basis, its inverse is \(V^{\mathsf T}\). Multiplying on the right gives

\[\boxed{B=U\Sigma V^{\mathsf T}.}\]

This is the singular value decomposition. Although the formula uses \(V^{\mathsf T}\), the right singular vectors are the columns of V; transposing puts them into the rows of \(V^{\mathsf T}\). The left singular vectors remain the columns of U.

The three matrices now have jobs you have already seen:

  • \(V^{\mathsf T}\) expresses an input along the right singular directions.
  • \(\Sigma\) stretches those coordinates by their singular values.
  • U combines the corresponding left singular directions to form the output.

For our square two-dimensional example, the orientation changes can be turns or reflections. They preserve lengths; the singular values describe the stretching.

Follow one vector through the three SVD operations

Input xxAfter VᵀVᵀxAfter ΣΣVᵀxAfter U = direct BxBx

The same unit input passes through Vᵀ, Σ and U. The nonsymmetric B requires different input and output directions. All four panels use the same coordinate scale.

Why does this describe every input vector?

You might reasonably ask: “We checked two input directions. How do we know the factorization works for every other arrow?”

Our perpendicular unit directions form a basis for the plane. Any input can be written as

\[\mathbf x=c_1\mathbf v_1+c_2\mathbf v_2.\]

Matrix multiplication preserves this combination:

\[B\mathbf x=c_1B\mathbf v_1+c_2B\mathbf v_2=c_1s_1\mathbf u_1+c_2s_2\mathbf u_2.\]

Once we know what B does to the two basis directions, we know what it does to every combination of them.

That is why we can decompose the whole matrix by understanding these particular vector pairs.

In the visual above, turn an arbitrary input arrow. Compare applying B directly with passing the same arrow through Vᵀ, then Σ, then U.

How did we find those special input directions?

We have described the useful directions. Let us now connect them to the eigenvectors you met earlier.

For any unit input x, the squared output length is

\[\|B\mathbf x\|^2=(B\mathbf x)^{\mathsf T}(B\mathbf x)=\mathbf x^{\mathsf T}(B^{\mathsf T}B)\mathbf x.\]

The symmetric matrix \(B^{\mathsf T}B\) contains the information about how B stretches different input directions.

For our example,

\[B^{\mathsf T}B=\begin{bmatrix}25&15\\15&25\end{bmatrix}.\]

Its perpendicular unit eigenvectors are precisely v₁ and v₂, with eigenvalues 40 and 10.

These eigenvalues give squared output lengths. Taking their square roots gives our singular values:

\[s_1=\sqrt{40},\qquad s_2=\sqrt{10}.\]

The largest eigenvalue identifies the greatest stretch. Once we know an input direction and its stretch, we obtain the output direction from

\[\mathbf u_i=\frac{B\mathbf v_i}{s_i}\quad(s_i>0).\]

In eigenvector language, the right singular vectors are eigenvectors of \(B^{\mathsf T}B\), and the matching left singular vectors are eigenvectors of \(BB^{\mathsf T}\). Both use the same squared singular values as their eigenvalues.

Now you can see where all three parts of the decomposition come from.

This explains the relationship mathematically. For numerical work, we calculate SVD directly with a library routine rather than explicitly forming BᵀB.

What changes when we move to n dimensions?

We drew two-dimensional arrows so that you could see the geometry. Let us now keep the same algebra and allow more components.

\[\mathbf x=\begin{bmatrix}x_1\\x_2\\\vdots\\x_n\end{bmatrix}.\]

An n-dimensional input has n components. If A has m rows and n columns, it takes an n-component input and produces an m-component output. The input and output spaces can have different dimensions.

Each column of A has m components. Matrix–vector multiplication still combines those columns:

\[A\mathbf x=x_1\mathbf a_1+x_2\mathbf a_2+\cdots+x_n\mathbf a_n.\]

The geometric ingredients extend too. Length comes from the sum of squared components, and the dot product still connects lengths and angles:

\[\|\mathbf x\|=\sqrt{\sum_{j=1}^{n}x_j^2},\qquad\mathbf a^{\mathsf T}\mathbf b=\|\mathbf a\|\,\|\mathbf b\|\cos\theta.\]

The angle formula applies to nonzero vectors in the same space. We no longer need a drawing to test whether directions are perpendicular: their dot product is zero.

Now carry our useful vector-pair relationship into these larger spaces:

\[A\mathbf v_i=s_i\mathbf u_i.\]

Here vᵢ is a unit input direction with n components, uᵢ is a unit output direction with m components, and \(s_i\) is its stretch. The input directions are mutually perpendicular, and so are the output directions.

Some input directions may collapse to zero. Let r be the number of nonzero singular values—the rank of A. Collecting the r contributing pairs gives the compact SVD:

\[\underbrace{A}_{m\times n}=\underbrace{U_r}_{m\times r}\underbrace{\Sigma_r}_{r\times r}\underbrace{V_r^{\mathsf T}}_{r\times n}.\]

The columns of \(U_r\) and \(V_r\) are perpendicular unit vectors. The diagonal of \(\Sigma_r\) holds the positive singular values, largest first. Directions that A sends to zero need no contribution to this sum.

\[A=\sum_{i=1}^{r}s_i\mathbf u_i\mathbf v_i^{\mathsf T}.\]

Each term is an m-by-n table made from one output column, one stretch and one input row. Adding all r terms recovers A. Keeping fewer terms gives an approximation.

The compact factors can be rectangular, so we should not call every factor a rotation matrix. \(V_r^{\mathsf T}\) extracts coordinates along the contributing input directions; \(\Sigma_r\) scales them; and \(U_r\) combines the output directions. The vector-pair idea remains the same even when we cannot draw either space.

Let us return to the geometry of PCA

So far, we have watched a matrix transform input arrows. For PCA, let us return to a collection of measured materials.

Begin with two properties so that we can draw each observation as a point. I will first subtract the mean of each property.

This moves the centre of the data to the origin. It preserves the relative positions of the points.

We would like to describe the data using a line through that centre. But which line should we choose?

Take one centred observation x and a unit direction q along a candidate line.

Its signed position along the line is

\[t=\mathbf x^{\mathsf T}\mathbf q.\]

This is its projection, or score, along q. The point on the line is \(t\mathbf q\).

The part left over is

\[\mathbf r=\mathbf x-t\mathbf q.\]

Look at the right triangle in the picture: the projected part lies along the line, while the remaining part is perpendicular to it.

Pythagoras gives

\[\|\mathbf x\|^2=t^2+\|\mathbf r\|^2.\]

Turn the candidate line. Watch the projected point move along it and the perpendicular distance change.

Follow one real observation onto a candidate PCA line

actual record 1projection at 45°011

Every faint point is one of the 5,205 actual ESTM observations. Axes are standardized log electrical and thermal conductivity, prepared as in our PCA lesson. Orange marks your observation; teal marks its projection; the dashed segment is the perpendicular residual. We use the full dataset to calculate both column means and standard deviations.

A closer look at the projection triangle

actual record 1projection at 45°011

This close-up follows the same observation and candidate line as the plot above. Its larger coordinate scale makes the projected part and perpendicular residual easier to see. The shaded triangle connects the origin, the projection and the original observation.

Why does keeping the greatest variance help?

Now repeat that separation for every observation.

\[\sum_i\|\mathbf x_i\|^2=\sum_i t_i^2+\sum_i\|\mathbf r_i\|^2.\]

The left-hand side is fixed: rotating our candidate line does not change the original data.

Therefore, keeping more squared variation along the line leaves less squared distance away from it.

This connects two descriptions of the same choice:

  • Find the line along which the projected observations have the greatest variance.
  • Find the line that minimizes the sum of squared perpendicular distances from the observations.

Because the data are centred, the projected scores have mean zero. Let N be the number of observations: N = 5,205 in this dataset. Their sample variance is

\[\frac{\sum_i t_i^2}{N-1}.\]

So maximizing that variance also minimizes the reconstruction error measured by those perpendicular distances.

This is why we seek maximum variance in PCA. We are choosing the line that preserves the most variation under this geometric measure.

That does not guarantee that the retained pattern is the most important one physically. We still need our materials knowledge to interpret it.

Retained variation and lost variation trade places

1RetainedResidualCandidate angle: 0° to 180°

The curves use all 5,205 standardized conductivity pairs. At every angle, retained variance plus residual variance equals the same total variance. The maximum retained variance and minimum loss occur together.

Where do SVD and PCA meet?

Let X contain our centred observations as rows.

For a candidate direction q, all the projection scores are collected in \(X\mathbf q\). Their total squared size is

\[\|X\mathbf q\|^2=\mathbf q^{\mathsf T}X^{\mathsf T}X\mathbf q.\]

You have seen this expression before. It is the same stretching question we investigated with our small matrix.

The first right singular vector of X gives the direction with the greatest projected variation. Its scores are

\[X\mathbf v_1=s_1\mathbf u_1.\]

Their sample variance is

\[\frac{s_1^2}{N-1}.\]

We can now return to our four-property thermoelectric table. We cannot draw all four coordinates together, but the algebra follows the same reasoning.

You have followed the path from arrows and their combinations to a matrix transformation, from that transformation to SVD, and from SVD to the geometry of PCA.

Let us now use those connections to look again at the materials data—and ask what its strongest patterns might be telling us.

The matrix B above is a geometric teaching example. The projection visuals return to the real ESTM conductivity data. The full four-property analysis also includes temperature and Seebeck coefficient, so it has different singular values and variance fractions from the two-property view.

Let us write the actual thermoelectric SVD

Our full table has N = 5,205 observation rows and n = 4 prepared property columns: temperature, Seebeck coefficient, log₁₀ electrical conductivity and log₁₀ thermal conductivity. We subtract each column’s mean and divide by its sample standard deviation, as in the main PCA walkthrough.

All four singular values are nonzero, so r = 4. The compact factors have these sizes:

\[\underbrace{X}_{5205\times4}=\underbrace{U}_{5205\times4}\underbrace{\Sigma}_{4\times4}\underbrace{V^{\mathsf T}}_{4\times4}.\]

The actual singular values give us

\[\Sigma\approx\begin{bmatrix}94.0490&0.0000&0.0000&0.0000\\0.0000&72.2134&0.0000&0.0000\\0.0000&0.0000&63.1130&0.0000\\0.0000&0.0000&0.0000&52.6571\end{bmatrix}\]

Here are the property directions, in the same column order as the prepared table:

\[V^{\mathsf T}\approx\begin{bmatrix}-0.11885&-0.50910&0.61775&0.58744\\0.97221&0.10871&0.18177&0.09977\\-0.17554&0.82732&0.16570&0.50722\\0.09935&-0.21105&-0.74692&0.62265\end{bmatrix}\]

U has a row for every observation. I show its first two rows, followed by an ellipsis for the remaining rows:

\[U\approx\begin{bmatrix}0.0015466&-0.0189471&-0.0150780&-0.0119934\\0.0004914&-0.0124694&-0.0193506&-0.0108868\\\vdots&\vdots&\vdots&\vdots\end{bmatrix}\]

These factors were fitted to all 5,205 observations. The displayed values are rounded; the full-precision factors reconstruct the prepared table to numerical precision.

\[X=s_1\mathbf u_1\mathbf v_1^{\mathsf T}+s_2\mathbf u_2\mathbf v_2^{\mathsf T}+s_3\mathbf u_3\mathbf v_3^{\mathsf T}+s_4\mathbf u_4\mathbf v_4^{\mathsf T}.\]

You can now read the decomposition both ways: as three matrix factors, or as four simple component tables whose sum rebuilds the data.

Inspect every row of U · Inspect the singular values · Inspect Vᵀ · Inspect the prepared X

Follow your observation through the numerical reconstruction →