Updating formula for the sample covariance and correlation
The second largest eigenvector is always orthogonal to the largest eigenvector, and points into the direction of the second largest spread of the data. In an earlier article we saw that a linear transformation matrix is completely defined by its eigenvectors and eigenvalues.Applied to the covariance matrix, this means that: where is an eigenvector of , and is the corresponding eigenvalue.If there are no covariances, then both values are equal.Now let’s forget about covariance matrices for a moment.Each of the examples in figure 3 can simply be considered to be a linearly transformed instance of figure 6:where and are the scaling factors in the x direction and the y direction respectively.In the following paragraphs, we will discuss the relation between the covariance matrix , and the linear transformation matrix .If the covariance matrix of our data is a diagonal matrix, such that the covariances are zero, then this means that the variances must be equal to the eigenvalues .
Figure 3 illustrates how the overall shape of the data defines the covariance matrix: In the next section, we will discuss how the covariance matrix can be interpreted as a linear operator that transforms white data into the data we observed.
To investigate the relation between the linear transformation matrix and the covariance matrix in the general case, we will therefore try to decompose the covariance matrix into the product of rotation and scaling matrices.
As we saw earlier, we can represent the covariance matrix by its eigenvectors and eigenvalues: where is an eigenvector of , and is the corresponding eigenvalue.
In this article, we provide an intuitive, geometric interpretation of the covariance matrix, by exploring the relation between linear transformations and the resulting data covariance.
Most textbooks explain the shape of data based on the concept of covariance matrices.