I am trying to resolve my confusion regarding vector projection and projected vectors with projection matrices.
Suppose the column vectors of $\mathbf{G}$ are orthonormal. To my understanding the vector projection of vector $\mathbf{a}$ onto vector $\mathbf{b}$ is $\hat{\mathbf{a}}(\frac{\mathbf{a}\cdot\mathbf{b}}{||\mathbf{a}||||\mathbf{b}||}$). Then how come in, say, PCA the projection of a data matrix $\mathbf{X}$ onto the new orthonormal basis vectors of $\mathbf{G}$ is given by $\mathbf{G}^T(\mathbf{X} - \mathbb{E}(\mathbf{X}))$? Specifically, the first column vector of $\mathbf{X}$ ends up being $\mathbf{G}^T(\mathbf{x}_1 - \mathbb{E}(\mathbf{x}_1)) = \sum_i\mathbf{g}_i^T(\mathbf{x}_1 - \mathbb{E}(\mathbf{x}_1))$, where the $i$th component of the projected version of $(\mathbf{x}_1 - \mathbb{E}(\mathbf{x}_1)$ is given by the dot product with the $i$th column vector of $\mathbf{G}$. But at what point are we dividing with the length of the vector $(\mathbf{x}_1 - \mathbb{E}(\mathbf{x}_1)$ like in the vector projection? Or is there a reason why this is not necessary?
$\endgroup$ 11 Answer
$\begingroup$Suppose that $\mathbf G$ has orthonormal columns $\mathbf g_1,\dots,\mathbf g_k$. Because the columns are orthonormal, we can get the projection onto the span of these vectors by adding up individual projections. That is, the projection a vector $\mathbf a$ is given by$$ \operatorname{proj}_{\mathbf G}(\mathbf a) = \frac{\mathbf g_1\cdot \mathbf a}{\|\mathbf g_1\|^2} \mathbf g_1 + \cdots + \frac{\mathbf g_k \cdot \mathbf a}{\|\mathbf g_k\|^2}\mathbf g_k \\ = (\mathbf g_1 \cdot \mathbf a)\mathbf g_1 + \cdots + (\mathbf g_k \cdot \mathbf a)\mathbf g_k. $$The coefficients of this sum are $\mathbf g_1\cdot \mathbf a, \dots, \mathbf g_k \cdot \mathbf a$, which are the entries of the product $\mathbf G^T\mathbf a.$ To get the corresponding projected vector, take the corresponding linear combination of the columns of $\mathbf G$. In other words, we have$$ \operatorname{proj}_{\mathbf G}(\mathbf a) = \mathbf G(\mathbf G^T \mathbf a). $$With that in mind, we can find the coefficients of the projection for each column of the matrix $\mathbf M = \mathbf X - \mathbb E(\mathbf X)$ by computing $\mathbf G^T\mathbf M$ and obtain the vectors corresponding to these columns of coefficients with the product $\mathbf G\mathbf G^T\mathbf M$.
$\endgroup$
