40
40.1. SVD of simple matrices
1. Consider the matrix
[latex]\begin{align*} A = \begin{pmatrix} 1 & 0 & 0 \\ 0 & -2 & 0 \\ 0 & 0 & 0 \\ 0 & 0 & 0 \end{pmatrix} \end{align*}[/latex]
a. Find the range, nullspace, and rank of [latex]A[/latex].
b. Find an SVD of [latex]A[/latex].
c. Determine the set of solutions to the linear equation [latex]Ax=y[/latex], with
[latex]\begin{center} \[ y=\left(\begin{array}{l} 2 \\ 3 \\ 0 \\ 0 \end{array}\right), \quad y=\left(\begin{array}{l} 2 \\ 0 \\ 0 \\ 3 \end{array}\right). \] \end{center}[/latex]
2. Consider the [latex]2 \times 2[/latex] matrix
[latex]\begin{center} \[ A=\frac{1}{\sqrt{10}}\left(\begin{array}{l} 2 \\ 1 \end{array}\right)\left(\begin{array}{ll} 1 & -1 \end{array}\right)+\frac{2}{\sqrt{10}}\left(\begin{array}{c} -1 \\ 2 \end{array}\right)\left(\begin{array}{ll} 1 & 1 \end{array}\right). \] \end{center}[/latex]
a. What is an SVD of [latex]A[/latex]? Express it as [latex]A = USV^T[/latex], with [latex]S[/latex] the diagonal matrix of singular values ordered in decreasing fashion. Make sure to check all the properties required for [latex]U,S,V[/latex].
b. Find the semi-axis lengths and principal axes (minimum and maximum distance and associated directions from [latex]\mathcal{E}[/latex] to the center) of the ellipsoid
[latex]\begin{center} \[ \mathcal{E}(A) = \{Ax: x \in \mathbb{R}^2, ||x||_2 \leq 1\}. \] \end{center}[/latex]
Hint: Use the SVD of [latex]A[/latex] to show that every element of [latex]\mathcal{E}(A)[/latex] is of the form [latex]y = U \overline{y}[/latex] for some element [latex]\overline{y}[/latex] in [latex]\mathcal{E}(S)[/latex]. That is, [latex]\mathcal{E}(A) = \{U\overline{y}: \overline{y} \in \mathcal{E}(S)\}[/latex]. (In other words, the matrix [latex]U[/latex] maps [latex]\mathcal{E}(S)[/latex] into the set [latex]\mathcal{E}(A)[/latex].) Then analyze the geometry of the simpler set [latex]\mathcal{E}(S)[/latex].
c. What is the set [latex]\mathcal{E}(A)[/latex] when we append a zero vector after the last column of [latex]A[/latex], that is [latex]A[/latex] is replaced with [latex]\tilde{A} = (A, 0) \in \mathbb{R}^{2 \times 3}[/latex]?
d. The same question when we append a row after the last row of [latex]A[/latex], that is, [latex]A[/latex] is replaced with [latex]\tilde{A} = [A^T, 0]^T \in \mathbb{R}^{3 \times 2}[/latex]. Interpret geometrically your result.
40.2. Rank and SVD
The image on the left shows a [latex]256 \times 256[/latex] matrix [latex]A[/latex] of pixel values. The lines indicate [latex]+1[/latex] values; at each intersection of lines, the corresponding matrix element is [latex]+2[/latex]. All the other elements are zero.
1. Show that for some permutation matrices [latex]P, Q[/latex], the permuted matrix [latex]B: = PAQ[/latex] has the symmetric form [latex]B = pq^T + qp^T[/latex], for two vectors [latex]p, q[/latex]. Determine [latex]P, Q, B[/latex] and [latex]p, q[/latex].
2. What is the rank of [latex]A[/latex]? Hint: find the nullspace of [latex]B[/latex].
3. Find an SVD of [latex]A[/latex]. Hint: Find an eigenvalue decomposition of [latex]S[/latex], using the results of an exercise on eigenvalue here.
40.3. Procrustes problem
The Orthogonal Procrustes problem is a problem of the form
[latex]\begin{align*} \min\limits_{X} ||AX-B||_F: X^TX = I_p, \end{align*}[/latex]
where [latex]||\cdot||_F[/latex] denotes the Frobenius norm, and the matrices [latex]A \in \mathbb{R}^{m \times n}[/latex], [latex]B \in \mathbb{R}^{m \times p}[/latex] are given. Here, the matrix variable [latex]X \in \mathbb{R}^{n \times p}[/latex] is constrained to have orthonormal columns. When [latex]n=m=p[/latex], the problem can be interpreted geometrically as seeking a transformation of points (contained in [latex]A[/latex]) to other points (contained in [latex]B[/latex]) that involves only rotation.
1. Show that the solution to the Procrustes problem above can be found via the SVD of the matrix [latex]A^TB[/latex].
2. Derive a formula for the answer to the constrained least-squares problem
[latex]\begin{align*} \min\limits_{x} ||Ax-b||_2: ||x||_2=1, \end{align*}[/latex]
with [latex]A \in \mathbb{R}^{m \times n}[/latex], [latex]b \in \mathbb{R}^m[/latex] given.
40.4. SVD and projections
1. We consider a set of [latex]m[/latex] data points [latex]x_i \in \mathbb{R}^n[/latex], [latex]i = 1, \cdots, m[/latex]. We seek to find a line in [latex]\mathbb{R}^n[/latex] such that the sum of the squares of the distances from the points to the line is minimized. To simplify, we assume that the line goes through the origin.
a. Consider a line that goes through the origin [latex]\mathcal{L} = \{tu: t\in\mathbb{R}\}[/latex], where [latex]u \in \mathbb{R}^n[/latex] is given. (You can assume without loss of generality that [latex]||u||_2=1[/latex].) Find an expression for the projection of a given point [latex]x[/latex] on [latex]\mathcal{L}[/latex].
b. Now consider the [latex]m[/latex] points and find an expression for the sum of the squares of the distances from the points to the line [latex]\mathcal{L}[/latex].
c. Explain how you would find the line via the SVD of the [latex]n\times m[/latex] matrix [latex]X = [x_1, \cdots, x_m][/latex].
d. How would you address the problem without the restriction that the line has to pass through the origin?
2. Solve the same problems as previously by replacing the line with a hyperplane.
40.5. SVD and least-squares
1. Consider the matrix [latex]A[/latex] formed as [latex]A = (c_1, c_2, c_3)[/latex], with
[latex]\begin{align*} c_1 &= (1, 2, 8), \quad c_2 = (3, 6, 9), \quad c_3 = c_1 - 4c_2 + \epsilon_1 v, \quad \epsilon_1 = .0000001, \end{align*}[/latex]
with [latex]v[/latex] is a vector chosen randomly in [latex][-0.5, 0.5]^3[/latex] (this represents a 3-dimensional cube with each dimension ranging from [latex]-0.5[/latex] to [latex]0.5[/latex]). In addition, we define
[latex]\begin{align*} y = 2c_1 - 7c_2 + \epsilon_2 w, \quad \epsilon_2 = .0001, \end{align*}[/latex]
with again [latex]w[/latex] chosen randomly in [latex][-0.5, 0.5]^3[/latex]. We consider the associated least-squares problem
[latex]\begin{align*} \min\limits_x ||Ax-y||_2. \end{align*}[/latex]
a. What is the rank of [latex]A[/latex]?
b. Apply the least-squares formula [latex]x^* = (A^TA)^{-1}A^Ty[/latex]. What is the norm of the residual vector, [latex]r = Ax-y[/latex]?
c. Express the least-squares solution in terms of the SVD of [latex]A[/latex]. That is, form the pseudo-inverse of [latex]A[/latex] and apply the formula [latex]x^* = A^{\dagger} y[/latex]. What is now the norm of the residual?
d. Interpret your results.
2. Consider a least-squares problem
[latex]\begin{align*} \min\limits_x ||Ax-y||_2 \end{align*}[/latex]
where the data matrix [latex]A \in \mathbb{R}^{m \times n}[/latex] has rank one.
a. Is the solution unique?
b. Show how to reduce the problem to one involving one scalar variable.
c. Express the set of solutions in closed form.
