Matrix Products

  • Matrix-vector product
  • Matrix-matrix product
  • Block matrix product
  • Trace and scalar product

Matrix-vector product

Definition

We define the matrix-vector product between a m \times n matrix and a n-vector x, and denote by Ax, the m-vector with i-th component

    \[(Ax)_{i}=\sum_{j=1}^{n}A_{ij}x_{j}, i=1,\dots,m.\]

alt text The picture on the left shows a symbolic example with n=2 and m=3. We have y=Ax, that is:

    \begin{align*}y_{1}&=A_{11}x_{1}+A_{12}x_{2},\\y_{2}&=A_{21}x_{1}+A_{22}x_{2},\\y_{3}&=A_{31}x_{1}+A_{32}x_{2}.\end{align*}

Interpretation as linear combinations of columns

If the columns of A are given by the vectors a_{i}, i=1,\dots,n so that A=(a_{1},\dots, a_{n}), then Ax can be interpreted as a linear combination of these columns, with weights given by the vector x:

    \[Ax=\sum_{i=1}^{n}x_{i}a_{i}.\]

alt text In the above symbolic example, we have y=Ax, that is:

    \[ y = x_{1} \begin{pmatrix} A_{11} \\ A_{21} \\ A_{31} \end{pmatrix} + x_{2} \begin{pmatrix} A_{12} \\ A_{22} \\ A_{32} \end{pmatrix}.\]

Example:

Interpretation as scalar products with rows

Alternatively, if the rows of A are the row vectors a_{i}^{T}, i=1,\dots,m:

    \[A = \begin{pmatrix} a_{1}^{T} \\ \vdots \\ a_{m}^{T} \end{pmatrix},\]

then Ax is the vector with elements a_{i}^{T}x, i=1,\dots,m:

    \[Ax = \begin{pmatrix} a_{1}^{T}x \\ \vdots \\ a_{m}^{T}x \end{pmatrix}.\]

alt text In the above symbolic example, we have y=Ax, that is:

    \[\begin{aligned} y_1 & =\begin{pmatrix} A_{11} \\ A_{12} \end{pmatrix}^{T} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = A_{11}x_{1} + A_{12}x_{2}, \\ y_2 & =\begin{pmatrix} A_{21} \\ A_{22} \end{pmatrix}^{T} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = A_{21}x_{1} + A_{22}x_{2}, \\ y_3 & =\begin{pmatrix} A_{31} \\ A_{32} \end{pmatrix}^{T} \begin{pmatrix} x_1 \\ x_2 \end{pmatrix} = A_{31}x_{1} + A_{32}x_{2} \\ \end{aligned}\]

Example: Absorption spectrometry: using measurements at different light frequencies.

Left product

If z \in \mathbb{R}^{m}, then the notation z^{T}A is the row vector of size n equal to the transpose of the column vector A^{T}z \in \mathbb{R}^{n}. That is:

    \[(z^{T}A)_{j} = \sum_{i=1}^{m} A_{ij}z_{i}, j=1,\dots,n.\]

Example: Return to the network example, involving a m \times n incidence matrix. We note that, by construction, the columns of A sum to zero, which can be compactly written as 1^{T}A=0, or A^{T}1=0.

Matlab syntax

The product operator in Matlab is *. If the sizes are not consistent, Matlab will produce an error.

Matlab syntax
>> A = [1 2; 3 4; 5 6]; % 3x2 matrix
>> x = [-1; 1]; % 2x1 vector
>> y = A*x; % result is a 3x1 vector
>> z = [-1; 0; 1]; % 3x1 vector
>> y = z'*A; % result is a 1x2 (i.e., row) vector

Matrix-matrix product

Definition

We can extend the matrix-vector product to the matrix-matrix product, as follows. If A \in \mathbb{R}^{m \times n} and B \in \mathbb{R}^{n \times p}, the notation AB denotes the m \times p matrix with i, j element given by

    \[(AB)_{ij} = \sum_{k=1}^{n}A_{ik}B_{kj}.\]

Transposing a product changes the order, so that (AB)^{T}=B^{T}A^{T}.

 

Column-wise interpretation

If the columns of B are given by the vectors b_{i}, i=1, \dots, n so that B=[b_{1}, \dots, b_{n}] then AB can be written as

    \[AB = A \begin{pmatrix}b_{1} & \dots & b_{n} \end{pmatrix} =. \begin{pmatrix}Ab_{1} & \dots & Ab_{n} \end{pmatrix}.\]

In other words, AB results from transforming each column b_{i} of B into Ab_{i}.

Row-wise interpretation

The matrix-matrix product can also be interpreted as an operation on the rows of A. Indeed, if A is given by its rows a_{i}^{T}, i = 1, \dots, m then AB is the matrix obtained by transforming each one of these rows via B, into a_{i}^{T}B, i = 1, \dots, m:

    \[AB = \begin{pmatrix}a_{1}^{T} \\ \vdots \\ a_{n}^{T}\end{pmatrix}B= \begin{pmatrix}a_{1}^{T}B \\ \vdots \\ a_{n}^{T}B\end{pmatrix}.\]

(Note that a_{i}^{T}B‘s are indeed row vectors, according to our matrix-vector rules.)

Block Matrix Products

Matrix algebra generalizes to blocks, provided block sizes are consistent. To illustrate this, consider the matrix-vector product between a m \times n matrix A and a n-vector x, where A, x are partitioned in blocks, as follows:

    \[A = \begin{pmatrix} A_{1} & A_{2} \end{pmatrix}, x = \begin{pmatrix} x_{1} & x_{2} \end{pmatrix},\]

where A_{i} is m \times n_{i}, x_{i} \in \mathbb{R}^{n}, i=1, 2, n_{1}+n_{2}=n. Then Ax = A_{1}x_{1} + A_{2}x_{2}.

Symbolically, it’s as if we would form the ‘‘scalar’’ product between the ‘‘row vector (A_{1}, A_{2}) and the column vector (x_{1}, x_{2})!

Likewise, if a n \times p matrix B is partitioned into two blocks B_{i}, each of size n_{i}, i=1,2, with n_{1}+n_{2}=n, then

    \[AB = \begin{pmatrix} A_{1} & A_{2} \end{pmatrix} \begin{pmatrix} B_{1} \\ B_{2} \end{pmatrix} = A_{1}B_{1} + A_{2}B_{2}.\]

Again, symbolically we apply the same rules as for the scalar product — except that now the result is a matrix.

Example: Gram matrix.

Finally, we can consider so-called outer products. Consider the case for example when A is a m \times n matrix partitioned row-wise into two blocks A_{1}, A_{2}, and B is a n \times p matrix that is partitioned column-wise into two blocks B_{1}, B_{2}:

    \[A = \begin{pmatrix} A_{1} & A_{2} \end{pmatrix}, B = \begin{pmatrix} B_{1} \\ B_{2} \end{pmatrix} .\]

Then the product C = AB can be expressed in terms of the blocks, as follows:

    \[C = AB = \begin{pmatrix} A_{1} & A_{2} \end{pmatrix}. \begin{pmatrix} B_{1} \\ B_{2} \end{pmatrix} = \begin{pmatrix} A_{1}B_{1} & A_{1}B_{2} \\ A_{2}B_{1} & A_{2}B_{2} \end{pmatrix}.\]

Trace, scalar product

Trace

The trace of a square n \times n matrix A, denoted by TrA, is the sum of its diagonal elements:  TrA = \sum_{i=1}^{n}A_{ii}.

Some important properties:

  • Trace of transpose: The trace of a square matrix is equal to that of its transpose.
  • Commutativity under trace: for any two matrices A \in \mathbb{R}^{m \times n} and B \in \mathbb{R}^{n \times m}, we have

    \[Tr(AB) = Tr(BA).\]

Matlab syntax
>> A = [1 2 3; 4 5 6; 7 8 9]; % 3x3 matrix
>> tr = trace(A); % trace of A

Scalar product between matrices

We can define the scalar product between two m \times n matrices A, B via

    \[\langle A,B \rangle = Tr(A^{T}B) = \sum_{i=1}^{m}\sum_{j=1}^{m}A_{ij}B_{ij}.\]

The above definition is symmetric: we have

    \[\langle A,B \rangle = Tr(A^{T}B) = Tr(A^{T}B)^{T} = Tr(B^{T}A) = \langle B,A \rangle.\]

Our notation is consistent with the definition of the scalar product between two vectors, where we simply view a vector in \mathbb{R}^{n} as a matrix in \mathbb{R}^{n \times 1}. We can interpret the matrix scalar product as the vector scalar product between two long vectors of length mn each, obtained by stacking all the columns of A, B on top of each other.

Matlab syntax
>> A = [1 2; 3 4; 5 6]; % 3x2 matrix
>> B = randn(3,2); % random 3x2 matrix
>> scal_prod = trace(A'*B); % scalar product between A and B
>> scal_prod = A(:)'*B(:); % this is the same as the scalar product between the
                            % vectorized forms of A, B

License

Hyper-Textbook: Optimization Models and Applications Copyright © by L. El Ghaoui. All Rights Reserved.

Share This Book