Lecture 4: Review of Linear Algebra

A vector is an ordered set of numbers arranged in a column¹. An n-dimensional column vector \(\mathbf{a}\) is

A matrix is a set of column vectors. An \(n \times k\) matrix \(\mathbf{A}\) is

• A square matrix: the number of rows equal the number of columns, that is, \(n = k\)
• A symmetric matrix: the \((i,j)\) element equal to the \((j, i)\) element.

• A diagonal matrix: a square matrix in which all off-diagonal elements equal zero, that is,

  \begin{equation*} \mathbf{A} = \begin{bmatrix} a_{11} & 0 & \cdots & 0 \\ 0 & a_{22} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & a_{nn} \end{bmatrix} \end{equation*}

• An identity matrix: a diagonal matrix in which all diagonal elements are 1. A subscript is sometimes included to indicate its size, e.g. \(\mathbf{I}_4\) indicate a \(4 \times 4\) identity matrix.

  \begin{equation*} \mathbf{I}_4 = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix} \end{equation*}

• A triangular matrix: have only zeros either above or below the main diagonal. A lower triangular matrix looks like

  \begin{equation*} \mathbf{A} = \begin{bmatrix} a_{11} & 0 & \cdots & 0 \\ a_{21} & a_{22} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nn} \end{bmatrix} \end{equation*}

The transpose of a matrix \(\mathbf{A}\), denoted \(\mathbf{A}^{\prime}\), is obtained by creating the matrix whose kth row is the kth column of the original matrix. That is, \[ \mathbf{B} = \mathbf{A}^{\prime} \Leftrightarrow b_{ik} = a_{ki} \text{ for all } i \text{ and } k \]

• For any \(\mathbf{A}\), we have \((\mathbf{A}^{\prime})^{\prime} = \mathbf{A}\)
• If \(\mathbf{A}\) is symmetric, then \(\mathbf{A} = \mathbf{A}^{\prime}\).

For two matrices \(\mathbf{A}\) and \(\mathbf{B}\) with the same dimensions, that is both are \(n \times k\).

\[\mathbf{A} + \mathbf{B} = [a_{ij} + b_{ij}] \text{ for all } i \text{ and } j\]

• Vector multiplication. The inner product of two \(n \times 1\) column vector \(\mathbf{a}\) and \(\mathbf{b}\) is \[ \mathbf{a}^{\prime} \mathbf{b} = \sum^n_{i=1} a_i b_i \]

  Since both \(\mathbf{a}\) and \(\mathbf{b}\) are \(n \times 1\) vectors, it must hold that \(\mathbf{a}^{\prime} \mathbf{b} = \mathbf{b}^{\prime} \mathbf{a}\).

• Matrix multiplication. The matrices \(\mathbf{A}\) and \(\mathbf{B}\) can be multiplied if they are conformable, that is, if the number of columns of \(\mathbf{A}\) equals the number of rows of \(\mathbf{B}\).

  Suppose that \(\mathbf{A}\) is an \(n \times m\) matrix and \(\mathbf{B}\) is an \(m \times k\) matrix, then the product \(\mathbf{C} = \mathbf{AB}\) is an \(n \times k\) matrix, where the \((i,j)\) element of \(\mathbf{C}\) is \(c_{ij} = \sum_{l=1}^m a_{il} b_{lj}\).

  In other words, if we write \(\mathbf{A}\) and \(\mathbf{B}\) with vectors, that is,

  \begin{equation*} \mathbf{A} = \begin{bmatrix} \mathbf{a}_1^{\prime} \\ \mathbf{a}_2^{\prime} \\ \vdots \\ \mathbf{a}_{n}^{\prime} \end{bmatrix} \text{ and } \mathbf{B} = \begin{bmatrix} \mathbf{b}_1 & \mathbf{b}_2 & \cdots & \mathbf{b}_k \end{bmatrix} \end{equation*}

  where \(\mathbf{a}_i = [a_{i1}, a_{i2}, \cdots, a_{im}]^{\prime}\) is the i^th row of \(\mathbf{A}\) for \(i = 1, 2, \ldots, n\), and \(\mathbf{b}_j = [b_{1j}, b_{2j}, \ldots, b_{mj}]^{\prime}\) is the j^th column of \(\mathbf{B}\) for \(j = 1, 2, \ldots, k\). Then,

  \begin{equation*} \mathbf{AB} = \begin{bmatrix} \mathbf{a}_1^{\prime} \mathbf{b}_1 & \cdots & \mathbf{a}_1^{\prime} \mathbf{b}_k \\ \mathbf{a}_2^{\prime} \mathbf{b}_1 & \cdots & \mathbf{a}_2^{\prime} \mathbf{b}_k \\ \vdots & \ddots & \vdots \\ \mathbf{a}_n^{\prime} \mathbf{b}_1 & \cdots & \mathbf{a}_n^{\prime} \mathbf{b}_k \end{bmatrix} \end{equation*}

• Commutative law: \(\mathbf{A} + \mathbf{B} = \mathbf{B} + \mathbf{A}\). No commutative law for matrix multiplication.
• Associative law: \((\mathbf{A} + \mathbf{B}) + \mathbf{C} = \mathbf{A} + (\mathbf{B} + \mathbf{C})\) and \((\mathbf{AB}) \mathbf{C} = \mathbf{A} (\mathbf{BC})\)
• Distributive law: \(\mathbf{A} (\mathbf{B} + \mathbf{C}) = \mathbf{AB} + \mathbf{AC}\)
• Transpose of a sum and a product: \((\mathbf{A} + \mathbf{B})^{\prime} = \mathbf{A}^{\prime} + \mathbf{B}^{\prime}\) and \((\mathbf{A} \mathbf{B})^{\prime} = \mathbf{B}^{\prime} \mathbf{A}^{\prime}\).

Let \(\mathbf{A}\) be an \(n \times n\) square matrix. \(\mathbf{A}\) is said to be invertible or nonsingular if such a matrix \(\mathbf{A}^{-1}\) exists that \(\mathbf{A}^{-1} \mathbf{A} = \mathbf{I}_n\). \(\mathbf{A}^{-1}\) is the inverse of \(\mathbf{A}\).

Let \(a^{ik}\) be the ik^th element of \(\mathbf{A}^{-1}\). The general formula for computing an inverse matrix is \[ a^{ik} = \frac{|\mathbf{C}_{ki}|}{|\mathbf{A}|} \] where \(| \mathbf{A} |\) is the determinant of \(\mathbf{A}\), \(| \mathbf{C}_{ki} |\) is the ki^th cofactor of \(\mathbf{A}\), that is, the determinant of the matrix \(\mathbf{A}_{ki}\) obtained from \(\mathbf{A}\) by deleting row \(k\) and column \(i\), pre-multiplied by \((-1)^{(k + i)}\).

• Example 1. Calculate the inverse of a \(2 \times 2\) matrix.

  \begin{equation*} \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix}^{-1} =\frac{1}{a_{11}a_{22} - a_{12}a_{21}} \begin{bmatrix} a_{22} & -a_{12} \\ -a_{21} & a_{11} \end{bmatrix} \end{equation*}

• Example 2. The inverse of a diagonal matrix.

  \begin{equation*} \begin{bmatrix} a_{11} & 0 & \cdots & 0 \\ 0 & a_{22} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & a_{nn} \end{bmatrix}^{-1} = \begin{bmatrix} 1/a_{11} & 0 & \cdots & 0 \\ 0 & 1/a_{22} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 1/a_{nn} \end{bmatrix} \end{equation*}

The set of \(k\) \(n \times 1\) vectors, \(\mathbf{a}_1, \mathbf{a}_2, \ldots, \mathbf{a}_k\) are linearly independent if there do not exist nonzero scalars \(c_1, c_2, \ldots, c_k\) such that \(c_1 \mathbf{a}_1 + c_2 \mathbf{a}_2 + \cdots + c_k \mathbf{a}_k = \mathbf{0}_{n \times 1}\).

The rank of the \(n \times k\) matrix \(\mathbf{A}\) is the number of linearly independent column vectors of \(\mathbf{A}\), denoted as \(\mathrm{rank}(\mathbf{A})\).

• If \(\mathrm{rank}(\mathbf{A}) = k\), then \(\mathbf{A}\) is said to have full column rank. Then, there do not exist a nonzero \(k \times 1\) vector \(\mathbf{c}\) such that \(\mathbf{A} \mathbf{c} = \mathbf{0}\).
• If \(\mathbf{A}\) is an \(n \times n\) square matrix and \(\mathrm{rank}(\mathbf{A}) = n\), then \(\mathbf{A}\) is nonsingular.
• If \(\mathbf{A}\) has full column rank, then \(\mathbf{A}^{\prime} \mathbf{A}\) is nonsingular.

Let \(\mathbf{V}\) be an \(n \times n\) square matrix. Then \(\mathbf{V}\) is positive definite if \(\mathbf{c}^{\prime} \mathbf{V} \mathbf{c} > 0\) for all nonzero \(n \times 1\) vector \(\mathbf{c}\). And \(\mathbf{V}\) is positive semidefinite if \(\mathbf{c}^{\prime} \mathbf{V} \mathbf{c} \geq 0\) for all nonzero \(n \times 1\) vector \(\mathbf{c}\).

• If \(\mathbf{V}\) is positive definite, then it is nonsingular.