Lecture 4: Review of Linear Algebra

Zheng Tian

Created: 2017-02-28 Tue 10:45

Vectors and Matrices

Vectors

  • A vector is an ordered set of numbers arranged in a column. An n-dimensional column vector \(\mathbf{a}\) is

    \begin{equation*} \mathbf{a} = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix} \end{equation*}

Matrices

  • A matrix is a set of column vectors. An \(n \times k\) matrix \(\mathbf{A}\) is

    \begin{equation*} \mathbf{A} = \begin{bmatrix} a_{11} & a_{12} & \cdots & a_{1k} \\ a_{21} & a_{22} & \cdots & a_{2k} \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nk} \end{bmatrix} \end{equation*}

Types of Matrices

Square and symmetric matrix

A square matrix
the number of rows equal the number of columns, that is, \(n = k\)
A symmetric matrix
the \((i,j)\) element equal to the \((j, i)\) element.

Diagonal matrix

  • A diagonal matrix: a square matrix in which all off-diagonal elements equal zero, that is,

    \begin{equation*} \mathbf{A} = \begin{bmatrix} a_{11} & 0 & \cdots & 0 \\ 0 & a_{22} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & a_{nn} \end{bmatrix} \end{equation*}

Identity matrix

  • An identity matrix: a diagonal matrix in which all diagonal elements are 1. A subscript is sometimes included to indicate its size, e.g. \(\mathbf{I}_4\) indicate a \(4 \times 4\) identity matrix.

    \begin{equation*} \mathbf{I}_4 = \begin{bmatrix} 1 & 0 & 0 & 0 \\ 0 & 1 & 0 & 0 \\ 0 & 0 & 1 & 0 \\ 0 & 0 & 0 & 1 \end{bmatrix} \end{equation*}

Triangular matrix

  • A triangular matrix: have only zeros either above or below the main diagonal. A lower triangular matrix looks like

    \begin{equation*} \mathbf{A} = \begin{bmatrix} a_{11} & 0 & \cdots & 0 \\ a_{21} & a_{22} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ a_{n1} & a_{n2} & \cdots & a_{nn} \end{bmatrix} \end{equation*}

Matrix Operations

Transpose

  • The transpose of a matrix \(\mathbf{A}\), denoted \(\mathbf{A}^{\prime}\), is obtained by creating the matrix whose kth row is the kth column of the original matrix. That is,

    \[ \mathbf{B} = \mathbf{A}^{\prime} \Leftrightarrow b_{ik} = a_{ki} \text{ for all } i \text{ and } k \]

  • For any \(\mathbf{A}\), we have \((\mathbf{A}^{\prime})^{\prime} = \mathbf{A}\)
  • If \(\mathbf{A}\) is symmetric, then \(\mathbf{A} = \mathbf{A}^{\prime}\).

Addition

  • For two matrices \(\mathbf{A}\) and \(\mathbf{B}\) with the same dimensions, that is both are \(n \times k\).

    \[\mathbf{A} + \mathbf{B} = [a_{ij} + b_{ij}] \text{ for all } i \text{ and } j\]

Multiplication

Vector multiplication

  • The inner product of two \(n \times 1\) column vector \(\mathbf{a}\) and \(\mathbf{b}\) is \[ \mathbf{a}^{\prime} \mathbf{b} = \sum^n_{i=1} a_i b_i \]

    Since both \(\mathbf{a}\) and \(\mathbf{b}\) are \(n \times 1\) vectors, it must hold that \(\mathbf{a}^{\prime} \mathbf{b} = \mathbf{b}^{\prime} \mathbf{a}\).

Matrix multiplication

  • Suppose that \(\mathbf{A}\) is an \(n \times m\) matrix and \(\mathbf{B}\) is an \(m \times k\) matrix, then the product \(\mathbf{C} = \mathbf{AB}\) is an \(n \times k\) matrix, where the \((i,j)\) element of \(\mathbf{C}\) is \(c_{ij} = \sum_{l=1}^m a_{il} b_{lj}\).

    In other words, if we write \(\mathbf{A}\) and \(\mathbf{B}\) with vectors, that is,

    \begin{equation*} \mathbf{A} = \begin{bmatrix} \mathbf{a}_1^{\prime} \\ \mathbf{a}_2^{\prime} \\ \vdots \\ \mathbf{a}_{n}^{\prime} \end{bmatrix} \text{ and } \mathbf{B} = \begin{bmatrix} \mathbf{b}_1 & \mathbf{b}_2 & \cdots & \mathbf{b}_k \end{bmatrix} \end{equation*}

    where \(\mathbf{a}_i = [a_{i1}, a_{i2}, \cdots, a_{im}]^{\prime}\) is the ith row of \(\mathbf{A}\) for \(i = 1, 2, \ldots, n\), and \(\mathbf{b}_j = [b_{1j}, b_{2j}, \ldots, b_{mj}]^{\prime}\) is the jth column of \(\mathbf{B}\) for \(j = 1, 2, \ldots, k\).

Matrix multiplication (cont'd)

\begin{equation*} \mathbf{AB} = \begin{bmatrix} \mathbf{a}_1^{\prime} \mathbf{b}_1 & \cdots & \mathbf{a}_1^{\prime} \mathbf{b}_k \\ \mathbf{a}_2^{\prime} \mathbf{b}_1 & \cdots & \mathbf{a}_2^{\prime} \mathbf{b}_k \\ \vdots & \ddots & \vdots \\ \mathbf{a}_n^{\prime} \mathbf{b}_1 & \cdots & \mathbf{a}_n^{\prime} \mathbf{b}_k \end{bmatrix} \end{equation*}

Properties of matrix addition and multiplication

  • Commutative law: \(\mathbf{A} + \mathbf{B} = \mathbf{B} + \mathbf{A}\). No commutative law for matrix multiplication.
  • Associative law: \((\mathbf{A} + \mathbf{B}) + \mathbf{C} = \mathbf{A} + (\mathbf{B} + \mathbf{C})\) and \((\mathbf{AB}) \mathbf{C} = \mathbf{A} (\mathbf{BC})\)
  • Distributive law: \(\mathbf{A} (\mathbf{B} + \mathbf{C}) = \mathbf{AB} + \mathbf{AC}\)
  • Transpose of a sum and a product: \((\mathbf{A} + \mathbf{B})^{\prime} = \mathbf{A}^{\prime} + \mathbf{B}^{\prime}\) and \((\mathbf{A} \mathbf{B})^{\prime} = \mathbf{B}^{\prime} \mathbf{A}^{\prime}\).

Matrix Inverse

Definition

  • Let \(\mathbf{A}\) be an \(n \times n\) square matrix. \(\mathbf{A}\) is said to be invertible or nonsingular if such a matrix \(\mathbf{A}^{-1}\) exists that \(\mathbf{A}^{-1} \mathbf{A} = \mathbf{I}_n\). \(\mathbf{A}^{-1}\) is the inverse of \(\mathbf{A}\).

Calculation

  • Let \(a^{ik}\) be the ikth element of \(\mathbf{A}^{-1}\). The general formula for computing an inverse matrix is

    \[ a^{ik} = \frac{|\mathbf{C}_{ki}|}{|\mathbf{A}|} \]

    where \(| \mathbf{A} |\) is the determinant of \(\mathbf{A}\), \(| \mathbf{C}_{ki} |\) is the kith cofactor of \(\mathbf{A}\), that is, the determinant of the matrix \(\mathbf{A}_{ki}\) obtained from \(\mathbf{A}\) by deleting row \(k\) and column \(i\), pre-multiplied by \((-1)^{(k + i)}\).

Example 1: The inverser of a \(2 \times 2\) matrix

\begin{equation*} \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \end{bmatrix}^{-1} =\frac{1}{a_{11}a_{22} - a_{12}a_{21}} \begin{bmatrix} a_{22} & -a_{12} \\ -a_{21} & a_{11} \end{bmatrix} \end{equation*}

Example 2: The inverse of a diagonal matrix

\begin{equation*} \begin{bmatrix} a_{11} & 0 & \cdots & 0 \\ 0 & a_{22} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & a_{nn} \end{bmatrix}^{-1} = \begin{bmatrix} 1/a_{11} & 0 & \cdots & 0 \\ 0 & 1/a_{22} & \cdots & 0 \\ \vdots & \vdots & \ddots & \vdots \\ 0 & 0 & \cdots & 1/a_{nn} \end{bmatrix} \end{equation*}

Linear Independence

Linear independence

  • The set of \(k\) \(n \times 1\) vectors, \(\mathbf{a}_1, \mathbf{a}_2, \ldots, \mathbf{a}_k\) are linearly independent if there do not exist nonzero scalars \(c_1, c_2, \ldots, c_k\) such that \(c_1 \mathbf{a}_1 + c_2 \mathbf{a}_2 + \cdots + c_k \mathbf{a}_k = \mathbf{0}_{n \times 1}\).

The rank of a matrix

  • The rank of the \(n \times k\) matrix \(\mathbf{A}\) is the number of linearly independent column vectors of \(\mathbf{A}\), denoted as \(\mathrm{rank}(\mathbf{A})\).
  • If \(\mathrm{rank}(\mathbf{A}) = k\), then \(\mathbf{A}\) is said to have full column rank. Then, there do not exist a nonzero \(k \times 1\) vector \(\mathbf{c}\) such that \(\mathbf{A} \mathbf{c} = \mathbf{0}\).
  • If \(\mathbf{A}\) is an \(n \times n\) square matrix and \(\mathrm{rank}(\mathbf{A}) = n\), then \(\mathbf{A}\) is nonsingular.
  • If \(\mathbf{A}\) has full column rank, then \(\mathbf{A}^{\prime} \mathbf{A}\) is nonsingular.

Positive definite matrices

  • Let \(\mathbf{V}\) be an \(n \times n\) square matrix. Then \(\mathbf{V}\) is positive definite if \(\mathbf{c}^{\prime} \mathbf{V} \mathbf{c} > 0\) for all nonzero \(n \times 1\) vector \(\mathbf{c}\).
  • \(\mathbf{V}\) is positive semidefinite if \(\mathbf{c}^{\prime} \mathbf{V} \mathbf{c} \geq 0\) for all nonzero \(n \times 1\) vector \(\mathbf{c}\).
  • If \(\mathbf{V}\) is positive definite, then it is nonsingular.

Calculus with Vectors and Matrices

  • We need to use the following results of matrix calculus in the future lectures.

    \begin{align*} & \frac{\partial \mathbf{a}^{\prime} \mathbf{x}}{\partial \mathbf{x}} = \mathbf{a},\; \frac{\partial \mathbf{x}^{\prime} \mathbf{a}}{\partial \mathbf{x}} = \mathbf{a},\; \text{ and } \\ & \frac{\partial \mathbf{x}^{\prime} \mathbf{A} \mathbf{x}}{\partial \mathbf{x}} = (\mathbf{A} + \mathbf{A}^{\prime}) \mathbf{x} \end{align*}

    When \(\mathbf{A}\) is symmetric, then \((\partial \mathbf{x}^{\prime} \mathbf{A} \mathbf{x}) / (\partial \mathbf{x}) = 2\mathbf{A} \mathbf{x}\)