Properties of Positive Definite Matrices

Say we have a

n\times n

matrix

\(A\)

then

\(A\)

is Positive Definite Matrix if any of the below condition is satisfies,

$$1.$$ All the eigenvalues are positive.

$\quad\lambda_1\gt 0,\lambda_2\gt 0,\cdots,\lambda_n\gt 0$

$$2.$$ All leading determinants are positive.

For Example:
Say we have a $3\times 3$ matrix $$A$$ $A=\begin{bmatrix} a_{11} & a_{12} & a_{13}\\ a_{21} & a_{22} & a_{23}\\ a_{31} & a_{32} & a_{33}\\ \end{bmatrix}$ $$A$$ is a Positive Definite Matrix if,
$\text{det}\left( \begin{bmatrix} a_{11} \end{bmatrix} \right)\gt 0;\quad$ $\text{det}\left( \begin{bmatrix} a_{11} & a_{12} \\ a_{21} & a_{22} \\ \end{bmatrix} \right)\gt 0;\quad$ $\text{det}\left( \begin{bmatrix} a_{11} & a_{12} & a_{13}\\ a_{21} & a_{22} & a_{23}\\ a_{31} & a_{32} & a_{33}\\ \end{bmatrix} \right)\gt 0$

$$3.$$ All pivots are positive.

$$4.$$ A $n\times n$ matrix say $$A$$ is Positive Definite Matrix, if

\begin{matrix} \vec{x}^TA\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0} \end{matrix}

More interesting properties of positive definite matrices

If a

n\times n

matrix

\(A\)

is a positive definite matrix then

A^{-1}

is also positive definite matrix

Proof:

Here we can use property number $$1$$ "A matrix is positive definite matrix if all the eigenvalues are positive."
Say that the eigenvalues of $$A$$ are $\lambda_1,\lambda_2,\cdots,\lambda_n$
Then the eigenvalues of $A^{-1}$ are $1/\lambda_1,1/\lambda_2,\cdots,1/\lambda_n$
$$A$$ is a positive definite matrix, it's eigenvalues are positive, so
$\lambda_1\gt 0,\lambda_2\gt 0,\cdots,\lambda_n\gt 0$
$\Rightarrow \quad1/\lambda_1\gt 0,\quad1/\lambda_2\gt 0,\cdots,1/\lambda_n\gt 0$
So by property $$1$$ we can say that " $A^{-1}$ is a Positive Definite Matrix"

Say we have two

n\times n

positive definite matrix

\(A\)

and

\(B\)

then

\(A+B\)

is also positive definite matrix

Proof:

Here we can use property number $$5$$ "A matrix (say A) is positive definite matrix if $\vec{x}^TA\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$ "
We know that $$A$$ is positive definite matrix so $\vec{x}^TA\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$
We know that $$B$$ is positive definite matrix so $\vec{x}^TB\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$
For $$A+B$$ to be positive definite matrix we need $\vec{x}^T(A+B)\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$
$\vec{x}^T(A+B)\vec{x}$ = $\underbrace{\vec{x}^TA\vec{x}}_{(\gt 0;\forall x\in\mathbb{R}^n \char`\\ \vec{0})} + \underbrace{\vec{x}^TB\vec{x}}_{(\gt 0;\forall x\in\mathbb{R}^n \char`\\ \vec{0})}$
$\Rightarrow \vec{x}^T(A+B)\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$
So by property $$5$$ we can say that " $$A + B$$ is a Positive Definite Matrix"

Say we have a

m\times n

matrix

\(A\)

and

\text{Rank}(A)=n

then

\(A^TA\)

is a positive definite matrix

Proof:

For $$A^TA$$ to be a positive definite matrix we need, $\vec{x}^T(A^TA)\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$
$\vec{x}^T(A^TA)\vec{x}=(\vec{x}^TA^T)(A\vec{x})$
$\Rightarrow \vec{x}^T(A^TA)\vec{x}=(A\vec{x})^T(A\vec{x})$
Here $A\vec{x}$ is a vector in $\mathbb{R}^n$ , so $(A\vec{x})^T(A\vec{x})=\|A\vec{x}\|^2$
$\|A\vec{x}\|^2$ is the sum of squares so it can't be negative $\|A\vec{x}\|^2\geq 0$ and $\|A\vec{x}\|^2$ is $$0$$ only for $\vec{x}=\vec{0}$
because $\text{Rank}(A)=n$ so columns are independent, so $A\vec{x}=\vec{0}$ if $\vec{x}=\vec{0}$ so it's Null space have only $\vec{0}$
$\Rightarrow \vec{x}^T(A^TA)\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}$
So we can say that " $$A^TA$$ is a Positive Definite Matrix"

Example $$1$$ :
Say

A=\begin{bmatrix} 2&6\\6&19 \end{bmatrix}

check if

\(A\)

is Positive Definite Matrix or not, using properties described above.
Let's try using the

4^{th}

property and see if

\vec{x}^TA\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}

Say

\vec{x}=\begin{bmatrix} x_1\\x_2 \end{bmatrix}

\vec{x}^TA\vec{x}= \begin{bmatrix} x_1&x_2 \end{bmatrix} \begin{bmatrix} 2&6\\6&19 \end{bmatrix} \begin{bmatrix} x_1\\x_2 \end{bmatrix}

\vec{x}^TA\vec{x}=2x_1^2+12x_1x_2+19x^2

Now we want to know that is

2x_1^2+12x_1x_2+19x^2\gt 0

Now let's use Calculus.

Our function is $$f(x_1,x_2)=2x_1^2+12x_1x_2+19x^2$$
Let's find the critical points,
First we take derivative w.r.t. $$x_1$$ .

$\begin{matrix} \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=& \displaystyle \frac{\partial (2x_1^2+12x_1x_2+19x^2)}{\partial x_1} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=&\displaystyle 4x_1+12x_2 \end{matrix}$ Now equate $\displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=0$
$\begin{matrix} 4x_1+12x_2=0 \end{matrix} \quad\tag{\color{red}{1}}$
Now we take derivative w.r.t. $$x_2$$ .

$\begin{matrix} \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_2}=& \displaystyle \frac{\partial (2x_1^2+12x_1x_2+19x^2)}{\partial x_2} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_2}=&\displaystyle 12x_1 + 38x_2 \end{matrix}$ Now equate $\displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=0$
$\begin{matrix} 12x_1 + 38x_2=0 \end{matrix} \quad\tag{\color{red}{2}}$
$$x_1=0$$ and $$x_2=0$$ satisfies this system of equations $(1)\text{ and }(2)$ .
Now we got a point $$(x_1,x_2)=(0,0)$$ but we don't know that, Is it minimum, maximum or saddle point.
To check that, Is it minimum, maximum or saddle point. We need $2^{nd}$ derivative test
$\frac{\partial^2 f(x_1,x_2)}{\partial x_1^2}$
$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1^2}=& \displaystyle \frac{\partial (4x_1+12x_2)}{\partial x_1} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=&\displaystyle 4 \end{matrix}$
$\frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}$
$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}=& \displaystyle \frac{\partial (12x_1 + 38x_2)}{\partial x_2} \\ \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}=&\displaystyle 38 \end{matrix}$
$\frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}$
$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}=& \displaystyle \frac{\partial (4x_1+12x_2)}{\partial x_2} \\ \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}=&\displaystyle 12 \end{matrix}$
(Recommended reading)
Now we have our Hessian matrix (say $$H$$ )
$Hf(x_1,x_2)=\begin{bmatrix} \displaystyle \frac{\partial f}{\partial x_1^2 } & \displaystyle \frac{\partial f}{\partial x_1x_2}\\ \displaystyle \frac{\partial f}{\partial x_2x_1} & \displaystyle \frac{\partial f}{\partial x_2^2} \end{bmatrix}$
$Hf(x_1,x_2)=\begin{bmatrix} 4&12\\ 12&38 \end{bmatrix}$
$\text{det}(Hf(x_1,x_2))\gt 0$ so it rules out the possibility of being a saddle point, so $$f(x_1,x_2)$$ either have a maximum or a minimum.
And because $\displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1^2}\gt 0$ so $$f(x_1,x_2)$$ have a minimum values at $$(0,0)$$ and that minimum value is $$0$$

So we had seen that the minimum value of

\(f(x_1,x_2)=2x_1^2+12x_1x_2+19x^2\)

\(0\)

\vec{x}^TA\vec{x}\gt 0

except at

\vec{x}=\vec{0}

, this means that

\(A\)

is a Positive Definite Matrix

Example $$2$$ :
Say

A=\begin{bmatrix} 2&6\\6&7 \end{bmatrix}

check if

\(A\)

is Positive Definite Matrix or not, using properties described above.
Let's try using the

4^{th}

property and see if

\vec{x}^TA\vec{x}\gt 0;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}

Say

\vec{x}=\begin{bmatrix} x_1\\x_2 \end{bmatrix}

\vec{x}^TA\vec{x}= \begin{bmatrix} x_1&x_2 \end{bmatrix} \begin{bmatrix} 2&6\\6&7 \end{bmatrix} \begin{bmatrix} x_1\\x_2 \end{bmatrix}

\vec{x}^TA\vec{x}=2x_1^2+12x_1x_2+7x^2

Now we want to know that is

2x_1^2+12x_1x_2+7x^2\gt 0

Now let's use Calculus.

Our function is $$f(x_1,x_2)=2x_1^2+12x_1x_2+7x^2$$
Let's find the critical points,
First we take derivative w.r.t. $$x_1$$ .

$\begin{matrix} \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=& \displaystyle \frac{\partial (2x_1^2+12x_1x_2+7x^2)}{\partial x_1} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=&\displaystyle 4x_1+12x_2 \end{matrix}$ Now equate $\displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=0$
$\begin{matrix} 4x_1+12x_2=0 \end{matrix} \quad\tag{\color{red}{1}}$
Now we take derivative w.r.t. $$x_2$$ .

$\begin{matrix} \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_2}=& \displaystyle \frac{\partial (2x_1^2+12x_1x_2+7x^2)}{\partial x_2} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_2}=&\displaystyle 12x_1 + 14x_2 \end{matrix}$ Now equate $\displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=0$
$\begin{matrix} 12x_1 + 14x_2=0 \end{matrix} \quad\tag{\color{red}{2}}$
$$x_1=0$$ and $$x_2=0$$ satisfies this system of equations $(1)\text{ and }(2)$ .
Now we got a point $$(x_1,x_2)=(0,0)$$ but we don't know that, Is it minimum, maximum or saddle point.
To check that, Is it minimum, maximum or saddle point. We need $2^{nd}$ derivative test
$\frac{\partial^2 f(x_1,x_2)}{\partial x_1^2}$
$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1^2}=& \displaystyle \frac{\partial (4x_1+12x_2)}{\partial x_1} \\ \displaystyle \frac{\partial f(x_1,x_2)}{\partial x_1}=&\displaystyle 4 \end{matrix}$
$\frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}$
$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}=& \displaystyle \frac{\partial (12x_1 + 14x_2)}{\partial x_2} \\ \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_2^2}=&\displaystyle 14 \end{matrix}$
$\frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}$
$\begin{matrix} \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}=& \displaystyle \frac{\partial (4x_1+12x_2)}{\partial x_2} \\ \displaystyle \frac{\partial^2 f(x_1,x_2)}{\partial x_1x_2}=&\displaystyle 12 \end{matrix}$
(Recommended reading)
Now we have our Hessian matrix (say $$H$$ )
$Hf(x_1,x_2)=\begin{bmatrix} \displaystyle \frac{\partial f}{\partial x_1^2 } & \displaystyle \frac{\partial f}{\partial x_1x_2}\\ \displaystyle \frac{\partial f}{\partial x_2x_1} & \displaystyle \frac{\partial f}{\partial x_2^2} \end{bmatrix}$
$Hf(x_1,x_2)=\begin{bmatrix} 4&12\\ 12&14 \end{bmatrix}$
$\text{det}(Hf(x_1,x_2))\lt 0$ so $$(0,0)$$ is the saddle point and $$f(0,0)=0$$ .

So we had seen that

\(f(x_1,x_2)=2x_1^2+12x_1x_2+19x^2\)

has neither maximum nor minimum point.

\((0,0)\)

is a saddle point of

\(f(x_1,x_2)\)

and

\(f(0,0)=0\)

\vec{x}^TA\vec{x}\ngtr 0 ;\quad\forall x\in\mathbb{R}^n \char`\\ \vec{0}

, this means that

\(A\)

is a not a Positive Definite Matrix