Inner Products and Norms

Properties of Inner Products and Norms

Definition 87: Inner Product Space

An inner product space is a vector space $𝑉$ over $𝔽$ equipped with a function $⟨ \cdot, \cdot ⟩ : 𝑉 \times 𝑉 \to 𝔽$ called the inner product that satisfies the following properties for all vectors $𝐮, 𝐯, 𝐰 \in 𝑉$ and scalars $𝛼, 𝛽 \in 𝔽$ :

Non-Negativity. $⟨ 𝐯, 𝐯 ⟩ \geq 0$ .
Positive-Definiteness. $⟨ 𝐯, 𝐯 ⟩ = 0$ iff $𝐯 = 𝟎$ .
Conjugate Symmetry. $⟨ 𝐮, 𝐯 ⟩ = \bar{⟨ 𝐯, 𝐮 ⟩}$ .
Linearity in the First Argument. $⟨ 𝛼 𝐮 + 𝛽 𝐯, 𝐰 ⟩ = 𝛼 ⟨ 𝐮, 𝐰 ⟩ + 𝛽 ⟨ 𝐯, 𝐰 ⟩$ .

Note

For an inner product space over $ℝ$ , since $\bar{𝛼} = 𝛼$ for all scalars $𝛼$ , conjugate symmetry implies actual symmetry, i.e. $⟨ 𝐮, 𝐯 ⟩ = ⟨ 𝐯, 𝐮 ⟩$ .

Theorem 72: Sesquilinearity of Inner Products

Let $𝑉$ be an inner product space over $𝔽$ . Then for all vectors $𝐮, 𝐯, 𝐰 \in 𝑉$ and scalars $𝛼, 𝛽 \in 𝔽$ , the inner product $⟨ \cdot, \cdot ⟩$ is sesquilinear, i.e., it is linear in the first argument and conjugate-linear in the second argument:

⟨ 𝐮, 𝛼 𝐯 + 𝛽 𝐰 ⟩ = \bar{𝛼} ⟨ 𝐮, 𝐯 ⟩ + \bar{𝛽} ⟨ 𝐮, 𝐰 ⟩ .

Proof: Theorem 72.

Let $𝑉$ be an inner product space over $𝔽$ , and let $𝐮, 𝐯, 𝐰 \in 𝑉$ and $𝛼, 𝛽 \in 𝔽$ . Then:

\begin{matrix} ⟨ 𝐮, 𝛼 𝐯 + 𝛽 𝐰 ⟩ & = \bar{⟨ 𝛼 𝐯 + 𝛽 𝐰, 𝐮 ⟩} & by conjugate symmetry \\ = \bar{𝛼 ⟨ 𝐯, 𝐮 ⟩ + 𝛽 ⟨ 𝐰, 𝐮 ⟩} & by linearity in the first argument \\ = \bar{𝛼} \bar{⟨ 𝐯, 𝐮 ⟩} + \bar{𝛽} \bar{⟨ 𝐰, 𝐮 ⟩} & by distributivity of conjugation \\ = \bar{𝛼} ⟨ 𝐮, 𝐯 ⟩ + \bar{𝛽} ⟨ 𝐮, 𝐰 ⟩ & by conjugate symmetry, \end{matrix}

as desired. ∎

Note

For an inner product space over $ℝ$ , since $𝛼 = \bar{𝛼}$ for all scalars $𝛼$ , sesquilinearity implies bilinearity, i.e. linearity in both arguments:

⟨ 𝐮, 𝛼 𝐯 + 𝛽 𝐰 ⟩ = 𝛼 ⟨ 𝐮, 𝐯 ⟩ + 𝛽 ⟨ 𝐮, 𝐰 ⟩ for 𝛼, 𝛽 \in ℝ .

Property: Inner Products with Zero Vectors

Let $𝑉$ be an inner product space over $𝔽$ . Then for all vectors $𝐯 \in 𝑉$ :

⟨ 𝟎, 𝐯 ⟩ = ⟨ 𝐯, 𝟎 ⟩ = 0 .

Theorem 73: Cauchy-Schwarz Inequality

Let $𝑉$ be an inner product space over $𝔽$ . Then for all vectors $𝐮, 𝐯 \in 𝑉$ :

{| ⟨ 𝐮, 𝐯 ⟩ |}^{2} \leq ⟨ 𝐮, 𝐮 ⟩ ⟨ 𝐯, 𝐯 ⟩ .

Proof: Cauchy-Schwarz Inequality.

Let $𝑉$ be an inner product space over $𝔽$ , and let $𝐮, 𝐯 \in 𝑉$ . If $𝐯 = 𝟎$ , then $⟨ 𝐮, 𝐯 ⟩ = ⟨ 𝐮, 𝟎 ⟩ = 0$ , so the inequality holds trivially. Now suppose $𝐯 \neq 𝟎$ . Then:

Theorem 74: Cauchy-Schwarz Equality

Let $𝑉$ be an inner product space over $𝔽$ . Then for all vectors $𝐮, 𝐯 \in 𝑉$ , equality holds in the Cauchy-Schwarz inequality if and only if $𝐮$ is a scalar multiple of $𝐯$ :

{| ⟨ 𝐮, 𝐯 ⟩ |}^{2} = ⟨ 𝐮, 𝐮 ⟩ ⟨ 𝐯, 𝐯 ⟩ if and only if 𝐮 = 𝛼 𝐯 for some 𝛼 \in 𝔽 .

Definition 88: Normed Vector Space

A normed vector space is a vector space $𝑉$ over $𝔽$ equipped with a function
$‖ \cdot ‖ : 𝑉 \to ℝ$ called the norm that satisfies the following properties for all vectors $𝐯, 𝐰 \in 𝑉$ and scalars $𝛼 \in 𝐹$ :

Non-Negativity. $‖ 𝐯 ‖ \geq 0$ .
Positive-Definiteness. $‖ 𝐯 ‖ = 0$ iff $𝐯 = 𝟎$ .
Absolute Homogeneity. $‖ 𝛼 𝐯 ‖ = | 𝛼 | ‖ 𝐯 ‖$ .
Triangle Inequality. $‖ 𝐯 + 𝐰 ‖ \leq ‖ 𝐯 ‖ + ‖ 𝐰 ‖$ .

Norms Induced by Inner Products

Definition 89: Induced Norm

Given an inner product space $𝑉$ over $𝔽$ with inner product $⟨ \cdot, \cdot ⟩$ , the norm induced by the inner product $⟨ \cdot, \cdot ⟩$ , called the induced norm or canonical norm, is the function $‖ \cdot ‖ : 𝑉 \to ℝ$ defined by:

‖ 𝐯 ‖ = \sqrt{⟨ 𝐯, 𝐯 ⟩} .

Corollary 39

All inner product spaces over $𝔽$ are normed vector spaces, whose norms are induced by Definition 89.

Using the induced norm, we can restate Theorem 73 in terms of norms as follows:

Note: Restatement of the Cauchy-Schwarz Inequality

Let $𝑉$ be an inner product space with induced norm $‖ \cdot ‖$ . Then $\forall 𝐮, 𝐯 \in 𝑉$ :

| ⟨ 𝐮, 𝐯 ⟩ | \leq ‖ 𝐮 ‖ ‖ 𝐯 ‖ .

Theorem 75: Validity of the Induced Norm

Let $𝑉$ be an inner product space over $𝔽$ with inner product $⟨ \cdot, \cdot ⟩$ . Then the induced norm $‖ \cdot ‖$ is a valid norm on $𝑉$ , i.e., it satisfies all four properties of a norm.

Proof: Validity of the Induced Norm.

Let $𝑉$ be an inner product space over $𝔽$ with inner product $⟨ \cdot, \cdot ⟩$ , and let $‖ \cdot ‖ = \sqrt{⟨ \cdot, \cdot ⟩}$ be the induced norm on $𝑉$ . Then for all vectors $𝐯, 𝐰 \in 𝑉$ and scalars $𝛼 \in 𝔽$ :

Non-Negativity. $⟨ 𝐯, 𝐯 ⟩ \geq 0$ , so $‖ 𝐯 ‖ = \sqrt{⟨ 𝐯, 𝐯 ⟩} \geq 0$ .
Positive-Definiteness. $⟨ 𝐯, 𝐯 ⟩ = 0$ iff $𝐯 = 𝟎$ , so $‖ 𝐯 ‖ = \sqrt{⟨ 𝐯, 𝐯 ⟩} = \sqrt{0} = 0$ iff $𝐯 = 𝟎$ .
Absolute Homogeneity.
$\begin{matrix} ‖ 𝛼 𝐯 ‖ & = \sqrt{⟨ 𝛼 𝐯, 𝛼 𝐯 ⟩} & by definition of the induced norm \\ = \sqrt{𝛼 \bar{𝛼} ⟨ 𝐯, 𝐯 ⟩} & by sesquilinearity \\ = \sqrt{{| 𝛼 |}^{2} ⟨ 𝐯, 𝐯 ⟩} & by properties of complex numbers \\ = | 𝛼 | \sqrt{⟨ 𝐯, 𝐯 ⟩} & by properties of square roots \\ = | 𝛼 | ‖ 𝐯 ‖ & by definition of the induced norm . \end{matrix}$
Triangle Inequality.
$\begin{matrix} {‖ 𝐯 + 𝐰 ‖}^{2} & = ⟨ 𝐯 + 𝐰, 𝐯 + 𝐰 ⟩ & by definition of the induced norm \\ = ⟨ 𝐯, 𝐯 ⟩ + ⟨ 𝐯, 𝐰 ⟩ + ⟨ 𝐰, 𝐯 ⟩ + ⟨ 𝐰, 𝐰 ⟩ & by sesquilinearity \\ = ⟨ 𝐯, 𝐯 ⟩ + ⟨ 𝐯, 𝐰 ⟩ + \bar{⟨ 𝐯, 𝐰 ⟩} + ⟨ 𝐰, 𝐰 ⟩ & by conjugate symmetry \\ = ⟨ 𝐯, 𝐯 ⟩ + 2 ℜ (⟨ 𝐯, 𝐰 ⟩) + ⟨ 𝐰, 𝐰 ⟩ & by properties of complex conjugates \\ \leq ⟨ 𝐯, 𝐯 ⟩ + 2 | ⟨ 𝐯, 𝐰 ⟩ | + ⟨ 𝐰, 𝐰 ⟩ & by properties of complex numbers \\ \leq ⟨ 𝐯, 𝐯 ⟩ + 2 ‖ 𝐯 ‖ ‖ 𝐰 ‖ + ⟨ 𝐰, 𝐰 ⟩ & by the Cauchy-Schwarz Inequality \\ = {‖ 𝐯 ‖}^{2} + 2 ‖ 𝐯 ‖ ‖ 𝐰 ‖ + {‖ 𝐰 ‖}^{2} & by definition of the induced norm \\ = {(‖ 𝐯 ‖ + ‖ 𝐰 ‖)}^{2} & by factorization . \end{matrix}$
Taking the square root of both sides, we have:
$‖ 𝐯 + 𝐰 ‖ \leq ‖ 𝐯 ‖ + ‖ 𝐰 ‖ as desired .$

Thus, the induced norm $‖ \cdot ‖$ satisfies the four axiomatic properties of a norm. ∎

Distance and Metric Spaces

Definition 90: Metric Space

A metric space is a set $𝑋$ equipped with a function $𝑑 : 𝑋 \times 𝑋 \to ℝ$ called the metric or distance function that satisfies the following properties for all points $𝑥, 𝑦, 𝑧 \in 𝑋$ :

Non-Negativity. $𝑑 (𝑥, 𝑦) \geq 0$ .
Positive-Definiteness. $𝑑 (𝑥, 𝑦) = 0$ iff $𝑥 = 𝑦$ .
Symmetry. $𝑑 (𝑥, 𝑦) = 𝑑 (𝑦, 𝑥)$ .
Triangle Inequality. $𝑑 (𝑥, 𝑧) \leq 𝑑 (𝑥, 𝑦) + 𝑑 (𝑦, 𝑧)$ .

Definition 91: Induced Metric

Given a normed vector space $𝑉$ over $𝔽$ with norm $‖ \cdot ‖$ , the metric induced by the norm $‖ \cdot ‖$ , called the induced metric, is the function $𝑑 : 𝑉 \times 𝑉 \to ℝ$ defined by:

𝑑 (𝐮, 𝐯) = ‖ 𝐮 - 𝐯 ‖ .

We call $𝑑 (𝐮, 𝐯)$ the distance between the vectors $𝐮$ and $𝐯$ .

Corollary 40

All normed vector spaces are metric spaces, whose metric is induced by Definition 91. Thus, all inner product spaces are metric spaces, whose metric is induced by the norm induced by the inner product.

Important

All inner product spaces are normed vector spaces, which are all metric spaces.

Theorem 76: Validity of the Induced Metric

Let $𝑉$ be a normed vector space over $𝔽$ with norm $‖ \cdot ‖$ . Then the induced metric $𝑑$ is a valid metric on $𝑉$ , i.e., it satisfies all four properties of a metric.

Special Inner Products and Norms

Definition 92: Dot Product

Let $𝑉 = 𝔽^{𝑛}$ be the vector space of $𝑛$ -tuples over a field $𝔽$ . The dot product is the function $⟨ \cdot, \cdot ⟩ : 𝑉 \times 𝑉 \to 𝔽$ defined by:

⟨ 𝐮, 𝐯 ⟩ = \sum_{𝑖 = 1}^{𝑛} 𝑢_{𝑖} \bar{𝑣_{𝑖}},

where $𝐮 = (𝑢_{1}, 𝑢_{2}, \dots, 𝑢_{𝑛})$ and $𝐯 = (𝑣_{1}, 𝑣_{2}, \dots, 𝑣_{𝑛})$ .

Terminology

For $𝑉 = ℂ^{𝑛}$ , the dot product is also called the Hermitian inner product.

Notation

The notation $𝐮 \cdot 𝐯$ can be used to emphasize that the dot product is being used as the inner product.

Proposition 19

The dot product is an inner product on $𝔽^{𝑛}$ .

Definition 93: $𝑝$ -Norm

Let $𝑉 = 𝔽^{𝑛}$ be the vector space of $𝑛$ -tuples over $𝔽$ . For any $𝑝 \in ℝ_{\geq 1}$ , the $𝒑$ -norm (or $𝐿^{𝑝}$ -norm) is the function ${‖ \cdot ‖}_{𝑝} : 𝑉 \to ℝ$ defined by:

{‖ 𝐯 ‖}_{𝑝} = {(\sum_{𝑖 = 1}^{𝑛} | 𝑣_{𝑖} |^{𝑝})}^{1 / 𝑝},

where $𝐯 = (𝑣_{1}, 𝑣_{2}, \dots, 𝑣_{𝑛})$ .

Proposition 20

For all $𝑝 \geq 1$ , the $𝑝$ -norm is a valid norm on $𝔽^{𝑛}$ .

Proposition 21

Extend the function ${‖ \cdot ‖}_{𝑝}$ from Definition 93 to accept $0 < 𝑝 < 1$ . Then for all $0 < 𝑝 < 1$ , ${‖ \cdot ‖}_{𝑝}$ is not a valid norm on $𝔽^{𝑛}$ .

Proof Sketch: Proposition 21.

Show that the triangle inequality does not hold for

{‖ \cdot ‖}_{𝑝}

when

0 < 𝑝 < 1

Terminology

The $2$ -norm is also called the Euclidean norm.

Definition 94: Maximum Norm

Let $𝑉 = 𝔽^{𝑛}$ be the vector space of $𝑛$ -tuples over $𝔽$ . The maximum norm (or $𝐿^{\infty}$ -norm) is the function ${‖ \cdot ‖}_{\infty} : 𝑉 \to ℝ$ defined by:

{‖ 𝐯 ‖}_{\infty} = \max_{1 \leq 𝑖 \leq 𝑛} | 𝑣_{𝑖} |,

where $𝐯 = (𝑣_{1}, 𝑣_{2}, \dots, 𝑣_{𝑛})$ .

Proposition 22

For all $𝐯 \in 𝔽^{𝑛}$ , ${‖ 𝐯 ‖}_{\infty} = \lim_{𝑝 \to \infty} {‖ 𝐯 ‖}_{𝑝}$ .

Exercises

Exercise 75: Inner Product of Functions.

Let $𝑉 = 𝐶_{[𝑎, 𝑏]} (𝔽)$ be the vector space of continuous functions from a closed interval $[𝑎, 𝑏] \subseteq ℝ$ to a field $𝔽$ . Define a function $⟨ \cdot, \cdot ⟩ : 𝑉 \times 𝑉 \to 𝔽$ by:

⟨ 𝑓, 𝑔 ⟩ = \int_{𝑎}^{𝑏} 𝑓 (𝑥) \bar{𝑔 (𝑥)} d 𝑥,

where $𝑓, 𝑔 \in 𝐶_{[𝑎, 𝑏]} (𝔽)$ . Show that $⟨ \cdot, \cdot ⟩$ is an inner product on $𝑉$ .

Exercise 76.

Show that for all functions $𝑓 \in 𝐶_{[𝑎, 𝑏]} (ℝ)$ with $𝑏 - 𝑎 \leq 1$ , we have:

{(\int_{𝑎}^{𝑏} 𝑓 (𝑥) d 𝑥)}^{2} \leq \int_{𝑎}^{𝑏} {(𝑓 (𝑥))}^{2} d 𝑥 .

Exercise 77.

Prove Proposition 21.