Orthogonality and Projections

Orthogonality

Definition 95: Orthogonality of Two Vectors

Let $𝑉$ be an inner product space over $𝔽$ . Two vectors $𝐮, 𝐯 \in 𝑉$ are orthogonal iff:

⟨ 𝐮, 𝐯 ⟩ = 0,

in which case we say that $𝐮$ is orthogonal to $𝐯$ , denoted $𝐮 ⟂ 𝐯$ .

Notation

If $𝐮$ is not orthogonal to $𝐯$ , we write

Example.

Let $ℝ^{2}$ be an inner product space with dot product as the inner product. Then the vectors $𝐮 = (1, 2)$ and $𝐯 = (2, - 1)$ are orthogonal because:

⟨ 𝐮, 𝐯 ⟩ = 𝐮 \cdot 𝐯 = (1, 2) \cdot (2, - 1) = 1 \cdot 2 + 2 \cdot (- 1) = 0 .

Property: Orthogonality is Symmetric

Let $𝑉$ be an inner product space. Then for all vectors $𝐮, 𝐯 \in 𝑉$ , $𝐮 ⟂ 𝐯$ if and only if $𝐯 ⟂ 𝐮$ .

Theorem 77: Pythagorean Theorem

Let $𝑉$ be an inner product space. Then for all vectors $𝐮, 𝐯 \in 𝑉$ , if $𝐮 ⟂ 𝐯$ , then:

⟨ 𝐮 + 𝐯, 𝐮 + 𝐯 ⟩ = ⟨ 𝐮, 𝐮 ⟩ + ⟨ 𝐯, 𝐯 ⟩ .

In terms of the induced norm $‖ \cdot ‖$ , this can be expressed as:

{‖ 𝐮 + 𝐯 ‖}^{2} = {‖ 𝐮 ‖}^{2} + {‖ 𝐯 ‖}^{2} .

Proof: Pythagorean Theorem.

Suppose $𝑉$ is an inner product space over $𝔽$ , and let $𝐮, 𝐯 \in 𝑉$ such that $𝐮 ⟂ 𝐯$ . Then:

\begin{matrix} ⟨ 𝐮 + 𝐯, 𝐮 + 𝐯 ⟩ & = ⟨ 𝐮, 𝐮 + 𝐯 ⟩ + ⟨ 𝐯, 𝐮 + 𝐯 ⟩ & by linearity in the first argument \\ = ⟨ 𝐮, 𝐮 ⟩ + ⟨ 𝐮, 𝐯 ⟩ + ⟨ 𝐯, 𝐮 ⟩ + ⟨ 𝐯, 𝐯 ⟩ & by sesquilinearity \\ = ⟨ 𝐮, 𝐮 ⟩ + 0 + 0 + ⟨ 𝐯, 𝐯 ⟩ & because 𝐮 ⟂ 𝐯 \\ = ⟨ 𝐮, 𝐮 ⟩ + ⟨ 𝐯, 𝐯 ⟩ . & ∎ \end{matrix}

Definition 96: Orthogonal Set of Vectors

Let $𝑉$ be an inner product space over $𝔽$ . A set of vectors $𝑆 \subseteq 𝑉$ is orthogonal iff every pair of distinct vectors in $𝑆$ is orthogonal, i.e., for all $𝐮, 𝐯 \in 𝑆$ , if $𝐮 \neq 𝐯$ , then $𝐮 ⟂ 𝐯$ :

{𝐯_{1}, 𝐯_{2}, \dots, 𝐯_{𝑝}} orthogonal \Leftrightarrow ⟨ 𝐯_{𝑖}, 𝐯_{𝑗} ⟩ = 0 for all 𝑖, 𝑗 \in {1, \dots, 𝑝} with 𝑖 \neq 𝑗 .

Theorem 78: Orthogonal Vectors are Linearly Independent

Let $𝑉$ be an inner product space over $𝔽$ . Then any set of nonzero orthogonal vectors in $𝑉$ is linearly independent.

Definition 97: Orthonormal Set of Vectors

Let $𝑉$ be an inner product space over $𝔽$ . A set of vectors $𝑆 \subseteq 𝑉$ is orthonormal iff $𝑆$ is orthogonal and every vector in $𝑆$ has norm $1$ , or equivalently:

{𝐯_{1}, 𝐯_{2}, \dots, 𝐯_{𝑝}} orthonormal \Leftrightarrow ⟨ 𝐯_{𝑖}, 𝐯_{𝑗} ⟩ = {\begin{matrix} 1 if 𝑖 = 𝑗 \\ 0 if 𝑖 \neq 𝑗 \end{matrix} for all 𝑖, 𝑗 \in {1, \dots, 𝑝} .

Orthogonal Complements

Definition 98: Orthogonal Complement

Let $𝑉$ be an inner product space, and let $𝑊$ be a subspace of $𝑉$ . The orthogonal complement of $𝑊$ in the ambient space $𝑉$ , denoted $𝑊^{⟂}$ , is the set of vectors in $𝑉$ that are orthogonal to every vector in $𝑊$ :

𝑊^{⟂} = {𝐯 \in 𝑉 | 𝐯 ⟂ 𝐰 for all 𝐰 \in 𝑊} .

Notation: Orthogonality of Vectors to Subspaces

Let $𝑉$ be an inner product space, and let $𝑊$ be a subspace of $𝑉$ . A vector $𝐯 \in 𝑉$ is orthogonal to $𝑊$ , denoted $𝐯 ⟂ 𝑊$ , iff $𝐯 ⟂ 𝐰$ for all $𝐰 \in 𝑊$ .

The above notation allows us to express Definition 98 more succinctly as:

𝑊^{⟂} = {𝐯 \in 𝑉 | 𝐯 ⟂ 𝑊} .

Notation: Orthogonality of Subspaces to Subspaces

Let $𝑉$ be an inner product space, and let $𝑋, 𝑌$ be subspaces of $𝑉$ . We say that $𝑋$ is orthogonal to $𝑌$ , denoted $𝑋 ⟂ 𝑌$ , iff every vector in $𝑋$ is orthogonal to every vector in $𝑌$ :

𝑋 ⟂ 𝑌 \Leftrightarrow 𝐱 ⟂ 𝐲 for all 𝐱 \in 𝑋 and 𝐲 \in 𝑌 .

Proposition 23: Orthogonal Complements are Subspaces

Let $𝑉$ be an inner product space and $𝑊$ be a subspace of $𝑉$ . Then $𝑊^{⟂}$ is a subspace of $𝑉$ .

Proof: Orthogonal Complements are Subspaces.

Suppose $𝑉$ is an inner product space over $𝐹$ , and let $𝑊$ be a subspace of $𝑉$ .

Let $𝐮, 𝐯 \in 𝑊^{⟂}$ and $𝛼, 𝛽 \in 𝐹$ . Then for all $𝐰 \in 𝑊$ :

\begin{matrix} ⟨ 𝛼 𝐮 + 𝛽 𝐯, 𝐰 ⟩ & = 𝛼 ⟨ 𝐮, 𝐰 ⟩ + 𝛽 ⟨ 𝐯, 𝐰 ⟩ & by linearity in the first argument \\ = 𝛼 \cdot 0 + 𝛽 \cdot 0 & because 𝐮 ⟂ 𝑊 and 𝐯 ⟂ 𝑊 \\ = 0 . \end{matrix}

Thus, $𝛼 𝐮 + 𝛽 𝐯 ⟂ 𝑊$ , which implies that $𝛼 𝐮 + 𝛽 𝐯 \in 𝑊^{⟂}$ . Therefore, $𝑊^{⟂}$ is closed under linear combinations, so by Theorem 11, $𝑊^{⟂}$ is a subspace of $𝑉$ . ∎

Theorem 79: Orthogonal Complements are Complements

Let $𝑉$ be a finite-dimensional inner product space, and let $𝑊$ be a subspace of $𝑉$ . Then:

𝑉 = 𝑊 \oplus 𝑊^{⟂} .

Corollary 41

By Corollary 10, $\dim (𝑉) = \dim (𝑊) + \dim (𝑊^{⟂})$ .

This can be arranged to give a formula for the dimension of the orthogonal complement:

Corollary 42: Dimension of Orthogonal Complements

Let $𝑉$ be a finite-dimensional inner product space, and let $𝑊$ be a subspace of $𝑉$ . Then:

\dim (𝑊^{⟂}) = \dim (𝑉) - \dim (𝑊) .

Theorem 80: Involutivity of Orthogonal Complements

Let $𝑊$ be a finite-dimensional subspace of an inner product space $𝑉$ . Then:

{(𝑊^{⟂})}^{⟂} = 𝑊 .

Theorem 81: Characterizing Orthogonal Complements using Spanning Sets

Let $𝑉$ be an inner product space, and let $𝑊$ be a subspace of $𝑉$ . Suppose $𝑆 \subseteq 𝑊$ is a spanning set for $𝑊$ . Then $𝐯 ⟂ 𝐬$ for all $𝐬 \in 𝑆$ is a sufficient condition for $𝐯 \in 𝑊^{⟂}$ .

Orthogonal Projections

Definition 99: Orthogonal Projection

Let $𝑉$ be an inner product space, and let $𝑊$ be a subspace of $𝑉$ . The orthogonal projection of a vector $𝐯 \in 𝑉$ onto the subspace $𝑊$ , denoted ${proj}_{𝑊} (𝐯)$ or $\hat{𝐯}$ , is the unique vector in $𝑊$ such that:

𝐯 - \hat{𝐯} \in 𝑊^{⟂} .

Exercises

Exercise 78.

Show that for any inner product space

𝑉

and subspace

𝑊

𝑉

𝑊 \cap 𝑊^{⟂} = {𝟎}

Exercise 79.

Show that for any inner product space

𝑉

𝑉^{⟂} = {𝟎}

and

{𝟎}^{⟂} = 𝑉

Exercise 80.

Show that if

𝑊

is a finite-dimensional subspace of an inner product space

𝑉

, then

𝑈^{⟂} = {𝟎}

if and only if

𝑈 = 𝑉

Exercise 81.

Let $𝑋, 𝑌$ be subspaces of an inner product space $𝑉$ . Show that

if 𝑋 \subseteq 𝑌, then 𝑌^{⟂} \subseteq 𝑋^{⟂} .

Exercise 82.

Let $𝑋, 𝑌$ be subspaces of an inner product space $𝑉$ . Show that:

{(𝑋 + 𝑌)}^{⟂} = 𝑋^{⟂} \cap 𝑌^{⟂} .

Exercise 83.

Let $𝑋, 𝑌$ be subspaces of an inner product space $𝑉$ . Show that:

{(𝑋 \cap 𝑌)}^{⟂} = 𝑋^{⟂} + 𝑌^{⟂} .

Exercise 84.

Prove or disprove that if

𝑉 = 𝑋 \oplus 𝑌

, then

𝑉 = 𝑋^{⟂} \oplus 𝑌^{⟂}

Exercise 85: Converse of the Pythagorean Theorem.

Let $𝑉$ be an inner product space, and let some $𝐮, 𝐯 \in 𝑉$ satisfy:

{‖ 𝐮 + 𝐯 ‖}^{2} = {‖ 𝐮 ‖}^{2} + {‖ 𝐯 ‖}^{2} .

Show that if $𝑉$ is a vector space over $ℝ$ , then $𝐮 ⟂ 𝐯$ , but that this is not necessarily true if $𝑉$ is a vector space over $ℂ$ .

Exercise 86: Parallelogram Law.

Let $𝑉$ be an inner product space, and let $𝐮, 𝐯 \in 𝑉$ . Show that:

{‖ 𝐮 + 𝐯 ‖}^{2} + {‖ 𝐮 - 𝐯 ‖}^{2} = 2 {‖ 𝐮 ‖}^{2} + 2 {‖ 𝐯 ‖}^{2} .

Exercise 87.

Let $𝑉$ be an inner product space, and let $𝐮, 𝐯 \in 𝑉$ . Show that:

| ‖ 𝐮 ‖ - ‖ 𝐯 ‖ | \leq ‖ 𝐮 - 𝐯 ‖ .

Exercise 88.

Suppose

𝑉

is an inner product space. Let

{𝐪_{1}, 𝐪_{2}, \dots, 𝐪_{𝑘}} \subseteq 𝑉

be an orthonormal set with

𝑘 \geq 2

, and let

𝐯 \in {(span {𝐪_{1}, 𝐪_{2}, \dots, 𝐪_{𝑘}})}^{⟂}

. Show that if

𝐯 \neq 𝟎

, then no two distinct vectors in the set

{𝐯 + 𝐪_{1}, 𝐯 + 𝐪_{2}, \dots, 𝐯 + 𝐪_{𝑘}}

are orthogonal.

Exercise 89.

Suppose

𝑉

is an inner product space and let

𝑃 : 𝑉 \to 𝑉

be a linear map such that

𝑃^{2} = 𝑃

(idempotent). Show that

im (𝑃) ⟂ \ker (𝑃)

if and only if

‖ 𝑃 (𝐯) ‖ \leq ‖ 𝐯 ‖

for every

𝐯 \in 𝑉