Inner Products, Norms and Metrics

Most people do have an intuitive understanding of the real number system: it is the number system that should be used for measurements of space and time as well as for other physical quantities that are thought of as varying continuously rather than discretely. This number system is also the basis for the modern understanding of geometrical concepts such as length, distance and angle. On the other hand, inner products, norms and metrics are needed for fundamental topics such as convergence, continuity and several concepts within topology.

Hence, the overall objective of this blog post is to foster the intuition, to explain the motivation behind and the interrelation between inner products, norms and metrics of inner products. All three function types are closely related to each other since an inner product can induce a norm, which itself can ultimately induce a corresponding metric.

Metrics are nothing but distance functions just like a reasonable person would imagine. A norm can also geometrically be interpreted as the distance of a real-valued point (e.g. in the Euclidean plane) to the origin measured via a corresponding distance function (, that can be induced by this norm). The interpretation of inner product is not that obvious, but one can see dot-products as something like a measure of how similar two vectors are. There are also helpful animated illustration of 3Blue1Brown and Zach Star (links below).

Relationship between Inner Products, Norms and Metrics
Fig. 1: Relationship between Inner Products, Norms and Metrics

All three function types life in a vector space. We assume the reader knows the basics about vector spaces. A rather short introduction to Euclidean vector spaces is provided, though.

Euclidean Spaces

It might turn out to be quite helpful to recall the basic knowledge about vector spaces. A very good overview is provided by the following videos of PacoVideoLectures:

Fundamental properties of vectors
Vector basis, span & linear independence
n-dimensional vector space

Ok, now that we have recalled the basics of vector spaces let us go back to Euclidean spaces. A point v\in \mathbb{R}^2 in the 2-dimensional plane can be modeled as an ordered pair v=\begin{pmatrix} x_1 \\ x_2 \end{pmatrix} \in\mathbb{R}^2 of real numbers. Similarly, a point in a 3-dimensional space can be interpreted as an ordered triple \begin{pmatrix} x_1 \\ x_2\\ x_3 \end{pmatrix} of real numbers. In general, we consider a set of n-tuples \begin{pmatrix} x_1 \\ \vdots \\ x_n \end{pmatrix} of real numbers. Usually, these type of vectors are column vectors.

Let v=(x_1, \ldots, x_n)^T and w=(y_1, \ldots, y_n)^T be in \mathbb{R}^n. We define equality v=w if, and only if, x_1=y_1, \ldots, x_n=y_n. The sum (v+w) and the difference (v-w)=(v+(-w)) are component-wise, that is, (x\pm y)=(x_1\pm y_1, \ldots, x_n \pm y_n)^T. The multiplication by real numbers \lambda\in \mathbb{R} (also called scalars) is defined by \lambda \cdot (x_1, \ldots, x_n)^T = (\lambda x_1, \ldots, \lambda x_n)^T.

In modern terms, an Euclidean space is a vector space equipped with an “inner product”. Actually, an Euclidean space is simply a space of classical geometry. In any case, we need to define what an inner product actually is.

Inner Products

An inner product \langle \ , \rangle, also called dot product, is a function that enables us to define and apply geometrical terms such as length, distance and angle in an Euclidean (vector) space (V, \langle \ , \rangle).

Please recall that metrics (distance functions) can be induced by inner products.

Definition 2.1:
Let V be a vector space over \mathbb{R}. An inner product on V is a function \langle \ , \rangle: V \times V \rightarrow \mathbb{R} such that for all u,v,w \in V and a,b \in \mathbb{R}, the following hold:

  1. \langle v , v \rangle \geq 0 and \langle v , v \rangle = 0 \Leftrightarrow v=0;
  2. \langle a u + b v , w \rangle = a \langle u , w \rangle + b \langle v , w \rangle;
  3. \langle u , v \rangle = \langle v , u \rangle.

Inner products can be generally defined as a symmetric bilinear form and a bilinear form is a bivariate functions that is linear with respect to each isolated argument. Note that the above definition can be generalized to complex vector spaces employing sesquilinear forms. We, however, restrict ourselves to real vector spaces and to the consideration of the standard inner product as defined in the following example.

Example 2.1:
The so-called standard inner product or Euclidean inner product of the real vector space \mathbb{R}^n is defined by \langle x, y\rangle := \sum_{i=1}^{n}{x_i y_i} for x, y\in \mathbb{R}^n. Please convince yourself that this function actually fulfills the required properties.

\square

For n=2 and x:=\begin{pmatrix} 1 \\ 3 \end{pmatrix}, y:=\begin{pmatrix} -4 \\ 2 \end{pmatrix}\in \mathbb{R}^2 we get \langle \begin{pmatrix} 1 \\ 3 \end{pmatrix},\begin{pmatrix} -4 \\ 2 \end{pmatrix}\rangle=1\cdot (-4) + 3\cdot 2 =-4+6=2.

Note that you can also use matrix multiplication to define the Euclidean inner product. For instance, if you have \begin{pmatrix} x_1 \\ y_1 \end{pmatrix}, \begin{pmatrix} x_2 \\ y_2 \end{pmatrix}\in \mathbb{R}^2, then \langle \begin{pmatrix} x_1 \\ y_1 \end{pmatrix},\begin{pmatrix} x_2 \\ y_2 \end{pmatrix}\rangle=(x_1, y_1) \cdot \begin{pmatrix} x_2 \\ y_2 \end{pmatrix}. Note that the last term of the equation shows a matrix multiplication of a 1-times-2 and a 2-times-1 matrix.

An interpretation of inner products is not that obvious. An inner product basically provides a meausre of how similar the two input vectors are.

A geometric illustration of dot products is explained by 3Blue1Brown. While watching bear in mind that an inner product is a symmetric bilinear form.

If you are interested in the duality of vector spaces, refer to this article (in German).

There is also another nice video of applications of inner products by Zach Star:

It might also help to learn (more) about norms and metrics. Knowing that inner products, norms and metrics are closely related to each other means that understanding one concept provides also heuristic information about the other concept.

Finally, another very nice video about the (mathematical) application of the inner product.

Norms

Functions closely related to inner products are so-called norms. Norms are specific functions that can be interpreted as a distance function between a vector and the origin.

Definition 3.1:
Let V be a real vector space over \mathbb{R}. A function ||\cdot||: V \rightarrow \mathbb{R} with v \mapsto ||v|| is called norm, and, (V, ||\cdot||) is called normed space if for all v,w \in V the following holds true:

(i) ||v|| > 0 if v\neq 0 (positivity);

(ii) ||\alpha v|| = \alpha ||v|| for all \alpha \in \mathbb{R} (homogenity);

(iii) ||v+w|| \leq ||v|| + ||w|| (triangle inequality).

\square

Properties (ii) and (iii) make only sense over a vector space. The former property is stating how the scalar product of a vector and a field element needs to behave to be a valid norm function.

An example will help to clarify what a norm function is.

Example 3.1:
(a) Let ||\cdot||: V:=\mathbb{R} \rightarrow \mathbb{R} defined by ||v||:=|v| the absolute value, that is a norm over the one-dimensional real vector space. It is always positive and only zero if v=0 and it also fulfills homogenity by definition.

To see that the triangle inequality (also called subadditivity) holds, first bear the definition of the absolute value in mind. If x,y\in \mathbb{R} and x\leq y as well as -x\leq y then |x|\leq |y|. In addition, x \leq |x|, -x \leq |x| as well as y \leq y, -y \leq y hold true. Hence, by adding both inequalities we get x+y \leq |x|+|y| as well as -(x+y)=-x-y\leq |x|+|y|, as desired.

(b) Let v=\begin{pmatrix} x \\ y \end{pmatrix}\in \mathbb{R}^2, ||\cdot||: V:=\mathbb{R}^2 \rightarrow \mathbb{R} defined by ||v|| := \langle v , v \rangle^{\frac{1}{2}} = \sqrt{\langle v , v \rangle} = ||\begin{pmatrix} x \\ y \end{pmatrix}|| = \left( x^2+y^2\right)^{\frac{1}{2}} = \sqrt{\left( x^2+y^2\right)}, which is called Euclidean norm. Note that \langle v , v \rangle is the Euclidean inner product as defined in Example 2.1.

\square

If the Euclidean norm ||v||= \sqrt{\langle v, v \rangle} can be interpreted as the length between the origin and the vector v, then the Euclidean inner product \langle v, v \rangle can be interpreted as the squared norm ||v||^2.

Example (a) is actually the most important example of a norm since basically every practically important norm can be traced back to the absolute value function in some sense.

The Euclidean norm of the vector \begin{pmatrix} x \\ y \end{pmatrix} equals ||\begin{pmatrix} x \\ y \end{pmatrix}|| = \sqrt{x^2+y^2}  \Leftrightarrow ||\begin{pmatrix} x \\ y \end{pmatrix}||^2 = x^2+y^2, which is exactly the length of hypotenuse of the corresponding triangle in Figure 2. That is, the Euclidean norm equals the distance between the origin and the point (x,y) in the Euclidean plane, which can also be derived from Pythagorean’s theorem. It also explains why the property (iii) is called triangle property ( and sometimes also subaditivity).

Illustration of Cauchy-Schwartz's inequality
Fig. 2: Illustration of the relation between Euclidean norm and Pythagorean theorem

The norm ||v-w|| is also called distance between the two real-valued vectors v and w. The angle 0\leq \alpha\leq 2\pi between two vectors v and w with v, w\neq 0 is defined by \cos(\alpha)=\frac{\langle v, w\rangle}{||v|| \cdot ||w||}. This naturally leads us to the definition of another closely related function type.

Metrics

Let us directly define the important term metric.

Definition 4.1:
Let V be a real vector space over \mathbb{R}. A function d:V \times V \rightarrow \mathbb{R} is called metric or distance function on V, and, (V, d) a metric space, if for all u,v,w\in V the following holds true:

(i) d(v,w) \geq 0 and d(v,w)=0 if and only if v=w;

(ii) d(v,w) = d(v,w);

(iii) d(v,w) \leq d(v, u) + d(u, w).

\square

Based on a day-to-day experience, the following basic considerations are reflected in (i) to (iii) of the definition of a metric:

  • Distances between two points can be represented by a positive real number;
  • Distances are independent from the order of measuring. That is, the distance between two points is always the same no matter at what point you start to measure;
  • Detours will imply longer distances.

Hence, the definition of a metric is nothing but an abstraction of the main features of a reasonable distance concept.

Example 4.1:
Consider d:\mathbb{R} \times\mathbb{R} \rightarrow \mathbb{R} defined by d(v,w):= |v-w|, whereby |.| is the absolute value. The function d is a metric since the absolute value is always positive and |v-w| = 0 if and only if v=w. Also (ii) is obvious. Property (iii), also called sub-additivity or triangle property, can be shown by distinguishing several cases. For more details reg. the properties of the absolute value function please refer to the corresponding wiki article.

\square

Topological features do play a big role in mathematical analysis as it is together with measure theory the foundation of how things can be measured. By using the norm and/or distances, we could also define topological terms such as open sets or balls.

Definition 4.2: Let (X, d) be a metric space and a\in X. Furthermore, let \epsilon\in \mathbb{R}, \epsilon>0.
(i) The set B(a, \epsilon):=\{x\in X | \ d(x,a)<\epsilon\} is called \epsilonball of a in (X, d).
(ii) Let U\subseteq X. The set U is called neighborhood of a in (X, d) if there is an \epsilon>0 such that B(a, \epsilon)\subseteq U.

\square

Probably the most prominent family of metric spaces is (\mathbb{R}^n, L_p), where L_p: V \times V \rightarrow \mathbb{R} is defined as

(1)   \begin{align*}      L_p(v,w) := \langle v, w\rangle_p := \left(\sum_{i=1}^n{|x_i - y_i|^p}\right)^{\frac{1}{p}}\end{align*}

with v, w\in \mathbb{R}^n and n\in \mathbb{N}. The case p=1 of the L_p metrics is the absolute value as outlined in Example 3.1. The corresponding normed spaces (V, l_p) can be defined as follows:

(2)   \begin{align*}l_p(v) := \ ||v||_p := \left(\sum{i=1}^n{|x_i|^p}\right)^{\frac{1}{p}}\end{align*}

l_p(v) is directly connected to the corresponding metric spaces L_p(v,w) via L_p(v,w)=\langle v, w \rangle =l_p(v-w)=||v-w||_p.

Illustration of the L_p norms and metrics

If (V, d) is a metric space, then (M, d) with M\subseteq V is also a metric space. The necessary properties are transmitted to the sub-space (M, d) simply by remembering the distances between two arbitrary points m, m'\in M \subseteq V. The metric (M, d) is also called metric induced by (V, d).

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.