Linear algebra is the branch of mathematics concerning linear equations, and linear maps, and their representations in vector spaces and through matrices.
Includes matrix math.
Linear algebra is central to geometry and to almost all areas of mathematics.
Linear algebra is also used in most sciences and fields of engineering, because it allows modeling many natural phenomena, and computing efficiently with such models.
linear - output is proportional to the input
nonlinear - output is NOT proportional to the input
chaotic - special case of nonlinear, where the model is sensitive to initial conditions
We have already learned equations, curves, Cartesian coordinates, polar coordinates. Now we learn three different notations to represent the same ideas.
These new notations allow us to deal with multiple dimensions.
vector: magnitude & direction
example vector: $$\begin{bmatrix} 4\\ 3 \end{bmatrix}$$
tuple - ordered list of numbers. The vector above is a 2-tuple.
${R}^2$ - real coordinate space, all possible vectors of real numbers in two dimensions
The vector can also be drawn as an arrow on a graph. Addition of two vectors, or multiplication by a scalar, can be shown as matrix math and as arrows on a graph.
Start with vector V as (2,3) or as: $$\vec{V} = \begin{bmatrix} 2\\ 3 \end{bmatrix}$$
Unit vector i moves 1 unit in the horizontal direction. $$\hat{i} = \begin{bmatrix} 1\\ 0 \end{bmatrix}$$
Unit vector j moves 1 unit in the vertical direction. $$\vec{j} = \begin{bmatrix} 0\\ 1 \end{bmatrix}$$
Now vector V can also be represented as: $$\vec{V} = 2 \hat{i} + 3 \hat{j}$$
Add two vectors using unit vector notation like this: \begin{align} \vec{b} &= -1\hat{i} + 4\hat{j} \\ \vec{V} + \vec{b} &= (2-1)\hat{i} + (3+4)\hat{j} = \hat{i} + 7 \hat{j} \end{align}
Viola. Straight algebra.
A parametric representation of a geometric object.
For example, this pair of equations forms a parametric representation of a circle. $t$ is the parameter.
$$ \begin{align*} x &= \cos t\\ y &= \sin t \end{align*} $$ A set of colinear vectors: $$ \begin{align*} \vec{V} &= \begin{bmatrix}2\\1\end{bmatrix} \\S &= \begin{Bmatrix}c \vec{V} \mid c \in \mathbb{R} \end{Bmatrix} \end{align*} $$ Given the vector $\vec{V}$, the set $S$ is the set of colinear vectors :
the constant $c$ times the vector $\vec{V}$ where $c$ is in the set of real numbers.
Multiple ways to express a 2D line:
\begin{align} y &= 3x + 2 && \text{with an algebraic equation} \\ \vec{a} &= \begin{bmatrix} 3\\ 1 \end{bmatrix} \vec{b} = \begin{bmatrix} 2\\ 0 \end{bmatrix} && \text{with vectors} \\L &= \begin{Bmatrix} \vec{x} + t \vec{V} \mid t \in \mathbb{R} \end{Bmatrix} && \text{and set notation} \end{align}
Define a 3D line, four different ways
\begin{align} x + y + z = k && \text{Nope. That is not a line. It is a plane.} \end{align}
It is not possible to define a 3D line with an algebraic equation.
For more than two dimensions, we must use vector and set notation and parameterized equations.
\begin{align} \vec{P}_{1} &= \begin{bmatrix} -1\\2\\7 \end{bmatrix} \vec{P}_{2} = \begin{bmatrix} 0\\3\\4 \end{bmatrix} && \text{for two vectors } \vec{P}_{1} \text{ and } \vec{P}_{2} \\L &= \begin{Bmatrix} \vec{P}_{1} + t (\vec{P}_{1} - \vec{P}_{2}) \mid t \in \mathbb{R} \end{Bmatrix} && L \text{ defines the line} \\L &= \begin{Bmatrix} \begin{bmatrix}-1\\2\\7\end{bmatrix} + t \begin{bmatrix}-1\\-1\\3\end{bmatrix} \mid t \in \mathbb{R} \end{Bmatrix} && \text{substitution and subtraction} \end{align}
\begin{align} x &= -1 + -1t \\y &= 2 + -1t \\z &= 7 + 3t \end{align}
Draw the line using Python and the parameterized equations.
import numpy as np import matplotlib.pyplot as plt from mpl_toolkits.mplot3d import Axes3D t = np.linspace(-10, 10, 100) # create parameter t x = -1 + -1*t # calc vectors x,y,z y = 2 + -1*t z = 7 + 3*t fig = plt.figure() ax = fig.add_subplot(projection='3d') ax.plot(x,y,z) # plot vectors x,y,z plt.show()
Multiply each vector by an arbitrary scalar and sum the resulting products.
\begin{align} c_{1} \vec{V}_{1} + c_{2} \vec{V}_{2} + \cdots + c_{n} \vec{V}_{n} \end{align}
\begin{align} \text{span } (\vec{a}, \vec{b}) \end{align}
$$\vec{0} = \begin{bmatrix}0\\0\end{bmatrix}$$
Given a set of vectors, a vector is dependent if it can be represented as a combination of the others. If not, then the vector is linearly independent.
If a vector is dependent it is redundant. In many data analysis problems, our first task will be to eliminate all dependent vectors. Our working set of data should include only independent vectors. Therefore we need ways to identify and separate the independent vs dependent vectors.
For example, consider these three vectors. $$ $$
The first two vectors define a 2D line. The third vector is on that line. It adds no additional information. It is redundant. It is dependent. Before doing any work, we want to eliminate dependent vectors.
The
One of the ways is to use the this rule.
Given two vectors, if one is dependent, then the other is also dependent?
Trick.
if both c_{1} and c_{2} are zero, then they are independent. If either c_{1} or c_{2} is non-zero, then they are dependent.
c_{1} + c_{2} = 0 \Rightarrow \text{independent}
if vectors are independent, then their span is the whole of real number set.
if vectors are dependent, then their span is a line.
c_{1} + c_{2} are independent, \Rightarrow \text{span}(\vec{a}, \vec{b}) = \mathbb{R}
Linear Algebra = doing multiply and divide operations on vectors. Specifically, scalar multiplication and tip-to-tail addition of vectors.
Vector Space = the set of all vectors.
Vector Subspace = a place where we can do linear algebra.
That is, where we can safely add and multiply vectors,
such that the resulting vectors are still in the same subspace.
$S$ is a subset of the vector space.
Is $S$ also a subspace of the vector space?
The answer is yes if three requirements have been met:
1. The zero vector must be included in S.
2. Assuming vectors x and y are in the subspace and scalar c is a Real number, if we multiple c times x, the resulting vector must still be within the S subspace. That is called “closure under scalar multiplication.”
3. Likewise, if we add a + b, the resulting vector must be in subspace S. That is called “closure under addition”.
Given:
\begin{align} S &\in \mathbb{R}\\ c &\in \mathbb{R}\\ \vec{x}, \vec{y} &\in S\\ \end{align}
S is a subspace of a Vector Space if the following three conditions are met
\begin{align} \vec{0} &\in S \\ c \vec{x} &\in S \\ \vec{x} + \vec{y} &\in S \\ \end{align}
The span of any set of vectors is a subspace of $\mathbb{R}^n$.
The basis of a subspace is the minimum set of vectors that spans the subspace.
By “minimum” set, we mean the vectors in this set must be linearly independent. No redundant vectors.
\begin{align} \vec{a} \cdot \vec{b} = \begin{bmatrix} a_1 \\ a_2 \\ \vdots \\ a_n \end{bmatrix} \cdot \begin{bmatrix} b_1 \\ b_2 \\ \vdots \\ b_n \end{bmatrix} = a_1 b_1 + a_2 b_2 + \cdots + a_n b_n \end{align}
The dot product equals a single number, a scalar.
The distributive, associative and commutative properties apply to dot products.
The length of a vector, taken from the Pythagorean theorem, is a kind of multidimensional hypotenuse.
\begin{align} \left \| \vec{a} \right \| = \sqrt{a^1+a^2+\cdots+a^n} \end{align}
The length of a vector can also be expressed as the square root of the dot product of a vector times itself.
\begin{align} \left \| \vec{a} \right \| &= \sqrt{\vec{a} \cdot \vec{a}} \\ {\left \| \vec{a} \right \|}^2 &= \vec{a} \cdot \vec{a} \end{align}
Given two non-zero vectors,
compare the dot product vs. the product of the lengths.
The dot product will be less than the product of the lengths, except
when one vector is a scalar of the other.
\begin{align} \vec{x}, \vec{y} \in \mathbb{R} \text{ and } \vec{x}, \vec{y} \ne \vec{0} \\ \left | \vec{x} \cdot \vec{y} \right | \le \left \| \vec{x} \right \| \left \| \vec{y} \right \| \\ \vec{x} = c\vec{y} \iff \left | \vec{x} \cdot \vec{y} \right | = \left \| \vec{x} \right \| \left \| \vec{y} \right \| \\ \end{align}
We started with Length of a vector above.
Now we define the angle between two vectors.
from Trigonometry, Triangle Inequality, the length any two sides is greater than or equal to the length of the remaining side. $$c \le a + b$$
Now we construct a triangle of vectors such that the length of the third side is equal to the difference between the lengths of the first two sides.
\begin{align} (\vec{a} - \vec{b}) = \left\|\vec{a}\right\| + \left\|\vec{b}\right\| \end{align}
from geometry:
law of cosines: $c^2 = a^2 + b^2 - 2ab\cos\theta$
The dot product of vectors $\vec{a}$ and $\vec{b}$ is equal to the length of $\vec{a}$ times the length of $\vec{b}$ times the cosine of theta.
\begin{align} (\vec{a} \cdot \vec{b}) = \left \| \vec{a} \right \| \left \| \vec{b} \right \| \cos \theta\\ \end{align}
Essence of Linear Algebra, 3blue1brown, playlist of 15 youtubes
Linear Algebra, Khan Academy, Salmon Khan, video course
Dr. Trefor Bazett explains subspaces https://youtu.be/rgxyxcTwvuo