DSC 40A Lecture 6

Source: https://dsc40a.com/resources/lectures/lec06/lec06-filled.pdf

Dot Products and Projections

Recap: Friends of the Simple Linear Regression Model

Dot products

Vectors

• A vector in is an ordered collection of numbers.
• We use lower-case letters with an arrow on top to represent vectors, and we usually write vectors as columns.
  
• Another way of writing the above vector is . (transpose)
• Since has four components (“elements”), we say .

The geometric interpretation of a vector

• A vector is an arrow to the point from the origin.
• The length, or norm, of is:
  
• A vector is sometimes described as an object with a magnitude/length and direction.

Dot product: coordinate definition

• The dot product of two vectors and in is written as:
  
• The computational definition of the dot product:
  
• The result is a scalar, i.e. a single number.

Dot product: geometric definition

• The computational definition of the dot product:
  
• The geometric definition of the dot product:
  
  where is the angle between and .
• The two definitions are equivalent. This equivalence allows us to find the angle between two vectors.

Orthogonal vectors

• Recall:
• Since , if the angle between two vectors is , their dog product is .
• If the angle between two vectors is , we say they are perpendicular, or more generally, orthogonal.
• Key idea:
  

---

Spans and projections

Adding and scaling vectors

• The sum of two vectors and in is the element-wise sum of their components:
  
• If is a scalar, then:
  

Linear combinations

• Let be all vectors in .
• A linear combination of is any vector of the form:
  
  where are all scalars.

Span

• Let all be vectors in .
• The span of is the set of all vectors that can be created using linear combinations of those vectors.
• Formal definition:
  

Projecting onto a single vector

• Let and be two vectors in .
• The span of is the set of all vectors of the form:
  
  where is a scalar.
• Question: What vector in is closest to ?
• The vector in that is closest to is the projection of onto .

Projection error

• Let be the projection error: that is, the vector that connects to .
• Goal: Find the that makes as short as possible.
• That is, minimize:
  
• Equivalently, minimize:
  
• Idea: To make as short as possible, it should be **orthogonal to .

Minimizing projection error

• Goal: Find the that makes as short as possible.
• Idea: To make as short as possible, it should be orthogonal to .
• Can we prove that making orthogonal to minimizes ?