L.

Vandenberghe EE133A (Fall 2014-15)

1. Vectors
notation
examples
vector operations
linear functions
complex vectors
operation counts
1-1
Vector
a vector is an ordered, nite list of numbers
we use two types of notation: vertical and horizontal arrays; for example
_

_
1.1
0.0
3.6
7.2
_

_
= (1.1, 0.0, 3.6, 7.2)
numbers in the list are the elements (components, entries, coecients)
number of elements is the length (size, dimension) of the vector
a vector with length n is called an n-vector
set of n-vectors with real elements is denoted R
n
Vectors 1-2
Conventions
we usually denote vectors by lowercase letters
a =
_

_
a
1
a
2
.
.
.
a
n
_

_
= (a
1
, a
2
, . . . , a
n
)
ith element of vector a is denoted a
i
i is the index of the ith element a
i
Note
several other conventions exist
well make exceptions, e.g., a
i
can refer to ith vector in a list of vectors
Vectors 1-3
Block vectors, subvectors
vectors can be stacked (concatenated) to create larger vectors
Example
stacking vectors b, c, d of length m, n, p gives an (m + n + p)-vector
a =
_
_
b
c
d
_
_
= (b
1
, . . . , b
m
, c
1
, . . . , c
n
, d
1
, . . . , d
p
)
other notation: a = (b, c, d)
b, c, d are blocks or subvectors of a
Vectors 1-4
Special vectors
Zero vector and vector of ones
0 = (0, 0, . . . , 0), 1 = (1, 1, . . . , 1)
length follows from context (if not, we add a subscript and write 0
n
, 1
n
)
Unit vectors
there are n unit vectors of length n, written e
1
, e
2
, . . . , e
n
ith unit vector is zero except its ith element which is 1; for n = 3,
e
1
=
_
_
1
0
0
_
_
, e
2
=
_
_
0
1
0
_
_
, e
3
=
_
_
0
0
1
_
_
length of e
i
follows from context (or should be specied explicitly)
Vectors 1-5
Outline
notation
examples
vector operations
linear functions
complex vectors
operation counts
Geometry
Position: coordinates of a point in a plane or three-dimensional space
x
1
x
2
x = (x
1
, x
2
)
Displacement: shown as arrow in plane or 3-D space
x
1
x
2
x = (x
1
, x
2
)
other quantities that have direction and magnitude, e.g., force vector
Vectors 1-6
Signal or time series
values of some quantity at dierent (and nitely many) times
x
k
is value at time k, or in period k
n-vector x can be displayed by plotting x
k
versus k
x
1
x
2
x
3
x
4
x
5
x
6
x
7
x
8
x
9
x
10
k
x can represent a sampled function f(t) of a continuous time variable t
x = (f(t
1
), f(t
2
), . . . , f(t
n
))
Vectors 1-7
Images
Monochrome (black-and-white) image
grayscale levels of M N pixels stored as MN-vector (e.g., columnwise)
Color image
three MN-vectors with R, G, B intensities
or concatenated as one vector of length 3MN
Video sequence
K frames of size M N as K vectors or length MN (if B&W)
or concatenated as one KMN-vector
Vectors 1-8
Feature vectors
collection of quantities or attributes that describe members of a set
Examples
age, weight, blood pressure, gender, . . . , of patients in a database
square footage, #bedrooms, list price, . . . , of houses in an inventory
Note
vector elements represent very dierent quantities, in dierent units
can contain categorical features (e.g., 0/1 for male/female)
ordering has no particular meaning
Vectors 1-9
Vectors of counts, histograms, occurrence vectors
Word counts
vector represents a document
length of vector is number of words in a dictionary
ith element is number of times word i occurs in document
Histogram
vector represents a histogram or distribution
ith element is the frequency in bin i
Occurrence, set membership
vector represents an object that can belong to n dierent sets
ith element is 1 if object is in set i; zero otherwise
Vectors 1-10
Probability
random event with n possible outcomes, numbered 1, 2, . . . , n
probability vector p has length n
ith element is probability of outcome i
elements of p are nonnegative and add up to one
Vectors 1-11
Economics and nance
Portfolio
vector represents portfolio of investments in n assets
ith element is amount invested in asset i, or #shares of asset i held
Cash ow
vector represents cash ow for n periods (e.g., quarters)
ith element represents payment to us (if positive), by us (if negative)
Resource vector
vector represents manufacturing process for a product
process requires n resources (energy, labor, material, . . . )
ith element is amount of resource i used
Vectors 1-12
Polynomials and generalized polynomials
a polynomial of degree n 1 or less
f(t) = c
1
+ c
2
t + c
2
t
2
+ + c
n
t
n1
can be represented by an n-vector (c
1
, c
2
, . . . , c
n
)
Extensions: for example, a cosine polynomial
f(t) = c
1
+ c
2
cos t + c
3
cos(2t) + + c
n
cos((n 1)t)
can be represented by an n-vector (c
1
, c
2
, . . . , c
n
)
Vectors 1-13
Summary
vectors are used in a wide variety of applications
can represent very dierent types of information
usefulness of vector representation for a particular application depends
on the relevance of vector operations for the application
Vectors 1-14
Outline
notation
examples
vector operations
linear functions
complex vectors
operation counts
Addition and subtraction
a + b =
_

_
a
1
+ b
1
a
2
+ b
2
.
.
.
a
n
+ b
n
_

_
, a b =
_

_
a
1
b
1
a
2
b
2
.
.
.
a
n
b
n
_

_
commutatitive
a + b = b + a
associative
a + (b + c) = (a + b) + c
a
a + b
b
Vectors 1-15
Scalar-vector and componentwise multiplication
Scalar-vector multiplication: for scalar and n-vector a,

_
a
1
a
2
.
.
.
a
n
_

_
=
_

_
a
1
a
2
.
.
.
a
n
_

_
Componentwise multiplication: for n-vectors a, b
a b =
_

_
a
1
b
1
a
2
b
2
.
.
.
a
n
b
n
_

_
(in MATLAB: a .* b)
Vectors 1-16
Linear combination
a linear combination of vectors a
1
, . . . , a
m
is a sum of scalar products

1
a
1
+
2
a
2
+ +
m
a
m
the scalars
1
, . . . ,
m
are the coecients of the linear combination
a
1
= (4, 0)
a
2
= (2, 2)
3
4
a
1
+
3
2
a
2
= (6, 3)
Vectors 1-17
Inner product
the inner product of two n-vectors a, b is dened as
a
T
b = a
1
b
1
+ a
2
b
2
+ + a
n
b
n
a scalar
meaning of superscript
T
will be explained when we discuss matrices
other notation: a, b, (a | b), . . .
(in MATLAB: a * b)
Vectors 1-18
Properties
for vectors a, b, c of equal length, scalar
a
T
a = a
2
1
+ a
2
2
+ + a
2
n
0
a
T
a = 0 only if a = 0
commutative:
a
T
b = b
T
a
associative with scalar multiplication:
(a)
T
b = (a
T
b)
distributive with vector addition:
(a + b)
T
c = a
T
c + b
T
c
Vectors 1-19
Simple examples
Inner product with unit vector
e
T
i
a = a
i
Dierencing
(e
i
e
j
)
T
a = a
i
a
j
Sum and average
1
T
a = a
1
+ a
2
+ + a
n
(
1
n
1)
T
a =
a
1
+ a
2
+ + a
n
n
Vectors 1-20
Applications
Weighted sum
f is vector of features; w is vector of nonnegative weights
w
T
f = w
1
f
1
+ w
2
f
2
+ + w
n
f
n
is total score
Cost
p is vector of prices of n goods; q is vector of quantities purchased
p
T
q = p
1
q
1
+ p
2
q
2
+ + p
n
q
n
is total cost
Expected value
p is vector of probabilities of n outcomes
f
i
is the value of a random variable in the event of outcome i
p
T
f is the expected value or average value of the random variable
Vectors 1-21
Discounted value
c is a cash ow over n 1 periods; d is vector of discount factors
d =
_
1,
1
1 + r
,
1
(1 + r)
2
, . . . ,
1
(1 + r)
n1
_
r > 0 is interest rate
d
T
c is net present value of cash ow
d
T
c = c
1
+
c
2
1 + r
+
c
3
(1 + r)
2
+ +
c
n
(1 + r)
n1
Co-occurrence
a, b are 0-1 occurrence vectors; represent membership of A, B in n sets
a
T
b is number of sets that contain both A and B
Vectors 1-22
Polynomial evaluation
c is vector of coecients of f(t) = c
1
+ c
2
t + c
3
t
2
+ + c
n
t
n1
x = (1, u, u
2
, . . . , u
n1
) is vector of powers of u
c
T
x is value of polynomial at u
c
T
x = c
1
+ c
2
u + c
3
u
2
+ + c
n
u
n1
= f(u)
Vectors 1-23
Outline
notation
examples
vector operations
linear functions
complex vectors
operation counts
Linear function
a function f : R
n
R is linear if superposition holds:
f(x + y) = f(x) + f(y) (1)
for all n-vectors x, y and all scalars ,
Extension: if f is linear, superposition holds for any linear combination:
f(
1
u
1
+
2
u
2
+ +
m
u
m
) =
1
f(u
1
) +
2
f(u
2
) + +
m
f(u
m
)
for all scalars
1
, . . . ,
m
and all n-vectors u
1
, . . . , u
m
(this follows by applying (1) repeatedly)
Vectors 1-24
Inner product function
for xed a R
n
, dene a function f : R
n
R as
f(x) = a
T
x = a
1
x
1
+ a
2
x
2
+ + a
n
x
n
any function of this type is linear:
a
T
(x + y) = (a
T
x) + (a
T
y)
for all scalars , and all n-vectors x, y
every linear function can be written as an inner-product function:
f(x) = f(x
1
e
1
+ x
2
e
2
+ + x
n
e
n
)
= x
1
f(e
1
) + x
2
f(e
2
) + + x
n
f(e
n
)
line 2 follows from superposition
Vectors 1-25
Examples in R
3
f(x) =
1
3
(x
1
+ x
2
+ x
3
) is linear: f(x) = a
T
x with a = (
1
3
,
1
3
,
1
3
)
f(x) = x
1
is linear: f(x) = a
T
x with a = (1, 0, 0)
f(x) = max{x
1
, x
2
, x
3
} is not linear: superposition does not hold for
x =
_
_
1
0
0
_
_
, y =
_
_
0
0
0
_
_
, = 1, = 1
we have f(x) = 1, f(y) = 0,
f(x + y) = 0 = f(x) + f(y) = 1
Vectors 1-26
Exercise
F(t)
x
1
x
2
x
3
x
4
x
5
x
6
x
7
x
8
x
9
x
10
t
F(t)
0 1 2 3 4 5 6 7 8 9 10
unit mass with zero initial position and velocity
subject to piecewise-constant force F(t) during interval [0, 10):
F(t) = x
j
for t [j 1, j), j = 1, . . . , 10
dene f(x) as position at t = 10, g(x) as velocity at t = 10
are f and g linear functions of x?
Vectors 1-27
Solution
from Newtons law s

(t) = F(t) where s(t) is the position at time t

integrate twice to get nal velocity and position
s

(10) =
_
10
0
F(t) dt
= x
1
+ x
2
+ + x
10
s(10) =
_
10
0
s

(t) dt
=
19
2
x
1
+
17
2
x
2
+
15
2
x
3
+ +
1
2
x
10
the two functions are linear: f(x) = a
T
x and g(x) = b
T
x with
a = (
19
2
,
17
2
, . . . ,
3
2
,
1
2
), b = (1, 1, . . . , 1)
Vectors 1-28
Ane function
a function f : R
n
R is ane if it satises
f(x + y) = f(x) + f(y)
for all n-vectors x, y and all scalars , with + = 1
Extension: if f is ane, then
f(
1
u
1
+
2
u
2
+ +
m
u
m
) =
1
f(u
1
) +
2
f(u
2
) + +
m
f(u
m
)
for all n-vectors u
1
, . . . , u
m
and all scalars
1
, . . . ,
m
with

1
+
2
+ +
m
= 1
Vectors 1-29
Ane functions and inner products
for xed a R
n
, b R, dene a function f : R
n
R by
f(x) = a
T
x + b = a
1
x
1
+ a
2
x
2
+ + a
n
x
n
+ b
i.e., an inner-product function plus a constant (oset)
any function of this type is ane: if + = 1 then
a
T
(x + y) + b = (a
T
x + b) + (a
T
x + b)
every ane function can be written as f(x) = a
T
x + b with:
a = (f(e
1
) f(0), f(e
2
) f(0), . . . , f(e
n
) f(0)) , b = f(0)
Vectors 1-30
Ane approximation
rst-order Taylor approximation of dierentiable f : R
n
R around z:

f(x) = f(z) +
f
x
1
(z)(x
1
z
1
) + +
f
x
n
(z)(x
n
z
n
)
generalizes rst-order Taylor approximation of function of one variable

f(x) = f(z) + f

(z)(x z)

f is a local ane approximation of f around z
in vector notation:

f(x) = f(z) + f(z)
T
(x z) where
f(z) =
_
f
x
1
(z),
f
x
2
(z), . . . ,
f
x
n
(z)
_
the n-vector f(z) is called the gradient of f at z
Vectors 1-31
Example
f(x
1
, x
2
) = x
1
3x
2
+ e
2x
1
+x
2
1
Gradient
f(x) =
_
1 + 2e
2x
1
+x
2
1
3 + e
2x
1
+x
2
1
_
First-order Taylor approximation around z = 0

f(x) = f(0) + f(0)

T
(x 0)
= e
1
+ (1 + 2e
1
)x
1
+ (3 + e
1
)x
2
Vectors 1-32
Regression model
y =
T
x + v
x is feature vector; elements x
i
are called regressors
is vector of weights
v is a scalar oset
y is (predicted) outcome or dependent variable
regression model expresses y as an ane function of x
model parameters are and v
Example
y is selling price of a house in some neighborhood
regressors (x
1
, x
2
, x
3
, x
4
) = (lot size, area, #bedrooms, #bathrooms)
Vectors 1-33
Outline
notation
examples
vector operations
linear functions
complex vectors
operation counts
Complex numbers
Complex number: x = + j with , real scalars
j =

1 (more common notation is i or j)

is the real part of x, denoted Re x
is the imaginary part, denoted Imx
set of complex numbers is denoted C
Modulus and conjugate
modulus (absolute value, magnitude): |x| =
_
(Re x)
2
+ (Imx)
2
conjugate: x = Re x j Imx
useful formulas:
Re x =
x + x
2
, Imx =
x x
2j
, |x|
2
= xx
Vectors 1-34
Polar representation
nonzero complex x = Re x + j Imx can be written as
x = |x| (cos + j sin ) = |x|e
j
[0, 2) is the argument (phase angle) of x (notation: arg x)
e
j
is complex exponential: e
j
= cos + j sin
real axis
imaginary axis
x
|x|
Re x
Imx
arg x
Vectors 1-35
Complex vector
vector with complex elements: a = + j with , real vectors
real and imaginary part, conjugate are dened componentwise:
Re a = (Re a
1
, Re a
2
, . . . , Re a
n
)
Ima = (Ima
1
, Ima
2
, . . . , Ima
n
)
a = Re a j Ima
set of complex n-vectors is denoted C
n
addition, scalar/componentwise multiplication dened as in R
n
:
a + b =
_

_
a
1
+ b
1
a
2
+ b
2
.
.
.
a
n
+ b
n
_

_
, a =
_

_
a
1
a
2
.
.
.
a
n
_

_
, a b =
_

_
a
1
b
1
a
2
b
2
.
.
.
a
n
b
n
_

_
Vectors 1-36
Complex inner product
the inner product of complex n-vectors a, b is dened as
b
H
a =

b
1
a
1
+

b
2
a
2
+ +

b
n
a
n
a complex scalar
meaning of superscript
H
will be explained when we discuss matrices
other notation: a, b, (a | b), . . .
for real vectors, reduces to real inner product b
T
a
(in MATLAB: b * a )
Vectors 1-37
Properties
for complex n-vectors a, b, c, complex scalars
a
H
a 0: follows from
a
H
a = a
1
a
1
+ a
2
a
2
+ + a
n
a
n
= |a
1
|
2
+ |a
2
|
2
+ + |a
n
|
2
a
H
a = 0 only if a = 0
b
H
a = a
H
b
b
H
(a) = (b
H
a)
(b)
H
a = (b
H
a)
(b + c)
H
a = b
H
a + c
H
a
b
H
(a + c) = b
H
a + b
H
c
Vectors 1-38
Example: power in electric networks
v(t) is voltage across circuit element at time t
(t) is current through element
+
v(t)

(t)
p(t) = v(t)(t) is instantaneous power absorbed by element at time t
for n elements, with voltages v
k
(t),
k
(t), k = 1, . . . , n, total power is
p(t) =
n

k=1
v
k
(t)
k
(t),
the (real) inner product of two n-vectors of voltages and currents
Vectors 1-39
Sinusoidal voltage and current
assume voltage and current are sinusoids with the same frequency
v(t) = V cos(t + ), (t) = I cos(t + ) (with V, I 0)
can be represented by complex numbers (phasors)
v =
V

2
e
j
, i =
I

2
e
j
instantaneous power
p(t) = V I cos(t + ) cos(t + )
=
V I
2
(cos( )(1 + cos 2(t + )) + sin( ) sin 2(t + ))
= Re(

iv) (1 + cos 2(t + )) + Im(

iv) sin 2(t + )

Vectors 1-40
Complex power
p(t) = Re(

iv) (1 + cos 2(t + ))

. .
average Re(

iv)
+Im(

iv) sin 2(t + )

. .
average zero
P = Re(iv) is called average (or real, active) power
Q = Im(iv) is reactive power
P + jQ = iv is complex power
for n elements: n-vectors of phasors v = (v
1
, . . . , v
n
) and i = (i
1
, . . . , i
n
)
i
H
v = Re(i
H
v) + j Im(i
H
v)
Re(i
H
v) is total average power
Im(i
H
v) is sum of reactive powers
i
H
v is sum of complex powers
Vectors 1-41
Outline
notation
examples
vector operations
linear functions
complex vectors
operation counts
Floating-point operation
Floating-point operation (op)
the unit of complexity when comparing vector and matrix algorithms
1 op = 1 basic arithmetic operation (+, ,

, . . . ) in R or C
Comments: this is a very simplied model of complexity of algorithms
we dont distinguish between the dierent operations
we dont distinguish between real and complex arithmetic
we ignore integer operations (indexing, loop counters, . . . )
we ignore cost of memory access
Vectors 1-42
Operation count
Operation count (op count)
total number of operations in an algorithm
in linear algebra, typically a polynomial of the dimensions in the problem
a crude predictor of run time of the algorithm:
run time
number of operations (ops)
computer speed (ops per second)
Dominant term: the highest-order term in the op count
1
3
n
3
+ 100n
2
+ 10n + 5
1
3
n
3
Order: the power in the dominant term
1
3
n
3
+ 10n
2
+ 100 = order n
3
Vectors 1-43
Examples
op counts of vector operations in this lecture (for vectors of length n)
addition, subtraction: n ops
scalar multiplication: n ops
componentwise multiplication: n ops
inner product: 2n 1 2n ops
these operations are all order n
Vectors 1-44