Intro to PySpark

Jason White, Shopify

What is PySpark?

Python interface to Apache Spark

Map/Reduce style distributed computing

Natively Scala

Interfaces to Python, R are well-maintained

Uses Py4J for Java <-> Scala interface

PySpark Basics

Distributed Computing basic premise:

Data is big

Program to process data is relatively small

Send program to where the data lives

Leave data on multiple nodes, scale horizontally

PySpark Basics

Driver: e.g. my laptop

Cluster Manager: YARN, Mesos, etc

Workers: Containers spun up by Cluster Manager

PySpark Basics

RDD: Resilient Distributed Dataset

Interface to parallelized data in the cluster

Map, Filter, Reduce functions sent from driver,

executed by workers on chunks of data in
parallel

PySpark: Hello World

Classic Word Count problem

How many times does each word appear in a

given text?

Approach: Each worker computes word counts

independently, then results aggregated together

PySpark: Hello World

Map

The
brown

dog

brown
dog

(The, 1)
(brown, 1)

(dog, 1)

(brown, 1)
(dog, 1)

Shuffle
Reduce

Collect

(The, 1)
(dog, 2)

(brown, 2)

(The, 1)
(dog, 2)
(brown, 2)

Demo
# example 1
text = "the brown dog jumped over the other brown dog"
text_rdd = sc.parallelize(text.split(' '))
text_rdd.map(lambda word: (word, 1)) \
.reduceByKey(lambda left, right: left + right).collect()
# example 2
import string
time_machine = sc.textFile('/user/jasonwhite/time_machine')
time_machine_tuples = time_machine.flatMap(lambda line: line.lower().split(' ')) \
.map(lambda word: ''.join(ch for ch in word if ch in string.letters)) \
.filter(lambda word: word != '') \
.map(lambda word: (word, 1))
word_counts = time_machine_tuples.reduceByKey(lambda left, right: left + right)

Monoids

Monoids are combinations of:

set of data; and

associative, commutative functions

Very efficient in M/R, strongly preferred

Examples:

addition of integers

min/max of records by timestamp

Demo
# example 3
dataset = sc.parallelize([
{'id': 1, 'value': 1},
{'id': 2, 'value': 2},
{'id': 2, 'value': 6}
])
def add_tuples(left, right):
left_sum, left_count = left
right_sum, right_count = right
return (left_sum + right_sum, left_count + right_count)
averages = dataset.map(lambda d: (d['value'], 1)) \
.reduce(add_tuples)
averages_by_key = dataset.map(lambda d: (d['id'], (d['value'], 1))) \
.reduceByKey(add_tuples) \
.map(lambda (key, (sum, count)): (key, sum * 1.0 / count))

# example 4
from datetime import date
dataset = sc.parallelize([
{'id': 1, 'group_id': 10,
{'id': 2, 'group_id': 10,
{'id': 3, 'group_id': 10,
{'id': 4, 'group_id': 11,
{'id': 5, 'group_id': 11,
])

Demo
'timestamp':
'timestamp':
'timestamp':
'timestamp':
'timestamp':

date(1978,
date(1984,
date(1986,
date(1956,
date(1953,

3,
3,
5,
6,
2,

2)},
24)},
19)},
5)},
21)},

def calculate_age(d):
d['age'] = (date.today() - d['timestamp']).days()
return d
def calculate_group_stats(left, right):
earliest = min(left['earliest'], right['earliest'])
latest = max(left['latest'], right['latest'])
total_age = left['total_age'] + right['total_age']
count = left['count'] + right['count']
return {
'earliest': earliest,
'latest': latest,
'total_age': total_age,
'count': left['count'] + right[count']
}
group_stats = dataset.map(calculate_age) \
.map(lambda d: (d['group_id'], {'earliest': d['timestamp'], 'latest':
d['timestamp'], 'total_age': d['age'], 'count': 1})) \
.reduceByKey(calculate_group_stats)

Joining RDDs

Like many RDD operations, works on (k, v) pairs

Each side shuffled using common keys

Each node builds its part of the joined dataset

Joining RDDs
{id: 1, field1: foo}
{id: 2, field1: bar}

{id: 1, field2: baz}

{id: 2, field2: baz}

(1, {id: 1, field1: foo})

(2, {id: 2, field1: bar})

(1, {id: 1, field2: baz})

(2, {id: 2, field2: baz})

(1, ({id: 1, field1: foo},

{id: 1, field2: baz}))

(2, ({id: 2, field1: bar},

{id: 2, field2: baz}))

Demo
# example 4
first_dataset = sc.parallelize([
{'id': 1, 'field1': 'foo'},
{'id': 2, 'field1': 'bar'},
{'id': 2, 'field1': 'baz'},
{'id': 3, 'field1': 'foo'}
])
first_dataset = first_dataset.map(lambda d: (d['id'], d))
second_dataset = sc.parallelize([
{'id': 1, 'field2': 'abc'},
{'id': 2, 'field2': 'def'}
])
second_dataset = second_dataset.map(lambda d: (d['id'], d))
output = first_dataset.join(second_dataset)

Key Skew

Achilles heel of M/R: key skew

Shuffle phase distributes like keys to like nodes

If billions of rows are shuffled to the same node,

may cause slight memory issues

Joining RDDs w/ Skew

When joining to small RDD, an alternative is to

broadcast the RDD

Instead of shuffling, entire RDD is sent to each

worker

Now each worker has all data needed

Each join is now just a map. No shuffle needed!

Demo
# example 5
first_dataset = sc.parallelize([
{'id': 1, 'field1': 'foo'},
{'id': 2, 'field1': 'bar'},
{'id': 2, 'field1': 'baz'},
{'id': 3, 'field1': 'foo'}
])
first_dataset = first_dataset.map(lambda d: (d['id'], d))
second_dataset = sc.parallelize([
{'id': 1, 'field2': 'abc'},
{'id': 2, 'field2': 'def'}
])
second_dataset = second_dataset.map(lambda d: (d['id'], d))
second_dict = sc.broadcast(second_dataset.collectAsMap())
def join_records((key, record)):
if key in second_dict.value.keys():
yield (key, (record, second_dict.value[key]))
output = first_dataset.flatMap(join_records)

Ordering

Row order isnt guaranteed unless you explicitly

sort the RDD

But: sometimes you need to process events in

order!

Solution: repartitionAndSortWithinPartitions

Ordering
{id: 1, value: 10}
{id: 2, value: 10}
{id: 3, value: 20}

{id: 1, value: 12}

{id: 1, value: 5}
{id: 2, value: 15}

Shuffle & Sort

{id: 1, value: 5}
{id: 1, value: 10}
{id: 1, value: 12}
{id: 3, value: 20

{id: 2, value: 10}

{id: 2, value: 15}

MapPartitions
{id: 1, interval: 5}
{id: 1, interval: 2}

{id: 2, interval: 5}

Thanks!