Professional Documents
Culture Documents
Course Statistics
W
ee
k
W "1"
ee
k
W "2"
ee
k
W "3"
ee
k
W "4"
ee
k
W "5"
ee
k"6
Ex "
am
"
h#p://www.openHPI.de
13,126
par8cipants,
4.161
ac8ve
contributors
in
discussions
4000"
100"
6
course
modules
for
6
weeks
80"
3000"
60"
35
individual
module
items
(videos
/
2000"
40"
slides
/
reading
material)
1000"
20"
300
exam
/
assignment
/
self-test
0"
0"
ques8ons
309
slides
explicitly
created
for
this
course
215
pages
book
reading
material
Number"of"Par8cipants"in"Assignments"&"Exam"
8h
cut
video
material
(12h
raw
material)
Average"Results"in"%"
Teaching
Team:
4
colleagues
Goals
Deep
technical
understanding
of
a
column-oriented
dic8onary-
encoded
in-memory
database
and
its
applica8on
in
enterprise
compu8ng
Learning Map
The
Future
of
Enterprise
Compu(ng
In-Memory
Database
Operators
Advanced
Database
Storage
Tech-
niques
Founda(ons
of
Database
Storage
Techniques
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Chapter 1:
The Future of
Enterprise Computing
Sensors
Tracing
Pharmaceu(cal
Packages
in
Europe
15
billion
packages
/
34
billion
read
events
per
year
Distributed
repositories
for
storing
read
events
References
to
read
events
are
stored
in
central
discovery
service
Combination of Structured
and Unstructured Data
Airplane
Maintenance
at
Boeing
Maintenance
workers
at
Boeing
write
reports
aYer
repairs
Reports
and
men8oned
part
numbers
get
indexed
Analy8cs
reveal
which
parts
in
other
planes
may
be
defec8ve
Mobile
Mobile
inverts
tradi(onal
corporate
structures
Enables
customer-facing
Consumers
/
Customers
personnel
Sales,
Service
Opera8ons
Controlling
Example: Dunning
Consolida8on
Strategy
CEO
10
Learning Map
The
Future
of
Enterprise
Compu(ng
Founda(ons
of
Database
Storage
Techniques
In-Memory
Database
Operators
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Advanced
Database
Storage
Tech-
niques
Enterprise Application
Characteristics
Challenge:
Diverse Applications
Transac(onal
Data
Entry
Real-(me
Analy(cs,
Structured
Data
Sources:
Machines,
Transac8onal
Apps,
User
Interac8on,
etc.
Event
Processing,
Stream
Data
Data
Management
Sources:
machines,
sensors,
high
volume
systems
Text
Analy(cs,
Unstructured
Data
Sources:
web,
social,
logs,
support
systems,
etc.
13
14
Online
Analy8cal
Processing
Drawbacks of
the Separation
15
16
Vision
Combine
OLTP
and
OLAP
data
using
modern
hardware
and
database
systems
to
create
a
single
source
of
truth,
enable
real-(me
analy(cs
and
simplify
applica8ons
and
database
structures.
Addi8onally,
extrac8on,
transforma8on,
and
loading
(ETL)
processes
17
Enterprise Data
Characteristics
CPG
Banking
Logis8cs
Discrete
manufacturing
High tech
19
80%
% of Columns
60%
Inventory Management
Financial Accounting
78 %
70%
64 %
50%
40%
30%
20%
24 %
10%
0%
20
13 %
9%
33 - 1023
1024 - 100000000
Number of Distinct Values
12 %
1 - 32
Wide Tables
Analysis
of
width
of
144
most
used*
tables
30
# Tables
25
20
15
10
0
1-9
10-19
20-29
30-39
40-49
50-59
60-69
70-79
82
99
110-119
120-129
138
140-149
156
180-189
200-209
230
312
399
# Columns
21
Learning Map
The
Future
of
Enterprise
Compu(ng
In-Memory
Database
Operators
Advanced
Database
Storage
Tech-
niques
Founda(ons
of
Database
Storage
Techniques
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Changes in Hardware
Advances in Hardware
A
Mul8-Core
Architecture
8
x
(8-16)
core
CPU
per
blade
Cost-performance
ra8o
rapidly
declining
Memory Hierarchy
CPU
Registers
Lower Price /
Higher Latency
Higher
Performance
CPU
Caches
Main Memory
Flash
Hard Disk
25
Latency Numbers
L1
cache
reference
(cached
data
word)
Branch
mispredict
5ns
L2 cache reference
7ns
Mutex lock/unlock
25ns
100ns
0.1s
20,000ns
20s
150,000ns
150s
250,000ns
250s
10,000,000ns
10ms
150,000,000ns
150ms
Disk
seek
Send
packet
CA
to
Netherlands
to
CA
26
0.5ns
Learning Map
The
Future
of
Enterprise
Compu(ng
Founda(ons
of
Database
Storage
Techniques
In-Memory
Database
Operators
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Advanced
Database
Storage
Tech-
niques
A Blueprint of SanssouciDB
SanssouciDB: An In-Memory
Database for Enterprise Applications
Cache-conscious
algorithms/
data
structures
are
crucial
(locality
is
king)
TA Manager
Time
travel
Combined
Column
Column
Differential
Store
Main Store
Data
aging
Distribution Layer
Layer
Distribution
at Blade
Serveri i
at
Main Memory
at Blade
Serveri i
Active Data
Column
Metadata
Merge
Query Execution
Combined
Column
Column
Column
Indexes
Inverted
Object
Data Guide
Logging
Recovery
Log
Passive Data (History)
Non-Volatile
Memory
Snapshots
Chapter 2:
Foundations of
Database Storage Techniques
Learning Map
The
Future
of
Enterprise
Compu(ng
Founda(ons
of
Database
Storage
Techniques
In-Memory
Database
Operators
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Advanced
Database
Storage
Tech-
niques
Dictionary Encoding
Motivation
33
Dictionary Encoding
Example
8 billion humans
A#ributes:
rst
name
last
name
gender
country
city
birthday
200
byte
per
tuple
34
Sample Data
Table:
world_popula8on
recID
fname
lname
gender
city
country
birthday
39
John
Smith
Chicago
USA
12.03.1964
40
Mary
Brown
London
UK
12.05.1964
41
Jane
Doe
Palo Alto
USA
23.04.1976
42
John
Doe
Palo Alto
USA
17.06.1952
43
Peter
Schmidt
Potsdam
GER
11.11.1975
35
Dictionary Encoding
a Column
36
recID
fname
valueID
Value
posi(on
39
John
40
Mary
23
John
39
23
41
Jane
24
Mary
40
24
42
John
25
Jane
41
25
43
Peter
26
Peter
42
23
43
26
valueID
37
Sorted Dictionary
38
Compression Rate
Entropy
39
Cardi-
nality
Bits
Item
Size
Plain
Size
Size
with
Dic(onary
Compression
Needed
(Dic(onary
+
Column)
Factor
First
names
5 millions
23 bit
50 Byte
373GB
238.4MB + 21.4GB
17
Last
names
8 millions
23 bit
50 Byte
373GB
381.5MB + 21.4GB
17
1 bit
1 Byte
7GB
2.0b + 953.7MB
1 million
20 bit
50 Byte
373GB
47.7MB + 18.6GB
20
Country
200
8 bit
47 Byte
350GB
9.2KB + 7.5GB
47
Birthday
40000
16 bit
2 Byte
15GB
78.1KB + 14.9GB
200 Byte
1.6TB
92GB
17
Gender
City
Totals
40
Basics
...
0x0
42
Row
Row
Row Operation
A B C A B C A B C A B C
43
Row
Row
Row
Row
Column
Column
Row Operation
A A A A B B B B C C C C
44
Column
Column
Column
Chapter 3:
In-Memory Database Operators
Learning Map
The
Future
of
Enterprise
Compu(ng
In-Memory
Database
Operators
Advanced
Database
Storage
Tech-
niques
Founda(ons
of
Database
Storage
Techniques
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Insert
Example
Example
Table:
world_popula8on
recID
fname
lname
gen country
der
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg 04-04-1968
Anton
Meyer
AUT
Innsbruck 10-20-1992
Sophie
Schulze
GER
Potsdam 09-03-1977
...
...
...
...
...
...
...
INSERT Without
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
AV
0
0
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Sophie
Schulze
GER
Potsdam
09-03-1977
...
...
...
...
...
...
...
Albrecht
1 1
Berg
2 3
Meyer
3 2
Schulze
4 3
49
INSERT Without
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
AV
0
0
Albrecht
1 1
Berg
2 3
Meyer
3 2
Schulze
4 3
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Sophie
Schulze
GER
Potsdam
09-03-1977
...
...
...
...
...
...
...
1.
50
Dic8onary (D)
INSERT Without
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
AV
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
4 3
Sophie
Schulze
GER
Potsdam
09-03-1977
5 3
...
...
...
...
0 0
Albrecht
1 1
Berg
2 3
Meyer
3 2
Schulze
...
1.
2.
Schulze
...
...
Dic8onary (D)
51
INSERT With
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
AV
0
0
Berlin
1 0
Hamburg
2 1
Innsbruck
3 2
Potsdam
4 3
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Sophie
Schulze
GER
Potsdam
09-03-1977
...
...
...
...
5
...
Schulze
...
...
52
INSERT With
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
AV
0
0
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Sophie
Schulze
GER
Potsdam
09-03-1977
...
...
...
...
Berlin
1 0
Hamburg
2 1
Innsbruck
3 2
Potsdam
4 3
5
...
1.
Schulze
...
...
Dic8onary (D)
53
INSERT With
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
AV
0
0
Berlin
1 0
Hamburg
2 1
Innsbruck
3 2
Potsdam
4 3
Rostock
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Sophie
Schulze
GER
Potsdam
09-03-1977
...
...
...
...
5
...
1.
2.
54
Schulze
...
...
INSERT With
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
AV
0
0
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Sophie
Schulze
GER
Potsdam
09-03-1977
Berlin
1 0
Hamburg
2 1
Innsbruck
3 2
Potsdam
4 3
Rostock
5 4
5
...
1.
2.
3.
Schulze
...
...
Rostock
...
...
...
...
Dic8onary (D)
55
INSERT With
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
AV
0
2
Anton
1 3
Hanna
2 1
Mar8n
3 0
Michael
4 4
Sophie
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Sophie
Schulze
GER
Potsdam
09-03-1977
5
...
Schulze
...
...
Rostock
...
...
...
...
56
INSERT With
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
AV
0
2
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Sophie
Schulze
GER
Potsdam
09-03-1977
Anton
1 3
Hanna
2 1
Mar8n
3 0
Michael
4 4
Sophie
5
...
1.
Schulze
...
...
Rostock
...
...
...
...
Dic8onary (D)
57
INSERT With
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
AV
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Sophie
Sophie
Schulze
GER
Potsdam
09-03-1977
Karen
0 2
Anton
1 3
Hanna
2 1
Mar8n
3 0
Michael
4 4
4
5
...
1.
2.
58
Schulze
...
...
Rostock
...
...
...
...
INSERT With
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
(old)
AV
0
2
Anton
D
(new)
0
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Sophie
Schulze
GER
Potsdam
09-03-1977
Anton
1 3
Hanna
Hanna
2 1
Mar8n
Karen
3 0
Michael
Mar8n
4 4
Sophie
Michael
Karen
Sophie
5
...
1.
2.
3.
Schulze
...
...
Rostock
...
...
...
...
Dic8onary (D)
59
INSERT With
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
AV
(old)
AV (new)
D
(new)
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Michael
Sophie
Schulze
GER
Potsdam
09-03-1977
Sophie
0 2
0 3
Anton
1 3
1 4
Hanna
2 1
2 1
Karen
3 0
3 0
Mar8n
4 4
4 5
4
5
...
1.
2.
3.
4.
60
Schulze
...
...
Rostock
...
...
...
...
INSERT With
New Dictionary Entry
INSERT
INTO
world_popula8on
VALUES
(Karen,
Schulze,
f,
GER,
Rostock,
06-20-2012)
D
AV
0
3
1.
2.
3.
4.
5.
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
GER
Potsdam
09-03-1977
Anton
1 4
Hanna
2 1
Karen
3 0
Mar8n
4 5
Michael
Sophie
Schulze
5 2
Sophie
Karen
Schulze
...
...
...
Rostock
...
...
...
...
Dic8onary (D)
61
Result
Example
Table:
world_popula8on
recID
fname
lname
gender country
city
birthday
Mar8n Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg 04-04-1968
Anton
Meyer
AUT
Innsbruck 10-20-1992
Ulrike
Schulze
GER
Potsdam 09-03-1977
Karen
Schulze
GER
Rostock
06-20-2012
...
...
...
...
...
...
...
Learning Map
The
Future
of
Enterprise
Compu(ng
Founda(ons
of
Database
Storage
Techniques
In-Memory
Database
Operators
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Advanced
Database
Storage
Tech-
niques
Scan Performance
65
...
Row
9
8 x 10
66
Birthday
City
Birthday
City
Row
2
Table
size:
8
billion
tuples
200
bytes
per
tuple
1.6
TB
Row
3
...
Row
9
8 x 10
67
Table
size:
8
billion
tuples
200
bytes
per
tuple
1.6
TB
Scan
through
all
rows
with
2MB/ms/core
800
s
with
1
core
Birthday
City
Row
2
Row
3
...
Row
9
8 x 10
68
Data loaded
and used
Data loaded
but not used
8
billion
cache
accesses
64
byte
512
GB
Read
with
2MB/ms/core
256
s
with
1
core
Birthday
City
Row
2
Row
3
...
Row
9
8 x 10
69
Data
not loaded
Data loaded
and used
Data loaded
but not used
70
Country
A#ribute
vector
size
91
GB
Dic8onary
size
700
MB
Table size 92 GB
Compression
factor
17
Birthday
City
71
Data
not loaded
Country
Size
of
a#ribute
vector
Gender:
8
billion
tuples
1
bit
per
tuple
1
GB
Scan
through
column
with
2MB/ms/core
0.5
s
with
1
core
Birthday
City
Data loaded
and used
Data loaded
but not used
72
Data
not loaded
Country
Size
of
a#ribute
vector
Birthday:
8
billion
tuples
2
byte
per
tuple
16
GB
Scan
through
column
with
2MB/ms/core
8
s
with
1
core
Birthday
City
Data loaded
and used
Data loaded
but not used
Scan Performance
Summary
How
many
women,
how
many
men?
Row Store
Column Store
Time in seconds
800
256
0.5
Factors compared
to best in class
1,600
512
73
Learning Map
The
Future
of
Enterprise
Compu(ng
In-Memory
Database
Operators
Advanced
Database
Storage
Tech-
niques
Founda(ons
of
Database
Storage
Techniques
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Tuple Reconstruction
76
All
a#ributes
are
stored
consecu8vely
Read
with
2MB/ms/core
0.128
s
with
1
core
77
78
6
cache
accesses
64
byte
384
byte
Read
with
2MB/ms/core
0.192
s
with
1
core
Learning Map
The
Future
of
Enterprise
Compu(ng
Founda(ons
of
Database
Storage
Techniques
In-Memory
Database
Operators
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Advanced
Database
Storage
Tech-
niques
Select
SELECT Example
SELECT fname, lname FROM world_population
WHERE country=Italy and gender=m
id
fname
country
gender
2394
Gianluigi Buon
lname
Italy
3010
Lena
Gercke
Germany
3040
Mario
Balotelli
Italy
3949
Manuel
Neuer
Germany
4902
Lukas
Podolski
Germany
20102
Klaas-Jan Huntelaar
Netherlands
81
Query Plan
country = "Italy"
gender = "m"
position
list
position
list
positional
AND
82
fname,
lname
Query Execution
ValueID
Dic(onary
for
country
id
fname
lname
country
gender
2394
Gianluigi
Buon
3010
Lena
Gercke
Algeria
France
3040
Mario
Balotelli
Germany
3949
Manuel
Neuer
Italy
4902
Lukas
Podolski
20102
Klaas-Jan Huntelaar
Netherlands
country = 3 (Italy)
ValueI
D
Dic(onary
for
gender
gender
=
1
(m)
Posi(on
0
2
Posi(on
AND
Posi(on
0
2
83
Learning Map
The
Future
of
Enterprise
Compu(ng
In-Memory
Database
Operators
Advanced
Database
Storage
Tech-
niques
Founda(ons
of
Database
Storage
Techniques
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Chapter 4:
Advanced Database
Storage Techniques
Differential Buffer
Motivation
87
Differential Buffer
Read
Write
world_population
Table
A#ribute Vector
4
5
0
1
2
3
Anton
Hanna
Michael
Sophie
Differential
1
3
2
8
Billion
entries
88
fname
Dic8onary
Main Store
A#ribute Vector
Main
0
Store
1
Table
2
3
Read Operations
fname
Data Modifying
Operations
(compressed)
Dic8onary
0
1
1
Angela
Klaus
Andre
Buffer
up
to
50,000
entries
Dieren8al Buer
CSB+
Differential Buffer
89
Tuple Lifetime
Michael
moves
from
Berlin
to
Potsdam
Table:
world_popula8on
recId
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Ulrike
Schulze
GER
Potsdam
09-03-1977
Sophie
Schulze
GER
Rostock
06-20-2012
...
...
...
...
...
...
...
8 109
Zacharias
Perdopolus
GRE
Athen
03-12-1979
Main Store
Dieren8al
Buer
UPDATE
world_popula*on
SET
city=Potsdam
WHERE
fname=
Michael
AND
lname=Berg
90
Tuple Lifetime
Michael
moves
from
Berlin
to
Potsdam
Table:
world_popula8on
recId
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Ulrike
Schulze
GER
Potsdam
09-03-1977
Sophie
Schulze
GER
Rostock
06-20-2012
...
...
...
...
...
...
...
8 109
Zacharias
Perdopolus
GRE
Athen
03-12-1979
Main Store
Dieren8al
Buer
UPDATE
world_popula*on
SET
city=Potsdam
WHERE
fname=
Michael
AND
lname=Berg
91
Tuple Lifetime
Michael
moves
from
Berlin
to
Potsdam
Table:
world_popula8on
recId
fname
lname
gender
country
city
birthday
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Ulrike
Schulze
GER
Potsdam
09-03-1977
Sophie
Schulze
GER
Rostock
06-20-2012
...
...
...
...
...
...
...
8 109
Zacharias
Perdopolus
GRE
Athen
03-12-1979
Michael
Berg
GER
Potsdam
03-05-1970
UPDATE
world_popula*on
SET
city=Potsdam
WHERE
fname=
Michael
AND
lname=Berg
92
Main Store
Dieren8al Buer
Tuple Lifetime
93
Tuple Lifetime
Michael
moves
from
Berlin
to
Potsdam
Table:
world_popula8on
recId
fname
lname
gender
country
city
birthday
valid
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Ulrike
Schulze
GER
Potsdam
09-03-1977
Sophie
Schulze
GER
Rostock
06-20-2012
...
...
...
...
...
...
...
8 109
Zacharias
Perdopolus
GRE
Athen
03-12-1979
Michael
Berg
GER
Potsdam
03-05-1970
UPDATE
world_popula*on
SET
city=Potsdam
WHERE
fname=
Michael
AND
lname=Berg
94
Main Store
Dieren8al
Buer
Tuple Lifetime
Michael
moves
from
Berlin
to
Potsdam
Table:
world_popula8on
recId
fname
lname
gender
country
city
birthday
valid
Mar8n
Albrecht
GER
Berlin
08-05-1955
Michael
Berg
GER
Berlin
03-05-1970
Hanna
Schulze
GER
Hamburg
04-04-1968
Anton
Meyer
AUT
Innsbruck
10-20-1992
Ulrike
Schulze
GER
Potsdam
09-03-1977
Sophie
Schulze
GER
Rostock
06-20-2012
...
...
...
...
...
...
...
8 109
Zacharias
Perdopolus
GRE
Athen
03-12-1979
Michael
Berg
GER
Potsdam
03-05-1970
UPDATE
world_popula*on
SET
city=Potsdam
WHERE
fname=
Michael
AND
lname=Berg
Main Store
Dieren8al
Buer
95
Learning Map
The
Future
of
Enterprise
Compu(ng
In-Memory
Database
Operators
Advanced
Database
Storage
Tech-
niques
Founda(ons
of
Database
Storage
Techniques
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Merge
Handling Write
Operations
98
Merge Process
Overview
Consists
of:
Prepare
merge
A#ribute
merge
Commit
merge
1.
2.
3.
Is triggered by:
99
Online Merge
Working
on
data
copies
allows
asynchronous
merge
Very
limited
interrup8on
due
to
short
lock
At
least
twice
the
memory
needed!
Before
Table
Main
Store
Merge Operation
Main
Store
Read Operations
100
Table
Differential
Buffer
Main
Store
(new)
Read Operations
Prepare
Data
Modifying
Operations
Data
Modifying
Operations
Table
Differential
Buffer
After
Data
Modifying
Operations
Attribute Merge
Differential
Buffer
(new)
Main
Store
(new)
Differential
Buffer
(new)
Read Operations
Commit
Attribute Merge
Step
1:
Extract
delta
dic8onary
from
delta
column
Merge
main
and
delta
dic8onary,
op8onally
remove
unused
values
Create
mapping
if
valueID
changed
Step
2:
Create
new
main
column
(if
needed)
Main
store
column
a#ribute
vector
needs
to
be
rewri#en
if
101
Example
Inputs Step 1
Outputs Step 1
Inputs Step 2
Output Step 2
AVm
Dm
Dc
AUX m
AVm
AVc
valueID
Dictionary Entries
Dictionary Entries
new valueID
valueID
valueID
anna
charlie
daniel
frank
hans
isabell
bernd
bernd
hans
frank
102
B1
D1
anna
bernd
charlie
daniel
frank
5
6
0
1
2
3
4
5
hans
isabell
10
11
AUX d
Delta Position
CSB+ Leafs
Lists
bernd
0
1
frank
hans
3
2
0
2
3
4
5
6
new valueID
old valueID
AVd
uncompressed
M1
old valueID
new valueID
old valueID
0
1
2
1
4
5
UM1
AVds
valueID
10
11
UM2
Learning Map
The
Future
of
Enterprise
Compu(ng
Founda(ons
of
Database
Storage
Techniques
In-Memory
Database
Operators
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Advanced
Database
Storage
Tech-
niques
105
106
Snapshots
of
primary
data
stores
are
wri#en
periodically
(e.g.
during
merge
process)
Log
les
for
updates
are
wri#en
between
snapshots
to
recover
the
database
Dic8onary-compressed
column
logging
allows
parallel
log
le
wri8ng
and
recovery
Snapshots
107
Chapter 5:
Foundations for a New
Enterprise Application
Development Era
Learning Map
The
Future
of
Enterprise
Compu(ng
Founda(ons
of
Database
Storage
Techniques
In-Memory
Database
Operators
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Advanced
Database
Storage
Tech-
niques
Implications on Application
Development
Simplified Application
Development
Tradi8onal
Column-oriented
Applica8on
cache
No caches needed
No
redundant
data/
objects
No
maintenance
of
indexes
or
aggregates
Database
cache
Prebuilt
aggregates
Raw data
111
Aggregate on-the-y
112
Current Financials
Solutions
113
The Target
Financials Solution
Only
base
tables,
algorithms,
and
some
indexes
114
In-Memory Financials on
SAP ERP
accounting documents
BKPF
secondary indices
BSEG
BSAD
BSAK
BSAS
BSID
dunning data
MHNK
MHND
CDHDR
CDPOS
BSIS
115
In-Memory Financials on
SAP ERP (prototype)
BKPF
BSEG
116
GLT0
change documents
BSIK
accounting documents
sum tables
LFC1
KNC1
Feasibility of Financials on
In-Memory Technology
Modica8ons
on
SAP
Financials
Financials:
(including inverted indices for join a#ributes and insert only extension)
117
Reduction by a Factor 10
118
DBMS
IMDB
BKPF
8.7 GB
1.5 GB
BSEG
255 GB
50 GB
263.7 GB
51.5 GB
Secondary Indices
255 GB
Sum Tables
0.55 GB
Complete
519.25 GB
51.5 GB
Dunning Run
119
Why?
120
121
122
1 SELECT
10,000 SELECTs
10,000 SELECTs
31,000 Entries
1 SELECT
One
single
Select
open
account
items
from
BSID,
for
each:
stored
procedure
10,000 SELECTs
Calculate
due
date
Select
dunning
procedure,
level
and
area
Create
MHNK
entries
10,000 SELECTs
31,000 Entries
123
1 SELECT
One
single
Select
open
account
items
from
BSID,
for
each:
stored
procedure
10,000 SELECTs
Calculate
due
date
Select
dunning
procedure,
level
and
area
Create
MHNK
entries
124
10,000 SELECTs
31,000 Entries
Calculated on-the-y
Results
Original
Version
needed
about
20
minutes
Factor
800x
accelera8on
achieved
#
Opera(on
HANA2
Version
0.63s
Variant
2
Variant
3
1.01s
0.6s
(incl.
T047
&
KNB5
Join)
(incl.
T047
&
KNB5
Join)
27s
deferred
to
aggrega8on
0.5s
19s
1.1s
0.5s
15s
0.8s
0.4s
done
in
#1
1.2s
done
in
#1
done
in
#1
140ms
done
in
#1
Total
1
Minute
3.0s
1.5s
125
Dunning Application
126
Dunning Application
127
Available-to-Promise
Check
Dynamic
aggrega8on
Instant
rescheduling
in
minutes
vs.
nightly
batch
runs
Real-8me
and
historical
analy8cs
Outcome
128
In-Memory Available-to-Promise
129
Demand Planning
130
GORFID
Key Findings:
131
Near Real-Time as a
Concept
ReadEvent
Event
Read
Read
Event
Repositories
Repositories
Repositories
Verification
Verification
Verification
Services
Services
Services
8,000read
read
upupto
toto8.000
8.000
up
read
eventnotifications
notifications
event
event
notifications
persecond
second
per
per
second
upto
to2.000
2,000
up
up
to 2.000
requests
requests
requests
persecond
second
per
per
second
DiscoveryService
Service
Discovery
Discovery
Service
SAP HANA
HANA
SAP
SAP HANA
PPA
132
A
A
Learning Map
The
Future
of
Enterprise
Compu(ng
Founda(ons
of
Database
Storage
Techniques
In-Memory
Database
Operators
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Advanced
Database
Storage
Tech-
niques
Views
Dynamic Views
Presentation Layer
Excel
Business
Objects
Explorer
SAP
View
View Layer
(Calculations, Filter, ...)
Any Software
View
View
...
View
View
Node Tables
Persistency Layer
View
View
Other DB
Node Tables
Node Tables
Node Tables
Store
(Main Memory)
DB
Persistence
Restart
Object Hierarchy
Write Complete
Objects
Recovery
Logical
Log
135
Learning Map
The
Future
of
Enterprise
Compu(ng
In-Memory
Database
Operators
Advanced
Database
Storage
Tech-
niques
Founda(ons
of
Database
Storage
Techniques
Founda(ons
for
a
New
Enterprise
Applica(on
Development
Era
Bypass Solution
140
Todays System
Traditional BI
SAP
ERP
ETL
OLAP
Engine
BIA
Business
Objects
...
Traditional DB
Excel
w/ Cubes
Traditional DB
141
Traditional BI
SAP
ERP
ETL
OLAP
Engine
BIA
Business
Objects
...
Traditional DB
w/ Cubes
Traditional DB
IMDB
SSD
Excel
Traditional BI
SAP
ERP
ETL
OLAP
Engine
BIA
Business
Objects
...
Traditional DB
Excel
w/ Cubes
Traditional DB
IMDB
SSD
New
Applications
ERP
EL
OLAP
Engine
BIA
Business
Objects
...
IMDB
w/o cubes
Traditional DB
IMDB
SSD
New
Applications
Excel
ERP
EL
OLAP
Engine
BIA
Business
Objects
...
IMDB
w/o Cubes
Traditional DB
IMDB
Excel
BI 2.0
OLAP
Engine
Business
Objects
New
Applications
...
SSD
SAP
Excel
Traditional DB
BI 2.0
IMDB
OLAP
Engine
New
Applications
Business
Objects
...
SSD
SAP
Excel
IMDB
OLAP
Engine
New
Applications
Business
Objects
...
SSD
SAP
Excel