CS631 Quick Reference
Author: Ramdas Rao
Data Storage
Overview of Physical Storage Media
In decreasing order of cost and performance / speed:
• Cache - Is volatile, L1 Cache operates at processor speed (e.g., if
processor is 3GHz, then the memory access time is 1/3 ns)
• Main Memory: Access speed is about 10 to 100 ns; 300 times slower
than cache; volatile
• Flash Memory: Read access speed about 100 ms (same as memory);
Writing is slower 4 to 10 µs; Limited number of erase cycles sup-
ported; NOR flash; NAND flash uses page-at-a-time read/write; non-
volatile
• Magnetic disk storage: Non-volatile; 10 ms access time; order of
magnitude slower
• Optical Storage: CD, DVD (Digital Versatile Disk); capacities of 700
MB to 17 GB; Write-once, read-many (WORM); optical disk juke-
box
• Tape Storage: Sequential access; mostly used for backup and
archival; High capacity (40 GB to 300 GB); Tape jukebozes (li-
braries) - 100s of TB and PB
Magnetic Disks
• Platter - Tracks - Sectors - Blocks
• A cylinder is a set of tracks one below the other on each platter
• Concept of zones: The number of sectors in outer zones is greater
than the number of sectors in the inner zones (e.g., 1000 v/s 500 sec-
tors)
• Disk Controller Interfaces: ATA (PATA), IDE, SATA (Serial ATA),
SCSI, Fiber Channel, Firewrite, SAN (Storage Area Network) - stor-
age on network made to appear as one large disk, NAS (Network
Attached Disks) - NFS or CIFS
• Performance Measures of disks: Access time, capacity, Data transfer
rate, Reliability
• Access time: Time from when a red or write is issues to the time
when the data transfer begins
• Access time = Seek Time (arm positioning) + Latency (waiting for
sector to rotate under head)
• Average Seek Time = 1/2 of Worst Case Seek Time = 4 to 10 ms
• Average Latency Time = 1/2 of time for full rotation = 4 to 10 ms
• Average Access Time = 8 to 20 ms
• Data Transfer Rate = Rate at which date can be transferred = 25 MB/s
to 100 MB/s
The transfer rate on the inner tracks are significantly lower (30 MB/s)
than the outer tracks (since the umber of sectors on the inner tracks is
lesser than the outer)
• Mean Time to Failure (MTTF):
– For a single disk, it is about 57 to 136 years
– If multiple disks are used, the MTTF reduces significantly -
with 1000 new disks, MTTF is 1200 hours = 47 days
– If 100 disks are in an array and each has a MTTF of 100000
hours, then the MTTF of the array is 100000/100 = 1000 hours
– If 2 disks have MTTF of 100000 hours and MTTR of 10 hours,
2
then the M T T DataLoss = 10000010+10
= 500x106 hours
• Mean Time to Data Loss = MTTF + MTTR, (MTTR is Mean Time
to Repair)
• Optimization of Disk Block Access:
– Scheduling of access of blocks (e.g., Elevator algo)
– File organization (File Systems, fragmentation, sequential
blocks, etc.)
– Non-volatile write buffers
∗ NVRAM to speed-up writes
∗ Log disk (since access to log disk is sequential)
∗ Log file (no separate disk - like in journaling file systems)
RAID
Redundant Array of Independent (Inexpensive) Disks; RAID improves reli-
ability via redundancy
• How RAID improves performance via parallelism:
Increases the number of I/O requests hanbdled per second or the
transfer rate or both
– Bit-level striping:
∗ The bits of each byte are split across several disks
∗ For an 8-disk configuration, transfer rate is 8 times that of
single disk, and number of I/Os are same as that for single
disk; Bit i of each byte does to disk i
∗ For a 4-disk config, bits i and 4+i of each byte do to disk i
– Block-level striping (most commonly used):
∗ Stripes blocks across multiple disks (one block on each
disk)
∗ Logical block i goes to disk (i mod n) + 1 and it uses the
floor(i/n)th physical block of the disk (forumlae assume
disk number starts from 1 and blocks from 0)
∗ For large reads (multiple blocks), the data transfer rate is
n times that of single disk (n is the number of disks)
∗ For single block read, transfer rate is the same as that of
single disk, but other disks are free to process other re-
quests
– Other forms of striping: Bytes of a sector, sectors of a block
• 2 main goals of parallelism in a disk system are:
– Load-balance smaller disk requests so that the throughput is
increased
– Parallelize large accesses so that the response time of large
accesses is reduced
• RAID Levels:
– RAID Level 0: No redundancy, block striping; used when
backup is easily restorable
– RAID Level 1: Mirroring w/ block striping (aka level 1 + 0 or
10); Mirroring without block striping is called Level 1 (2M
number of disks required; used when number of writes are
more (e.g., log disk)
– RAID Level 2: Memory Style ECC (w/ parity bits); Fewer
number of disks required than level 1; Some disks store par-
ity (e.g., 3 Parity disks for 4 disks of data); Subsumed by level
3
– RAID Level 3: Bit-interleaved parity; a single parity bit can be
used for error detection as well as correction (e.g., 1 P disk for
3 disks of data)
– RAID Level 4: Block-interleaved parity; separate disk for par-
ity (at block level); Parity disk will be involved for every read
/ write; A single write requires 4 disk accesses: 2 to read the
2 old blocks and 2 to write the new blocks (parity and data);
Subsumed by level 4
– RAID Level 5: Block interleaved distributed parity; all disks
store parity for the other disks; subsumes level 4
– RAID Level 5: P+Q redundancy (like RAID Level 5, but stores
extra redundant information to guard against multiple disk fail-
ures); ECC such as Reed-Solomon used; 4-bits of parity instead
of 2 can tolerate upto 2 disk failures
• Choice of RAID:
– RAID Level 0 : Use where data safety is not critical (and
backup easily restorable)
– RAID Level 1: Offers best write performance, use for High I/O
requirements for moderate storage (e.g., log files in a database
system)
– RAID Level 5: Storage intensive apps such as vide data stor-
age; More frequent reads and rare writes
– RAID Level 6: Use when data safety is very important
• Hardware RAID v/s Software RAID: Hot-swapping, etc.
Buffer Manager
• Buffer Replacement Strategy: LRU (not good for nested loop), MRU
(depends on the join strategy), Toss Immediate
• Pinning - pinning a block in memork (block that is not allowed to be
written back to disk)
• Forced output of blocks: even if space is not required (used when
xact log records need to go to stable storage)
File Organization
• Fixed Length Records:
– On deleting, move the records (expensive)
– On deleting, move final record to free space (requires addi-
tional disk access)
– Store header and pointers to link (chain) free space for records
• Variable Length Records:
– Use slotted page structure for each block
– Each block has a header that stores:
∗ Number of record entries in the header
∗ End of free space in the block
∗ An array whose entries contain the location and the size
of each record in the block
• Organization of the records within a file:
– Heap file organization
– Seq. file organization (may require overflow blocks or periodic
reorganization)
– Hashing file organization
– Multi-table clustering file organization: For example, for join
of depositor nad customer, after on depositor record, store the
customer records for that depositor
Data Dictionary Storage
• Store like a miniature database
• Types of information stored:
– Names of the relations
– Names and attributes of each relation
– Domains and lengths of each attribute
– Names and definitions of views
– Integrity constraints
– User, Auth and Accounting info, Passwords
– Statistical data
– Info on indices
Indexing and Hashing
• 2 basic types of indices: Ordered Indices (based on a sorted order-
ing of values) and Hash Indices (based on a uniform distribution of
values across a range of buckets)
• Indexing techniques must be evaluated on these factors:
– Access time
– Access type (range of values or point-based)
– Insertion time
– Deletion time
– Space overhead for storing the index
• Ordered Indices:
– Clustering or Primary Index: If the file containing the records
is sequentially ordered, then a clustering or primary index is an
index whose search key also defines the sequential order of the
file
– Non-clustering or Secondary Index: Indices whose search key
specifies an order different from the sequential order of the file
• Indexed Sequential File: A file with a clustering index on the search
key is called an Indexed Sequential File.
– Dense index: An index record appears for every search-key
value in the file
– Sparse index: An index record appears for only some of the
search-key values (some sequential scan required to locate the
records)
– Main disadvantages of Indexed Sequential File: Performance
degrades (both for sequential scans as well as for index
lookups) as file grows; can be remedied by periodic reorg of
the file, but expensive
• Multi-level Index:
– n-level sparse indices
– If an index occupies 100 blocks, using binary search requires
ceiling (log2 (b)) disk accesses
– Closely related to tree structures
Read update of indices pseudo code from notes / book
• Secondary Indices:
– Cannot be sparse (must be dense)
– Pointers in a secondary index (on search keys that are not can-
didate keys) do not point directly to the file (instead, each
points to a bucket that contains pointers to the file)
– Disadvantages: Sequential scan in secondary-key order is very
slow; they impose significant overhead on the modification of
the file (note that when a file is modified, every index must be
updated)
B+-Tree Index Files
P1 K1 P2 K2 . . . Kn−1 Pn
• Most widely used index structure
• Maintains its efficiency despite insertions and deletions of data
• A b+-tree index takes the form of a balanced tee in which every path
from the root of the tree to a leaf of the tree is of the same length
• Each non-lead node has between ceil(n/2) and ceil(n) children, where
nis fixed for a particular tree
• Each leaf must have at least ceil((n-1)/2) values and at most ceil(n-1)
values
• Each non-leaf must have at least ceil(n/2) pointers and at most ceil(n)
pointers
• Imposes performance overhead for insertion and deletion and space
overhead (as much as half of a node maybe empty), but is still pre-
ferred (since periodic file reorg is not needed)
• A B+-tree index is like a multi-level search index
• Queries on B+-trees:
– If there are K-search key values in a file, then the path from the
root to the leaf is no larger than ceil(logn/2 (K)). For exam-
ple, if K = 1000000 and n = 100, then ceil(log50 (1000000))=
4. Therefore, at most 4 nodes need to be accessed. For binary
search, it would require ceil(log2 (1000000)) = 20 nodes.
Algo for B+tree from book
B+-Tree File Organization
• Leaf node stores records than pointers
• Need to consider capacity of mode while splitting and coalescing
(since records are stored in the leaf nodes)
• SQL CLOBs and large objects are split into sequence of smaller
records and organized in a B+-Tree file organization
• Forumla here
• Indexing Strings: Strings are variable length (need to consider capac-
ity); prefix compression can be used to reduce the size
• Advantages: No repetition of search-key values; Non-leaf nodes con-
tain pointer to data (so additional ptr is required in non-leaf nodes -
height of the tree may increase compared to B+-tree); not much gain
compared to B+-tree since anyway majority of the data is in the leaf
node
• Disadvantages: Deletion and other ops more complex
Other properties
• Mutliple Key Access:
1. Use multiple-single key indices: Perform intersection. Perfor-
mance is poor if there are many records satistying condition
individually, but few satisfying both conditions
2. Use indices on multiple keys: composite search key; queries
with conjunction predicates with equality on primary key is still
okay (since we can treat as (P, -inf) to (P, inf); but if inequality
for first, then inefficient
3. Bitmap indices can be used: existence bitmaps and presence of
NULLs need to be handled
4. R-tree (extension of B+-tree) to handle indexing on multiple
dimensions (e.g., for geographical data)
• Non-unique search keys: Use unique record if to prevent buckets and
extra page lookups; Search for customer name = ’X’ internally be-
comes (’X’, -inf) to (’X’, inf)
• Covering indices: Store multiple (extra) attributes alongwith pointer
to records (e.g., balance can be stored if it is required frequently);
saves one disk access
• Secondary indices and index relocation:
– Some file organaiztions (such as B+-tree) change the location
of records even when the records may not have been updated
– To overcome problems due to this in Secondary indices, we can
store the values of the search-key indices (instead of pointers)
in the secondary index and use the primary index to lookup
– Cost of access increases, but no change is required on file reorg
Hashing
• Hash function must be chosen so that: distribution is uniform and is
random
• Hashing can be used for:
– Hash file organization: compute address of block directly
– Hash index organization: organizes index into a hash file struc-
ture
• Example hash function: s[0[ to s[n-1] is a string of n characters long
s[0] ∗ 31n−1 + s[1] ∗ 31n−2 + . . . + s[n − 1] mod number of buckets
• Bucket overflows can still occur due to: insufficient buckets, and
skew
• Overflow can be handled through Overflow chaining, or Open hash-
ing (linear or quadratic probing, etc.); open hashing is not good for
db since deletion in this is troublesome
• Dynamic hashing: One form of extendable hashing; use of hash pre-
fix; increasing and decreasing of bucket address table
• Advantages of dynamic hashing: No space reservation required; Per-
formance does not degrade as file grows or shrinks
• Disadvantages of dynamic hashing: Additional lookup for bucket ad-
dress table required
• Linear hashing avoids extra level of indirection at the possible cost of
more buckets
• Ordered Indexing: Can handle range queries better
• Hashing: Bad for range queries; suitable for single-value compar-
isons; good for temporary files during query processing
Bitmap Index Structure
Bitmaps and B+-trees can be combined
Query Processing
• Steps in Query Processing: Parser and Translator (gives rel. algebra
expression), optimizer (also consults statistics about data and give ex-
ecution plan), Evaluation Engine (evaluates plan and outputs results
of the query)
Mention about join and sorting techniques here
Cost of Selections
ts : Seek time, br : number of blocks in file, tT : Transfer time for one block.
Index structures are called access paths since they provide a path through
which data can be located and accessed.
• Linear Search (A1): Can be applied to any file
– Cost = ts + (br ∗ tT ) (one seek + search all blocks)
– For key attributes, we can stop after finding the match. Average
Cost is : ts + b2r ∗ tT . Worst-case cost is: ts + br ∗ tT
• Binary Search (A2):
– Cost for key searches: dlog2 (br )e ∗ (ts + tT )
– Cost for non-key searches: dlog2 (br )e ∗ (bn + tT ), where n is
the number of items with duplicate keys
• Primary Index, equality on key (A3): For a B+-tree, if hi is the height
of the tree, then
– Cost = (hi + 1) ∗ (tT + ts )
• Primary Index, equality on non-key (A4):
– Cost = hi ∗ (tT + ts ) + ts + b ∗ tT , where b is the number of
blocks containing the matched duplicate keys
• Secondary Index, equality on key (A5):
– For key: Cost is same as that for A3
– For non-key: Cost = (hi + n) ∗ (tT + ts ), n is the number of
blocks contianing matching keys
• Primary Index, comparison (A6):
– For A > v, first locate V and then sequential access. Cost is
similar to A4
– For A < v or A ≤ v, no index is used. Similar to A1.
• Secondary Index, comparison (A7):
– Searching index is similar to A6
– But, retrieving each block may require access a different block
– Therefore, linear search may be better
• Conjunctive Selection using one index (A8):
– Use one index to retrieve (use A2 through A7)
– Compare each for satisfying the other condition
– To reduce the cost, we choose a θi and one of A1 through
A7 for which the combination results in the least cost for
σthetai (r)
– Cost is cost of chosen algo
• Conjunctive Selection using composite index (A9):
– Use composite index (same as A3, A4 or A5)
• Conjunctive Selection by using intersection of identifiers (A10):
– Cost is sum of (cost of individual index scans) + (cost of re-
trieval of records in the intersection)
– Sorting can be used so that all pointers in a block come to-
gether; blocks are read in sorted physical order to minimize
disk arm movement
• Disjunctive Selection by using union of pointers (A11):
– If access paths are available on all conditions, each index is
scanned to get the pointers, union is taken and records are re-
treived
– Even if one of the condition does not have an access path, the
most efficient method could be a linear scan
Cost of Joins
• Two reasons why sorting is important: the query may require output
to be sorted, and joins and some other operations can be implemented
efficiently, if the input relations are first sorted. Sorting physically is
more important than sorting logically (to reduce disk arm movement)
• Natural Join can be expressed as a θ join followed by elimination of
repeated attributes by a projection
• nr : number of tuples in r, ns : number of tuples in s, br : number of
blocks of r, bs : number of blocks of s
• Nested loop Join:
– If both relations can be read into memory, cost = (br + bs )
– Else, if only one block of each relation fits into memory,
cost = nr ∗ bs + br , assuming “r” is the outer relation
• Block Nested loop Join:
– Assumption: M+1 is the total blocks available (1 for o/p); else
the denom will be (M-2)
 – r is the outer relation: Cost = d Mb−1r
∗bs +br , where M blocks • Set operations of union and intersection are commutative
are allocated to r
E1 ∪ E2 = E 2 ∪ E 1
• Sort Merge Join
– Soting cost of each reltion assuming they are not sorted is (M E1 ∩ E2 = E 2 ∩ E 1
is the number of pages available for sorting - 1 is for o/p and However, Set Difference is not commutative.
M-1 for input):
• Set union and intersection are associative
∗ for r, br (2dlogM −1 (br /M )e + 1 + 1
∗ for s, bs (2dlogM −1 (bs /M )e + 1 + 1 (E1 ∪ E2 ) ∪ E3 = E1 ∪ (E2 ∪ E3 )
– After sorting, assuming that all the tuples with the same value
(E1 ∩ E2 ) ∩ E3 = E1 ∩ (E2 ∩ E3 )
for the join attributs fir in memory, the cost is: (Sorting Cost) +
br + bs • Selection operation distributes over union, intersection and set differ-
• Hash Join ence operations:
– Assume no overflow occurs σP (E1 − E2 ) = σP (E1 ) − σP (E2 )
– Use smaller relation (say r) as the build relation and larger re-
lation (say s) as the probe relation Also,
– If M > br /M , no need of recursive paritioning and cost is:
Cost = 3(br + bs ) σP (E1 − E2 ) = σP (E1 ) − E2
– Else, if recursive partioning occurs: Cost = 2(br + (this does not hold for intersection???)
bs )dlogM −1 (br )−1e+br +bs (included is the cost for reading • Projection operation distributes over union:
and writing partitions)
πL (E1 ∪ E2 ) = (πL (E1 )) ∪ (πL (E2 ))
• A B-tree organization has b(m − 1)n/mc entries for each node
Join ordering: choose such that the size of th temporary results are re-
Query Optimization duced
Enumeration of Equivalent Expressions:
Equivalence Rules • Space requirements can be optimized by pointing to shared sub ex-
(Set Version) pressions
• Cascade of σ for conjunction selections • Time requirements can be reduced by optimization (dynamic pro-
gramming, etc.)
σθ1 ∧θ2 (E) = σθ1 (σθ2 (E))
Estimating Statistics of Expr Results
• Commutativity of selection operations
• Catalog Information:
σθ1 (σθ2 (E)) = σθ2 (σθ1 (E)) – nr = number of tuples in r
– br = number of block containing tuples of r
• Only final projection in a sequence of projections
– lr = size of a tuple of r in bytes
– fr = blocking factor of r (= number of tuples of r that fit in one
πL1 (πL2 (. . . πLn ))) = πL1 (E)
block)
• Selections can be combined with Cartersian products and theta joins: – V (A, r) = number of distinct values that appear in r for at-
– tribute A
σθ (E1 xE2 ) = E1 on θ E2 – If A is a key for r, then V (A, r) = nr
– – If tuples of r are phsyically stored together, br = d nfrr e
σθ1 (E1 σθ2 E2 ) = E1 o
nθ1 ∧θ2 E2 – Histogram for a range of values of attribute can be used for
estimating (histograms can be equi-width or equi-height)
• Theta joins and natural joins are commutative
• Selection Size Estimation:
E1 o
n E2 = E2 o
n E1 – Equality (σA=a (r))
∗ Assuming equi-probable, N um = nr /V (A, r)
• Associativity of joins ∗ With histogram, num = nrange /V (A, range)
– Natural Joins are associative: – Comparison (σA≤v (r))
∗ If v < min(A, r), num = 0
(E1 o
n E2 ) o
n E3 = E1 o
n (E2 o
n E3 )
∗ If v ≥ max(A, r), N um = nr
v−min(A,r)
– Theta joins are associative in the following manner: If θ2 has ∗ Else, num = nr . max(A,r)−min(A,r) (this can be modi-
only attributes from E2 and E3 , then: fied to use a histogram, where available - use the number
in the ranges, instead of in the entire relation)
(E1 o
n θ1 E 2 ) o
nθ1 ∧θ3 E3 = E1 o
nθ1 ∧θ3 (E2 o
n θ2 E 3 ) ∗ If v is not known as in the case of stored procedures, as-
sume num = nr /2
– Cartesian products are also associative – Complex selections
• Distributivity of selections ∗ Conjunctions (σθ1 ∧θ2 ...∧n (r)): num = nr . s1 ∗sn2 ∗...s
n
n
,
r
– If θ0 only involves E1 , then where si is the number of tuples that satisfy the selection
σθ1 (r).
σθ0 (E1 o
nθ E2 ) = (σθ0 (E1 )) o
n θ E2 si /nr is called the selectivity of the selection σθ1 (r)
∗ Disjunctions (σθ1 ∨θ2 ...θn (r)): num = nr ∗ [1 − (1 −
– If θ1 only involves E1 and θ2 only involves E2 , then s1
nr
)(1 − ns2r ) . . . (1 − n
sn
r
)
∗ Negations: Compute num as: num = nr − num(σθ (r).
σθ1 ∧θ2 (E1 o
nθ E2 ) = (σθ1 (E1 )) o
nθ σθ2 (E2 )) If NULLs are present, compute as: num = nr −
• Distributivity of projections If L1 are only attributes of E1 and L2 num(σθ (r) − num(N U LLs).
only of E2 , then • Join Size Estimation:
– Cartesian Product: N um(rxs) = nr xns
πL1 ∪L2 (E1 o
nθ E2 ) = πL1 (E1 ) o
nθ (πL2 (E2 )) – Natural Joins
 ∗ R ∩ S = φ: same as Cartesian Product Materialized Views
∗ R ∩ S is a key for R: num ≤ ns . Similarly, when it is a
Normally, only query defintion is stoed. In materialized views, we compute
key for S.
the contents of the view and store. View Maintenance is required to keep
∗ R ∩ S is a foreign key of S, referencing R: N um = ns
the materialized view up-to-date. View Maintenance can be relegated to the
∗ R ∩ S is neither a key for R nor S: Choose minimum of
programmer or be taken care by the system (can be immediate or deferred)
the following, where R ∩ S = {A}
· N um = nr ∗ ns /V (A, s) • Incremental View Maintenance: Update can be treated conceptually
· N um = nr ∗ ns /V (A, r) to Delete followed by Insert
• Size Estimation for Other Operations: • Join Operation:
– Projection (πA (r)): N um = V (A, r) (since projection elimi- – For inserts, vn ew = vo ld ∪ (ir o
n s)
nates duplicates) – For deletes, vn ew = vo ld − (dr on s)
– Aggregation (A GF (r)): N um = V (A, r) (since one tuple in • Selection and Projection Operation:
output for each distinct value of A)
– For selection inserts, vn ew = vo ld ∪ σθ (ir )
– Set Operations
– For selection deletes, vn ew = vo ld − σθ (dr )
∗ Same relation operations: Rewrite as conjunctions, dis-
– For projection, need to handle duplicates:
junctions or negations and use previous results (e.g.,
σθ1 (r) ∪ σθ2 (r) = σθ1 ∨θ2 (r) ∗ Keep count for each tuple in projection πA (r)
∗ Different relation operations: Inaccurate, but provides up- ∗ Decrement count on delete and delete record from view
per bound when count is 0
· N um(r ∪ s) = nr + ns ∗ Increment count on insert or add to view if not present
· N um(r ∩ s) = min(nr , ns ) • Aggregation Operations:
· N um(r − s) = nr – Count: Similar to projection
– Outer Joins: Inaccurate, but provides upper bound – Sum: Similar to count (but need to keep sum as well as count)
∗ N um(rlef touters) = N um(r o n s) + nr . Similarly, – Avg: Keep sum as well as count
for (r right outer s)
– Min, Max: Insetion - easy; deletion - expensive - need to find
∗ N um(routers) = N um(r o n s) + nr + ns
new min, max
• Estimation of number of distinct values:
• Other Operations:
– If selection condition θ forces A to take a single value, N um =
V (A, σθ (r)) = 1 – Set Intersection: (r ∩ s)
– If range of values, then Num = Number of specified values in ∗ On insertion in r, check if it is in s. If so, add to view
the selection condition ∗ On deletion in r, check if it is in s. If so, delete from view
– If selection condition of the form (A op V), then – Outer Joins: (routerjoins)
V (A, σθ (r)) = V (A, r) ∗ s, where s is the selectivity of the ∗ On insertion in r, check if it is in s. If so, add to view. If
selection not is s, still add to view, but padded with NULLs
– In all other cases, num = min(V (A, r), nσθ (r)) ∗ On deletion from s, pad with NULLs if it is in r and no
– For joins: longer in S
∗ On deletion from r, remove from view
∗ If all attrs. in A are from r, then V (A, r o n s) =
min(V (A, r), nro ns )
∗ If A has A1 from r and A2 from S, then N um = Query Optimization using Materialized Views
min(V (A1 , r) ∗ V (A2 − A1 , s), V (A2 , s) ∗ V (A1 − • Optimizer may need to substitute the query (or sub-query) by materi-
A2 , r), nro
ns )
alized view, if it exists
– For projections: N um = nr
• Replacing a use of a materialized view by the view definition. For
– For aggregates like sum, count, avg, it is same as nr example, if σA=10 (V ), where V is defined as r on s and there is an
– For min(A) and max(A), N um = min(V (A, r), V (G, r)), index on A in r, but not in r o
n s.
where G denotes the grouping attributes

Transactions
Choice of Evaluation Plans
Transaction: a set of operations that form a single logical unit of work.
• To choose the best overall algo, we must consider even nonoptimal
algos for individual operations • ACID properties of a transaction:
• Cost-based optimization: With n relations, there are 2(n−1)! different – A - Atomicity: all or none (handled by Transaction Mgmt.
(n−1)!
join orders. component)
• Time complexity is O(3n ) – C - Consistency: if db was consistent before xact, then it should
be consistent after the xact (handled by programmer or con-
• Dynamic Programming Algo: Outline algo here
straints)
– I - Isolation: an xact does not see the effects of a concurrent
Heuristics in Optimization running xact (handled by the Concurrency Control component)
1. Perform selection operations as early as possible (may cause prob- – D - Durability: once committed, stays committed (handled by
lems if no index on selection attribute and r relation is small in the Recovery Mgmt. component)
σθ (r o
n s) • Transaction States: Active, Failed, Aborted (perform rollback), Par-
2. Perform projections early (similar problems as in 1 above) tially committed, Committed
3. Left-Deep Join Orders: convenient for pipelining • Shadow-copy technique: Ensures atomicity and durability, and is
4. Avoid Cartesian products used by text editors. Disadvantage: Very expensive to make copies
of entire db; no support for concurrent xacts
5. Cached plan can be reused
• Need for Concurrent Executions: Improved throughput (tps), Im-
proved resource utilization, Reduced waiting time (e.g., smaller xacts
Optimizing Nested Queries queued up behind a large xact), Reduced average response time
“where exsits” type of query can be optimized by using “decorrelation”; • Schedules: represent the chronological order in which the instruc-
rewrite as join of temporary table (remember to use select distinct and to tions are executed in the system. For a set for n transactions, there
take care of NULL values to preserve the number of tuples) exist n! different serial schedules
 • Consistency of the db under concurrent execution can be ensured by
making sure that any schedule that is executed has the same effect as
a serial schedule (that is, one w/o concurrent execution)
Conflict Serializability:
• Instructions I1 and I2 conflict if they are operations by different xacts
on the same data item and at least one of them is a write operation
• If a schedule S can be transformed into a schedule S 0 by a series of
swaps of non-conflicting instructions, S and S 0 are said to be conflict
equivalent
• A schedule S is said to be conflict serializable if it is conflict equiva-
lent to some serial schedule
• This prohibits certain types of schedule even though there would be
no problem (e.g., ops that simply add and subtract). However, these
cases are harder to analyze.
View Serializability:
• Less stringent than Conflict Serializability
• View Equivalence: 2 schedules S and S 0 are view equivalent if ALL
3 conditions mentioned below are met:
– For each data item Q, if xact Ti read the initial value of Q in S,
then xact Ti in S 0 must also read the initial value of Q
– For each data item Q, if xact Ti executes read(Q) in S and if
that value was produced by xact Tj , then that read(Q) op of
xact Ti in S 0 must also read the value of Q that was produced
by that same write op of xact Tj
– For each data item Q, the xact (if any) that performs the final
write(Q) op in S, must also perform the final write(Q) op in S 0
• A schedule is said to be view serializable if it is view equivalent to
some serial schedule
• Blind Writes: Writing a value w/o reading it first
• Blind Writes appear in any view serializable schedule that is not con-
flict serializable
Other properties
• Recoverable Schedule: is one where, for each pair of xacts Ti and
Tj such that Tj reads a data item previously written by Ti , then the
commit operation of Ti appears before the commit operation of Tj
• Cascading rollback is undesirable since it can lead to undoing a sig-
nificant amount of work
• Cascadeless Schedule: is one where, for each pair of xacts Ti andTj
such that Tj reads a data item previosuly written by Ti , the commit
operation of Ti occurs before the read operation of Tj .
• A cascadeless schedule is also recoverable, but not vice-versa.
• The goal of concurrency control schemes is to provide a high degree
of concurrency, while ensuring that all schedules that can be gener-
ated are conflict or view serializable, and are cascadeless.
• Testing for Conflict Serializability: (to show that the generated sched-
ules are serializable)
– Construct precedence graph for a schedule S (vertices are xacts,
edges indicate read/write dependencies)
– If the graph contains no cycles, then the schedule S is conflict
serializable
– A serializability order of the xacts can be obtained through
topological sorting of the precedence graph
– Cycle detection algos are O(n2 )
• Testing for View Serializability:
– NP-complete problem
– Sufficient conditions can be used
– If sufficient conditions are satisfied, the schedule is view-
serializable
– But, there may be view-serializable schedules that do not sat-
isfy the sufficient conditions
See examples and exercises of schedules from the book
Concurrency Control
Shared locks (S) and Exclusive locks (X): Compatibility matrix: (S,S) true,
(S,X) false, (X,S) false, (X,X) false
Starvation can be avoided: by processing the lock requests in the order
in which they were made
2PL
• Ensures serializability
• Growing phase, Shrinking phase
• Does not prevent deadlock
• Cascading rollbacks may occur (e.g., if T7 reads a data item that was
written by T5 and then T5 aborts
– To avoid cascading rollbacks, strict 2PL can be used where
exclusive locks must be held till the xact aborts or commits
(prevents xacts from reading uncommitted writes)
– rigorous 2PL can be used where ALL locks are held till the
xact aborts or commits; xacts are serialized in their commit or-
der
• Upgrading and Downgrading of locks can be done; upgrading should
be allowed only in the growing phase, while downgrading only in the
shrinking phase (e.g., series of reads followed by write to a data item
- in other forms of 2PL above, the xact must obtain an X lock on the
data item to be updated, even if it is much later)
Implementation of locking: Hash table for data items with linked list (of
xacts that have been granted locks for that data item plus those that are wait-
ing). Overflow chaining can be used.
Graph-based Protocols
• Acyclic graph of data item locking order
• A data item can be locked by Ti only if its parent is currently locked
by Ti
• Locks can be released earlier; so shorter waiting times and increased
concurrency
• Deadlock free; so no rollbacks are required
• Disadvantages: may need to lock more data items than needed (lock-
ing overhead and increased waiting time), w/o prior knowledge of
which data items to lock, xacts may have to lock the root of the tree
and that can reduce concurrency greatly
• Cascadelessness can be obtained by tracking commit dependencies
such that a transaction is not allowed to commit until the ones that it
had read values written by have not commited
Timestamp-based Protocols
• Determines the serializability order by selecting the order in advance
• Using timestamps: could be the system clock or a logical counter
• Each xact is given a timestamp when it enters the system
• Each data item has 2 timestamps: W-timestamp (the largest ts of
any xact that wrote the data item successfully) and R-timestamp (the
largest ts of any xact that read the data item successfully)
• Timestamp-Ordering Protocol is:
– If Ti issues read(Q)
∗ If T S(Ti ) < W − timestamp(Q), reject the read and
rollback Ti
∗ If T S(Ti ) ≥ W − timestamp(Q), execute the read
and set the R-timestamp of Q to maximum of T Si and
R-timestamp(Q)
– If Ti issues write(Q)
∗ If T S(Ti ) < R − timestamp(Q), reject the write and
rollback Ti
∗ If T S(Ti ) < W − timestamp(Q), reject the write and
rollback Ti
∗ In all other cases, execute the write and set the W-
timestamp of Q to T Si
– Rolled-back TS get a new timestamp when they are restarted
– Freedom from deadlocks
 – However, xacts could starve (e.g., long duration xact getting
restarted repeatedly due to conflicts with short duration xacts)
– Recoverability and cascadelessness can be ensured by:
∗ Performing all writes together at the end of the xact; no
xact is permitted to access any of the data items that have
been written
∗ Using a limited form of locking, whereby uncommitted
reads are postponed until the xact that updated the item
commits
– Recoverability alone can be guaranteed by using commit de-
pendencies, that is, tracking uncommited writes and allowing a
xact Ti to commit only after the commit of all xacts that wrote
a value that Ti read.
• Thomas’ Write Rule:
– Allows greater potential concurrency
– Ignores writes if T Si < W − timestamp(Q), instead of
rolling it back
Validation-based Protocols
• Also called optimistic concurrency control
• Each xact goes through 3 phases (for update xacts and 2 for read-only
xacts):
– Read phase: The system executes the xact Ti ; it reads all data
items and performs all write operations on temporary local
variables, w/o updates to the actual db
– Validation phase: Checks if the updates can be copied over to
the db w/o conflict
– Write phase: Done only if the xact succeeds in the validation
phase. If so, the system applies the updates to the db; otherwise
the xact is rolled back
• Validation test for xact Tj requires that for all xacts Ti with
T S(Ti ) < T S(T j), one of the following conditions must hold:
– F inish(Ti ) < Start(Tj )
– The set of data item written by Ti does not intersect with the set
of data items read by Tj and Ti completes its write phase before
Tj starts its validation phase. (Start(Tj ) < F inish(Ti ) <
V alidation(Tj ). This ensures that the writes of Ti and Tj do
not overlap
Multiple Granularity
• Hierarchy of granularity: DB, Areas, Files, Records; visualize as a
tree with the DB at the root of the tree
• Explicit locking at one level will mean implicit locking at all nodes
below it
• Care must be taken not to grant explicit lock at a level above which
another lock has been granted already (e.g., cannot lock a record ex-
plicitly, if the file has been locked). The tree must be traversed from
the root to the required level to find out.
• Also, a db cannot be locked, if someone else is holding a lock at a
lower level. Instead of searching the entire tree to determine this,
intention lock modes are used.
– When an xact locks a node, it acquires an intention lock on all
the nodes from the root to that node.
– IS (Intention-Shared) lock: If a node is locked in IS mode, then
explicit shared locking is being at the lower level
– IX (Intention-Exclusive) lock: If a node is locked in IX mode,
then explicit exclusive or shared locking is being at the lower
level
– SIX (Shared and Intention-Exclusive) lock: If a node is locked
in SIX mode, then the subtree rooted at that node is being
locked in explicitly shared mode and explicit exclusive lock-
ing is being at the lower level
– Compatibility Matrix:
IS IX S SIX X locks is relatively high; otherwise detection and recovery are more efficient.
IS true true true true false
IX true true false false false
S true false true false false
SIX true false false false false
X false false false false false
– Multiple-granularity protocol:
∗ The compat matrix above must be followed for granting
locks
∗ It must lock the root of the tree first, and it can lock it in
any mode
∗ It can lock a node Q in S or IS mode only if it currently
has the parent of Q locked in either IS or IX mode
∗ It can lock a node Q in X, SIX, or IX mode only if it cur-
rently has the parent of Q locked in either IX or SIX mode
∗ It can lock a node only if it has not previously unlcoked
any node (that is, Ti is 2P)
∗ It can unlock a node Q only it it currently has none of the
children of Q locked
∗ Locking is done top-down, whereas unlocking is done
bottom-up
– This protocol enhances concurrency and reduces lock overhead
and is good for apps that include a mix of:
∗ Short xacts that access only a few data items
∗ Long xacts that produce reports from the entire file or set
of files
– Deadlock is possible
Multiversion Schemes
Instead of delaying the reads or aborting an xact, these schemes use old
copies of the data. Each write produces a new version of a data item, and
read is given one of the versions of the data item. The protocol must ensure
that the version given ensures serializability and that an xact be able to easily
determine which version to read.
• Multiversion Timestamp Ordering:
– Each xact has unique ts as before (for the TS Scheme)
– Each version of data item has content of the data item, R-ts and
W-ts
– Whenever an xact writes to Q, a new version of Q is produced
whose R-ts and W-ts are initialized to T S(Ti ).
– Whenever an xact reads Q, the R-ts of Q is set to T S(Ti ) only
if R − ts(Q) < T S(Ti )
– The protocol is (an xact Ti wants to read or write Q):
∗ Find a version Qk whose w-ts is the largest ts ≤ T S(Ti )
∗ If xact Ti issues read(Q), the value returned is the content
of Qk
∗ If xact Ti issues write(Q) and T S(Ti ) < R − ts(Qk ),
then rollback Ti (some other xact already read the value
and so we cannot change it now). On the other hand, if
T S(T i) = W − ts(Qk ), overwrite the contents of Qk
(w/o creating a new version); else, create a new version.
– Older versions of a data item are removed by: If there are 2 ver-
sions of a data item with W-ts less than the oldest transaction
in the system, the older of these 2 versions can be removed
– A read request never fails and is never made to wait
– Disadv: Reading requires updating of R-ts (2 disk accesses
than one), and conflicts between xacts are resolved through
rollbacks rather than waits (Multiversion 2PL solves the roll-
back problem).
– Does not ensure recoverability and cascadelessness; can be ex-
tended in the same manner as the basic TS-ordering scheme
• Multiversion 2 PL: Attempts to combine the adv. of multiversion
with 2PL; it differentiates between read-only xacts and update xacts.
TODO
Deadlock Handling
2 methods to deal with deadlocks: Deadlock prevention, and Deadlock de-
tection and recovery. Deadlock prevention is used if the probability of dead-
Detection scheme requires overhead to maintain information while running
to detect deadlocks as well as losses that can occur due to recovery from
deadlocks.
• Deadlock Prevention using partial ordering: Use partial ordering
technique like tree protocol
 • Deadlock Prevention using total ordering and 2PL: Use total or-
dering and 2PL; in this case, the xact cannot request locks on items
that precede that item in the ordering
• Deadlock Prevention using wait-die: Using xact rollback; older
xacts are made to wait; younger ones are rolled back if the lock is
currently held by an older one; the older the xact gets, the more it
must wait
• Deadlock Prevention using wound-wait: Pre-emptive technique;
younger xact is wounded by older one; younger one is made to wait,
if older xact has a lock on the item; there may be fewer rollbacks in
this scheme
Both wait-die and wound-wait avoid starvation and both may cause
unnecessary rollbacks
• Timeout-based schemes: In between deadlock prevention and de-
tection schemes; allow an xact to wait for sometime; if timeout, as-
sume that deadlock may have occurred and rollback xact; Easy to
implement, but difficult to determine the correct duration of time to
wait; suitable for short xacts
• Deadlock Detection and Recovery: Must check periodically if a
deadlock has occurred (detection); Can be done to see if cycles exist
in a wait-for graph; Selection of a victim can be done on the basis
of minimum cost (how many xacts will be involved; how many data
items have been used; how much longer to complete, etc.); Total roll-
back or partial rollback (just enough rollback to the point where the
appropriate lock is released that breaks the deadlock); Starvation can
be prevented by including the number of times an xact has been rolled
back in the cost factor while deciding the victim
Insert and Delete Operations
• Delete operation similar to write (X lock for delete op in 2PL; treated
similar to write op in TS-ordering)
• Insert operation: X-lock in 2PL; in TS-ordering, assign TS of the xact
that is inserting the item to the R-ts and W-ts of the data item
Phantom Phenomenon
Consider computation of sum by using a select and an insert statement; this
can result in a non-serializable schedule if locking is done at the granularity
of the data item; neither access any tuple in common - so the conflict would
go undetected.
Can be alleviated by:
• Associating a virtual data item with every relation and having the
xacts lock this (in addition to the tuples), if they are updating or read-
ing info about the relation
• Index-locking protocol using 2PL can be used: nodes of the index
must be locked in shared mode for lookups; writes must lock the ap-
propriate nodes of the index in exlcusive mode
• Variants of index-locking can be used to implement the other schemes
(apart from 2PL)
Weak Levels of Consistency
Serializability allows programmers to ignore issues related to concurrency
when they code xacts.
• Degree-Two Consistency: Purpose is to avoid cascading aborts w/o
necessarily ensuring serializability; S-locks may be acquired and re-
leased at any time; X-locks can be acquired at any time, but cannot be
released until the xact aborts or commits; results in non-serializable
schedules; therefore, this approach is undesirable for many apps
• Cursor Stability: Form of two-degree consistency for programs
written in host languages where iteration of tuples is done using a
cursor; Instead of locking the entire relation, the tuple that is cur-
rently being processed is locked in S-mode; Any modified tuples are
locked in X-mode until the xact commits; 2PL is not used, Serializ-
ability is not guaranteed; Heavily accessed relations gain increased
concurreny and improved system performance. Programmers must
take care at the app level so that db consistency is ensured.
• Weak Levels of Consistency in SQL: SQL-92 levels:
– Serializable (default)
– Repeatable Read (xact may not be serializable wrt other xacts;
e.g., when an xact is searching for records satisfying some con-
ditions, the xact may find some records inserted by a committed
xact, but not others)
– Read committed
– Read uncommitted (lowest level of consistency allowed in
SQL-92)
Concurrency in Index Structures
Since indices are accessed frequently, they would become a point of great
lock contention, leading to a low degree of concurrency. It is acceptable to
have nonserializable concurrent access to an index, as long as the accuracy
of the index is maintained.
2 technique:
• Crabbing Protocol:
– When searching, lock root node in shared mode, then the child
node. After acquiring lock on child, release lock on parent.
– When inserting, traverse tree as in search mode. Then, lock
the node affected in X-mode. If coalescing, splitting or redis-
tribution is required, lock the parent in X-mode; then perform
the operations on the node(s) and release the locks on the node
and the siblings; retain lock on parent, if parent needs further
splitting, coalescing, or resitribution.
– Progress of locking goes from top to bottom while searching
and bottom to up when splitting, coalescing, redistributing
• B-link-tree locking protocol: Achieves more concurrency by avoid-
ing holding the lock on one node while holding lock on another node,
by using a modified version of B+-trees called B-link trees; these
require that every node including the internal nodes and leaf nodes
maintain a pointer to its right sibling
– Lookup: Each node must be locked in S mode before access-
ing it; Split may occur concurrently with lookup, so the search
value may have moved to the right node; Leaf nodes are locked
in 2PL to avoid phantom phenomenon
– Insertion and deletion: Follows the rules to locate the lead node
into which the insertion or deletion will take place; Upgrades
the shared lock to X lock on the affected leaf; Leaf nodes are
locked in 2PL to avoid phantom phenomenon
– Split: Create the new node (split); change the right-sibling
pointers accordingly; release X-lock on the original node (if
it is non-leaf; leaf nodes are locked in 2PL to avoid phantom
phenomenon)
– Coalescing: Node into which coalescing will be done should
be locked in X mode; once coalescing has been done, parent
node is locked in X mode to remove the deleted node; then,
xact releases the locks on the coalesced nodes, if parent is not
to be coalesced, lock on parent can be released
– Note: An insertion or deletion may lock a node, ulock it, and
subsequently relock it. Furthermore, a lookup that runs concur-
rently with a split or coalescence operation may find that the
desired value has shifted to the right-sibling node by the split
or coalescence operation; this can be accessed by following the
right-sibling pointer.
– Coalescing of nodes can cause inconsistencies; lookups may
have to restart
– Instead of 2PL on leaf nodes, key-value locking on individ-
ual key-values can be done. However, must be done carefully;
else, phantom phenomenon can occur for range lookups; can be
taken by locking one more key value than the range (next-key
value).
Recovery System
• Fail-stop assumption: Hardware errors and bugs in software bring
the system to a halt, but do not corrupt the nonvolatile storage con-
tents.
• Stable Storage Implementation: Keep 2 physical blocks for each
logical database block. Write the info onto the first physical block.
 When the first write completes successfully, write the same info onto
the second physical block. The o/p is completed only after the second
write completes successfully.
During recovery, the system examines each pair of physical blocks.
If contents same, nothing to be done. If error in one, replace with the
other. If contents differ, replace first’s contents with the contents of
the second block.
Number of blocks to compare can be reduced by keeping list of on-
going writes in NVRAM (so that only these need to be compared).
Log-Based Recovery
Recovery is used for rolling back transactions as well as for crash recovery.
Update log record has: Xact Id, Data-item id, Old Value, New Value
• <T Start>, <T commit> or <T abort> records are written at start,
commit or abort of a transaction
• Deferred Database Modification: Only new values need to be stored
(for redo; no need for undo)
• Immediate Database Modification: Requires to store both old and
new values (for undo and redo)
– Undo is performed before redo
• Checkpoints: Helps in reducing scanning the log after a crash to lo-
cate the transactions to be undone and redone; also helps in reducing
the time to redo (since the changes before the checkpoint would have
been applied already).
Transactions are not allowed to perform any update actions, such as
writing to a buffer block or writing a log record, while a checkpoint
is in progress.
– Output all log records to stable storage
– Output all modified buffer blocks to stable storage
– Write the checkpoint record to stable storage
For recovery, the log must be scanned backward to find the most re-
cent checkpoint record. It needs to further continue searching back-
ward until it finds all the transactions that have some record after the
most recent checkpoint record. Only these transactions need to be
redone / undone. No commit record, do undo; else do redo.
• Recovery with Concurrency Control:
– List of active transactions are stored as part of the checkpoint
record
– The log can be used to rollback even failed xacts.
– If strict 2PL is used (that is, excl. locks till end of xact),
the locks held by an xact may be released only after the xact
has been rolled back. So, when an xact is being rolled back,
no other xact may have updated the same data item (the xact
should have locked the data item since it was to update it in the
first place). Therefore, restoring the old value of a data item
will not erase the effects of any other xact.
– Undo must be done by processing the log backward
– Redo must be done by processing the log forward
– For recovery: Scan the log backward until it finds the ¡check-
point L¿ record performing the following steps as it reads each
record while scanning backward:
∗ If ¡Ti commit¿ record found, add Ti to the redo list
∗ If ¡Ti start¿ record found and Ti is not on the redo list,
add to the undo list
∗ Finally, for all Ti in the checkpoint record list, that does
not appear in the redo list, add to the undo list. This is to
take care of long running xacts that may not have updated
anything since the checkpoint record was written.
∗ Undo must be done prior to redo
• WAL (Write-ahead logging): Before a block of data in main memory
can be output to the database (in non-volatile storage), all log records
pertaining to data in that block must have been output to stable stor-
age.
Strictly speaking, the WAL rule requires only that the undo info in
the log have been output to stable storage, and permits the redo info
to be written later. This is relevant only in systems where undo and
redo info are stored in separate log records.
• Dump database procedure: Output all log records to stable, then the
buffer blocks, copy the contents of the db to stable storage, then out-
put a log record ¡dump¿ onto the stable storage. To recover, only
records after the ¡dump¿ record must be redone. But copying of en-
tire db is impractical and the xact processing must be halted during
the dump. Fuzzy dumps can be used to allows xacts to be active while
the dump is in progress.
Advanced Recovery Techniques
Using logical logging for undo process for achieving more concurrency
(faster release of locks on certain structures such as B+-tree index pages)
Fuzzy Checkpointing:
• Normal checkpointing may halt the xact processing for a long time if
the number of pages to be written is large
• Allows xacts to modify buffer blocks once the checkpoint record has
been written
• While performing fuzzy checkpointing, the xact processing is halted
only briefly to make a list of buffers modified. The checkpoint is
record before the buffers are written out.
• The locks are released and xacts can modify the buffer blocks; the
checkpointing process proceeds to output the modified blocks in its
list in parallel. However, the block being written out by the check-
pointing process still needs to be locked; other blocks need not be.
• Concept of “last-checkpoint” record at a fixed position on disk can
be used to guard against failures. This record should be updated only
after ALL the buffers in the checkpoint’s list have been written to
stable storage.
ARIES
• Features:
– Uses LSN (log sequence number)
– Physiological Redo
– Dirty Page Table
– Fuzzy Checkpointing (allows dirty pages to be written contin-
uously in the background, removing in bottle necks when all
pages need to be written at once)
• LSN:
– Every log record has a unique LSN that uniquely identifies the
log record
– LSN is most often file number and an offset within that file
– Each page has an LSN that indicates the LSN of the last record
that modified that page.
– PageLSN is essential to ensure idempotence in the presence of
phsyiological redo operations
– Physiological redo cannot be reapplied to a page since it would
result in incorrect changes on the page
– Each log record contains a field called ”PrevLSN” that points to
the previous log record for this transactioon (helps in locating
transaction log records easily without reading the whole log)
– CLRs (Compenstation Log Records) have an additional field
UndoNextLSN that is used in the case of the operation-abort
log record to point to the log record that is to be undone next
• Dirty Page Table:
– Stores the list of pages that have been updated in the buffer
– For each page, the PageLSN and the RecLSN is also stored
– RecLSN indicates which log records have already been applied
to the disk version of the page
– Intially, when the page is brought in from the disk, the RecLSN
is set to the current end of the log
• Checkpointing:
– A checkpoint log record contains the Dirty Page Table and the
list of active transactions
– For each transaction, the checkpoint record also stores the last
LSN for that transaction
– A fixed position on the disk notes the LSN of the last complete
checkpoint log record
• Recovery: 3 Phases (in recovery):
 – Analysis Pass: Determines which xacts to undo, which pages
were dirty at the time of the crash, and the LSN from which the
redo pass should start.
– Redo Pass: Starts from a position determined during the anal-
ysis phase, and performs a redo, repeating history, to bring the
database to a state it was in before the crash.
– Undo Pass: Rolls back all xacts that were incomplete at the
time of the crash. Need to elaborate here about CLRs, etc.
While undoing, if a CLR is found, it uses the UndoNextLSN to
locate the next record to be undone; else it undoes the record
whose number is found in the PrevLSN field
• Advantages of ARIES:
– Recovery is faster (no need to reapply already redone records;
pages need not even be fetched if the changes are already ap-
plied)
– Lesser data needs to be stored in the log
– More concurrency is possible
– Recovery Independence (e.g., for pages that are in error, etc.)
– Savepoints (e.g., rolling back to a point where deadlock can be
broken)
– Allows fine-grained locking
– Recovery optimizations (fetch-ahead of pages, out-of-order
redo)
Remote Backup Systems
Several issues must be addressed:
• Detection of failure: Using “heartbeat” messages and multiple links
of communication
• Transfer of control: When original comes back up, it must update
itself (by receiving the redo logs from the old backup site and replay-
ing them locally). The old backup can then fail itself to allow the
recovered primary to take over.
• Time to recover: Hot-spare configuration can be used.
• Time to commit:
– One-safe: Commit as soon as commit log record is written to
stable storage at primary
– Two-very safe: Commit only when both primary and secondary
have written the log records to stable storage (problem is when
secondary is down)
– Two-safe: Same as Two-very safe when both primary and sec-
ondary are up; when secondary is down, proceed as One-safe)
Database System Architectures
Main Types: Client-Server, Parallel, Distributed
Centralized Systems:
• Coarse-grained parallelism:
A single query is not partitioned among multiple processors.
Such systems support a higher throughput; that is, they allow a
greater number of transactions to run per second, although individ-
ual transactions do not run any faster.
• Fine-grained parallelism: Single tasks are parallelized (split) among
multiple processors
Client-Server Systems:
Clients access functionality through API (JDBC, ODBC, etc.) or transac-
tional remote procedure calls
Server System Architectures:
2 types: Transaction-server v/s Data-server systems
• Transaction-server systems (aka query-server systems):
– Components of a Transaction-server system include:
∗ Server Processes
∗ Lock Manager Process
∗ Log Writer Process
∗ Database Writer Process
∗ Process Monitoring Process
∗ Checkpoint Process
– The shared memory contains all the shared data:
∗ Buffer Pool
∗ Lock Table
∗ Log Buffer
∗ Cached Query Plans
– Semaphores or ”Test and Set” atomic operations must be used
to ensure concurrent access to the shared memory
– Even if the system handles lock requests through shared mem-
ory, it still uses the lock manager process for deadlock detection
• Data-server systems (aka query-server systems):
– This architecture is used typically when:
∗ High-speed connection between clients and servers
∗ Client systems have comparable computational power as
those of servers
∗ Tasks to be executed are computationally intensive
– The client needs to have full backend functionality
Parallel Systems
• Speedup v/s Scaleup
• Factors affecting Scaleup / Speedup
• Interconnection Networks:
– Bus
√ √
– Mesh (max. distance is 2( n − 1) or n, if wrapping is al-
lowed from the ends)
– Hypercube (max. distance is log n)
• Parallel System Architectures: Shared-memory, Shared-disks,
Shared nothing, Hierarchical
– Hierarchical: Share nothing at the top-level(???), but internally
each node has either shared-memory or shared-disk architec-
ture)
Distributed Systems
• Reasons: Sharing data, Autonomy, Availability
• Multidatabase or heterogeneous distributed database systems
• Issues in distributed database systems: Software development cost,
Greater potential for bugs, Increased processing overhead
• Local-Area Networks, Storage Area Networks (SAN)
• Wide-Area Networks: Disconintuous Connection WANs v/s Contin-
uous Connection WANs
Distributed DB
• Each site may participate in the execution of transactions that access
data at one site, or several sites.
• The difference between centralized and distributed databases is that,
in the centralized case, the data reside in one location, whereas in the
distributed case, the data reside in several locations.
• Homogeneous Distributed DB: All sites have identical dbms soft-
ware, are aware of one another, and agree to cooperate in processing
users’ requests
• Heterogeneous Distributed DB: Different sites may use different
schemas and different dbms software, and may provide only limited
facilities for cooperation in transaction processing
Distributed Data Storage
Two approaches to storing a relation in a distributed db:
• Replication: Several identical copies of a relation are stored; each
replica at a different site. Full replication: a copy is stored at every
site
– Advantages: Availability, Increased parallelism (minimizes
movement of data between sites)
– Disadvantages: Increased overhead on update
– In general, replication increases the performance of and the
availability of data for read operations; but update transactions
incur greater overhead
 – Concept or primary copy of a relation
• Fragmentation: The relation is partitioned into several fragments, and
each fragment is stored at a different site
– Horizontal Fragmentation: Each tuple to one or more sites
ri = σPi (r). r is reconstructed using: r = r1 ∪r2 ∪r3 . . .∪rn
– Vertical Fragmentation: Decomposition of scheme of relation
(so that columns are at one or more sites) ri = πRi (r). The
original relation can be obtained by taking the natrual join of
all the fragmented relations. Primary key (e.g., tuple id) needs
to exist in each fragment.
– For privacy reasons, vertical fragmentation can be used for hid-
ing columns.
• Fragmentation and Replication can be combined
• Transparency: Users should get: Fragmentation transparency, Repli-
cation transparency, Location transparency
• To prevent name clashes: a name server can be used (single point-
of-failure) or site id prepended to each relation name. Aliases can
be used to map aliases to real names stored at each site. This helps
when the administrator decides to move a data item from one site to
another.
Distributed Transactions
• Need to worry about failure of a site or failure of communication link
while participating in a transaction
• Transaction Manager (handles ACID for local) and Transaction Co-
ordinator (coordinates the execution of both local and global transac-
tions initiated at its site)
• Transaction Coordinator Responsibilities: Start execution of a trans-
action, Break a xact into sub-parts and distribute to various sites, co-
ordinate termination of xact (abort or commit)
• Failures: failure of a link, loss of messages, network partition, failure
of a site
2PC
• Protocol: When all sites inform the coordinator that the transaction is
complete:
– P1: Coord sends all sites ¡prepare T¿, Sites reply with ¡ready
T¿ or ¡no T¿
– P2: Coord sends ¡commit T¿ or ¡abort T¿ (based on whether
all sites were ready to commit or not)
– All such comm. must be logged to stable storage before it sends
the msg. out so that recovery is possible
– In some implementations, each site sends ¡ack T¿ msg to the
coord. The coord records ¡complete T¿ after it receives ¡ack T¿
from all the sites
• Handling of failures:
– Failure of site: Handling by coordinator:
∗ If site failed before replying ¡ready¿, the coord treats it
similar to a reply of ¡abort¿
∗ If site failed after replying ¡ready¿, the coord ignores the
failure and proceeds normally (the site will take care after
it comes back up)
Handling by site: When the site comes back up, it checks it log:
∗ If no control records in log, execute undo
∗ If commit in log, commit the xact
∗ If abort in log, execute undo
∗ If ¡ready¿ is present in log, it needs to find out from the co-
ord about the status. If coord is down, it can ask the other
sites. If this info is not available, then the site can neither
commit nor abort T. It needs to postpone the decision for
T until it gets the needed info.
– Failure of coord: When coord fails, then the participating sites
must try to determine the outcome (but cannot be done in all
cases)
∗ If site has ¡commit T¿, then it needs to commit the xact
∗ If site has ¡abort T¿, then it needs to undo
∗ If site does not have ¡ready T¿, then it can undo
∗ Otherwise, site has a ¡ready T¿. In this case, it must wait
for the coord to recover. This is the “blocking problem”.
If locking is used, other transactions may be forced to
wait.
– Network Partition:
∗ If the coord and all participants are in the same partition,
then no effect.
∗ Otherwise, the sites that are in the partition other than the
coord, treat the failure as if the coord failed. Similarly, for
the sites in the same partition as the coord and the coord,
they treat the failure as if the sites in the other partition
had failed.
• To allow the recovered site to proceed, the list of items locked can
also be recorded with the ¡ready T¿ message in the log. The recovery
proceeds to relock those items, whereas other xact can proceed.
3PC
• Tries to avoid blocking in certain cases by informing at least “k” other
sites of its decision
• It is assumed that no network partition occurs and not more than “k”
sites fail, where “k” is a predetermined number
• If the coord fails, then the sites elect a new coord. The new coord
tries to find out if any site knows about the old coord’s intentions. If
it finds any one site, then it starts the third phase (to commit or abort).
If it cannot, the new coord aborts the xact.
• If a n/w partition occurs, it may appear to be the same as “k” sites
failing and blocking may occur.
• 3PC has overheads; so it is not used widely.
• Also, it should be implemented carefully; else the same xact may be
committed in one partition and may be aborted in another
Alternative methods of xact processing
Using persistent messaging; this requires complicated error handling (e.g.,
by using compensating xacts). Persistent messaging can be used for xacts
that cross organizational boundaries.
Implementation of persistent messaging:
• Sending site protocol: Messages must be logged to persistent stor-
age within the context of the same xact as the originating xact be-
fore sending it out; On receiving an ack from the receiver, this can
be deleted. If no ack is recd., the site tries repeatedly. After pre-
determined number of failures, error is reported to the application
(compensating xact must be applied).
• Receiving site protocol: On receipt, the receiver must first log into
persistent storage; Duplicates must be rejected; After the xact for
logging the message to the log relation commits, the receiver send
an ack. Ack is also sent for duplicates. Deleting received messages
from the receiver must be done carefully, since the ack may not have
reached the sender and a duplicate may be sent. Each message can be
given a timestamp to deal with this problem. If the ts of a recd. msg.
is older than some predetermined cutoff, then that msg is discarded
and all other messages recorded that have ts older than the cutoff can
be deleted.
Concurrency Control in Dist. DB
• Each site participates in the execution of a commit protocol to ensure
global transaction atomicity.
Locking Protocols:
• Single Lock-Manager Approach:
– A single lock manager (residing at a single site) for the entire
system
– Request for lock is delayed until it can be granted; message is
sent to the site from which the lock request was initiated.
– The xact can read from any of the site where the replica is avail-
able; but all sites where a replica of the data item exists must
be involved in the writing.
 – Advantages: Simple Implementation, Simple Deadlock Han-
dling
– Disadvantages: Bottleneck, Vulnerability (Single point-of-
failure)
• Distributed Lock Manager:
– Lock Manager function is distributed over several sites
– Each site maintains a lock manager whose function is to ad-
minister the lock and unlock requests for those data items that
are stored at that site
– this works as for the single case when the data item is not repli-
cated
– for replicated case, see the methods below
– Advantages: Simple implementation; reduces the degree to
which the coord is the bottleneck; reasonably low overhead
requiring 2 messages for lock requests and one for unlock re-
quests.
– Disadvantages: Deadlock handling is more complex since the
lock/unlock requests are not made at a single site. There may
be intersite deadlocks even when there is no deadlock within
a single site.
• Primary Copy:
– Single primary copy for each replicated data item
– Lock / unlock requests are always made to the site that has the
primary copy
– This is handled similar to the case for unreplicated data
– Advantages: Simple Implementation
– Disadvantages: Single point-of-failure (if the site that has the
primary copy fails, the data item is inaccessible, although other
sites containing a replica may be accessible)
• Majority Protocol:
– If a data item Q is replicated at n different sites, then a lock re-
quest must be sent to more than one-half of the n sites in which
Q is stored; the transaction proceeds only when more than one-
half of the n sites grant a lock on the data item Q; otherwise, it
is delayed
– Writes are performed on all replicas
– Protocol can be extended to deal with site failures (see later
points ¡which one¿)
– Advantages: Distributed lock manager functionality
– Disadvantages: More complicated to implement; requires at
least 2(n/2 + 1) messages for handling lock requests and at
least (n/2 + 1) messages for handling unlock requests; Dead-
lock handling is more complicated - deadlocks can occur even
if a single data item is being locked (unless the requests are
made to the sites in the same predetermined order by all the
sites)
• Biased Protocol:
– Requests for shared locks are given more favorable treatment
– Shared locks: Request from one site that has a replica of Q
– Exclusive locks: Request locks at all sites that have a replica of
Q
– Advantages: Lesser overhead on read operations than the ma-
jority protocol; savings are significant when reads are more
– Disadvantages: Writes require more overhead; same complex-
ity for deadlock handling as for the Majority Protocol
• Quorum Consensus Protocol:
– Generalization of the majority protocol
– Each site is assigned a nonnegative weight; read and write oper-
ations are assigned 2 integers called read quorum Qr and write
quorum Qw
– Following condition must be satisfied: Qr + Qw > S and
2 ∗ Qw > S, where S is the total weight of all the sites at
which the data item exists
– For read locks, enough replicas must be locked so that their
total weight ≥ Qr
– For write locks, enough replicas must be locked so that their
total weight ≥ Qw
– Advantages: The cost of read or write locking can be selec-
tively reduced by choosing the read and write quorums; by set-
ting appropriate weights, this protocol can simulate the major-
ity and biased protocols
• Timestamping:
– Each xact is given a unique timestamp that the system uses in
deciding the serialization order
– 2 methods for generating unique timestamps: (1). Centralized,
or (2). Distributed (concatenate the site id at the end of the lo-
cal unique timestamp - this is done to ensure that the global ts
generated in one site are not always greater than those gener-
ated in other sites)
– Handling of faster clocks: Use logical counter clock; When-
ever a transaction with timestamp ¡x,y¿ visits a site and x is
greater than the current value of local clock counter, set local
clock counter to x + 1 (Similar technique can be use for system
clock based timestamps)
Replication with weak degrees of consistency
• Master-slave replication: updates only at a primary site; xacts can
read from anywhere
• Multi-master replication: Update-anywhere replication
• Laxy propagation: instead of updating all replicas as part of the xact
performing the update.
• Updates at replicas as translated into updates at a primary site, which
are then propagates lazily to all replicas. (This ensures updates to an
item are ordered serially, although serializability problems can occur,
since xacts may read an old value of some other data item and use it
to perform an update)
• Updates are performed at any replica and propagated to all other
replicas. Can cause even more problems since the same data item
may be updated concurrently at multiple sites.
Deadlock Handling
• Deadlock can occur if the union of the local wait-for graphs contains
a cycle (even though each local wait-for graph is acyclic)
• Centralized deadlock detection: Global wait-for graph is maintained
at the central site and is updated whenever a new edge is removed
or inserted from one of the local wait-for graphs (or periodically or
whenever the coord needs to invoke the cycle detection algo)
• When a cycle is detected, the coord selects a victim to be rolled back
and it must notify all sites about this. The sites, in turn, roll back the
victim xact.
• May produce unnecessary roll-backs:
– False cycles: Message for adding edge arrives before message
for removing edge
– One of the xact was to be aborted: If an xact was to be aborted
for reasons other than the deadlock, it may be possible that the
deadlock would have been broken and there would not be the
need to select (another) victim
• Deadlock detection can be done in a distributed manner, but is more
complicated.
Availability
Multiple-links can be used between sites; however, multiple links may still
fail. So there are cases where we cannot distinguish between site failure and
network partition.
Must take care to ensure that these situations are avoided: 2 or more
central servers are elected in distinct partitions, and more than one partition
updates a replicated data item
• Majority-based Approach:
– Each data item stores with it a version number to detect when
it was last written to. This is updated on every write.
– If a data item is replicated at n sites, then the xact will not pro-
ceed until it has obtained locks from majority of those n sites
– Read operations look at all versions and choose the highest one
(the sites will lower numbered versions can be informed of the
new version)
 – Write ops write to all replicas that have been locked; the
version number is one more than the highest numbered one
amongst them
– This works even when a failed site comes back up (it will told
about its stale data). Site reintegration is trivial - nothing needs
to be done. This is since writes would have updated a majority,
while reads will read a majority of the replicas and find at least
one replica that has the highest version.
– Same version numbering technique can be used with the quo-
rum consensus to make it work in the presence of failures.
However, failures may prevent xacts from proceeding if some
sites are given higher weights.
• Read One, Write All Approach:
– Unit weights to all sites, read quorum = 1 and write quorum =
n (all sites)
– No need of version number since even one site failed will not
allow write to the data item to happen
– To allow this to work in the case of failures, we could use “read
one, write all available” - but there are severla complications
that can arise in the case of network partitions or temporary site
failures (the site will not know and may have to explicitly cathc
up). Inconsistencies can arise in the case of network partitions.
• Site Reintegration: The recovering site must ensure that it gets the
latest values and in addition must continue to receive updates as it is
recovering. An easy solution is to halt entire system temporarily, but
this is usually not feasible. Recovery of a link must be informed to
all sites.
• Comparison with Remote Backup: In remote backup, concurrency
control and recovery are performed at a single site (overhead with
2PC are avoided); only data and log records are shipped across.
Transaction code is only at one site. Remote backup system offer
a lower-cost approach to availability than replication. On the other
hand, replication can provide greater availability by having multiple
replicas and using the majority protocol.
• Coordinator Selection:
– When there is not enough information avlbl to continue from
the failed coord, the backup coord can abort all (or several) cur-
rent xacts and restart them under the control of the new coord
– bully algorithm: If some site is electing itself the coord, then
the site must wait to hear the election message within a prede-
termined time interval. If it does not hear this message, this site
will restart the election algo.
Distributed Query Processing
Must take into account: cost of data transmission over the n/w as well as the
hard disk access and the potential gain in performance from having several
sites process parts of the xact in parallel
• Query Transformation: Choose the replica for which the transmis-
sion cost is the lowest; can make use of the fact that the selection
only fetches tuples from a (fragmented) replica.
• Simple Join Processing:
– Ship all copies to S1
– Ship to S1 ; compute join; ship result to S3 ; compute join; ship
result to S1 (or roles interchanged)
– Need to worry about the volume of data being shipped; also,
indices may have to be re-created at the shipped site
• Semijoin Strategy: Compute r1 o n r2 o n ΠR1 ∩R2 (r1 ) Semijoin:
r1 n r2 = ΠR1 (r1 o n r2 ). That is, the semijoin selects those tuples
of relation r1 that contributed to r1 o
n r2
• Join Strategies that exploit parallelism: Pipelined-join technique can
be used, for example, for r1 o n r2 o
n r3 on r4
Hetergeneous Dist. DB
• Unified View of Data: Difficult because: endianness, ASCII v/s
EBCDIC, units of measurement, Strings in different languages
(“Cologne” v/s “Koln”)
• Query Processing: Wrappers for local schema to global schema map-
ping and back; mediator system do not bother about xact processing
LDAP
• Can be used for storing bookmarks, browser settings, etc.
• Provides a simple mechanism to name objects in a hierarchical fash-
ion
• RDN=value can be collected to form the full distinguished name
• Querying consists of just selections and projections; no joins
• Distributed Directory Trees: A node in a DIT (directory info tree)
may contain a referral to another node in another DIT; this helps in
distributed trees.
• Many LDAP implementations support master-slave replication and
multimaster replication even though replication is not part of the cur-
rent std.
Parallel Databases
Parallelism is used to: speedup (queries are executed faster because more
resources, such as processors and disks, are provided) and scaleup (increas-
ing workloads are handled without increased response time, via an increase
in the degree of parallelism)
I/O Parallelism
• Horizontal Partioning: Tuples of a relation are divided (or declus-
tered) among many disks, so that each tuple resides on one disk
• Partioning Techniques
– Round-robin (ith tuple to disk number Dimodn : ensures even
distribution of tuples across disks (each disk has approx. the
same number of tuples)
– Hash partioning (hashing is on the chosen partioning attributes
of the tuples)...if the has function returns i, the tuple is placed
on disk Di
– Range partitioning: Contiguous attribute-value ranges to each
disk based on a partitioning attribute.
• Comparison of partitioning technique based on the access technique:
scanning entire relation, point queries, range queries
– Round-robin: Good for sequential scan of entire data; bad for
range and point queries (since each of the n disks must be
searched)
– Hash-partioning: Best for point queries; also suited for sequen-
tial scans of the entire relation (since the hash function could
ensure that the data are evenly distributed); not good for range
(since all disks must be searched)
– Range-partitioning: Suited for range as well as point queries;
point-queries can be answered by looking at the partition vector
Range parititioning results in higher throughput while main-
taining good response time when a query is sent to one disk
(only a few tuples in the queried range - other disks can be
used for other queries). On the other hand, when many tuples
are to be fetched from a few disks, this could result in an I/O
bottleneck (hostpot) at those disks.
• Choice of partioning affects other operations such as joins; in general,
range or hash partioning are preferred to round-robin.
• If a relation consists of m disk blocks and there are n disks, then the
relation should be allocated to min(m, n) disks (preferably, try to fit
relations that fit in a block to a single disk).
• Handling of skew: Attribute-value skew and partition skew
– Attribute-value skew: all tuples with the same value for the
partitioning attribute end up in the same partition; can occur
regardless of whether range or hash partioning is used.
– Partition-skew: Load imbalance in the partioning, even when
there is no attr. skew; range partitioning may result in partition
skew, if the partition vector is not chosen carefully; partition
skew is less likely to occur with hash partitioning, if a good
hash function is chosen.
• Loss of speedup due to skew increases with parallelism
• Techniques to overcome skew:
– Balanced range-partitioning vector: Sort by partitioning attr.
and distribute equally (1/n); can still result in skew; also cost
for sorting is high
 – Use histogram to reduce skew further
– Use concept of virtual processors: mapping of real virtual pro-
cessors to real in round-robin
Interquery Parallelism
• Different queries or xacts execute in parallel with one another
• Xact throughput can be increased, but the response times of individ-
ual xacts are no faster than they would be if the xacts were run in
isolation
• Pimary use of interquery parallelism is to scaleup a xact processing
system to support a large number of xacts per second
• Not useful for speeding up long running tasks, since each task is ex-
ecuted sequentially
• Easiest form to support (esp. in shared-memory parallel system)
• Cache-coherency problem: can be solved by locking a page in mem-
ory before any read or write access and flushes the page to the shared
disk before it releases the lock
• Other way is to access the latest value from the buffer pool of some
other processor
Intraquery Parallelism
• Execution of a single query in parallel on multiple processors and
disks
• Useful for speeding up long-running tasks
• Intraoperation Parallelism: Speed up processing of a query by par-
allelizing the execution of each individual operation (such as sort,
select, project, and join)
• Interoperation Parallelism: Speed up processing by executing in
parallel the different operations in a query expr
• Both can be used simulatenously on a query
• Since the number of ops in a typical query is small compared to the
number of tuples processed by each operation, intraop parallelism
scales better with increasing parallelism. However, with relatively
small number of processors, both forms of parallelism are important
Intraoperation Parallelism:
• Parallel Sort using Range-Partioning Sort:
– Range partition the data as per the sorting attribute and send
it to the respective processors
– Each processor sorts within the range
– The final merge is trivial since the range partitioning in the first
phase ensures that all key values in processor Pi are less than
those in Pj , for all i ≤ j.
• Parallel Sort using Parallel External Sort-Merge:
– Each processor locally sorts the data on its disk
– Merging of the sorted runs is done similar to external sort-
merge. Merging can be done by range-partitioning the sorted
data at each processor and then each processor sending the val-
ues in each partition to the respective processor.
Could result in execution skew where each processor will be-
come a hot-spot when it is its turn to receive the tuples. To
avoid this, each processor send the first block of every parti-
tion, then the second block of every partition, and so on. As a
result, all processors receive data in parallel.
• Parallel Join using Partitioned Join:
– Works only for equi-joins or natural joins
– Range-parition or hash partition can be used to partition the 2
relations (r and s) to be joined
– Each processor can perform the join for the ith partition of r
and s
– To prevent skew, the range partitioning vector must be such that
the sum of the sizes of ri and si is roughly equal over all i
• Parallel Join using Fragment-and-replicate Join:
– Works for any kind of join
– Asymmetric Frag-and-rep join: Fragment one of the relations
“r” using any partitioning technique. Replicate the other rela-
tion “s”. Each processor performs the join of ri and s using
any join technique.
– (Symmetric) Frag-and-rep join: Fragment both of the relations
using any partitioning technique; the paritions need not be of
the same size. Each processor performs join of ri and sj .
– Asymm. frag-and-rep join is useful when one of the relations
“s” is smaller; it can be replicated to all processors.
• Parallel Join using Partitioned Parallel Hash Join:
– Hash paritition each relation and send to the respective proces-
sor
– As each processor receives the tuples, it performs a local hash
join (build and probe relation)
– Hybrid hash-join could also be used locally to cache the incom-
ing tuples in memory, and thus avoid the costs of writing them
and of reading them back in.
• Parallel Join using Parallel Nested Join:
– Asymm. frag-and-rep. can be used along with indexed nest
loop join at each processor
– The indexed nested loop join can be overlapped with the dis-
tribution of the tuples in “s” to reduce the costs of writing the
tuples of “s” to disk and to read them back.
• Other operations:
– Selection for a range can proceed in parallel at each proces-
sor whose range partition overlaps with the specified range of
values in the selection.
– Duplicate elimination can be parallelized using parallel sort-
ing technique or hash / range partitioning and eliminating the
duplicates locally at each processor.
– Projection w/o duplicate elimination can be done as tuples are
read in from the disk in parallel.
– Aggregation can be done in parallel by paritioning on the
grouping attributes and then computing the aggregate locally
at each processor. (Either range or hash partitioning can be
used). The cost of transferring tuples can be reduced by partly
computing the aggregate values before partitioning (and then
using partitioning as before).
• Cost of Parallel Evaluation of Operations: Start-up costs, Skew,
Contention for resources, Cost of assembling the final result
T otalT ime = Tpart + Tasm + max(T0 , T1 , . . . , Tn−1 , where
T part is the time for partitioning the relations.
A paritioned parallel evaluation is only as fast as the slowest of
the parallel executions.
Interoperation Parallelism:
• 2 forms: Pipelined Parallelism and Independent Parallelism
• Pipelined: Major advantage is that the intermediate results are not
written to disk; they are just fed to the other processors in the pipeline
• Independent: r1 join r2 can be computed independently of r3 join r4;
has lower degree of parallelism
Query Optimization:
• Avoid long pipelines (resources will be hoarded) and it will take time
for the first input to reach the last processor in the pipeline
• Advatnage of parallelism could get negated by the overhead of com-
munication
• Heuristic 1: Consider only evaluation plans that parallelize every op-
eration across all processors, and that do not use any pipelining
• Heuristic 2: Exchange-operator model: Exchange operators can be
introduced into an evaluation plan to transform it into a parallel eval-
uation plan
A large parallel databse system must also address these availability is-
sues: Resilience to failure of some processors or disks; On line reorganiza-
tion of data and schema changes.
Online index construction : should not lock the entire relation in shared
mode as it is done usually; instead it should keep track of updates that oc-
cur while it is active and incroporate the changes into the index being con-
structed.
XML
• Self-documenting (because of presence of tags)
• Format of the document is not rigid (e.g., extra tags)
• XML allows nested structures
• Wide-variety of tools available
• XML Schema Definitions:
– DTD (Document Type Definition)
– XSD (XML Schema Definition)
– Relax NG
• DTD:
– ELEMENT, ATTLIST, default values supported, #PCDATA,
(+, *, ? for repetitions), empty and any
– Attributes can have #REQUIRED or #IMPLIED
– ID, IDREF and IDREFS for uniqueness, references and list of
references (e.g., “owns”)
– Limitations of DTD:
∗ Text elements and attributes cannot be constrained to be
of specific types
∗ Difficult to specify unordered sets of subelements
∗ Lack of typing in IDs and IDREF or IDREFS
∗ (?) No support for user-defined types
• XSD:
– Specified in XML syntax
– Support for type checking (simple as well as user-defined (com-
plexType and sequence)
– Specification of keys and key references using xs:key and
xs:keyref
– Benefits over DTD:
∗ Text can be constrained to specific types or sequences
∗ Allows user-defined types
∗ Allows uniqueness and foreign-key constraints
∗ Integrated with namespaces to allow different parts of a
document to conform to different schemas
∗ Allows maximum and minimum value checking
∗ Allows complex types to be inherited through a form of
inheritance
Querying and Transformation:
• XPath, XQuery (FLWOR expressions), XSLT
• XPath:
– Nodes are returned in the same order as they appear in the doc-
ument
– @ is used for attributes
– /bank/account[bal > 400]/@account_no
– count function for counting the nodes matched
– — operator for union of the results
– // for slipping multiple levels, .. specifies parent
– function doc(name) allows to look into the document whose
name is specified
(e.g., doc("bank.xml")/bank/account)
• XQuery:
– Uses XPath and is based on XQL and XML-QL
– Uses FLWOR Expressions: for, let, where, order by, return
– for statement is like “from” in SQL
– return statement treats everything as plain text to be output ex-
cept for strings within which are treated as expressions to be
evaluated
– return can have nested queries
– User-defined functions and types are allowed
– some and every can be used for testing existential and universal
qualification
• XSLT:
– Templates are used
– “match” and “xsl:value-of select” are used
– xsl:key and xsl:sort
• API for XML Processing: DOM (Document Object Model) and SAX
(Simple API for XML)
– SAX is useful when the application needs to create its own data
representation of the data read from the XML document
Storage of XML Data
• Non-relational Data Stores: Flat file and special XML database (suf-
fers from no support for atomicity, transactions, concurrency, data
isolation and security)
• Relational Databases:
– Store as a String: Database does not know the schema of the
stored elements; searching is inefficient; additional fields may
be stored (at the cost of redundancy) for indexing; function in-
dices can be used A
– Tree Representation: nodes(id, type, label, value) and
child(child id, parent id); position can be added, if order must
be preserved; many XML queries can be converted to relational
ones; disadvantage of large number of joins
– Map to Relations: All attributes are stored as string-valued at-
tributes of the relation; if subelement of simple type, add as an
attribute of the relation; else, add as a separate relation; par-
ent id needs to be added; position can be added for position;
relations that can occur at most once can be “flattened” into
the parent relation by moving all their attributes into the parent
relations
– Publishing and Shredding XML Data: “Publish” means “to
XML from relational”; “Shredding” means “to relational from
XML”
∗ Publishing: An XML element for every tuple and every
column of the relation as a subelement of the XML ele-
ment (more complicated for nesting)
∗ Shredding: Similar to “Map to Relations”
– Native Storage within a Relational Database: Using CLOB
and BLOB; binary representations of the XML can be stored
directly as a BLOB; some dbs provide xml data type; Xquery
can be executed on a XML document within a row and a SQL
query can be used for iterating over the required rows
– SQL / XML: XML extensions to SQL; xmlelement, xmlat-
tributes, xmlforest, xmlagg, xmlconcat
XML Applications:
• Storing Data with complex structure (such as bookmarks)
• Standardized Data Exchange Formats (e.g., ChemML, RosettaNet)
• Web Services (SOAP) - Web Services provide a RPC call interface
with XML as the mechanism for encoding parameters as well as re-
sults
• Data Mediation (collecting data from various web sites / sources and
presenting a single XML view to the user; e.g., showing user’s bank
account details from various banks)
Additional Research Papers
Mention about tree algo here...keeping id and pre-order numbering, etc.
Advanced Transaction Processing
TP-Monitor Architectures:
• Process-per-client model
• Single-server model (multithreaded; a bug in one app can affect all
other apps; not suited for parallel or distributed databases)
• Many-server, single-router model (PostgreSQL, Web apps)
• Many-server, many-router model (very high performance web sys-
tems, Tandem Pathway)
Main Memory DB:
Since disk I/O is often the bottleneck for reads/writes, we can make the db
system less disk bound by increasing the size of the database buffer. Since
memory sizes are increasing and costs are decreasing, an increasing number
of apps can be expected to have data fit into main memory. Larger main
memories allow faster processing of transactions, since data are memory
resident. But there are still disk-related limitations:
• Log records must be written to stable storage (logging process will
become a bottleneck). Could use NVRAM or group commit to re-
duce the overhead imposed by logging.
• Buffer blocks marked as modified by committed xacts still have to be
written so that the amount of log that needs to be replayed at recovery
time is reduced.
• After a crash recovery, even after recovery is complete, it takes some
time before the db is fully loaded in main memory
Opportunities for optimization:
• Data Structures with pointers can be used across pages (unlike those
on the disk)
• There is no need to pin pages in memory before they are accessed,
since buffer pages will never be replaced
• Query-processing techniques should be designed to minimize space
overhead (otherwise, main memory limits may be exceeded and
swapping may take place slowing the query processing)
• Operations such as locking and latching may become bottlenecks -
these should be improved.
• Recovery algos can be optimized, since pages rarely need to be writ-
ten out to make space for other pages.
“Group commit” is to reduce the overhead of logging by delaying writes
to a log until a batch is ready. This results in a slight delay in the commit of
transactions that perform updates.
Real-Time Xact Systems:
Systems with deadlines are called “real-time systems”.
• Hard deadline: Serious problems, such as system crash, may occur if
a task is not completed by its deadline.
• Firm deadline: The task has zero value if not completed by its dead-
line.
• Soft deadline: The task has diminishing value if it completed after
the deadline.
• Pre-emption of lock or rolling back a xact may be required
• Variance in xact execution time (disk access v/s in memory, locking,
xact aborts, etc.) can cause difficulty in supporting real-time con-
straints.
Long-Duration Xacts:
• Properties:
– Long duration (human interaction)
– Exposure of uncommited data
– Subtasks: User may want to abort a subtask only without
rolling back the entire xact
– Recoverability: Aborting a long-duration interactive xact be-
cause of a system crash is unacceptable
– Performance: Fast response time is expeted in contrast to
throughput (number of xacts per second)
• Nonserializable Executions: Enforcement of serializability can cause
problems for long-duration xacts
– 2 PL: Longer waiting times (since data items locked are not re-
leased until no other data items are needed to be locked). This,
in turn, leads to longer response time and increased chance of
deadlock.
– Graph-based protocols: An xact may have to lock more data
than it needs. Long-duration lock waits are likely to occur.
– Timestamp-based protocols: No waiting for locks, but xact
could get aborted. Cost of aborting a long -duration xact may
be prohibitive
– Validation protocols: Same as that for timestamp-based proto-
cols
Thus, it appears that the enforcement of xact atomicity must either
lead to an increased probability of long-duration waits or create a
possibility of cascading rollback.
• Concurrency Control:
– Correctness may be achievable without serializability
– Could split db into sub-dbs on which concurrency can be man-
aged separately
– Could use concurrency techniques that exploit multiple ver-
sions
• Nested and Multilevel xacts:
– A long-duration xact may be viewed as a set of sub xacts
– If a sub xact of T is permitted to release locks on completion,
T is called a “multilevel xact”
– If locks held by a sub xact of T are automatically assigned to T
on completion of the sub xact, it is called “nested xact”
• For large data items:
– Difficult to store both old and new values; therefore, we can
use the concept of logical logging
– Shadow-copy technique can be used to keep copies of pages
that have been modified
Xact Mgmt in Multidb
See Practice Exercise 25.5 and 25.8
Strong correctness
Two-Level Serializability (2LSR): Ensure serializability at 2 levels:
• Each local db system ensures local serializability among its local
xacts, including those that are part of a global xact
• The mutidb system ensures serializability among the global xacts
alone - ignoring the orderings induced by the locak xacts.
• Global-read protocol: allows global xacts to read, but not to update
local data item, while disallowing all access to global data by local
xacts
– Local xacts access only local data items
– Global xacts may access global data items, and may read local
data items (though they must not write local data items)
– There are no consistency constraints between local and global
data items
• Local-read protocol: allows local xacts to read global data, but disal-
lows all access to local data by global xacts
– Local xacts may access local data items, and may read global
data items stored at that site (though they must not write global
data items)
– Global xacts access only global data items
– No xact may have a value dependency (A xact has value de-
pendency if the value that it writes to a data item at one site
depends on a value that it read for a data item on another site).
• Global-read-write / local-read protocol: most generous; allows global
xacts to read and write local data, and allows local xacts to read global
data.
– Local xacts may access local data items, and may read global
data items stored at that site (though they must not write global
data items)
– Global xacts may read and write global as well as local data
items
– There are no consistency constraints between local and global
data items
– No xact may have a value dependency (A xact has value de-
pendency if the value that it writes to a data item at one site
depends on a value that it read for a data item on another site).
Ticket-based systems can also be used.
Data Warehousing
• A data warehouse is a repository of data gathered from multiple
sources and stored under a common, unified database schema.
• 2 types of db apps: Transaction Processing and Decision Support
• Transaction Processing: Record info about transactions
• Decision Support: Aim to get high-level of info from the detailed
info stored in transaction-processing systems, and to use the high-
level info to make decisions
• DSS aim to get high-level information from the detailed information
stored in a transaction-processing system.
• Issues related DSS:
– OLAP deals with tools and techniques that can give nearly in-
stantaneous answers to queries requesting summarized data,
even though the database may be extremely large
– Database query languages are not suited to the performance of
detailed statistical analyses of data (SAS and S++ do much bet-
ter)
– For performance as well as for organization control, data
sources will not permit other parts to retrieve data. DW gather
data from multiple sources under a unified schema at a single
site
– Data Mining combined knowledge-discovery with efficient im-
plementations that can be used on extremely large databases
• Measure attributes: Those that can be measured (e.g., price, quantity
sold)
• Dimension attributes: Other attributes; these are the dimensions on
which the measure attributes, and the summary of measure attributes,
are viewed.
• Multidimensional Data: Data that can be modeled as dimension attrs
and measure attrs are called multi-dimensional data
• Cross-tabulation (aka cross-tab or pivot table): A table where values
for one attribute form the row headers, values for another attributes
form the column headers, and where the cell values represent some
aggregate. For example, a table that has the sum of quantity sold for
item name as row headers and color as column headers for all sizes.
• A change in the data may result in more columns being added to the
cross-tab (e.g., when a new colored item is added to the data, it will
appear as a new column in the above cross-tab)
• Data Cube: Generalization of a cross-tab to “n” dimensions
• For a table with n dimensions, aggregation can be performed with
grouping on of the 2n subsets of the n dimensions. Grouping on the
set of all n dimensions is useful only if the table may have duplicates.
• Operations on a data cube:
– Pivoting: The operation of changing the dimensions used in a
cross-tab.
– Slicing / Dicing: Viewing the data cube for a particular value
of a dimension (aka called dicing particularly when the values
for multiple dimensions are fixed)
– Rollup: From finer to coarser granularity (e.g., rollup a table on
the size attribute) Drill Down: From coarser to finer granularity
• Hierarchy on dimensions: Date/Time (and Hour of day), Date, Month
(and Day of Week), Quarter, Year
• OLAP Implementation:
– MOLAP: OLAP cube stored in “multi-dimensional arrays”
– ROLAP: Relational OLAP (data stored in relational database)
– HOLAP: Hybrid OLAP (some in memory and some in rela-
tional)
• Simple Optimization: compute aggregates from an already computed
aggregration, instead of from the original relation; this does not work
for non-decomposable aggregate functions such as “median”
• For n dimension attributes, there can be 2n groupings
• SQL-1999 constructs:
– rank, dense rank, stddev, variance group by cube, group by
rollup, percent rank, ntile, cume dist
– Windowing: rows unbounded preceding, between rows 1 pre-
ceding and 1 following range between 10 preceding and cur-
rent row
Example: Finding the cumulative balance in an account, given
a relation specifying the deposits and withdrawals on an ac-
count
Data Warehousing:
• When and what data to gather: Source-based / Destination-based
(push / pull)
• What schema to use
• Data transformation and cleansing: merge-purge, deduplication,
householding, other types such as units of measurement
• How to propagate updates: Same as view-maintenance problem
• What data to summarize
• ETL (Extract, Transform, Load)
• Warehouse Schemas:
– Fact tables: tables containing multi-dimensional data
– Dimensional tables: To minimize storage requirements
(foreign-key looked up into other tables)
– Star schema, Snowflake schema
• Components of a DW: Data Loaders, DBMS, Query and Analysis
Tools (+ data sources)
Data Mining:
• Classifiers:
– Decision-Tree Classifiers
– Bayesian Classifiers (easier to construct than decision-tree
classifiers and work better in the case of null or missing attibute
values)
• Other types of data mining: clustering, text mining, data visualization
• TODO: Details about classifiers here
Advanced App Dev
• Benchmarks are standardized sets of tasks that help to characterize
the performance of db systems. They help to get a rough ideas of the
hardware and software requirements of an app, even before the app
is built.
• Tunable parameters at 3 levels:
– Hardware Level: CPU, Memory, Adding disks or using RAID
– DB System params: Buffer sizes, checkpointing intervals
– Higher level: Schema (indices), transactions
These must be considered together; a tuning at one level may result
in a bottleneck in another (e.g., tuning at a higher level may result in
a bottleneck at the CPU level)
• Tuning of hardware:
– For today’s disk, average access time is 10 ms and avg. transfer
rate of 25 MB/s
– A reduction of 1 I/O per second saves: (price per disk drive) /
(access per second per disk)
– Storing a page in memory costs: (price per MB of memory) /
(number of pages per MB of memory)
– Break-even point is:
price per disk drive price per MB of memory
n∗ =
access per second per disk pages per MB of memory
– 5-minute rule
– For sequentially accessed, we get 1-minute rule
– RAID 5 is much slower than RAID 1 on random writes: RAID
5 requires 2 reads and 2 writes to execute a single randown
write
– If an app performs r reads and w writes (random), then RAID
5 will require r + 4w I/O ops per second; RAID 1 will require r
+ w I/O ops per second
– If we take the current disks performance as 100 I/Os / second,
we can find the number of disks required (e.g., (r+w)/100).
This value is enough to hold 2 copies of all the data. For such
apps, if RAID 1 is used, the number of disks required is actu-
ally less than if RAID 5 is used.
 – RADI 5 is useful only when the data storage requirements are
large and the data xfer and I/O rates are small (that is, for very
large and very “cold” data)
• Tuning of schema: Use denormalized relation or materialized views
• Tuning of indices:
– Removing of indices may speed up updating
– For range queries, B+-tree indices are preferable to hash in-
dices
– If most number of queries and updates are clustered, clustered
indices could be used
• Materialized Views: Using deferred view maintenance reduces the
burden on updates.
• Automated tuning of physical design:
– Greedy heuristics: Estimates costs of using materialized differ-
ent indices / views and the cost of maintaining it
– Choose the one that prvoides max benefit per unit storage space
– Once this has been chosen, recompute the cost of other indices
/ views
– Continue the process until the space avaiable for storing the
mat. indices / views is exhausted or the cost of maintaining
the remaining candidates is more than the benefit to the queries
that could use indices / views.
• Tuning of transactions:
– Improve set of orientation (e.g., by using proper group by or by
using stored procedures)
– Reduce lock contention (maybe use weaker levels of consis-
tency)
– Minibatch transactions
Performance Benchmarks
Use harmonic mean for different xact types:
n – Ranges may overlap (as compared to the B+-trees, k-d trees
1
+ 1
+ ... + 1 and quadtrees.
t1 t2 tn
• TPC-A: Single type xact that models cash withdrawal and deposit at
a bank teller (not used widely currently)
• TPC-B: Same as TPC-A but focuses only on back-end db server) (not
used widely currently)
• TPC-C: More complex system model; order entry, etc.
• TPC-D: For decision support (scale factor is used - scale factor of 1
represents the benchmark on a 1 GB db)
• TPC-R: Db is permitted to use mat. views and other redundant info
• TPC-H: Ad-hoc (prohibits mat. views and other redundant info)
• TPC-W: Web commerce (performance metrics are in WIPS - Web
instructions per second)
• App migration: Big-bang approach v/s Chicken-little approach
Spatial Data
• Nearness Queries and Region queries (inside a region, etc.)
• Hash joins and sort-merge joins cannot be used on spatial data
Indexing of Spatial Data
• k-d Trees:
– Partitioning is done along one-dimension at the node ath the
top level of the tree, along another dimension in nodes at the
next level (and so on, cycling through all the dimensions)
– One-half point in one partition and one-half in the other
– Partioning stops when a node has less than given max. number
of points
– Each line in the diag. corresponds to a node in the k-d tree
– k-d-B tree extends the k-d tree to allow multiple child nodes
for each internal node (just like B-tree extends a binary tree) to
reduce the height of the tree. k-d-B trees are better suited for
secondary storage than k-d trees.
• Quadtrees:
– Each node of a quadtree is associated with a rectangualr region
of space.
– Each non-leaf node divides its region into 4 equal-sized quad-
rants
• R-trees:
– A rectangular bounding box is associated with each tree node
– A search for objects containing a given point has to follow all
child nodes whose bounding boxes contain the point
– The storage efficiency of R-trees is better than that of k-d trees
or quadtrees, since a object is stored only once. However,
searching is not efficient in R-trees since multiple paths may
have to be searched. Inspite of this, R-trees are popular (be-
cause of space efficiency and similarity to B-trees)
Read insertion, deletion and searching from book