You are on page 1of 55

<Insert Picture Here>

Oracle Exadata Database Machine Overview

Copyright 2012, Oracle Corporation and/or its affiliates

Revised March 26, 2012

Exadata Database Machine


Best Platform to Run the Oracle Database

Best Machine for Data Warehousing Best Machine for OLTP

Best Machine for Database Consolidation

Unique Architecture Makes it


Fastest, Lowest Cost

Copyright 2012, Oracle Corporation and/or its affiliates

Exadata in the Marketplace


Rapid adoption in all geographies and industries

Copyright 2012, Oracle Corporation and/or its affiliates

Exadata Hardware Architecture


Scaleable Grid of industry standard servers for Compute and Storage Eliminates long-standing tradeoff between Scalability, Availability, Cost

Database Grid
8 Dual-processor x64 database servers OR 2 Eight-processor x64 database servers

Intelligent Storage Grid


14 High-performance low-cost storage servers

100 TB High Performance disk, or 504 TB High Capacity disk

InfiniBand Network
Redundant 40Gb/s switches Unified server & storage network

5.3 TB PCI Flash Data mirrored across storage servers

Copyright 2012, Oracle Corporation and/or its affiliates

Standardized and Simple to Deploy


All Database Machines are the same
Delivered ready-to-run Tested Highly supportable No unique configuration issues Identical to config used by Oracle Engineering

Runs existing OLTP and DW applications


Full 30 years of Oracle DB capabilities No Exadata certification required Deploy in Days, Not Months

Leverages Oracle ecosystem


Skills, knowledge base, people, partners

Copyright 2012, Oracle Corporation and/or its affiliates

Exadata Hardware

Copyright 2012, Oracle Corporation and/or its affiliates

Exadata Storage Server Building Block


Hardware by Sun

Software by Oracle

High-performance storage server built from industry standard components 12 disks - 600 GB 15000 RPM High Performance SAS or 3TB 7200 RPM High Capacity SAS 2 Six-Core Intel Xeon Processors (L5640) Dual ported 40 Gb/sec InfiniBand 4 x 96 GB Flash Cards

Intelligent Exadata Storage Server Software

Copyright 2012, Oracle Corporation and/or its affiliates

Start Small and Grow


Field Upgradeable

Quarter Rack

Half Rack

Full Rack

Balanced Incremental Scaling for OLTP and DW


Copyright 2012, Oracle Corporation and/or its affiliates

Exadata Storage Expansion Rack


Big Data for Less
Expands storage capacity of existing Exadata Database Machines and SuperCluster
Available with either 600GB High Performance disks or 3 TB High Capacity disks

Essentially Database Machine with database servers replaced by more Storage Servers For Unbalanced Applications that need more space but not more compute

Copyright 2012, Oracle Corporation and/or its affiliates

On-disk Backups (up to 27 TB/hour) Historical or Archive Data File Data, LOBs, XML Data, Documents Images, and other large unstructured data
9

Exadata Storage Expansion Racks


Expand Database Machine Storage Capacity Online
Quarter Rack Half Rack Full Rack Multi Rack
8+ Racks

Up to 144 TB Disk 1.5 TB Flash 4 Storage Servers 48 CPU cores

Up to 324 TB Disk 3.4 TB Flash 9 Storage Servers 108 CPU cores

Up to 648 TB Disk InfiniBand Connected 6.75 TB Flash 18 Storage Servers 216 CPU cores

Database Backups, Historical Data, Files, Images, XML


Copyright 2012, Oracle Corporation and/or its affiliates

10

Complete Family Of Exadata Machines


For OLTP, Data Warehousing & Consolidated Workloads Storage Expansion Exadata X2-8 Exadata X2-2

Copyright 2012, Oracle Corporation and/or its affiliates

11

Scale to 8 Racks by Just Adding Cables


Full Bandwidth and Redundancy

Scale to more than 8 Racks by adding InfiniBand switches


Copyright 2012, Oracle Corporation and/or its affiliates

12

Capacity on Demand
License the capacity you need when you need it! License the Oracle Database for only the database servers required initially Expand your database processing capability by licensing the Oracle Database for additional database servers as your needs grow License the Exadata Storage Server Software for the Exadata Storage servers that match your needs initially License the Exadata Storage Server Software for additional storage servers as your storage needs grow
Copyright 2012, Oracle Corporation and/or its affiliates

13

Capacity on Demand
Example Create a 3/4 Rack Situation: Need a Three-Quarter Rack
Need the storage capacity/performance of 10 Exadata Storage Servers Need the processing power of 6 database servers

Solution:
Purchase a X2-2 Full Rack (contains 8 database severs and 14 storage servers) License and use the Oracle Database on only 6 of the 8 database servers License and use the Exadata Storage Server Software on 10 of the 14 storage servers License remaining database servers and/or storage servers in the future, as needed
Copyright 2012, Oracle Corporation and/or its affiliates

14

Database Server Operating System Choices


Two Operating System Choices on the x64 database servers Oracle Linux Solaris 11 Customers will choose their preferred Database Server OS at installation time Exadata Storage Servers will continue to embed Oracle Linux

Copyright 2012, Oracle Corporation and/or its affiliates

15

Database Machine Capacity (Uncompressed)


X2-8 or X2-2 Full Rack Raw Disk Capacity1 High Perf Disk High Cap Disk 100 TB 504 TB 5.3 TB X2-2 Half Rack 50 TB 252 TB 2.6 TB 22.5 TB 112 TB 15 TB 75 TB X2-2 Quarter Rack 21.6 TB 108 TB 1.1 TB 9.5 TB 48 TB 6.5 TB 32 TB

Raw Flash Capacity2

Usable Mirrored Capacity 2,3


Usable Triple Mirrored Capacity2,4

High Perf Disk


High Cap Disk High Perf Disk High Cap Disk

45 TB 224 TB 30 TB 150 TB

1- Raw Disk Capacity defined using standard disk drive terminology of 1 TB = 1000 * 1000 * 1000 * 1000 bytes. 2- Capacity calculated using normal space terminology of 1 TB = 1024 * 1024 * 1024 * 1024 bytes. 3 - Actual space available for a database after mirroring (ASM normal redundancy) and allowing one disk (Quarter and Half) or two disks (Full Rack) of free space to automatically remirror after disk failures. 4 - Actual space available for the database computed after triple mirroring (ASM high redundancy).

Copyright 2012, Oracle Corporation and/or its affiliates

16

Database Machine IO Performance


X2-2 or X2-8 Full Rack
Disk Data Bandwidth1,3 Flash Cache Data Bandwidth1,3 Disk IOPS High Perf Disk

X2-2 Half Rack 12.5 GB/s 9 GB/s 37.5 GB/s 34 GB/s 25,000

X2-2 Quarter 5.4 GB/s 4 GB/s 16 GB/s 14.5 GB/s 10,800

25 GB/s 18 GB/s 75 GB/s 68 GB/s 50,000

High Cap Disk


High Perf Disk High Cap Disk High Perf Disk

High Cap Disk


Flash IOPS2,3 Data Load Rate4

28,000
1,500,000 12 TB/hr

14,000
750,000 6 TB/hr

6,000
375,000 3 TB/hr

1 - Bandwidth is peak physical scan bandwidth achieved running SQL, assuming no compression. Effective data bandwidth will be much higher when compression is factored in. 2 - IOPS Based on read IO requests of size 8K running SQL. Note that the IO size greatly effects flash IOPS. Others quote IOPS based on 2K, 4K or smaller IOs that are not relevant for databases. Exadata Flash read IOPS are so high they are typically limited by database server CPU, not IO. 3- Actual Performance varies by application. 4 Exadata load rates are typically limited by database server CPU, not IO. Rates vary based on load method, indexes, data types, compression, and partitioning

Copyright 2012, Oracle Corporation and/or its affiliates

17

Exadata Storage Expansion Capacity (Uncompressed)


Full Rack High Perf Disk High Cap Disk 128 TB 648 TB 6.75 TB High Perf Disk High Cap Disk High Perf Disk High Cap Disk 58 TB 288 TB 38 TB 192 TB Half Rack 64 TB 324 TB 3.4 TB 29 TB 144 TB 19 TB 96 TB Quarter Rack 28 TB 144 TB 1.5 TB 13 TB 64 TB 8.5TB 43 TB One Cell 7.2 TB 36 TB 0.375 TB 3.25 TB 16 TB 2.1 TB 10.75 TB Raw Disk Capacity1

Raw Flash Capacity2 Usable Mirrored Capacity 2,3 Usable Triple Mirrored Capacity2,4

1- Raw Disk Capacity defined using standard disk drive terminology of 1 TB = 1000 * 1000 * 1000 * 1000 bytes. 2- Capacity calculated using normal space terminology of 1 TB = 1024 * 1024 * 1024 * 1024 bytes. 3 - Actual space available for a database after mirroring (ASM normal redundancy) and allowing one disk (Quarter and Half) or two disks (Full Rack) of free space to automatically remirror after disk failures. 4 - Actual space available for the database computed after triple mirroring (ASM high redundancy).

Copyright 2012, Oracle Corporation and/or its affiliates

18

Exadata Storage Expansion Performance


Full Rack
Disk Data Bandwidth1,3 Flash Cache Data Bandwidth1,3 Disk IOPS High Cap Disk High Perf Disk High Cap Disk High Perf Disk High Cap Disk High Perf Disk

Half Rack 16 GB/s 11.5 GB/s 48.5 GB/s 44 GB/s 32,000 18,000 950,000

Quarter 7.2 GB/s 5 GB/s 21.5 GB/s 19 GB/s 14,400 8,000 425,000

Single Cell 1.8 GB/s 1.3 GB/s 5.4 GB/s 4.9 GB/s 3,600 2,000 107,000

32 GB/s 23 GB/s 97 GB/s 88 GB/s 64,000 36,000 1,900,000

Flash IOPS2,3

1 - Bandwidth is peak physical scan bandwidth achieved running SQL, assuming no compression. Effective data bandwidth will be much higher when compression is factored in. 2 - IOPS Based on read IO requests of size 8K running SQL. Note that the IO size greatly effects flash IOPS. Others quote IOPS based on 2K, 4K or smaller IOs that are not relevant for databases. Exadata Flash read IOPS are so high they are typically limited by database server CPU, not IO. This is especially true for expansion racks. 3- Actual Performance varies by application.

Copyright 2012, Oracle Corporation and/or its affiliates

19

Exadata Innovations

Copyright 2012, Oracle Corporation and/or its affiliates

20

Exadata Innovations

Exadata Storage Server Software


Intelligent storage
Smart Scan query offload Scale-out storage

Hybrid Columnar Compression


10x compression for warehouses 15x compression for archives

Smart Flash Cache


Accelerates random I/O up to 30x Doubles data scan rate

Data remains compressed for scans and in Flash

Uncompressed

primary backup

Benefits Multiply

test standby devt

Compressed

Copyright 2012, Oracle Corporation and/or its affiliates

21

Exadata Intelligent Storage Grid Most Scalable Data Processing


Data Intensive processing runs in Exadata Storage Grid
Filter rows and columns as data streams from disks (168 Intel Cores)

Example: How much product X sold last quarter


Exadata Storage Reads 10TB from disk Exadata Storage Filters rows by Product & Date Sends 100GB of matching data to DB Servers

Scale-out storage parallelizes execution and removes bottlenecks

Copyright 2012, Oracle Corporation and/or its affiliates

22

Simple Query Example


What were my sales yesterday?
Optimizer Chooses Partitions and Indexes to Access

Exadata Storage Grid

Oracle Database Grid


Select sum(sales) where Date=24-Sept

Scan compressed blocks in partitions/indexes


Retrieve sales amounts for Sept 24

SUM

10 TB scanned 1 GB returned to servers

Copyright 2012, Oracle Corporation and/or its affiliates

23

Exadata Intelligent Storage


Exadata storage servers also run more complex operations in storage
Join filtering Incremental backup filtering I/O prioritization Storage Indexing Database level security Offloaded scans on encrypted data Data Mining Model Scoring

Exadata Intelligent Storage Grid

10x reduction in data sent to DB servers is common

Copyright 2012, Oracle Corporation and/or its affiliates

24

Exadata is Smart Storage

Storage Server is smart storage, not a DB node


Storage remains an independent tier Compute and Memory Intensive Processing

Database Servers
Perform complex database processing such as joins, aggregation, etc.

Exadata Storage Servers


Search tables and indexes filtering out data that is not relevant to a query Cells serve data to multiple databases enabling OLTP and consolidation Simplicity, and robustness of storage appliance

Data Intensive Processing


Copyright 2012, Oracle Corporation and/or its affiliates

25

Exadata Storage Index


Transparent I/O Elimination with No Overhead Table
A B C D
1 3 5 5 8 3

Index

Exadata Storage Indexes maintain summary information about table data in memory
Store MIN and MAX values of columns Typically one index entry for every MB of disk

Min B = 1 Max B =5 Eliminates disk I/Os if MIN and MAX can never match where clause of a query Min B = 3 Completely automatic and transparent Max B =8

Select * from Table where B<2 - Only first set of rows can match
26

Copyright 2012, Oracle Corporation and/or its affiliates

Storage Index with Partitions Example


Orders Table
Order# Order_Date Partitioning Column Ship_Date Item

1
2 3

2007
2008 2009

2007
2008 2009

Queries on Ship_Date do not benefit from Order_Date partitioning


However Ship_date and Order# are highly correlated with Order_Date e.g. Ship dates are usually near Order_Dates and are never less

Storage index provides partition pruning like performance for queries on Ship_Date and Order#
Takes advantage of ordering created by partitioning or sorted loading

Copyright 2012, Oracle Corporation and/or its affiliates

27

Storage Index with Joins Example


Select count(*) from fact, dim
where fact.m=dim.m and dim.name=Camry

Dimension
Name M
Bloom filter constructed with min/max for M Perform IO and apply bloom filter

Fact
A M C D 1 3 5 5
Skip IO

Accord 1
Camry 3

Civic
Prius

5
8

Due to Storage Index

5 5

Copyright 2012, Oracle Corporation and/or its affiliates

28

Exadata Smart Flash Cache Extreme Performance OLTP & DW

Exadata has 5 TB of flash


56 Flash PCI cards avoid disk controller bottlenecks

Intelligently manages flash


Smart Flash Cache holds hot data Avoids large scan wipe-outs of cache Gives speed of flash, cost of disk

5X More I/Os than 1000 Disk Enterprise Storage Array

Exadata flash cache achieves:


Over 1.5 million IO/sec from SQL (8K) Sub-millisecond response times
29

Copyright 2012, Oracle Corporation and/or its affiliates

Exadata Smart Flash Log


Accelerate Transaction Response Times using Flash
Transaction Response Times
Default (on left) - Choppy response - High Outliers Smart Flash Log - 3x faster response - Much lower outliers Smart Flash Log Enabled

Uses Flash for Database Logs in a clever way


Flash is fast but has slow outliers Erase cycles, wear leveling, etc.

Smart Flash Log feature transparently uses Flash as a parallel write cache to disk controller cache Automatic and Transparent
Whichever write completes first wins (disk or flash)

Better response time and more throughput Uses almost no flash capacity (0.1% of capacity)
30

Copyright 2012, Oracle Corporation and/or its affiliates

Hybrid Columnar Compression Highest Capacity, Lowest Cost


Data is organized and compressed by column
Dramatically better compression

Speed Optimized Query Mode for Data Warehousing Query


10X compression typical Runs faster because of Exadata offload!

Space Optimized Archival Mode for infrequently accessed data


15X to 50X compression typical

Faster and Simpler Backup, DR, Caching, Reorg, Clone Benefits Multiply

Copyright 2012, Oracle Corporation and/or its affiliates

31

Hybrid Columnar Compression How it works


Compression Unit
Tables are organized into sets of a few thousand rows
Compression Units (CUs)

Within CU, data is organized by column, then compressed


Column organization brings similar values close together, enhancing compression

Useful for data that is bulk loaded and queried

Reduces 4x to 50x Table Size Reduction 4x to 40x

Update activity is light

Compared to best conventional algorithms Gzip, Bzip2


Typically 2X the compression, 10X the performance

Exadata servers offload filtering, projection, etc. for scans on compressed data
Indexed accesses return compressed blocks to database so buffer cache benefits from compression

Copyright 2012, Oracle Corporation and/or its affiliates

32

Compression Ratio of Real-World Data


Compression Ratio varies by customer and table Trials were run on largest table at 10 ultra large companies
Average revenue > $60 BB

Average Query Compression ratio was 13x


On top of Oracles already highly efficient format

Copyright 2012, Oracle Corporation and/or its affiliates

33

DBFS - Scalable Shared File System


Database Machine comes with DBFS shared Linux file system
Shared storage for ETL staging, scripts, reports and other application files

Files stored as SecureFile LOBs in database tables stored in Exadata


Protected like any DB data mirroring, DataGuard, Flashback, etc.

5 to 7 GB/sec file system I/O throughput

Load into database using External Tables

ETL Files in DBFS

ETL

More File Throughput than High-End NAS Filer


Copyright 2012, Oracle Corporation and/or its affiliates

34

Exadata In-Memory Database


Best of Disk, Flash, Memory in One Solution
Technology Disks Flash DRAM Units 168 56 PCI Cards 256 DIMM Size 504 TB 5 TB 4 TB Throughput 25 GB/sec 50 GB/sec 200 GB/sec

All Data stored on disk Warm data cached in flash


3x Bandwidth, 20x IOs/sec Much faster recovery from crash

Hot data cached in DRAM

Cost of Disk IOs of Flash Speed of DRAM

Effectively Infinite IOs/sec 4x Bandwidth of Flash


Note there is no 100x DRAM speedup in Exadata Flash and Disk much faster than typical systems

Copyright 2012, Oracle Corporation and/or its affiliates

35

Exadata I/O Resource Management


Mixed Workloads and Multi-Database Environment
Ensure different databases are allocated the correct relative amount of I/O bandwidth
Database A: 33% I/O resources Database B: 67% I/O resources

Database A

Database B

Ensure different users and tasks within a database are allocated the correct relative amount of I/O bandwidth
Database A: Reporting: 60% of I/O resources ETL: 40% of I/O resources Database B: Interactive: 30% of I/O resources Batch: 70% of I/O resources

InfiniBand Switch/Network

Exadata Cell

Exadata Cell

Exadata Cell

Copyright 2012, Oracle Corporation and/or its affiliates

36

Best Machine for Database Consolidation

ERP CRM

Exadata serves as farm/cloud for databases


Large memory enables many databases to be consolidated Extreme performance for complex workloads that mix OLTP, DW, batch, reporting I/O and CPU resource management isolates workloads

Warehouse
Data Mart HR

Copyright 2012, Oracle Corporation and/or its affiliates

37

First Secure Database Machine

Moves decryption from software to hardware


Over 5x faster

Near zero overhead for fully encrypted database Queries decrypt data at hundreds of Gigabytes/second

Copyright 2012, Oracle Corporation and/or its affiliates

38

Exadata Management

Copyright 2012, Oracle Corporation and/or its affiliates

39

Exadata Storage Management & Administration


Enterprise Manager
Manage & administer Database and ASM Monitor the Exadata Database Machine Hardware

Auto Service Request (ASR)


File SRs automatically for common hardware faults

Comprehensive CLI
Local Exadata Storage cell management Distributed shell utility to execute CLI across multiple cells

Embedded Integrated Lights Out Manager (ILOM)


Remote management and administration of hardware

Copyright 2012, Oracle Corporation and/or its affiliates

40

Oracle Enterprise Manager for Exadata Management Integrated View of Hardware and Software
Hardware view
Schematic of cells, compute nodes and switches Hardware components alerts

Software/system view
Performance, availability, usage by databases, services, clusters Software alerts db, cluster, ASM Topology view of DB systems/clusters

Configuration view
Version summary of all components along with patch recommendations

Copyright 2012, Oracle Corporation and/or its affiliates

41

ASR for Exadata


Customer Datacenter FRU replaced by Field Engineer
Oracle Field Engineer

Oracle Support Services

FRU dispatched by Support Engineer

Oracle Support Engineer

Fault occurs
Customer

SR creation email notification to customer

Service Request routed to Support Engineer


Oracle Case Management System

Product's auto-diagnosis facility sends SNMP trap to ASR Manager

ASR Manager

ASR Service

Fault telemetry securely transmitted to Oracle

Service Request (SR) created

Copyright 2012, Oracle Corporation and/or its affiliates

42

Quality of Service Management


The Oracle Approach in Action
Define and Enable Classify and Measure

QoS Policy Set


Maintenance Policy

Evaluate and Report

Server Pools

Weekend Policy After Hours Policy Business Hours Policy


Performance Objectives Business Rankings

Policy-Driven Architecture Performance


Classes

Analyze and Recommend

Server Pool Allocations

Implement and Control


Copyright 2012, Oracle Corporation and/or its affiliates

43

Integrated, Task-Based Management


New EM Quality of Service Management Tool

Copyright 2012, Oracle Corporation and/or its affiliates

44

Exadata Comparisons

Copyright 2012, Oracle Corporation and/or its affiliates

45

Faster than DW Appliances


Query Throughput Fastest Disk Throughput Much Faster with Flash
20
GB/sec Uncompressed Data Single Rack

75 GB/sec!

Flash

36

24 19
25

50 TB of data fits in Flash


Using 10x Query Compression

Flash

10 Disk

Flash

Disk

Effective Query Throughput on compressed data is even higher Netezza


Hundreds of GB/sec

Disk

Disk

Disk

Teradata

EMC Exadata

TwinFin 12

2650 4600 6680 Greenplum

Copyright 2012, Oracle Corporation and/or its affiliates

46

Faster than High End Arrays

75 GB/sec!

Flash

Exadata is fastest
and scales with more racks

Storage Data Bandwidth


(Uncompressed GB/sec)

25

Storage Arrays already cant deliver disk bandwidth


Disk

9.3

11

???

No extra bandwidth from Flash < 6 No CPU offload No Columnar Compression NetApp IBM IBM Hitachi EMC Exadata Exadata No InfiniBand 6080 XIV G3 DS8700 USP V VMAX V2 X2

Multiple Racks
Copyright 2012, Oracle Corporation and/or its affiliates

1 Rack
47

More Data Capacity


Exadata 10x Compression IBM Netezza TwinFin 2x to 4x Compression

Equivalent Capacity Systems


All with Largest Disks, Best Compression

4X Racks

EMC VMAX 3x Oracle Compression


EMC Greenplum DCA up to 4x Compression

3X Racks 3.5X Racks


for 2650 20X for 6680 41X for 4600
48

Teradata 1.4x Compression (block compression is archival)

7X

Copyright 2012, Oracle Corporation and/or its affiliates

Exadata Outperforms IBM Power7 for Less


More CPU cores More memory More I/O performance Same storage capacity
(excludes compression)

2 Exadata X2-8

$3,300,000
5.7:1 advantage*

HA configuration

$18,860,000
IBM P795 + 4 DS8700s w/ Flash
* Hardware list price comparisons 49

Copyright 2012, Oracle Corporation and/or its affiliates

Exadata Outperforms HP Superdome for Less


More CPU performance More I/O performance More storage capacity
(excludes compression)

1 Exadata X2-2

$1,100,000
8.5:1 advantage*

HA configuration

$9,420,000
HP Superdome 2 + XP24000 w/ Flash
* Hardware list price comparisons 50

Copyright 2012, Oracle Corporation and/or its affiliates

Turkcell: 10x Compression, 10x Speedup


250 TB warehouse compresses to 25 TB
1 Exadata rack 50,000 Reports run 10x faster each month (avg 27 min to 3 min) 1.5 Billion records (2-3 TB raw) loaded daily (data doubles yearly) Redundancy/HA built-in

25 TB compressed data

10:1 advantage
10 storage racks 1 large SMP server 250 TB raw data

Hitachi USP-V 5 Racks

EMC DMX-4 5 Racks


51

Copyright 2012, Oracle Corporation and/or its affiliates

Softbank Replaces 36 Teradata Racks


3 Exadata racks up to 8x faster
3 Exadata racks Billions of CDRs processed in 7 hours (from 25 with Teradata) Power, cooling, space savings Maintenance charges slashed

150% more data capacity

12:1 advantage

DBA

36 total racks
Twice the operational cost of Exadata

Copyright 2012, Oracle Corporation and/or its affiliates

52

Oracle Exadata Database Machine


Best Machine for
All Workloads Warehousing All Tiers Disk

OLTP
DB Consolidation
DB Consolidation Lower Costs Increase Utilization Reduce Management

Flash
Memory
Tier Unification Cost of Disk IOs of Flash Speed of DRAM

Copyright 2012, Oracle Corporation and/or its affiliates

53

Resources
Oracle.com:
http://www.oracle.com/exadata

Oracle Exadata Technology Portal on OTN:


http://www.oracle.com/technology/products/bi/db/exadata

Copyright 2012, Oracle Corporation and/or its affiliates

54

Copyright 2012, Oracle Corporation and/or its affiliates

55

You might also like