etdTT.27 Gallagher-etdTT.27 PDF

#EMCECS
Copyright 2015 EMC Corporation. All rights reserved.
ECS ARCHITECTURE
DEEP DIVE
#EMCECS
ROADMAP INFORMATION DISCLAIMER

EMC makes no representation and undertakes no obligations with
regard to product planning information, anticipated product
characteristics, performance specifications, or anticipated release
dates (collectively, Roadmap Information).
Roadmap Information is provided by EMC as an accommodation to the
recipient solely for purposes of discussion and without intending to be
bound thereby.
Roadmap information is EMC Restricted Confidential and is provided
under the terms, conditions and restrictions defined in the EMC NonDisclosure Agreement in place with your organization.
#EMCECS
@scale
accessible
unstructured
web
global
many devices
#EMCECS
5
MODERN APPS ARE CHANGING

INDUSTRIES
#EMCECS
MODERN APPS ARE BREAKING

TRADITIONAL STORAGE
NOT DESIGNED FOR MODERN APPLICATIONS AND HYPERSCALE
Architecture is too complex

Locking, replication, High Availability, geodistribution is complex
Not Web or firewall friendly

Distributed (WAN) access is complex
Storage silos impede development

Different hardware for every data type and
access protocol
#EMCECS
NEW DEMANDS FOR HYPERSCALE
SOFTWARE-DEFINED STORAGE AND COMMODITY ARE FOUNDATIONAL

The Potential
Reduce storage costs with

commodity off-the-shelf (COTS)
components and standard, open
technologies
7:1
Savings
The Reality
Individual components provide

lower performance, lower
reliability, etc.
You need sufficient scale, the right

software, and operational expertise
to provide a reliable, scalable
infrastructure
Copyright 2014
2015 EMC Corporation. All rights reserved.
Storage Appliance
Commodity Hardware
#EMCECS
Source: IDC DSS Tracker Q4 2014 , Dec. 2014
WHAT DO CUSTOMERS NEED?
Customer need simple, APIaccessible, software-based

storage
that can be deployed on
commodity off-the-shelf
components
in a hyperscale architecture
#EMCECS
CUSTOMER PAIN POINTS
Help Me
Reduce
Costs
Give Me
Freedom of
Choice
I Want a
Path to
the Cloud
#EMCECS
10
ECS DESIGN PHILOSOPHY
Software
Defined Storage
#EMCECS
11
RACK & STACK MODEL

SCALE OUT ARCHITECTURE
#EMCECS
12
ECS SOFTWARE STACK
#EMCECS
13
GEO-DISTRIBUTED CLOUD STORAGE

ACTIVE-ACTIVE ARCHITECTURE
#EMCECS
14
SYSTEM OVERVIEW
User & System data persisted
through the storage engine
/bucket
Scalable Key-Value pair
/images
/data
Multi-protocol access
/metadata
/docs
/data
/metadata
#EMCECS
15
Namespace
Admin
Namespace 1
Object User
ECS CONCEPTS
Namespace 2
VDC 2
VDC 1
Storage Pool
Storage Pool
Sys Admin
Replication Group
#EMCECS
16
BUILD YOUR OWN ECS (DIY)

HW & OS agnostic
Self-contained
Docker containers
Timeline
Dev/Test (now)
Production (EOY 2015)
#EMCECS
17
DEEP DIVE
#EMCECS
Copyright
Copyright 2015
2015 EMC
EMC Corporation.
Corporation. All
All rights
rights reserved.
reserved.
18
CONSISTENCY SEMANTICS
#EMCECS
19
CONSISTENCY
Strong consistency
Resilient to temporary site outage
#EMCECS
20
LAYERS IN SYSTEM
#EMCECS
21
DATA PATH
#EMCECS
22
TEMPORARY SITE FAILOVER
#EMCECS
23
GLOBAL STATE
MANAGEMENT
#EMCECS
24
GLOBAL STATE MANAGEMENT (VNEST)

Service discovery
Distributed lock management
Paxos based coordination
#EMCECS
25
ECS DESIGN
Performance optimized for small and large data sets
File system on Key-Value
#EMCECS
26
PARTITIONING SCHEME
Partition Table
Partition
ID
Owner
Node
P1
N1
P2
N1
P3
N2
P4
N2
A logical table is shared into

partitions, based on hashing for
random distribution of load.
A partition is owned by one node at
a given time. A node can own many
partitions.
Node 1
Node 2
P1
P3
P2
P4
#EMCECS
27
PARTITION COMPONENTS
A partition presents a table
of key-value.
Node 1
A node that owns a

partition, does the
execution of operations
that are associated with
the keys in that partition.
The key-value information
is persisted on chunks
using BTree and Journal.
P1
BTree
Journal
Chunks
#EMCECS
28
PERFORMANCE
#EMCECS
29
#EMCECS
30
CHUNK WRITE (PARALLEL SYNC

WRITE)
Acks
Node
Buffered Writer
#EMCECS
31
ECS AT EMC WORLD

ECS is Everywhere at EMC World!
On The Show Floor
ECS at CTD/ETD Booth 813 and kiosks #8 and #9
Breakout Sessions
EMC ECS:
Scale IT
EMC ECS:
Deploying
EMC ECS:
EMC ECS:
2015
ECS and Unstructured Content Strategies for Web

ECS and VMware vCloud Air - Architecting and
an Exabyte+ Cloud
ECS Architecture Deep Dive
ECS Storage for 3rd Platform and What's New in
HOLs (Hands-on Labs)
EMC ECS: Empowering Storage Admins for Cloud Storage
#EMCECS
32
ECS SOFTWARE STACK

Object
Storage Services
HDFS
ECS Portal Services
Licensing, Authentication, Multi-tenancy and Provisioning

Services
ECS Provisioning Services
Cluster manager, health, configuration management upgrade,
installer, alerting
OS + networking + tools
Fabric
Infrastructure
#EMCECS
34
TEMPORARY SITE FAILOVER
Successful operations
Create Object
List Bucket
Edit Bucket
List Buckets in
namespace on site-1
Read and Update Object
created on site-1 &
site-2
Failed Operations
x Create bucket
x Create User
x Create Namespace
Bucket ownership transferred to site-2

(secondary)
Read / Write objects succeed
Read and Update
Objects created on
site-1 & site-2
List Bucket
Create Objects
Edit Bucket
Site-1
Failed Operations
x Create Bucket
x Create User
x Create Namespace
Bucket A
Site-2
Bucket A
Read and Update
Objects created on
site-1 & site-2
List Bucket
Create Objects
Edit Bucket
Bucket A
Failed Operations
x Create Bucket
x Create User
x Create Namespace
Site-3
#EMCECS
35
TEMPORARY SITE FAILURE (EXISTING
Bucket A is owned by Site-1 (primary)

BEHAVIOR)
Create Object
List Bucket
created on site-1
Failed Operations
List Bucket created on
site-2
created on site-2
Create bucket
Edit bucket
Outage leads to temp site failure of Site-1

ECS detects the temp site outage
Successful
operations
Read and Update
Object created on
site-2
Failed Operations
x Create bucket,
user, namespace
x Edit bucket
x Create Object
x List Bucket
x Read and Update
Object created on
site-1
Site-1
Site-2
Bucket A
Bucket A
#EMCECS
36
TEMPORARY SITE FAILOVER 2 SITE SCENARIO
Create Object
List Bucket
Edit Bucket
List Buckets in
namespace on site-1
created on site-1 &
site-2
Failed Operations
x Create bucket
x Create Namespace
x Create User

(secondary)
Read / Write objects succeed
Read and Update
Objects created on
site-1 & site-2
List Bucket
Create Objects
Edit Bucket
Failed Operations
x Create Bucket
x Create Namespace
x Create User
Site-2
Site-1
Bucket A
Bucket A
#EMCECS
37

RECONCILIATION (RESYNC)
Objects created at site-1 and site-2 will be
Create Object
List Bucket
created on site-1 &
site-2
Create / edit Bucket
retained
Objects edited at site-2 will be the copy
retained
Creating buckets, user, namespace succeed
created on site-1 &
site-2
Create / edit Bucket
List Bucket
Site-2
Site-1
Bucket A
Bucket A
#EMCECS
38

etdTT.27 Gallagher-etdTT.27 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

etdTT.27 Gallagher-etdTT.27 PDF

Uploaded by

Copyright:

Available Formats

#EMCECS

Copyright 2015 EMC Corporation. All rights reserved.

ROADMAP INFORMATION DISCLAIMER

MODERN APPS ARE CHANGING

MODERN APPS ARE BREAKING

NOT DESIGNED FOR MODERN APPLICATIONS AND HYPERSCALE

Architecture is too complex

Not Web or firewall friendly

Storage silos impede development

NEW DEMANDS FOR HYPERSCALE

SOFTWARE-DEFINED STORAGE AND COMMODITY ARE FOUNDATIONAL

Reduce storage costs with

Individual components provide

You need sufficient scale, the right

WHAT DO CUSTOMERS NEED?

Customer need simple, APIaccessible, software-based

CUSTOMER PAIN POINTS

ECS DESIGN PHILOSOPHY

RACK & STACK MODEL

ECS SOFTWARE STACK

GEO-DISTRIBUTED CLOUD STORAGE

Scalable Key-Value pair

BUILD YOUR OWN ECS (DIY)

TEMPORARY SITE FAILOVER

GLOBAL STATE MANAGEMENT (VNEST)

A logical table is shared into

Copyright 2015 EMC Corporation. All rights reserved.

A node that owns a

Copyright 2015 EMC Corporation. All rights reserved.

CHUNK WRITE (PARALLEL SYNC

ECS AT EMC WORLD

ECS at CTD/ETD Booth 813 and kiosks #8 and #9

ECS and Unstructured Content Strategies for Web

HOLs (Hands-on Labs)

EMC ECS: Empowering Storage Admins for Cloud Storage

ECS SOFTWARE STACK

ECS Portal Services

Licensing, Authentication, Multi-tenancy and Provisioning

Copyright 2015 EMC Corporation. All rights reserved.

TEMPORARY SITE FAILOVER

Bucket ownership transferred to site-2

Copyright 2015 EMC Corporation. All rights reserved.

TEMPORARY SITE FAILURE (EXISTING

Bucket A is owned by Site-1 (primary)

Outage leads to temp site failure of Site-1

Copyright 2015 EMC Corporation. All rights reserved.

TEMPORARY SITE FAILOVER 2 SITE SCENARIO

Bucket ownership transferred to site-2

Objects created at site-1 and site-2 will be

Copyright 2015 EMC Corporation. All rights reserved.

You might also like