OpenStack Object Storage - Swift

1he CpenSLack
1M

Cb[ecL SLorage sysLem

ueploylng and managlng a scalable, open-source cloud
sLorage sysLem wlLh Lhe SwlfLSLack laLform

8y SwlfLSLack, lnc. conLacL[swlfLsLack.com
1he CpenSLack Cb[ecL SLorage sysLem Iebruary 2012
2012 SwlfLSLack, lnc. All rlghLs reserved.

2
ConLenLs
Introduct|on............................................................................................................... 4
Introduc|ng Sw|ft ....................................................................................................... S
!"#$% '()*)+%,*#-%#+-.............................................................................................................................. /
!"#$% #- !+)0)10, ....................................................................................................................................... /
!"#$% #- 23%*,4,05 67*)10,.................................................................................................................. 8
!"#$% #- 9:,; !<7*+, !<$%")*,........................................................................................................... 8
!"#$% #- !#4#0)* %< =>! !?................................................................................................................... @
!"#$% #- A7#0% <; B;C7-%*5D-%);C)*C '<4:<;,;%- ...................................................................... @
!"#$% '); A, 6,:0<5,C B;DE<7-, <* =-D)D!,*F#+,...................................................................... G
Us|ng Sw|ft............................................................................................................... 10
!"#$% '<44);C- D H(, A)-#+- ......................................................................................................... IJ
'0#,;% K#1*)*#,-...................................................................................................................................... II
now Sw|ft Works ..................................................................................................... 12
A7#0C#;L A0<+M-...................................................................................................................................... IN
O*<35 !,*F,*- ......................................................................................................................................... IN
H(, P#;L................................................................................................................................................... IN
Q<;,-R S)#07*, A<7;C)*#,- ............................................................................................................... I?
=++<7;%- T '<;%)#;,*-....................................................................................................................... IU
O)*%#%#<;-................................................................................................................................................. IU
P,:0#+)%#<;.............................................................................................................................................. IU
E<" %(,-, )*, )00 %#,C %<L,%(,*...................................................................................................... IV
Sw|ft C|uster Arch|tecture ........................................................................................ 17
=++,-- H#,*............................................................................................................................................... I8
!%<*)L, W<C,-........................................................................................................................................ I@
Conf|gur|ng Network|ng ........................................................................................... 20
K)*L,D!+)0, W,%"<*M#;L................................................................................................................... NJ
X,C#74D!+)0, W,%"<*M#;L .............................................................................................................. NI
X);)L,4,;% W,%"<*M....................................................................................................................... NI

3
nardware kecommendat|ons................................................................................... 22
O*<35 W<C,-............................................................................................................................................ NN
!%<*)L, W<C,-........................................................................................................................................ NN
W,%"<*M#;L............................................................................................................................................. N?
!#Y#;L Z<7* !"#$% '07-%,* .................................................................................................................. N?
|ann|ng a Dep|oyment ............................................................................................ 24
6)%)+,;%,* S)+#0#%5 O0);;#;L.......................................................................................................... NU
B;%,L*)%#<;-............................................................................................................................................ NV
X<;#%<*#;L .............................................................................................................................................. NV
=C4#;#-%*)%#<; T X)#;%,;);+, ..................................................................................................... N/
[;C,*-%);C#;L H'9............................................................................................................................. N/
P<00#;L !"#$% 97% #; Z<7* 9*L);#Y)%#<; ..................................................................................... N8
Manag|ng Sw|ft w|th Sw|ftStack............................................................................... 28
L|ke to Learn More? ................................................................................................. 29


4
lnLroducLlon
ln Lhls era of connecLed devlces, Lhe demands on sLorage sysLems are lncreaslng
exponenLlally. users are produclng and consumlng more daLa Lhan ever. Soclal medla,
onllne vldeo, user-uploaded conLenL, gamlng, and sofLware-as-a-servlce appllcaLlons
are all conLrlbuLlng Lo Lhe vasL need for easlly consumable sLorage sysLems LhaL can
grow wlLhouL bounds.
1o accommodaLe Lhese demands, sLorage sysLems musL be able Lo handle web-scale
workloads wlLh many concurrenL readers and wrlLers Lo a daLa-sLore. Some daLa ls
frequenLly wrlLLen and reLrleved, such as daLabase flles and vlrLual machlne lmages.
CLher daLa, such as documenLs, lmages, and backups are generally wrlLLen once and
rarely accessed. Web and moblle daLa asseLs also need Lo be accesslble over Lhe web
vla a u8L Lo supporL Loday's web/moblle appllcaLlons. A one-slze-flLs-all daLa sLorage
soluLlon ls Lherefore nelLher pracLlcal nor economlcal.
ubllc cloud sLorage sysLems have rlsen Lo Lhe Lask of handllng web-scale workloads.
Cloud sLorage-as-a-servlces lnclude Amazon.com's Slmple SLorage Servlce (S3) and
8ackspace's Cloud llles, whlch boLh has grown Lremendously ln usage. lor lnsLance, ln
Aprll 2010, Amazon.com reporLed LhaL 100 bllllon ob[ecLs were sLored ln S3, a 100
growLh from Lhe prevlous year. ln CcLober 2011, Amazon.com reporLed 366 bllllon
ob[ecLs were sLored ln S3. 1echnology Lrend analyzers aL 8ullLWlLh.com now Lracks
more Lhan 60,000 webslLes LhaL serve conLenL dlrecLly from S3.
Powever, noL every organlzaLlon wlll uLlllze a publlc sLorage cloud such as S3 for cosL,
regulaLory, or conLrol reasons. WlLh Lhe CpenSLack Cb[ecL SLorage sysLem, aka SwlfL",
Lhere ls now an open-source opLlon for organlzaLlons needlng a hlghly durable ob[ecL
sLorage sysLem LhaL ls deployed on lndusLry-sLandard hardware ln Lhelr own
daLacenLer. 1hese sysLems can serve as Lhe core for a prlvaLe sLorage cloud or publlc
sLorage-as-a-servlce offerlngs.
1he purpose of Lhls whlLe-paper ls Lo help Lhose who are conslderlng deploylng an
ob[ecL sLorage sysLem based on CpenSLack SwlfL and complemenLs Lhe offlclal SwlfL
documenLaLlon whlch ls avallable aL hLLp://swlfL.opensLack.org. noL every Loplc relaLed
Lo geLLlng SwlfL up and runnlng ln your envlronmenL ls covered ln Lhls whlLe-paper, buL
lL provldes an overvlew of Lhe key areas Lo be aware of, lncludlng whaL SwlfL ls, how lL
works, how Lo deploy, manage and monlLor SwlfL wlLh Lhe SwlfLSLack laLform, how Lo
use SwlfL and some general deploymenL conslderaLlons.

3
lnLroduclng SwlfL
SwlfL ls a mulLl-LenanL, hlghly scalable and durable ob[ecL sLorage sysLem LhaL ls
deslgned Lo sLore large amounLs of unsLrucLured daLa aL low cosL. Plghly scalable",
means LhaL lL can scale from a few nodes and a handful of drlves Lo Lhousands of
machlnes wlLh mulLlple eLabyLes of sLorage. SwlfL ls deslgned Lo be horlzonLally
scalable-Lhere ls no slngle polnL-of-fallure.
SwlfL ls used by lorLune 300 enLerprlses, web companles and servlce provlders
worldwlde. lL ls Lyplcally used Lo sLore unsLrucLured daLa such as documenLs, web
conLenL, backups, lmages and vlrLual machlne snaphoLs. Crlglnally developed as Lhe
englne behlnd 8ackSpace Cloud llles, lL was open-sourced under Lhe Apache 2 llcense
under Lhe CpenSLack pro[ecL ln 2010. WlLh now more Lhan 100 companles and
Lhousands of developers parLlclpaLlng ln Lhe CpenSLack pro[ecL, Lhe usage of SwlfL ls
lncreaslng rapldly.
SwlfL ls noL a LradlLlonal flle sysLem or a raw block devlce. lnsLead, lL enables you Lo
sLore, reLrleve and deleLe ob[ecLs (wlLh lLs assoclaLed meLadaLa) ln conLalners
(buckeLs" ln Amazon S3 Lermlnology) vla a 8LS1ful P11 Al.

.

uevelopers can elLher wrlLe dlrecLly Lo Lhe SwlfL Al or use one of Lhe many cllenL
llbrarles LhaL exlsL for all popular programmlng languages, such as !ava, yLhon, 8uby
and C#. Amazon S3 and 8ackSpace Cloud llles users should feel very famlllar wlLh SwlfL.
lor users who have noL used an ob[ecL sLorage sysLem before, lL wlll requlre a dlfferenL
approach and mlndseL Lhan uslng a LradlLlonal fllesysLem.
8enef|ts of Sw|ft for deve|opers |nc|ude:
! uaLa can be dlrecLly served over Lhe lnLerneL
! 8LS1ful P11 lnLerface
! Access Lo sLorage ln mlnuLes, noL days
! Cne mulLl-LenanL sLorage sysLem for all your apps

6
! locus on app developmenL, noL lnfrasLrucLure plumblng
! A rlch ecosysLem of Lools and llbrarles

8enef|ts to I1 operat|ons teams |nc|ude:
! use low-cosL, lndusLry-sLandard servers and dlsks
! Manage more daLa and use cases wlLh ease
! Lnable new appllcaLlons qulckly
! Plghly durable archlLecLure wlLh no slngle-polnL of fallure
Sw|ft Character|st|cs
SwlfL can'L be mounLed llke a folder ln your operaLlng sysLem. 1here ls no random
access wlLhln a flle's conLenL and Lhere can be mulLlple concurrenL wrlLers, whlch
makes lL unsulLable for LransacLlonal appllcaLlons such as LradlLlonal relaLlonal
daLabases for whlch a SLorage Area neLwork (SAn) or neLworked ALLached SLorage
(nAS) sysLem may be a beLLer flL. Also, slnce ob[ecL sLorage sysLems don'L provlde raw
daLa blocks LhaL an operaLlng sysLem can form lnLo a fllesysLem, SwlfL ls unsulLable for
booLlng an operaLlng sysLem.
1he key characLerlsLlcs and beneflLs of SwlfL lnclude:
! All ob[ecLs have a u8L
! All ob[ecLs have Lhelr own meLadaLa
! uevelopers lnLeracL wlLh Lhe ob[ecL sLorage sysLem Lhrough a 8LS1ful P11 Al
! Cb[ecL daLa can be locaLed anywhere ln Lhe clusLer
! 1he clusLer scales by addlng addlLlonal nodes - wlLhouL sacrlflclng performance,
whlch allows a more cosL-effecLlve llnear sLorage expanslon vs. fork-llfL upgrades
! uaLa doesn'L have Lo be mlgraLed Lo an enLlrely new sLorage sysLem
! new nodes can be added Lo Lhe clusLer wlLhouL downLlme
! lalled nodes and dlsks can be swapped ouL wlLh no downLlme
! 8uns on lndusLry-sLandard hardware, such as uell, P, Supermlcro eLc.
Sw|ft |s Sca|ab|e
1o supporL Lhousands of concurrenL users, Loday's appllcaLlon archlLecLs musL Lake
advanLage of Lhe laLesL ln dlsLrlbuLed archlLecLures, uslng dlsLrlbuLed noSCL daLabases
(Couchu8, Cassandra, Mongou8), dlsLrlbuLed message / queulng sysLems (AcLlveMC,
8abblLMC) and dlsLrlbuLed processlng sysLems llke Padoop. 1o LhaL end, appllcaLlon
archlLecLs need Lhelr sLorage sysLem Lo scale along wlLh Lhelr appllcaLlon.
Avallable space lsn'L a useful sLaLlsLlc on lLs own. A key benchmark ls Lhe sLorage
sysLem's concurrency. 1he ablllLy Lo handle a greaL number of slmulLaneous

7
connecLlons from wlLhln a daLacenLer or across Lhe web ls crlLlcal Lo saLlsfy Lhe needs
of appllcaLlons LhaL are bullL for web scale usage.
SwlfL ls deslgned Lo have llnear growLh characLerlsLlcs. As Lhe sysLem grows ln usage
and Lhe number of requesLs lncrease, performance doesn'L degrade. 1o scale up, Lhe
sysLem ls deslgned Lo grow where needed - by addlng sLorage nodes Lo lncrease
sLorage capaclLy, addlng proxy nodes as requesLs lncrease, and growlng neLwork
capaclLy where choke polnLs are deLecLed.
Sw|ft |s Lxtreme|y Durab|e
8ullL on an archlLecLure slmllar Lo Amazon S3, whlch clalms 99.999999999 durablllLy,
SwlfL ls exLremely durable. 1o achleve Lhls level of durablllLy, ob[ecLs are dlsLrlbuLed ln
LrlpllcaLe across Lhe clusLer. A wrlLe musL be conflrmed ln Lwo of Lhe Lhree locaLlons Lo
be consldered successful. AudlLlng process run Lo ensure Lhe lnLegrlLy of daLa.
8epllcaLors run Lo ensure LhaL a sufflclenL number of coples are ln Lhe clusLer. ln Lhe
evenL LhaL a devlce falls, daLa ls repllcaLed LhroughouL Lhe clusLer Lo ensure LhaL Lhree
coples remaln.
AnoLher feaLure ls Lhe ablllLy Lo deflne fallure zones. lallure zones allow a clusLer Lo be
deployed across physlcal boundarles, each of whlch could lndlvldually fall. lor example,
a clusLer could be deployed across several nearby daLa cenLers, enabllng lL Lo survlve
mulLlple daLacenLer fallures.
1he servers LhaL handle lncomlng Al requesLs scale up [usL llke any fronL-end Ller for a
web appllcaLlon. 1he sysLem uses a shared-noLhlng approach and employs Lhe same
proven Lechnlques LhaL have been used Lo provlde hlgh avallablllLy by many web
appllcaLlons.
Sw|ft |s Cpen Source Software
SwlfL ls llcensed under Lhe permlsslve Apache 2 open source llcense. WhaL makes SwlfL
dlfferenL from mosL oLher open source pro[ecLs ls LhaL Lhe sofLware has already been
sLressed LesLed ln a large-scale producLlon deploymenL aL 8ackspace before lLs flrsL
publlc release. As an open source pro[ecL, SwlfL provldes Lhe followlng beneflLs Lo lLs
users:

1. No vendor |ock-|n - As an open source pro[ecL, you have Lhe opLlon Lo work wlLh a
varleLy of provlders or as a ul? pro[ecL
2. Commun|ty support - ?ou can access and share Lools, besL pracLlces and
deploymenL know-how wlLh oLher organlzaLlons and communlLy members LhaL
are uslng SwlfL

8
3. Large ecosystem - WlLh Lhe large number or organlzaLlons and developers
parLlclpaLlng ln Lhe CpenSLack pro[ecL, Lhe developmenL veloclLy and breadLh of
Lools, uLlllLles and servlces for SwlfL wlll only lncrease over Llme

As Lhe source code ls publlcly avallable, lL can be revlewed by many more developers
Lhan whaL ls Lhe case for proprleLary sofLware. 1hls means LhaL poLenLlal bugs also
Lend Lo be more vlslble and more rapldly correcLed Lhan for proprleLary sofLware.
ln Lhe long Lerm, open" generally wlns - and SwlfL mlghL be consldered Lhe Llnux of
sLorage.
Sw|ft |s S|m||ar to AWS S3
Access Lo Lhe SwlfL ob[ecL sLorage sysLem ls enLlrely Lhrough a 8LS1 Al, whlch ls
slmllar Lo Lhe Amazon.com S3 Al and compaLlble wlLh Lhe 8ackspace Cloud llles Al.
1hls means LhaL (a) appllcaLlons LhaL are currenLly uslng S3 can use SwlfL wlLhouL ma[or
re-facLorlng of Lhe appllcaLlon code and (b) appllcaLlons LhaL llke Lo Lake advanLage of
boLh prlvaLe and publlc cloud sLorage can do so as Lhe Als are comparable.
Slnce SwlfL ls comparable wlLh publlc cloud servlces, developers & sysLems archlLecLs
can also Lake advanLage of a rlch ecosysLem of commerclal and open-source Lools ls
avallable for Lhese ob[ecL sLorage sysLems. CllenLs such as Cyberduck, fllers llke nasunl
and programmlng llbrarles whlch are avallable ln C#, P, erl, yLhon, !ava, and 8uby
are [usL some examples.
Sw|ft |s 8u||t on Industry-standard Components
lf you look under-Lhe-hood, SwlfL ls bullL uslng maLure, sLandard componenLs such as
rsync, Mu3, sqllLe, memcache, xfs and pyLhon. SwlfL runs on off-Lhe-shelf Llnux
dlsLrlbuLlons such as ubunLu, whlch ls dlfferenL from mosL oLher sLorage sysLems,
whlch run on proprleLary or hlghly-cusLomlzed operaLlng sysLems.
lrom a hardware perspecLlve, SwlfL ls deslgned ground up Lo handle fallures so LhaL
rellablllLy on Lhe lndlvldual componenL level ls less crlLlcal. 1hus, regular deskLop drlves
can be used ln a SwlfL clusLer raLher Lhan more expenslve enLerprlse" drlves.
Pardware quallLy and conflguraLlon can be chosen Lo sulL Lhe Lolerances of Lhe
appllcaLlon and Lhe ablllLy Lo replace falled equlpmenL.
Slnce commodlLy hardware can be used wlLh Lhe sysLem, Lhere ls consequenLly no lock-
ln wlLh any parLlcular sLorage vendor. 1hls means deploymenLs can conLlnually Lake
advanLage of decreaslng hardware prlces and lncreaslng drlve capaclLy.


9
Sw|ft Can 8e Dep|oyed In-nouse or As-a-Serv|ce
lor organlzaLlons uncomforLable wlLh ouLsourclng Lhelr daLa sLorage Lo a publlc cloud
sLorage vendor, SwlfL can help achleve lower cosLs and slmllar performance whlle
reLalnlng greaLer conLrol over neLwork access, securlLy, and compllance. CosL ls also a
ma[or facLor for brlnglng cloud sLorage ln-house. ubllc cloud sLorage cosLs lnclude per-
C8 prlclng plus daLa LranslL charges, whlch can become very expenslve.
1he neLwork laLency Lo publlc sLorage servlce provlders may be unaccepLable. A
prlvaLe deploymenL can provlde lower-laLency access Lo sLorage, as requlred by many
appllcaLlons. Also, appllcaLlons may have large volumes of daLa ln fllghL, whlch can'L go
over Lhe publlc lnLerneL.
lor Lhe above reasons, organlzaLlons can use SwlfL Lo bulld an ln-house sLorage sysLem
LhaL has slmllar durablllLy/accesslblllLy properLles and ls compaLlble wlLh Lhe sulLes of
Lools avallable for publlc cloud sLorage sysLems.
SwlfL can also be deployed as a publlc sLorage-as-a-servlce. CperaLors and servlce
provlders, who wanL Lo offer S3-llke sLorage servlces, can geL sLarLed by offerlng SwlfL
as a value-added servlce Lo Lhelr cusLomers. WlLh SwlfL, lL ls now cosL-effecLlve Lo bulld
and deploy an ob[ecL sLorage clusLer for publlc use.
Sw|ft Is Supported
As an CpenSLack pro[ecL, SwlfL has Lhe beneflL of a rlch communlLy, whlch lncludes
more Lhan 100 parLlclpaLlng companles and 1000+ developers. 1he followlng supporL
opLlons are avallable for SwlfL:
! Commerclal supporL and Lools are avallable Lhrough SwlfLSLack, whlch has
experlence deploylng, runnlng and supporLlng SwlfL aL scale
! CommunlLy supporL ls provlded Lhrough CpenSLack communlLy, where besL
pracLlces can be shared wlLh oLher organlzaLlons and users LhaL are uslng SwlfL
! SwlfL's documenLaLlon ls publlcly avallable aL hLLp://swlfL.opensLack.org

10
uslng SwlfL
Cnce deployed, all communlcaLlon wlLh SwlfL ls done over a 8LS1-ful P11 Al.
AppllcaLlon uevelopers who'd llke Lo Lake advanLage of SwlfL for sLorlng conLenL,
documenLs, flles, lmages eLc. can use one of Lhe many cllenL llbrarles LhaL exlsL for all
all popular programmlng languages, lncludlng !ava, yLhon, 8uby, C# and P. LxlsLlng
backups, daLa proLecLlon and archlvlng appllcaLlons whlch currenLly supporL elLher
8ackspace Cloud llles or Amazon S3 can also use SwlfL as Lhelr sLorage back-end wlLh
mlnor modlflcaLlons.
Sw|ft Commands - 1he 8as|cs
As SwlfL has a 8LS1-ful Al, all communlcaLlon wlLh SwlfL ls done over P11, uslng Lhe
P11 verbs Lo slgnal Lhe requesLed acLlon. A SwlfL sLorage u8L looks llke Lhls:

hLLp://swlfL.example.com/v1/accounL/conLalner/ob[ecL

SwlfL's u8Ls have four baslc parLs. uslng Lhe example above, Lhese parLs are:

! 8ase: swlfL.example.com/v1/
! AccounL: An accounL ls deLermlned by Lhe auLh server when Lhe accounL ls
creaLed.
! ConLalner: ConLalners are namespaces used Lo group ob[ecLs wlLhln an accounL
! Cb[ecL: Cb[ecLs are where Lhe acLual daLa ls sLored ln swlfL. Cb[ecL names may
conLaln /, so pseudo-nesLed dlrecLorles are posslble.

1o geL a llsL of all conLalners ln an accounL, use Lhe CL1 command on Lhe accounL:
CL1 hLLp://swlfL.example.com/v1/accounL/

1o creaLe new conLalners, use Lhe u1 command wlLh Lhe name of Lhe new conLalner:
u1 hLLp://swlfL.example.com/v1/accounL/new_conLalner

1o llsL all ob[ecL ln a conLalner, use Lhe CL1 command on Lhe conLalner:
CL1 hLLp://swlfL.example.com/v1/accounL/conLalner/


11
1o creaLe new ob[ecLs wlLh a u1 on Lhe ob[ecL:
u1 hLLp://swlfL.example.com/v1/accounL/conLalner/new_ob[ecL

1he CS1 command ls used Lo change meLadaLa on conLalners and ob[ecLs.When
plannlng a SwlfL deploymenL, Lhe flrsL sLep ls Lo deflne Lhe appllcaLlon workloads and
funcLlonal requlremenLs LhaL wlll deLermlne how your SwlfL
C||ent L|brar|es
Several cllenL llbrarles for SwlfL are avallable, lncludlng:

C#/.nL1: hLLp://glLhub.com/rackspace/csharp-cloudflles
!ava: hLLp://glLhub.com/rackspace/[ava-cloudflles
P: hLLp://glLhub.com/rackspace/php-cloudflles
yLhon: hLLp://glLhub.com/rackspace/pyLhon-cloudflles
8uby: hLLp://glLhub.com/rackspace/ruby-cloudflles

ln addlLlon, a 8uby llbrary ls also avallable Lhrough Lhe fog cllenL:
hLLp://glLhub.com/geemus/fog

1he luse cllenL can be used Lo map a fllesysLem Lo SwlfL-
hLLp://glLhub.com/redbo/cloudfuse/

lor more lnformaLlon bulldlng cllenL llbrarles for SwlfL see:
hLLps://glLhub.com/rackspace/docs-cloudflles


12
Pow SwlfL Works
8u||d|ng 8|ocks
1he componenLs LhaL enable SwlfL Lo dellver hlgh avallablllLy, hlgh durablllLy and hlgh
concurrency are:
! roxy Servers: Pandles all lncomlng Al requesLs.
! k|ngs: Maps loglcal names of daLa Lo locaLlons on parLlcular dlsks.
! 2ones: Lach Zone lsolaLes daLa from oLher Zones. A fallure ln one Zone doesn'L
lmpacL Lhe resL of Lhe clusLer because daLa ls repllcaLed across Lhe Zones.
! Accounts & Conta|ners: Lach AccounL and ConLalner are lndlvldual daLabases LhaL
are dlsLrlbuLed across Lhe clusLer. An AccounL daLabase conLalns Lhe llsL of
ConLalners ln LhaL AccounL. A ConLalner daLabase conLalns Lhe llsL of Cb[ecLs ln
LhaL ConLalner.
! Cb[ects: 1he daLa lLself.
! art|t|ons: A arLlLlon sLores Cb[ecLs, AccounL daLabases and ConLalner
daLabases. lL's an lnLermedlaLe 'buckeL' LhaL helps manage locaLlons where daLa
llves ln Lhe clusLer.
roxy Servers
1he roxy Servers are Lhe publlc face of SwlfL and handle all lncomlng Al requesLs.
Cnce a roxy Server recelve a requesL, lL wlll deLermlne Lhe sLorage node based on Lhe
u8L of Lhe ob[ecL, e.g. hLLps://swlfL.example.com/v1/accounL/conLalner/ob[ecL. 1he
roxy Servers also coordlnaLes responses, handles fallures and coordlnaLes
LlmesLamps.
roxy servers use a shared-noLhlng archlLecLure and can be scaled as needed based on
pro[ecLed workloads. A mlnlmum of Lwo roxy Servers should be deployed for
redundancy. Should one proxy server fall, Lhe oLhers wlll Lake over.
1he k|ng
1he 8lng maps arLlLlons Lo physlcal locaLlons on dlsk. When oLher componenLs need
Lo perform any operaLlon on an ob[ecL, conLalner, or accounL, Lhey need Lo lnLeracL
wlLh Lhe 8lng Lo deLermlne lLs locaLlon ln Lhe clusLer.
1he 8lng malnLalns Lhls mapplng uslng zones, devlces, parLlLlons, and repllcas. Lach
parLlLlon ln Lhe 8lng ls repllcaLed Lhree Llmes by defaulL across Lhe clusLer, and Lhe

13
locaLlons for a parLlLlon are sLored ln Lhe mapplng malnLalned by Lhe 8lng. 1he 8lng ls
also responslble for deLermlnlng whlch devlces are used for handoff should a fallure
occur.

"#$ %&'( )*+, +*-.&.&/', ./ +#0,&1*2 2/1*.&/', /' 3&,45
2ones: Ia||ure 8oundar|es
SwlfL allows zones Lo be conflgured Lo lsolaLe fallure boundarles. Lach plece of daLa
resldes ln mulLlple zones. AL Lhe smallesL level, a zone could be a slngle drlve or a
grouplng of a few drlves. lf Lhere were flve ob[ecL sLorage servers, Lhen each server
would represenL lLs own zone. Larger deploymenLs would have an enLlre rack (or
mulLlple racks) of ob[ecL sLores, each represenLlng a zone. 1he goal of zones ls Lo allow
Lhe clusLer Lo LoleraLe slgnlflcanL ouLages of sLorage servers.
As we learned earller, everyLhlng ln SwlfL ls sLored, by defaulL, Lhree Llmes. 1hree zones
may seem sufflclenL for holdlng Lhree coples of daLa, buL conslder Lhe case when a
zone goes down. 1here would be no fourLh zone lnLo whlch daLa may be repllcaLed,
leavlng only Lwo coples of all sLored daLa. 1herefore, lL ls recommended LhaL aL leasL
four zones and preferably flve zones be deployed.
lf a zone goes down, daLa wlll be repllcaLed Lo oLher zones. Pavlng aL leasL flve zones
leaves enough margln Lo accommodaLe Lhe occaslonal Zone fallure and enough
capaclLy Lo repllcaLe daLa across Lhe sysLem.

6#$' * 3&,47 '/3$7 /- 8/'$ 9*&2,7 -$+2&1* 3*.* &, 3&,.-&:;.$3 ./ .#$ /.#$- 8/'$, ./ $',;-$
.#$-$ *-$ .#-$$ 1/+&$, /9 .#$ 3*.*

14
Accounts & Conta|ners
Lach accounL and conLalner ls an lndlvldual SCLlLe daLabase LhaL ls dlsLrlbuLed across
Lhe clusLer. An accounL daLabase conLalns Lhe llsL of conLalners ln LhaL accounL. A
conLalner daLabase conLalns Lhe llsL of ob[ecLs ln LhaL conLalner.

"/ 4$$+ .-*14 /9 /:<$1. 3*.* 2/1*.&/'7 $*1# *11/;'. &' .#$ ,0,.$) #*, * 3*.*:*,$ .#*.
-$9$-$'1$, *22 &., 1/'.*&'$-,7 *'3 $*1# 1/'.*&'$- 3*.*:*,$ -$9$-$'1$, $*1# /:<$1.5
art|t|ons
A arLlLlon ls a collecLlon of sLored daLa, lncludlng AccounL daLabases, ConLalner
daLabases, and ob[ecLs. arLlLlons are core Lo Lhe repllcaLlon sysLem.
1hlnk of a arLlLlon as a bln movlng LhroughouL a fulflllmenL cenLer warehouse.
lndlvldual orders geL Lhrown lnLo Lhe bln. 1he sysLem LreaLs LhaL bln as a coheslve
enLlLy as lL moves LhroughouL Lhe sysLem. A bln full of Lhlngs ls easler Lo deal wlLh Lhan
loLs of llLLle Lhlngs. lL makes for fewer movlng parLs LhroughouL Lhe sysLem.
1he sysLem repllcaLors and ob[ecL uploads/downloads operaLe on arLlLlons. As Lhe
sysLem scales up, behavlor conLlnues Lo be predlcLable as Lhe number of arLlLlons ls a
flxed number.
1he lmplemenLaLlon of a arLlLlon ls concepLually slmple - a parLlLlon ls [usL a
dlrecLory slLLlng on a dlsk wlLh a correspondlng hash Lable of whaL lL conLalns.

=>&9. +*-.&.&/', 1/'.*&' *22 3*.* &' .#$ ,0,.$)5
kep||cat|on
ln order Lo ensure LhaL Lhere are Lhree coples of Lhe daLa everywhere, repllcaLors

13
conLlnuously examlne each arLlLlon. lor each local arLlLlon, Lhe repllcaLor compares
lL agalnsL Lhe repllcaLed coples ln Lhe oLher Zones Lo see lf Lhere are any dlfferences.
Pow does Lhe repllcaLor know lf repllcaLlon needs Lo Lake place? lL does Lhls by
examlnlng hashes. A hash flle ls creaLed for each arLlLlon, whlch conLalns hashes of
each dlrecLory ln Lhe arLlLlon. Lach of Lhe Lhree hash flles ls compared. lor a glven
arLlLlon, Lhe hash flles for each of Lhe arLlLlon's coples are compared. lf Lhe hashes
are dlfferenL, Lhen lL ls Llme Lo repllcaLe and Lhe dlrecLory LhaL needs Lo be repllcaLed ls
copled over.
1hls ls where Lhe arLlLlons come ln handy. WlLh fewer Lhlngs" ln Lhe sysLem, larger
chunks of daLa are Lransferred around (raLher Lhan loLs of llLLle 1C connecLlons, whlch
ls lnefflclenL) and Lhere are a conslsLenL number of hashes Lo compare.
1he clusLer has evenLually conslsLenL behavlor where Lhe newesL daLa wlns.

?9 * 8/'$ (/$, 3/>'7 /'$ /9 .#$ '/3$, 1/'.*&'&'( * -$+2&1* '/.&1$, *'3 +-/*1.&@$20
1/+&$, 3*.* ./ * #*'3/99 2/1*.&/'5
now these are a|| t|ed together
1o descrlbe how Lhese pleces all come LogeLher, leL's walk Lhrough a few scenarlos and
lnLroduce Lhe componenLs.
Up|oad
A cllenL uses Lhe 8LS1 Al Lo make a P11 requesL Lo u1 an ob[ecL lnLo an exlsLlng
ConLalner. 1he clusLer recelves Lhe requesL. llrsL, Lhe sysLem musL flgure ouL where
Lhe daLa ls golng Lo go. 1o do Lhls, Lhe AccounL name, ConLalner name and Cb[ecL
name are all used Lo deLermlne Lhe arLlLlon where Lhls ob[ecL should llve.
1hen a lookup ln Lhe 8lng flgures ouL whlch sLorage nodes conLaln Lhe arLlLlons ln

16
quesLlon.
1he daLa Lhen ls senL Lo each sLorage node where lL ls placed ln Lhe approprlaLe
arLlLlon. A quorum ls requlred - aL leasL Lwo of Lhe Lhree wrlLes musL be successful
before Lhe cllenL ls noLlfled LhaL Lhe upload was successful.
nexL, Lhe ConLalner daLabase ls updaLed asynchronously Lo reflecL LhaL Lhere ls a new
ob[ecL ln lL.

Down|oad
A requesL comes ln for an AccounL/ConLalner/ob[ecL. uslng Lhe same conslsLenL
hashlng, Lhe arLlLlon name ls generaLed. A lookup ln Lhe 8lng reveals whlch sLorage
nodes conLaln LhaL arLlLlon. A requesL ls made Lo one of Lhe sLorage nodes Lo feLch
Lhe ob[ecL and lf LhaL falls, requesLs are made Lo Lhe oLher nodes.


17
SwlfL ClusLer ArchlLecLure
Access 1|er

Large-scale deploymenLs segmenL off an Access 1ler". 1hls Ller ls Lhe Crand CenLral"
of Lhe Cb[ecL SLorage sysLem. lL flelds lncomlng Al requesLs from cllenLs and moves
daLa ln and ouL of Lhe sysLem. 1hls Ller ls composed of fronL-end load balancers, ssl-
LermlnaLors, auLhenLlcaLlon servlces, and lL runs Lhe (dlsLrlbuLed) braln of Lhe ob[ecL
sLorage sysLem - Lhe proxy server processes.
Pavlng Lhe access servers ln Lhelr own Ller enables read/wrlLe access Lo be scaled ouL
lndependenLly of sLorage capaclLy. lor example, lf Lhe clusLer ls on Lhe publlc lnLerneL
and requlres ssl-LermlnaLlon and has hlgh demand for daLa access, many access servers
can be provlsloned. Powever, lf Lhe clusLer ls on a prlvaLe neLwork and lL ls belng used
prlmarlly for archlval purposes, fewer access servers are needed.
As Lhls ls an P11 addressable sLorage servlce, a load balancer can be lncorporaLed lnLo

18
Lhe access Ller.
1yplcally, Lhls Ller comprlses a collecLlon of 1u servers. 1hese machlnes use a moderaLe
amounL of 8AM and are neLwork l/C lnLenslve. As Lhese sysLems fleld each lncomlng
Al requesL, lL ls wlse Lo provlslon Lhem wlLh Lwo hlgh-LhroughpuL (10CbL) lnLerfaces.
Cne lnLerface ls used for 'fronL-end' lncomlng requesLs and Lhe oLher for 'back-end'
access Lo Lhe ob[ecL sLorage nodes Lo puL and feLch daLa.
Iactors to Cons|der
lor mosL publlcly faclng deploymenLs as well as prlvaLe deploymenLs avallable across a
wlde-reachlng corporaLe neLwork, SSL wlll be used Lo encrypL Lrafflc Lo Lhe cllenL. SSL
adds slgnlflcanL processlng load Lo esLabllsh sesslons beLween cllenLs, more capaclLy ln
Lhe access layer wlll need Lo be provlsloned. SSL may noL be requlred for prlvaLe
deploymenLs on LrusLed neLworks.
Storage Nodes


19
1he nexL componenL ls Lhe sLorage servers Lhemselves. Cenerally, mosL conflguraLlons
should have each of Lhe flve Zones wlLh an equal amounL of sLorage capaclLy. SLorage
nodes use a reasonable amounL of memory and Cu. MeLadaLa needs Lo be readlly
avallable Lo qulckly reLurn ob[ecLs. 1he ob[ecL sLores run servlces noL only Lo fleld
lncomlng requesLs from Lhe Access 1ler, buL Lo also run repllcaLors, audlLors, and
reapers. Cb[ecL sLores can be provlsloned wlLh slngle glgablL or 10 glgablL neLwork
lnLerface dependlng on expecLed workload and deslred performance.
CurrenLly 218 or 318 SA1A dlsks dellver good prlce/performance value. ueskLop-grade
drlves can be used where Lhere are responslve remoLe hands ln Lhe daLacenLer, and
enLerprlse-grade drlves can be used where Lhls ls noL Lhe case.
Iactors to Cons|der
ueslred l/C performance for slngle-Lhreaded requesLs should be kepL ln mlnd. 1hls
sysLem does noL use 8Alu, so each requesL for an ob[ecL ls handled by a slngle dlsk.
ulsk performance lmpacLs slngle-Lhreaded response raLes.
1o achleve apparenL hlgher LhroughpuL, Lhe ob[ecL sLorage sysLem ls deslgned wlLh
concurrenL uploads/downloads ln mlnd. 1he neLwork l/C capaclLy (1CbL, bonded 1CbL
palr, or 10CbL) should maLch your deslred concurrenL LhroughpuL needs for reads and
wrlLes.

20
Conflgurlng neLworklng
8elow are Lwo examples of deploymenLs aL Lwo scales: Lhe larger deploymenLs wlLh a
Lwo-Ller neLworklng archlLecLure, and smaller deploymenLs wlLh a slngle neLworklng
Ller.
noLe LhaL when a wrlLe comes lnLo Lhe proxy server, Lhere ls Lhree Llmes Lhe Lrafflc
golng Lo Lhe ob[ecL sLores Lo wrlLe Lhe Lhree repllcas. SysLems musL be deslgned Lo
accounL for Lhe expecLed read/wrlLe Lrafflc.
Large-Sca|e Network|ng

Aggregat|on
A palr of aggregaLlon swlLches wlLh Lwo llnks back Lo Lhe access neLwork / border
neLwork are used Lo connecL Lo Lwo pools of Lhe Access 1ler and Lo each of Lhe flve
Zone swlLches LhaL connecL Lhe Cb[ecL SLores. All connecLlons Lo Lhe Access 1ler and
Lhe Zones are 10CbL.
2one Network
Lach Zone has a swlLch Lo connecL lLself Lo Lhe aggregaLlon neLwork. lL's posslble Lo use
a slngle, non-redundanL swlLch as Lhe sysLem ls deslgned Lo susLaln a Zone fallure.

21
uependlng on overall concurrency deslred, a deploymenL can use elLher a 1CbL or a
10CbL neLwork Lo Lhe ob[ecL sLores.
Med|um-Sca|e Network|ng

A slngle neLwork Ller ls used for smaller deploymenLs ln Lhe range of 300-30018. LlLher
1CbL or 10CbL swlLches can be used for Lhls purpose dependlng on Lhe LhroughpuL Lhe
clusLer ls expecLed Lo susLaln. 1he Access 1ler servlces sLlll conLaln Lwo lnLerfaces and a
vLAn ls creaLed for each fronL-faclng" Al requesL and back-end" neLwork
connecLlng Lhe ob[ecL server Zones.
Management Network
A managemenL neLwork ls crlLlcal Lo malnLalnlng Lhe healLh of Lhe clusLer. A separaLe
1CbL managemenL neLwork ls creaLed for lMl, monlLorlng, and ouL-of-band access Lo
every machlne ln Lhe clusLer. Powever, lL ls Lyplcally posslble Lo use Lhe hlgher-
bandwldLh connecLlons durlng provlslonlng for operaLlng sysLem lnsLallaLlon.


22
Pardware 8ecommendaLlons

SwlfL ls deslgned Lo sLore and reLrleve whole flles vla P11 across a clusLer of lndusLry-
sLandard x86 servers and drlves, uslng repllcaLlon Lo ensure daLa rellablllLy and faulL
Lolerance. Whlle Lhls model provldes greaL flexlblllLy (and low cosL) from a hardware
perspecLlve, lL requlres some upfronL plannlng, LesLlng and valldaLlon Lo ensure LhaL
Lhe hardware you selecL ls sulLable noL [usL for SwlfL lLself, buL also for Lhe expecLed
workload LhaL you are deslgnlng your clusLer for. ?our operaLlons Leam may also have
oplnlons on Lhe hardware selecLlon, as Lhey prefer Lo work wlLh hardware Lhey are
already famlllar wlLh.
roxy Nodes
roxy nodes use a moderaLe amounL of 8AM and are neLwork lC lnLenslve. 1yplcally,
roxy nodes are 1u sysLems wlLh a mlnlmum of 12 C8 8AM. As Lhese sysLems fleld
each lncomlng Al requesL, lL ls wlse Lo provlslon Lhem wlLh Lwo hlgh-LhroughpuL
(10CbL) lnLerfaces. Cne lnLerface ls used for 'fronL-end' lncomlng requesLs and Lhe
oLher for 'back-end' access Lo Lhe ob[ecL sLorage nodes Lo puL and feLch daLa. lor small
SwlfL deploymenLs, Lhe sLorage nodes can serve as proxy nodes.
Storage Nodes
SLorage nodes are Lyplcally hlgh-denslLy 3u or 4u nodes wlLh 16-36 SA1A dlsks each.
1hese nodes use a reasonable amounL of memory and Cu. 1he sLorage nodes run
servlces noL only Lo fleld lncomlng requesLs from Lhe proxy nodes, buL also repllcaLlon,
audlLlng and oLher processes Lo ensure durablllLy. SLorage nodes can be provlsloned
wlLh slngle glgablL or 10CbL neLwork lnLerface dependlng on expecLed workload and
deslred performance.

lor sLorage nodes, we recommend Lhe followlng speclflcaLlons:

CU - 64-blL x86 Cu (lnLel/AMu), quad-core or greaLer, runnlng aL leasL 2-2.3CPz

kAM - A good rule of Lhumb ls approxlmaLely 1 C8 of 8AM for each 18 of ulsk. l.e. for
a node wlLh 24 drlves, 36-48C8 of 8AM should be used. 1he memory ls used for Lhe
many processes used fleld lncomlng ob[ecL requesLs and xlS lnode cachlng.

Dr|ves - LlLher 218 or 318 7200 8M SA1A drlves, whlch dellver good
prlce/performance value. ueskLop-grade drlves can be used where Lhere are
responslve remoLe hands ln Lhe daLa cenLer, and enLerprlse-grade drlves can be used

23
where LhaL ls noL Lhe case. We don'L recommend uslng green" drlves. SwlfL ls
conLlnuously ensurlng daLa lnLegrlLy and Lhe power-down funcLlons of green drlves
may resulL ln excess wear.

Lxtreme conta|ner update work|oad cons|derat|on - Where Lhe appllcaLlon needs Lo
lngesL many mllllons of flles ln a slngle conLalner, lL may be necessary Lo use hlgher-
performlng medla (8Alu 10 wlLh 13k drlves or SSus) for Lhe conLalner lndexes. 1he daLa
seL ls relaLlvely very small ln slze, so few space ls needed on Lhe hlgher performlng
medla Lo sLore Lhls daLa.

Contro||er Cards - SwlfL repllcaLes daLa across zones so Lhere ls no need for daLa
redundancy Lo be provlded by Lhe conLroller. SwlfL Lherefore uses sLandard SA1A
conLroller cards wlLhouL 8Alu, such as LSl 9211-8l 6Cb/s SAS / SA1A P8A. Powever, lf
Lhe conLroller card requlres 8Alu volumes Lo be creaLed, seL up a 8Alu 0 group
(wlLhouL sLrlplng) for each drlve.

Network Cards - uependlng on Lhe use case, slngle glgablL eLherneL (1CbL) on each
hosL may be all LhaL ls requlred. Powever, lL ls posslble Lo conflgure bonded 1CbL or
10CbL lf Lhe workload demands lL.
Network|ng
A Lyplcal deploymenL would have a fronL-faclng 'access' neLwork and a back-end
'sLorage' neLwork. When deslgnlng Lhe neLwork capaclLy, keep ln mlnd LhaL wrlLes fan-
ouL ln LrlpllcaLe ln Lhe sLorage neLwork. As Lhere are Lhree coples of each ob[ecL, an
lncomlng wrlLe ls senL Lo Lhree sLorage nodes. 1herefore neLwork capaclLy for wrlLes
needs Lo be consldered ln proporLlon Lo overall workload.
S|z|ng our Sw|ft C|uster
Lach node and drlve LhaL ls added Lo a SwlfL clusLer wlll noL only provlde addlLlonal
sLorage capaclLy, buL wlll also lncrease Lhe aggregaLe lC capaclLy of Lhe clusLer as Lhere
are more sysLems and drlves avallable Lo serve lncomlng read requesLs. When selecLlng
a speclflc hardware conflguraLlon for your SwlfL clusLer, lL ls Lherefore lmporLanL Lo
deLermlne whlch conflguraLlon provldes Lhe besL balance of lC performance, capaclLy
and cosL for a glven workload. lor lnsLance, cusLomer-faclng web appllcaLlons wlLh a
large number of concurrenL users wlll have a dlfferenL proflle Lhan one LhaL ls used
prlmarlly for archlvlng.


24
lannlng a ueploymenL

8enchmark|ng and 1est|ng
When plannlng a SwlfL deploymenL, Lhe flrsL sLep ls Lo ldenLlLy Lhe appllcaLlon
workloads and deflne Lhe correspondlng funcLlonal requlremenLs, whlch wlll deLermlne
how your SwlfL envlronmenL wlll be deslgned, conflgured and deployed. 8enchmarklng
goes hand-ln-hand wlLh deslgnlng Lhe rlghL clusLer. Some of Lhe areas Lo conslder when
conducLlng your benchmarklng lnclude:

1. 1he spread of flle slzes
2. number of concurrenL workers
3. 1he proporLlon of creaLlon / read / updaLe / deleLe LhaL ls expecLed

1o conducL your benchmarklng, a pre-populaLed clusLer wlLh a seL of daLa Lo mlmlc an
evenLual sLaLe should be seL up. lL ls also lmporLanL Lo be able Lo parallellze Lhe
benchmarklng workloads as lL may requlre many sysLems Lo susLaln Lhe aggregaLe
LhroughpuL a SwlfL clusLer can puL ouL.

SwlfLSLack ls avallable Lo asslsL ln deLermlnlng whlch hardware conflguraLlon ls mosL
opLlmal for your workload. As parL of a SwlfLSLack deploymenL we provlde a
benchmarklng LesL sulLe Lo help ensure and Lune your CpenSLack SwlfL clusLer so lL can
handle Lhe workloads you're abouL Lo puL on lL.

A LesLlng plan also needs Lo be developed whlch ouLllnes how Lhe envlronmenL wlll be
LesLed and Lhe overall LesLlng crlLerla. Some of Lhe areas Lo address ln Lhe LesLlng plan
lnclude:

! SwlfL Al calls
! lallure scenarlos
! SwlfL cllenL llbrarles LhaL wlll be used
Datacenter Iac|||ty |ann|ng
1he daLacenLer space and power requlremenLs are crlLlcal areas Lo plan for a successful
SwlfL deploymenL. 1hey requlre plannlng wlLh your daLacenLer faclllLles Leam or
daLacenLer vendor Lo deploy a full rack of ob[ecL sLores. 8e sure Lo meeL early wlLh Lhls
Leam Lo explaln your pro[ecL and plan for:
! ower provlslonlng

23
! Coollng
! hyslcal space requlremenLs
! neLworklng capaclLy and plannlng
! orL layouLs
! 8ack layouLs
SwlfLSLack can asslsL wlLh besL pracLlces and daLacenLer faclllLles plannlng advlce for a
successful SwlfL lmplemenLaLlon.
Integrat|ons
When geLLlng SwlfL up and runnlng ln your daLa-cenLer, Lhere are several poLenLlal
lnLegraLlons wlLh 3rd parLy sysLems and servlces Lo conslder, lncludlng:

! AuLhenLlcaLlon sysLem, such as LuA
! CperaLlons supporL sysLems
! 8llllng sysLems
! LxLernal monlLorlng sysLems - Lo consume SnM polllng of sysLem lnformaLlon,
and SnM Lraps eLc.
! ConLenL ulsLrlbuLlon neLworks (Cuns)
! 8epllcaLlon Lo off-slLe SwlfL clusLers for addlLlonal redundancy

Lach of Lhese areas can be lnLegraLed wlLh your SwlfL envlronmenL buL requlremenLs
wlll dlffer based on your speclflc requlremenLs and use-case. Whlle Lhls whlLe-paper
does noL cover Lhese areas ln any depLh, SwlfLSLack can provlde advlce and besL
pracLlces on how Lo lnLegraLe wlLh 3rd parLy sysLems and servlces.
Mon|tor|ng
1here are many Lools avallable for appllcaLlon developers and l1 operaLlons Leams Lo
measure Lhe healLh of appllcaLlons and servers. Whlle many of Lhese Lools are helpful,
for a SwlfL clusLer Lhey may be more complex and provlde more daLa Lhan you need. 1o
ensure LhaL users can qulckly measure Lhe overall healLh of Lhelr SwlfLSLack
envlronmenL, Lhe SwlfLSLack laLform ls Lracklng Lhe key meLrlcs LhaL allow you Lo
qulckly deLermlne Lhe sLaLus of your overall SwlfL clusLer and of Lhe lndlvldual nodes.

lor Lhe overall SwlfL clusLer, Lhe key meLrlcs monlLored by Lhe SwlfLSLack laLform are
node Cu uLlllzaLlon, Lop 3 leasL free dlsks, dlsk l/C and neLwork l/C. lor lndlvldual
nodes, Lhe same key meLrlcs are reporLed, whlch can be used Lo Lell Lhe overall healLh
of Lhe node. ln addlLlon, exLernal monlLorlng sysLems can be conflgured Lo consume
SnM polllng of sysLem lnformaLlon, and SnM Lraps provlded Lhrough Lhe SwlfLSLack
plaLform.

26

Adm|n|strat|on & Ma|ntenance
1he SwlfLSLack laLform doesn'L ellmlnaLe Lhe need for a sLorage admlnlsLraLor buL wlll
slgnlflcanLly slmpllfy Lhe [ob of admlnlsLraLlng and malnLalnlng a SwlfL envlronmenL.
Some common areas LhaL wlll need Lo be admlnlsLered and malnLalned lnclude
performance Lunlng of Lhe overall clusLer, addlng/removlng capaclLy Lo Lhe clusLer,
ldenLlfylng and replaclng falled hardware, applylng sofLware updaLes eLc. Whlle Lhe
SwlfLSLack laLform auLomaLes and slmpllfles many of Lhese Lasks, lL wlll requlre some
hands-on admlnlsLraLlon. 1he auLomaLlon of many of Lhese Lasks ln Lhrough Lhe
SwlfLSLack laLfrom ls noL [usL a Llme saver for a sLorage admlnlsLraLor, buL crlLlcal Lo
ensurlng sysLem upLlme and prevenLlng Lhe loss of daLa or puLLlng daLa aL rlsk.
Understand|ng 1CC
llnally, when plannlng a deploymenL, undersLandlng Lhe LoLal-cosL of ownershlp for Lhe
clusLer ls a crlLlcal so all dlrecL and lndlrecL cosLs are lncluded. CosLs should lnclude:
! ueslgn/uevelopmenL
! Pardware
! Pardware SLandup
! uaLacenLer Space
! ower/Coollng

27
! neLwork Access
! Cngolng SofLware MalnLenance and SupporL
! MonlLorlng and CperaLlonal SupporL

ko|||ng Sw|ft Cut |n our Crgan|zat|on
lf your organlzaLlon does noL already have experlence bulldlng appllcaLlons wlLh an
ob[ecL sLorage sysLems, such as Amazon S3 and 8ackspace Cloud llles, lL ls also crlLlcal
Lo Lraln lnLernal developmenL and producL lnLegraLlon Leams on how Lo use and Lake
advanLage of SwlfL. 8ecause SwlfL and oLher ob[ecL sLorage sysLems requlres a dlfferenL
approach Lo appllcaLlon developmenL from a LradlLlonal flle sysLem, lL ls lmporLanL Lo
sLarL Lhls process early. SwlfLSLack provldes boLh Lralnlng and a vlrLual Lralnlng
appllance wlLh SwlfL, whlch can be lnsLalled and use on a lapLop. 1hls enables
developers and l1 operaLlons sLaff Lo become famlllar wlLh how SwlfL works and how Lo
Lake advanLage of some of lLs key beneflLs vls-a-vl a LradlLlonal fllesysLem based
approach, lncludlng:

! Self-servlce provlslonlng, where developers and users can access sLorage ln
mlnuLes
! 1he ablllLy for end-users Lo creaLe huge namespaces for appllcaLlons wlLhouL
needlng Lo segmenL sLorage sysLems
! Access Lo a wlde selecLlon of avallable Lools and llbrarles LhaL provlde lnLegraLlon
convenlence.


28
Managlng SwlfL wlLh SwlfLSLack
SwlfLSLack provldes Lhe deploymenL, managemenL and monlLorlng plane for SwlfL,
whlch:
! urasLlcally slmpllfles Lhe process of geLLlng SwlfL up and runnlng ln your daLa-
cenLer
! ueploys SwlfL on nodes and conflgures Lhe clusLer
! Lnables you Lo sLarL wlLh one node and add nodes as your daLa grows
! rovldes a cenLral managemenL console for your SwlfL nodes and clusLer
! MonlLors and alerLs for lssues ln nodes, dlsks and oLher resources
! Lnables you Lo easlly expand your clusLer and Lune for performance
! rovldes dlagnosLlcs for lssues, whlch slmpllfles supporL and admlnlsLraLlon

1he SwlfLSLack laLform lncorporaLes deploymenL and operaLlonal besL pracLlces for
SwlfL and provldes a slngle plane of glass" for your enLlre SwlfL envlronmenL. 1o geL
sLarLed wlLh SwlfLSLack, Lhe flrsL sLep ls Lo download Lhe SwlfLSLack lSC conslsLlng of:

! ubunLu
! CpenSLack SwlfL
! SwlfLSLack AgenLs


29
AfLer logglng ln aL Lhe SwlfLSLack laLform, lL wlll gulde you Lhrough Lhe process of
creaLlng a new clusLer, creaLlng accounLs and users, lnsLalllng and provlslonlng SwlfL on
clusLer nodes, formaLLlng drlves, conflgurlng zones and Lhe oLher Lasks requlred Lo seL
up your SwlfL envlronmenL. Cnce your SwlfL envlronmenL has been deployed, Lhe
SwlfLSLack laLform helps provldes Lhe on-golng admlnlsLraLlon, managemenL and
monlLorlng of your SwlfL envlronmenL.

Llke Lo Learn More?
SwlfL and SwlfLSLack offers a real alLernaLlve Lo proprleLary ob[ecL sLorage sysLems and
ls much easler Lo use Lhan LradlLlonal flle-sysLem based approaches. SwlfL ls provlded
under Lhe Apache 2 open source llcense, ls hlghly scalable, exLremely durable and runs
on lndusLry sLandard hardware. SwlfL also has a compelllng seL of compaLlble Lools
avallable from Lhlrd parLles and oLher open source pro[ecLs. WlLh Lhe SwlfLSLack
laLform, deploymenL, on-golng managemenL and monlLorlng can now be done wlLh
ease.
lf you'd llke Lo learn more abouL SwlfL and SwlfLSLack, conLacL us aL
conLacL[swlfLsLack.com.
1he SwlfLSLack 1eam, lebruary 2012

OpenStack Object Storage - Swift

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

OpenStack Object Storage - Swift

Uploaded by

Copyright:

Available Formats

1he CpenSLack

You might also like