Cisco OperatingApplicationCentricInfrastructure

Operating Cisco
Application Centric
Infrastructure
Alejandra Sanchez, Andres Vega, Arvind Chari, Carly Stoughton,

Christopher Stokes, Gabriel Fontenot, Jonathan Cornell, Ken Fee,
Kevin Corbin, Lauren Malhoit, Loy Evans, Lucien Avramov,
Paul Lesiak, Steven Lym, Rafael Muller, Robert Burns
Operating Cisco
Application Centric
Infrastructure
Copyright
Op
er
at
ing Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture
Ale
jan
dra Sanchez, An
dres Vega, Arvind Chari, Carly Stoughton, Christo
pher Stokes,
Gabriel Fontenot, Jonathan Cor
nell, Ken Fee, Kevin Corbin, Lau
ren Mal
hoit, Loy Evans,
Lu
cien Avramov, Paul Lesiak, Rafael Muller, Robert Burns. Steven Lym
Copy
right 2015 Cisco Sys
tems, Inc.
Licensed under the Apache License, Version 2.0 (the "License"); you may not use this
file except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or
agreed to in writing, software distributed under the License is distributed on an "AS IS"
BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
implied. See the License for the specific language governing permissions and
limitations under the License.
Cisco Press logo is a trade

mark of Cisco Sys
tems, Inc.
Pri
vately pub
lished by Cisco Sys
tems, Inc.
Warning and Disclaimer

This book is de signed to pro
vide in
for
ma
tion about Cisco ACI. Every effort has been
made to make this book as com plete and as ac
cu
rate as pos
si
ble, but no warranty or
fit
ness is im
plied.
The infor
mation is pro vided on an as is basis. The au thors, and Cisco Sys tems, Inc.
shall have nei
ther li
abil
ity nor re
spon
si
bil
ity to any per
son or en tity with re
spect to any
loss or damages arising from the infor
mation con tained in this docum
ent.
The opinions ex
pressed in this book be
long to the au
thor and are not nec
es
sar
ily those
of Cisco Sys
tems, Inc.
Feed
back In
for
ma
tion
Read
ers feed
back is a nat
ural con
tin
ua
t
ion of this process. If you have any com
ments
re
garding how we could im
prove the qual
ity of this book, or oth
er
wise alter it to bet
ter
suit your needs, you can contact
us through email at ops-booksprint@ cisco.com .
Contents
. . . . . Prologue
........................................................................ 1
. . . . . Abstract
................................................................... 3
. . . . . Authors
................................................................... 5
. . . . . Book
. . . . . Writing
. . . . . . . .Methodology
...................................................... 7
. . . . . Hardware
. . . . . . . . . .and
. . . .Software
. . . . . . . . .Included
. . . . . . . . in
. . the
. . . .Book
.............................. 9
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
. . . . . Introduction
. . . . . The
. . . . Story
. . . . . .of
. . ACME
. . . . . . Inc.
................................................. 13
. . . . . The
. . . . Why,
. . . . . .Who,
. . . . .What,
. . . . . .When
. . . . . .and
. . . .How
.................................... 15
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
. . . . . Management
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
. . . . . APIC
. . . . . Overview
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
. . . . . Configuring
. . . . . . . . . . . .Management
. . . . . . . . . . . . Protocols
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
. . . . . Role-Based
. . . . . . . . . . . Access
. . . . . . .Control
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
. . . . . Import
. . . . . . . and
. . . . Export
. . . . . . .Policies
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
. . . . . Upgrading
. . . . . . . . . . .and
. . . Downgrading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
. . . . . . . . . . . . . Firmware
. . . . . Section
. . . . . . . .Content
........................................................... 51
. . . . . Firmware
. . . . . . . . . .Management
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
. . . . . Upgrading
. . . . . . . . . . .and
. . . Downgrading
. . . . . . . . . . . . . Considerations
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
. . . . . Upgrading
. . . . . . . . . . .the
. . . Fabric
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
. . . . . Fabric . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
. . . . . . .Connectivity
. . . . . Section
. . . . . . . .Content
........................................................... 71
. . . . . Understanding
. . . . . . . . . . . . . . .Fabric
. . . . . .Policies
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
. . . . . Adding
. . . . . . . New
. . . . .Devices
. . . . . . . .to
. .the
. . . .Fabric
......................................... 81
. . . . . Server
. . . . . . .Connectivity
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
. . . . . Virtual
. . . . . . . Machine
. . . . . . . . Networking
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
. . . . . Deploying
. . . . . . . . . . the
. . . .Application
. . . . . . . . . . .Virtual
. . . . . . Switch
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125
. . . . . External
. . . . . . . . .Connectivity
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
. . . . . Application
. . . . . . . . . . . Migration
. . . . . . . . . .Use
. . . .Case
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
. . . . . Tenants
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .159
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
. . . . . ACI
. . . . Tenancy
. . . . . . . . .Models
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163
. . . . . Application
. . . . . . . . . . . Profile
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
. . . . . Endpoint
. . . . . . . . . Group
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171
. . . . . Endpoint
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
. . . . . Private
. . . . . . . Network
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
. . . . . Bridge
. . . . . . .Domain
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
. . . . . Tenant
. . . . . . . Networking
. . . . . . . . . . . .Use
. . . .Cases
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
. . . . . Working
. . . . . . . . .with
. . . . Contracts
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .215
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
. . . . . Contracts
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
. . . . . Filters
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229
. . . . . Taboo
. . . . . . .Contracts
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
. . . . . Inter-Tenant
. . . . . . . . . . . . .Contracts
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
. . . . . Contracts
. . . . . . . . . .Use
. . . .Cases
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
. . . . . Layer
. . . . . .4. .to
. .Layer
. . . . . .7. Services
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .249
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
. . . . . . . . . . . . . . .Layer
. . . . . 4. .to
. . Layer
. . . . . .7. Integration
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
. . . . . Services
. . . . . . . . .Deployment
. . . . . . . . . . . Guide
. . . . . . Reference
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
. . . . . Service
. . . . . . . .Graph
. . . . . .Monitoring
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
. . . . . ASAv
. . . . . Sample
. . . . . . . .Configuration
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
. . . . . Health
. . . . . . .Scores
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .269
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
. . . . . . . . . . . . . . .Health
. . . . . . Scores
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
. . . . . . . . . . . . . . .Faults
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
. . . . . How
. . . . . Health
. . . . . . .Scores
. . . . . . Are
. . . . Calculated
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
. . . . . Health
. . . . . . .Score
. . . . . .Use
. . . .Cases
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
. . . . . Monitoring
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .285
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
. . . . . Proactive
. . . . . . . . . .Monitoring
. . . . . . . . . . -. .Tenant
. . . . . . .and
. . . .Fabric
. . . . . .Policies
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
. . . . . Proactive
. . . . . . . . . .Monitoring
. . . . . . . . . . -. .Infrastructure
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
. . . . . Proactive
. . . . . . . . . .Monitoring
. . . . . . . . . . Use
. . . . Cases
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 329
. . . . . Reactive
. . . . . . . . .Monitoring
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 333
. . . . . Reactive
. . . . . . . . .Monitoring
. . . . . . . . . . Tools
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
. . . . . Reactive
. . . . . . . . .Monitoring
. . . . . . . . . . Use
. . . . Cases
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
. . . . . Scripting
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .349
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
. . . . . Leveraging
. . . . . . . . . . .Network
. . . . . . . . Programmability
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353
. . . . . ACI
. . . . and
. . . . Scripting
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
. . . . . API
. . . .Inspector
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 363
. . . . . Development
. . . . . . . . . . . . . Techniques
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 367
. . . . . STman
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 369
. . . . . Cobra
. . . . . . SDK
. . . . .and
. . . .Arya
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 375
. . . . . ACI
. . . . Toolkit
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 381
. . . . . GitHub
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
. . . . . Hardware
. . . . . . . . . .Expansion
. . . . . . . . . .and
. . . .Replacement
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .389
. . . . . Section
. . . . . . . .Content
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 391
. . . . . Expanding
. . . . . . . . . . .and
. . . Shrinking
. . . . . . . . . .the
. . . Fabric
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 393
. . . . . Hardware
. . . . . . . . . .Diagnostics
. . . . . . . . . . . and
. . . .Replacement
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 399
. . . . . Appendix
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .413
. . . . . Classes
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 415
. . . . . Package
. . . . . . . . Decoder
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 435
. . . . . Acronyms
. . . . . . . . . . and
. . . .Definitions
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 445
. . . . . Reference
. . . . . . . . . . Material
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 453
....
1
Prologue
Prologue 3
Abstract
Cisco's Appli
ca
tion Centric In
fra
struc
ture (ACI) pro
vides power
ful new ways to dy
nami
-
cally manage infra
struc
ture in the mod ern world of IT automation and De
vOps. Hav
ing
the tools to change how in
fra
struc
ture is built is one thing, but being able to ef
fec
tively
oper
ate the in
fra
structure be
yond the Day Zero build ac
tiv
i
ties is cru
cial to long term ef
-
fec
tive
ness and ef fi
ciency. To ef
fec
tively har
ness the power of ACI, or gani
za
tions will
need to un
der
stand how to in
cor
po
rate ACI into their daily op
er
a
tions. This book ex
am
-
ines some of the com
mon op
er
a
tional ac
tiv
i
ties that IT teams use to pro
vide con
tin
ued
in
fra
struc
ture op
er
a
tions and gives the reader ex
po
sure to the tools, method
olo
gies, and
processes that can be em
ployed to sup
port day 1+ op
er
a
tions within an ACI-based fab
ric.
Prologue 5
Authors
This book repre

sents a joint in
tense col
lab
or
a
tive ef
fort over the course of a week, with
par
tic
ip
a
tion of a number of Cisco func tional or
ga
niza
tions in
clud
ing Cisco Advanced
Ser
vices, Tech
ni
cal Ser
vices, Prod
uct Mar
ket
ing, So
lu
tion Val
id
a
tion Test
ing, IT, and
Sales.
Authors
Ale
jan
dra Sanchez - Cus
tomer Sup
port En
gi
neer, Tech
ni
cal Ser
vices
Andres Vega - Cus tomer Sup port En gi
neer, Tech nical Ser
vices
Arvind Chari - So lutions Ar chitect, Ad vanced Ser vices
Carly Stoughton - Tech nical Mar keting En gi
neer, INSBU
Christopher Stokes - Cisco IT Net work Con sulting En gi
neer
Gabriel Fontenot - Net work Con sulting En gi
neer, Ad vanced Ser vices
Jonathan Cor nell - Sys tems En gi
neer, Sales
Ken Fee - Con sulting Ar chitect, Busi ness Tech nology Architects
Kevin Corbin - Tech nical Solu
tions Ar chitect, Sys tems En gi
neering
Lauren Mal
hoit - Mar keting Con sul
tant, INSBU
Loy Evans - Con sult ing Sys tem En gi
neer, Sales
Lu
cien Avramov - Tech nical Mar keting En gineer, INSBU
Paul Lesiak - So
lu tions Ar chitect, Ad vanced Ser vices
Rafael Muller - Tech ni
cal Leader, So lution Valid
ation Services
Robert Burns - Tech ni
cal Leader, Tech ni
cal Services
Steven Lym - Tech ni
cal Writer, INSBU
6 Prologue
Book Sprint Facilitation

Bar
bara Rhling
Faith Bosworth
Illustrations
Hen
rik van Leuwen
Book Production
Julien Taquet
Clean Up
Raewyn Whyte
Prologue 7
Book Writing Methodology
The Book Sprint (booksprints.

net) method
ol
ogy was used for writ
ing this book. The
Book Sprint method ol
ogy is an in
no
vat
ive new style of coop
er
at
ive and col
lab
o
ra
tive
book pro
duction. Book Sprints are strongly fa
cil
it
ated and lever
age team-oriented in
-
spi
ra
tion and mo
ti
va
tion to rapidly de
liver large amounts of well-au
thored and re
-
viewed con
tent, and in
cor
po
rate it into a com
plete nar
ra
tive in a short amount of time.
By lever
aging the input of many experts, the com
plete book was written in only five
days, but in
volved hundreds of au
thor
ing per
son-hours, and in
cluded thou
sands of ex-
pe
ri
enced en
gi
neer
ing hours, al
low
ing for ex
tremely high qual
ity in a very short pro
-
duc
tion time pe
riod.
Prologue 9
Hardware and Software Included in the Book
The Cisco Appli

ca
tion Cen
tric In
fra
struc
ture (ACI) com
bines hard
ware, soft
ware, and
ASIC in
no
va
tions into an in
te
grated systems approach and pro
vides a common man -
age
ment frame
work for net
work, ap
pli
ca
tion, se
cu
rity, and vir
tu
al
iza
tion.
For the pur

pose of writ
ing this book, the fol
low
ing hard
ware de
vices were used:
Cisco Application Policy Infrastructure Controller (APIC)

ACI Spine switches, including Cisco Nexus 9508 and 9336PQ
ACI Leaf switches, including Cisco Nexus 9396PX, 9396TX, and 93128TX
Cisco Application Virtual Switch (AVS)
Cisco UCS B and C series servers
Several models of switches and routers, including Cisco Nexus 5000 switches
and Cisco Integrated Services Routers (ISR)
A variety of hypervisors, including KVM, Microsoft Hyper-V, and VMware
vSphere
IXIA IxVM product family traffic generators
This book was writ

ten based on the Cisco ACI 1.1 soft
ware re
lease.
11
Introduction
Introduction 13
The Story of ACME Inc.
ACME Inc. is a multi na

tional corpo
ra
tion that special
izes in man
uf
ac
tur
ing, sales, and
dis
tri
bu
t
ion of a di
verse product port
fo
lio, in
clud
ing rocket-powered roller skates, jet-
pro
pelled uni
cy
cles, and var
io
us ex
plo
sive ma
te
ri
als. These prod
uct groups op
er
ate as
separate busi
ness groups within the company, and have pre
vi
ously maintained sep
arate
in
frastructure and ap
pli
ca
tions. They have largely fo
cused on re
tail routes to market,
but have re
cently de
cided to pur
sue a more di
rect-to-con
sumer busi
ness model due
to in
tense pres
sure from new com
peti
tors who have dom
in
ated the on
line sales chan
-
nels. In an ef
fort to be more com
pet
it
ive, ACME has un
der
taken a pro
ject to build a mo
-
bile ap
pli
ca
tion plat
form to sup port or
der
ing and lo
gis
tics for prod
uct de
liv
ery to their
customers for their entire port
fo
lio.
Traditionally, ACME busi ness units have lever aged third party soft ware com pa
nies and
com mercially avail able software to meet their IT de mands, but would like to cre ate a
more in ti
mate re la
tion
ship with their con sumers and be able to take feed back on the
platform di rectly from those users, while in corporat
ing an on go
ing improvement cycle
so they can react to chang ing mar ket dynam ics in a more nim ble fash
ion. Where they
have used cus tom software in the past, they have lever aged a tradi
tional in
fra
struc
ture
and soft ware model that does not allow them to keep up with the chang ing re
quire-
ments, and there fore ACME is look ing for a new ap proach to both ap pli
ca
tion and in -
fra
struc ture life cycle man age ment. The ap plica
tion devel
opers have been look ing at
new ap pli
ca tion devel
opment trends such as Con tin
uous Deliv
ery and Con tin
uo
us Inte-
gration, and the new ap pli
cation platform is to be de vel
oped in this man ner. To support
this, the in frastructure com po nents need to be ca pa
ble of map ping to these new par a-
digms in a way that is not pos si
ble using tra
di
tional con cepts.
One of the largest chal lenges ACME has his tor

ic
ally faced is that oper
at
ions and infra-
structure has been an af ter
thought to prod uct de vel
opment. This has led to several sit
-
uat
ions where ap plica
tion de ployments have meant long week end hours for all of the
teams, caused cus tomer-im pacting outages, and taken longer to ac com plish than the
business lead
ers would have liked. For this rea son, ACME Inc. has de cided to change by
creat
ing an envi
ronment where in fra
struc
ture ar ti
facts are treated as part of the appli
-
cation, can be checked into ver sion control, can be tested along side the actual ap
pli
ca-
tion, and can con tin
ually improve.
14 Introduction
While ACME is in
tensely fo
cused on de
liv
er
ing the new ap
pli
ca
tion plat
form in a timely
man
ner, ACME is also in
ter
ested in cre
at
ing a foun
da
tion on which it can grow to de
-
liver a com
mon pool of in
fra
struc
ture that is shared across all busi
ness groups and op
-
er
ated in a multi-ten
ant fash
ion to in
crease ef
fi
ciency.
At an ex
ec
ut
ive brief
ing, John Cham
bers, the CEO of Cisco Sys
tems, told ACME:
"The world is chang
ing. Every com
pany is a tech
nol
ogy com
pany, and if you don't
adapt, you'll get left be
hind."
As ev
id
enced by the success of cloud plat
forms, such as Ama zon Web Ser vices
(AWS) and Open
stack, con
sumption mod
els of tech
nol
ogy de
liv
ery have the abil
ity to
adapt tech
nol
ogy more quickly to rapid busi
ness re
quire
ments changes. This is the type
of con
sump
tion that ACME Inc.'s busi
ness own
ers need. Con
trol of op
er
at
ions is what
op
er
at
ions groups are fo
cused on, but con
trol can be a bar
rier to a pure con
sump
tion
model. Unless compa
nies make in
vestments in technolo
gies that allow for con
sumption
of au
to
mated com po
nents, the only other way to scale is by break ing the human level
component, and few peo
ple would really choose to work for that type of company.
After an al
yzing cur rent of fers from var io
us tech nology ven dors, ACME Inc. se lected
Cisco Ap pli
cation Cen tric Infra
structure (ACI). The abil ity to ab stract all phys
ic
al and
vir
tual infra
struc ture con fig
ura
tion into a sin
gle con fig
ura
tion that is con sis
tent across
dev, test, and prod en vi
ron ments, as well as portable across the var io
us data center lo-
cations cur rently main tained by ACME, is highly de sir
able. ACI has been built from the
ground up to change the sub structure used to build net work de vices and pro to
cols. In-
novation at this level will pro vide more op portuni
ties for ex pand ing the tools with
which users in teract. This is where the ful crum will tilt in the favor of IT and in fra
struc -
ture being more dy namic, thus al lowing IT to op
erate and man age at the speed of busi -
ness. How ever, with a change of this na ture comes fear, un certainty, and doubt. This
book will at tempt to bring some level of com fort and fa miliar
ity with op er
at
ions ac
tiv
i-
ties within an ACI fab ric.
While ACME Inc. is a fic ti

tious company, this is the true story of every com pany, and
just impor
tant this is the story of the em ploy
ees of those com pa
nies. Work ers in the IT
in
dustry need to adapt to keep up with the rapid change of the busi ness. How ever,
this runs contrary to how most op er
at
ions groups exist in the re la
tion
ship be tween
business and tech nology. Most IT op er
at
ions groups invest a lot of time in the tools
needed to de liver ser vices today and there is an or ganic re
sis
tance to re-in vest
ing.
The thought is, "Why fix what is al ready working?"
Introduction 15
The Why, Who, What, When and How
Within ACI, ACME is look ing to simplify how it oper

ates in
fra
struc
ture, but rec
og
nizes
that this ini
tia
tive, this ap
pli
ca
tion, and this in
fra
structure are new to ACME Inc. ACME
must ad
dress fun
da
men
tal ques
tions in
clud
ing Who man
ages What, and How they go
about their tasks. When dif
fer
ent groups perform regul
ar op
er
at
ions, and Where they
go to per
form these oper
at
ions, are also con
sid
er
at
ions, but more tacti
cal and point-
in-time-rel
e
vant. This sec
tion dis
cusses the rel
e
vant as
pects of these monikers as it re
-
lates to ACI fab
ric op
er
at
ions and how a com
pany such as ACME Inc. can di
vide the
work
load.
Why
"Why" is the most im portant aspect of what should be con sid
ered in op erational
iz
ing an
ACI fab ric. In the case of ACME Inc. the key suc cess cri teria is to stream line processes
and pro ce dures re lated to the de ployment of in fra
struc ture re quired to sup port the ap -
pli
cation initia
tives. To achieve the de sired re sult, a high de gree of au toma tion is re -
quired. Au toma tion adds speed to repet it
ive tasks and elim in
ates er rors or missed
steps. Ini tially, au
toma tion can be a scary propo si
tion for some of the key stake hold ers,
as it could be looked at as a threat to their own job. Quite the op po
site, au toma tion is
about mak ing work more en joy
able for all team mem bers, al low ing them the free dom to
in
novate and add value, while re mov ing mun dane, repet itive tasks. Look ing at why an
auto
mated fab ric is benefi
cial to an organiza
tion is im por tant for set ting ex pectations
of return on in vestment. Also, look ing at why an op erational prac tice is done a spe cific
way can help with fram ing the tools and processes that are em ployed.
Who
As with most or ga ni
za
tions, ACME Inc. tra di
tionally had differ
ent types of stake holders
in
volved in mak ing any IT initia
tive successful, and each has a spe cific el
ement of the
in
fra
structure in which they have spe cific ex
per tise and about which they care most. In
any IT or gani
zation, these groups can have dis tinct orga
ni
za
tional bound aries, but
more likely the bound aries are blurred to some de gree. Listed below are some char ac
-
ter
is
tics of these groups, but keep in mind that some of these char ac
teris
tics might be
com bined. At the macro level, the fact that these dif fer
ent orga
ni
zations exist should
16 Introduction
not be ev
id
ent to the end-user. In
stead, the en
tire or
ga
ni
za
tion should be seen as one
team with a com
mon goal of de
liv
er
ing value to their or
ga
ni
za
tion.
ACME's De
vel
op
ment and Ap
pli
ca
tion Team is fo
cused on the soft
ware and ap
pli
ca
tions
the com
pany uses in
ter
nally and the soft
ware that it de
liv
ers to its cus
tomers. The Ap
-
pli
ca
tion part of the team con
tains ap
pli
ca
tion own
ers and sub
ject mat
ter ex
perts that
en
sure other busi
ness units are able to do their jobs by uti
liz
ing the busi
ness ap
pli
ca
-
tions avail
able. The De
vel
op
ment part of the team will be writ
ing the mo
bile ap
pli
ca
-
tion soft
ware platform. Both parts of this team will need to work closely with the other
teams in this sec
tion to en
able the best design, per
for
mance, and avail
abil
ity of ap
pli
ca
-
tions for the end users.
ACME's Net
work Team is pri
mar
ily fo
cused on cre
at
ing and man
ag
ing net
work
ing con
-
structs to for ward pack ets at Layer 2 (MAC/switch ing) and Layer 3 (IP rout ing). The
team is chal lenged with jug gling appli
ca
tion re quirements, man ag
ing SLA, and as sist
ing
in the en forcement of in formation security, all while main tain
ing high lev els of avail-
abil
ity. What the team needs to know is how to con fig
ure the network ing con structs,
how to tie Layer 2 to Layer 3, how to ver ify for
ward ing, and how to trou bleshoot net -
work for ward ing aspects in the fabric. With ACI, the team is the most in terested in de -
coupling over loaded net work con structs and re turning to the spe cific network prob -
lems that the team was in tended to solve, while al lowing other groups to lever age their
specific ex
pertise to manipul
ate security and ap pli
ca
tion level poli
cies. The team is also
in
terested in allowing more trans parency in the per for mance of the net work forward -
ing, and the team is mak ing key met rics avail able on de mand in a self-ser vice ca
pacity.
ACME's Stor age Team is pri mar ily fo

cused on de liv
ery of data stor age resources to the
or
ganiza
tion. The stor age team is con cerned with pro tecting the data in terms of avail -
abil
ity, as well as mak ing sure that sen si
tive data is se cure. The stor age team has been
very suc cessful in main tain
ing very tight SLAs and has tra ditionally man aged sep arate
in
frastructure for stor age ac cess. The ca pabil
ities pro vided by the ACI fab ric allow
them to con fi
dently de ploy newer IP-based stor age and clus tering tech nologies. The
team is also very in terested in being able to see how the stor age ac cess is per form -
ing and would like to be no ti
fied in the event of con tention. The team typ ic
ally
has some spe cific re
quire ments around QoS, multi-pathing, and so on. His tor
ic
ally, the
team had to worry about de livering a stor age fab ric in addition to man ag
ing storage de -
vices them selves. ACI will pro vide the stor age team with the vis ib
il
ity they will require.
These ca pabil
it
ies are primar ily discussed in the mon itor
ing sec tions.
Introduction 17
The Com
pute and Vir
tu
al
iza
tion Team at ACME Inc. is wrap
ping up a major ini
tia
tive to
vir
tu
al
ize the server farms that it is re
spon
si
ble for main
tain
ing. The team also re
cently
em
ployed new con
fig
ur
a
tion man
age
ment tools to ac
count for new work
loads that fell
out
side of the vir
tu
al
iza
tion effort to get sim
il
ar agility for bare metal servers that the
team gained from its virtual
iza
tion ef
forts. This is timely as the ap
pli
ca
tion roll
out will
have both vir
tu
al
ized and non-vir
tu
al
ized work
loads. Ad
di
tion
ally, the ap
pli
ca
tion de
-
vel
op
ers are in
creas
ingly in
ter
ested in lever
ag
ing Linux con
tainer tech
nolo
gies to allow
for even greater ap
pli
ca
tion porta
bil
ity. The Com
pute and Vir
tu
al
iza
tion teams are in
-
ter
ested in ACI for its abil
ity to pro
vide com
mon ac
cess to phys
ic
al and vir
tual servers,
al
low
ing the team to publish end
point groups to vir
tu
aliza
tion clus
ters from a central
-
ized place across mul
ti
ple hyper
vi
sors. These ca
pa
bil
it
ies are discussed fur
ther in the
Fab
ric Con
nec
tiv
ity chap
ter.
The In
for
ma
tion Se
cu
rity Team at ACME Inc. has tra
di
tion
ally been en
gaged late in an
appli
cation de ploy ment process, and has been re spon si
ble for perform ing vulnerab
il
ity
assess
ment and data clas si
fi
ca
tion ef
forts. With the cur rent project, the new ap pli
ca-
tion will be stor ing sensitive customer in for
mation, including credit card num bers. Due
to the sen si
tiv
ity of this in formation and the se cu
rity aspects of the ACI fab ric, the In-
formation Se cu rity Team is able to pro vide input ear lier in the process and avoid re-do -
ing work be cause of se curity or com pli
ance is
sues. The In for
mation Se cu
rity Team is
in
terested in the op er
ational aspects of the ACI se cu rity model as it re lates to the fol-
lowing ca pabil
it
ies: tenancy, Role Based Ac cess Con trol (RBAC), mon it
oring, and Layer 4
to Layer 7 ser vices.
What
The aspect of "what" can be looked at in many dif fer
ent ways, but the main con cept in
the context of this book is what tools are used to man age op
er
at
ions of an ACI fab
ric. In
a tra
di
tional network, you have some tra di
tional tools, such as CLI and SNMP, to man -
age network op er
at
ions, and these tools in
tegrate into management platforms and con -
fig
ur
a
tion and man age
ment processes.
In ACI there are some el ements of the tra di

tional tools, but the fab ric manage
ment is
rooted in an ab stracted object model that pro vides a more flex ib
le base. With this base,
the op er
at
or of the fabric can choose from mul ti
ple modes of man agement, such as
GUI, CLI, API in te
gra
tion, program ming, script ing, or some com bi
nation of these. How
a tool is se
lected in ACI will often be a prod uct of what is being done and the as pects of
how the tool is used. For ex am ple, if an oper
ations staff is try
ing to gather a bunch of
in
for
ma tion across a num ber of inter faces and switches or is man ag
ing the config
ur
a
-
18 Introduction
tion of many dif

fer
ent ob
jects at once, script
ing might be more ef
fi
cient, whereas sim
-
ple dash
board mon
it
or
ing might be more suited to a GUI.
When
"When" refers to when the teams listed above are involved in the plan
ning. It is a good
idea to in
volve the dif
fer
ent teams early when build
ing poli
cies and processes for how
the fab
ric is im
ple
mented and then man
aged. The col
lab
o
ra
tive na
ture of ACI al
lows for
a high degree of par
al
leliza
tion of work flow. This is a key dif
fer
ence between ACI and
tra
di
tional processes that were very se ri
al in na
ture, re
sult
ing in a longer de
ploy
ment
time for ap
pli
ca
tions and a higher mean-time to res
ol
u
tion when is
sues arise.
How
"How" an
swers the fol
low
ing basic ques
tions:
How does a networking person go about configuring the network forwarding?

How does the compute team get information from the infrastructure to make
optimal workload placement decisions?
How do the application team track performance and usage metrics?
How does a storage team track the access to storage subsystems and ensure
that it is performant?
When "how" in volves mak ing a change to the con fig

ur
a
tion of an en vi
ron ment, an im -
portant con sid
er
at
ion is change con trol. Change con trol is a fact of life in the mission-
crit
ic
al envi
ron
ments that ACI has been de signed to sup port. The ACI pol icy model has
been de signed to reduce the over all size of a fault domain and pro vide a mech an
ism for
in
cremen tal change. There are mech an
isms for backup and re store that will be dis -
cussed in fol low-on chap ters. We will also dis cuss the model and which ob jects af
fect
the tenants and the fab ric as a whole.
An eval
ua
t
ion of cur
rent change con trol and contin
uo
us inte
gra
tion/deliv
ery strategies
is war
ranted as oper
at
ional proce
dures evolve. Through out this book we will high light
the methods and procedures to proactively and re
ac
tively manage the fabric.
As a baseline, most orga

ni
za
tions are im
plementing some kind of struc tured change-
control method ol
ogy to mit
ig
ate busi
ness risk and en hance system availabil
ity. There
are a num ber of change/IT man age
ment prin ci
ples (Cisco Lifecy
cle ser
vices, FCAPS,
Introduction 19
and ITIL) that are good guides from which to start. A com mon sense ap proach to
change man age
ment and con
tin
uo
us in
te
gra
tion should be a premise that is dis
cussed
early in the de
sign and im
ple
menta
tion cycle before handing the fab
ric to the oper
a-
tions teams for day-to-day main
tenance, monit
oring, and pro
vi
sion
ing. Training op
er-
a
tions teams on norms (a stated goal of this book) is also key. Ap
ply
ing change man
-
age
ment prin
ci
ples based on tech
nol
ogy from five years ago would not en
able the rapid
de
ploy
ment of tech
nol
ogy that ACI can de
liver.
The multi-ten
ant and role-based ac
cess con
trol fea
tures in
her
ent to the ACI so
lu
tion
allow the iso
la
tion or draw
ing of a very clean box around the scope and im pact of the
changes that can be made. For more de tails, see the RBAC sec
tion of this book.
Ul
ti
mately each change must be eval
ua
ted pri
mar
ily in terms of both its risk and value
to the busi
ness. A way to enable a low-overhead change man age
ment process is to re-
duce the risk of each change and in crease its value. Con tin
uo
us de
liv
ery does exactly
this by en
suring that re
leases are performed reg ul
arly from early on in the de liv
ery
process, and ensur
ing that de
liv
ery teams are work ing on the most valuable thing they
could be at any given time, based on feedback from users.
In the In
for
ma
tion Man
age
ment Sys
tems world, there are three fun
da
men
tal kinds of
changes:
Emergency changes
Normal
Standard
Emergency changes are by de f

i
n
i
tion a re
sponse to some kind of technical out
age (hard
-
ware, soft
ware, in
fra
struc
ture) and are performed to re
store ser
vice to af
fected sys
tems.
Normal changes are those that go through the reg ul

ar change man agement process,
which starts with the creation of a request for change which is then reviewed, assessed,
and then either au
thorized or re jected, and then (assuming it is au
thorized) planned
and imple
mented. In an ACI en viron
ment a nor mal change could apply to any thing
within the fol
low
ing com ponents:
Fabric Policies (fabric internal and access will be discussed in detail later)
Configuration objects in the Common tenant that are shared with all other
tenants (things that affect the entire fabric)
Private Networks
20 Introduction
Bridge Domains
Subnets
Virtual Machine Manager (VMM) integrations
Layer 4 to Layer 7 devices
Device packages
Creation of logical devices
Creation of concrete devices
Layer 2 or Layer 3 external configuration
Attachable Entity Profile (AEP) creation
Server or external network attachment
Changes to currently deployed contracts and filters that would materially
change the way traffic flows
Stan
dard changes are low-risk changes that are pre-au
tho
rized. Each or
ga
ni
za
tion will
de
cide the kind of standard changes that they allow, who is al lowed to ap prove
them, the cri
te
ria for a change to be considered "stan
dard", and the process for manag
-
ing them. As with nor mal changes, they must still be recorded and ap proved. In the ACI
en
viron
ment some ex amples of "stan
dard" changes could be:
Tenant creation
Application profile creation
Endpoint group (EPG) creation
Contracts scoped at a tenant level
Layer 4 to Layer 7 service graphs
Domain associations for EPGs
The items mentioned above are not in

tended to be all-in
clu
sive, but are rep
re
sen
ta
tive
of common tasks per
formed day-to-day and week-to-week.
The ability to audit changes that are hap pening to the envi
ronment is a re
quirement for
ACME Inc. APIC main tains an audit log for all con
fig
ur
a
tion changes to the sys tem. This
is a key troubleshooting tool for "when some thing magic
ally stops working". Im
medi
ate
ac
tion should be to check the audit log as it will tell who made what change and when,
cor
re
lat
ing this to any faults that re
sult from said change. This en
ables the change to be
re
verted quickly.
Introduction 21
A more in-depth dis

cus
sion of con
tin
uo
us de
liv
ery in the con
text of in
fra
struc
ture
man
age
ment is out
side of the scope of this book.
The re
main
der of this book an
swers these ques
tions, pro
vid
ing you with a frame
work
of how to take the con
cepts and pro
ce
dures and apply them to sim
il
ar ini
tia
tives within
your or
gani
za
tions. The book is laid out in a spe
cific order. However, ACI en ables
ACME Inc. to com plete these tasks in par al
lel with the var
io
us stakehold
ers who are
highlighted through out, and this book il lus
trates how the stake holders can work to -
gether in a more col labor
a
tive manner than they have in the past. While some script ing
opportuni
ties are called out through out the book, there is a more in-depth sec tion at
the end that ex plains how to use the ACI API to au to
mate most op er
at
ional tasks. While
or
ganiza
tional struc tures might be siloed into these teams, to pro vide the greatest
value to the cus tomer, user, and ul ti
mately the busi ness, the most im por
tant thing is
for these groups to work in sieme (to
gether).
ACI addresses IT requirements from across the organization

23
Management
Management 25
Section Content
APIC Overview
Configuring Management Protocols

Cisco Discovery Protocol
Link Layer Discovery Protocol
Time Synchronization and NTP
Out-of-Band Management NTP
In-Band Management NTP
Verify NTP Operation
Verifying that the NTP Policy Deployed to Each Node
Domain Name Services (DNS)
Verifying DNS Operation
Role-Based Access Control

Multiple Tenant Support
User Roles
Security Domains
Creation of a Security Domain
Adding Users
Remote Authentication
Import and Export Policies

Configuration Export (Backup)
Add a Remote Location (SCP)
Create a One Time Export Policy
Verify Export Policy was Successful
Extract and View Configuration Files
Configuration Import (Restore/Merge)
Management 27
APIC Overview
There are a number of fun

da
men
tal dif
fer
ences between the oper
at
ions of tra
di
tional
net
working hard
ware and an ACI fab
ric. These dif
fer
ences serve to sim
plify the man-
age
ment greatly, re
duce the num
ber of touch
points, and de
cou
ple the switch
ing hard
-
ware from the de
sired con
fig
ur
a
tion in
tent. These changes in
clude:
Single point of management controller-based architecture

Stateless hardware
Desired state-driven eventual consistency model
The sin gle point of man age

ment within the ACI ar chi
tecture is known as the Ap pli
ca-
tion Policy In
fra
structure Con troller (APIC). This con troller pro vides ac
cess to all con -
fig
ur
ation, management, mon it
oring, and health func tions. Hav ing a centralized con -
troller with an appli
cation pro
gram ming inter
face (API) means that all func tions con fig
-
ured or ac cessed through the fab ric can be uniformly ap proached through a graph ic
al
user inter
face (GUI), com mand line in ter
face (CLI), and ap pli
ca
tion program ming in ter-
face (API), with no risk of in consis
tency be tween the var io
us data inter
faces. This re -
sults in a clean and pre dictable transi
tion between the in terfaces. The under ly
ing inter-
face for all ac
cess meth ods is provided through a REST-based API, which mod if
ies the
contents of a syn chro nized database that is repli cated across APICs in a clus ter and
provides an abstraction layer between all of the inter
faces.
This con troller-based ar chi

tecture also makes pos si
ble a state less con fig
uration
model that de cou
ples the hard ware from the con fig
ur a
tion run ning on it. This trans -
lates to an APIC clus ter that man ages indi
vid
ual fabric nodes of leaf and spine switches
that derive their identity from what the con troller de fines as being the de sired in tent,
and not from the se ri
al num ber of the chas sis, nor from a con fig
ura
tion file residing on
the devices. Each node re ceives a unique node iden ti
fier, which al lows for the de vice to
down load the cor rect con figur
a
tion attrib
utes from the con troller. The de vice can
also be sub sti
tuted in a state less fashion, mean ing that hard ware swaps can be faster,
topology changes are less im pactful, and net
work man age ment is sim pli
fied.
The de sired state model for con

fig
ur
ation fur
ther comple
ments these concepts of con
-
troller-based man age
ment and state less
ness by taking ad
van
tage of a con
cept known
as declarat
ive con
trol-based man agement, based on a con cept known as the promise
28 Management
the
ory. De
clar
at
ive con
trol dic
tates that each ob
ject is asked to achieve a de
sired state
and makes a "promise" to reach this state without being told precisely how to do so.
This stands in con
trast with the tra
di
tional model of im
perat
ive control, where each
man
aged el
e
ment must be told pre
cisely what to do, be told how to do it, and take into
ac
count the spe
cific sit
ua
t
ional as
pects that will im
pact its abil
ity to get from its cur
-
rent state to the con
fig
ured state. A sys
tem based on de
clar
at
ive con
trol is able to scale
much more ef
fi
ciently than an im
per
at
ive-based sys
tem, since each en
tity within the
do
main is responsi
ble for know
ing its cur
rent state and the steps re
quired to get to the
de
sired state, dic
tated by the managing con
troller.
Management 29
Configuring Management Protocols
For the op
ti
miza
tion of ACME's Cisco Ap pli
ca
tion Cen
tric In
fra
structure (ACI) fab
ric,
the net
work team created the fol
low
ing man
agement pro
tocol poli
cies:
Cisco Discovery Protocol (CDP) - These policies are primarily consumed by the
network team, as CDP is the standard for their existing network equipment.
Link Layer Discovery Protocol (LLDP) - Discovery protocols are no longer
limited to just network devices. LLDP is a standards-based protocol for
discovering topology relationships. ACME is deploying LLDP on servers, Layer 4
to Layer 7 services devices, and storage arrays. Therefore, these policies will be
available for consumption by all of the teams.
Network Time Protocol (NTP) - Maintaining accurate time across the
application logs and infrastructure components is extremely important for
being able to correlate events. ACME has existing NTP servers and will
leverage them for maintaining time in their ACI fabric deployment. With ACI,
maintaining accurate time is of increased importance as accurate time is a
prerequisite for features such as atomic counters.
Domain Name Services (DNS) - providing name resolution to fabric components
consistent with the application and server teams. In addition to be able to
resolve names within the fabric, the important ACI components are also
registered with ACME's DNS server so teams can easily access them in their
management tasks.
Cisco Discovery Protocol

Cisco Dis covery Protocol (CDP) is a valu able source of in for
ma tion in troubleshooting
physic
al and data-link layer is sues. In the course of ACI op er
at
ions, there may be times
when you must pro vi
sion a partic
ul
ar host-fac ing port man ua
lly with a CDP on or off
pol
icy. By de fault, the ACI fab ric provides a de fault pol
icy where CDP is dis abled. As
a rec
om mended prac tice, man ua
lly cre
ate a CDP-EN ABLED" pol icy with CDP en abled,
as well as a CDP-DIS ABLED pol icy with CDP dis abled that can be ref er
enced through -
out all in
terface policy config
ura
tions. It is also impor
tant to note that CDP might be
re
quired for cer tain con fig
ur
a
tions, such as when to cre ate switch pro files to con
nect
to Cisco UCS B Se ries servers.
30 Management
To cre
ate CDP poli
cies:
1 On the menu bar, choose Fabric > Access Policies.
2 In the Navigation pane, choose Interface Policies > Policies > CDP Interface.
3 In the Work pane, choose Actions > Create CDP Interface Policy.
4 In the Create CDP Interface Policy dialog box, perform the following actions:
a. In the Name field enter the name of the policy, such as "CDP-ENABLED".
b. For the Admin State radio buttons, click Enabled.
c. Click Submit.
a. In the Name field enter the name of the policy, such as "CDP-DISABLED".
b. For the Admin State radio buttons, click Disabled.
c. Click Submit.
Your CDP pol

icy is now ready for de
ploy
ment to the ACI fab
ric in
ter
faces.
Link Layer Discovery Protocol

Link Layer Dis covery Pro to
col (LLDP) is a valu able source of in
for
ma tion for trou -
bleshoot ing phys ic
al and data-link layer is
sues. In the course of ACI op erat
ions, by de-
fault, the LLDP fea ture is enabled on the fabric. There might be times in the op er
at
ion
of the ACI fab ric in which you must man ually adjust the LLDP pro
tocol to con form with
in
teroper
abil
ity requirements of end-host de vices. For ex
ample, when con necting to
Cisco Uni fied Com puting System (UCS) Blade servers, you must dis able LLDP.
Cisco rec
om mends that you pre-pro vi
sion LLDP en able and dis
able pro
tocol poli
cies to
make future in
ter
face pol
icy de
ploy
ment de ci
sions streamlined and eas
ily config
ured.
To cre
ate an LLDP pol
icy:
2 In the Navigation pane, choose Interface Policies > Policies > LLDP Interface.
3 In the Work pane, choose Actions > Create LLDP Interface Policy.
4 In the Create LLDP Interface Policy dialog box, perform the following actions:
Management 31
a. In the Name field, enter "LLDP-TX-ON-RX-ON".

b. For the Receive State radio buttons, click Enabled.
c. For the Transmit State radio buttons, click Enabled.
d. Click Submit.
a. In the Name field, enter "LLDP-TX-ON-RX-OFF".
b. For the Receive State radio buttons, click Disabled.
c. For the Transmit State radio buttons, click Enabled.
d. Click Submit.
a. In the Name field, enter "LLDP-TX-OFF-RX-ON".
b. For the Receive State radio buttons, click Enabled.
c. For the Transmit State radio buttons, click Disabled.
d. Click Submit.
a. In the Name field, enter "LLDP-TX-OFF-RX-OFF".
b. For the Receive State radio buttons, click Disabled.
c. For the Transmit State radio buttons, click Disabled.
d. Click Submit.
Your LLDP pol

ploy
ment to the ACI fab
ric in
ter
faces.
Time Synchronization and NTP

Within the Cisco Ap pli
cation Centric In
frastructure (ACI) fab ric, time syn chronization is
a crucial ca
pability upon which many of the mon it
oring, operat
ional, and trou bleshoot -
ing tasks depend. Clock syn chronization is important for proper analy sis of traf
fic flows
as well as for cor relat
ing debug and fault time stamps across mul ti
ple fabric nodes.
An offset present on one or more de vices can ham per the abil ity to prop erly diagnose
and resolve many com mon op er
at
ional is
sues. In ad dition, clock syn chro niza
tion allows
for the full utilization of the atomic counter ca pa bil
ity that is built into the ACI upon
which the ap plication health scores de pend. Nonex istent or im proper con fig
ura
tion of
time syn chro
niza tion does not nec es
sarily trig
ger a fault or a low health score. You
32 Management
should config
ure time syn
chroniza
tion before de
ploy
ing a full fab
ric or ap
pli
ca
tions so
as to en
able proper usage of these features. The most widely adapted method for syn -
chro
niz
ing a de
vice clock is to use Net
work Time Pro
to
col (NTP). For more in
for
ma
tion
on atomic coun
ters, see the Re
ac
tive Mon
it
or
ing Tools chap
ter.
Prior to con
fig
ur
ing NTP, con
sider what man
age
ment IP ad
dress scheme is in place
within the ACI fab
ric. There are two op
tions for con
fig
ur
ing man
age
ment of all ACI
nodes and Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
trollers (APICs), in-band man
age
ment
and/or out-of-band man agement. De
pend ing on which man
age
ment op
tion was cho
-
sen for the fab
ric, con
fig
ur
a
tion of NTP will vary.
An
other con
sid
er
at
ion in de
ploy
ing time syn
chro
niza
tion where the time source is lo
-
cated. The re
li
ab
il
ity of the source must be care
fully con
sid
ered when de
ter
min
ing if
you will use a pri
vate in
ter
nal clock or an ex
ter
nal pub
lic clock.
Out-of-Band Management NTP

When an ACI fab ric is de
ployed with out-of-band man age
ment, each node of the fab
ric,
inclu
sive of spines, leaves and all mem bers of the APIC cluster, is man
aged from out -
side the ACI fab ric. This IP reach
ab
il
ity will be lever
aged so that each node can in
di
vid-
ually query the same NTP server as a con sistent clock source.
To con fig
ure NTP, a Date and Time pol icy must be cre ated that ref
er
ences an out-of-
band man agement end
point group. Date and Time poli cies are con
fined to a single
pod and must be de ployed across all pods pro
vi
sioned in the ACI fab ric. As of the pub
-
lish
ing of this doc
um
ent, only one pod per ACI fabric is al
lowed.
1 On the menu bar, choose Fabric > Fabric Policies.
2 In the Navigation pane, choose Pod Policies > Policies.
3 In the Work pane, choose Actions > Create Date and Time Policy.
4 In the Create Date and Time Policy dialog box, perform the following actions:
a. Provide a name for the policy to easily distinguish between your
environment's different NTP configurations, such as "Production-NTP".
b. Click Next.
c. Click the + sign to specify the NTP server information (provider) to be used.
d. In the Create Providers dialog box, enter all relevant information, including
the following fields: Name, Description, and Minimum Polling Intervals, and
Maximum Polling Intervals.
Management 33
i. If you are creating multiple providers, click the Preferred check box for
the most reliable NTP source.
ii. In the Management EPG drop-down list, if the NTP server is reachable by
all nodes on the fabric through out-of-band management, choose Out-of-
Band. If you have deployed in-band management, see the "In-Band
Management NTP" section.
iii. Click OK.
e. Repeat steps c and d for each provider that you want to create.
Your NTP pol

ploy
ment to the ACI fab
ric nodes.
In-Band Management NTP

When an ACI Fab ric is de
ployed with in-band man agement, con sider the reachab
il
ity of
the NTP server from within the ACI in-band man agement net work. In-band IP ad dress-
ing used within the ACI fab ric is, by def
in
i
t
ion, not reach able from any where out side
the fab
ric. To leverage an NTP server ex ternal to the fabric with in-band man age
ment,
you must con struct a policy to en able this com mu ni
ca
tion. The steps used to con figure
in-band man agement poli cies are iden ti
cal to those used to es tablish an out-of-band
man age
ment pol icy: the distinction is around how to allow the fab ric to con
nect to the
NTP server.
Once you have es tablished a pol

icy to allow com
muni
ca
tion, you can cre
ate the NTP
pol
icy as es
tab
lished in the "Out-of-Band Man age
ment NTP" section.
Verifying NTP Operation

Full verif
i
ca
tion of NTP functional
ity is best accom plished by leverag
ing both the ACI
CLI and the APIC GUI. Start ver if
ying time syn chronization con
fig
ur
ation by ensur
ing
that all poli
cies are con
fig
ured prop erly in
side of the APIC GUI. Policy config
ur
a
tion can
be verif
ied by these steps:
2 In the Navigation pane, choose Pod Policies > Policies > Date and Time >
ntp_policy > server_name. The ntp_policy is the previously created policy.
3 In the Work pane, verify the details of the server.

34 Management
Verifying that the NTP Policy Deployed to Each Node
To ver
ify that the pol
icy has been suc cessfully de
ployed to each node, you should use
the NX-OS Vir tual Shell that is pre
sent in the APIC. To ac
cess the NX-OS Virtual Shell,
open an SSH session to the out-of-band man age
ment IP in
ter
face of any of the APIC
nodes. From this prompt, exe
cute the "ver
sion" com
mand to ob tain a con
sol
id
ated list
of node host
names.
1 SSH to an APIC in the fabric.
2 Press the Tab key twice after the entering the attach command to list all of the
available node names:
admin@apic1:~> attach <Tab> <Tab>
3 Log in to one of the nodes using the same password that you used to access the
APIC.
admin@apic1:~> attach node_name
4 View the NTP peer status:
leaf-1# show ntp peer-status
A reach able NTP server has its IP ad

dress pre
fixed by an as
ter
isk (*), and the
delay is a non-zero value.
5 Repeat steps 3 and 4 for each node on the fabric.
Domain Name Services (DNS)

Set
ting up a DNS server al
lows the APIC to re
solve var
io
us host
names to IP ad
dresses.
This is use
ful when in
te
grat
ing VMM domains or other Layer 4 to Layer 7 devices and
the host
name is ref
er
enced.
2 In the Navigation pane, choose Global Policies > DNS Profiles > default.
3 In the Work pane, in the Management EPG drop-down list, choose the
appropriate management EPG.
Management 35
Note: The default is default (Out-of-Band).
4 Click + next to DNS Providers to add a DNS provider.

a. In the Address field, enter the provider address.
b. In the Preferred field, click the check box if you want to have this address as
the preferred provider.
Note: You can have only one preferred provider.
c. Click Update.
5 Repeat step 4 for each additional DNS provider.
6 Click + next to DNS Domains to add a DNS domain.

a. In the Name field, enter the domain name, such as "cisco.com".
b. In the Default field , click the check box to make this domain the default
domain.
Note: You can have only one domain name as the default.
c. Click Update.
7 Repeat step 6 for each additional DNS Domain suffix.
8 Click Submit.
Verifying DNS Operation
1 Check the resolv.conf file from the APIC CLI:
admin@apic1:~> cat /etc/resolv.conf

# Generated by IFC
search cisco.com
nameserver 171.70.168.183
nameserver 173.36.131.10
36 Management
2 Ping a host by DNS name that will be reachable from the APIC out-of-band
management.
admin@apic1:~> ping cisco.com

PING cisco.com (72.163.4.161) 56(84) bytes of data.
64 bytes from www1.cisco.com (72.163.4.161): icmp_seq=1 ttl=241 time=42.7 ms
^C
--- cisco.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2102ms
rtt min/avg/max/mdev = 42.485/43.038/43.903/0.619 ms
admin@apic1:~>
Management 37
Role-Based Access Control
Role-based access con

trol (RBAC) and tenancy are impor
tant con cepts to mas
ter when
op
er
at
ing a Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) fab
ric. Role-based ac
cess and
the con
cept of ten
ancy are two core foun
da
tions upon which the ACI man
age
ment and
pol
icy mod
els are built.
In an ACI fabric, the Cisco Ap

pli
ca
tion Pol
icy In
fra
structure Controller (APIC) grants ac
-
cess to all ob
jects based on a user's es
tablished role in the con
fig
ura
tion. Each user can
be assigned to a set of roles, a priv
il
ege type, and a se cu
rity domain. Think of this in
terms of Who, What, and Where. The role is the Who (log ic
al col
lec
tion of priv
il
eges),
priv
il
eges define What func tions a user can per form, and the se curity do
main de -
fines Where the user can per form these ac tions. When com bin
ing these objects, you
can im ple
ment very granul
ar access control in the sys
tem.
The role clas

si
fi
ca
tion is an estab
lished group ing of per
mis sions. For ex am ric-
ple, a fab
ad
min can have ac cess to the en tire fab
ric as well as to assigning other user roles and
as
so
ci
ate them to se curity domains, whereas a ten ant-ad min can only have ac cess to
the components within the ten ant to which they're as soci
ated, and is un able to man -
age other user roles at a fabric wide level. ACME's IT or ga
nization fits per
fectly into this
ac
cess control model.
The APIC provides access ac

cord
ing to a users role through RBAC. An ACI fab
ric user is
as
so
ci
ated with the fol
lowing:
A set of roles
For each role, privileges and privilege types (which can be "no access", "read-
only", or "read-write")
One or more security domain tags that identify the portions of the management
information tree (MIT) that a user can access
38 Management
Authenticated user creation lifecycle
The ACI fab ric manages access priv

ileges at the man aged ob ject (MO) level. A priv
il
ege
is an MO that en ables or re
stricts access to a partic
ul
ar function within the system. For
example, fabric-equipment is a privil
ege flag. This flag is set by the APIC on all ob jects
that correspond to equip ment in the phys ic
al fab
ric.
APIC poli
cies man age the ac
cess, au
thenti
cation, and ac count ing (AAA) func tions of the
Cisco ACI fabric. The combi
na
tion of user priv il
eges, roles, and se curity domains with
ac
cess rights inherit
ance en
ables admin is
trators to con figure AAA func tions at the
managed object level in a very granul
ar fash ion. These con fig
ura
tions can be im ple
-
mented using the REST API, CLI, or GUI.
Multiple Tenant Support

A core APIC in
ter
nal data ac
cess con
trol sys
tem pro
vides multi-ten
ant iso
la
tion and
prevents in
for
ma
tion pri
vacy from being compro
mised across tenants. Read/write re -
stric
tions pre
vent any tenant from see
ing any other ten
ants con
fig
ur
a
tion, sta
tis
tics,
Management 39
faults, or event data. Un

less the ad
min
is
tra
tor as
signs per
mis
sions to do so, ten
ants are
re
stricted from read
ing fab
ric con
fig
ur
a
tion, poli
cies, sta
tis
tics, faults, or events.
If a vir
tual ma
chine man
age
ment (VMM) do
main is tagged as a se
cu
rity do
main, the
users con
tained in the se
cu
rity do
main can ac
cess the cor
re
spond
ing VMM do
main. For
ex
am ant named solar is tagged with the se
ple, if a ten cu main called sun and a
rity do
VMM do
main is also tagged with the se
cu main called sun, then users in the solar
rity do
ten
ant can ac
cess the VMM do
main ac
cord
ing to their ac
cess rights.
User Roles
You can view the built-in user roles as well as cre
ate cus
tom roles to meet spe
cific re
-
quire
ments.
1 On the menu bar, choose Admin > AAA.
2 In the Navigation pane, choose Security Management > Roles.

Note: From here, you can see each built-in role and the associated privileges.
Additionally, you can create a custom role from the Actions menu.
Security Domains
The se cu
rity do main concept is crucial to proper operat
ion of the ACI fab ric. By using
se
curity domains, users can be or ganized into vario
us per mission struc tures, most
commonly ap plied to ten
ants. Using the ten ancy ca
pabil
it
ies of the ACI fab ric in con-
junc
tion with prop erly con
fig
ured se curity domains, it will be pos si
ble to com pletely
sep
arate con fig
ur
a
tion of appli
ca
tion work loads while only pro viding access to those
who need it.
A key concept to keep in mind when con fig

ur
ing secu
rity domains is that the config
u-
ra
tion only ap
plies at a ten
ant level. The poli
cies cannot currently be ap
plied to in
di
vid-
ual ob
jects de
spite ref
erences at the ob
ject level in
side of the RBAC screen in the APIC.
Cisco rec ommends that you con fig

ure security do
mains and access levels prior to de-
ployment of ten ants. Cisco does not rec om mend that you pro vide user access con fig
u-
ra
tion by mod if
ing permis sions of the "all" domain. Changes in the "all" domain will af-
fect access permissions for all users. If you need to make se lec
tive changes to allow ac -
cess out side of a user's security domain, be sure to set up a dis
crete user ac cess policy
just for that com muni
cation.
40 Management
Creation of a Security Domain

There are three se
cu
rity do
mains that are pro
vi
sioned by de
fault on the ACI fab
ric: "all",
"common", and "mgmt". The "all" se
cu
rity do
main usu
ally in
cludes access to every
thing
within the man
age
ment in
for
ma tion tree (MIT). "Com
mon" is usually used when there is
a need for shared re
sources be
tween ten
ants, such as DNS or di
rec
tory ser
vices. The
"mgmt" se
cu
rity do
main is for man
age
ment traf
fic poli
cies. You can add se
cu
rity do
-
mains as nec
es
sary. For ex
am
ple, for a multi-ten
ant en
vi
ron
ment, a se
cu
rity do
main
would be cre
ated for each ten
ant. Users are then cre
ated and as
signed to cer
tain se
cu
-
rity do
mains, or ten
ants. RBAC poli
cies are con
fig
ured under the "Admin" tab of the
APIC GUI.
For more in
for
ma
tion about the MIT, see the doc
um
ent at the fol
low
ing URL:
http://www.cisco.
com/c/en/
us/products/
collateral/
cloud-systems-management/
aci-fabric-controller/
white-paper-c11-729586.
html
To cre
ate a se
cu
rity do
main:
2 In the Navigation pane, choose Security Management > Security Management.
3 In the Work pane, choose Actions > Create a Security Domain.
4 In the Create a Security Domain dialog box, perform the following actions:
a. Give the security domain a name and an optional description.
Adding Users
You might need to add new users within the ACI fab ric. ACI can have local users man u-
ally en
tered by an admin, or use some thing such as LDAP, RA DIUS, Active Di
rec
tory, or
TACACS+ to spec ify users that will be al
lowed to man age cer
tain parts of the ACI net-
work.
Con
fig
ure Local Users:
2 In the Navigation pane, choose AAA Authentication.
3 In the Work pane, in the Realm drop-down list, choose Local and click
Submit if Local is not already chosen.
Management 41
4 In the Navigation pane, choose Security Management.
5 In the Work pane, choose Actions > Create Local User.
6 In the Create Local User dialog box, perform the following actions:
a. Specify any security information that is necessary and click Next.
b. Select the roles to be given to this user, such as Read/Write for admin or
tenant admin, and click Next.
c. Specify login information for the user and
7 Click Finish.
Remote Authentication
Creating remote user accounts in the ACI is sim
il
ar to most other data cen ter systems.
If creat
ing an LDAP ac count, then the LDAP provider must be con fig
ured first. The IP
address or host name of the LDAP server is needed, along with the port it uses to com -
mu ni
cate as well as any other rel e
vant in
for
mation that will allow con nection to that
server. The same is true for RA DIUS and TACACS+. The next op tion is to con fig
ure
groups from which it is al lowed to read data and grab it for the pur poses of se lect
ing
remote users granted ac cess to the ACI network. Re mote authenti
cation is covered in
de da
tail in the Cisco ACI Fun mentals Guide.
Management 43
Import and Export Policies
All of the stake

holders at ACME involved in the de
ploy
ment and ad minis
tra
tion of the
new mo bile ap
pli
ca
tion plat
form need to know that they will be able to easily re
cover
from any loss of con
fig
ur
a
tion quickly and eas
ily. Since all APIC poli
cies and con
fig
ur
a
-
tion data can be ex
ported to create backups and tech sup port files for Dis
as
ter Re
cov
-
ery, trou
bleshoot
ing, and of
fline analy
sis, ACI is a good choice on this front as well. As
with all things APIC, a pol
icy needs to be con
fig
ured to ex
port or backup the con
fig
ur
a
-
tion poli
cies. This can be done from any ac
tive and fully fit APIC within the ACI fab
ric.
The backup and re
store process does not re
quire backup of in
di
vid
ual com
po
nents.
Backups are con

fig
urable through an ex
port policy that al
lows ei
ther sched uled or im-
medi
ate backups to a remote server (preferred) or, in the case where an ex ternal
SCP/FTP server is not avail
able, back
ups to be writ
ten to the local APIC file sys
tem.
Backup/ex port policies can be con fig

ured to be run on-de mand or based on a re cur
-
ring schedule. Cisco rec ommends that a cur rent Backup be per formed before making
any major sys tem con fig
ur
a
tion changes or ap ply
ing software up dates. Con
fig
ur
a
tion
Import poli
cies (pol
icy restore) will be dis
cussed later in this sec
tion.
By default, all poli

cies and tenants are backed up, but the adminis
tra
tor can optionally
spec
ify only a spe cific sub
tree of the management in
for
ma tion tree. This is use ful in
multi-tenant de ployments where in di
vid
ual ten
ants wish to backup or re store their re-
spec
tive policies.
Another Ex port policy re

quired on oc ca
sion is for gath er
ing techni
cal sup
port infor
ma -
tion. If is
sues are en coun tered within the sys tem, you might need to con tact the Cisco
Technical Assis
tance Cen ter (TAC). Pro vid
ing a tech ni
cal support bundle will give Cisco
support en gi
neers the informa tion needed to help iden tify and suggest re
me di
at
ion to
is
sues. Tech ni
cal sup port export poli cies can be con figured to run on-de mand or
scheduled for re cur
ring purposes. Tech ni
cal support bun dles are config
ured very sim i-
lar to con fig
ura
tion export poli cies, and so the bun dles will not be cov ered in de tail
here. For de tails on tech
nical sup port export policies, see the Cisco APIC Trou bleshoot-
ing Guide that is ref er
enced in the Ap pendix of this book.

44 Management

Since ACI is pol icy dri
ven, more than a backup job is re quired. First, a remote loca
tion
is created (the ex ter
nal server to which you want to ex port in
for
ma tion), then an Ex -
port pol icy task, and option
ally a Scheduler (for re
cur
ring tasks). The pro cedure below
details how to cre ate a re
mote lo ca
tion and export pol
icy in ACI to transfer the config
-
u
ra
tion file bun
dle to an SCP server. The in di
vid
ual XML files can be extracted and
viewed after the config
ur
a
tion bun
dle is trans
ferred to the desk
top. An ex
trac
tion util
-
ity is needed to de
com
press the tar.
gz file that is cre
ated.
Configuration Export Policy
Add a Remote Location (SCP)

1 On the menu bar, choose Admin > Import/Export.
2 In the Navigation pane, choose Remote Locations.
3 In the Work pane, choose Actions > Create Remote Location
4 In the Create Remote Location dialog box, perform the following actions:
a. Enter a Remote location name
b. Enter a Hostname/IP address
c. Choose a Protocol
d. Enter a Remote path
e. Enter a Remote port
f. Enter a Username
g. Enter a Password
h. Choose a Management EPG
Note: default (Out-of-Band)
5 Click Submit.
Management 45
Create a One Time Export Policy

The pro
ce
dure below de
tails a con
fig
ur
a
tion ex
port pol
icy, but the pro
ce
dure for a
tech
ni
cal sup
port ex
port pol
icy is very sim
il
ar.
2 In the Navigation pane, choose Export Policies > Configuration.
3 In the Work pane, choose Actions > Create Configuration Export Policy
4 In the Create Configuration Export Policy dialog box, perform the following
actions:
a. Name = Export_Policy_Name
b. Format = XML
c. Start Now = Yes
d. Export Destination = Choose_the_Remote_location_created_above
5 Click Submit.
Two optional config

ur
a
tions are ap
ply
ing a Scheduler pol
icy if you want to setup a re
-
cur
ring op
er
ation, and specif
y
ing a spe
cific Dis
tin
guished Name (DN) if you want to
backup only a subset of the Manage
ment Infor
mation Tree (MIT).
Verify Export Policy was Successful

2 In the Navigation pane, choose Export Policies > Configuration

> Export_Name.
3 In the Work pane, choose the Operational tab.

a. The State should change from "pending" to "success" when the export
completes correctly.
b. (Optional) Confirm on the SCP server that the backup filename exists.
46 Management
Extract and View Configuration Files

A con
fig
ur
a
tion ex
port task ex
ports a com
pressed bun
dle of in
di
vid
ual XML files. These
XML con
fig
ur
a
tion files can be ex
tracted and viewed on an
other work
sta
tion.
1 From the workstation where the exported bundle has been copied, select the
archive utility of choice.
2 Navigate to the folder where the config export is located (tar.gz), highlight the
file, and then select Extract.
3 Double-click one of the XML files to view the contents in a browser.
4 Examine the various XML configuration files for parameters that have been
configured.
Configuration Import (Restore/Merge)

To re
store a con
fig
ur
a
tion from a pre
vi
ous backup:
If the re
mote loca
tion does not exist, cre
ate a re
mote lo
ca
tion per the "Adding a Re
-
mote Lo ca
tion (SCP)" sec
tion.
2 In the Navigation pane, choose Import Policies > Configuration.
3 In the Work pane, choose Actions > Create Import Configuration Policy.
4 In the Create Import Configuration Policy dialog box, perform the following
actions:
a. Enter a name and the filename for the import policy, such as "backup-
file.tar.gz". Do not include the file path.
b. Choose the import type:
Merge - Will merge backup configuration with existing running
configurations. Will not overrite any existing policies.
Replace - Will replace all configurations with those only from the backup file.
Management 47
c. The import mode must be specified when you attempt to perform

a Merge import. The configuration data is imported per shard with each
shard holding a certain part of the system configuration objects. The default
is best-effort. The Replace Import Mode can be either:
best-effort - Each shard is imported, skipping any invalid objects
atomic - Attempts to import each shard, skipping those which contain an

invalid configuration. A shard's entire configuration must be valid to be
imported in this mode.
d. Choose Start Now.
e. Choose the Remote Location that was previously created in the "Adding a
Remote Location (SCP)" section.
49
Upgrading and
Downgrading Firmware
Upgrading and Downgrading Firmware 51
Section Content
Firmware Management
Firmware Versions
Firmware Components
Firmware Policies
Firmware Groups
Maintenance Groups
Controller Firmware
Catalogue Firmware
Upgrading and Downgrading Considerations
Upgrading the Fabric

Downloading the Firmware Images
Upgrading the APIC Controller Software
Upgrading the Switch Software Using the GUI
Upgrading the APIC Controller Software Using the CLI
Upgrading the Switch Software Using the CLI
Verifying Cluster Convergence
Troubleshooting Failures During the Upgrade Process
Firmware Management
ACME Inc., in partner ship with Cisco, has eval ua

ted the re quirements for their deploy
-
ment based on the soft ware features re quired, the sup port for the hardware platforms
they have selected, and the ma turity of the software releases. They have se lected a tar
-
get version of soft
ware for their de ployment. Ad di
tionally, they have put a proac tive
plan in place to re
visit this de
ci
sion pe ri
od
ic
ally to deter
mine if future up
grades are re-
quired.
Firmware Versions
The soft
ware ver
sions for ACI are listed in the fol
low
ing for
mat:
major.minor(mntnc)
major - Represents major changes in the product architecture, platform, or

features content.
minor - Represents a minor release with new software features.
mntnc - Represents bug fixes to a feature release of APIC. This changes when
there are fixes for product defects in the software, but no additional new
features.
Ex
am
ple:
APIC version: 1.1(1d)
Both the soft ware for the APIC and the fab
ric nodes are de
noted by the same ver
sion
-
ing scheme, For ex ample, APIC release 1.0(2j) corre
sponds to switch software
11.0(2j). Re
lease notes for the APIC ver
sions ref
er
ence the corre
spond
ing switch ver
-
sions and vice-versa.
All com
po
nents of the ACI in
fra
struc
ture in
clud
ing the Cisco Ap
pli
ca
tion Pol
icy In
fra
-
struc
ture Con
troller (APIC), leaf switches, and spine switches, should be on the same
54 Upgrading and Downgrading Firmware
ver
sion. While at the time of up
grad
ing, dis
parate ver
sions may exist be
tween APIC and
the switches, do not op
er
ate the fab
ric for ex
tended pe
ri
ods of time in this state.
When con
sid
er
ing the im
pact and risk of up
grad
ing, you can as
sume that a main
te
-
nance ver
sion upgrade, such as 1.1.(1d) => 1.1(1d), will have less im
pact than a
major/minor ver
sion upgrade, as there will be only bug fixes and no new features
added.
Firmware Components
There are three main com
po
nents that can be up
graded:
Switches (leaf and spine)

Application Policy Infrastructure Controller (APIC)
Catalog firmware
Firmware Policies
Firmware Groups
Firmware Group poli cies on the APIC de

fine which group of nodes on which firmware
will be up
graded. For most deploy
ments, a sin
gle firmware group is ad
e
quate. Con
trol
over which switches up grade to the new version can be deter
mined through mainte
-
nance groups.
Maintenance Groups
Mainte
nance Group policies define a group of switches that will be jointly upgraded to
the as
so
ci
ated firmware set. Main tenance groups can be up graded on de mand or ac-
cord
ing to a sched
ule, making it possi
ble to defer an up
grade task to a busi
ness main
te
-
nance window.
Controller Firmware
The APIC controller firmware pol
icy always applies to all con
trollers in the clus
ter, but
the up
grade is al
ways done sequentially. The APIC GUI pro vides real-time status in
for
-
ma
tion about firmware upgrades. Con troller Firmware poli
cies can be up
graded on de
-
mand or also ac
cord
ing to a sched
ule.
Catalogue Firmware
Each firmware image in
cludes a com
pat
ib
il
ity cat
al
og that iden
ti
fies sup
ported switch
mod els. The APIC maintains a cat
al
og of the firmware im ages, switch types, and mod els
that are al
lowed to use that firmware image. The APIC, which per forms image man age-
ment, has an image repos it
ory for compatib
il
ity cat
al
ogs, APIC controller firmware im-
ages, and switch im
ages.
Firmware upgrade policy relationships

Upgrading and Downgrading Considerations
Be
fore start
ing the up
grade or down
grade process, con
sider the fol
low
ing things:
Before starting the upgrade process, your controllers should be in good health.
On the menu bar, choose System > Controllers. In the navigation pane,
choose Controllers > apic_controller_name > Cluster. Verify that the health
state of all of the controllers in the cluster are Fully Fit before you proceed. To
resolve issues for controllers that are not fully fit see the Troubleshooting Cisco
Application Centric Infrastructure document:https://datacenter.github.io/aci-
troubleshooting-book/
Configuration backup. Before starting any upgrade, always export your
configuration to an external source. For information about exporting
configurations, see the Import and Export Policies chapter.
Permissions. A user must have the fabric administrator role to perform
firmware upgrade tasks.
Verify free space. On the menu bar, choose System > Controllers. In the
navigation pane, choose Controllers > apic_controller_name > Storage.
Confirm that the /firmware partition is not filled beyond 75%. If the partition
is filled beyond 75%, you might be required to remove some unused firmware
files from the repository to accommodate the compressed image as well as
provide adequate space to extract the image. The APIC automatically extracts
the image.
Upgrade order. Typically, the controllers should be upgraded first, followed by
the switch nodes. Always refer to the relevant release notes of the destination
firmware version for any changes to this order.
Maintenance windows. Although it is possible to upgrade the fabric without
impacting the dataplane, you should perform an upgrade during a scheduled
maintenance window according to your change control policy. This window
should account for any unforseen issues that might arise during the
upgrade, and allocate enough time to troubleshoot or perform a rollback.
Maintenance groups. To help minimize the impact to hosts during an upgrade,
you should set up at least two separate maintenance groups. A common
separation is by odd and even node IDs. Assuming that your hosts are dual-
connected to at least one odd and one even leaf node, there should not be any
impact to your hosts. Maintenance group creation is covered in detail later in

the chapter. Another consideration is that your leaf VPC pairs should contain
one odd and one even node.
Upgrading a fabric with the Application Virtual Switch (AVS) deployed. The AVS
software is not specifically tied to the APIC or switch software version.
Device packages. Device packages are not always tied to the APIC software.
You can confirm the device compatability for Layer 4 to Layer 7 devices using
the online Application Centric Infrastructure (ACI) Compatability
tool: http://www.cisco.com/web/techdoc/aci/acimatrix/matrix.html
Upgrading the Fabric
Downloading the Firmware Images

You must down
load both the con
troller soft
ware pack
age and switch soft
ware pack
-
age for the Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller (APIC) from Cisco.
com.
To down
load the firmware im
ages using the APIC GUI:
1 On the menu bar, choose Admin > Firmware.
2 In the Navigation pane, choose Fabric Node Firmware.

Note: In the Work pane, the list of all switches in the fabric and the status of
when the firmware was last upgraded are displayed.
3 In the Navigation pane, choose Download Tasks.
4 In the Work pane, choose Actions > Create Outside Firmware Source.
5 In the Create Outside Firmware Source dialog box, perform the following
actions:
a. In the Source Name field, enter a name for the switch image, such as
"apic_1.1.3d".
b. For the Protocol radio buttons, click the Secure copy or HTTP radio button.
c. In the URL field, enter the URL from where the image must be downloaded.
HTTP Example: http://192.168.0.50/aci-apic-dk9.1.1.3d.iso
SCP Example: 192.168.0.50:/tmp/aci-firmware/aci-apic-
dk9.1.1.3d.iso
For SCP, enter your username and password.
d. Click Submit.
6 In the Navigation pane, choose Download Tasks.
7 In the Work pane, choose the Operational tab to view the download status of
the images.
8 Repeat the steps for the switch image.
9 Once the download reaches 100%, in the Navigation pane, choose Firmware
Repository.
10 In the Work pane, the downloaded version numbers and image sizes are
displayed.
To down
load the firmware im
ages using the APIC CLI:
1 SSH to an APIC and log in as "admin".
2 Pull the software using SCP:
admin@apic1:~> scp
username@IP_address_with_the_image:/absolute_path_to_image_plus_image_filename &#10!
3 Place the image into the image repository:
admin@apic1:~> firmware add ver_no.iso &#10!
The firmware image ver_no.iso is added to the repos

it
ory.
4 Verify the software has been added to the respository:
admin@ifc1:~> firmware list

Name : aci-apic-dk9.1.1.1d.bin
Type : controller
Version : 1.1(1d)
Upgrading the APIC Controller Software

The cat
al
og firmware image is up
graded when an APIC con
troller image is up
graded.
You do not need to up
grade the cat
al
og firmware image sep
ar
ately.
To up
grade the APIC con
troller soft
ware using the GUI:
2 In the Navigation pane, click Controller Firmware.
3 In the Work pane, choose Actions > Upgrade Controller Firmware Policy.
4 In the Upgrade Controller Firmware Policy dialog box, perform the following
actions:
a. In the Target Firmware Version field, from the drop-down list, choose the
image version to which you want to upgrade.
b. In the Apply Policy field, click the Apply now radio button. Alternately, you
can apply a schedule policy if you wish to defer the task to a specific
date/time.
c. Click Submit to complete the task.
The Sta
tus di
al
og box dis
plays the Changes Saved Suc
cess
fully mes
sage, and the
up
grade process be
gins. The APIC con
trollers are up
graded se
ri
ally so that the
con
troller clus
ter is avail
able dur
ing the up
grade.
5 Verify the status of the upgrade in the Work pane by clicking Controller
Firmware in the Navigation pane.
The con trollers upgrade in random order. Each APIC con troller takes about 10
minutes to up grade. Once a con troller image is up graded, it drops from the
cluster, and it reboots with the newer ver sion while the other APIC con trollers
in the cluster are still op
er
at
ional. Once the con troller re
boots, it joins the clus -
ter again. Then the clus ter converges, and the next con troller image starts to
upgrade. If the clus ter does not im me di
ately converge, and is not fully fit, the
upgrade will wait until the cluster con verges and is fully fit. During this period, a
Wait ing for Cluster Con ver
gence mes sage is dis
played in the Sta tus column for
each APIC as it up grades.
When the APIC con troller that the browser is con

nected to is up
graded and it
re
boots, the browser dis
plays an error mes
sage.
6 In the browser URL field, enter the URL for the APIC controller that has already
been upgraded, and sign in to the APIC controller as prompted.
Upgrading the Switch Software Using the GUI

Before you upgrade the switches, the APICs must have com
pleted up
grad
ing and have
a health state of Fully Fit.
To up
grade the switch soft
ware using the GUI:

2 In the Navigation pane, choose Fabric Node Firmware.

Note: In the Work pane, the switches that are operating in the fabric are
displayed.
3 If you have not created a firmware group, perform the following substeps:
a. In the Work pane, choose the Policy tab.
b. Choose Actions > Create Firmware Group.
c. In the Create Firmware Group dialog box, perform the following actions:
i. In the Group Name field, enter the name of the firmware group.
ii. In the Target Firmware Version drop-down list, choose the
firmware version to which you will upgrade.
iii. In the Group Node IDs field, enter a comma-separated list or a range of
node IDs to include in the group. For example, "101, 103-105, 108".
iv. Click Submit.
d. To verify that the firmware group was created, in the Navigation pane,
choose Fabric Node Firmware > Firmware Groups >
new_firmware_group. The Work pane displays details about the firmware
policy that was created earlier.
4 If you have not created maintenance groups, perform the following substeps:
a. In the Navigation pane, click Maintenance Groups.
Note: Cisco recommends that you create two maintenance groups for all of
the switches. For example, create one group with the even-numbered
devices and the other group with the odd-numbered devices.
b. In the Work pane, choose Action > Create Maintenance Group.
c. In the Create Maintenance Group dialog box, perform the following actions:
i. In the Group Name field, enter the name of the maintenance group. For
example, "Even-Nodes".
ii. For the Run Mode radio buttons, click the Pause Upon Upgrade
Failure radio button, which is the default mode.
iii. In the Group Node IDs field, enter a comma-separated list or a range of
node IDs to include in the group. For example, "102, 104, 106, 108, 110".
iv. In the Scheduler drop-down list, you can choose to create a schedule
for upgrading or leave the drop-down list blank so that you can upgrade on
demand.
v. Click Submit.
vi. To verify that the maintenance group was created, in the Navigation pane,
choose Fabric Node Firmware > Maintenance Groups
> new_maintenance_group, and click the name of the maintenance
group that you created.
Note: The Work pane displays details about the maintenance policy.
vii. Repeat this step for the second maintenance group. For example, a group
named "Odd-Nodes".
5 Right-click one of the maintenance groups that you created and

choose Upgrade Now.
6 In the Upgrade Now dialog box, for Do you want to upgrade the maintenance
group policy now?, click Yes.
7 Click OK.
Note: In the Work pane, the Status displays that all the switches in the group
are being upgraded simultaneously. The default concurrency in a group is set at
20. Therefore, up to 20 switches at a time will get upgraded, and then the next
set of 20 switches are upgraded. In case of any failures, the scheduler pauses
and manual intervention is required by the APIC administrator. The switch
upgrade takes up to 12 minutes for each group. The switches will reboot when
they upgrade, connectivity drops, and the controllers in the cluster will not
communicate for some time with the switches in the group. Once the switches
rejoin the cluster after rebooting, you will see all the switches listed under the
controller node. If there are any VPC configurations in the cluster, the upgrade
process will upgrade only one switch at a time out of the two switches in a VPC
domain.
8 In the Navigation pane, click Fabric Node Firmware.

Note: In the Work pane, view all of the switches that are listed. In the Current
Firmware column, view the upgrade image details listed against each switch.
Verify that the switches in the fabric are upgraded to the new image.
Upgrading the APIC Controller Software Using the CLI

The cat
alog firmware image is up graded when an APIC con troller image is upgraded.
You do not need to upgrade the cat al
og firmware image separately. Cisco rec
ommends
that you to perform the firmware up grade from the GUI. When you use the GUI, the
APIC performs addi
tional ver
if
i
ca
tion and in
tegrity checks on the software image.
To up
grade the APIC con
troller soft
ware using the CLI:
1 List the current software in the respository that was previously downloaded.
Ex
am
ple:
admin@apic1:~> firmware list

Name : aci-apic-dk9.1.1.1d.bin
Type : controller
Version : 1.1(1d)
2 Upgrade the firmware on the controllers.
Ex
am
ple:
admin@apic1:~> firmware upgrade controllers ver_no.bin
The APIC con trollers are up

graded se
ri
ally so that the con
troller clus
ter is avail
-
able dur
ing the upgrade. The upgrade occurs in the background.
3 Check the status of the upgrade.

Ex
am
ple:
admin@apic1:~> firmware upgrade status
Node-Id Role Current- Target- Upgrade- Progress-Percent
Firmware Firmware Status (if inprogress)
--------- ----------- ------------ ------------------ ---------- ------------------
1 controller 1.0(1.200) apic-1.0(1.202a) inqueue 0
2 controller 1.0(1.200) apic-1.0(1.202a) inqueue 0
3 controller 1.0(1.200) apic-1.0(1.202a) inprogress 0
The Up
grade-Sta
tus field will show "in
queue", "in
progress", or "com
ple
teok". If you
see "un
known" in this field, that con
troller has prob
a
bly up
graded and is re
boot
ing.
When the APIC con

troller to which you have con
nected com
pletes up
grad
ing
and re
boots, you can close the SSH win
dow.


Be
fore you up grade the switches, the APICs must have com
pleted up
grad
ing and have a
health state of Fully Fit.
To up
grade the switch soft
ware using the CLI
1 Check that the output of the following command appears like the output shown
below, with the correct version number:
Ex
am
ple:
admin@apic1:~> firmware list
Name : aci-n9000-dk9.11.1.1d.bin
Type : switch
Version : 11.1(1d)
The name changes from ".iso" to ".bin".
2 Upgrade the switches.
Ex
am
ple:
admin@apic1:~> firmware upgrade switch node 101 ver_no.bin
Firmware Installation on Switch Scheduled
You must up
grade each switch sep
ar
ately.
3 Check the upgrade status for the switch. The output that appears from the
following command will appear like the following sample:
Ex
am
ple:
admin@apic1:~> firmware upgrade status node node_id
Node-Id Role Current- Target- Upgrade- Progress-Percent
Firmware Firmware Status (if inprogress)
--------- ----------- ------------------- ------------------ ---------- ------------------
1017 leaf n9000-11.0(1.869S1) n9000-11.1(1d) completeok 100

You can check the sta

tus of all nodes at once, by en
ter
ing the firmware up
grade
sta
tus com
mand.
4 Repeat Steps 2 and 3 for each additional switch.
Verifying Cluster Convergence

You can mon
it
or the progress of the clus
ter con
ver
gence after a sched
uled main
te
-
nance. You view the Con
troller Firmware screen on the GUI, which pre
sents you with
a se
ries of mes
sages during the process of one clus ter con
verg
ing and then the next
clus
ter. These mes played in the Sta
sages are dis tus field.
As the con
troller and switches move through the up
grade, you will see mes
sages about
the num
ber of nodes queued and the num
ber in the process of up
grad
ing, as well as
how many have up
graded suc
cess
fully.
The fol
low
ing are the pos
si
ble up
grade states for a node:
NotScheduled: No upgrade is currently scheduled for this node.

Scheduled: Upgrade is scheduled for this node.
Queued: There is a currently active window (schedule) and the node is
requesting permission to upgrade.
Inprogress: Upgrade is currently in progress on this node.
CompleteOK: Upgrade completed successfully.
CompleteNOK: Upgrade failed on this node.
Inretryqueue: Node is queued again for upgrade retry (5 attempts are made
before declaring failure).
This may take a while. When all the clus

ters have con verged successfully, you will see
"No" in the Wait
ing for Clus
ter Con
ver
gence field of the Controller Firmware screen.
Troubleshooting Failures During the Upgrade Process

There is one sched
uler per main
te
nance pol
icy. By de
fault, when an up
grade fail
ure is
de
tected, the sched
uler pauses, and no more nodes in that group begin to upgrade. The
scheduler ex
pects manual in
ter
ven
tion to debug any up
grade fail
ures. Once manual in
-
ter
ven
tion is com
plete, you must resume the paused scheduler.
If you no
tice that switches are in the queued state, then check the fol
low
ing:
Is the controller cluster healthy? The controller cluster must be healthy. If you
see waitingForClusterHealth = yes in the API or "Waiting for Cluster
Convergence" showing "Yes" in the GUI, that means the controller cluster is not
healthy. Until the controller cluster is healthy, switches which have not already
started their upgrade will be in queued state.
Is the switch maintenance group paused? The group will be paused if any
switch fails its upgrade.
If the sys
tem takes longer than about 60 min
utes for a switch to dis
play wait
ing
For
-
Clus
ter
Health = no in the API or "Wait
ing for Clus
ter Con
ver
gence" show
ing "No" in
the GUI, you should work through the steps for ver
if
y
ing a pause in the sched
uler.
For additional trou

bleshoot
ing pro
ce
dures, see the Trou
bleshoot
ing Cisco Ap
pli
ca
tion
Centric Infra
struc
ture doc
um
ent at the fol
low
ing URL:
https://
datacenter.
github.
io/
aci-troubleshooting-book/
69
Fabric Connectivity
Fabric Connectivity 71
Section Content
Understanding Fabric Policies
Fabric - Access Policies

Domains
VLAN Pools
AEPs
Policy Types
Interface Policies
Switch Policies
Interface Policy Groups
Switch Policy Groups
Interface Profiles
Switch Profiles
Best Practices
Adding New Devices to the Fabric

Sample Configuration
Creating VLAN Pools
Create VLAN Pool
Create Physical Domain
Create an Attachable Access Entity Profile (AEP)
Interface Policies
Interface Profile

Switch Profiles
Reusability
Sample vPC Creation
Create VLAN Pool

Create a Physical Domain
72 Fabric Connectivity
Create Access Entity Profile

Interface Policies
Switch Profile
Create vPC domain
Validate Operation of Configured vPC
Server Connectivity
Cisco UCS B-Series Servers
Standalone Rack Mount Servers or Non-Cisco Servers
Virtual Machine Networking

Understanding VM Networking in ACI
ACI VM Integration Workflow
VMware Integration
VMM Policy Model Interaction
Publishing EPGS to a VMM Domain
Connecting VMs to the EPG Port Groups on vCenter
Verifying Virtual Endpoint Learning
Verifying VM Endpoint Learning on the APIC from the CLI
VMware Integration Use Case
Deploying the Application Virtual Switch

Prerequisites
Getting Started
Install the AVS VIB
Manual Installation
DHCP Relay
Attachable Access Entity Profile (AEP) and AVS
VMM Domains for vCenter
AVS Switching Modes
Create the VMM Domain for AVS
Verify AVS Deployment on vCenter
Add vSphere Hosts to the AVS
Verify AVS on ESX

VXLAN Load Balancing
IGMP Snooping Policy for AVS
External Connectivity
Extending ACI to External Layer 2
Extending an ACI Bridge Domain Outside of the Fabric
Extending Endpoint Groups Outside the ACI Fabric
Supported Routing Protocols
Configure MP-BGP Spine Route Reflectors
Layer 3 Integration Through Tenant Network with OSPF NSSA
External Layer 3 for Multiple Tenants
Application Migration Use Case

Extending the Network to ACI
Understanding Fabric Policies
Now that ACME has been pro vi

sioned with ACI fab ric and infrastruc
ture space has
been con
fig
ured be
tween the leaf and spine switches, access priv
il
eges, and basic man
-
age
ment poli
cies, it is time to start cre
at
ing con
nec
tiv
ity poli
cies within the ACI fab
-
ric. The fab
ric tab in the APIC GUI is used to con
fig
ure sys
tem-level fea
tures including,
but not limited to, device dis
cov
ery and in
ven
tory management, di
ag
nostic tools, con-
fig
ur
ing do
mains, and switch and port be
hav
ior. The fab
ric pane is split into three sec
-
tions: in
ven
tory, fab
ric poli
cies, and ac
cess poli
cies. Un
der
stand
ing how fab
ric and ac
-
cess poli
cies con
fig
ure the fab
ric is key for the ACME net
work teams, as they will be
maintain
ing these poli cies for the pur poses of in
ter
nal con
nec
tions be tween fab
ric leaf
nodes, con nections to ex ter
nal en ti
ties such as servers, net
work ing equipment, and
stor
age arrays. It is im
portant that other teams such as server teams un der
stand these
concepts as well, as they will be in teract
ing with them, par
tic
ul
arly in the case of their
build processes for adding ad ditional capacity.
This chapter will re

view the key objects in the ac
cess policies subsec
tion of the fabric
tab many of which are reusable; demon strate how to add and pre-pro vi
sion switches,
and walk through the steps and ob jects required when new de vices are added to the
fab
ric to ef
fec
tively op
er
ate an ACI fabric. While many poli cies are reusable, it is also
key to under
stand the impli
ca
tions of delet
ing poli
cies on the ACI fabric.
The ac cess policies subsection is split into folders separat

ing out differ
ent types of
poli
cies and ob jects that affect fab
ric be havior. For exam ple, the inter
face poli
cies
folder is where port be hav
ior is configured, like port speed, or whether or not to run
proto
cols like LACP on leaf switch in terfaces is set. Domains and AEPs are also cre ated
in the access policies view. The fabric access policies provide the fabric with the base
config
ur
ation of the ac
cess ports on the leaf switches.
Fabric - Access Policies
Domains
EPGs are considered the who in ACI; con tracts are con sid
ered the
what/when/why; AEPs can be con
sid
ered the "where" and do
mains can be thought
of as the how of the fab

ric. Dif
fer
ent do
main types are cre ated de
pending on how a
device is con
nected to the leaf switch. There are four dif
fer
ent do
main types: phys
ic
al
do
mains, ex
ter
nal bridged do
mains, ex
ter
nal routed do
mains, and VMM do
mains.
Phys
ic
al do
mains are gen
er
ally used for bare metal servers or servers where hy
per
vi
sor
in
te
gra
tion is not an op
tion.
Ex
ter
nal bridged do
mains are used for Layer 2 con
nec
tions. For ex
am
ple, an ex
ter
nal
bridged do
main could be used to con
nect an ex
ist
ing switch trunked-up to a leaf
switch.
Ex
ter
nal routed do
mains are used for Layer 3 con
nec
tions. For ex
am
ple, an ex
ter
nal
routed do
main could be used to con
nect a WAN router to the leaf switch.
Do
mains act as the glue be tween the con fig
ur
a
tion done in the fabric tab to the policy
model and EPG con fig
ur
ation found in the ten ant pane. The fab
ric operat
or cre
ates the
do
mains, and the tenant adminis
tra
tors asso
ci
ate do
mains to EPGs.
For an in-depth white

board expla
na
tion on do mains, check out the fol
low
ing video ti
-
tled "How De
vices Con
nect to the Fabric: Under
stand
ing Cisco ACI Do
mains": https://
www. youtube.
com/watch?
v=_ iQvoC9zQ_ A
VLAN Pools
VLAN pools con tain the VLANs used by the EPGs the do main will be tied to. A do
main is
as
so
ci
ated to a sin gle VLAN pool. VXLAN and mul ti
cast ad
dress pools are also config
-
urable. VLANs are in stan
ti
ated on leaf switches based on AEP con fig
ur
a
tion. For
warding
de
ci
sions are still based on contracts and the pol
icy model, not subnets and VLANs.
AEPs
At
tachable Ac cess En tity Profiles (AEPs) can be considered the "where" of the fab ric
config
ura
tion, and are used to group do mains with simil
ar re
quire
ments. AEPs are tied
to in
terface policy groups. One or more do mains are added to an AEP. By group ing do-
mains into AEPs and as so
ci
at
ing them, the fabric knows where the var io
us devices in
the domain live and the APIC can push the VLANs and pol icy where it needs to be. AEPs
are configured under global poli cies.
Policy Types
Most of the poli
cies fold
ers have sub
fold
ers. For ex
am
ple, under the in
ter
face poli
cies
folder there are fold
ers for con
fig
ur
a
tion called poli
cies, pol
icy groups, and pro
files.
Interface Policies
First, in
ter
face poli
cies are cre
ated to dic
tate in
ter
face be
hav
ior, are are later tied to
in
ter
face pol
icy groups. For ex
am
ple, there should be a pol
icy that dic
tates CDP is dis
-
abled, and a pol
icy that dictates CDP is en
abled - these can be reused as new de
vices
are connected to the leaf switches.
Switch Policies
There are also poli

cies for switches - for ex
am
ple, con
fig
ur
ing vPC do
mains, which are
called ex
plicit vPC pro
tec
tion groups in the APIC GUI. Ide
ally, poli
cies should be cre
-
ated once and reused when con nect
ing new devices to the fab
ric. Maxim
izing reusabil
-
ity of pol
icy and ob
jects makes day-to-day oper
ations ex
po
nentially faster and eas
ier to
make large-scale changes.
Inter face policy groups are tem plates to dic

tate port behavior and are as soci
ated to an
AEP. In ter
face pol icy groups use the poli cies described in the pre vi
ous para graph to
spec ify how links should be have. These are also reusable ob jects as many de vices are
likely to be con nected to ports that will require the same port con fig
ura
tion. There are
three types of in ter
face policy groups de pending on link type: in di
vid ual, Port Chan nel,
and vPC. Note that the ports on the leaf switches de fault to 10GE, and a 1GE link level
pol icy must be cre ated for de vices con nected at that speed. Just like poli cies, there
should ide ally be a set of pol icy groups cre ated once and reused as new de vices are
con nected to the fab ric. For example, there may be a pol icy that dic tates 10GE, CDP-
en abled. Note the in ter
face policy groups sim ply dictate pol icy. Policy groups do not
actu ally spec ify where the pro tocols and port be havior should be im plemented. The
"where" hap pens by as so
ci
at
ing one or more in ter
face profiles to a switch pro file, cov
-
ered in the fol lowing paragraphs.
Switch Policy Groups
Switch pol
icy groups allow lever
ag
ing of ex
ist
ing switch po
lices like Span
ning Tree and
monit
or
ing poli
cies.
Interface Profiles
In
ter
face profiles help tie the pieces to
gether. In
ter
face pro
files con
tain blocks of ports
- in
ter
face selec
tors - and are also tied to the in
ter
face pol
icy groups de scribed in the
pre
vi
ous para
graphs. Again, this is just an arbi
trary port, such as e1/1, the pro
file must
be as
so
ci
ated to a spe
cific switch pro
file (dis
cussed in the next paragraph) to con fig
ure
the ports.
Switch Profiles
Lastly, switch pro files allow the se lec

tion of one or more leaf switches and asso
ci
ate in
-
ter
face pro files to con fig
ure the ports on that spe cific node. This as
so
ci
at
ion pushes
the con fig
ura
tion to the inter
face, and cre ates a Port Channel or vPC if one has been
config
ured in the in ter
face policy.
The fol
lowing fig
ure high
lights the re
la
tion
ship be
tween the var
io
us global, switch, and
in
ter
face poli
cies:
Relationships to allow a physical interface or interfaces to be attached to an EPG
Layer 2 In
ter
face Pol
icy
In Cisco ACI ver

sion 1.1, a new con
fig
urable In
ter
face Pol
icy was added to allow a per
port-VLAN sig
nif
ic
ance.
To con
nect de
vices to the ACI fab
ric we can use un
tagged traf
fic,VLAN en
cap
su
la
-
tion or VXLAN en
cap
su
la
tion.
In tra
di
tional net
work
ing one of the lim
it
a
tions re
lated to VLAN en
cap
su
la
tion is scal
a-
bil
ity and re-us
abil
ity due to the limit of 4096 VLANs in net
work
ing de
vices.
In ACI, with the de

fault con
fig
ur
a
tion (global), EPGs can use the same VLAN en
cap
su
la
-
tion as long as EPGs are bound to sep arate switches. This al
lows tenants to re-use
VLAN en cap
su
la
tion IDs thor
ough the fabric with
out al
low
ing communi
ca
tion be
tween
ten
ants. How
ever, global con
fig
ur
a
tion as
sumes that ten
ants do not share leaf switches
and there
fore there is no VLAN over
lap
ping within the same leaf.
Per Port-VLAN limitations and considerations

When per port-VLAN is used, a port and VLAN pair (P,V) is registered
internally instead of just a VLAN encapsulation ID. This increases the
consumption of hardware resources at a per switch level.
Two EPGs belonging to a single Bridge Domain cannot share the same
encapsulation ID on a given leaf switch.
It is expected that the port will flap when the Layer 2 interface policy
changes between global and local. That is, traffic will get affected.
Best Practices
Cisco has es
tablished several best prac
tices for fab
ric con
fig
ura
tion. These are not re
-
quire
ments and might not work for all en vi
ronments or ap pli
cations, but might help
sim
plify day-to-day op er
at
ion of the ACI fab
ric.
Policies
Reuse policies whenever possible. For example, there should be policies for
LACP active/passive/off, 1GE port speed, and 10GE port speed.
When naming policies, use names that clearly describe the setting. For
example, a policy that enables LACP in mode active could be called "LACP-
Active". There are many "default" policies out of the box. However, it can be
hard to remember what all the defaults are, which is why policies should be
clearly named to avoid making a mistake when adding new devices to the
fabric.
A set of interface policy groups should be created for each type of similar
devices connected. For example, there can be a set of interface policy
groups for all VMware ESXi servers connected via 10GE vPCs, and a
different set of interface policy groups for all bare metal servers running
1GE with CDP disabled. Since interface policy groups are tied to a single
AEP, each AEP will have its own set of interface policy groups.
Create a switch profile for each leaf switch individually, and additionally,
create a switch profile for each vPC pair (if using vPC).
Domains
Build one physical domain per tenant for bare metal servers or servers
without hypervisor integration requiring similar treatment.
Build one physical domain per tenant for external connectivity.
If a VMM domain needs to be leveraged across multiple tenants, a single
VMM domain can be created and associated with all leaf ports where
VMware ESXi servers are connected.
AEPs
Multiple domains can be associated to a single AEP for simplicity's sake.
There are some cases where multiple AEPs may need to be configured to
enable the infrastructure VLAN, such as overlapping VLAN pools, or to limit
the scope of the presence of VLANs across the fabric.
Adding New Devices to the Fabric
This sec tion will demonstrate how to con figure ACI to re-use the fab ric ac
cess poli
cies,
sim
plify
ing day-to-day op er
ation of the fabric. This section will walk through set ting up
pro
files from scratch, with a focus on how to re-use these pro files across the fab
ric. As
out
lined in the pre vi
ous sec
tion, these var io
us profiles are linked to gether and have de -
penden cies. The fol
low
ing di
ag
ram re it
er
ates the ob ject re
la
tionships:
Object relationships
Whereas a tra
di
tional com
mand line inter
face on a switch gen
er
ally re
quires a port-by-
port confugu
ra
tion, ACI al
lows de
f
in
i
t
ion of ob
jects and poli
cies that can be re-used.
The re-us
abil
ity of these policies makes it pos
si
ble to repli
cate the con fig
ur
a
tion of a
switch very easily. The fol
lowing di
ag
ram depicts how this re-us abil
ity simpli
fies the
op
er
at
ion of the fab
ric over time.
Policy Re-use
In any data cen

ter, the con fig
ur
a
tion of a cou ple of switches does not re
quire many
processes or au
toma tion. As the data center size in
creases, au
toma
tion be
comes more
and more crit ic
al as it has a di
rect impact on the cost of busi ness op er
at
ions. In tradi
-
tional net
works, when changes that im pact a large set of devices need to be made, the
oper
at
or is faced with the cost of de signing processes to man age these de vices. These
can be net work man agement tools, scripts, or spe cial
ized appli
ca
tions. Lever aging the
Cisco ACI pol icy model, an op er
at
or can leverage profiles to stream line the oper
ation of
adding de vices and man ag
ing those devices. This is what is de
picted as the pol
icy re-
use in
flec
tion point in the pre
vi
ous di
ag
ram.
Sample Configuration
The fol
lowing sec
tions will walk through sample config
ur
a
tion of setting up in
di
vid
ually
connected devices, Port Channel-con nected de
vices, and vPC-con nected devices from
scratch, and will in
clude a re
view of the objects as they are con fig
ured. These are the
steps to be taken in the APIC GUI when new de vices are connected to the leaf switches
to en
sure the access ports on the leaf switches have the right switch port config
ur
a
tion,
and the verif
i
ca
tion steps to en
sure proper config
ur
a
tion. The fol
lowing steps rep
re
-
sent the use case of adding a new bare metal server connected to a leaf switch.
Be
fore get
ting into the con
fig
u
ra
tion of vPC's, which are a pop
u
lar server con
nec
tiv
ity
method
ol
ogy, it is im
por
tant to un
der
stand what vPC's are and how they are dif
fer
ent from
tra
ditional meth
ods of server con nec
tiv
ity. This section of the chap
ter at
tempts to clar
ify
at a high level what vPC's are, the ben
e
fits they provide and how vPC's in the ACI fab
ric dif
-
fer from how they are de ployed on Cisco Nexus switches run ning NX-OS software.
At a high level, vPC ex

tends link ag
gre
ga
tion to two sep
ar
ate phys
ic
al switches.
vPC Topology
In the fig
ure above, a single server is dual homed to two dif fer
ent switches for re dun
-
dancy. With out vPC's, the server will likely use an ac
tive-standby con fig
ura
tion, or a
special con
fig
ura
tion on the NIC dri
ver or the ker
nel that al
lows it to in
tel
li
gently load-
bal
ance traffic using an al
go
rithm.
By config
ur
ing ports on two differ
ent switches as the same port-chan nel and using an
in
ter-switch mes sag
ing chan nel (such as the in
ter-switch port-chan nel in the green
box on the left hand side) to cover re
dundancy scenar
ios, we pro
vide a log
ic
al topol
ogy
that greatly sim
pli
fies server pro
vi
sion
ing and man
age
ment.
This al
lows for sev
eral key ad
van
tages from a server de
ploy
ment per
spec
tive:
You can create resilient Layer 2 topologies based on link aggregation

You do not need STP
You have increased bandwidth, as all links are actively forwarding
Your server configurations are simplified since the configurations simply
appears as port-channels without the need for special software, from a driver
or kernel-tuning standpoint
vPCs can also be used to con

nect other down
stream de
vices, such as Cisco UCS fab
ric-
in
ter
con
nects, to pro
vide sim
il
ar ben
e
fits.
The fig
ure below shows a sin
gle tra
di
tional Layer 2 switch con
nected to a VPC en
-
abled Cisco switch pair.
Legacy connectivity compared to vPC
The com
po
nents of a tra
di
tional vPC do
main are il
lus
trated below:
Traditional vPC topology

As il
lus
trated above, in Cisco switching products run
ning NX-OS soft ware, vPC config
-
u
rations need to be done man ua
lly by the op
er
at
or and re
quire a pair of ded
ic
ated "in
-
ter-switch" links also called a peer-link. There is also a peer-keepalive link, typ
ic
ally on
the out-of-band management port, that is used to de
ter
mine peer live
li
ness to de
tect a
vPC peer-switch fail
ure. Mak
ing config
ura
tion changes in such scenarios with
out the
con
fig-sync fea ture enabled may lead to sce nar ios where there are mis matched vPC
pa
ra
me ters be
tween the vPC pri mary and the vPC sec ondary switches that may cause
par
tial con
nectiv
ity loss dur
ing the change it
self if a type-1 in
con
sis
tency is de
tected.
The ACI fab

ric greatly sim
pli
fies VPC con
fig
ur
a
tions.
vPC topology in ACI
The key dif

fer
ences to note here are that rel
at
ive to tra
di
tional vPC de
sign, there is no
re
quire
ment for set
ting up vPC peer-links. There are also no keepalives being sent on
the man age
ment ports. The fab
ric it
self serves as the peer-link. The rich inter
con
nec
-
tiv
ity be
tween fab
ric nodes makes it un likely that peers will have an active path be
-
tween them.
Note that at
tempt
ing to cable a leaf switch to an
other leaf switch will lead to a "wiring
mis
match" fault in the GUI and re
sult in a black
listed port that will have to be man
ua
lly
re
cov
ered.
The fol
low
ing are some other key be
hav
ioral changes to vPC as it ap
plies to the ACI
fab
ric rel
at
ive to clas
sic vPC that are im
por
tant for op
er
at
ors to un
der
stand:
Configurations are automatically synchronized to avoid an error-free

configuration by the APIC which is the central point of control for all
configurations in the ACI fabric.
In traditional vPC solution, the slave switch brings down all its vPC links if the
MCT goes down.
In the ACI fabric, it is very unlikely that all the redundant paths between vPC
peers fail at the same time. Hence if the peer switch becomes unreachable, it is
assumed to have crashed. The slave switch does not bring down vPC links.
Role election still happens, peers assume master-slave roles.
Role is used in case of vpc type-1 consistency failure. Slave switch brings down
all its vPC ports. A list of type-1 parameters used for consistency checking for a
given vPC domain specific to the ACI fabric are listed below.
Global type-1 parameters:
STP
Interface type-1 parameters:
STP: Only BPDU Guard is configurable
EthPM
Port speed
Duplex mode
Port mode
MTU
Native VLAN
PCM: Channel mode, static vs lacp
LACP: Lag ID
The fol
low
ing di
ag
rams il
lus
trate how the ACI fabric for
wards traf
fic from a vPC do
main
to a non-vPC connected host in the fab
ric, and vice-versa.
vPC forwarding
Uni
cast packet flow H2 -> H1
1 H2 sends a pkt towards H1 on its link to S3.
2 S3 does a table lookup and routes with vPC Virtual IP (VIP).
3 Spine switch sees multiple routes for VIP and picks one of them (S2 in this case).
4 S2 delivers the pkt to locally attached host H1.
H1 -> H2
1 H1 sends a pkt towards H2 on one of its PC link (S1 in this case).
2 S1 does a table lookup and routes with IP of S3.
3 Spine switch routes to S3.
4 S3 delivers the pkt to locally attached host H2.
Creating VLAN Pools

In this ex
am
ple, con
fig
ur
ing newly-con
nected bare metal servers first re
quires cre
ation
of a phys
ic
al do
main and then as
so
ci
at
ion of the do
main to a VLAN pool. As men tioned
in the pre
vi
ous sec
tion, VLAN pools de
fine a range of VLAN IDs that will be used by the
EPGs.
The servers are con

nected to two dif
fer
ent leaf nodes in the fab
ric. Each server will be
tag
ging using 802.1Q or VXLAN en capsu
la
tion. The range of VLANs used in the con fig
u
ra
-
tion ex
ample is 100-199. As de
picted in the fol
low
ing fig
ure, the ACI fab
ric can also act as
a gateway between disparate en
capsu
la
tion types such as un tagged traf fic, 802.1Q VLAN
tags, VXLAN VNIDs, and NVGRE tags. The leaf switches nor malize the traffic by strip
ping
off tags and reap
ply
ing the re
quired tags on fabric egress. In ACI, it is im
portant to under
-
stand that the de
f
i
n
i
tion of VLANs as they per tain to the leaf switch ports is uti lized only
for iden ti
fi
ca
tion pur
poses. When a packet ar rives ingress to a leaf switch in the fabric,
ACI has to know be fore
hand how to clas
sify packets into the differ
ent EPGs, using iden
ti
-
fiers like VLANs, VXLAN, NVGRE, phys i
cal port IDs, vir
tual port IDs.
Encapsulation normalization
Create VLAN Pool
2 In the Navigation pane, choose Pools > VLAN.
3 In the Work pane, choose Actions > Create VLAN Pool.
4 In the Create VLAN Pool dialog box, perform the following actions:
a. Define a meaningful name for the VLAN pool.
b. Optionally, provide a description for the VLAN pool.
c. There are two allocation modes: dynamic or static.
Note: When dynamic allocation is selected, the APIC selects VLANs from the
pool dynamically. This is common in VMM integration mode where the actual
VLAN ID is not important. Remember it's the EPG itself which policies are
applied to. Static allocation is typically used when the pool will be referenced
from a static source like a static path binding for an EPG for use with bare
metal servers.
d. The encap blocks are used to define the range of VLANs in the VLAN pool.
Note multiple ranges can be added to a single pool.
XML Ob
ject
<fvnsVlanInstP allocMode="static" childAction="" configIssues="" descr=""

dn="uni/infra/vlanns-[bsprint-vlan-pool]-static" lcOwn="local" modTs="2015-02-
23T15:58:33.538-08:00" monPolDn="uni/fabric/monfab-default" name="bsprint-vlan-pool"
ownerKey="" ownerTag="" status="" uid="8131">
<fvnsRtVlanNs childAction="" lcOwn="local" modTs="2015-02-25T11:35:33.365-08:00"
rn="rtinfraVlanNs-[uni/l2dom-JC-L2-Domain]" status="" tCl="l2extDomP" tDn="uni/l2dom-
JC-L2-Domain"/>
<fvnsRtVlanNs childAction="" lcOwn="local" modTs="2015-02-23T16:13:22.007-08:00"
rn="rtinfraVlanNs-[uni/phys-bsprint-PHY]" status="" tCl="physDomP" tDn="uni/phys-
bsprint-PHY"/>
<fvnsEncapBlk childAction="" descr="" from="vlan-100" lcOwn="local" modTs="2015-02-
23T15:58:33.538-08:00" name="" rn="from-[vlan-100]-to-[vlan-199]" status="" to="vlan-
199" uid="8131"/>
</fvnsVlanInstP>
Create Physical Domain

A physic
al do
main acts as the link between the VLAN pool and the Ac cess Entity Pro
file
(AEP). The domain also ties the fabric con
fig
ur
a
tion to the ten
ant con
fig
ur
a
tion, as the
ten
ant adminis
tra
tor is the one who asso
ci
ates do
mains to EPGs, while the domains are
cre
ated under the fab ric tab. When config
ur
ing in this order, only the pro
file name and
the VLAN pool are con fig
ured. The cre
ation of the AEP and its as so
ci
at
ion will be cov
-
ered later in this sec
tion.
2 In the Navigation pane, choose Physical and External Domains > Physical
Domains.
3 In the Work pane, choose Actions > Create Physical Domain.
4 In the Create Physical Domain dialog box, perform the following actions:
a. Define a meaningful name for the profile.
b. Select the VLAN pool you just created.
XML Ob
ject
<physDomP childAction="" configIssues="" dn="uni/phys-bsprint-PHY" lcOwn="local"

modTs="2015-02-23T16:13:21.906-08:00" monPolDn="uni/fabric/monfab-default"
name="bsprint-PHY" ownerKey="" ownerTag="" status="" uid="8131">
<infraRsVlanNs childAction="" forceResolve="no" lcOwn="local" modTs="2015-02-
23T16:13:22.065-08:00" monPolDn="uni/fabric/monfab-default" rType="mo" rn="rsvlanNs"
state="formed" stateQual="none" status="" tCl="fvnsVlanInstP" tDn="uni/infra/vlanns-
[bsprint-vlan-pool]-static" tType="mo" uid="8131"/>
<infraRsVlanNsDef childAction="" forceResolve="no" lcOwn="local" modTs="2015-02-
23T16:13:22.065-08:00" rType="mo" rn="rsvlanNsDef" state="formed" stateQual="none"
status="" tCl="fvnsAInstP" tDn="uni/infra/vlanns-[bsprint-vlan-pool]-static"
tType="mo"/>
<infraRtDomP childAction="" lcOwn="local" modTs="2015-02-23T16:13:52.945-08:00"
rn="rtdomP-[uni/infra/attentp-bsprint-AEP]" status="" tCl="infraAttEntityP"
tDn="uni/infra/attentp-bsprint-AEP"/>
</physDomP>
Create an Attachable Access Entity Profile (AEP)

The AEP links the phys ic
al do
main and its VLAN Pool to the in
ter
face poli
cies. The con
-
fig
ur
a
tion for an AEP is straight
for
ward.
2 In the Navigation pane, choose Global Policies > Attached Acess Entity Profle.
3 In the Work pane, choose Actions > Create Attached Entity Profile.
4 In the Create Attached Entity Profile dialog box, perform the following actions:
a. Define the a meaningful name for the profile.
b. Optionally, enter a description.
c. Click + to associate the domain to the AEP.
d. Select the physical domain that was previously configured.
5 Click Next.
6 Click Submit.
XML Ob
ject
<infraAttEntityP childAction="" configIssues="" descr="" dn="uni/infra/attentp-

bsprint-AEP" lcOwn="local" modTs="2015-02-23T16:13:52.874-08:00"
monPolDn="uni/fabric/monfab-default" name="bsprint-AEP" ownerKey="" ownerTag=""
status="" uid="8131">
<infraContDomP childAction="" lcOwn="local" modTs="2015-02-23T16:13:52.874-08:00"
rn="dompcont" status="">
<infraAssocDomP childAction="" dompDn="uni/phys-bsprint-PHY" lcOwn="local"
modTs="2015-02-23T16:13:52.961-08:00" rn="assocdomp-[uni/phys-bsprint-PHY]" status=""/>
<infraAssocDomP childAction="" dompDn="uni/l2dom-JC-L2-Domain" lcOwn="local"
modTs="2015-02-25T11:35:33.570-08:00" rn="assocdomp-[uni/l2dom-JC-L2-Domain]"
status=""/>
</infraContDomP>
<infraContNS childAction="" lcOwn="local" modTs="2015-02-23T16:13:52.874-08:00"
monPolDn="uni/fabric/monfab-default" rn="nscont" status="">
<infraRsToEncapInstDef childAction="" deplSt="" forceResolve="no"
lcOwn="local" modTs="2015-02-23T16:13:52.961-08:00" monPolDn="uni/fabric/monfab-
default" rType="mo" rn="rstoEncapInstDef-[allocencap-[uni/infra]/encapnsdef-
[uni/infra/vlanns-[bsprint-vlan-pool]-static]]" state="formed" stateQual="none"
status="" tCl="stpEncapInstDef" tDn="allocencap-[uni/infra]/encapnsdef-
[uni/infra/vlanns-[bsprint-vlan-pool]-static]" tType="mo">
<fabricCreatedBy childAction="" creatorDn="uni/l2dom-JC-L2-Domain"
deplSt="" domainDn="uni/l2dom-JC-L2-Domain" lcOwn="local" modTs="2015-02-
25T11:35:33.570-08:00" monPolDn="uni/fabric/monfab-default" profileDn="" rn="source-
[uni/l2dom-JC-L2-Domain]" status=""/>
<fabricCreatedBy childAction="" creatorDn="uni/phys-bsprint-PHY" deplSt=""
domainDn="uni/phys-bsprint-PHY" lcOwn="local" modTs="2015-02-23T16:13:52.961-08:00"
monPolDn="uni/fabric/monfab-default" profileDn="" rn="source-[uni/phys-bsprint-PHY]"
status=""/>
</infraRsToEncapInstDef>
</infraContNS>
<infraRtAttEntP childAction="" lcOwn="local" modTs="2015-02-24T11:59:37.980-08:00"
rn="rtattEntP-[uni/infra/funcprof/accportgrp-bsprint-AccessPort]" status=""
tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-bsprint-AccessPort"/>
<infraRsDomP childAction="" forceResolve="no" lcOwn="local" modTs="2015-02-
25T11:35:33.570-08:00" monPolDn="uni/fabric/monfab-default" rType="mo" rn="rsdomP-
[uni/l2dom-JC-L2-Domain]" state="formed" stateQual="none" status="" tCl="l2extDomP"
tDn="uni/l2dom-JC-L2-Domain" tType="mo" uid="8754"/>
<infraRsDomP childAction="" forceResolve="no" lcOwn="local" modTs="2015-02-
23T16:13:52.961-08:00" monPolDn="uni/fabric/monfab-default" rType="mo" rn="rsdomP-
[uni/phys-bsprint-PHY]" state="formed" stateQual="none" status="" tCl="physDomP"
tDn="uni/phys-bsprint-PHY" tType="mo" uid="8131"/>
</infraAttEntityP>
Create Interface Policies

Next, de
fine the in
ter
face pro
files and show
case the re-us
abil
ity of the fab
ric poli
-
cies. In
ter
face poli
cies can be re-used as needed by dif
fer
ent in
ter
face pro
file de
f
in
i
t
ion
re
quire
ments. This sec tion will il
lus
trate cre
ation of new pro
files, but ide
ally there are
al
ready poli
cies in place that can sim ply be se
lected.
Create Link Level Policies
Link level poli

cies dic
tate con
fig
ur
a
tion like the speed of ports.
2 In the Navigation pane, choose Interface Policies > Policies > Link Level.
3 In the Work pane, choose Actions > Create Link Level Policy.
4 In the Create Link Level Policy dialog box, perform the following actions:
a. Define the meaningful name for the policy.
b. Optionally, provide a description for the policy.
c. Select the auto negotiation mode for the interface.
d. Select the interface speed. Leaf switch ports default to 10GE.
e. Change the de-bounce interval if required.
5 Click Submit.
XML Ob
ject
<fabricHIfPol autoNeg="on" childAction="" descr="" dn="uni/infra/hintfpol-1G-Auto"

lcOwn="local" linkDebounce="100" modTs="2015-01-14T06:47:15.693-08:00" name="1G-Auto"
ownerKey="" ownerTag="" speed="1G" status="" uid="15374">
<fabricRtHIfPol childAction="" lcOwn="local" modTs="2015-01-14T06:48:48.081-
08:00" rn="rtinfraHIfPol-[uni/infra/funcprof/accportgrp-UCS-1G-PG]" status=""
tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-UCS-1G-PG"/>
<fabricRtHIfPol childAction="" lcOwn="local" modTs="2015-02-25T11:48:11.331-
08:00" rn="rtinfraHIfPol-[uni/infra/funcprof/accportgrp-L3-Example]" status=""
tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-L3-Example"/>
</fabricHIfPol>
Create a CDP Interface Policy
2 In the Navigation pane, choose Interface Policies > Policies > CDP Interface.
a. Define a meaningful name for the policy such as 'CDP-Enable'.

c. Choose either admin state enabled or disabled.
5 Click Submit.
XML Ob
ject
<cdpIfPol adminSt="enabled" childAction="" descr="" dn="uni/infra/cdpIfP-CDP-Enable"

lcOwn="local" modTs="2015-01-14T06:47:25.470-08:00" name="CDP-Enable" ownerKey=""
ownerTag="" status="" uid="15374">
<cdpRtCdpIfPol childAction="" lcOwn="local" modTs="2015-01-14T07:23:54.957-
08:00" rn="rtinfraCdpIfPol-[uni/infra/funcprof/accportgrp-UCS-10G-PG]" status=""
08:00" rn="rtinfraCdpIfPol-[uni/infra/funcprof/accbundle-ACI-VPC-IPG]" status=""
tCl="infraAccBndlGrp" tDn="uni/infra/funcprof/accbundle-ACI-VPC-IPG"/>
08:00" rn="rtinfraCdpIfPol-[uni/infra/funcprof/accportgrp-UCS-1G-PG]" status=""
08:00" rn="rtinfraCdpIfPol-[uni/infra/funcprof/accportgrp-bsprint-AccessPort]"
status="" tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-bsprint-
AccessPort"/>
08:00" rn="rtinfraCdpIfPol-[uni/infra/funcprof/accportgrp-L3-Example]" status=""
tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-L3-Example"/>
</cdpIfPol>
Create an LLDP Interface Policy
2 In the Navigation pane, choose Interface Policies > Policies > LLDP Interface.
a. Define a meaningful name for the policy.
c. Choose the receive state.
d. Choose the transmit state.
5 Click Submit.

XML Object descr=""
<lldpIfPol adminRxSt="enabled" adminTxSt="enabled" childAction=""

dn="uni/infra/lldpIfP-LLDP-Enable" lcOwn="local" modTs="2015-02-11T07:40:35.664-08:00"
name="LLDP-Enable" ownerKey="" ownerTag="" status="" uid="15374"> status=""
/>
<lldpRtLldpIfPol childAction="" lcOwn="local" modTs="2015-02-24T14:59:11.154-
08:00" rn="rtinfraLldpIfPol-[uni/infra/funcprof/accbundle-ACI-VPC-IPG]"
tCl="infraAccBndlGrp" tDn="uni/infra/funcprof/accbundle-ACI-VPC-IPG" status=""
/>
<lldpRtLldpIfPol childAction="" lcOwn="local" modTs="2015-02-25T11:48:11.331-
08:00" rn="rtinfraLldpIfPol-[uni/infra/funcprof/accportgrp-L3-Example]"
tCl="infraAccPortGrp" tDn="uni/infra/funcprof/accportgrp-L3-Example"
</lldpIfPol>
l
ows
Create an LACP Interface Policy
faces. These can be
Link Aggre
gation Con trol Pro
to
col is part of an IEEE specif
i
ca
tion (802.3ad) that al
ra
tion re
quire
ments
an optional nego
tiat
ion proto
col to be run on Port Chan nel in
ter
pre-emp tively de
fined to be used as needed for the var io
us con fig
u
in the data center.
1 On the menu bar, choose Fabric > Access Policies. .
2 In the Navigation pane, choose Interface Policies > Policies > LACP
3 In the Work pane, choose Actions > Create LACP Policy.
4 In the Create LACP Policy dialog box, perform the following actions:
c. Select the LACP mode required for the server. Note if LACP is enabled on the
leaf switch, LACP must also be enabled on the server or other connected
device.
d. Optionally, specify the minimum and maximum number of links in the Port
Channel.
5 Click Submit.
XML Ob
ject
<lacpLagPol childAction="" ctrl="fast-sel-hot-stdby,graceful-conv,susp-individual"

descr="" dn="uni/infra/lacplagp-LACP-Active" lcOwn="local" maxLinks="16" minLinks="1"
modTs="2015-02-24T11:58:36.547-08:00" mode="active" name="LACP-Active" ownerKey=""
ownerTag="" status="" uid="8131">
<lacpRtLacpPol childAction="" lcOwn="local" modTs="2015-02-24T14:59:11.154-
08:00" rn="rtinfraLacpPol-[uni/infra/funcprof/accbundle-ACI-VPC-IPG]" status=""
tCl="infraAccBndlGrp" tDn="uni/infra/funcprof/accbundle-ACI-VPC-IPG"/>
</lacpLagPol>
Create an LACP Member Profile (optional)
Op
tion
ally, the LACP mem
ber pro
file pro
vides the abil
ity to pro
vide pri
or
ity spec
if
i
ca
-
tions to mem
bers of an LACP group.
2 In the Navigation pane, choose Interface Policies > Policies > LACP Member.
3 In the Work pane, choose Actions > Create LACP Member Policy.
4 In the Create LACP Member Policy dialog box, perform the following actions:
c. If required, change the priority.
d. If required, change the transmit rate.
5 Click Submit.
Create a Spanning Tree Interface Policy (optional)
The Span ning Tree pol

icy dic
tates the behav
ior of south
bound leaf port Spanning Tree
fea
tures. It is a com
mon best practice to en
able BPDU guard on in
terfaces con
nected to
servers.
Note: ACI does not run Spanning Tree on the fabric be

tween the leaves and spines. The
Span
ning Tree in
ter
face pol
icy sim
ply de
fines the port be
hav
ior.
2 In the Navigation pane, choose Interface Policies > Policies > Spanning Tree
Interface.
3 In the Work pane, choose Actions > Create Spanning Tree Interface Policy.
4 In the Create Spanning Tree Interface Policy dialog box, perform the following
actions:
c. Enable BPDU filter and/or BPDU guard.
5 Click Submit.
Create a Storm Control Policy (optional)
A traf
fic storm oc
curs when pack ets flood the LAN, cre at
ing exces
sive traf
fic and de grad
-
ing network perfor
mance. The traffic storm con trol fea
ture can be used to pre vent dis
-
ruptions on ports by a broad
cast, multi
cast, or uni
cast traf
fic storm on phys i
cal in
ter
faces.
2 In the Navigation pane, choose Interface Policies > Policies > Storm Control.
3 In the Work pane, choose Actions > Create Storm Control Policy.
4 In the Create Storm Control Policy dialog box, perform the following actions:
c. Specify how the control policy is to be applied, either through percentage of
the total bandwidth or as a packet per second definition that matches the
requirement for the data center
5 Click Submit.
Creating a Layer 2 Interface Policy to enable per port-VLAN
2 In the Navigation pane, choose Interface Policies > Policies > L2 Interface.
3 In the Work pane, choose Actions > Create L2 Interface Policy.
4 In the Create L2 Interface Policy dialog box, perform the following actions:
a. Give the L2 Interface name and an optional description.
b. Select VLAN scope to Port Local scope to enable per port-VLAN.
Create Interface Policy Groups

The in
ter
face pol
icy groups com prise the in
ter
face poli
cies as a functional group that is
as
so
ci
ated to an in
ter
face. The fol
lowing di
ag
ram shows how pre vi
ously created items
are grouped under the policy group.
Policies contained in a policy group
Once all the in

ter
face poli
cies have been de fined, the in
di
vid
ual poli
cies can be brought
to
gether to form a pol icy group that will be linked to the in ter
face profile. The pol
icy
group is de
fined from a mas ter de
f
in
i
t
ion that en
com passes being one of the fol low
ing:
Access Policy Group

Port Channel Policy Group
VPC Policy Group
Create Access Port Policy Group
The ac
cess port pol
icy is de
fined for an in
di
vid
ual link (non-Port Chan
nel or vPC).
2 In the Navigation pane, choose Interface Policies > Policy Groups.
3 In the Work pane, choose Actions > Create Access Policy Group.
4 In the Create Access Policy Group dialog box, perform the following actions:
a. Define a meaningful name for the policy group.
b. Optionally, provide a description for the policy group.
c. Use the profiles created previously that are relevant for this policy group.
5 Click Submit.
Create Port Channel Interface Policy Group
Port Chan
nel
ing also load-bal
ances traf
fic across the phys
ic
al in
ter
faces that are mem
-
bers of the channel group. For every group of in ter
faces that needs to be con fig
ured
into a port chan
nel, a dif
fer
ent pol
icy group has to be cre
ated. This pol
icy group defines
the be
hav
iour. For ex
am
ple, if ports 1/1-4 are to be con
fig
ured into one port chan
-
nel, and ports 1/5-8 into a sep
ar
ate port chan
nel, each of those groups would re
quire
the cre
ation of a sep
ar
ate pol
icy group.
3 In the Work pane, choose Actions > Create PC Interface Policy Group.
4 In the Create PC Interface Policy Group dialog box, perform the following
actions:
c. Select the policies created previously that are relevant for this PC policy
group.
5 Click Submit.
Create VPC Interface Policy Group
Note: This ob
ject must be unique for each VPC cre
ated.
A vir
tual PortChan nel (vPC) al
lows links that are physic
ally connected to two differ
ent
devices to ap
pear as a sin
gle Port Channel to a third de
vice. In the world of ACI, pairs of
leaf switches may be con fig
ured in a vPC do main so that down stream devices can be
ac
tive-active dual-homed.
For every group of inter

faces that are to be con
figured into a vPC, a dif fer
ent inter
face
pol
icy group needs to be created. The vPC policy group con tains both the de f
in
i
t
ion for
the behav
iour of the port channel, and the iden
ti
fier. For ex
ample, if ports 1/1-4 are to
be config
ured into one vPC across two switches, and ports 1/5-8 into a sep arate vPC
across two switches, each of those groups would re quire the def
in
it
ion of a sep ar
ate
pol
icy group.
Note: For vPC you will also require a unique vPC do

main de
f
in
i
t
ion be
tween the two
paired switches. More de
tails to fol
low.

3 In the Work pane, choose Actions > Create VPC Interface Policy Group.
4 In the Create VPC Interface Policy Group dialog box, perform the following
actions:
c. Choose the policies created previously that are relevant for this vPC policy
group.
5 Click Submit.
Interface Profile
The in ter
face profile in ACI links the pol icy groups that de fine how the in terface is
going to be have, and as signs them to spe cific ports via the con cept of inter
face selec
-
tor. In turn, the inter
face profile is even
tually tied to a switch profile to specify which
leaf switch the ref er
enced ports should be con fig
ured. As we con tinue the process of
defining the port pro files, you can observe how we have started at the bot tom of this
object tree config
ur
ing the differ
ent profiles. The purposes for all these in
di
vid
ual poli
-
cies that tie to
gether is to max im
ize pol
icy re-use.
Interface Profile links to Interface Selector and Interface Policy Group
The diag
ram in the previ
ous section provides a vi
sual descrip
tion of what can be ac -
complished by grouping the poli
cies that have been de fined under the inter
face pro
file,
and then as
signed to ports with inter
face selec
tors and the access port pol
icy groups.
Create Interface Profile
The in
ter
face pro
file is com
posed of two pri
mary ob
jects. The in
ter
face se
lec
tor and
the ac
cess port pol
icy group. The in
ter
face se
lec
tor de
fines what in
ter
faces will apply
the ac
cess port pol
icy. The ports that share the same at
trib
utes can then be grouped
under the same in
ter
face pro
file.
2 In the Navigation pane, choose Interface Policies > Profiles.
3 In the Work pane, choose Actions > Create Interface Profile.
4 In the Create Interface Profile dialog box, perform the following actions:
b. Optionally, provide a description for the profile.
5 Click Submit.
Next, add the in

ter
face se
lec
tors that are as
so
ci
ated to this in
ter
face pro
file.
Create Interface Selector
2 In the Navigation pane, choose Interface Policies > Profiles

> Name_of_Interface_Profile_Created.
3 In the Work pane, choose Actions > Create Access Port Selector.
4 In the Create Access Port Selector dialog box, perform the following actions:
c. Enter the interface IDs.
d. Choose the interface policy group that should be associated to these ports.
5 Click Submit.
Create Interface Profile for Port Channel
If a server has two or more up links to a leaf switch, the links can be bun dled into a Port
Chan nel for resiliency and load dis
tri
bu
tion. In order to config
ure this in ACI, cre
ate an
inter
face pol icy group of type Port Chan nel to bun dle the inter
faces. Dif
ferent Port
Chan nels re
quire differ
ent pol
icy groups.
Port Channel Policy Group
5 Click Submit.
Next, create an inter

face port se lec
tor. Since you will be config
ur
ing a Port Chan nel,
the oper
at
or will add all of the in
ter
faces required in the Port Channel in
ter
face. In this
ex
ample inter
faces Ethernet 1/1-2 will be con fig
ured in one Port Chan nel and inter-
faces Ether
net 1/3-4 will be con fig
ured in another Port Channel.

c. Enter interface IDs for the first port channel.
d. Choose the interface policy group.
5 Click Submit.
6 Repeat this process for the second Port Channel (if you have another Port
Channel to add).
Create an Interface Profile for Virtual Port Channel
A vPC do
main is al
ways made up of two leaf switches, and a leaf switch can only be a
member of one vPC do
main. In ACI, that means that the de
f
in
i
t
ion of the poli
cies is sig
-
nif
ic
ant be
tween the two switches. The same pol icy can be reused be tweeen the two
switches, and through the vPC do main the pair as soci
at
ion can be defined. vPC Switch
domain mem bers should be taken into con sid
erat
ion when con fig
ur
ing firmware main -
te
nance groups. By keep ing this in mind, firmware up grades should never im pact both
vPC switch peers at the same time. More de tails on this can be found in the Up grad
ing
and Down
grad
ing Firmware sec
tion.
For this rea

son, a switch pro file that would repre
sent two sep
ar
ate switch IDs needs to
be created. The re la
tion
ship of these switches to the two ports in the same pol icy
group is defined through the in ter
face pro
file.
vPC Policy Group

The same process would have to be re

peated for every grouped in
ter
face on each side
that will be a mem
ber of the vPC.
5 Click Submit.

c. Enter interface IDs.
d. Select of the interface policy group to be used for the vPC port behavior.
9 Click Submit.
Create a vPC Domain for Virtual Port Channel
When con fig

ur
ing a vPC, there is one addi
tional step to be con
fig
ured once to put two
leaf switches into the same vPC do main.
Creating a vPC Domain

2 In the Navigation pane, choose Switch Policies > VPC Domain > Virual Port
Channel default.
3 In the Work pane, choose Actions > Explicit VPC Protection Group.
4 In the Explicit VPC Protection Group dialog box, perform the following actions:
a. Define a meaningful name for the vPC domain.
b. Provide a unique ID to represent the vPC domain.
c. Choose the first switch you want to be part of the vPC domain.
d. Choose the second switch you want to be part of the vPC domain.
5 Click Submit.
Switch Profiles
A switch pro file groups all the inter
face profiles that de fine the be
havior of its re
spec
-
tive switch ports. A switch pro file could be the de f
in
i
t
ion of a sin
gle switch or it could
be the de f
in
i
tion of mul
ti
ple switches. As a best prac tice, there should be a switch pro -
file for each leaf switch, and an ad di
tional switch pro file for each vPC do main pair of
leaf switches.
The in ter
face pro files that you have cre ated can be as sociated to the switch through a
sin
gle switch pro file or they can be as soci
ated through dif fer
ent switch profiles. If you
have var io
us racks that are iden ti
cal in the way the in terface ports are con fig
ured, it
could be ben e
fi
cial to uti lize the same switch pro file. This would make it pos si
ble to
mod ify the config
ur ation of many switches dur ing op er
ations without hav
ing to con fig
-
ure each switch in dividua
lly.
Reusability
The ca pabil
ity of policy reusability is cru
cial to re-em phasize from an op er
at
ional per-
spective. If a profile has been de fined to con figure a port as 1GB speed for ex ample, that
pro
file can be reused for many in ter
face pol icy groups. When look ing at whole switch
con
figura
tions, the re-us abil
ity of the pro file can be ex tended to sim plify data cen ter
op
erations and en sure com pliance. The fol lowing fig
ure illus
trates the reusabil ity of
pro
files across racks of switches.
Policy re-use at scale
In the pre vi
ous diag
ram, each of the top of rack switches is based on the same switch
profile. If all these racks are con
fig
ured in the same fash
ion (meaning they are wired in
the same way) the same poli cies could be reused by simply as
sign
ing the switches to
the same switch profile. It would then in
herit the pro
file tree and be config
ured the
exact same way as the other racks.
It is also im
portant to be aware of the im pli
ca
tion of delet
ing profiles. If a pro
file has
been reused across many de vices, make sure to check where it is being used be fore you
delete the profile or pol
icy.
Sample vPC Creation

The fol
lowing procedure demon strates what a vPC bringup looks like and also API
POST configur
a
tion as
s
es
ment of the vPC. The fol
low
ing topol
ogy will be con
fig
ured:
Sample Topology
Create VLAN Pools
2 In the Navigation pane, choose Pools > VLAN.
3 In the Work pane, choose Actions > Create VLAN Pool.
4 In the Create VLAN Pool dialog box, perform the following actions:
a. Define a meaningful name for the pool.
b. Optionally, provide a description for the pool.
c. Click Static Allocation for the allocation mode.
Note: For this example the pool will be from VLAN 100 to VLAN 199.
REST :: /api/node/class/fvnsVlanInstP.xml
CLI :: moquery -c fvnsVlanInstP
Create a Physical Domain
2 In the Navigation pane, choose Physical and External Domains > Physical
Domains.
3 In the Work pane, choose Actions > Create Physical Domain.
4 In the Create Physical Domain dialog box, perform the following actions:
a. Define a meaningful name for the domain.
b. Choose the VLAN pool that you previously created.
REST :: /api/node/class/physDomP.xml
CLI :: moquery -c physDomP
Create Access Entity Profile
2 In the Navigation pane, choose Global Policies > Attachable Access Entity
Profles.
3 In the Work pane, choose Actions > Create Attached Entity Profile.
4 In the Create Attached Entity Profile dialog box, perform the following actions:
a. Define a meaningful name for the AEP.
b. Provide a unique ID to represent the AEP.
c. Click + to add a domain that will be associated to the interfaces.
d. Choose the physical domain that you previously created.
5 Click Next.
6 Click Submit.
REST :: /api/node/class/infraAttEntityP.xml
CLI :: moquery -c infraAttEntityP
Interface Policies
Create Link Level Policy
2 In the Navigation pane, choose Interface Policies > Policies > Link Level.
3 In the Work pane, choose Actions > Create Link Level Policy.
4 In the Create Link Level Policy dialog box, perform the following actions:
a. Define a meaningful name for the pool.
c. Choose the auto negotiation for the interface.
d. Choose the speed to match the interface requirement.
5 Click Submit.
REST :: /api/node/class/fabricHIfPol.xml
CLI :: moquery -c fabricHIfPol
Create LACP Policy
2 In the Navigation pane, choose Interface Policies > Policies > LACP.
3 In the Work pane, choose Actions > Create LACP Policy.
4 In the Create LACP Policy dialog box, perform the following actions:
b. Optionally, provide a description for the pool.
c. In mode click on Active.
5 Click Submit.
REST :: /api/node/class/lacpLagPol.xml
CLI :: moquery -c lacpLagPol
Create vPC Interface Policy Group
3 In the Work pane, choose Actions > Create vPC Interface Policy Group.
4 In the Create vPC Interface Policy Group dialog box, perform the following
actions:
c. Choose a Link Level Policy.
d. Choose an LACP Policy.
Note: LACP is recommended for vPCs. However, ensure LACP is configured
on the device connected to the leaf switch.
e. Choose an AEP to associate the policy group to.
5 Click Submit.
REST :: /api/node/class/infraAccBndlGrp.xml
CLI :: moquery -c infraAccBndlGrp
Create an Interface Profile
5 Click Submit.
6 In the Navigation pane, choose Interface Policies > Profiles > Profiles > ACI-
VPC-int-profile.
b. Optionally, provide a description.
c. Select the proper interfaces.
d. Select an interface policy group.
9 Click Submit.
REST :: /api/node/class/infraAccPortP.xml
CLI :: moquery -c infraAccPortP
Create a Switch Profile
2 In the Navigation pane, choose Switch Policies > Profiles.
3 In the Work pane, choose Actions > Create Switch Profile.
4 In the Create Switch Profile dialog box, perform the following actions:
c. In switch selectors click on the + symbol.
i. Name: 103-104 (example node numbers).
ii. Blocks: Select switch 103 and switch 104.
5 Click Next.
6 Select the previously created interface selector.
7 Click Finish.
REST :: /api/node/class/infraNodeP.xml
CLI :: moquery -c infraNodeP
Create vPC domain
2 In the Navigation pane, choose Switch Policies > VPC Domain > Virual Port
Channel default.
3 In the Work pane, choose Actions > Explicit VPC Protection Group.
a. In the Explicit VPC Protection Group dialog box, perform the following
actions:
b. In the Explicit VPC Protection Groups section, click + to create a vPC
protection group.
c. Define a meaningful name for the vPC domain.
d. Provide a unique ID for the vPC domain.
e. Select the first switch ID that is part of this vPC pair: 103.
f. Select the second switch ID that is part of this vPC pair: 104.
4 Click Submit.
REST :: /api/node/class/fabricExplicitGEp.xml
CLI :: moquery -c fabricExplicitGEp
Validate Operation of Configured vPC
1 On the menu bar, choose Fabric > Inventory.
2 In the Navigation pane, choose POD 1 > Interfaces > vPC Interfaces.
3 In the Work pane, there will be a table that shows the status of the vPC
interface. If configured correctly, the status should be displayed and you should
see successful establishment of the vPC domain.
You can also val

id
ate the op
er
at
ion of the vPC di
rectly from the CLI of the switch it
self.
If you con
nect to the con
sole or the out of band man
age
ment in
ter
face of the leaf node
you should be able to see the sta
tus with the com
mand show vpc.
Leaf-3# show vpc

Legend:
(*) - local vPC is down, forwarding via vPC peer-link
vPC domain id : 100

Peer status : peer adjacency formed ok
vPC keep-alive status : Disabled
Configuration consistency status : success
Per-vlan consistency status : success
Type-2 consistency status : success
vPC role : primary
Number of vPCs configured : 1
Peer Gateway : Disabled
Dual-active excluded VLANs : -
Graceful Consistency Check : Enabled
Auto-recovery status : Enabled (timeout = 240 seconds)
Operational Layer3 Peer : Disabled
vPC Peer-link status

---------------------------------------------------------------------
id Port Status Active vlans
-- ---- ------ --------------------------------------------------
1 up -
vPC status
----------------------------------------------------------------------
id Port Status Consistency Reason Active vlans
-- ---- ------ ----------- ------ ------------
1 Po1 up success success -
The fol
low
ing REST API call can be used to build vPCs and at
tach vPCs to sta
tic port
bind
ings.
URL: https://{{apic-ip}}/api/policymgr/mo/.xml
<polUni>
<infraInfra>

<infraNodeP name="switchProfileforVPC_201">
<infraLeafS name="switchProfileforVPC_201" type="range">
<infraNodeBlk name="nodeBlk" from_="201" to_="201"/>
</infraLeafS>
<infraRsAccPortP tDn="uni/infra/accportprof-intProfileforVPC_201"/>
</infraNodeP>
<infraNodeP name="switchProfileforVPC_202">
<infraLeafS name="switchProfileforVPC_202" type="range">
<infraNodeBlk name="nodeBlk" from_="202" to_="202"/>
</infraLeafS>
<infraRsAccPortP tDn="uni/infra/accportprof-intProfileforVPC_202"/>
</infraNodeP>

<infraAccPortP name="intProfileforVPC_201">
<infraHPortS name="vpc201-202" type="range">
<infraPortBlk name="vpcPort1-15" fromCard="1" toCard="1" fromPort="15"
toPort="15"/>
<infraRsAccBaseGrp tDn="uni/infra/funcprof/accbundle-intPolicyGroupforVPC"/>
</infraHPortS>
</infraAccPortP>
<infraAccPortP name="intProfileforVPC_202">
<infraHPortS name="vpc201-202" type="range">
<infraPortBlk name="vpcPort1-1" fromCard="1" toCard="1" fromPort="1"
toPort="1"/>
<infraRsAccBaseGrp tDn="uni/infra/funcprof/accbundle-intPolicyGroupforVPC"/>
</infraHPortS>
</infraAccPortP>

<infraFuncP>
<infraAccBndlGrp name="intPolicyGroupforVPC" lagT="node">
<infraRsAttEntP tDn="uni/infra/attentp-AttEntityProfileforCisco"/>
<infraRsCdpIfPol tnCdpIfPolName="CDP_ON" />
<infraRsLacpPol tnLacpLagPolName="LACP_ACTIVE" />
<infraRsHIfPol tnFabricHIfPolName="10GigAuto" />
</infraAccBndlGrp>
</infraFuncP>
</infraInfra>
</polUni>
https://{{hostName}}/api/node/mo/uni.xml
<polUni>
<fvTenant descr="" dn="uni/tn-Cisco" name="Cisco" ownerKey="" ownerTag="">
<fvAp descr="" name="CCO" ownerKey="" ownerTag="" prio="unspecified">
<fvAEPg descr="" matchT="AtleastOne" name="Web" prio="unspecified">

<fvRsPathAtt encap="vlan-1201" instrImedcy="immediate" mode="native"
tDn="topology/pod-1/protpaths-201-202/pathep-[vpc201-202] />
</fvAEPg>
<fvAEPg descr="" matchT="AtleastOne" name="App" prio="unspecified">
tDn="topology/pod-1/protpaths-201-202/pathep-[vpc201-202] />
</fvAEPg>
</fvAp>
</fvTenant>
</polUni>
Server Connectivity
Server connec
tiv
ity is nec
es
sary for all appli
ca
tion work loads to function prop erly
on the Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) fab
ric. The fab
ric connec
tiv
ity re
-
quire
ments that are dic
tated by the server in
fra
struc
ture must be care
fully con
sid
-
ered. In the case of Cisco Uni
fied Comput
ing Sys
tem (UCS), fab
ric access poli
cies must
be provi
soned to match these re quire
ments. These poli
cies are all gov
erned by in ter
-
face pol
icy groups. ACME Inc. has sev
eral dif
fer
ent mod
els of servers in their data cen
-
ters, such as Cisco UCS B and C se
ries, as well as some third party servers that all need
to be con
nected to the ACI fab
ric.
Cisco UCS B-Series Servers

If UCS B-series Fab
ric Inter
con
nects are being con nected to your ACI fab ric and Cisco
UCS, the Link Level Discovery Pro
to
col (LLDP) must be dis abled. Cisco Dis
covery Pro
to-
col (CDP) must be enabled on any ACI fab ric in
ter
faces that are connect
ing to UCS Fab -
ric In
ter
con
nects. You can pre-pro vi
sion these policies as part of the ini
tial manage-
ment tasks. See the Con fig
ur
ing Management Pro to
cols chapter for more in
forma
tion.
When con nect

ing UCS to the ACI fab ric, the type of Layer 2 con nec
tion needed on the
Fabric In
ter
connect facing ports must be de ter
mined first. A best prac
tice is to lever
age
a vir
tual pri
vate cloud (vPC) to con nect the UCS en vi
ron
ment so as to cre ate a multi-
chassis ether
chan nel. In this sce
nario, in
di
vid
ual link and fab
ric switch fail
ures are miti-
gated to main tain a higher ex pected up time.
For more in for

mation on the process needed to con
fig
ure links to UCS as either a vPC
or a tra
di
tional port chan
nel, see the Adding New De
vices to the Fab
ric sec
tion.
Standalone Rack Mount Servers or Non-Cisco Servers

Any non-UCS server archi
tec
ture can also be con
nected di
rectly to the ACI fabric or to a
Cisco Nexus 2000 Fab
ric Ex
ten
der (FEX). When being connected to the ACI fabric, the kind
of traf
fic ex
pected out of the server links needs to be de ter
mined. If the work load is a
bare metal server, traf
fic can be classi
fied on a per port basis and as soci
ated AEPs and
EPGs can be mapped ap propri
ately to match the encap
sulated traf
fic. If a supported hy-
per
vi
sor is to be used, a Vir
tual Ma
chine Man
ager (VMM) domain must be prop erly con
fig
-
ured, and then as so
ci
ated with the cor
re
spond
ing ports on the fab
ric as a hy
per
vi
sor that
is fac
ing through EPG and AEP map
ping. The key is to map the ex
pected traf
fic clas
si
fi
ca
-
tion to the ports that are con
nected to the server in
fra
struc
ture.
Uti
liz
ing a FEX is an al
ter
na
tive way to con
nect host de
vices into the ACI fab
ric. Re
-
stric
tions that are pre
sent in NX-OS mode such that non-host-fac
ing ports are not
sup
ported, are still true. Ports must only be con
nected to hosts, and con
nec
tiv
ity to any
other net
work de
vice will not func
tion prop
erly. When uti
liz
ing a FEX, all host-fac
ing
ports are treated the same way as if they were di
rectly at
tached to the ACI fab
ric.
Virtual Machine Networking
Understanding VM Networking in ACI

One of the most com
mon uses of the Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) will
be to help man
age and de
ploy ap
pli
ca
tions in vir
tual en
vi
ron
ments. The ACI pro
vides
the abil
ity to man
age both vir
tual and phys
ic
al end
points with the same set of poli
cies.
This chap ter will look at var
io
us op
er
at
ional tasks that will be per
formed through
out
the daily oper
at
ions.
The fol
low
ing list de
scribes some vir
tual ma
chine man
ager (VMM) sys
tem terms:
A virtual machine controller is an external VMM entity, such as VMware

vCenter, VMware vShield, and Microsoft Systems Center Virtual Machine
Manager (SCVMM). The Application Policy Infrastructure Controller
(APIC) communicates with the VMM to publish network policies that are
applied to virtual workloads. A virtual machine controller administrator
provides an APIC administrator with a virtual machine controller authentication
credential; multiple controllers of the same type can use the same credential.
Credentials represent the authentication credentials to communicate with
virtual machine controllers. Multiple controllers can use the same credentials.
A pool represents a range of traffic encapsulation identifiers, such as VLAN
and VXLAN IDs, and multicast addresses. A pool is a shared resource and can be
consumed by multiple domains, such as VMM and Layer 4 to Layer 7 services. A
leaf switch does not support overlapping VLAN pools. Different overlapping
VLAN pools must not be associated with the same attachable entity profile
(AEP).
The two types of VLAN-based pools are as follows:
Dynamic pools - Managed internally by the APIC to allocate VLANs for
endpoint groups (EPGs). A VMware vCenter domain can associate only to a
dynamic pool. This is the pool type that is required for VMM integration.
Static pools - The EPG has a relation to the domain, and the domain has a
relation to the pool. The pool contains a range of encapsulated VLANs and
VXLANs. For static EPG deployment, the user defines the interface and the
encapsulation. The encapsulation must be within the range of a pool that is
associated with a domain with which the EPG is associated.
When cre
at
ing dy
namic VLAN pools for VMM in
te
gra
tion, the VLAN range must also be
cre
ated on any inter
me di
ate de
vices, such as tra
di
tional switches or blade switches.
This in
cludes cre ing the VLANs on Uni
at fied Computing Sys
tem (UCS).
For de
tailed in
for
ma
tion on how to de
ploy the VMware vSphere Dis
trib
uted Switch
with the Cisco APIC, see the fol
low
ing doc
um
ents:
Cisco APIC Getting Started Guide
VMware Integration
When inte
grat
ing ACI into your VMware in
fra
struc
ture you have two options for de
-
ploy
ing net
working. VMware do mains can be de ployed, lever
ag
ing the VMware
vSphere Dis
trib
uted Virtual Switch (DVS) or the Cisco Appli
ca
tion Vir
tual Switch (AVS).
Both pro
vide simil
ar basic vir
tual net
work
ing func
tion
al
ity; however, the AVS provides
ad
di
tional ca
pa
bil
it
ies, such as VXLAN and mi
croseg
men
ta
tion sup
port. ACME Inc. has
chosen to lever
age the addi
tional fea
tures pro
vided by AVS. For or ga
niza
tions in
ter
-
ested in using the stan
dard DVS pro vided by VMware, please refer to the fol
lowing doc
-
u
ments:
http://www.cisco.
com/c/en/
us/
td/docs/
switches/
datacenter/aci/
apic/
sw/ 1-x/
getting-started/video/
cisco_
apic_
create_
vcenter_
domain_profile_using_
gui.
html

Shown below are some of the var io
us ACI poli
cies which are in
volved with set ting up
VM In
te
gra
tion. This serves as a ref
er
ence for the ways the var
io
us poli
cies are related
to each other.

Publishing EPGs to a VMM Domain

This sec
tion will de
tail how to pub
lish an ex
ist
ing end
point group (EPG) to a Vir
tual Ma
-
chine Man
ager (VMM) do
main. For de
tails on how to cre
ate EPGs, see the Ten
ants sec
-
tion.
For an EPG to be pushed to a VMM do

main, a do
main bind
ing within the ten
ant EPG
must be cre
ated.
1 On the menu bar, choose Tenants > ALL TENANTS
2 In the Work pane, choose the Tenant_Name.
3 In the Navigation pane, choose Tenant_Name > Application

Profiles > Application_Profile_Name > Application
EPGs > Application_EPG_Name > Domains (VMs and Bare-Metals).
4 In the Work pane, choose Actions > Add VM Domain Association.
5 In the Add VM Domain Association dialog box, choose the VMM Domain Profile
that you previously created.
a. For the Deployment & Resolution Immediacy, Cisco recommends keeping
the default option of On Demand. This provides the best resource usage in
the fabric by only deploying policies to Leaf nodes when endpoints assigned
to this EPG are connected. There is no communication delay or traffic loss
by keeping the default selections.
6 Click Submit.
Note: The EPG will now be available as a Port Group to your VMM.
Connecting VMs to the EPG Port Groups on vCenter

1 Connect to your vCenter using the VI Client.
2 From the Host and Clusters view, right click on your Virtual Machine and select
"Edit Settings".
3 Click on the Network Adapter, and in the Network Connection dropdown

box select the Port Group which corresponds to your EPG. It should display in
the format of
[TENANT|APPLICATION_PROFILE|EPG|VMM_DOMAIN_PROFILE]
If you do not see your ACI EPG in the Net

work Con
nec
tion list, it means one of the fol
-
lowing:
The VM is running on a host which is not attached to the Distributed Switch

managed by the APIC.
There may be a communication between your APIC and vCenter either through
the OOB or INB management network.
Verifying Virtual Endpoint Learning

Once the VMs are connected to the appro
pri
ate port group/EPG, you should ver
ify the
APIC has learned your vir
tual end point.
1 On the menu bar, choose Tenants > ALL TENANTS.
3 In the Navigation pane, choose Tenant_Name > Application

Profiles > Application_Profile_Name > Application
EPGs > Application_EPG_Name.

Note: The current tab should display CLIENT ENDPOINTS. All endpoints either
virtual or physical will be displayed. From here you should be able to find your
Virtual Machine by filtering the "Learning Source" column for rows with values
of "Learned VMM".
Verifying VM Endpoint Learning on the APIC from the CLI

You can verify the same info as above from the CLI by using the 'moquery' (Man
aged Ob
-
ject Query) com mand and adding two filters. One for the Dis
tin
guished Name (DN) name
of your EPG, and one for the Class Name of 'fvCEp' (Fab
ric Vec
tor Client End
point)
moquery -c fvCEp --dn uni/tn-<TENANT_NAME>/ap-<APP_PROFILE_NAME>/epg-<EPG_NAME>
You can de termine the DN of your EPG by right click

ing on the EPG in the GUI, se lect
-
ing "Save As" and looking at the XML ob
ject. From this file you will see the DN entry for
the par
tic
ular EPG:
<imdata totalCount="1"><fvAEPg uid="15374" triggerSt="triggerable" status=""

scope="2588672" prio="unspecified" pcTag="49159" name="epg-od"
monPolDn="uni/tn-common/monepg-default" modTs="2015-02-06T06:46:24.729+11:00"
matchT="AtleastOne" lcOwn="local" dn="uni/tn-mb-tennant1/ap-mb-app-pro/epg-epg-od"
descr="" configSt="applied" configIssues="" childAction=""/></imdata>
Next, use this DN with the mo

query to re
turn the list of client En
points for this EPG:
admin@apic1:~> moquery -c fvCEp --dn uni/tn-mb-tennant1/ap-mb-app-pro/epg-epg-od

Total Objects shown: 1
# fv.CEp
name : 00:50:56:BB:8C:6A
childAction :
dn : uni/tn-mb-tennant1/ap-mb-app-pro/epg-epg-od/cep-00:50:56:BB:8C:6A
encap : vlan-211
id : 0
idepdn :
ip : 10.10.10.10
lcC : learned,vmm
lcOwn : local
mac : 00:50:56:BB:8C:6A
mcastAddr : not-applicable
modTs : 2015-02-06T06:48:52.229+11:00
rn : cep-00:50:56:BB:8C:6A
status :
uid : 0
uuid :
VMware Integration Use Case

A VMWare ad
min
is
tra
tor in ACME re
quests the net
work team to trunk a set of VLANs
down to the ESX hosts to pro vide con
nectiv
ity to their DVS switches. Rather than
trunk
ing VLANs on a per server basis, the net work team de cides to leverage a new
methodol
ogy to be more agile and lever
age the on-de mand pro vi
sion
ing of re
sources
where and when they are needed, as well as pro
vid
ing un
lim
ited Layer 2 mo
bil
ity for all
the VM hosts within the ACI fab
ric.
To do so, the net

work ad
mins work with the VMware ad
mins to de
cide on a range of
VLANs that will be pro
vided dy
nam
ic
ally by APIC to the ESX hosts that need them. They
de
cide on an un
used VLAN range of (600 - 800). This is their dy
namic VLAN pool. Once
this is de
cided, the APIC ad min
is
tra
tor pro
ceeds to con fig
ure VMM in
tegra
tion in the
APIC GUI by pro vid
ing the vCen
ter cre
dentials to APIC. APIC dy
nam
ic
ally pro
vi
sions all
EPGs and makes them avail
able to the ESX hosts as a port-group.
Note: The APIC does not auto

mat
ic
ally move VM
NICs into the port-group. This al
lows
VMware ad mins to main
tain con
trol and move vir
tual NICs into these port-groups on
de
mand.
As the VMware admin pro

vi
sions ESX hosts and se
lects the ap
pro
pri
ate port-groups for
VMs, the APIC dynamic
ally communi
cates with vCen
ter to make EPGs avail
able through
port groups. The APIC also config
ures VLAN IDs on the leaf-switches as needed.
During a vMo
tion event, APIC is auto
matic
ally in
formed of the VM move and then up -
dates the end
point track
ing table to allow seam less commu ni
ca
tion. VMs are al
lowed to
move any where within the ACI fabric with no restric
tions other than those imposed by
vCenter.
It is im portant to note that ACME can still choose to de ploy tra

di
tional VLAN trunk ing
down to VMware DVS switches by sta t
i
cally pro
vi
sion
ing EPGs on a per-port basis, and
still reap the ad vantages of the Layer 2-any where ACI fabric. How
ever, ACME chose VMM
in
te gration as the pre ferred deployment model as it is the most ef fec
tive method of
break ing down or ganiza
tional chal
lenges, doing on-de mand resource al
lo
ca
tion, and get
-
ting en hanced visi
bil
ity and telemetry into both the vir
tual and phys
i
cal en
vi
ron
ments.
Deploying the Application Virtual Switch
Prerequisites
All switch nodes have been discovered by the fabric
INB or OOB management connectivity is configured.
VMware vCenter is installed, configured, and available.
One or more vSphere hosts are available for deployment to the AVS.
A DNS server policy has been configured to enable connection to a VMM using
a hostname.
A dynamic VLAN pool has been created with enough VLAN IDs to accommodate
one VLAN per EPG you plan on deploying to each VMM domain.
Getting Started
The AVS software was de signed to oper
ate in
dependently of the APIC soft
ware ver
sion.
This al
lows ei
ther device to be upgraded independently. Always refer to the AVS re-
lease notes to con
firm if any spe
cial con
sid
erat
ions may exist.
Just like any soft

ware, new ver sions of the AVS will be re leased to in
clude new features
and im provements. The ini tial AVS software released was ver sion 4.2.1, fol
lowed by
sion 5.2.1. Refer to the ACI Ecosys
ver tem Com pati
bil
ity List doc
ument to ensure your
de
sired version of AVS is compatib
le with the APIC and vSphere ver sions being run.
The AVS pack age for either ver

sion will in
clude vSphere In stal
la
tion Bundles (VIBs).
Each ver sion of AVS software in
cludes the VIB files for all sup
ported vSphere ver sions.
As of this pub li
ca
tion there are two VIBs to sup port vSphere ver sions 5.1 and 5.5
(vSphere 5.0 is not sup ported). These can be down loaded from CCO at the fol lowing
lo
ca
tion:
Downloads Home > Prod

ucts > Switches > Vir
tual Net
work
ing > Ap
pli
ca
tion Vir
tual
Switch
The VIBs can be iden

ti
fied as fol
lows:
AVS 4.2.1 Bundle
cross_cisco-vem-v165-4.2.1.2.2.3.0-3.1.1.vib
AVS 5.2.1 Bundle
Install the AVS VIB

Be
fore set
ting up the AVS con
fig
ur
a
tion on the APIC, the AVS soft
ware must be in
-
stalled in vSphere, re ferred to as the Virtual Eth
er
net Mod ule (VEM). This can be
achieved in a va ri cussed in Cisco Ap
ety of ways, all of which are dis pli
ca
tion Virtual
Switch Instal
la
tion Guide. For a few hosts, this can eas
ily be done manua
lly, but for 10+
hosts it may be easier to lever
age the Vir
tual Switch Up
date Man
ager (VSUM) to help
au
to
mate the in
stal
lation process.
Manual Installation
1 Copy the VIB file to a host. The easiest way to copy the VIB to the host is to
leverage the VMware VI Client, navigate to the Host > Configuration > Storage
> Datastore_X. Right click on the desired datastore and select Browse. From
here,VIB can be uploaded directly to the host's datastore.
2 SSH into the vSphere host on which the AVS VIB is to be installed.
3 Install or upgrade the VIB using the esxcli command:
To in
stall the AVS VIB:
esxcli software vib install -v /<path>/<vibname> --maintenance-mode --no-sig-check

To Up
grade an ex
ist
ing AVS VIB:
esxcli software vib update -v /<path>/<vibname> --maintenance-mode --no-sig-check
A sam
ple out
put is shown below:
# esxcli software vib install -v /vmfs/volumes/datastore1/cross_cisco-vem-v172-
5.2.1.3.1.3.0-3.2.1.vib --maintenance-mode --no-sig-check
Installation Result
Message: Operation finished successfully.
Reboot Required: false
VIBs Installed: Cisco_bootbank_cisco-vem-v172-5.2.1.3.1.3.0-3.2.1
VIBs Removed:
VIBs Skipped:
/vmfs/volumes/53cab6da-55209af3-0ef2-24e9b391de3e # vem version
Running esx version -1623387 x86_64
VEM Version: 5.2.1.3.1.3.0-3.2.1
VSM Version:
System Version: VMware ESXi 5.5.0 Releasebuild-1623387
4 Confirm the VEM is loaded and running.
# vem status
VEM modules are loaded
Switch Name Num Ports Used Ports Configured Ports MTU Uplinks
vSwitch0 3072 6 128 1500 VMNIC0
VEM Agent (vemdpa) is running
DHCP Relay
The first task is to cre
ate a DHCP relay pol
icy in the infra ten
ant on the APIC. This will
allow the AVS to cre
ate a vmk Vir
tual Tun
nel End
point (VTEP) in
ter
face on each host.
This VTEP in
ter
face will be used for the Open
Flex con
trol chan
nel and/or the VXLAN
tun
nel source be
tween the VTEP and ACI fabric.
If the de
fault TEP Ad
dress pool was used dur
ing ini
tial setup (10.
0.
0.
0/
16), this will
allow the AVS VTEP in
ter
faces to pull an ad
dress from this pool. The way APIC would
do it is through over the infra vlan. For this rea
son it is crit
ic
al to ex
tend the Infra
VLAN from a leaf through to each vSphere host that will host the AVS.
Create the Infra DHCP Relay Policy

Cre
ate a DHCP relay pol
icy that will setup the DHCP server (the APIC in this case) and a
label using the Infra Ten
ant and de
fault EPG. The APIC IP ad
dresses au
to
mat
ic
ally
tion as the DHCP providers and DO NOT need to be ex
func plic
itly added.
STEPS
2 In the Work pane, choose the infra.
3 In the Navigation pane, choose infra > Networking > Protocol Policies > DHCP
> Relay Policies.
4 In the Work pane, choose Actions > Create DHCP Relay Policy.
5 In the Create DHCP Relay Policy dialog box, perform the following actions:
a. Provide a name for the Relay policy, such as "avs-dhcp-relay-pol".
b. Leave other fields blank and click OK.
6 Click Submit.
Create the DHCP Relay Label
2 In the Work pane, choose the infra.
3 In the Navigation pane, choose infra > Networking > Bridge

Domains > default > DHCP Relay Labels.
4 In the Work pane, choose Actions > Create DHCP Relay Policy.
5 In the Create DHCP Relay Policy dialog box, perform the following actions:
a. Change the Scope to TENANT.
b. From the Name drop-down list, choose the DHCP Relay Policy that you
created previously.
6 Click Submit.
Attachable Access Entity Profile (AEP) and AVS

An im
por
tant com
po
nent used by the AVS is the At
tach
able En
tity Pro
file (AEP). Re
-
gard
less of using an ex
ist
ing AEP or cre ing a new one, the En
at able In
fra
structure
VLAN check box must be checked for the AEP pol icy. This is to en
sure that the traf
fic
of in
ter
est (DHCP re
quest/offer can flow through the in
fra
struc
ture VLAN to the AVS).
The AEP de
fines which VLANs will be per
mit
ted on a host fac
ing in
ter
face. This can be
com
pared to a "switch
port trunk al
lowed VLAN xxx" com
mand in tra
di
tional net
work
-
ing. Re
fer
ring back to the "VMM Pol
icy Model In
ter
ac
tion" di
ag
ram from the "VM Net
-
work
ing Overview" chap
ter, the AEP is what ties the VMM do
main to the phys
ic
al in
ter
-
faces where the vSphere hosts are con
nected. The AEP can be cre ated on-the-fly dur
-
ing the cre
ation of the VMM domain it
self, but this guide will de
tail cre
at
ing the AEP
sep
ar
ately first.
Create a New AEP
Profile.
3 In the Work pane, choose Actions > Create Attachable Access Entity Profile.
4 In the Create Attachable Access Entity Profile dialog box, perform the
following actions:
a. Fill in the AEP wizard information then click Next.
i. Name: Provide any name to identify the AEP, such as "AVS-AEP".
ii. Enable Infrastructure VLAN: Check this box.
iii. Domains (VMs or Baremetal): Leave blank. This will be covered later in the
Publishing EPGs to VMM Domains chapter.
b. From the next page of the wizard, select the Interface Policy Group your
AEP will be associated to. This procedure assumes your Interface Policy
Group has already been created. Click the All Interfaces radio button for the
desired Interface Policy Group.
Note: In
ter
face Pol icy Group cre ation is covered else where in this guide. Es sen
tially
the In
ter
face Policy Group is a col lec
tion of Inter
face policies which de fine In
ter
faces
Se
lec
tors and prop erties, such as speed/ne goti
at
ion, LLDP, and CDP. See the "Adding
New De vices to the Fab ric" chapter for more de tail on creat
ing the inter
face pol
icy
group and inter
face pro files.
Modify an Exisiting AEP
Profile.
a. In the Navigation pane, choose the existing AEP
b. In the Work pane, check the Enable Infrastructure VLAN check box.
Note: As men
tioned early in this chap
ter, the In
fra
struc
ture VLAN is re
quired for AVS
com
mu
ni
ca
tion to the fab
ric using the Open
Flex con
trol chan
nel.
VMM Domains for vCenter

A Vir
tual Machine Man ager (VMM) domain de fines a vir
tual in
frastruc
ture that will be
in
te
grated into ACI. This al
lows the same policies ap
plied to phys ic
al end
points, to also
be applied to vir
tual endpoints. vCen
ter VMM Do mains are cre ated using either the
VMware DVS or Cisco AVS. You can not change from one to the other. A new VMM Do -
main will be cre
ated from scratch to support AVS de ployment.
AVS Switching Modes

The AVS can op
er
ate in the fol
low
ing switch
ing modes:
Local Switching: Supports VXLAN encapsulation or VLAN encapsulation.

This switching mode allows Inter-EPG traffic to be switched locally
on the AVS.
No Local Switch only supports VLAN encapsulation.
This switching mode sends all traffic (Inter-EPG included) to the
Leaf switch.
AVS Switching Modes: Non-Local and Local switching mode
The de
ci
sion be
tween using VLAN or VXLAN en cap
sula
tion will man
date dif
fer
-
ent VLAN exten
sion re
quire
ments out
side of the fab
ric. When using VXLAN en -
capsula
tion, only the infra VLAN is required to be extended to the AVS hosts. All
traf
fic between the AVS up links and ACI fabric will be en
capsu
lated by VXLAN
and trans fered using the infrastruc
ture VLAN.
If VLAN encapsu
la
tion is pre
ferred, you will need to ensure every VLAN in the
VM Do main VLAN pool has been ex tended be tween the fabric and AVS host.
This in
cludes creat
ing the VLANs on inter
me di
ate de
vices such as UCS and the
vNICs for any AVS vSphere hosts.
Create the VMM Domain for AVS

Now that the DHCP server pol
icy has been cre
ated and AEP cre
ated/mod
if
ied, you can
cre
ate the VMM do
main for the AVS.
1 On the menu bar, choose VM NETWORKING.
2 In the Navigation pane, choose the Policies tab.
3 In the Work pane, choose Actions > Create VCenter Domain.

4 In the Create vCenter Domain dialog box, perform the following actions:
a. Name: This value will be used as the AVS "Switchname" displayed in vCenter.
b. Virtual Switch: Cisco AVS
c. Switching Preference: <Choose Local or No Local Switching>
For No Local Switching mode:
Multicast Address: <Assign a multicast address to represent your AVS>
Multicast Address Pool: <Create a unique Multicast Address Pool large
enough to include each AVS vSphere host.>
For Local Switching mode:
Encapsulation: <Choose VLAN or VXLAN based on preference>
For VLAN Encapsulation
VLAN Pool: <Choose/Create a VLAN pool>
For VXLAN Encapsulation
Multicast Address: <Assign a multicast address to represent your
AVS>
Multicast Address Pool: <Create a unique Multicast Address Pool
large enough to include each AVS vSphere host.>
d. Attachable Access Entity Profile: <Choose the AEP previously
created/modified>
e. vCenter Credentials: Create a credential set with administrator/root acces
to vCenter
f. vCenter: Add the vCenter details.
Name: Friendly name for this vCenter
Hostname/IP Address: <DNS or IP Address of vCenter>
DVS Version: vCenter Default
Datacenter: <Enter the exact Datacenter name displayed in vCenter>
Management EPG: <Set to oob or inb Management EPG>
Associated Credentials: <Choose the Credential set previously created>
Click OK to complete the creation of the vCenter.
5 Click Submit.
Verify AVS Deployment on vCenter
1 In the vCenter client, navigate to HOME > INVENTORY > NETWORKING and
confirm a new Distributed Virtual Switch folder has been created.
2 Expand this folder to find your AVS, and a few default Port Groups including
"uplink" and "vtep".
Add vSphere Hosts to the AVS

After the AVS has been cre
ated in vCen
ter, you then need to at
tach hosts to it. To do
this you will need at least one un
used phys
ic
al in
ter
face (VMNIC) to act as the up
link on
each host. AVS up
links can not be shared with any other ex
ist
ing vSwitch or vDS.
1 From the vCenter client, navigate to HOME > INVENTORY > NETWORKING.
2 Right click on the newly created AVS switch (not the folder) and choose Add
Host....
3 In the Add Host dialog box, choose any vSphere hosts to add to the AVS, and
select an unassigned VMNIC uplink.
4 Click Next until the wizard completes, skipping the migration of any virtual
adapters or virtual machine networking at this time.
Note: For blade switch sys tems such as UCS, the VMNIC in ter
face used must have all
neces
sary VLANs al lowed on the in ter
face. In UCS terms, this re
quires the vNIC within
the ser
vice pro
file to have all rel
e
vant VLANs ac tive on the vNICs.
Adding Virtual host physical NICs to participate in Virtual Switch
5 Assuming the ACI fabric can reach the vSphere host over the infra VLAN, you
should see a new vmk interface created on your distributed switch within
vCenter and assigned to the 'vtep' port group. This vmk is your Virtual Tunnel
Endpoint (vtep) interface and should have pulled a DHCP address from the APIC
from the TEP subnet. As can be seen from the screenshot below, we see that
the VMkernel port has received the IP address from the APIC. The APIC uses
the same 10.0.0.0/16 pool that is created during the APIC setup to provision the
IP address. This implies that we are ready for Opflex communication in
between the AVS and the APIC.
~ # esxcfg-VMKNIC -l
Interface Port Group/DVPort IP Family IP Address
Netmask Broadcast MAC Address MTU TSO MSS Enabled Type
vmk0 Management Network IPv4 172.16.176.54
255.255.255.0 172.16.176.255 00:25:b5:00:00:29 1500 65535 true STATIC
vmk1 vmotion IPv4 192.168.99.54
255.255.255.0 192.168.99.255 00:50:56:61:1c:92 1500 65535 true STATIC
vmk2 9 IPv4 10.0.16.95
255.255.0.0 10.0.255.255 00:50:56:65:3d:b3 1500 65535 true DHCP

Verify AVS on ESX

On the ESX com cmd show open
mand line, issue the vem flex com
mand.
Ver
ify that the sta
tus: 12 (Ac
tive) is seen as well as the switch
ing mode. Also ver
ify that
the GIPO ad dress is the same as the mul ti
cast ad
dress that was used while cre
at
ing the
VMM do main.
~ # vemcmd show openflex

Status: 12 (Active)
Dvs name: comp/prov-VMware/ctrlr-[AVS-TEST]-VC/sw-dvs-87
Remote IP: 10.0.0.30 Port: 8000
Infra vlan: 4093
FTEP IP: 10.0.0.32
Switching Mode: LS
NS GIPO: 225.127.1.1
Ver
ify on the AVS host - there should be one mul ti
cast group per de
ployed EPG on the
host. In the out
put below, there are three dif
fer
ent Vir
tual Ma
chines connected to dif
-
fer
ent EPGs.
~ # vemcmd show epp multicast

Number of Group Additions 3
Number of Group Deletions 0
Multicast Address EPP Ref Count
225.0.0.58 1
225.0.0.76 1
225.0.0.92 1
These multi
cast ad
dresses will corre
spond to the EPG de tails found in the APIC GUI
under Tenants > Ten antX > Ap pli
ca
tion Pro
files > Ap
pli
cation fileX > End Point
Pro
Groups > EndPointGroupX and click the Oper
a
tional Tab > Client End Points.
VXLAN Load Balancing

VXLAN load balanc
ing is au
to
mat
ic
ally en
abled as soon as more than one VMKNIC is
con
nected to the Cisco AVS. Each VMKNIC can use only one up link port; you should
have an equal num
ber of VMKNICs and up
links. A max
im
um of eight VMKNICs can be
at
tached to a Cisco AVS switch. Each of the VMKNICs that you cre ate has its own soft-
ware-based MAC ad dress. In VXLAN load bal anc
ing, the VMKNICs pro vide a unique
MAC address to packets of data that can then be di
rected to use cer
tain phys ic
al NICs
(VM
NICs).
You need to have as many VMKNICs as the host has VM NICs, up to a maxim
um of eight.
For exam
ple, if the host has five VM NICs, you need to add four VMKNICs to en able
VXLAN load-bal anc
ing; the Cisco Ap
pli
ca
tion Pol
icy In
fra
structure Controller (APIC) al
-
ready cre
ated one VMKNIC when the host was added to the dis trib
uted vir
tual switch
(DVS).
In VMware vSphere Client, you will need to cre

ate an addi
tional vir
tual adapter (VMK)
for each AVS up
link. Each vmk in
ter
face cre
ated for the AVS should be at tached to the
vtep port group and con fig
ured for DHCP. In the screen shot below you can see the
four VMNIC up links to the AVS and the four vmk vir
tual in
ter
faces to pro
vide equal load
bal
anc
ing traf
fic across all uplinks. Note that each VMK in ter
face added is as
signed a
unique DHCP ad dress from the fabric TEP pool.
Distributed Switch configured with APIC port groups

IGMP Snooping Policy for AVS

Cisco UCS-B Series Considerations with AVS Deployments
This sec
tion of the ar
ti
cle will focus the nec
es
sary steps to en
able AVS through the
Cisco UCS-B series.
By de
fault, USC-B FI has IGMP snooping en
abled. Due to this, we have to con
fig
ure an
IGMP querier pol
icy on the APIC. The IGMP snooping pol
icy needs to be en
abled on the
infra ten
ant.
If we dis
able IGMP snoop
ing on UCS or other in
ter
me
di
ate blade switches, then IGMP
pol
icy is not needed since the blade switch will flood the mul
ti
cast traf
fic on all the rel
-
e
vant ports.
Create an IGMP Snooping Policy for AVS
2 In the Work pane, choose infra.
3 In the Navigation pane, choose infr > Networking > Bridge Domain > default.
a. In the IGMP Snoop Policy drop-down list, choose Create IGMP Snooping
Policy.
b. Provide a name for the policy, such as "IGMP_Infra".
c. Click the Fast Leave check box.
d. Click the Enable Querier check box.
4 Click Submit.
Note: Verify if IGMP snooping is working properly on the vSphere host CLI
using 'vemcmd show epp multicast' as shown above.
The al
ter
nate method would be to cre ate an IGMP pol icy on UCS to disable IGMP
snoop
ing. This will cause flood
ing of the mul
ti
cast traf
fic to all end
points.
External Connectivity

As men
tioned in the in
tro
duc
tion of this book, ACME Inc. is a multi
na
tional com
pany
with mul
ti
ple data cen
ters. There
fore, ACME Inc. must con
fig
ure some Layer 2 con
nec
-
tiv
ity. This is nec
es
sary for ex
tend
ing Layer 2 con
nec
tiv
ity to a Data Cen
ter In
ter
con
-
nect (DCI) plat
form, to fur
ther extend connectiv
ity to a re
mote data cen ter, or sim
ply
to ex
tend a Layer 2 domain outside of the fab
ric to con nect in an ex
ist
ing Layer 2 net
-
work in a non-ACI fab
ric.
Extending Endpoint Groups Outside the ACI Fabric

The sim plest way to extend an end point group (EPG) out side of the ACI fab ric is to sta
t-
i
cally as
sign a leaf port and VLAN ID to an ex ist
ing endpoint group. After doing so, all of
the traffic re
ceived on this leaf port with the con fig
ured VLAN ID will be mapped to the
EPG, and as such, the con fig
ured pol
icy for this EPG will be enforced. The end points do
not need to be di rectly con nected to the ACI leaf, as the traf fic clas
si
fi
ca
tion will be
based on the en capsu
lation re
ceived on a port.
To as
sign a Layer 2 con
nec
tion sta
ti
c
ally on an ACI leaf port to an EPG:
3 In the Navigation pane, choose Tenant_Name > Application Profiles >

App_Profile_Name > Application EPGs > EPG_Name > Static Bindings
(Paths).
4 In the Work pane, choose Action > Deploy Static EPG on PC, vPC or Interface.
5 In the Deploy Static EPG on PC, vPC or Interface dialog box, perform the
following actions:
a. In the Path field, specify a port as well as a VLAN ID.
b. Click one of the Deployment Immediacy radio buttons. Deployment
immediacy determines when the actual configuration will be applied on the
leaf switch hardware. The immediacy also determines when the hardware
resource, such as a VLAN resource and policy content-addressable memory
(CAM) to support the related contract for this EPG, will be consumed on the
leaf switch. The option Immediate means that the EPG configuration and its
related policy configuration will be programmed in the hardware right away.
The option On Demand instructs the leaf switch to program the EPG and its
related policy in the hardware only when traffic matching this policy is
received for this EPG.
c. Click one of the Mode radio buttons. The mode option specifies whether the
ACI leaf expects incoming traffic to be tagged with a VLAN ID or not.
i. Tagged - The tagged option means that the leaf node expects incoming
traffic to be tagged with the specified VLAN ID previously established. This
is the default deployment mode. Choose this mode if the traffic from the
host is tagged with a VLAN ID. Multiple EPGs can be statically bound to
the same interface as long as the encap VLAN/VXLAN ID is unique.
ii. Untagged - The untagged option means that the leaf expects untagged
traffic without a VLAN ID. Similar to the switchport access
vlan vlan_ID command, with this option you can only assign the interface
to one EPG. This option can be used to connect a leaf port to a bare metal
server whose network interface cards (NICs) typically generate untagged
traffic. A port can have only one EPG statically bound to a port as
untagged.
iii. 802.1P - The 802.1P option refers to traffic tagged with 802.1P headers.
802.1P mode is useful when its necessary to handle the traffic on one EPG
as untagged to the interface (similar to the switchport trunk native vlan
vlan_ID command), but (unlike the untagged mode) 802.1P will allow other
'tagged' EPGs to be statically bound to the same interface. Any traffic
received on links with this mode classification will have the following
conditions applied to them:
d. Create a physical domain and VLAN pool that are associated to this physical
domain.
e. Associate the physical domain to the EPG in question.
f. Create an attachable access entity profile (AEP) to map the interfaces and
policies together.
See the Adding New De vices to the Fab

ric sec
tion for more in
for
ma
tion on how to con
-
fig
ure an AEP and a phys
ic
al do
main.
Extending an ACI Bridge Domain Outside of the Fabric

A Layer 2 out
side con
nec
tion is as
so
ci
ated with a bridge do
main and is de
signed to ex
-
tend the whole bridge do
main, not just an in
di
vid
ual EPG under the bridge do
main, to
the out
side net
work.
To ac
com
plish an ex
ten
sion of the bridge do
main to the out
side, a Layer 2 out
side con
-
nection must be cre ated for the bridge do main. During this process, create a new ex-
ter
nal EPG to clas sify this exter
nal traf
fic. This new EPG will be part of the ex ist
ing
bridge do main. Clas sify any outside connections or endpoints into this new ex ter
nal
EPG. With two sep arate EPGs, you will also need to select which traf
fic you would like
to traverse be
tween the two EPGs. Sim il
ar to the pre
vi
ous example of adding an end -
point to a pre-exist
ing EPG, this method will also allow the end points to share the same
subnet and default gate way.
To cre
ate an ex
ter
nal Layer 2 do
main:
3 In the Navigation pane, choose Tenant_Name > Networking > External

Bridged Network.
4 In the Work pane, choose Action > Create Bridged Outside.
5 In the Create Bridged Outside dialog box, perform the following actions:
a. Associate the Layer 2 outside connection with the bridge domain and a VLAN
ID. This VLAN must be configured on the external Layer 2 network. This
Layer 2 outside connection will put this VLAN and the bridge domain of the
ACI fabric under the same Layer 2 domain. This VLAN ID must be in the range
of the VLAN pool that is used for the Layer 2 outside connection.
i. For the External Bridged Domain drop-down list, create a Layer 2 domain
if one does not already exist.
ii. While creating the Layer 2 domain, if it does not already exist, create a
VLAN pool to associate to the VLAN on the Layer 2 outside
connection. This is a means to specify the range of the VLAN IDs that will
be used for creating a Layer 2 outside connection. This helps avoid any
overlapping in the VLAN range between VLANs used for an EPG and those
in use for a Layer 2 outside connection.
b. Add a Layer 2 border leaf node and Layer 2 interface for a Layer 2 outside
connection.
c. After adding a Layer 2 border leaf and Layer 2 interface, click Next to start
creating a Layer 2 EPG. Simply provide a name for the Layer 2 EPG. All of the
traffic entering the ACI fabric with the designated VLAN (the VLAN ID
provided in step 1) will be classified into this Layer 2 EPG.
d. Configure a contract to allow communication between your existing
endpoints in the existing EPG and your new external Layer 2 EPG. In the
Navigation pane, choose External Bridged Networks > Networks and specify
a contract to govern this policy as the consumed contract. After specifying
this contract as the provided contract for your internal EPG, the
communication between this external Layer 2 EPG and your existing internal
EPG will be allowed.
e. Create an AEP. This is a policy object that tells the APIC to allow certain
encap (VLANs) on selected ports. For more information on how to create a
Access Attachable Entity Profile, see the Adding New Devices to the
Fabric section.
You should now have the de

sired reach
ab
il
ity be
tween the in
side and out
side
Layer 2 seg
ments.

The most im portant com ponent of any ap pli
ca
tion is the user or cus tomer, which gen -
er
ally does not di rectly at
tach to the fab ric, and there fore there must be con nected to
the external net
work. ACME must be able to con nect to both their in ter
nal cor
porate
backbone, as well as to the In ter
net to pro vide ac cess to the mo bile ap
pli
ca
tion. This
in
te
gration is possi
ble with Cisco Ap pli
cation Cen tric Infra
struc
ture (ACI) at the tenant
pol
icy level. Layer 3 con nectiv
ity to a de vice like a router is known as an Ex ternal
Routed Net work, and pro vides IP connec tiv
ity between a ten ant private net
work and
an exter
nal IP network. Each Layer 3 ex ternal connection is asso
ci
ated with one ten ant
pri
vate net
work. The require
ment of the Layer 3 ex ter
nal net
work is only needed when
a group of de
vices in the appli
ca
tion pro
file re
quire Layer 3 con
nectiv
ity to a net
work
out
side of the ACI fab
ric.
An ap
pli
ca
tion pro
file en
ables an op
er
at
or to group the dif
fer
ent com
po
nents, or tiers,
of an ap
pli
ca
tion into endpoint groups (EPGs). These ap pli
cation com po
nents might
have re
quire
ments for exter
nal con
nec
tiv
ity into them. The fol
lowing fig
ure shows part
of a sim
pli
fied fab
ric:
A sample application profile for a three-tiered application with contracts between

the tiers
For ex
ample, web servers need a con nection to the outside world for users to reach
them. With ACI, the con nectiv
ity is de
fined by a contract to a defined exter
nal Layer 3
end
point group. As the op er
ator of the fab
ric, you can provide the ten ant ad
minis
tra
tor
with the abil
ity to in
ter
face to an ex ter
nal Layer 3 connection in vario
us ways by using
a uniquely-defined Layer 3 con
struct for the ten
ant ap
pli
ca
tion pro
file or a shared
common infra
struc
ture.
Ex
ter
nal Layer 3 con
nec
tions are usu
ally es
tab
lished on the bor
der leaf con
struct of the
ACI. Any ACI leaf can become a border leaf. In large scale ACI de
signs it might be pro
-
duc
tive to have ded ic
ated ACI leafs as border leafs. It is im
por
tant to note that the
spine nodes cannot have con nec
tions to ex
ter
nal routers. A bor
der leaf is simply ter
mi
-
nol
ogy to refer to a leaf that hap
pens to be con nected to a Layer 3 de vice. Other de-
vices, like servers, can still con
nect to the bor
der leaves. In the ACI fab
ric, the ex
ter
nal
Layer 3 con nection can be one of the fol
lowing types:
1 Physical Layer 3 interface
2 Subinterface with 8021.Q tagging
3 Switched Virtual Interface (SVI)

Bridge domains con tain one or more sub

nets. These sub
nets can be clas
si
fied as pri
-
vate, pub
lic, or shared:
Public - Indicates that this subnet will be advertised to the external router.
Private - Indicates that this subnet will be contained only within the ACI fabric
private network.
Shared - Indicates that this subnet will be leaked to one or more private
networks inside of the ACI fabric.
The fol
low
ing fig
ure de
picts the logic of pub
lic and pri
vate net
works:
Application profile with external consumers, public and private networks annotated
With de
vices connecting through the external Layer 3 con nec tion, the exter
nal net
-
work has learned of the in
ter
nal ACI net
work 10.1.
1.
0/24, as it is adver
tised to the ad
ja
-
cent router through the Layer 3 ex ter
nal connec tion. For the pri vate networks, ACI
does not ad
ver
tise the networks through the rout ing protocol to the ad ja
cent Layer 3
router, and the net
works are not reach
able to de
vices ex
ter
nal to the fab
ric.
In re
leases prior to ver
sion 1.1 of Cisco Appli
ca
tion Pol
icy In
frastruc
ture Controller
(APIC), the fab
ric only ad
ver
tises subnets that are marked pub lic in the as
so
ci
ated
bridge do
main. Routes that are learned exter
nally from the fabric are not adver
tised
through other ports. This be
hav
ior is known as a non-tran
sit fab
ric. In re
lease ver
sion
1.1 and later, ACI is ca
pa
ble of act
ing as a tran
sit net
work, and routes learned from one
exter
nal Layer 3 con nec
tion can be ad ver
tised out to a dif
fer
ent ex
ter
nal Layer 3 con
-
nec
tion, not just fab
ric routes.
The net
work team will provide the ex ter
nal Layer 3 connectiv
ity for their ten
ants. One
common mech an
ism is to use sub-in ter
faces on a router to create dif
fer
ent Layer 3 do
-
mains since each ten
ant will likely not have their own exter
nal router.
Supported Routing Protocols

The fol
low
ing rout
ing pro
to
cols are sup
ported at time of writ
ing:
Static routesYou can define static routes to the outside world. Using static
routes reduces the sizing and complexity of the routing tables in the leaf nodes,
but increases administrator overhead. With static routes, you must also
configure the static path back to the interanal network that you wish to be
reachable from the outside world.
OSPF NSSAUsing not-so-stubby area (NSSA) reduces the size of the Open
Shortest Path First (OSPF) database and the need to maintain the overhead of
the routing protocols with large tables of routes. With OSPF NSSA, the router
learns only a summarization of routes, including a default path out of the fabric.
OSPF NSSA advertises to the adjacent router the internal public subnets part of
the Layer 3 external.
iBGP peering leaf and external routerWith internal Border Gateway Protocol
(iBGP), ACI supports only one autonomous system (AS) number that has to
match the one that is used for the internal Multiprotocol Border Gateway
Protocol (MP-BGP) route reflector. Without MP-BGP, the external routes
(static, OSPF, or BGP) for the Layer 3 outside connections are not propagated
within the ACI fabric, and the ACI leaves that are not part of the border leaf
does not have IP connectivity to any outside networks. Given that the same AS
number is used for both cases, the user must find out the AS number on the
router to which the ACI border leaf will connect, and use that AS number as the
BGP AS number for the ACI fabric.
Configure MP-BGP Spine Route Reflectors

The ACI fab ric route re flec
tors use multi
pro to
col border gateway pro to
col (MP-BGP) to
dis
trib
ute external routes within the fab ric so a full mesh BGP topol ogy is not required.
To enable route re flec
tors in the ACI fab ric, the fab
ric adminis
tra
tor must select at least
one spine switch that will be a route re flector, and pro vide the au
tonomous sys tem (AS)
num ber for the fab ric. Once route re flectors are con fig
ured, adminis
tra
tors can setup
connectiv
ity to ex
ternal networks.
To connect ex ter
nal Layer 3 devices to the ACI fab
ric, the fab
ric in
fra
struc
ture op
er
at
or
must con fig
ure a Route Reflec
tor policy to des
ig
nate which spines act as the route re -
flec
tor(s). For re
dundancy pur poses, con fig
ure more than one spine as a router re flec
-
tor node.
When a ten
ant re
quires a Layer 3 con
nec
tion, the in
fra
struc
ture op
er
at
or con
fig
ures
the leaf node to which the WAN router is being con
nected as bor
der leaf node, which
pairs the bor
der leaf node with one of the route re
flec
tor nodes as a BGP peer. After
the route re
flec
tors are con
fig
ured, they can ad
ver
tise routes in the fab
ric.
Each leaf node can store up to 4000 routes at time of writ

ing. If a WAN router must ad
-
ver
tise more than 4000 routes, the router should peer with mul
ti
ple leaf nodes. The in
-
fra
structure oper
at
or con
fig
ures each of the paired leaf nodes with the routes (or route
prefixes) that the nodes can ad
vertise.
To con
fig
ure the Route Re
flec
tor pol
icy:
2 In the Navigation pane, choose Pod Policies > Policies > BGP Route
Deflector default.
3 In the Work pane, perform the following actions:

a. Change the Autonomous System Number to match the required number for
your network.
b. Add the two spine nodes that will be members of this reflector policy and
c. Click Submit.
4 In the Navigation pane, choose Pod Policies.
5 In the Work pane, choose Actions > Create Pod Policy Group.
6 In the Create Pod Policy Group dialog box, perform the following actions:
a. In the BGP Route Reflector Policy drop-down list, choose default.
b. In the Navigation pane, choose Pod Policies > Profiles > default.
c. In the Work pane, in the Fabric Policy Group drop-down list, choose Create
Pod Policy Group.
d. In the Create Pod Policy Group dialog box, in the Date Time Policy drop-
down list, choose default.
e. In the BGP Route Reflector Policy drop-down list, choose default.
f. Complete the remainder of the dialog box as appropriate to your setup.
7 Click Submit.
The fol
low
ing fig
ure il
lus
trates the ob
jects and their re
la
tion
ships for ex
ter
nal Layer 3
con
nec
tions:
Object relationships for Layer 3 outside objects
Layer 3 Integration Through Tenant Network with OSPF NSSA

The fol
low
ing fig
ure shows a sim
ple in
te
gra
tion of a Layer 3 ex
ter
nal into ACI using
OSPF:
Logical topology for an external OSPF router communicating with two border leafs
The setup in
cludes a sin
gle router with two in
ter
faces con
nected to leaf switches.
Note: See the "Adding New De vices to The Fab

ric" sec
tion to setup the ac
cess poli
cies
for the in
ter
faces of the leaves that are con
nected to the router.
To in
te
grate Layer 3 through a ten
ant net
work with OSPF/NSSA:
2 In the Work pane, choose the tenant.
3 In the Navigation pane, choose Tenant_Name > Networking > External Routed
Networks.
4 In the Work pane, choose Action > Create Routed Outside.
5 In the Create Routed Outside dialog box, perform the following actions:
a. In the Name field, enter a name for the profile.
b. In the Private Network drop-down list, choose the private network for this
tenant.
c. Click the OSPF check box.
d. In the OSPF Area ID field, enter the OSPF area ID, such as "1".
e. In the OSPF Area Control section, click the Send redistributed LSAs into
NSSA area check box.
f. In the OSPF Area Type section, click the NSSA Area radio button.
g. In the Nodes and Interfaces Protocol Profiles section, click + to add a profile.
h. In the Create Node Profile dialog box, perform the following actions:
i. In the Name field, enter a name for the profile.
ii. In the Nodes section, click + to add a node.
iii. In the Select Node dialog box, perform the following actions:
1. In the Node ID drop-down list, choose a node, such as Leaf-1.
2. In the Router ID field, enter the router's IP address as the ID, such
as "10.0.1.1".
3. Uncheck the Router ID as Loopback Address check box.
4. In the Loopback Addresses section, click + to add a loopback address.
5. Enter the loopback address, such as "10.254.254.1", and click Update.
6. Click OK.
iv. In the OSPF Interface Profiles section, click + to create an OSPF interface
profile.
v. In the Create Interface Profile dialog box, perform the following actions:
1. In the Name field, enter a name for the profile.
2. In the OSPF Policy drop-down list, choose Create OSPF Interface
Policy. When defining the interaction with another OSPF router, you
must specify the policy interaction. This document does not explain the
different OSPF parameters.
3. In the Create OSPF Interface Policy dialog box, perform the following
actions:
a. In the Name field, enter a name for the OSPF policy, such as "OSPF-
Point2Point".
b. In the Network Type section, click the radio button that matches the
adjacent router, such as Point to Point.
c. Complete the remainder of the dialog box as appropriate to your
setup.
d. Click Submit.
4. In the Interfaces section, click on the Routed Intefaces tab.

5. Click the + sign to select a routed interface.
6. In the Select Routed Interface dialog box, perform the following actions:
a. In the Path drop-down list, choose the interface on the leaf, such as
e1/9 on Leaf-1.
b. In the IP Address field, enter the IP address of the path that is
attached to the layer 3 outside profile, such as "10.0.1.1/24".
c. In the MTU (bytes) field, enter the maximum MTU of the external
network, such as "1500" to match the example peering router.
d. Complete the remainder of the dialog box as appropriate to your
setup and click OK.
7. Click OK.
vi. Click OK.
i. Click Next.
j. In the External EPG Networks section, click + create an external network.
k. In the Create External Network dialog box, perform the following actions:
i. In the IP Address field, enter 0.0.0.0/0 to permit the learning of any

subnet and click OK.
l. Click Finish.
Next, you must configure the external network EPG.
External Layer 3 for Multiple Tenants

In ACI, you can use var
io
us mech
an
isms to reuse the same ex
ter
nal Layer 3 router for
multi
ple tenants. If the ad
ja
cent router is a Cisco Nexus Series Switch with a Layer 2
trunk inter
face, the exter
nal Layer 3 con
nection can be con
fig
ured to route via SVI. For
routers ca
pa
ble of using sub-in
ter
faces, those can be used to pro
vide mul
ti
ple ex
ter
nal
Layer 3 con nection for multi
ple VRFs and/or ten ants. The fabric oper
at
or can con
fig
-
ure multi
ple ex
ternal Layer 3 connec
tions using ei
ther sub-interface or SVI and pro
vide
that to each tenant.
Application Migration Use Case
When operat
ing the ACI fab
ric, there can be oc
ca
sions when you will have to mi grate
work
loads, servers or vir
tu
al
iza
tion hosts from out
side the ACI fab
ric, onto the fab
ric.
One com
mon ex
am
ple is when mi
grat
ing from a tra
di
tional data cen
ter con
fig
ur
a
tion
over to a pol
icy-dri ven data center using ACI. As ACME starts to use ACI in more of
their data cen
ters, it will be come neces
sary to per
form these mi
gra
tions. In this ex
am
-
ple, ACME must man
age the mi
gra
tion of SVI in
ter
faces as well as the pol
icy al
low
ing
traf
fic to tra
verse a Layer 2 out
side net
work and then on to the ACI fab
ric.
At a high level, you must start by config

ur
ing the Layer 2 out side net
work to allow traf-
fic from the source VLAN to com muni
cate with the same VLAN re sid
ing on the ACI fab-
ric. You will also need to config
ure Layer 3 con nec
tiv
ity from the fab ric out to the ex
-
ist
ing Layer 3 networks in prepara
tion for full con
nectiv
ity after SVI migration.
Further in
for
mation and steps on how to cre ate this Layer 2 and Layer 3 con nec
tiv
-
ity in
clud
ing the pol
icy, can be found in the Fab
ric Con
nectiv
ity chap
ter of this book.
Once they have suc cessfuly estab

lished connectiv
ity be
tween the out side Layer 2 net-
work (where the work load or host is com ing from) and the ex ist
ing internal fab
ric EPG,
you can then start the mi gration process of mov ing appli
ca
tion work loads onto the fab -
ric. One key con sid
er
at
ion should be when to switch over the SVI in terfaces from the
exist
ing en
vi
ronment into the ACI fab ric and when to start ad ver
tising routes to this
SVI network. Assuming that the SVIs reside on the ex ter
nal Layer 2 net work, Cisco rec-
om mends that you move the SVIs over to the ACI fab ric once a ma jority of the hosts
have been mi grated over.
Extending the Network to ACI

One of the ACME sites would like to mi grate from the legacy data cen ter ar
chi
tecture
to the next gener
at
ion ACI Fabric. They would like to mi grate with min im
al ser
vice in
-
terrup
tion while tak
ing ad
vantage of ACI in nova
tions where applic
ab
le. ACME would
like to per
form the mi
gra
tion in mul ti
ple stages.
The Legacy data cen

ter prior to mi
gra
tion:
Traditional pre-migration data center
The ACI data cen

ter fol
low
ing mi
gra
tion:
Post-migration ACI based data center topology

The first stage will pro

vide con
nec
tiv
ity from the legacy data cen
ter to the ACI fab
ric.
In this state, you log
ic
ally map a VLAN=EPG. The in
ter
con
nect from the legacy net
work
to the ACI fab
ric will be ac
com
plished through stan
dard Layer 2 ex
ten
sions
(VLAN/VXLAN).
Provide physic
al connectiv
ity from the exist
ing ag
grega
tion layer to the ACI border
leafs. This con
nectiv
ity can be accomplished in ei
ther the form of a Vir
tual Port Chan
-
nel, Port Chan
nel, or a sin
gle in
ter
face.
1 Provide a physical connection from aggregation switch #1 to the ACI border
leaf #1.
2 Provide a physical connection from aggregation switch #2 to the ACI border

leaf #1.
Note: Before con necting exter

nal phys
ic
al con
nections into the fab
ric, the Fab
ric Ac-
cess Poli
cies for the access ports that you will be used for the DCI must be con fig
ured.
For de
tails on con fig
uring the ac
cess poli
cies please ref
er
ence the Fab ric Con
nectivty
sec
tion of this book.
Con
fig
ure the ag
gre
ga
tion links as a Layer 2 trunk.
1 Trunk the VLAN representing the host connectivity. This allows for the host
VLAN to be extended into the fabric.
In the APIC con troller you will now config

ure a single ten
ant. The cre
ated ten
ant will
repre
sent the legacy data cen
ter into the ACI fab
ric.
1 On the menu bar, choose Tenants > Add Tenant.
2 In the Create Tenant dialog box perform the following actions:

a. In the Name field enter a name for the tenant.
b. Click Next.
3 Click Finish.
Con
fig
ure a sin
gle pri
vate net
work.

3 In the Navigation pane choose Tenant_Name > Networking > Private

Networks.
4 In the Work pane, choose Actions > Create Private Network.
5 In the Create Private Network dialog box, perform the following actions:
a. In the Name field enter a name for the private network.
b. Click Next.
c. In the Name field enter a name for the bridge domain.
d. In the Forwarding drop-down list, choose Custom.
e. For the Layer 2 Unknown Unicast radio buttons, click Flood.
f. For the Multi Destination Flooding radio buttons, click Flood in BD.
g. Click the ARP Flooing check box.
6 Click Finish.
Note: The con cept of flooding the unknown uni cast and arp within the Fab ric is to
allow for the Layer 2 seman tics from the legacy data cen
ter to be ex
tended into the ACI
Fab ric. When a host in the legacy data cen ter sends an ARP request and/or floods an
unknown uni cast frame, the bridge domain will then mimic the behav
ior in the ACI Fab-
ric. By default BPDU frames are flooded within EPG.
Con
fig
ure a sin
gle ap
pli
ca
tion pro
file:
1 On the menu bar, choose Tenants > ALL TENANTS
3 In the Navigation pane choose Application Profiles.
4 In the Work pane, choose Actions > Create Application Profile.
5 In the Create Application Profile dialog box, perform the following actions:
a. In the Name field enter a name for the Application Profile.
b. Click Submit.
Con
fig
ure a sin
gle end
point group:
3 In the Navigation pane choose Tenant_Name > Application Profiles >

Application_Profile_Name.
4 In the Work pane, choose Actions > Create Application EPG.
5 In the Create Application EPG dialog box, perform the following actions:
a. In the Name field, enter a name for the endpoint group.
b. In the Bridge Domain field, choose the appropiate bridge domain.
c. Click Finish.
Note: The EPG within the fab ric will map to a sin
gle VLAN in the legacy data cen ter.
The use of a sin
gle VLAN per EPG will provide a path for a net
work-cen
tric mi
gra
tion
min
im
iz
ing im
pact while in
tro
duc
ing fab
ric in
no
va
tions.
Con
fig
ure a VPC for the con
nec
tiv
ity to the legacy data cen
ter. See the "Fab
ric Con
nec
-
tiv
ity" sec
tion. Then, con
fig
ure a sta
tic trunk bind
ing using the VPC under the EPG,
Acme
Out
Side. The en
cap
su
la
tion VLAN should match the de
fined legacy data cen
ter
VLAN de
f
in
i
t
ion.
3 In the Navigation pane choose Tenant_Name > Application

Profiles > Application_Profile_Name > Application EPGs > EPG_Name
> Static Bindings (Paths).
4 In the Work pane, choose Actions > Deploy Static EPG on PC, VPC or
Interface.
5 In the Deploy Static EPG on PC, VPC or Interface dialog box, perform the
following actions:
a. Choose the Path Type.
b. Choose the Path.
c. Enter the encapsulation VLAN.
d. Click Submit.
Fol
lowing the Stage 1 mi gra
tion, the Legacy host VLAN is now ex tended to the ACI fab-
ric and all hosts from the Fab ric point of view are in the EPG, Acme OutSide. From the
ACI Fabric per
spective, the local VLAN num ber is not sig
nif
ic
ant as it will be mapped to
the tagged VLAN on the VPC to ward the Legacy data cen ter. This is what is known as
nor
mal
iza
tion where ACI uses the tagged vlan to map ex
ter
nal Layer 2 con
nec
tions into
ACI End Point Groups.
The connectiv
ity noted below rep
re
sents a BD=EPG=VLAN. The Layer 3 gate way for the
ACI fab
ric and the legacy data cen
ter or pro
vided by the Legacy data cen
ter.
An EPG as a VLAN datacenter deployment
The second stage of the mi gration takes advantage of the fabric pol
icy model to de
fine
fur
ther the appli
ca
tion mod el
ing. During this stage, multi
ple EPGs will be cre
ated with
APIC con tracts that define reachab
il
ity. The fol
lowing list is an overview of the steps
that you must per form:
1 In the APIC:
a. Configure additional Tenants (optional).
b. Configure additional Private Networks (optional).
c. Configure additional Bridge Domains (optional).
d. Configure additional Application Profiles (optional).
e. Configure additional End Point Groups.
i. Create the endpoint group "AcmeInSide".
f. Configure contracts for inter-EPG communications.

i. For the Scope drop-down list, choose Tenant.
ii. For the Allow drop-down list, choose Any-Any.
2 Begin migrating the host to the fabric EPG, "AcmeInSide".

a. Create a static binding for each physical host to be migrated.
b. Create a VMM domain to deploy the host within the ACI Fabric (optional).
Note: En
sure the ap
pro
pri
ate re
sources have been cre
ated to sup
port the con
-
nectiv
ity, VLAN pools, phys ic
al do
main or VMM Do main, AEP, and switch or in
-
ter
face pro files. For more infor
mation, see the "Fab
ric Connec
tivty" sec
tion.
Fol
low
ing the stage 2 mi
gra
tion, the host con
nec
tiv
ity across both the legacy data cen
-
ter and ACI fab
ric are now gov
erned by the APIC pol
icy (con
tracts).
Note: The Layer 3 gate

way for the ACI fab
ter are pro
vided by
the legacy data cen
ter.
Datacenter migration with Layer 3 provided by existing DC and Layer 2 extended to

new ACI datacenter
In the third stage of the migra

tion the Layer 3 gate way will move from the legacy data
center to the ACI Fabric. The fol
lowing list is an overview of the steps that you must
perform:
1 In the APIC:
a. Configure a Layer 3 out.
b. Migrate the gateway from the legacy data center to the fabric.
i. In the Bridge Domain drop-down list, choose AcmeBD.
ii. In the Flood Layer 2 drop-down list, choose Unknown Unicast.
iii. In the ARP drop-down list, choose Flooding.
iv. In the Unicast drop-down list, choose Routing.
Note: The con cept of uni

cast rout
ing within the bridge domain al
lows for the con
fig
u-
ra
tion of the per
va
sive gate
way across the fabric. The Layer 3 gate
way for the ACI fab
-
ter are pro
vided by the ACI fab
ric.
159
Tenants
Tenants 161
Section Content
ACI Tenancy Models
Application Profile
Application Profile Configuration
Create a New Application Profile
Modify Application Profile
Remove Application Profile
Verify Application Profile
Endpoint Group
Endpoint Group Configuration
Create a New Endpoint Group
Modify Endpoint Group
Remove Endpoint Group
Verify Endpoint Group
Endpoint
Private Network
Private Network Configuration Parameters
Creating a New Private Network
Modify Private Network
Remove Private Network
Verify Private Network
162 Tenants
Bridge Domain
Bridge Domain Configuration Parameters
Create a new Bridge Domain
Modify a Bridge Domain
Remove a Bridge Domain
Verify Bridge Domain
Tenant Networking Use Cases

Common Private Network for All Tenants
Multiple Private Networks with Intra-Tenant Communication
Multiple Private Networks with Inter-Tenant Communication
Tenants 163
ACI Tenancy Models
ACME Inc. will be using ten ancy for a cou ple of use cases. They will be using tenant
con
structs for the ap
pli
ca
tion life
cy
cle of their cur
rent de
ployment, maintain
ing a sep
-
a
rate ten
ant for the re
sources that de
vel
op
ers will be using to build the ap
pli
ca
tion, a
tenant that will be used for the au
tomated test
ing, and fi
nally a pro
duction ten
ant. Ad-
di
tion
ally, as men tioned in the intro
duc
tion, they are also look ing to build an infra
-
struc
ture which can be lever
aged for sim
il
ar ini
tia
tives in the fu
ture. Ten
ants will be
used to draw vir
tual bound
aries for dif
fer
ent lines of busi
ness. The in
for
ma
tion se
cu
-
rity team will be able to in
te
grate this into the cor
po
rate LDAP sys
tem, and pre
vent
changes which would im
pact other groups.
ACI has been de signed from the be gin

ning to be multi-tenant. This means dif ferent
things to dif
ferent people (much like the term Cloud) based on their per spective. In the
case of a classic ser
vice provider, a ten
ant is a unique customer, while in a typic
al end-
customer en vi
ronment a tenant could be an op er
at
ing group, busi
ness unit, appli
cation
owner, etc.
The de
ci
sion on how to lever
age ten
ancy mod
els is dri
ven by a num
ber of fac
tors:
1 Overall IT operations and support models in your organization to manage
application, networking, servers, security, and so on.
2 Separation of environments from a software development lifecycle perspective:

Development, Quality Assurance, and Production.
3 Separation of duties by domain owner, such as web, app, and database owners.
4 Fault domain size and scope to limit the impact of failures, such as different
business units.
In tra
di
tional net
working envi
ron ments, mak ing a routing proto
col change on a router
or Layer 3 switch could po tentially af
fect hun dreds of unique VLANs/sub nets. This in
-
tro
duces a war ranted level of cau tion around change con trol and appli
ca
tion im
pact.
Leveraging the ACI pol
icy model, the phys ic
al hardware is ab stracted from the logic
al
constructs. The ten ant ob
ject gives us the abil
ity to draw a box around the logic
al and
concrete objects that we use to provide a uni
fied view of the config
ur
a
tion de
penden-
cies for un
derlay and over
lay networks.
164 Tenants
A Ten
ant in the ACI Ob
ject model rep
re
sents the high
est-level ob
ject. In
side, you can
dif
fer
en
ti
ate be
tween the ob
jects that de
fine the ten
ant net
work
ing, such as Pri
vate
Net
works, Bridge Do
mains and sub
nets; and the ob
jects that de
fine the ten
ant poli
cies
such as Ap
pli
ca
tion Pro
files and End
point Groups.
In ACI, the ten

ant poli
cies are where you de
fine ap
pli
ca
tions. An ap
pli
ca
tion could con
-
sist of a com
bi
na
tion of phys
ic
al servers or VMs that we will call servers from now on.
For ex
am ple, a web
site could use a 3-tier ap
pli
ca
tion model, com
prised of web servers,
ap
pli
ca
tion servers and data base servers. When a user browses the web site, they
might ac
tu
ally be communi
cat
ing with a vir
tual IP ad
dress on a load bal
ancer that in
turn can dis
trib
ute the web request to a num ber of dif
fer
ent web servers. The web
servers in turn com
mu
ni
cate with core ap
pli
ca
tions that can be di
vided amongst sev
-
eral ap
pli
ca
tions servers for load bal
anc
ing or high avail
abil
ity pur
poses. Fi
nally, the
ap
pli
ca
tion servers com
mu
ni
cate with the data
base which could also be a clus
ter of
servers.
Each server is referred to as an End

point in ACI. End
points are classi
fied in ACI to apply
poli
cies. You create endpoint groups with end points that share the same type of poli -
cies, such as with whom are they going to com muni
cate and what type of com mu ni
ca
-
tion or restric
tions are required. Therefore, an appli
ca
tion can be formed by sev -
eral endpoint groups and they are grouped in an Ap pli
ca
tion Pro
file.
The ten ant net

working is used to de
fine net
work ing poli
cies and will be ap
plied to the
un
derly
ing hardware in a transpar
ent way thanks to the layer of ab strac
tion provided
by ACI using private net
works, bridge domains and sub nets. In the next sections of this
chapter these concepts will be cov
ered in de
tail. Below you can find an il
lus
tra
tion with
the dif
fer
ent ob
jects that compound a tenant and how they are re lated.
Al
though the ten ant net
working and the tenant poli
cies are defined separ
ately, the
net
working poli
cies used by an ap
pli
ca
tion are de
fined with a re
la
tion
ship between the
Endpoint Groups and the Bridge Domain.
The follow
ing image shows all of the components that can be con
fig
ured within a ten
-
ant. In the fol
low
ing sec
tions each di
ag
ram shows the progress of how ACME Inc. adds
each com ponent.
Tenants 165
Tenant Logical Model
There are 3 Ten

ants pre
con
fig
ured in the sys
tem by de
faut:
1 Common a special tenant with the purpose of providing common services to

other tenants in the ACI fabric. Global reuse is a core principle in the common
tenant. Some examples of common services are:
a. Shared Private Networks
b. Shared Bridge Domains
c. DNS
d. DHCP
e. Active Directory
2 Infra The Infrastructure tenant that is used for all internal fabric
communications, such as tunnels and policy deployment. This includes switch
to switch (Leaf, Spine, Application Virtual Switch (AVS)) and switch to APIC.
The infra tenant does not get exposed to the user space (tenants) and it has its
own private network space and bridge domains. Fabric discovery, image
management, and DHCP for fabric functions are all handled within this tenant.
3 Mgmt - The management tenant provides convenient means to configure

access policies for fabric nodes. While fabric nodes are accessible and
configurable through the APIC, they can also be accessed directly using in-band
and out-of band connections. In-band and out-of-band policies are configured
under the mgmt tenant:
In-Band Management Access
Out-of-Band Management Access
Tenants 167
Application Profile
An applica
tion pro
file is a convenient logic
al con
tainer for mul
ti
ple hosts (physic
al or
vir
tual). You can cre ate Ap pli
ca
tion Profile contain
ers based on a va ri
ety of cri
te
ria,
such as what func tion the ap pli
ca
tion pro vides, how the appli
cation looks from the
end-user per spec
tive, where they are lo cated within the context of the data center, or
any other log
ic
al group
ing rel
at
ive to the imple
men
ta
tion. Appli
ca
tion Pro
file servers
are grouped in EPGs depending on the use of common poli
cies.
Ap
pli
ca
tion Profiles pro
vide a mech
an
ism to under
stand groups of servers as a sin
gle
ap
pli
ca
tion. This approach makes an ACI ap
pli
ca
tion aware and al
lows us to check the
op
erational state for an ap
pli
ca
tion mon it
or
ing all the servers that are part of an appli
-
ca
tion as a whole and be come in formed about rel evant faults and health sta
tus for that
par
ticul
ar appli
ca
tion. Each Appli
ca
tion Profile created can have a unique mon it
or
ing
pol
icy and QOS pol
icy.
An Appli
ca
tion Profile is a child ob
ject of the Ten
ant and a sin
gle Ten
ant can con
tain
mul
ti
ple Appli
ca
tion Pro files.
Adding components to a Tenant - 1. Application Profile

168 Tenants
Application Profile Configuration

Name - The name of the ap
pli
ca
tion pro
file.
Tags - A tag or meta

data is a non-hi
er
ar
chi
cal key
word or term as
signed to the fab
ric
module.
Mon
i
tor
ing Pol
icy - The mon
it
or
ing pol
icy name for the EPG se
man
tic scope (op
-
tional).
Create a New Application Profile
3 In the Navigation pane choose Tenant_Name > Application Profiles.
a. Enter an Application Profile Name.
b. Enter a TAG (optional).
c. Choose a Monitoring Policy (optional).
6 Click Submit.
Modify Application Profile

Application_Profile.
4 In the Work pane, choose policy.
a. Enter an Application Profile Name.
b. Enter an appropriate TAG (optional).
c. Choose the Monitoring Policy (optional).
6 Click Submit.
Tenants 169
Remove Application Profile
3 In the Navigation pane choose Tenant_Name > Application Profiles

> Application_Profile.
4 In the Work pane, choose Actions > Delete.
Verify Application Profile
REST :: /api/node/class/fvAp.xml
CLI :: moquery -c fvAp
Tenants 171
Endpoint Group
Endpoint Groups are used to cre ate log

ic
al group
ings of hosts or servers that per
form
sim
il
ar functions within the fab
ric. Each End point Group cre ated can have a unique
mon it
or
ing pol
icy and QoS pol
icy and must be asso
ci
ated with a Bridge Domain.
An End point group is a child ob ject of the Ap

pli
ca
tion Profile and an Ap
pli
ca
tion Pro
-
file can contain multi
ple Endpoint Groups. Each end point within an Endpoint Group is
suscepti
ble to the same policy in the Fabric.
Adding components to a tenant - 2. End Point Group in the Application Profile
All the Endpoints inside an EPG can com municate with each other. How com mu ni
ca
-
tions between EPGs con tracts will be required is governed by con tracts and not tra
di
-
tional Layer 2/Layer 3 for warding con structs. For example, Host-A in EPG-A can have
the IP ad dress/mask of 10. 1.
1.
10/24 and Host B in EPG B can have the IP ad -
dress/mask 10. 1.
1.
20/24 (note that both hosts be lieve they are "in the same subnet"). In
this case they would not be al lowed to com
muni
cate un
less a con tract that permit
ted
connec tiv
ity ex
isted between EPG-A and EPG-B. Contracts will be explained in greater
detail in a fol
lowing sec
tion.
Note that there are some types of End point Groups within the fabric that are not con-
tained under Appli
ca
tion Pro
files such as, Ap
pli
ca
tion End
point Group, Ex ternal Bridge
Networks (aka Layer2 Ex
ter
nal), Ex
ter
nal Routed Networks (aka as Layer3 Exter
nal) and
Man
age
ment End
point Groups. These End
point Groups might have spe
cial re
quire
-
172 Tenants
ments, for ex
am
ple, in Ex
ter
nal Bridge Net
works, MAC ad
dresses of the end
points are
not learnt by the leaf switches.
End
point Groups are linked to Bridge Do
mains but they will re
ceive a VLAN ID dif
fer
ent
from the bridge do
main, un
less Bridge Do
main legacy mode is used.
It is im
por
tant to un
der
stand that a sin
gle sub
net can be ex
tended across sev
eral EPGs.
Each EPG is iden
ti
fied by an en
cap
su
la
tion VLAN or VXLAN so that the same sub
net will
be using dif
fer
ent en
cap
su
la
tion IDs across the fab
ric. This con
cept is dif
fer
ent from
tra
di
tional net
work
ing.
Endpoint Group Configuration

Name - The name for the end
point group.
Tag - A tag or meta

data is a non-hi
er
ar
chi
cal key
word or term as
signed to the fab
ric
module.
Qos Class - The QoS pri

or
ity class iden
ti
fier.
The class can be:
Unspecified
Level1-Class 1 Differentiated Services Code Point (DSCP) value.
Level2-Class 2 DSCP value.
Level3-Class 3 DSCP value.
Custom Qos - The QoS traf

fic pri
or
ity class iden
ti
fier. The Cus
tom class is a user-con
-
fig
urable DSCP value.
Bridge Do
main - The name of the bridge do
main as
so
ci
ated with this ob
ject.
Mon
it
or icy The mon
ing Pol it
or
ing pol
icy name for the EPG se
man
tic scope (op
tional).
Asso
ci
ated Do
main Pro file A source rela
tion to an in
fra
struc
ture do
main pro
file as
so
-
ci
ated with ap
pli
ca
tion end point groups.
Tenants 173
Sub
net - An End
point Group sub
net is rel
e
vant when and only when con
fig
ur
ing route
leak
ing be
tween VRF's/Pri
vate Net
work within a Ten
ant (op
tional).
Static End
point - The static client end
point rep
re
sents a silent client end
point at
tached
to the network (optional).
Create a New Endpoint Group

Application_Profile > Application EPGs.
5 In the Create Application EPG dialog box, perform the following.

a. Enter an Application EPG Name.
b. Enter an Tag (optional).
c. Enter an Qos Class (optional).
d. Enter an Custom Qos (optional).
e. Enter a Bridge Domain Name.
f. Choose a Monitoring Policy (optional).
g. Enter an Associated Domain Profile Name.
6 Click Finish.
Modify Endpoint Group

> Application_Profile > Application EPGs > Application_EPG.
4 In the Work pane, select policy.

a. Enter an Application EPG Name.
b. Enter an Tag (optional).
c. Enter an Qos Class (optional).
d. Enter an Custom Qos (optional).
174 Tenants
e. Enter a Bridge Domain Name.

f. Choose the appropriate Monitoring Policy if applicable (optional).
g. Enter an Associated Domain Profile Name.
5 Click Finish.
Remove Endpoint Group

> Application_Profile > Application EPGs > Application_EPG..
REST :: /api/node/class/fvAEPg.xml
CLI :: moquery -c fvAEPg
Tenants 175
Endpoint
End
points are de
vices that are connected to the net work either di
rectly or indi
rectly.
End
points have an address (iden
tity), a lo
ca
tion, and at
trib
utes, and can be either vir
-
tual or phys
ic
al. Each end
point has a path, an en
cap
su
la
tion, and a de
ploy
ment Im
me
-
di
acy mode as
so
ci
ated with it.
An End
point is a child ob
ject of the End
point Group and an End point Group con struct
can con
tain multi
ple End
points. The Endpoints ref
er
enced within the fab
ric can be ei
-
ther sta
tic (de
fined within the APIC) or dy
namic (au
to
mated by vCen
ter/Open
stack).
You can add Static Endpoints by creat

ing Sta
tic Bind
ings within the End
point Group.
Below is an exam
ple of a sta
tic bind
ing. See the VVM sec tion for an ex
am
ple of a dy
-
namic binding.

Profiles > Application_Profile_Name > Application EPGs > EPG_Name
> Static Bindings (Paths).
4 In the Work pane, choose Actions > Deploy Static EPG on PC, VPC or
Interface.
5 In the Deploy Static EPG on PC, VPC or Interface dialog box, perform the
following actions:
a. Choose the Path Type.
b. Choose the Path.
c. Enter the encapsulation VLAN.
d. Click Submit.
In order to show the end

points that are con
nected to the fab
ric under cer
tain EPGs:

176 Tenants
3 In the Navigation Pane choose Tenant_Name > Application Profiles

> Application_Profile > Application EPGs > Application_EPG.
4 In the Work pane, choose Operational.
Verify Endpoint
REST :: /api/node/class/fvCEp.xml
CLI :: moquery -c fvCEp
Tenants 177
Private Network
A Pri
vate Network is also re ferred to as a VRF, private Layer 3 net
work, or con
text. It is
a unique Layer 3 for ward ing and ap pli
ca
tion policy do
main. Private net
works are a
child of the Tenant ob ject. All of the endpoints within the Private Net
work must have
unique IP ad dresses be cause it is pos si
ble to for
ward packets di
rectly be
tween these
de
vices if the pol
icy al
lows it. One or more bridge do mains are as
so
ci
ated with a pri
vate
net
work.
Adding components to a Tenant - 3. Private Network

as part of the Tenant Logical Model
The most com mon method to share Pri vate Networks be tween ten ants is through the
com mon ten ant. For more in for
ma tion about com mon ten ants, see the overview sec -
tion of this chapter. Pri
vate Networks cre ated in the com mon ten ant are shared glob-
ally within the fab
ric. How ever, a Pri
vate Network that is in
tended to be used by mul ti
-
ple ten
ants and is not cre
ated in the com
mon ten
ant re
quires ex
plicit con
fig
ur
a
tion to
be shared.
When there is a re
quire
ment to route traf
fic be
tween sep
ar
ate Pri
vate Net
work in
-
stances, spe cial con
sid
er
at
ion for sub
net config
ura
tion is needed. This will be dis
cussed
in de
tail in the Bridge Domain and EPG con fig
ur
ation sections.
178 Tenants
Private Network Configuration Parameters

Name - The name of the Pri
vate Net
work.
Policy En
force
ment - The preferred policy con ues can be en
trol. The val forced or un-
enforced. When enforced is cho sen, contracts between EPGs are re quired to allow
traf
fic. Un
en
forced al
lows all traf
fic within the Private Network. The default is en
-
forced.
BGP Timers - Name of the BGP timers pol

icy as
so
ci
ated with this ob
ject.
OSPF Timers - Name of the OSPF timers pol

icy as
so
ci
ated with this ob
ject.
End Point Re
ten
tion Pol
icy - The end point re
ten
tion pol
icy name (op
tional).
Mon
i
tor
ing Pol
icy - The mon
i
tor
ing pol
icy name for the Ten
ant se
man
tic scope (op
tional).
DNS Label - The net

work do
main name label. Labels enable clas
si
fy
ing which ob
jects
can and can
not com
mu
ni
cate with one an
other (op
tional).
Creating a New Private Network


Networks.
a. Enter an Private Network Name.
b. Choose a Policy Enforcement (optional).
c. Choose a BGP Policy Name (optional).
d. Choose an OSPF Policy Name (optional).
e. Choose an End Point Retention Policy Name (optional).
g. Choose the appropriate DNS Label if applicable (optional).
6 Click Finish.
Tenants 179
Modify Private Network


Networks > Private_Network.
4 In the Work pane, select policy.

a. Choosesa Policy Enforcement (optional).
b. Choose a BGP Policy Name (optional).
c. Choose an OSPF Policy Name (optional).
d. Choose an EIGRP Policy Name (optional).
e. Choose an End Point Retention Policy Name (optional).
g. Choose the appropriate DNS Label if applicable (optional).
5 Click Finish.
Remove Private Network

Networks > Private_Network.
Verify Private Network
REST :: /api/node/class/fvCtx.xml
CLI :: moquery -c fvCtx
Tenants 181
Bridge Domain
A Bridge Do main is the ab

stract repre
senta
tion of a Layer 2 for
warding domain within
the fab
ric. A Bridge Domain is a child of the Tenant object and must be linked to a Pri
-
vate Network.
The bridge domain de fines the unique Layer 2 MAC ad dress space and a Layer 2 flood
do
main if flooding is enabled. While a Private Net work defines a unique IP ad dress
space, that ad
dress space can con sist of mul
ti
ple sub
nets. Those subnets will be spread
across one or more bridge do mains contained in the Pri
vate Net
work.
Bridge do
mains will span all switches in which as
so
ci
ated EPG are config
ured. A bridge
domain can have mul ti
ple subnets. However, a subnet is con
tained within a sin gle
bridge do
main.
Adding components to a Tenant - 4. Bridge Domain

as part of the Tenant Application Profile
The fol
low
ing image pro vides an ex
am
ple of a ten
ant to show how bridge domains are
con
tained in
side of Pri
vate Net
works and how they are linked to EPGs and the other el
-
e
ments.
182 Tenants
End Point Group as part of the Tenant Application Profile
It is important to understand that a bridge domain is NOT a VLAN, al though it can act
sim il
ar to a VLAN. You in stead should think of it as a distrib
uted switch, which, on a
leaf, can be trans lated lo
cally as a VLAN with local sig
nif
ic
ance.
From a prac ti
cal per
spective, each Bridge Do main will exist in a par tic
ul
ar leaf if there
is a con
nected end point that belongs to that EPG. Each Bridge Do main re ceives a VLAN
ID in the leaf switches. This VLAN ID is signif
ic
ant lo
cally in the leaf switches and there -
fore it might be dif
fer from one to other leaf switch.
The VLAN ID used is also called Plat form Inde

pendent VLAN or PI VLAN. This VLAN
concept is dif
ferent from tradi
tional networking and it is not used to for
ward traf
fic but
as an identi
fier. Each PI VLAN is then linked to a VXLAN ID that will be used for for -
warding purposes in side of the fab
ric.
In the fol
low
ing ex
ample, under the Tenant Acme, the bridge do
main Acme-Ap
pli
ca
-
tions-BD was assigned the PI VLAN ID 42 in the Leaf-1.
Tenants 183
VLAN output from Leaf node
EPGs are also assigned with a PI VLAN ID that is lo

cally signif
ic
ant in each leaf. This
VLAN ID is dif
fer
ent from the Bridge Do
main. There fore in ACI, sev eral VLANs will be
used for EPs in
side on one bridge do
main. For more de tails refer to the EP section in
this chap
ter.
In some sit
ua
t
ions where tra
di
tional net
work
ing is re
quired, for ex
am
ple dur
ing some
mi
gration processes, it might be re quired to have only one en cap
su
la
tion VLAN per
Bridge do main across all the leaf switches and EPGs that refer
ence that Bridge Do
main.
For this sit
ua
t
ion, there is a fea
ture called Bridge do
main legacy mode.
Bridge domain in legacy mode is not rec om mended under ACI best prac tices. It is
thought to use tra
di
tional net
working within ACI but it lim
its the ACI fea
tures that can
be used. EPGs contained in bridge domains in legacy mode can not commu ni
cate with
EPGs in ACI bridge domains.
When a Sub
net is de
fined in a Bridge Do
main, the leaf switches will be the de
fault gate
way
for the EPGs using that sub net. If the EPGs have end points on multi
ple leaves, each
leaf will con
fig
ure the de
fault gate
way. In that way, the de
fault gate
way for the end
points
will al
ways be the first switch of the fab
ric that is reached, also know as a per
va
sive gate
-
way. This means that an SVI will be con fig
ured under the VRF that rep re
sents the pri
vate
184 Tenants
network that the Bridge Do main is linked to. If a Bridge Do

main has sev
eral sub
nets, there
will be only one SVI per Bridge Domain but it will use secondary IP ad
dresses.
Bridge Domain Configuration Parameters

Name - The name of the Bridge Do
main.
Net
work - The as
so
ci
ated layer 3 con
text.
For
ward
ing - Op
ti
mize/Cus
tom
L2 Un
known Uni
cast - The for
ward
ing method for un
known layer 2 des
ti
na
-
tions. De
fault is Proxy.
Un
known Mul ti
cast Flood
ing - The node for
ward
ing pa
ra
me
ter for un
known
Mul
ti
cast des
ti
na
tions.
ARP Flood ing - A property to specify whether ARP flooding is en

abled. If flood
-
ing is disabled, unicast rout
ing will be performed on the tar get IP ad dress.
Flooding is dis
abled by de
fault but can be en
abled by click
ing the check box.
Unicast Routing - The forwarding method based on pre de

fined for
warding cri
-
te
ria (IP or MAC ad dress). Uni
cast rout
ing is en
abled by de
fault but can be dis
-
abled by clicking the check box.
Config BD MAC Ad dress - The MAC ad dress of the bridge do main (BD) or
switched vir
tual in
ter
face (SVI). Every BD by de
fault takes the fab
ric wide de
fault
mac address.
IGMP Snoop Pol icy - The IGMP Snoop ing pol

icy name. By ex
am in
ing (snoop ing) IGMP
membership re
port messages from in
ter
ested hosts, mul
ti
cast traf
fic is lim
ited to the
sub
set of VLAN in
ter
faces on which the hosts re
side.
As
so
ci
ated L3 Outs - The name of the Layer 3 out
side in
ter
face as
so
ci
ated with this
ob
ject.
L3 Out for Route Pro file - The Layer 3 out

side in
ter
face iden
ti
fier con
trol
ling con
nec
-
tiv
ity to out
side net
works.
Tenants 185
Route Pro
file - The as
so
ci
ated route pro
file name.
Mon
i
tor
ing Pol
icy - The mon
it
or
ing pol
icy name for the Ten
ant se
man
tic scope (op
-
tional).
Subnets - The IP ad
dress and mask of the de
fault gate
way. It will be con
fig
ured in the
leaf nodes which have EPGs in that bridge do
main.
The net
work vis
ib
il
ity of the sub
net. The sub
net is a por
tion of a net
work shar
-
ing a par
tic
ul
ar sub
net ad
dress. The scope can be:
Shared - Defines subnets under an endpoint group, with the Shared option
configured, to route leak to other Tenants within the Fabric.
Public - Defines subnets under a bridge domain, with the Public option
configured, to share with Layer 3 outbound.
Private - Defines subnets under a bridge domain, with the Private option
configured, to only be used in that Tenant (will not be leaked). The default
is Private.
Sub
net Control - The control can be specific pro
to
cols ap
plied to the sub
net
such as IGMP Snooping. The control can be:
Unspecified - The default is Unspecified.

Querier IP - Enables IGMP Snooping on the subnet.
DHCP La
bels - The net
work do
main name label.
Create a new Bridge Domain
3 In the Navigation pane choose Tenant_Name > Networking > Bridge Domains.
4 In the Work pane, choose Actions > Create Bridge Domain.
5 In the Create Bridge Domain dialog box, perform the following actions:
a. Enter an Bridge Domain Name.
b. Choose the Network.
c. Choose the Forwarding Semantics (optional).
186 Tenants
d. Choose the IGMP Snoop Policy (optional).

e. Choose the Associated L3 Outs (optional).
f. Choose the L3 Out for Route Profile (optional).
g. Choose the Route Profile (optional).
h. Choose the Monitoring Policy if applicable (optional).
i. Choose the Subnets (optional).
j. Choose the DNS Label if applicable (optional).
6 Click Submit.
Modify a Bridge Domain
3 In the Navigation pane choose Tenant_Name > Networking > Bridge Domains
> Bridge_Domain_Name.
4 In the Work pane, choose the Policy tab and perform the following actions:
a. Choose the Network.
b. Choose the Forwarding Semantics (optional).
c. Choose the IGMP Snoop Policy (optional).
d. Choose the Associated L3 Outs (optional).
e. Choose the L3 Out for Route Profile (optional).
f. Choose the Route Profile (optional).
g. Choose the Monitoring Policy if applicable (optional).
h. Choose the Subnets (optional).
i. Choose the DNS Label if applicable (optional).
5 Click Finish.
Remove a Bridge Domain
3 In the Navigation Pane choose Tenant_Name > Networking > Bridge Domain >
Bridge Domain_Name.

Tenants 187
Verify Bridge Domain
REST :: /api/node/class/fvBD.xml
CLI :: moquery -c fvBD
Tenants 189
Tenant Networking Use Cases
Common Private Network for All Tenants

This use case may be typ
ic
al for en
vi
ron
ments where an ACI ad
min
is
tra
tor wishes to
cre
ate mul
ti
ple ten
ants, but place all within a sin
gle pri
vate net
work in the fab
ric.
This method has the fol

low
ing ad
van
tages and dis
ad
van
tages:
Ad
van
tages:
Ability to use a single private network for all internal and external fabric
connectivity
No route leaking needed between EPGs in different VRFs
Single Layer 3 Outside can be used by all tenants
Dis
ad
van
tages:
Changes to routing will impact all tenants
From a con
tain
ment and re
la
tion
ship per
spec
tive, this topol
ogy looks as fol
lows:
Common Private Network for all Tenants
To Con
fig
ure the com
mon Ten
ant pri
vate net
work:
2 In the Work pane, choose the common.

190 Tenants
3 In the Navigation pane choose common > Networking > Private Networks >
default.

a. Choose a Policy Enforcement (optional).
b. Choosea BGP Policy Name (optional).
c. Choose an OSPF Policy Name (optional).
d. Choosean End Point Retention Policy Name (optional).
e. Choose the appropriate Monitoring Policy if applicable (optional).
f. Choosethe appropriate DNS Label if applicable (optional).
5 Click Finish.
The Ten
ant has been cre
ated. Now the net
work ad
min
is
tra
tor will have to as
so
ci
ate the
com
mon pri
vate net
work to the Ten
ant by first cre
at
ing a bridge do
main.
3 In the Navigation pane choose Tenant_Name > Networking > Bridge Domains.
4 In the Work pane, choose Actions > Create Bridge Domain.
5 In the Create Bridge Domain dialog box, perform the following actions:
a. Enter a Bridge Domain Name.
b. Choose network.
c. In the Subnets field, click +.
d. In the Gateway IP field enter the IP address for this subnet.
e. In the Scope field you have the option to select Private, Public and Shared.
Note: By default the Private option is selected. For more information on what
to select please reference the External Layer 3 section.
6 Click OK.
7 Click Finish.
Tenants 191
The con
fig
ur
a
tion for this use case can be ap
plied via the fol
low
ing CLI con
fig
ur
a
tion:
CLI : Ten
ant Cisco
# tenant
cd '/aci/tenants'
mocreate 'Cisco'
moconfig commit
# bridge-domain
cd '/aci/tenants/Cisco/networking/bridge-domains'
mocreate 'Cisco'
cd 'Cisco'
moset network 'default'
moconfig commit
# subnet
cd '/aci/tenants/Cisco/networking/bridge-domains/Cisco/subnets'
mocreate '172.16.0.1/24'
moconfig commit
# application-profile
cd '/aci/tenants/Cisco/application-profiles'
mocreate 'App1'
moconfig commit
# application-epg
cd '/aci/tenants/Cisco/application-profiles/App1/application-epgs'
mocreate 'EPG1'
cd 'EPG1'
moset bridge-domain 'Cisco'
moconfig commit
# criterion
cd '/aci/tenants/Cisco/application-profiles/App1/application-epgs/EPG1/vm-attributes-
criteria'
mocreate 'default'
moconfig commit
This con
fig
ur
a
tion can also be ap
plied using the fol
low
ing XML posted to the APIC
REST API
192 Tenants
XML : Ten
ant Cisco
<fvTenant name="Cisco">
<fvBD arpFlood="no" multiDstPktAct="bd-flood" name="Cisco" unicastRoute="yes"
unkMacUcastAct="proxy" unkMcastAct="flood">
<fvRsCtx tnFvCtxName="default"/>
<fvSubnet ctrl="nd" descr="" ip="172.16.0.1/24" preferred="no"
scope="private"/>
</fvBD>
<fvAp name="App1">
<fvAEPg matchT="AtleastOne" name="EPG1">
<fvRsBd tnFvBDName="Cisco"/>
</fvAEPg>
</fvAp>
<fvRsTenantMonPol tnMonEPGPolName=""/>
</fvTenant>
For many multi-ten

ant envi
ron
ments it is de
sir
able to allow each ten
ant to man
age and
own their own address space and not be con cerned with over laps be
tween other ten
-
ants. This par
tic
ul
ar use case demonstrates how a pri
vate net
work can be asso
ci
ated
with each tenant. One Pri
vate Net
work per Ten
ant with In
tra-EPG communi
ca
tions
Ad
van
tages:
Allow for maximum isolation between tenants

Ability to address hosts in tenants with overlapping IP addresses
Dis
ad
van
tages:
Increased complexity when needing to allow EPG communication between

different tenants with dedicated VRF
Tenants 193
The ob
ject con
tain
ment for this par
tic
ul
ar setup can be de
picted as shown below:
Private Network per Tenant
To cre
ate the ten
ant:
1 On the menu bar, choose Tenants > ADD TENANT.
2 In the Create Tenant dialog box, perform the following actions:

a. Enter a Tenant Name.
b. Click Next.
3 Click Finish.
The Ten
ant has been cre
ated. Now the ten
ant ad
min
is
tra
tor can cre
ate the pri
vate net
-
work.

Networks.
a. Enter Private Network Name.
b. Click Next.
194 Tenants
c. Enter Associated Bridge Domain Name.

d. In the Subnets field, click +.
e. In the Gateway IP field enter the IP address for this subnet.
f. In the Scope field you have the option to select Private, Public and Shared.
Note: By default the Private option is selected. For more information on what
to select please reference the External Layer 3 section.
6 Click OK.
7 Click Finish.
The con
fig
ur
a
plied via the fol
low
ing CLI con
fig
ur
a
tion:
CLI : Ten
ant Cisco
# tenant
cd '/aci/tenants'
mocreate 'Cisco'
moconfig commit
# bridge-domain
mocreate 'Cisco'
cd 'Cisco'
moset network 'Cisco'
moconfig commit
# subnet
cd '/aci/tenants/Cisco/networking/bridge-domains/Cisco/subnets'
mocreate '172.16.0.1/24'
moconfig commit
# private-network
cd '/aci/tenants/Cisco/networking/private-networks'
mocreate 'Cisco'
moconfig commit
mocreate 'App1'
moconfig commit
# application-epg
mocreate 'EPG1'
cd 'EPG1'
moconfig commit
Tenants 195
This con
fig
ur
a
tion can also be ap
plied using the fol
low
REST API:
XML : Ten
ant Cisco
<fvTenant name="Cisco">
<fvBD arpFlood="no" multiDstPktAct="bd-flood" name="Cisco" unicastRoute="yes"
unkMacUcastAct="proxy" unkMcastAct="flood">
<fvRsCtx tnFvCtxName="Cisco"/>
<fvSubnet ctrl="nd" ip="172.16.0.1/24" name="" preferred="no" scope="private"/>
</fvBD>
<fvCtx knwMcastAct="permit" name="Cisco" pcEnfPref="enforced"/>
<fvAp name="App1" prio="unspecified">
<fvAEPg name="EPG1">
<fvRsBd tnFvBDName="Cisco"/>
</fvAEPg>
</fvAp>
</fvTenant>
Multiple Private Networks with Intra-Tenant Communication

Another use case that may be de sir
able to sup
port is the option to have a sin
gle ten
ant
with multi
ple pri
vate networks. This may be a result of need ing to provide multi-ten-
ancy at a network level, but not at a management level. It may also be caused by need -
ing to sup
port overlap
ping subnets within a sin
gle tenant, due to merg ers and acqui
si
-
tions or other busi
ness changes.

low
ing ad
van
tages and dis
ad
van
tages:
Ad
van
tages:
Ability to have overlapping subnets within a single tenant
Dis
ad
van
tages:
EPGs residing in overlapping subnets cannot have policy applied between one
another
196 Tenants
The ob
ject con
tain
ment for this par
tic
ul
ar setup can be de
picted as shown below:
Multiple Private Networks with Intra-Tenant communication
To cre
ate the ten
ant:
1 On the menu bar, choose Tenants > ADD TENANT.
2 In the Create Tenant dialog box, perform the following actions:

a. Enter a Tenant Name.
3 Click Next.
4 Click Finish.
The Ten
ant has been cre
ated. Now the ten
ant ad
min
is
tra
tor can cre
ate the pri
vate net
-
work.

Networks.
a. In the Name field enter a name for the Private Network.
b. Click Next.
Tenants 197
d. In the Subnets field click +.

f. In the Scope field select Shared.
Note: The shared subnet type causes what is known in ACI as a route
leak between two Private Networks (VRF).
6 Click OK.
7 Click Finish.

Networks.
a. In the Name field enter a name for the second Private Network.
b. Click Next.
c. In the Name field enter a name for the second bridge domain.
11 Click OK.
12 Click Finish.
Now that the two pri vate networks and bridge do

mains have been cre
ated, you can
move to the Ap
pli
ca
tion Pro
file.
3 In the Navigation pane choose Tenant_Name > Networking > Application

Profiles.
4 In the Action Tab select Create Application Profile.
6 Click Submit.
198 Tenants
To cre
ate the two end
point groups:

Profiles > Application Profile_Name.
a. In the Name field enter a name for the End Point Group.
b. In the Bridge Domain field, select the appropiate bridge domain.
6 Click Finish.
Re
peat these steps for the sec
ond EPG.
The ten
ant ad
min
is
tra
tor will have to now cre
ate a con
tract and fil
ter be
tween the two
EPGs.
3 In the Navigation pane choose Tenant_Name > Networking > Security

Policies > Filters.
4 In the Work pane, choose Actions > Create Filters.
5 In the Create Filter dialog box, perform the following actions:

a. In the Name field enter a name for the Filter.
b. Click +.
c. In the Name column enter ICMP.
d. In the Ethertype select IP.
e. In the IP Protocol column select ICMP.
6 Click Update.
7 Click Submit.
As the ten
ant ad minis
tra
tor, you would have the knowl edge of the fil
ters re
quired to
per
mit traf
fic across the two EPGs. In the fil
ter, you can re
peat the process of defin
ing
Tenants 199
var
io
us dif
fer
ent network pro to
cols as re
quired for your appli
ca
tions. Now you have to
de
fine the contract that will be con
sumed and pro vided by the two EPGs.

Policies > Contracts.
4 In the Work pane, choose Actions > Create Contract.
5 In the Create Contract dialog box, perform the following actions:

b. In the Scope field select Global.
c. Click Subjects field click +.
d. In the Name field enter a name for the Subject.
e. In the Filter Chain field click +.
f. Select the Filter you just created.
6 Click Update.
7 Click OK.
8 Click Submit.
As
sign the Contract be
tween the EPGs. A con tract is as
signed to an EPG as ei ther a
consumed or a pro
vided con
tract. Each EPG that you have cre ated will then ei
ther con
-
sume or pro
vide that con
tract to es
tab
lish a re
la
tion
ship between both EPGs.

Application Profile Name > Application EPGs > EPG_Name > Contract.
4 In the Work pane, choose Actions > Add Provided Contract.
5 In the Add Provided Contract dialog box, perform the following actions:
a. Select the created contract.
b. Click Submit..
6 In the Navigation pane under the second EPG select Contracts.
7 In the Action Tab select Add Consume Contract.

a. Select the created contract
200 Tenants
8 Click Submit.
The con
fig
ur
a
plied via the fol
low
ing CLI con
fig
ur
a
tion:
CLI : Ten
ant Cisco
# tenant
cd '/aci/tenants'
mocreate 'Cisco'
moconfig commit
# bridge-domain
mocreate 'Cisco'
cd 'Cisco'
moconfig commit
# bridge-domain
mocreate 'Cisco1'
cd 'Cisco1'
moset network 'Cisco1'
moconfig commit
# private-network
mocreate 'Cisco'
moconfig commit
# private-network
mocreate 'Cisco1'
moconfig commit
mocreate 'App1'
moconfig commit
# application-epg
mocreate 'EPG1'
cd 'EPG1'
Tenants 201
moconfig commit
# fv-rscon
cd '/aci/tenants/Cisco/application-profiles/App1/application-
epgs/EPG1/contracts/consumed-contracts'
mocreate 'ICMP'
moconfig commit
# fv-subnet
cd '/aci/tenants/Cisco/application-profiles/App1/application-epgs/EPG1/subnets'
mocreate '172.16.1.1/24'
cd '172.16.1.1:24'
moset scope 'private,shared'
moconfig commit
# application-epg
mocreate 'EPG2'
cd 'EPG2'
moset bridge-domain 'Cisco1'
moconfig commit
# fv-rsprov
cd '/aci/tenants/Cisco/application-profiles/App/application-
epgs/EPG2/contracts/provided-contracts'
mocreate 'ICMP'
moconfig commit
# fv-subnet
cd '/aci/tenants/Cisco/application-profiles/CCO/application-epgs/EPG2/subnets'
mocreate '172.16.2.1/24'
cd '172.16.2.1:24'
moconfig commit
This con
fig
ur
a
tion can also be ap
plied using the fol
low
REST API:
XML : Ten
ant Cisco
<fvTenant dn="uni/tn-Cisco" name="Cisco">
<vzBrCP name="ICMP" scope="tenant">
<vzSubj consMatchT="AtleastOne" name="icmp" provMatchT="AtleastOne"

202 Tenants
revFltPorts="yes">
<vzRsSubjFiltAtt tnVzFilterName="icmp"/>
</vzSubj>
</vzBrCP>
<fvCtx knwMcastAct="permit" name="CiscoCtx" pcEnfPref="enforced"/>
<fvCtx knwMcastAct="permit" name="CiscoCtx2" pcEnfPref="enforced"/>
<fvBD arpFlood="yes" mac="00:22:BD:F8:19:FF" name="CiscoBD2" unicastRoute="yes"
unkMacUcastAct="flood" unkMcastAct="flood">
<fvRsCtx tnFvCtxName="CiscoCtx2"/>
</fvBD>
<fvBD arpFlood="yes" mac="00:22:BD:F8:19:FF" name="CiscoBD" unicastRoute="yes"
<fvRsCtx tnFvCtxName="CiscoCtx"/>
</fvBD>
<fvAp name="CCO">
<fvAEPg matchT="AtleastOne" name="Web">
<fvRsCons tnVzBrCPName="ICMP"/>
tDn="topology/pod-1/paths-201/pathep-[eth1/16]"/>
<fvSubnet ip="172.16.2.1/24" scope="private,shared"/>
<fvRsDomAtt instrImedcy="lazy" resImedcy="lazy" tDn="uni/phys-
PhysDomainforCisco"/>
<fvRsBd tnFvBDName="CiscoBD2"/>
</fvAEPg>
<fvAEPg matchT="AtleastOne" name="App">
<fvRsBd tnFvBDName="CiscoBD"/>
<fvRsProv matchT="AtleastOne" tnVzBrCPName="ICMP"/>
</fvAEPg>
</fvAp>
</fvTenant>

Tenants 203

ic
al for en
vi
ron
min
is
tra
tor wishes to
cre
ate mul
ti
ple ten
ants with the abil
ity to sup
port in
ter-ten
ant com
mu
ni
ca
tions.

low
ing ad
van
tages and dis
ad
van
tages:
Ad
van
tages
Each tenant container can be managed separately

Allows for maximum isolation between tenants
Dis
ad
van
tages
Tenant address space must be unique
From a con
tain
ment and re
la
tion
ship per
spec
tive, this topol
ogy looks as fol
lows:
To cre
ate the ten
ant:
1 In the GUI Navigate to Tenants > ADD TENANT.

a. In the Name field enter a name for the first Tenant.
3 Click Next.
204 Tenants
4 Click Finish.
The ten
ant has been cre
ated. Now the ten
ant ad
min
is
tra
tor can cre
ate the pri
vate net
-
work.
3 In the Navigation Pane choose Tenant_Name > Networking > Private

Networks.
b. Click Next.
6 Click OK.
7 Click Finish.
To cre
ate the ap
pli
ca
tion pro
file:
3 In the Navigation Pane choose Tenant_Name > Networking > Application

Profiles.
6 Click Submit.
Tenants 205
To cre
ate the end
point group:

Profiles > Application Profile_Name
6 Click Finish.
To cre
ate the sec
ond ten
ant and ap
pli
ca
tion pro
file:
1 In the GUI Navigate to Tenants > ADD TENANT.

a. In the Name field enter a name for the first Tenant.
3 Click Next.
4 Click Finish.
The ten
ant has been cre
ated. Now the ten
ant ad
min
is
tra
tor can cre
ate the pri
vate net
-
work.

Networks.
b. Click Next.
206 Tenants

6 Click OK.
7 Click Finish.
To cre
ate the ap
pli
ca
tion pro
file:

Profiles.
6 Click Submit.
To cre
ate the end
point group:

Profiles > Application Profile_Name
6 Click Finish.
The ten
ant ad
min
is
tra
tor will have to now cre
ate a con
tract and fil
ter be
tween the two
EPGs.

Tenants 207

Policies > Filters.
4 In the Work pane, choose Actions > Create Filters.

b. Click +.
c. In the Name column enter ICMP.
d. In the Ethertype select IP.
e. In the IP Protocol column select ICMP.
6 Click Update.
7 Click Submit.
As the tenant ad minis

tra
tor you would have the knowl edge of the filters re
quired to
per
mit traffic across the two EPGs. In the filter you can repeat the process of defining
var
io
us dif
ferent network pro to
cols as re
quired for your appli
ca
tions. Now you have to
de
fine the con tract that will be con
sumed and pro vided by the two EPGs.

Policies > Contracts.

a. In the Name field enter a name for the filter.
b. In the Scope field choose Global.
c. Click Subjects field click +.
d. In the Name field enter a name for the Subject.
e. In the Filter Chain field click +.
f. Select the filter you just created.
6 Click Update.
7 Click OK.
8 Click Submit.
208 Tenants
As
sign the Contract be
tween the EPGs. A con tract is as
signed to an EPG as ei ther a
consumed or a pro
vided con
tract. Each EPG that you have created will then ei
ther con
-
sume or pro
vide that con
tract to es
tab
lish a re
la
tion
ship be
tween both EPGs.

Profiles > Application Profile Name > Application EPGs > EPG_Name >
Contracts.
4 In the Work pane, choose Actions > Add Provided Contract.
5 In the Add Provided Contract dialog box, perform the following actions:
a. Select the created Contract.
b. Click Submit.
6 In the GUI Navigate to Tenants > ALL TENANTS.
7 Select the second created Tenant.

Profiles > Application Profile Name > Application EPGs > EPG_Name >
Contracts.
9 In Work pane, choose Action > Add Consume Contract.

a. Select the created Contract.
10 Click Submit.
The con
fig
ur
a
plied via the fol
low
ing CLI con
fig
ur
a
tion:
CLI : TEN
ANT Cis
co1
# tenant
cd '/aci/tenants'
mocreate 'Cisco1'
moconfig commit
# bridge-domain
cd '/aci/tenants/Cisco1/networking/bridge-domains'
mocreate 'Cisco1'
cd 'Cisco1'
moset network 'Cisco1'
moconfig commit
Tenants 209
# private-network
cd '/aci/tenants/Cisco1/networking/private-networks'
mocreate 'Cisco1'
moconfig commit
cd '/aci/tenants/Cisco1/application-profiles'
mocreate 'App1'
moconfig commit
# application-epg
cd '/aci/tenants/Cisco1/application-profiles/App1/application-epgs'
mocreate 'EPG1'
cd 'EPG1'
moset bridge-domain 'Cisco1'
moconfig commit
# fv-rsprov
cd '/aci/tenants/Cisco/application-profiles/CCO/application-
epgs/App/contracts/provided-contracts'
mocreate 'ICMP'
moconfig commit
# fv-subnet
cd '/aci/tenants/Cisco/application-profiles/CCO/application-epgs/App/subnets'
mocreate '172.16.1.1/24'
cd '172.16.1.1:24'
moconfig commit
# contract
cd '/aci/tenants/Cisco/security-policies/contracts'
mocreate 'ICMP'
cd 'ICMP'
moset scope 'global'
moconfig commit
# contract-subject
cd '/aci/tenants/Cisco/security-policies/contracts/ICMP/subjects'
mocreate 'icmp'
moconfig commit
# vz-rssubjfiltatt
cd '/aci/tenants/Cisco/security-policies/contracts/ICMP/subjects/icmp/common-filters'
mocreate 'icmp'
moconfig commit
210 Tenants
CLI : TEN
ANT Cis
co2
# tenant
cd '/aci/tenants'
mocreate 'Cisco'
moconfig commit
# bridge-domain
mocreate 'Cisco'
cd 'Cisco'
moconfig commit
# private-network
mocreate 'Cisco'
moconfig commit
mocreate 'App1'
moconfig commit
# application-epg
cd '/aci/tenants/Cisco2/application-profiles/App1/application-epgs'
mocreate 'EPG1'
cd 'EPG1'
moconfig commit
# fv-rsconsif
cd '/aci/tenants/Cisco1/application-profiles/CCO/application-
epgs/Web/contracts/consumed-contract-interfaces'
mocreate 'CiscoInterTenantICMP'
moconfig commit
# fv-subnet
cd '/aci/tenants/Cisco1/application-profiles/CCO/application-epgs/Web/subnets'
mocreate '172.16.2.1/24'
cd '172.16.2.1:24'
moset scope 'shared-subnet'
moconfig commit
# imported-contract
cd '/aci/tenants/Cisco1/security-policies/imported-contracts'
mocreate 'CiscoInterTenantICMP'
cd 'CiscoInterTenantICMP'
moset contract 'tenants/Cisco/security-policies/contracts/ICMP'
moconfig commit
Tenants 211
This con
fig
ur
a
tion can also be ap
plied using the fol
low
REST API:
XML : TEN
ANT Cis
co1
<fvTenant dn="uni/tn-Cisco1" name="Cisco1">
<vzBrCP name="ICMP" scope="global">
<vzSubj consMatchT="AtleastOne" name="icmp" provMatchT="AtleastOne"
revFltPorts="yes">
<vzRsSubjFiltAtt tnVzFilterName="icmp"/>
</vzSubj>
</vzBrCP>
<vzCPIf dn="uni/tn-Cisco1/cif-ICMP" name="ICMP">
<vzRsIf consMatchT="AtleastOne" name="icmp" provMatchT="AtleastOne"
revFltPorts="yes">
<vzRsSubjFiltAtt tDn="uni/tn-Cisco2/brc-default"/>
</vzRsIf>
</vzCPIf>
<fvRsCtx tnFvCtxName="CiscoCtx2"/>
</fvBD>
<fvBD arpFlood="yes" name="CiscoBD" unicastRoute="yes" unkMacUcastAct="flood"
unkMcastAct="flood">
</fvBD>
<fvAp name="CCO">

212 Tenants
<fvRsProv matchT="AtleastOne" tnVzBrCPName="ICMP"/>
</fvAEPg>
</fvAp>
</fvTenant>
XML : TEN
ANT Cis
co2
<fvTenant dn="uni/tn-Cisco2" name="Cisco2">
</fvBD>
<fvBD arpFlood="yes" name="CiscoBD" unicastRoute="yes" unkMacUcastAct="flood"
unkMcastAct="flood">
</fvBD>
<fvAp name="CCO">

Tenants 213
<fvRsConsIf matchT="AtleastOne" tnVzBrCPIfName="ICMP"/>
</fvAEPg>
</fvAp>
</fvTenant>
215
Working with Contracts

Working with Contracts 217
Section Content
Contracts
Contract Configuration Parameters
Create/Modify/Remove Contracts
Create Contracts
Modify Contracts
Remove Contracts
Verify Contracts
Apply/Remove EPG Contracts

Apply a Contract to an EPG
Remove a Contract from EPG
Verify Contract on EPG
Apply/Remove External Network Contracts

Apply a Contract to an External Network
Remove a Contract from an External Network
Verify External Network Contracts
Apply/Remove Private Network Contracts

Apply a Contract to a Private Network (vzAny)
Remove a Contract From a Private Network (vzAny)
Verify Private Network Contracts
Filters
Filter Entry Configuration Parameters
Create Filters
Modify Filters
Remove Filters
Verify Filters
218 Working with Contracts
Taboo Contracts
Taboo Contract Configuration Parameters
Create, Modify, or Delete Taboo Contracts
Create Taboo Contracts
Modify Taboo Contracts
Delete Taboo Contracts
Verify Taboo Contracts
Apply/Remove Taboo Contracts

Apply Taboo Contract to an EPG
Remove Taboo Contract from EPG
Verify Taboo Contracts Applied to EPG
Inter-Tenant Contracts
Configuration Parameters
Create/Modify/Remove Export Contracts

Export Contract
Modify Exported Contracts
Remove Exported Contracts
Verify Exported Contracts
Contracts Use Cases

Tenant Cisco-1/EPG-1
Inter-Private Network Contracts Communication

Single Contract Bidirectional Reverse Filter
Single Contract Unidirectional with Multiple Filters
Multiple Contracts Uni-Directional Single Filter

Contracts
Contracts pro
vide a way for the ACI ad
minis
tra
tor to con
trol traf
fic flow within the ACI
fab
ric be
tween EPGs. These con tracts are built using a provider-con sumer model
where one EPG pro vides the ser
vices it wants to offer and an other EPG con sumes
them.
Con
tracts are com
prised of the fol
low
ing items:
Subjects - A group of filters for a specific application or service

Filters - Used to classify traffic based upon layer 2 to layer 4 attributes (such as
Ethernet type, protocol type, TCP flags and ports)
Actions - Permit, deny, mark, log, redirect, copy, and service graph to perform
matches based upon filters
Labels - Used optionally to group objects such as subjects and EPGs for the
purpose of further defining policy enforcement
EPGs can only com municate with other EPGs based upon the con tract rules de
-
fined. There is no contract re quired for in
tra-EPG com
mu
ni
ca
tion: in
tra-EPG com mu-
ni
ca
tion is al
lowed by default.
The ex ample below, shows how con tracts would control traf
fic flow between EPGs in a
3-tiered ap pli
ca
tion. The Web EPG pro vides a con
tract which is con sumed by the Ap-
pli
cation EPG, and the Ap pli
ca
tion EPG pro vides a contract which the Data base EPG
would con sume. EPGs may acts a both a provider and con sumer of the same con tract
easily by making the con
tract bidi
rec
tional.
Contract Policies Between End Point Groups
Con
tracts gov
ern the fol
low
ing types of end
point group com
mu
ni
ca
tions:
Between application EPGs

Between application EPGs and external networks
Between application EPGs and in-band management EPG, for example if in-
band management is configured for the ACI fabric and certain EPGs are to be
allowed to access it
Contract Configuration Parameters

When con
fig
ur
ing con
tracts you can de
fine the fol
low
ing op
tions:
Con
tract Scope - The scope of a ser
vice con
tract be
tween two or more par
tic
ip
at
ing
peer en
ti
ties.
The states are:
Context - This contract will be applied for endpoint groups associated with the
same private network.
Application-profile - This contract will be applied for endpoint groups in the
same application profile.
Tenant - This contract will be applied for endpoint groups within the same tenant.
Global - This contract will be applied for endpoint groups throughout the
fabric.
fault is Con
The de text.
QoS Class - The pri

or
ity level of the ser
vice con
tract.
The pri
or
ity level can be:
Unspecified
Level1 - Class 1 Differentiated Services Code Point (DSCP) value.
Level2 - Class 2 DSCP value.
Level3 - Class 3 DSCP value.
fault is Un
The de spec
i
fied.
Tags - The search key word or term that is assigned to the ap

pli
ca
tion pro
file. A tag al
-
lows you to group mul ti
ple ob
jects by a de
scriptive name. You can as
sign the same tag
name to multi
ple ob
jects and you can assign one or more tag names to an ob ject.
Match - The sub

ject match cri
te
ria across con
sumers. The op
tions are:
AtleastOne
AtmostOne
None
All
fault is Atlea
The de st
O
ne.
Create/Modify/Remove Contracts
Create Contracts
3 In the Navigation pane choose Tenant_Name > Security Policies > Contracts.

a. Enter a Contract Name.
b. Choose a Contract Scope (optional).
c. Choose a QoS Class (optional).
d. Click + next to the Subject to add a Contract Subject.
i. In the Create Contract Subject dialog box, perform the following actions:
1. Enter a Contract Subject Name.
2. Click + in the Filter Chain field.
For information regarding filter creation, see the "Filters" section.
6 Click Update.
7 Click OK.
8 Click Submit.
Modify Contracts
3 In the Navigation pane choose Tenant_Name > Security Policies > Contracts >
Contract_Name.
4 In the Work pane, choose the Policy tab.

a. Choose a Contract Scope (optional).
b. Choose a Qos Class (optional).
c. Click + next to the Subject field. to add a Contract Subject.
i. In the Create Contract Subject dialog box, perform the following actions:
1. Enter a Contract Subject Name.
2. Click + next to Filter Chain.
Note: For information regarding filter creation, see the "Filters" section.
5 Click Update.
6 Click OK.
7 Click Submit.
Remove Contracts
Remove Contracts
3 In the Navigation pane choose Tenant_Name > Security Policies > Contracts
> Contract_Name.
Verify Contracts
REST :: /api/node/class/vzBrCP.xml
CLI :: moquery -c vzBrCP
Apply/Remove EPG Contracts
Apply a Contract to an EPG

Application_Profile_Name > Application EPGs > EPG_Name > Contracts.
4 In the Work pane, choose Actions > Add Provided Contract or Actions > Add
Consumed Contract.
Note: Choose the action depending on how the contract is to be deployed.
5 In the Add Contract dialog box, perform the following actions:

a. Enter a Contract_Name.
b. Choose a QOS policy (optional).
c. Choose a Label (optional).
6 Click Submit.
Remove a Contract from EPG


> Application_Profile_Name .> Application EPGs > EPG_Name > Contracts >
Contract_Name.
Verify Contract on EPG
Provider
REST :: /api/node/class/fvRsProv.xml
CLI :: moquery -c fvRsProv
Consumer

REST :: /api/node/class/fvRsCons.xml
CLI :: moquery -c fvRsCons
Apply/Remove External Network Contracts
Apply a Contract to an External Network
3 In the Navigation pane choose Tenant_Name > Networking > External Routed
Networks > Routed Outside_Name > Networks >
External_Network_Instance_Profile.
4 In the Work pane, click + next to either Add Provided Contract or Add
Consumed Contract.
Note: Make a selection depending on how the contract is to be deployed.
a. Choose a Contract_Name.
b. Choose a QOS Type.
c. Choose a Match Criteria.
5 Click Update.
Remove a Contract from an External Network
3 In the Navigation pane choose Tenant_Name > Networking > External Routed
Networks > Routed Outside_Name > Networks
> External_Network_Instance_Profile.
4 In the Work pane, choose the Contract_Name and click x.
Verify External Network Contracts
Provider
Consumer
Apply/Remove Private Network Contracts

In order to apply con tracts to all endpoint groups within a pri
vate net
work, con tracts
can be applied di
rectly to the private net
work. This con
cept is also re
ferred as "vzAny"
endpoint group. It eases con
tract manage
ment by al
low
ing the contract con
fig
ura
tion
for all end
point groups within a pri
vate net
work from a sin
gle lo
ca
tion as well as opti
-
mizing hard
ware re source consumption.
For in
stance, if an ACI Ad
min
is
tra
tion has 100 EPGs that are all part of the same pri
vate
net
work, they can apply the con
tracts to this one vzAny group under the pri
vate net
-
work, rather than to each EPG.
Apply a Contract to a Private Network (vzAny)


Networks > Private_Network_Name > EPG Collection for Context.
4 In the Work pane, click + next to either Add Provided Contract or Add
Consumed Contract.
Note: Make a selection depending on how the contract is to be deployed.
a. Enter a Contract_Name.
b. Choose a QOS Type.
c. Choose a Match Criteria.
Click Update.
Remove a Contract From a Private Network (

vzAny)

Networks > Private_Network_Name > EPG Collection for Context.
4 In the Work pane, choose the Contract_Name and click x.
Verify Private Network Contracts
REST :: /api/node/class/vzBrCP.xml
CLI :: moquery -c vzBrCP
Filters
A fil
ter pol
icy is a group of resolv
able fil
ter en
tries. Each fil
ter entry is a com
bi
na
tion of
network traffic clas
si
fi
ca
tion proper
ties.
Fil
ters are Layer 2 to Layer 4 fields, TCP/IP header fields such as Layer 3 pro
to
col type,
Layer 4 ports, and so on. According to its re
lated con
tract, an EPG provider dictates the
proto
cols and ports in both the in and out di rec
tions. Contract sub
jects con
tain as
soci
-
a
tions to the fil
ters (and their di
rec
tions) that are applied between EPGs that pro duce
and con
sume the con
tract. See the Use Cases below for an ex
am
ple.
Filter Entry Configuration Parameters

When con
fig
ur
ing a Fil
ter, the fol
low
ing op
tions can be de
fined:
Name - The name of a fil

ter entry.
Ether
Type - The Ether
Type of the fil
ter entry. The Ether
Types are:
ARP
FCOE
IP
MAC Security
MPLS Unicast
Trill
Unspecified
fault is ARP.
The de
Arp Flag - The Address Resol

ution Protocol flag for a fil
ter entry. The fil
ter entry is a
combi
na
tion of net
work traf
fic clas
si
fi
ca
tion prop er
ties.
IP Pro
to
col - The IP pro to
col for a fil
ter entry. The fil
ter entry is a com
bi
na
tion of net
-
work traf
fic clas
si
fi
ca
tion proper
ties.
Allow Frag
ment - The start of the source port range. The start of the port range is de
-
ter
mined by the server type. The range is 0 to 0xffff. The port can be for the fol
low
ing
server types:
Source Port: From - The start of the source port range. The start of the port range is
de
termined by the server type. The range is 0 to 0xffff. The port can be for the fol
low
-
ing server types:
Unspecified
ftpData
SMTP
DNS
HTTP
POP3
HTTPS
RTSP
Source Port: To - The end of the source port range. The end of the port range is de
ter
-
mined by the server type. The range is 0 to 0xffff. The port can be for the fol low
ing
server types:
Unspecified
ftpData
smtp
DNS
HTTP
POP3
HTTPS
RTSP
Des ti
na
tion Port: From - The end of the des ti
na
tion port range. The end of the port
range is deter
mined by the server type. The range is 0 to 0xffff. The port state set
tings
are:
Unspecified
ftpData
SMTP
DNS
HTTP
POP3
HTTPS
RTSP
Des
ti
na
tion Port: To - The start of the des
ti
na
tion port range. The port range is de
ter
-
mined by the server type. The range is 0 to 0xffff. The des
ti
na
tion port can be set for
the fol
low
ing server types:
unspecified
ftpData
smtp
dns
http
pop3
https
rtsp
fault is un
The de spec
i
fied.
TCP Session Rules - The TCP ses sion rules for a fil
ter entry. The fil
ter entry is a com
bi
-
na
tion of net
work traf
fic clas
si
fi
ca
tion proper
ties.
Create Filters
3 In the Navigation pane choose Tenant_Name > Security Policies > Filters.
4 In the Work pane, choose Actions > Create Filter.

a. Enter an Filter Name.
b. Click + to add a Filter Entry.
i. In the Filter Entry dialog box, perform the following actions:

1. Enter a Filter Entry Name.
2. In the drop-down list select an Ethertype.
3. In the drop-down list select an ARP Flag (optional).
4. In the drop-down list select an IP Protocol (optional).
5. If required, check the Allow Fragment check box (optional).
6. In the drop-down list select the Source Port From (optional).
7. In the drop-down list select the Source Port To (optional).
8. In the drop-down list select the Destination Port From (optional).
9. In the drop-down list select the Destination Port To (optional).
10. In the drop-down list select the TCP Session Rules (optional).
6 Click Update.
7 Click Submit.
Modify Filters
3 In the Navigation pane choose Tenant_Name > Security Policies > Filters >
Filter_Name.
4 In the Work pane, double click the filter entry.

a. In the drop-down list select an Ethertype (optional).
b. In the drop-down list select an ARP Flag (optional).
c. In the drop-down list select an IP Protocol (optional).
d. If required, click the Allow Fragment check box (optional).
e. In the drop-down list select the Source Port From (optional).
f. In the drop-down list select the Source Port To (optional).
g. In the drop-down list select the Destination Port From (optional).
h. In the drop-down list select the Destination Port To (optional).
i. In the drop-down list select the TCP Session Rules (optional).
5 Click Update.
6 Click Submit.
Remove Filters
3 In the Navigation pane choose Tenant_Name > Security Policies > Filters >
Filter_Name.
Verify Filters
REST :: /api/node/class/vzFilter.xml
CLI :: moquery -c vzFilter
Taboo Contracts
There may be times when the ACI ad minis

tra
tor might need to deny traf fic that is al
-
lowed by an
other con
tract. Taboos are a special type of con
tract that an ACI admin is
-
tra
tor can use to deny spe
cific traf
fic that would other
wise be al
lowed by an
other con-
tract. Taboos can be used to drop traf fic match
ing a pat
tern (any EPG, a spe
cific EPG,
match
ing a fil
ter, and so forth). Taboo rules are ap
plied in the hard
ware be
fore the
rules of reg
ul
ar con
tracts are ap
plied.
Taboo con
tracts are not rec
om
mended as part of the ACI best prac
tices but they can
be used to transi
tion from tradi
tional net
work ing to ACI. To imit
ate the tra
di
tional net
-
work ing con
cepts, an "allow-all-traf
fic" contract can be applied, with taboo con tracts
config
ured to re
strict cer
tain types of traf
fic.
Taboo Contract Configuration Parameters

When con
fig
ur
ing Taboo Con
tracts you can de
fine the fol
low
ing op
tions:
Name - The name of the con

tract or con
tract ob
ject.
Sub
jects - The net
work do
main name label. La
bels enable clas
sifi
ca
tion of the ob
jects
which can and can
not commu
ni
cate with one an
other (optional).
Di
rec
tive - The fil
ter di
rec
tives as
signed to the taboo con
tract.
Create, Modify, or Delete Taboo Contracts
Create Taboo Contracts
3 In the Navigation pane choose Tenant_Name > Security Policies > Taboo
Contracts.
4 In the Work pane, choose Action > Create Taboo Contract.
5 In the Create Taboo Contract dialog box, perform the following actions:
a. Enter a Taboo Contract Name.
b. Click + to next to the Subject field to add a Taboo Subject.
i. Enter a Filter Name.
ii. Choose Directives.
6 Click Update.
7 Click OK.
8 Click Submit.
Modify Taboo Contracts
3 In the Navigation pane choose Tenant_Name > Security Policies > Taboo
Contracts > Taboo_Contract_Name.

a. Click + to next to the Subject field.
b. In the Create Taboo Contract Subject dialog box, perform the following
actions:
i. Enter a Taboo Contract Subject Name.
ii. Click + in the Filter Chain field.
1. Enter a Filter Name.
2. Choose Directives.
5 Click Submit.
Delete Taboo Contracts
3 In the Navigation pane choose Tenant_Name > Security Policies

> Taboo Contracts > Taboo_Contract_Name.
4 In the Work pane, choose Action > Delete.
Verify Taboo Contracts
REST :: /api/node/class/vzTaboo.xml
CLI :: moquery -c vzTaboo
Apply/Remove Taboo Contracts
Apply Taboo Contract to an EPG

> Application_Profile_Name > Application EPGs > EPG_Name > Contracts.
4 In the Work pane, choose Actions > Add Taboo Contract.
5 In the Add Taboo Contract dialog box,

a. Choose the Taboo Contract.
6 Click Submit.
Remove Taboo Contract from EPG

> Application_Profile_Name > Application EPGs > EPG_Name > Contracts.
4 In the Work pane, choose the Taboo Contract_Name > Actions > Delete.
Verify Taboo Contracts Applied to EPG
Provider
C
onsumer
There may be times when the ACI ad minis

tra
tor might need to allow traffic be
tween
two ten
ants. In
ter
face con
tracts are a spe
cial type of con
tract that an ACI adminis
tra
-
tor can use to allow specific traf
fic through the use of a con
tract export. The contract
in essence is ex
ported in the source ten ant and im
ported into the tar
get ten
ant. Simil
ar
to tra
di
tional con
tracts, the source EPG will be of type provider. How
ever, in the tar
get
ten
ant, the con
tract is im
ported as type con
tract in
ter
face. Some use case ex
am
ples
show the com
plete process in the next chap
ter.
Configuration Parameters
When Im
port
ing a Con
tract, the fol
low
ing op
tions can be de
fined:
Name - The name of the con

tract in
ter
face.
Global Con tract - Name of a ser

vice con
tract to be shared be
tween two or more par
-
tic
ip
at
ing peer enti
ties.
Ten
ant - The Ten
ant name of the tar
geted Ex
port con
tract.
Create/Modify/Remove Export Contracts
Export Contract
3 In the Navigation pane choose Tenant_Name > Security Policies > Contracts.
4 In the Work pane, choose Actions > Export Contract.
5 In the Export Contract dialog box, perform the following actions:

a. Enter an Export Contract Name.
b. Choose the Global Contract.
c. Enter the Tenant Name.
6 Click Finish.
Modify Exported Contracts
3 In the Navigation pane choose Tenant_Name > Security Policies > Contracts >
Contract_Name.

a. Enter an Export Contract Name.
b. Choose the Global Contract.
c. Enter the Tenant Name.
5 Click Finish.
Remove Exported Contracts
3 In the Navigation pane choose Tenant_Name > Security Policies > Contracts
> Imported Contracts > Contact_Name.
Verify Exported Contracts
REST :: /api/node/class/vzCPif.xml
CLI :: moquery -c vzCPif
Contracts Use Cases
These use cases all assume the ob jec

tive is for a host in EPG-1 to talk to a host in EPG-
2, achiev
ing bidi
rec
tional traf
fic. How these sce nar
ios are im
ple
mented will de pend on
the op
er
at
ional model cho
sen, and whether the sys
tem is more fo
cused on ob
ject re-
use or tenant au
tonomy. Re
view the con
tracts sec
tion on Con
tract Scop
ing for a more
de
tailed dis
cus
sion.
There are four com

mon sce
nar
ios:
1 Inter-Tenant Contracts.
2 Inter-Private Network Contracts.
3 Single Contract Bidirectional forwarding with reverse filter.
4 Single Contract Unidirectional with multiple Filters.
5 Multiple Contracts Unidirectional with single Filter.
ACME Inc., as with most com panies, makes use of shared ser vices such as DNS for name
reso
lu
tion and Active Di
rec
tory for user man agement. These ser vices will be used across
most of their ten ants and so ACME Inc. must allow this traf fic across the whole fab -
ric. Com muni
ca
tion between EPGs that be long to dif
fer
ent ten ants is only al
lowed when
they share the same con tract. To use the same con tract, it will need to be ex ported from
the source ten ant to the ap
propri
ate des
ti
nation tenant. That con tract will ap
pear under
the Imported Con tract sec
tion in the Se
curity Poli
cies of the des ti
na
tion tenant.
A Con
sumed Con tract In
ter
face will be used to as
so
ci
ate an EPG from the des
ti
na
tion
ten
ant with the im
ported con tract.
Note: A contract con

sump tion in
terface rep
re
sents one or more subjects de
fined under
the con
tract. By as
so
ci
at
ing to an inter
face, an end
point group starts consuming all the
sub
jects repre
sented by the in
terface.
In the use case below, EPG-1 in tenant Cisco-1 requires com muni
ca
tion with EPG-2 in
tenant Cisco-2. This is ac
com
plished by uti
liz
ing contact in
ter
faces. In ten
ant Cisco-1
the user will ex

port the in
tended contract in
ter
faces. In ten
ant Cisco-1 the user will ex
-
port the intended contract and se
lect provider to pro vide the contrast to EPG-2. The
user will then con
firm the imported contract in ten
ant Cisco-2 and se lect the con
tract
as con
sumed. To ad
ver
tise the routes from the source VRF to the in
tended VRF, the
user must cre
ate the sub
net within the EPG.
Exporting Contracts Between Tenants
1 Create an Export Contract under security policies.
2 Create the host subnet (default Gateway IP) under EPG1 - subnet scope shared.
3 Add the Contract under EPG1 - contract type provider.
4 Create the host subnet under the bridge domain - subnet scope private/public.
1 Confirm the exported contract is listed under Imported Contracts.
3 Add the Interface Contract under EPG2 - contract type consumed.
4 Create the host subnet (default Gateway IP) under the bridge domain - subnet
scope private/public.
Inter-Private Network Contracts Communication

In the use case below, EPG-1 in VRF Cisco-1 re
quires com
mu
ni
ca
tion with EPG-2 in VRF
Cisco-2. This is ac
complished by uti
liz
ing the sub
net field within the EPG. By cre
at
ing
the sub
net under the EPG and se lecting shared, the route will be leaked to the VRF
noted within the Tenant scoped con tract.
Exporting Contracts Between Private Networks
1 Create the contract under Security Policies - contract scope Tenant.
Single Contract Bidirectional Reverse Filter

This use case is use
ful when imple
ment
ing a con
tract with the option to apply the con -
tract sub
ject in both di
rec
tions and with the option to apply the re
verse fil
ter. This is
the most com

mon of the use cases and al
lows for a sin
gle sub
ject/fil
ter to be im
ple
-
mented with a sin
gle Provider/Con
sumer re
la
tion
ship.
In the use case below, EPG-1 is pro

vid
ing a con
tract with a sub
ject of www and EPG-2
is con
sum ing the con tract. This al
lows the Web Client in EPG-2 to ac
cess the Web
Server in EPG-1. i.e. EPG-1 is pro
viding a ser
vice to EPG-2.
Default Bi-directional Contract with Reverse Filter
Re
sult:
A sin
gle con
tract with (1) Sub
ject and (1) Fil
ter with a sin
gle provider and a sin
gle con
-
sumer. In this ex
am
ple, www.
Sample Contract using Single Bi-directional Subject, Single Filter with Reverse
Filtering Ports Enabled


This use case in
volves im
ple
ment
ing a con
tract with
out the op
tion to apply the con
-
tract subject in both di
rec
tions. When se lect
ing this op
tion the user no longer has the
option to se
lect the re
verse fil
ter op
tion.
In the use case below, EPG-1 is pro vid

ing a con
tract with a sub
ject of icmp and EPG-2 is
consuming the contract. This al
lows the Host in EPG-1 to ac cess the Host in EPG-2 via
icmp. When uti liz
ing a sin
gle subject without the use of Apply Both Di rec
tions, the
user must then con
fig
ure two fil
ters, one in each di
rec
tion.
Single Contract, Single Unidirectional Subject, Multiple Filters
Re
sult:
A sin
gle con
tract with (1) Sub
ject (2) Fil
ters and a sin
gle provider and a sin
gle con
sumer.
In this ex
am
ple, icmp.
Sample Single Contract, Single Unidirectional Subject, Multiple Filters

Multiple Contracts Uni-Directional Single Filter

This use case is useful when implementing a con
tract with the option to apply the con -
tract sub
ject in both direc
tions, and without the op
tion to apply the reverse fil
ter. This
al
lows the end-user the most gran ul
ar
ity when de ploy
ing contracts, but is also the
most com prehensive.
In the use case below, EPG-1 is pro

vid
ing a con
tract with a sub
ject of www and EPG-2
is con
sum ing the contract. This al
lows the Web Client in EPG-2 to ac
cess the Web
Server in EPG-1. That is, EPG-1 is provid
ing a ser
vice to EPG-2.
Multiple Contracts, Unidirectional Subjects, Single Filters
Re
sult:
Two con tracts with (1) Subject (1) Fil

ters. Each con tract will have a sin
gle provider and a
sin
gle con sumer ref er
encing the same con tract. The dif
ference here is that the con -
tract is ex
plic
itly ap
plied in BOTH di rections.
Sample Contract with Single Bi-Directional Subjects and single Filter

249
Layer 4 to Layer 7
Services
Layer 4 to Layer 7 Services 251
Section Content
Understanding Layer 4 to Layer 7 Integration

Service Insertion Design Principles
Applying Service Graphs to EPG Communications
Rendering Service Graphs
Integration Support Function
Services Deployment Guide Reference

Import Device Package
Create a Device
Modify a Device
Create Layer 4 to Layer 7 Service Graph Template
Apply a Service Graph Template to EPGs
Service Graph Monitoring

Monitoring a Service Graph Instance
Monitoring and Resolving Service Graph Faults
Monitoring a Virtual Device
ASAv Sample Configuration

Verify ASAv VM configuration
Remove GW IP from Relevant Fabric Bridge Domains
Create a Layer 4 to Layer 7 Device
Create a Layer 4 to Layer 7 Service Graph Template
Apply the Service Graph Template
Associate the Web and App Bridge domains to the Interfaces
Verifying service node configuration
Display the Access List configuration pushed-down from APIC
Display the Interface configuration pushed-down from APIC
Display the current Interface names
Understanding Layer 4 to Layer 7 Integration
When ACME Inc. designed their new appli

ca
tion, they knew they would need to in
cor
-
po
rate some ser
vices, such as fire
walls, load balancers, IDS/IPS, or other types of
higher-layer ser
vice de
vice.
In tra
di
tional in
fra
struc
ture, ser
vice in
ser
tion would re
quire some in
line de
vice place
-
ment or redirec
tion in order to get traffic to the ser
vice de
vices. As an ex
am
ple, fire
-
walls might be placed di
rectly in
line as a "bump in the line" or might be an ad
junct ser-
vice de
vice near a gate
way. Firewalls are typic
ally config
ured per de
vice by building up
blocks of sta
tic con
fig
ur
a
tion. These sta tic config
ur
a
tion blocks build up over time to
cre
ate a sit
ua
t
ion where the config
ur
ation works, but can be come in
flex
ib
le and frag
ile,
which can cause change man agement to be come chal lenging.
One of the key tech nol

ogy in
no
va
tions in Cisco's Ap pli
ca
tion Cen tric In
frastruc
ture
(ACI) is pol
icy-based man agement of service in
ser
tion through ap pli
cation of a Ser
vice
Graph. Through the rest of this chap ter, we will discuss the high level overview of how
the process works and how ACME will uti lize Ser
vice Graphs for effi
cient man age
ment
of Layer 4 to Layer 7 ser
vices.
The main pur pose of data cen ter fab

ric equipment is fast and ef fi
cient for
ward ing
of traf
fic from ingress to the fab
ric, be
tween phys ic
al and vir
tual hosts within the fab ric
and egress back out of the fab ric. Use
ful in
fra
struc
ture imple
men ta
tions also uti
lize this
fast fabric in a smart way to also inte
grate Layer 4 to Layer 7 ser vices. Some pos si
ble
Layer 4 to Layer 7 services in
clude:
Firewalls
Load balancers
Traffic inspection appliances
SSL offload functions
Application flow acceleration functions
In
te
grat
ing ser
vices with Cisco ACI Ser
vice Graphs will pro
vide ACME with the fol
low
-
ing ben
efits:
Policy based configuration management

254 Layer 4 to Layer 7 Services
Flexible configuration state abstraction through the ACI object model

Integrated configuration management using the APIC GUI, REST API or Python
scripts, all based on a consistent ACI object model
Complex topology modeling with logical flow stitching allowing abstracted links
between multiple service devices
Policy-based provisioning allowing rapid complex topology deployment
Configuration synchronization allowing dynamic workload provisioning and de-
provisioning without manual intervention
Application centric template-based configuration management and object
reuse to shorten infrastructure implementation timelines
Infrastructure multi-tenancy within the fabric and the service devices
Service Insertion Design Principles

With the spine-leaf ar chitec
ture and holis tic fabric management aspects of ACI, traffic
flow to/from ser vice appli
ances is man aged very ef fi
ciently, and thus, the appli
ances
do not need to be placed at any spe cific location within the net work fab
ric. Ser
vices ap -
pli
ances can be phys ic
al or vir
tual and can be con nected to any leaf under man agement
of the ACI fabric. Where ap plic
ab
le, phys ic
al ap pli
ances can also be run in multi
ple con -
text mode, al
low ing multi-ten ant map ping of fab ric for
warding and ten
ant-spe cific ser-
vice con
fig
urations.
With vir
tual ser
vices ap
pli
ances, cur
rently only VMware Hy
per
vi
sors and VLAN trans
-
port modes are sup ported.
Applying Service Graphs to EPG Communications

To allow com mu nica
tions between EPGs within an ACI fab ric, a con tract must be put in
to place. This con tract may take the form of a spe cific consumer/provider re lationship
defined by specified pro to
col and port. There could also be an "allow all" con tract that
al
lows com pletely open com mu ni
ca
tions. This contract essentially controls com muni
-
cations flow between EPGs, and can be ex tended to include ser vice inser
tion via at -
tach ment of a Ser vice Graph to a con tract. The Ser
vice Graph then ties the con tract to
the resolved ser
vice de vice with the policy-based con fig
ur
ation in place.
Rendering Service Graphs

The ACI al
lows users to de
fine a pol
icy using the fol
low
ing ways:
APIC GUI
API with a programmatic tool, such as Python or a RESTful API post through
POSTman
These pol
icy ob
jects can be cre
ated, ma
nip
ul
ated, and reused. As it re
lates to the Layer
4 to Layer 7 ser
vices, these ob
jects ex
press the in
tent of use for that ob
ject in re
la
tion
to the ap
pli
ca
tion.
When an ap
pli
ca
tion pro
file is de
ployed and end
points are at
tached to the leaf
switches, the ser
vice graph ob
jects are then trans
lated into spe
cific de
vice con
fig
ur
a
-
tions that gets pushed the ser
vice nodes through a process called ren
der
ing. The APIC
also sets up the network forwarding path to make sure the cor
rect for
ward
ing ac
tion is
taken to get traf
fic flow to the ser
vice nodes for treat
ment.
This abstracted process of con figur

a
tion man agement works like a policy tem plate
where you can de fine the ex
pected be hav
ior, then link two groups and sub
ject their re
-
la
tion
ship to that pol
icy. This pol
icy can be copied, reused and repack aged as nec es
-
sary.
The ren der

ing in
volves al
lo
ca
tion of the nec es
sary bridge domains, con fig
ur
a
tion of IP
addresses on the firewall and load balancer inter
faces, cre
ation of the VLAN on these
devices to cre
ate the path for the functions, and perfor
mance of all the work nec es
sary
to make sure that the path be tween EPGs is the path de fined in the ser
vice graph.
As is the case with many cus tomers, ACME has a few cookie cut ter templates for fire
-
wall and load-bal ancing ser
vices. Though the initial de
f
in
i
t
ion of these tem plates can be
po
ten tially cumber
some, sub se
quently reusing the tem plates is very straightfor
ward
simply by replac
ing IP addresses, ports, ob
ject-groups, and other val ues.
Integration Support Function

In the ACI model, com muni
cations with the service de
vices is supported by im port
ing
device packages. These de vice pack ages carry the de vice descrip
tion, ex
posed func -
tions, and have con fig
ur
a
tion script content. When the de vice package is imported, the
ACI fabric has full under
standing of what a device can do, how it con nects to the fab
ric,
how to build path for

ward
ing to bring traf
fic into and get traf
fic back from the de
vice,
and how to trans
late pol
icy in
tent to a de
vice-spe
cific con
fig
ur
a
tion. This de
vice pack
-
age is a ven
dor-de
vel
oped pack
age that is read
ily avail
able from the orig
in
al ven
dor or
from Cisco on the soft
ware down
load page.
Some of the De
vice Pack
ages can be down
loaded at: http://
www.
cisco.
com/
c/
en/
us/
solutions/
collateral/
data-center-virtualization/
application-centric-
infrastructure/
solution-overview-c22-732445.
html
De
vice pack
ages from mul
ti
ple ven
dors that lever
age the rich open API that ACI pro
-
vides are avail
able from ven
dors such as Cit
rix and F5 at the time of writ
ing of this
book.
An up to date list
ing of part
ners that lever
age the API are avail
able at: http://
www.
cisco.
com/
c/en/ us/solutions/
unified-fabric/aci_
ecosystem.
html
Services Deployment Guide Reference
The di
ag
ram below shows a high level overview of the Layer 4 to Layer 7 ser
vices work
-
flow when at
tempt
ing to in
te
grate a de
vice.
L4-L7 Workflow
For in
for
ma
tion about deploy vices, see the Cisco APIC Layer 4
ing Layer 4 to Layer 7 ser
to Layer 7 Services Deploy
ment Guide: http:// www. cisco.com/c/en/us/
td/docs/
switches/datacenter/aci/apic/
sw/1-x/L4-L7_Services_ Deployment/ guide/b_
L4L7_
Deploy.
html.
The key sec

tions of the De
ploy
ment Guide are listed below:
Import Device Package

To begin the process of Ser
vice Node in
te
gra
tion, ACME will im
port the ven
-
dor/model-spe
cific de
vice pack
ages, as shown in this sec
tion of the De
ploy
ment
Guide.
Create a Device
Once the de
vice pack
age is im
ported, the de
vices will be added through a process of
cre
at
ing a log
ic
al de
vice clus
ter and creat
ing a re
la
tion
ship between this log
ic
al de
vice
and the physic
al ap
pli
ance. This can be done with a phys ic
al or VM de
vice. The config
u-
ra
tion steps dif
fer slightly for phys
ic
al de
vices and vir
tual de
vices, but are very sim
il
ar.
Modify a Device
You can mod
ify a de
vice's con
fig
ur
a
tion through the GUI as de
scribed in this sec
tion of
the De
ploy
ment Guide.
Create Layer 4 to Layer 7 Service Graph Template

This sec
tion of the De
ploy
ment Guide ex
plains how to cre
ate a ser
vice graph.
Apply a Service Graph Template to EPGs

Once the ap
pli
ca
tion End
point Groups (EPGs) have been cre
ated, the process to apply
a ser
vice graph tem
plate to EPGs can be found in this sec
tion of the De
ploy
ment Guide.
Service Graph Monitoring
Once ACME Inc. has deployed Service Graphs for ap

pli
ca
tion ser
vice in
sertion, the re
-
quire
ment of ob
serv
ing the Ser
vice Graph be
comes an op er
at
ional im
perative. To sup
-
port these ef
forts, there are a few tech
niques that can be em
ployed, in
clud
ing:


Once a service graph is con fig
ured and asso
ci
ated with a con
tract that is at
tached to an
EPG, there are some pri mary mon it
or
ing aspects that should be con sid
ered: State of
the Ser
vice Graph, Func tions of a Graph instance, re
sources al
lo
cated to a function and
pa
ra
meters specif
ied for a function.
To mon
it
or a ser
vice graph in
stance:
3 In the Navigation pane, choose Tenant_Name > L4-L7 Services > Deployed
Graph Instances. The Work pane displays information about the deployed
graph instances, including a list of the deployed service graphs, the associated
contracts, and the current state of the graph policy. A state of "Applied" means
the graph has been applied and is active in the fabric and the service device.
For fur
ther details of the possi
ble states and other relvant states, see the Cisco APIC
e
Layer 4 to Layer 7 Services Deploy
ment Guide at: http://www. cisco.
com/ c/
en/us/td/
docs/switches/ datacenter/ aci/apic/sw/1-x/L4-L7_
Services_ Deployment/ guide/
b_
L4L7_ Deploy/ b_L4L7_ Deploy_ chapter_ 01010.
html#task_
F2BFF7545D9142EFB208C10F5DFBB1B4

To mon
it
or a ser
vice graph's faults:
3 In the Navigation pane, choose Tenant_Name > L4-L7 Services > Deployed
Graph Instances.
4 In the Work pane, choose the Faults tab. The Work pane displays the faults that
are related to the active service graph.
To un
der
stand the faults listed and pos
si
ble res
o
lu
tions, see Ta
bles 1 and 2 in the Cisco
APIC Layer 4 to Layer 7 Ser
vices De
ploy
ment Guide at: http://
www. cisco.
com/
c/
en/
us/td/
docs/switches/datacenter/aci/
apic/
sw/
1-x/L4-L7_ Services_Deployment/
guide/b_
L4L7_Deploy/ b_L4L7_Deploy_
chapter_
01010.html# concept_
307C0CA3EB57469EAF7EF87AAE5A240F

After you con fig
ure a service graph and at tach the graph to an end point group (EPG)
and a con tract, you can mon it
or the vir
tual devices asso
ci
ated with the ser vice graphs
of a ten
ant. Mon it
or
ing the virtual de
vices tells you what de vice clusters are as
so
ci
ated,
which VLANs are con fig
ured for a device cluster, the func
tions in use and the func tion
para
meters passed to the de vices, the statis
tics from the de vices, and the health of the
devices in a de
vice cluster.
To mon
it
or a vir
tual de
vice:
3 In the Navigation pane, choose Tenant_Name > L4-L7 Services > Virtual
Devices. The Work pane displays information about the virtual devices, such as
faults and health scores.
ASAv Sample Configuration
One of the service nodes that ACME Inc. will in te

grate is an ASAv de
ployed in sin
gle-
node, routed FW mode. This ex am ple de
tails the process they fol
lowed to con
fig
ure the
ASAv fire
wall vir
tual ser
vice ap
pli
ance as a sin gle node in routed mode. As a routed
node, the ASAv will become the de fault gate
way for the hosts in the 2 EPGs that are
con
nected by the contract that this ser
vice graph gets as
so
ci
ated to.
The high level steps are:
Create Logical and Concrete device clusters

Define a firewall ruleset/graph
Deploy the graph between 2 EPGs
Verify ASAv VM configuration

The first set of steps performed is to ver ify ASAv VM con
fig
ur
a
tion and up
load the ASA
de
vice package (or verify if it has al
ready been done)
1 Log in to the ASAv VM.
2 Enter enable mode.
3 Issue the following command:
# show ip int brief
4 Verify that the ASAv has a correct IPv4 address on the management 0/0
interface.
5 Verify connectivity from the APIC to the management 0/0 interface of the
ASAv.
a. SSH to the APIC cluster IP address
b. Issue the following command:
# ping {ASAv management 0/0 interface}

c. The return response should be similar to the following example:
64 bytes from 172.16.10.10: icmp_seq=1 ttl=64 time=0.050 ms
If the re
sponse is dif
fer
ent, then there is likely some sort of con
nec
tiv
ity
issue. Ad
dress the con
nec
tiv
ity prob
lems be
fore mov
ing on.
6 In the APIC GUI, on the menu bar, choose Tenant > Tenant_Name.
7 In the Navigation Pane expand Tenant_Name > L4-L7 Services >

Packages > L4-L7 Service Device Types.
8 If an ASA device package is not listed, then perform the following actions:
a. Right click on the Device Types.
b. Choose Import Device Package.
c. Follow the prompts to upload the device package.
Remove GW IP from Relevant Fabric Bridge Domains

Once the ASAv pack age and VM are verif
ied, the next step is to re
move SVI/GW IP ad-
dresses from the fabric Bridge Do
mains so the Layer 3 routed fire wall can be
come the
de
fault gate
way.
To re
move the rout
ing de
fault gate
way func
tion on the EPG1 and EPG2 bridge do
mains:
3 In the Navigation pane, choose Tenant_Name > Networking > Bridge Domains
> BD-EPG1.
4 In the Work pane, perform the following actions:

a. For the L2 Unknown Unicast radio buttons, click Flood.
b. For the L3 Unknown Multicast Flooding radio buttons, click Flood.
c. Uncheck the Unicast Routing check box.
d. Click Submit.
Re
peat this process for the Bridge Do
mains of the af
fected EPGs.


Per
form the fol
low
ing steps to add log
ic
al and con
crete de
vice clus
ters to a ten
ant:
3 In the Navigation Pane choose Tenant_Name > L4-L7 Services.
4 In the Work pane, click Create a L4-L7 Function Profile.
5 In the Create a L4-L7 Function Profile dialog box, perform the following
actions:
a. Enter a name that is relevant to the tenant and fuction, such as "TXX-ASAv-
FP".
b. In the Profile Group drop-down list, choose Create Function Profile Group.
c. In the Create L4-L7 Services Function Profile Group dialog box, perform the
following actions:
i. Enter a meaningful name, such as "TXX-FP-Group".
ii. Click SUBMIT.
d. In the Profile drop-down list, choose WebServiceProfileGroup

or WebPolicyForRoutedMode.
e. In the Feature and Parameters section, choose the All Parameters tab. You
will configure IP addressing under Interface Related Configuration for both
external and internal interfaces (externalIf and interalIf).
f. Expand Interface Related Configuration for externalIf.
g. Expand Interface Specific Configuration.
h. Double-click IPv4 Address.
i. Enter: ipv4 address in a.b.c.d/m.n.o.p format
j. Click Update.
k. Repeat steps 4f - 4j as needed.
6 Choose L4-7 Services > Function Profiles > ALL PARAMETERS.
7 Verify the external and internal interfaces IPv4 addresses.
The fol
low
ing steps will cre
ate a Layer 4 to Layer 7 de
vice:

3 In the Navigation pane, choose Tenant_Name > L4-L7 Services.
4 In the Work pane, click Create a L4-L7 virtual device.
5 In the Create L4-L7 Devices dialog box, perform the following actions:
a. Enter a meaningful name: Txx-ASAv-Cluster
b. Device Package: CISCO-ASA-1.1 Model: ASAv
c. Mode: Single Node
d. Function Type: Goto
e. Connectivity VMM Domain: Txx-vCenter
f. APIC to Device: Out of Band
g. Credentials: {uid/pwd}
h. Under Device 1, specify the following values:
i. Management IP address: ASAv IP address
ii. Management Port: https
iii. VM: Tenant ASAv Controller (in the dropdown box)
iv. Virtual Interfaces: Create two entries; click + twice; enter interface values
accordingly:
1. Name: GigabitEthernet0/0 vNIC: Network Adapter 2 Direction:
provider
2. Name: GigabitEthernet0/1 vNIC: Network Adapter 3 Direction:
consumer
v. Click UPDATE after each entry.

vi. Click NEXT.
vii. Click FINISH.
Ver
ify the Log
ic
al and Con
crete De
vice Clus
ters have been con
fig
ured:
3 In the Navigation Pane choose Tenant_Name > L4-L7 Services > L4-L7 Devices.
4 Expand both TXX-ASAv-Cluster and TXX-ASAv-Cluster_Device_1 to view the

logical and physical interfaces.
5 Select TXX-ASAv-Cluster_Device_1 to see a graphic view of the concrete

device.
Note This com

pletes the Log
ic
al and Con
crete de
vice cre
ation.
Create a Layer 4 to Layer 7 Service Graph Template

3 In the Navigation pane, choose Tenant_Name > L4-L7 Services > L4-L7 Service
Graph Templates.
4 In the Work pane, choose Actions > Create a L4-L7 Service Graph Template.
5 In the Create L4-L7 Devices dialog box, perform the following actions:
a. In the Name field, enter TXX-ASAv-L3-Routed-Template.
b. In the Type drop-down list, choose Single Node - Firewall in Routed Mode.
c. In the Device Function drop-down list, choose CISCO-ASA-1.1/Firewall.
d. In the Profile drop-down list, choose TXX-ASAv-FP, which is the functional
profile you created previously.
e. Click Submit.
6 You can verify that the template was created successfully by expanding
Tenant_Name > L4-L7 Services > L4-L7 Service Graph Templates > Txx-ASAv-
L3-Routed-Template > Function Node - Firewall.
Apply the Service Graph Template

3 In the Navigation pane, choose Tenant_Name > L4-L7 Services > L4-L7 Service
Graph Templates.
4 Right click Txx-ASAv-L3-Routed-Template and choose Apply L4-L7 Service

Graph Template.
5 In the Apply L4-L7 Service Graph Template to EPGs dialog box, perform the
following actions:
a. In the Consumer EPG / External Network drop-down list, choose EPG1.
b. In the Provider EPG / External Network drop-down list, choose EPG2.
c. In the Service Graph Template drop-down list, choose Txx-ASAv-L3-

Routed-Template.
d. For the Contract radio buttons, click Create A New One.
e. Check the No Filter (Allow All Traffic) check box.
f. In the Contract Name field, enter TXX-EPG1-to-EPG2.
g. Click Next.
h. In the L4-L7 Devices drop-down list, choose Txx-ASAv-Cluster.
i. In the Features and Parameters section, choose the All Parameters tab.
j. Double click Device Config > Interface Related Configuration externalIf >
Access Group > Inbound Access List.
k. In the Value drop-down list, choose access-list-inbound.
l. Click Update.
m. Expand Device Config > Interface Related Configuration externalIf >
Interface Related Configuration to verify the IP address assignment. If the
mask is missing, the configuration will not push to the ASA.
n. Double click Device Config > Interface Related Configuration internalIf >
Access Group > Inbound Access List.
o. Click Reset. This unassigns the inbound access list from the internal
interface. This inbound access list is not desirable for the lab traffic-flow.
p. Expand Device Config > Interface Related Configuration internalIf >
Interface Specific Configuration to verify the IP address assignment with
the mask.
q. Click Finish. The Devices Selection Polices folder and Deployed Graph
Instances folder are now populated.
6 Choose TXX-EPG1-to-EPG2-TXX-ASAv-L3-Router-Template-TXXProduction.
You can see that the Consumer and Provider EPGs are associated with the EPG1
and EPG2 server EPGs.
Associate the Web and App Bridge domains to the Interfaces

1 Expand Devices Selection Policies.
2 Expand TXX-EPG1-to-EPG2-TXX-ASAv-L3-Router-Template-Firewall.
3 Select External.
4 Assign the Bridge Domain to BD-EPG1.

5 Repeat the process to assign the Bridge Domain to BD-EPG2.
6 Click SUBMIT.
7 Expand Layer 4 to Layer 7 Service Graph Templates.
8 Expand TXX-ASAv-L3-Router-Template.
9 Expand Function Node Firewall > choose external.
10 Filters: common/default.
11 CTX Terms: TXX-ASAv-L3Routed Template/T1/Out.
12 Click SUBMIT.
13 Repeat the process for internal.
14 Filters: common/default.
15 CTX Terms: TXX-ASAv-L3Routed Template/T1/In.
16 Click SUBMIT.
The ASAv will now have IP ad dresses asigned in the Service Graph Pro
file. This can be
ver
if
ied by going into the ASAv VM console and issue the fol
low
ing com
mand:
# show ip int brief
Verifying service node configuration

Once the De vice has been in
tegrated and the Ser
vice Graph has been config
ured and
as
so
ci
ated, the re
sult
ing con
fig
ura
tion pushed to the ser
vice node can be ver
if
ied by
ple show com
sim mands on the real ser
vice node de
vice.
Display the Access List configuration pushed-down from APIC

SSH to the ASAv ser
vice node for ac
cess list val
id
a
tion. Issue the fol
low
ing com
mand:
# show run | grep access
This will show the ac

cess list con
fig
ur
a
tion and you can re
late that to the con
fig
ur
a
tion
that was done in the APIC.
Display the Interface configuration pushed-down from APIC

SSH to the ASAv ser
vice node for in
ter
face con
fig
ur
a
tion val
id
a
tion. Issue the fol
low
ing
com
mand:
# show run interface
This will show the inter

face con
fig
ur
a
tion where the IP ad
dress con
fig
ur
a
tion was
pushed from the APIC.
Display the current Interface names

SSH to the ASAv ser
vice node for in
ter
face name con
fig
ur
a
tion val
id
a
tion. Issue the fol
-
low
ing command:
# show nameif
This will show the in

ter
face names pushed from the APIC and will show the re
lated in
-
ter
face names to the log
ic
al in
ter
face names that were con
fig
ured in the APIC above.
269
Health Scores
Health Scores 271
Section Content
Understanding Health Scores
Understanding Faults
How Are Health Scores Calculated
Health Score Use Cases

Using Health Scores for Proactive Monitoring
Using Health Scores for Reactive Monitoring
Health Scores 273
Understanding Health Scores
ACME's Op er
at
ions team has been chal lenged on a reg
ul
ar basis to answer basic ques
-
tions re
gard
ing the cur
rent sta
tus, per
for
mance, and avail
abil
ity of the sys
tem they are
re
spon
si
ble for op
er
at
ing. To ad
dress these chal
lenges they can now uti
lize the Ap
pli
-
cation Cen
tric In
fra
struc
ture (ACI), which pro vides Health Scores that make in for
ma-
tion on sta
tus, perfor
mance, and availabil
ity readily avail
able. It is worth not
ing that
while pro
vid
ing such an
swers may be easy as it re
lates to an in
de
pen
dent de
vice or link,
this in
for
ma
tion by it
self is of lit
tle to no value with
out ad
di
tional data on its ef
fect on
the over
all health of the net
work. To man
ua
lly col
lect and cor
re
late in
for
ma
tion would
have previ
ously been a long and te dious task, but with health scores, data throughout
the fab
ric is col
lected, computed, and cor
re
lated by the APIC in real time and then pre
-
sented in an un derstand
able for
mat.
Traditional network mon it

oring and man agement sys tems attempt to pro vide a model
of infra
structure that has been pro vi
sioned, and de scribe the rela
tionship between the
vario
us de vices and links in at tempt to pro vide a cor
rela
tion. The ob ject model at the
heart of ACI is in herent to the infrastruc
ture, and there fore the cur rent status of all of
the ob jects includ
ing links, devices, their re
la
tion
ships, as well as the real time sta tus of
their utiliza
tion, can be rep re
sented in a single consol
id
ated health score.
The visib
il
ity provided by health scores give the op er
at
or a quick at-a-glance as sess -
ment of the cur rent sta
tus of the entire system, or any sub set of the sys
tem. This visi-
bil
ity has a num ber of practi
cal use cases, and in this chap ter we will clas
sify these use
cases as re active and proac tive. ACI also pro vides the flexib
il
ity to mon
it
or some as -
pects of how the health scores are cal culated, and how var io
us faults impact the cal
cu-
la
tion of the health score.
Most objects in the model will have an as soci

ated health score, which can be found
from the Dash board or Policy tabs of the object from the GUI. Addi
tion
ally, all health
scores are in
stan
ti
ated from the healthInst class and can be ex
tracted through the API.
274 Health Scores
Tenant Health Score
In a re
active ca
pacity, ACI health scores pro vide a quick check as to whether an issue
being reported is con firmed in a degradation of the health score. If so, the root cause
of the issue can be found by ex ploring the faults and how these get rolled up in the
larger model. Health scores also pro vide a real-time corre
la
tion in the event of a fail
ure
scenario, immedi
ately provid
ing feedback as to which ten ants, appli
ca
tions, and EPGs
are impacted by that failure.
As a example, if you nav

ig
ate to the ap
pli
cation profile it has a Health tab. In this tab is
a tree that will show the vario
us ob
jects in a tree form to re veal faults.
Health Scores 275
Object with a fault
Proactively, ACI health scores can help identify poten

tial bottle
necks in terms of hard -
ware re sources, bandwidth uti
liza
tion, and other ca pacity planning ex
ercises. Oper
a-
tions teams also stand a bet ter chance of iden ti
fy
ing issues before they im pact cus-
tomers or users.
Ide
ally, the health of all appli
ca
tion and infra
struc
ture components would always be at
100%, how ever, this is not al
ways real
is
tic given the dy
namic nature of data cen
ter en
-
vi
ronments. Links, equip ment, and end points have fail
ures. In
stead the health score
should be seen as a met ric that will change over time, with the goal of in creas
ing the
av
er
age health score of a given set of com
po
nents over time.
Health Scores 277
Understanding Faults
From a man agement point of view we look at the Ap

pli
ca
tion Pol
icy In
fra
struc
ture Con
-
troller (APIC) from two perspectives:
1 Policy Controller - Where all fabric configuration is created, managed and

applied. It maintains a comprehensive, up-to-date run-time representation of
the administrative or configured state.
2 Telemetry device - All devices (Fabric Switches, Virtual Switches, integrated

Layer 4 to Layer 7 devices) in an ACI fabric report faults, events and statistics to
the APIC.
Faults, events, and sta

tis
tics in the ACI fab
ric are rep
re
sented as a col
lec
tion of Man
-
aged Ob jects (MOs) within the over all ACI Object Model/Man age
ment Informa tion
Tree (MIT). All ob
jects within ACI can be queried, in clud
ing faults. In this model, a fault
is rep
re
sented as a muta
ble, state
ful, and persis
tent MO.
Fault Lifecycle
278 Health Scores
When a spe
cific con
di
tion oc
curs, such as a com
po
nent fail
ure or an alarm, the sys
tem
cre
ates a fault MO as a child object to the MO that is pri
marily as
so
ci
ated with the fault.
For a fault ob
ject class, the fault con
ditions are de
fined by the fault rules of the par
ent
ob
ject class. Fault MOs appear as regul
ar MOs in MIT; they have a par ent, a DN, RN,
and so on. The Fault code is an al
phanumeral string in the form FXXX. For more in
ic -
for tion about fault codes, see the Cisco APIC Faults, Events, and Sys
ma tem Mes
sages
Manage
ment Guide.
The fol
low
ing ex
ample is a REST query to the fab
ric that re
turns the health score for a
ten
ant named "3tier
app":
https://hostname/api/node/mo/uni/tn-3tierapp.xml?query-target=self&rsp-subtree-
include=health
The fol
low
ing ex
am
ple is a REST query to the fab
ric that re
turns the sta
tis
tics for a ten
-
ant named "3tier
app":
https://hostname/api/node/mo/uni/tn-3tierapp.xml?query-target=self&rsp-subtree-
include=stats
The fol
low
ing ex
am
ple is a REST query to the fab
ric that re
turns the faults for a leaf
node:
https://hostname/api/node/mo/topology/pod-1/node-103.xml?query-target=self&rsp-
subtree-include=faults
As you can see, MOs can be queried by class and DN, with prop
erty fil
ters, pag
i
na
tion, and
so on.
In most cases, a fault MO is au

tomatic
ally cre
ated, es
ca
lated, de-es
ca
lated, and deleted
by the sys
tem as specific con
di
tions are detected. There can be at most one fault with a
given code under an MO. If the same con di
tion is de
tected mul ti
ple times while the
corre
spond ing fault MO is active, no addi
tional in
stances of the fault MO are cre ated.
In other words, if the same con di
tion is detected mul ple times for the same af
ti fected
object, only one fault is raised while a counter for the re currence of that fault will be
in
cre
mented. A fault MO remains in the system until the fault con
di
tion is cleared. To
re
move a fault, the con
di
tion rais
ing the fault must be cleared, whether by con fig
ur
a
-
Health Scores 279
tion, or a change in the run time state of the fab

ric. An ex
cep
tion to this is if the fault is
in the cleared or retained state, in which case the fault can be deleted by the user by
ac
knowl
edg
ing it.
Sever
ity pro
vides an in
di
ca
tion of the es
ti
mated im
pact of the con
di
tion on the ca
pa
bil
-
ity of the sys
tem or com
po
nent to pro
vide ser
vice.
Pos
si
ble val
ues are:
Warning (possibly no impact)

Minor
Major
Critical (system or component completely unusable)
The cre
ation of a fault MO can be trig
gered by in
ter
nal processes such as:
Finite state machine (FSM) transitions or detected component failures

Conditions specified by various fault policies, some of which are user
configurable
For ex
am ple, you can set fault thresh
olds on sta
tis
ti
cal mea
sure
ments such as health
scores, data traf
fic, or tem
per
at
ures.
Health Scores 281
How Health Scores Are Calculated
Health scores exist for systems and pods, ten ants, managed ob jects (such as switches
and ports), as well as an over
all health score for the overall sys
tem. All health scores are
cal
culated using the num ber and im portance of faults that apply to it. Sys
tem and pod
health scores are a weighted av erage of the leaf health scores, di vided by the total
num ber of learned end points, mul ti
plied by the spine coef
fi
cient which is derived from
the num ber of spines and their health scores. In other words:
Health Score calculation
Ten
ant health scores are sim
il
ar, but con
tain health scores of log
ic
al com
po
nents
within that tenant. For ex
am
ple, it will only be weighted by the end points that are in
-
cluded in that ten
ant.
You can see how all of these scores are aggre

gated by looking at how man aged object
scores are cal
cu
lated, which is di
rectly by the faults they have asso
ci
ated with them.
Each fault is weighted de
pend
ing on the level of im
por
tance. Crit
ic
al faults might have a
282 Health Scores
high fault level at 100%, while warn

ings might have a low fault level at only 20%. Faults
that have been iden
ti
fied as not im
pact
ing might even be re
as
signed a per
cent
age value
of 0% so that it does not af
fect the health score com
pu
ta
tion.
Luck
ily there is re
ally no need to un
der
stand the cal
cu
la
tions of the health scores to
use them ef
fec
tively, but there should be a basic un
der
stand
ing of whether faults
should have high, medium, low, or none fault lev
els. Though faults in ACI come with
de
fault val
ues, it is pos
si
ble to change these val
ues to bet
ter match your en
vi
ron
ment.
Keep in mind, be
cause of the role-based ac
cess con
trol, not all ad
min
is
tra
tors will be
able to see all of the health scores. For ex
am
ple, a fab
ric admin will be able to see all
health scores, but a ten
ant admin would only be able to see the health scores that per-
tain to the ten
ants to which they have ac
cess. In most cases, the ten
ant admin should
be able to drill into the health scores that are vis
ib
le to them, but it is pos
si
ble a fault
may be oc cur
ring that is af
fecting more than that one ten ant. In this case the fab
ric ad
-
minis
tra
tor may have to start trou bleshooting. The ten ant and fab ric ad
mins may also
see health scores of any layer four through seven de vices, such as firewalls, load bal
-
ancers, and intru
sion pre ven tion/detec
tion sys tems. These, along with faults within
our VMM do mains will all roll up into our tenant, pod, and over all sys
tem health scores.
For more in formation on how to use faults, see the Troubleshoot

ing Cisco Ap
pli
ca
tion
Centric In
frastruc
ture book: http://
datacenter.github.
io/
aci-troubleshooting-book/.
Health Scores 283
Health Score Use Cases
Using Health Scores for Proactive Monitoring

While ACME ad
min
is
tra
tors have tra
di
tion
ally spent a lot of time re
act
ing to is
sues on
the net
work, ACI health scores will allow them to start pre
vent
ing is
sues. Health scores
not only act as in
di
ca
tors of faults, they are es
sen
tially base
lines to which you can make
compar
isons later. If you see that one of the leaf switches is at 100% (green for good)
one week, and the next week the leaf is show ing a warning, you can drill down to see
what changed. In this sce
nario, it is pos
si
ble the links are over
sub
scribed and so it can
be time to either move some of of the work load to an other leaf or maybe to add more
bandwidth by con nect
ing more cables. Since it is still only a warning, there is time to
re
solve the issue be
fore any bot
tle
necks on the net work are no ticeable.
The same sce nario can ob served with a load bal

ancer or fire
wall that is getting over-
loaded. In these cases adding an other load bal
ancer, or fire
wall, or maybe even op ti
-
mizing the rules may be needed to make traf fic flow more ef fi
cient. As shown in the
above examples, this baselin
ing method can be used as a ca pac
ity planning tool.
Other ways health scores can be used to proac tively mon itor your ACI en
vi
ronment are
by giv
ing vis
ib
il
ity of cer
tain com ponents to other groups. Since you can ex port the
scores and faults, it is pos si
ble to send these no ti
fi
cations to ap
pli
ca
tion own ers,
VMware ad minis
tra
tors, Database Adminis
tra
tor, and so on. This would pro vide mon i-
tor
ing of the en vi
ron
ment across the net work that has not pre vi
ously been available
and which is not able to be re trieved by any other means.
Using Health Scores for Reactive Monitoring

Reactively, health scores can be used to di ag
nose prob lems with the ACI fab ric. Upon
noti
fi
cation that a health score has been de graded, an op er
ator can use the GUI to eas
-
ily navig
ate the rela
tion
ships and faults that are con tributing to that health score. Once
the root cause faults have been iden ti
fied, the fault it
self will contain in
for
ma
tion about
possible re
me di
at
ion steps.
Most ob jects will have a Health tab which can be used to ex plore the re
la
tionship be
tween
ob
jects, and their asso
ci
ated faults. This pro
vides the abil
ity to double-click to root cause.
285
Monitoring
Monitoring 287
Section Content
Proactive Monitoring - Tenant and Fabric Policies

Stats Collection Policies
Stats Export Policies
Diagnostics Policies
Call Home/SNMP/syslog
Event Severity and Fault Severity Assignments
Fault Lifecycle Policies
TCAM Policy Usage
Create TCAM Policy Monitor
TCAM Prefix Usage
Health Score Evaluation Policy
Communication Policy
Proactive Monitoring - Infrastructure

Monitoring APICs
CPU utilization and Memory
Disk Utilization
Physical and Bond Interface Statistics
APIC Fan Status
Temperature Status
Power Supply Status
Monitoring Leaf Switches
Monitoring Switch CPU Utilization
Monitoring Switch Memory Utilization
Monitoring File System Health
Monitoring CoPP (Control Plane Policing) Statistics
Physical Interface Statistics and Link State
Module Status
Switch Fan Status
288 Monitoring
Power Supply Status

LLDP Neighbor Status
GOLD Diagnostic Results
Proactive Monitoring Use Cases

Monitoring Workload Bandwidth
EPG Level Statistics
Reactive Monitoring
Reactive Monitoring Tools
Switch Port Analyzer (SPAN)
Traceroute
Atomic Counters
Traffic Map
Enhanced Troubleshooting Wizard
IPing
Audit Logs
Reactive Monitoring Use Cases

Loss of Connectivity to Endpoint
Users Report that an Application Running in the Fabric is Slow
Monitoring 289
Proactive Monitoring -
Tenant and Fabric Policies
Proac
tive mon
it
or
ing is a very im
por
tant piece of the net
work ad
min
is
tra
tor's job, but
is often ne
glected be
cause putting out fires in the net
work usu ally takes pri
or
ity. How
-
ever, since the APIC makes it in
cred
ib
ly easy to gather sta
tis
tics and per form analyses,
this will save net
work ad
min
is
tra
tors both time and frus
tra
tion. Since sta
tis
tics are
gath
ered au
to
matic
ally and policies are used and can be re-used in other places, the
human error and effort is min
im
al.
Sta
tis
tics gathering has been a some what man ual and even re source in tensive process
for ACME in the past. Even when they have used tools to gather data on layer one
through seven de vices, it has still been nec es
sary to man ua
lly spec ify which de vices are
to be mon it
ored and how they should be mon it
ored. For ex am ple, SNMP and a third
party tool may have been used to mon it
or the CPU of switches or band width uti liza tion
on ports, but they strug gled with en ter ing correct SNMP in for
ma tion on each de vice,
or often for got to add a new de vice to their Net work Mon it
oring Sys tem (NMS). ACI
provides an APIC which will do all of the sta tis
tics gathering, and pro vides the abil ity to
proactively mon it
or your en tire en vi
ron ment with out all of the has sle of main tain ing a
third party mon itor
ing tool.
The APIC, whether ac cessed through the GUI, CLI, or API, can be used to drill into any
of the com ponents and pro vides the abil ity to click on a Stats tab to display on-de mand
sta
tis
tics, but more im por tantly it enables the setup of poli cies to keep per sis
tent data
to anal
yze trends in the en vi
ronment, as well as to trou bleshoot or pre dict any is sues
that may be aris ing. When plan ning to move an ap pli
ca
tion from a legacy net work to
the ACI in fra
struc ture, it is sensi
ble to start by test ing before going straight to pro duc-
tion. Add test VMs to port groups on ei ther a DVS or AVS as soci
ated with the APIC,
and add phys ic
al test servers to VPCs on the leaf switches. This could also be in a test -
ing ten ant which is com pletely sep ar
ate from the pro duction envi
ronment. At this
point the APIC is al ready gath er
ing statis
tics for the VMM do main and the phys ic
al de
-
vices. The next step is to con fig
ure a pol icy for trend analy sis.
There are four dif fer

ent scopes for sta
tis
tics gath
er
ing: Com mon or Fab ric Wide, Fab-
ric, Tenant, or Access. A Fab
ric Wide policy would be cre ated as a default pol
icy to be
applied to all ten
ants. However, to over
ride that pol
icy for a par
tic
ul
ar tenant, the ten
-
290 Monitoring
ant pol
icy will over
ride the Fab
ric pol
icy. In the fol
low
ing test
ing ex
am
ple, a Ten
ant
pol
icy is cre
ated to gather sta
tis
tics. Even if this ten
ant is shared with other ap
pli
ca
-
tions, cus
tomers, test cases, it will pro
vide a real world ex
am
ple of how the ap
pli
ca
tion
will be
have in a pro
duc
tion en
vi
ron
ment.
Create Tenant Monitor policy

To cre
ate a ten
ant mon
it
or
ing pol
icy:
3 In the Navigation pane, choose Tenant_Name > Monitoring Policies.
4 In the Work pane, choose Actions > Create Monitoring Policies.
5 In the Create Monitoring Policies dialog box, perform the following actions:
a. In the Name field enter a name for the Monitoring Policy.
b. Click Submit.
6 In the Navigation pane, choose Tenant_Name > Monitoring Policies >

Policy_Name to display the following information:
Callhome, SNMP, and syslog
Event Severity Assignment Policies

Click ing on Stats Col lec
tion Policies will dis play the de fault reten
tion periods and
admin states (En abled/Dis abled) for ALL Mon itored Objects. Most likely the de faults
will be kept, but a dou ble click on them will change the admin state or re ten tion peri
-
ods. For ex ample, to have it poll a com po nent every 5 min utes, but be retained for 2
hours, just click on the pol icy that spec if
ies a 5 minute gran ul
ar
ity and change the re -
tention pe riod to 2 hours. It is sim il
arly pos sible to change the poli cies for spe cific
Mon it
or
ing Ob jects. A mon it
or
ing ob ject tells the APIC which com ponents to gather
Monitoring 291
sta
tis
tics about. For ex
ample, to change the in
forma
tion gathered for Bridge Do
mains,
use the Bridge Do main (infra.
RSOInfraBD) Mon it
or
ing Ob
ject.
To add mon
it
or
ing ob
jects:
3 In the Navigation pane choose Tenant_Name > Monitoring

Policies > Monitoring Policy_Name > Stats Collection Policies
a. Click on the Pencil icon to edit the Monitoring Objects.
b. Put a checkmark next to the Monitoring Objects to be included, and remove
any checkmarks next to Monitoring Objects to be left out.
c. Click Submit.
For this ex
ample, changes might be made to Mon it
oring Ob ject poli
cies for Tenant,
VXLAN Pool, Leaf Port, and/or Taboo Con tract. There are sev eral options and this will
all de
pend on what is impor
tant to monit
or in the en vi
ronment. Click on the pull down
menu to select a mon
it
or
ing ob
ject and add a retention policy to it.
To add a pol
icy to a Mon
it
or
ing Ob
ject:
3 In the Navigation pane choose Monitoring Policies > Monitoring

Policy_Name > Stats Collection Policies.
4 In the Work pane, in the Stats Collection Policy dialog box, perform the
following actions:
a. Select the Monitoring Object.
b. Click + to add the policy.
c. Select the granularity with which it is to poll.
d. Leave the state as inherited to stick with the defaults as set for ALL, or
explicitly select enabled or disabled.
e. The retention policy may either be inherited or explicitly specified as enabled
or disabled as well.
f. Click Update.

292 Monitoring

It is de
sir
able to col
lect these on
go
ing sta
tis
tics as well as to see how this data be
haves
over time. Use the Stats Ex
port Poli
cies op
tion in the left nav
ig
a
tion pane, lo
cated
under the mon it
or
ing pol
icy. Much like the Stats Col lec
tion Poli
cies, it is possi
ble to
cre
ate a pol
icy for ALL mon it
or
ing ob
jects, or se
lect specific monit
or
ing ob jects and
spec
ify where this in
for
ma
tion will be saved.
To cre
ate a Stats Ex
port Pol
icy:
3 In the Navigation pane choose Tenant_Name > Monitoring

Policies > Monitoring Policy_Name > Stats Export Policies.
4 In the Work pane, in the Stats Export Policy dialog box, perform the following
actions:
a. Select ALL or a specific monitoring object from the drop-down list.
b. Click + to add the policy.
c. Now define the Stats Export Policy in the wizard.
d. Choose either JSON or XML as the format. There's really no difference other
than personal preference, or it may be dictated by the tool used to read it.
e. Choose to compress it using GZIP, or leave it uncompressed.
f. Click + under Export Destinations to specify a server where this information
is to be collected. Another wizard will pop up to enable specification of the
protocols and credentials used to connect to this server.
g. Click Ok.
5 Click Submit.
Next are the di agnos
tics poli
cies in the nav ig
ation pane on the left. This is a re ally slick
fea
ture that allows the setup of di ag
nostics test for the Mon it
oring Ob jects that were
specif
ied in the Stats Collec
tion Poli cies. Next to the Mon it
oring Ob ject is the Pen cil
but
ton which en ables se
lec
tion of the mon it
oring ob
jects to be con figured with di ag-
nostics poli
cies. There are two dif fer
ent kind of policies for con figur
a
tion - Boot-Up di -
ag
nos tics or Ongo
ing di
agnos
tics.
Monitoring 293
To con
fig
ure di
ag
nos
tic poli
cies:
2 In the Navigation pane choose Tenant_Name > Monitoring Policies > default
> Diagnostics Policies.
3 In the Work pane, in the Diagnostic Policies dialog box, perform the following
actions:
Note: Click on the Pencil Icon and put checks next to the Monitoring Objects
which diagnostics tests are to be added to.
a. Select one of the Monitoring Objects.
b. Click + to add an Object.
i. select either Boot-Up or Ongoing.
ii. Boot-Up runs the tests while the devices are booting, and Ongoing will run
the tests as often as specified within the wizard.
iii. In the wizard give it a name and select the admin state.
iv. There are five different diagnostics tests available: ASIC, CPU, Internal
Connectivity, Peripherals, and System Memory. Double-click on each to
obtain the option of specifying no tests, full tests, or recommended tests.
v. Click Submit.
The di
ag
nostics found here can be useful in find
ing failed com
po
nents be
fore they
cause major is
sues within your en
vi
ron
ment.
Call Home/SNMP/syslog
There are a few dif fer
ent ways to setup no ti
fi
ca
tion or alert poli cies. The Call
Home/SNMP/sys log policy will allow alert ing to be con fig ured in a flexib
le man ner.
Cisco Call Home is a fea ture in many Cisco prod ucts that will pro vide email or web-
based no ti
fi
cation alerts in several different formats for crit ic
al events. This allows ad-
minis
tra
tors to re solve issues before they turn into out ages. SNMP or sys log poli
cies
can also be used with cur rent noti
fi
cation systems. Dif fer
ent log ging lev
els may be se -
lected for no ti
fi
cations and alert lev els spec if
ied for Mon itor
ing Ob jects from which
alerts are to be re ceived.
294 Monitoring
Event Severity and Fault Severity Assignments

Event and fault sever
it
ies can be changed for events raised by Mon
it
or
ing Ob
jects.
Most likely, the de
fault sever
ity as
sign
ments for Events and Faults will be kept, but
there are exam
ples where an ACI ad min
is
tra
tor may de
cide the event or fault is more
or less se
vere than the de
fault value. For ex
am
ple, if only crit
ic
al faults are being no
ti
-
fied, but there is a major fault you'd also like to be no
ti
fied about im
me
di
ately, you can
change the sever
ity for that par
tic
ul
ar fault code.
3 In the Navigation pane, choose Tenant_Name > Monitoring Policies

> Monitoring_Policy > Fault Lifecycle Policies.
4 In the Work pane, in the Fault Severity Assignment Policies dialog box,
perform the following actions:
a. Select a Monitoring Object, which will dictate the fault codes for which you
are changing the fault severity.
b. Click + to add an Object.
c. Select the particular fault code for which severity is to be changed.
d. Select the severity: Cleared, Critical, Major, Minor, Squelched, Inherit,
Warning, Info.
Note: Squelched gives it a weight of 0%, meaning it does not affect health
scores.
5 Click Update.
The Event Sever

ity As
sign
ment Poli
cies are con
fig
ured in the same way.

Fault Life
cycle is the term Cisco uses to de scribe the life of a fault. Once a fault is de -
tected it is in the "soak ing" state. After a certain amount of time, re ferred to as the
"soaking in
ter val" it will move on to the "raised" state. "Raised" means the fault is still
pre
sent after the soak ing in ter
val. After the fault clears it's in a state called "raised
clear
ing." It is only in this state briefly and moves on to the "clear ing time" state. It re
-
mains in the "clear ing time" state for the amount of time spec if
ied in the "clear
ing in-
ter
val." Lastly it moves on to the "re tain
ing" state and does not get re moved until the
end of the "re taining in
ter val."
Monitoring 295
To change Fault Life

cy
cle In
ter
vals:
3 In the Navigation pane, choose Tenant_Name > Monitoring Policies >

Monitoring_Policy > Fault Lifecycle Policies.
4 In the Work pane, in the Fault Lifecycle Policies dialog box, perform the
following actions:
a. Select a Monitoring Object, which will dictate the fault codes for which you
are changing the default intervals.
b. Click +.
c. Specify times for the Clearing Interval, Retention Interval, and Soaking
Interval (all in seconds).
Note: The default for the Clearing Interval is 120 seconds; the Retention
Interval is 3600 seconds; and the Soaking Interval is 120 seconds.
At this point there will be a fully work ing tenant monit

oring pol
icy. ACME will have
other poli
cies to con
fig
ure in the fab
ric as out
lined in the fol
lowing sec
tions.
TCAM Policy Usage

The phys
ic
al ternary content-address
able memory (TCAM) in which pol
icy is stored for
en
force
ment is an ex pensive component of switch hard
ware and there fore tends to
lower pol
icy scale or raise hard
ware costs. Within the Cisco ACI fabric, pol
icy is ap
-
plied based on the EPG rather than the end point it
self. This pol
icy size can be ex-
pressed as n*m*f, where n is the num
ber of sources, m is the num ber of des
ti
na
tions,
and f is the num
ber of pol
icy fil
ters. Within the Cisco ACI fab
ric, sources and des ti
na
-
tions become one entry for a given EPG, which re duces the num ber of total en
tries re
-
quired.
TCAM is a fab ric re

source that should be mon it
ored. There is a a sys
tem wide view of
available TCAM re sources. To see this click on Fabric > In
ven
tory > Pod1 and then se -
lect the Op er
a
tional tab in the work pane, and you will see a table sum ma
riz
ing ca
pac
i-
tiy for all nodes.
296 Monitoring
Switch TCAM capacity dashboard
TCAM is a crit ic
al sys
tem resource in an ACI fab
ric and should be mon it
ored for uti
liza
-
tion. The ar chitec
ture/design team should ar tic
ul
ate what the as sump tions were for
TCAM uti lization. There is a Fabric Re
source Calcula
tion tool on Github that will help
with es
ti
ma tion of normal oper
at
ing pa
ra
meters:
https://
github.
com/
datacenter/
FabricResourceCalculation
As a general rule, the de fault mon it

oring poli
cies will alert you to a re
source short
age
and lower over all fab
ric health score. If your en vi
ronment has a high rate of change, or
you antic
ip
ate the pos si
bil
ity of consis
tently being oversubscribed, you may want to set
dif
fer
ent thresholds.
Create TCAM Policy Monitor
2 In the Navigation pane, choose Monitor Policies > default > Stats Collection
Policies.
Monitoring 297
3 In the Work pane, in the Stats Collection Policies dialog box, perform the
following actions:
a. Select the Monitoring Object Equipment Capacity Entity
(eqptcapacity.Entity).
b. Select the Stats Type Policy Entry.
c. Click + under Config Thresholds.
d. In the Thresholds For Collection 5 Minute window, select the blue pencil
icon next to policy CAM entries usage current value.
TCAM Prefix Usage

This pro
ce
dure man
ages a TCAM Pre
fix Usage.
2 In the Navigation pane, choose Monitor Policies > default > Stats Collection
Policies.
3 In the Work pane, in the Stats Collection Policies dialog box, perform the
following actions:
a. Select the Monitoring Object Equipment Capacity Entity
(eqptcapacity.Entity).
b. Select the Stats TypeLayer3 Entry.
c. Click + under Config Thresholds.
d. In the Thresholds For Collection 5 Minute window, select the blue pencil
icon next to policy CAM entries usage current value.
Health Score Evaluation Policy

2 In the Navigation pane, choose Monitor Policies > Common Policies > Health
Score Evaluation Policy > Health Score Evaluation Policy.
3 In the Work pane, in the Properties dialog box, perform the following actions:
a. In the Penalty of fault severity critical dropdown menu, select the desired %.
b. In the Penalty of fault severity major dropdown menu, select the desired %.
c. In the Penalty of fault severity minor dropdown menu, select the desired %.
d. In the Penalty of fault severity warning dropdown menu, select the desired %.
298 Monitoring
4 Click Submit.
Communication Policy
2 In the Navigation pane, expand Pod Policies > Policies > Communication.
3 In the Work pane, choose Actions > Create Communication Policy.
4 In the Create Communication Policy dialog box, perform the following actions:
a. Enter Communication Policy Name.
b. From the HTTP Admin State dropdown menu select the desired state.
c. From the HTTP Port dropdown menu select the desired port.
d. Select the desired HTTP redirect state.
e. From the HTTPS Admin State dropdown menu select the desired state.
f. From the HTTPS Port dropdown menu select the desired port.
g. Select the desired HTTPS redirect state.
h. From the SSH Admin State dropdown menu select the desired state.
i. From the Telnet Admin State dropdown menu select the desired state.
j. From the Telnet Port dropdown menu select the desired port.
5 Click Submit.
Monitoring 299
Proactive Monitoring - Infrastructure
While health scores provide a com pre

hen
sive view of the health sta
tus of var
io
us ob
-
jects, and fab
ric and tenant poli
cies show us a narrower view of fab ric and tenant
health, ACME will also need to set up poli
cies to mon
it
or spe
cific re
sources.
This sec
tion of the book at
tempts to cover some key per
for
mance in
di
ca
tors that
should be mon
it
ored, pro
ce
dures for mon it
or
ing them, and when pos
si
ble, suggest rec
-
ommended thresholds for trig
ger
ing alerts or alarms from Network Mon it
or
ing Sys-
tems (NMS).
There are sev

eral meth
ods to obtain this data, and ref
er
ences are pro
vided to ways of
ob
tain
ing the data when possi
ble. Oper
ators can use a wide array of tools in
clud
ing
Sys
log, SNMP, GUI, REST API calls and CLI.
A full list of MIBs sup

ported on the 1.x is avail
able at:
http://www.cisco.
com/c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/list/
mib-support.
html
It is im
por
tant to note that this list changes as newer soft
ware ver
sions are made avail
-
able, and more variables may be ex posed via SNMP MIBs in the fu
ture.
Monitoring APICs
CPU utilization and Memory

GUI
The easie
st way to quickly ver
ify the health of the con
trollers is the APIC. When log
ging
into the system dashboard, the health of the APICs and the health of the clus ter it
self
are dis
played right at the dash
board.
The screen
shot shows where this in forma
tion is vis
ib
le. In this ex
ample, two out of the
three APICs are in a sub-op
ti
mal state and APIC 1 is also experi
enc
ing is
sues.
300 Monitoring
System health dashboard
The normal state for these is to have them all green in a "fully fit" state im
ply
ing the
APICs are syn
chronized with each other.
A more de
tailed drill
down is avail ing on Sys
able by click tem > Con
trollers.
REST API
Con trollers pro

vide in
for
mation regard
ing the cur rent sta
tus of CPU and mem ory uti
-
liza
tion by cre at stances of the pro
ing in cEntity class. procEntity is a con tainer of
processes in the sys tem. This ob
ject holds detailed in
for
mation about var io
us processes
running on the APIC. The pro cEntity ob
jects contain the fol
low
ing use ful properties:
cpuPct - CPU uti

liza
tion
maxMemAl
loc - The max
im
um mem
ory al
lo
cated for the sys
tem
Free - The max

mem im
um amount of avail
able mem
ory for the sys
tem
Sam
ple Usage: This in
for
ma
tion can be re
trieved for all APIC con
trollers using the fol
-
low
ing REST call:
http[s]://apic_ip/api/node/class/procEntity.xml?
Monitoring 301
CLI
The Linux Top util

ity also comes built into the APIC con
trollers and can be used for
trou
bleshoot
ing and/or ver
if
i
ca
tion.
user@apic1:~> top
top - 11:41:51 up 16:50, 4 users, load average: 4.19, 4.27, 4.29
Tasks: 354 total, 1 running, 353 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.3%us, 0.4%sy, 0.0%ni, 99.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 131954932k total, 7473180k used, 124481752k free, 409540k buffers
Swap: 0k total, 0k used, 0k free, 1952656k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND

32102 root 20 0 556m 205m 85m S 3.3 0.2 38:11.04 svc_ifc_applian
32120 ifc 20 0 660m 343m 86m S 2.0 0.3 27:58.73 svc_ifc_dbgr.bi
32121 ifc 20 0 631m 286m 86m S 2.0 0.2 17:41.92 svc_ifc_topomgr
32105 root 20 0 659m 258m 85m S 1.7 0.2 17:08.35 svc_ifc_bootmgr
32113 ifc 20 0 1083m 721m 69m S 1.7 0.6 20:03.37 svc_ifc_observe
32128 ifc 20 0 639m 315m 69m S 1.7 0.2 16:28.34 svc_ifc_reader.
32132 ifc 20 0 657m 252m 71m S 1.7 0.2 17:13.74 svc_ifc_scripth
1291 root 20 0 834m 419m 94m S 1.3 0.3 20:35.24 nginx.bin
Disk Utilization
GUI
There are several disks and file sys

tems pre sent on the APICs. The GUI provides ready
ac
cess to disk space uti
liza
tion of all par
ti
tions on the sys
tem and can be used for mon-
i
tor
ing this in
for
ma
tion.
The disk uti

liza ing on Sys
tion can be viewed by click tem > Con
trollers > Apic-X > Stor
age
The work pane dis

plays the uti
liza
tion of all par
ti
tions in the sys
tem.
302 Monitoring
CLI
user@apic1:~> df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/dm-1 41282880 10518960 28666872 27% /
tmpfs 4194304 56456 4137848 2% /dev/shm
tmpfs 65977464 964 65976500 1% /tmp
/dev/mapper/vg_ifc0-data
41282880 10518960 28666872 27% /data
/dev/mapper/vg_ifc0-firmware
41284928 13860672 25327104 36% /firmware
/dev/mapper/vg_ifc0-data2
583149656 1281104 552246280 1% /data2
* Note that not all file sys

tems are vis
ib
le from the CLI as some re
quire root ac cess to
reach the mount points. The GUI should be used as a sin gle source of truth for file sys
-
tem uti
liza
tion.
REST API
This in
for
ma
tion can be re
low
ing REST call:
http[s]://apic-ip/api/node/class/eqptStorage.xml?
Physical and Bond Interface Statistics

APICs use a bonded in ter
face that is typ
ic
ally dual-homed to two leaves for con
nectiv
-
ity to the ACI fab
ric and have the abil ity to use a bonded inter
face that can be dual
homed to the out-of-band man
age
ment net
work.
Bond0 is the bond interface used to con

nect to the fab
ric it
self (to con
nect to leaves
that con
nect into the fabric).
Bond1 is the bond in

ter
face used to connect to the out-of-band seg
ment (to con
nect to
an OOB seg ment that al
lows setup of the APIC it
self).
Monitoring 303
The bond inter

faces rely on under
ly
ing physic
al in
ter
faces and it is impor
tant to note
that the GUI pro
vides link in
for
ma
tion for both the phys ic
al and log
ic
al bond in
ter
faces.
GUI
To view in ter
face sta
tus for the in
ter
faces on the APICs, navate to Sys
ig tem > Con
-
trollers > Apic-x > In
ter
faces
CLI
Both "if
con
fig" and the "ip link" CLI com
mands can be used to ver
ify link state. The CLI
also pro
vides in
for
ma
tion on de
tailed in
ter
face sta
tis
tics such as RX and TX coun
ters.
REST API
This in
for
ma
tion can be re
low
ing REST call:
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-1/sys.json?query-
target=subtree&target-subtree-class=l3EncRtdIf
APIC Fan Status

The fol
lowing sec
tion de
scribes method
olo
gies to re
trieve the sta
tus of the fan trays on
the APICs.
GUI
To view in
ter
face sta
tus for the in
ter
ig tem > Con
-
trollers > Apic-x > Equip
ment-Fans
REST API
This in
for
ma
tion can be re
low
ing REST call:
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-1.json?query-
target=subtree&target-subtree-class=eqptFan
CLI
304 Monitoring
CLI
The Fan sta
tus for the APICs can be mon
it
ored using the CLI on the CIMC port of the
APIC. To ob
tain this, login to the CIMC using the cre dentials used for setting up the
CIMC (may not be the same as the cre den
tials used for APIC). If this has not been setup
pre
vi
ously, the de
fault user
name is admin and the de
fault pass
word is pass
word.
The CIMC port is the inte

grated lights-out man agement port that can be used to re
-
cover an APIC in the event of a cat
as
trophic fail
ure.
user@apic1:~> ssh -l admin 172.16.176.179
Warning: Permanently added '172.16.176.179' (RSA) to the list of known hosts.
admin@172.16.176.179's password:
C220-FCH1807V02V# scope sensor
C220-FCH1807V02V /sensor # show fan
Name Sensor Reading Units Min. Max. Min. Max.
Status Warning Warning Failure Failure
----------- ------- -------- ----- --------- --------- -------- ---------
FAN1_TACH1 Normal 7490 RPM 1712 N/A 1284 N/A
Temperature Status
To mon it
or the tem
per
at
ure state of the var
io
us sen
sors avail
able on the APICs use the
fol
low
ing steps.
GUI
To view in
ter
face sta
tus for the in
ter
ig tem > Con
-
trollers > Apic-x > Equip
ment-Sen
sors
REST API
Monitoring 305
REST API
This in
for
ma
tion can be re
low
ing REST call:
https://{{apic-ip}}//api/node/mo/topology/pod-1/node-1.json?query-
target=subtree&target-subtree-class=eqptSensor
CLI
C220-FCH1807V02V /sensor # show temperature
C220-FCH1807V02V /sensor # show temperature
------------------ -------- -------- ------- ------- --------- -------- ---------
P1_TEMP_SENS Normal 49.5 C N/A 81.0 N/A 86.0
RISER1_INLET_TMP Normal 45.0 C N/A 60.0 N/A 70.0
RISER2_INLET_TMP Normal 41.0 C N/A 60.0 N/A 70.0
RISER1_OUTLETTMP Normal 50.0 C N/A 60.0 N/A 70.0
RISER2_OUTLETTMP Normal 41.0 C N/A 60.0 N/A 70.0
FP_TEMP_SENSOR Normal 37.0 C N/A 60.0 N/A 70.0
DDR3_P1_A1_TEMP Normal 42.0 C N/A 65.0 N/A 85.0
DDR3_P1_B1_TEMP Normal 43.0 C N/A 65.0 N/A 85.0
DDR3_P1_C1_TEMP Normal 44.0 C N/A 65.0 N/A 85.0
DDR3_P1_D1_TEMP Normal 44.0 C N/A 65.0 N/A 85.0
DDR3_P2_E1_TEMP Normal 43.0 C N/A 65.0 N/A 85.0
DDR3_P2_F1_TEMP Normal 43.0 C N/A 65.0 N/A 85.0
DDR3_P2_G1_TEMP Normal 42.0 C N/A 65.0 N/A 85.0
DDR3_P2_H1_TEMP Normal 41.0 C N/A 65.0 N/A 85.0
VICP81E_0_TMP3 Normal 56.0 C N/A 85.0 N/A 90.0
PSU1_TEMP Normal 37.0 C N/A 60.0 N/A 65.0
PCH_TEMP_SENS Normal 51.0 C N/A 80.0 N/A 85.0
Power Supply Status

To mon it
or the tem
per
at
ure state of the var
io
us sen
sors avail
able on the APICs use the
fol
low
ing steps.
306 Monitoring
CLI
C220-FCH1807V02V /sensor # show psu
------------------ -------- -------- -------- ------- --------- -------- ---------
POWER_USAGE Normal 160 Watts N/A N/A N/A 800
PSU1_POUT Normal 136 Watts N/A 624 N/A 648
PSU1_PIN Normal 160 Watts N/A 720 N/A 744
PSU1_STATUS Normal present
PSU2_STATUS Normal absent
PSU1_PWRGD Normal good
PSU1_AC_OK Normal good
Monitoring Leaf Switches

Leaf switches in the ACI fabric typ
ic
ally equate to the Nexus 9300 fam
ily of switches
(with the ex
cep
tion of the Nexus 9336 switch).
The leaf switches pro

vide first hop con
nec
tiv
ity to any
thing that at
taches to the fab
ric.
Note that un like tradi
tional two or three tier designs where the "WAN" layer at taches to
ei
ther the dis tri
bu
t
ion/ag grega
tion layer, in the ACI fabric, all ex
ter
nal connec
tiv
ity is
provided through a set of leaf switches that pro vide high den sity 10gig or 40gig con -
nectiv
ity to ele
ments out side the fab
ric.
To ac
cess the dash
board to mon
it
or switches navate to Fab
ig ric > In
ven
tory > Pod-1 >
Leaf-*
Monitoring 307
Leaf switch monitoring dashboard
No
tice that the dash
board for this switch de
faults to pre
sent
ing the health score of the
switch at a node level on the dashboard, and the sub tabs on the right hand side (topol
-
ogy/gen eral/stats/health/faults/trou bleshoot
ing/his
tory) can be used to quickly drill
down into the var io
us proper
ties of the switch to under
stand how the switch is de -
ployed from a hard ware config
ur
ation stand
point, re
me
diate faults on the switch, and
trou
bleshoot the switch from a hard
ware per
spec
tive.
Monitoring Switch CPU Utilization

There are several meth
ods to poll CPU utiliza
tion and trend it over dif
fer
ent pe
ri
ods of
time. The fol
low
ing sec
tions de
scribe a few of the meth ods avail
able.
REST API
Spine and Leaf switches CPU uti
liza
tion can be mon
it
ored using the fol
low
ing classes,
based on the de
sired timescale and gran
ul
ar
ity.
proc:SysCPU5min A class that rep re

sents the most current sta
tis
tics for Sys
tem cpu in
a 5 minute sam
pling in
ter
val. This class up
dates every 10 sec
onds.
308 Monitoring
proc:SysCPU15min A class that rep

re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in
a 15 minute sam
pling in
ter
val. This class up
dates every 5 min
utes.
proc:SysCPU1h A class that rep

re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in a 1
hour sam
pling in
ter
val. This class up
dates every 15 min
utes.
proc:SysCPU1d A class that rep resents the most cur

rent sta
tis
tics for Sys
tem cpu in a 1
day sampling in
ter
val. This class updates every hour.
proc:SysCPU1w A class that rep

re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in a 1
week sam
pling in
ter
val. This class up
dates every day.
proc:SysCPU1mo A class that rep resents the most cur

rent sta
tis
tics for Sys
tem cpu in a
1 month sam
pling in
ter
val. This class up
dates every day.
proc:SysCPU1qtr A class that rep

re
sents the most cur
rent sta
tis
tics for Sys
tem cpu in a
1 quar
ter sam
pling in
ter
val. This class up
dates every day.
proc:SysCPU1year A class that rep resents the most current sta

tis
tics for Sys
tem cpu in
a 1 year sam
pling in
ter
val. This class up
dates every day.
ACME would like to see the av

er
age CPU uti
liza
tion of all of the fab
ric switches over the
last day.
http[s]://apic_ip//api/node/class/procSysCPU1d.xml?
CLI
Leaf-1# show proc cpu sort
PID Runtime(ms) Invoked uSecs 1Sec Process

----- ----------- -------- ----- ------ -----------
4012 69510 493837 140 1.3% t2usd_tor
4065 7239 27609 262 1.3% python
4292 3841 134758 28 0.8% svc_ifc_opflexe
4391 2355 4423 532 0.4% nginx
4067 1911 206 9278 0.4% svc_ifc_policye
4302 1904 1862 1022 0.3% svc_ifc_observe
Monitoring 309
4311 1811 1018 1779 0.3% svc_ifc_confele

4123 1407 251 5606 0.3% svc_ifc_eventmg
4310 1802 689 2616 0.3% svc_ifc_dbgrele
4846 119693 36527 3276 0.2% stats_manager
3923 15406 2645 5824 0.1% pfmclnt
4864 2361 2812 839 0.1% ospfv3
4865 2402 2717 884 0.1% ospf
13606 435 211 2065 0.0% bgp
4296 6263 7413 844 0.0% snmpd
4297 6667 4542 1467 0.0% dc3_sensor
4299 8559 8225 1040 0.0% policy_mgr
4301 1860 19152 97 0.0% plog
4866 2792 3269 854 0.0% isis
5025 1611 1743 924 0.0% mcecm
In order to ob
tain a his
tor
ic
al view of CPU uti
liza
tion from the CLI it may be nec
es
sary
to jump into an alter
na
tive shell from the switch bash prompt. This shell is called vsh
(or v-shell).
Leaf-1# show processes cpu history
1 1 33 1
746554661885833055376572545534667663554785033943645665335644
100
90
80
70
60
50
40 #
30 ##
20 ##
10 # ### ####### ######## # ## ##### ## #### # # #### ##
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per second (last 60 seconds)

# = average CPU%
310 Monitoring
32 13311134111111111311111111131 11111113 1111 11231 1111111

749513800432206328353370732175609342000769025791144192680117
100
90
80
70
60
50
40 * * * *
30 * ** ** * * * *
20 ** **** ** * * ** * * *** ** ** ** ** *
10 ############################################################
0....5....1....1....2....2....3....3....4....4....5....5....
0 5 0 5 0 5 0 5 0 5
CPU% per minute (last 60 minutes)

* = maximum CPU% # = average CPU%
1
440
030
100 *
90 *
80 *
70 *
60 *
50 *
40 ***
30 ***
20 ***
10 ###
0....5....1....1....2....2....3....3....4....4....5....5....6....6....7.
0 5 0 5 0 5 0 5 0 5 0 5 0
CPU% per hour (last 72 hours)

* = maximum CPU% # = average CPU%
Monitoring 311
Monitoring Switch Memory Utilization

There are sev
eral meth
ods to poll mem
ory uti
liza
tion and trend it over dif
fer
ent pe
ri
-
ods of time. The fol
low
ing sec
tions de
scribe a few of the meth
ods avail
able.
REST API
Spine and Leaf switches mem ory uti
liza
tion can be mon
it
ored using the fol
low
ing
classes, based on the de
sired timescale and granul
ar
ity.
proc:Sys
Mem5min A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
-
ory in a 5 minute sam
pling in
ter
val. This class up
dates every 10 sec
onds.
proc:Sys
Mem15min A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
-
ory in a 15 minute sam
pling in
ter
val. This class up
dates every 5 min
utes.
proc:Sys
Mem1h A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
ory
in a 1 hour sam
pling in
ter
val. This class up
dates every 15 min
utes.
proc:Sys
Mem1d A class that rep
re
sents the most cur
rent sta
tis
tics for Sys
tem mem
ory
in a 1 day sam
pling in
ter
val. This class up
dates every hour.
proc:SysMem1w A class that rep resents the most current sta

tis
tics for Sys
tem mem
ory
in a 1 week sam
pling in
ter
val. This class up
dates every day.
proc:Sys Mem1mo A class that represents the most current sta

tis
tics for Sys
tem mem
-
ory in a 1 month sam
pling in
ter
val. This class up
dates every day.
proc:Sys Mem1qtr A class that represents the most cur rent sta
tis
tics for Sys
tem mem
-
ory in a 1 quar
ter sam
pling in
ter
val. This class up
dates every day.
proc:Sys Mem1year A class that rep re

sents the most current sta
tis
tics for Sys
tem mem
-
ory in a 1 year sam
pling in
ter
val. This class up
dates every day.
ACME would like to mon

it
or mem
ory over the last day, and would use the fol
low
ing
REST call:
http[s]://apic_ip/api/node/class/procSysMem1d.xml?
312 Monitoring
CLI
Leaf-1# show system resources

Load average: 1 minute: 1.21 5 minutes: 1.14 15 minutes: 0.80
Processes : 513 total, 2 running
CPU states : 4.1% user, 2.5% kernel, 93.4% idle
Memory usage: 16401072K total, 9054020K used, 7347052K free
Current memory status: OK
SNMP
As men
tioned in the URL for the SNMP refer
ence guide for ACI release 1.x, the fol
lowing
SNMP ob jects are sup
ported from an SNMP polling per
spective. See http:// www. cisco.
com/c/en/us/ td/
docs/switches/
datacenter/aci/
apic/sw/ 1-x/mib/ list/
mib-
support.
html
Monitoring File System Health
CLI
Cur rently, the CLI is only way to mon it

or the uti
liza
tion of the file sys
tem on the leaves.
It is normal to see a higher % utiliza
tion on some of the mount points in the file sys tem
hier
archy. The critic
al vol
umes to keep track of in terms of uti liza
tion are /volatile,
boot flash and logflash
Leaf-1# df
df: `/nxos/tmp': No such file or directory
df: `/var/home': No such file or directory
df: `/var/tmp': No such file or directory
df: `/nginx': No such file or directory
df: `/debugfs': No such file or directory
df: `/recovery': No such file or directory
df: `/cfg0': No such file or directory
df: `/cfg1': No such file or directory
df: `/logflash/INXOS_SYSMGR/startup-cfg': No such file or directory
df: `/mnt/plog': No such file or directory
Filesystem 1K-blocks Used Available Use% Mounted on
Monitoring 313
rootfs 512000 1064 510936 1% /

rootfs 512000 1064 510936 1% /
none 512000 1064 510936 1% /isan
none 512000 1064 510936 1% /var
none 51200 2288 48912 5% /etc
none 51200 108 51092 1% /var/log
none 3145728 336664 2809064 11% /dev/shm
none 512000 0 512000 0% /volatile
/dev/sda4 7782036 1080636 6306088 15% /bootflash
/dev/sda5 60485 5356 52006 10% /mnt/cfg/0
/dev/sda6 60485 5356 52006 10% /mnt/cfg/1
/dev/sda3 120979 13349 101384 12% /mnt/pss
/dev/sda7 512000 1064 510936 1% /logflash
/dev/sda9 15748508 591216 14357292 4% /mnt/ifc/cfg
/dev/sda8 15748508 991204 13957304 7% /mnt/ifc/log
/dev/sda8 15748508 991204 13957304 7% /var/log/dme/oldlog
/dev/sda9 15748508 591216 14357292 4% /controller
rootfs 716800 665728 51072 93% /mnt/ifc/cfg/mgmt/opt/controller/sbin
rootfs 716800 665728 51072 93% /controller/sbin
/dev/sda8 15748508 991204 13957304 7% /data/techsupport
rootfs 716800 665728 51072 93% /bin
/dev/sda4 7782036 1080636 6306088 15% /bootflash
rootfs 716800 665728 51072 93% /data/challenge.old.plugin
/dev/sda9 15748508 591216 14357292 4% /controller
rootfs 716800 665728 51072 93% /controller/sbin
rootfs 716800 665728 51072 93% /dev
none 3145728 336664 2809064 11% /dev/shm
none 51200 2288 48912 5% /etc
none 2097152 682360 1414792 33% /isan/plugin/0/isan/utils
none 2097152 682360 1414792 33% /isan/plugin/0/lc/isan/utils
none 2097152 682360 1414792 33% /isan/plugin/0/lc/isan/lib
none 2097152 682360 1414792 33% /isan/plugin/0/isan/lib
none 2097152 682360 1414792 33% /isan/lib
none 2097152 682360 1414792 33% /isan/plugin/0/lib
none 2097152 682360 1414792 33% /isan/utils
rootfs 716800 665728 51072 93% /lc/isan/utils
rootfs 716800 665728 51072 93% /lib
rootfs 716800 665728 51072 93% /mnt/cfg
/dev/sda5 60485 5356 52006 10% /mnt/cfg/0
314 Monitoring
/dev/sda6 60485 5356 52006 10% /mnt/cfg/1
rootfs 716800 665728 51072 93% /mnt/ifc

/dev/sda9 15748508 591216 14357292 4% /mnt/ifc/cfg
rootfs 716800 665728 51072 93% /mnt/ifc/cfg/mgmt/opt/controller/sbin
/dev/sda8 15748508 991204 13957304 7% /mnt/ifc/log
/dev/sda3 120979 13349 101384 12% /mnt/pss
rootfs 716800 665728 51072 93% /sbin
/dev/sda8 15748508 991204 13957304 7% /data/techsupport
none 1572864 39444 1533420 3% /tmp
rootfs 716800 665728 51072 93% /usr
none 51200 108 51092 1% /var/log
none 51200 108 51092 1% /var/log/messages
rootfs 716800 665728 51072 93% /var/log/dme
rootfs 716800 665728 51072 93% /var/log/dme/nginx
rootfs 716800 665728 51072 93% /usr/share/vim
/dev/sda7 11811760 375608 10836140 4% /var/log/dme/log
none 512000 1064 510936 1% /var/run/mgmt/log
none 512000 1064 510936 1% /var/run/utmp
/dev/sda7 11811760 375608 10836140 4% /var/sysmgr
none 40960 8 40952 1% /var/sysmgr/startup-cfg
/dev/sda7 11811760 375608 10836140 4% /logflash/core
rootfs 716800 665728 51072 93% /usb
none 512000 0 512000 0% /volatile
Monitoring CoPP (Control Plane Policing) Statistics
CLI
CoPP is en abled by de fault and the para

me
ters can
not be changed at this time. CoPP
sta
tis
tics are avail
able through the CLI.
To show the CoPP pol

icy that is pro
gramed by the sys
tem use the fol
low
ing CLI com
mand:
Leaf-1# show copp policy

COPP Class COPP proto COPP Rate COPP Burst
Monitoring 315
mcp mcp 500 500

ifc ifc 5000 5000
igmp igmp 1500 1500
nd nd 1000 1000
cdp cdp 1000 1000
pim pim 500 500
dhcp dhcp 1360 340
lacp lacp 1000 1000
ospf ospf 2000 2000
arp arp 1360 340
lldp lldp 1000 1000
acllog acllog 500 500
stp stp 1000 1000
eigrp eigrp 2000 2000
coop coop 5000 5000
traceroute traceroute 500 500
isis isis 1500 5000
icmp icmp 500 500
bgp bgp 5000 5000
To show drops against spe

cific CoPP classes, use the fol
low
ing CLI com
mand:
Leaf-1# show copp policy stats
COPP Class COPP proto COPP Rate COPP Burst AllowPkts AllowBytes DropPkts DropBytes
mcp mcp 500 500 0 0 0 0
ifc ifc 5000 5000 195072 161613961 0 0
igmp igmp 1500 1500 3 192 0 0
nd nd 1000 1000 6 564 0 0
cdp cdp 1000 1000 494 140543 0 0
pim pim 500 500 0 0 0 0
dhcp dhcp 1360 340 4 1400 0 0
lacp lacp 1000 1000 0 0 0 0
ospf ospf 2000 2000 0 0 0 0
arp arp 1360 340 1284 90068 0 0
lldp lldp 1000 1000 5029 1717208 0 0
acllog acllog 500 500 0 0 0 0
stp stp 1000 1000 0 0 0 0
eigrp eigrp 2000 2000 0 0 0 0
coop coop 5000 5000 4722 470546 0 0

316 Monitoring
traceroute traceroute 500 500 0 0 0 0
isis isis 1500 5000 17141 2167565 0 0
icmp icmp 500 500 0 0 0 0
bgp bgp 5000 5000 864 73410 0 0
Physical Interface Statistics and Link State
GUI
To ac
cess in
ter
face link state infor
mation, in the APIC GUI, nav ate to Fab
ig ric > In
ven
-
tory > Pod-1 > Leaf-X > In terfaces > Phys i
cal In
ter
faces. In the work pane, the oper
state col
umn displays the op
er
at
ional state of the link. Note that there are other tabs
avail
able in the work pane that ref
er
ence other types of interfaces like port-chan
nels,
vir
tual port-chan
nels, routed in
ter
faces, loop
backs, etc.
To ac
cess in
ter
face sta
tis
tics, in the APIC GUI, nav ate to Fab
ig ric > In
ven
tory > Pod-1 >
Leaf-X > In
ter
faces > Phys i
cal inter
faces > Eth X/Y, and then click on the Stats tab in
the work pane on the right-hand side.
Physical interface throughput statistics
Note that click

ing on the check icon en
ables you to se
lect ad
di
tional sta
tis
tics that can
be graphed sim
il
ar to the fol
low
ing screen
shot.
Monitoring 317
Granular physical interface statistics
REST API
For customers that pre fer the REST API in
ter
face to poll for in
ter
face sta
tis
tics, sev
eral
ob
jects are avail able. There are several such coun ters that are avail
able (e.g. RX/TX,
input/out put / du plex, 30 sec ond rates, 5 minute rate, uni cast packets, mul ti
cast
packets, etc.). As a pointer, the par
ent man aged ob ject is pro
vided below, as the chil -
dren can be de rived from it.
It is ex
pected that the reader has a good un der
standing of the ob
ject model and is able
to nav ig
ate through the model to ob tain the in
for
ma tion de
siered using the example
below, in for
mation pro
vided in pre
ced
ing sec
tions and the tools described therein
An ex
am
ple of the base API call for phys
ic
al in
ter
face sta
tis
tics is:
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/phys-[eth1/1].json
318 Monitoring
For ex
am
ple, to de
ter
mine the total ingress bytes on Leaf 101 port Eth1/1, the ACME
op
er
at
or could issue the fol
low
ing API call:
/topology/pod-1/node-101/sys/phys-[eth1/1].json
Vi
sore al
lows the op
er
at
or to dig deeper into the hi
er
ar
chi
cal tree. From the prior com
-
mand, the op er
at
or could see chil dren of the interface object, such as ingress and
egress bytes. One of the child ob
jects in
cludes the fol
lowing:
topology/pod-1/node-101/sys/phys-[eth1/1]/dbgEtherStats&#10
CLI
The show in
ter
face eth x/y command can be used to mon it
or in
ter
faces from the CLI.
Other sup
ported commands include "show in
ter
face port-chan nel x/y"
Leaf-1# show int e1/1

Ethernet1/1 is up
admin state is up, Dedicated Interface
Hardware: 1000/10000/auto Ethernet, address: 7c69.f60f.8771 (bia 7c69.f60f.8771)
MTU 9000 bytes, BW 1000000 Kbit, DLY 1 usec
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, medium is broadcast
Port mode is trunk
full-duplex, 1000 Mb/s, media type is 1G
Beacon is turned off
Auto-Negotiation is turned on
Input flow-control is off, output flow-control is off
Auto-mdix is turned off
Rate mode is dedicated
Switchport monitor is off
EtherType is 0x8100
EEE (efficient-ethernet) : n/a
Last link flapped 04:19:13
Last clearing of "show interface" counters never
1 interface resets
30 seconds input rate 169328 bits/sec, 97 packets/sec
30 seconds output rate 424528 bits/sec, 115 packets/sec
Monitoring 319
Load-Interval #2: 5 minute (300 seconds)

input rate 644416 bps, 134 pps; output rate 365544 bps, 114 pps
RX
2474537 unicast packets 8434 multicast packets 2 broadcast packets
2482973 input packets 1686129815 bytes
0 jumbo packets 0 storm suppression bytes
0 runts 0 giants 0 CRC 0 no buffer
0 input error 0 short frame 0 overrun 0 underrun 0 ignored
0 watchdog 0 bad etype drop 0 bad proto drop 0 if down drop
0 input with dribble 712 input discard
0 Rx pause
TX
1673907 unicast packets 575 multicast packets 7 broadcast packets
1674489 output packets 455539518 bytes
0 jumbo packets
0 output error 0 collision 0 deferred 0 late collision
0 lost carrier 0 no carrier 0 babble 0 output discard
0 Tx pause
SNMP
As men
tioned in the URL for the SNMP ref
er
ence guide for ACI re
lease 1.x, the fol
lowing
SNMP ob jects are sup ported from an SNMP polling per spective for in terfaces
See: http://www. cisco.
com/c/
en/us/
td/
docs/switches/datacenter/ aci/apic/ sw/
1-x/
mib/list/
mib-support.html
Module Status
Even though the leaves are con sid
ered fixed switches, they have a su per
visor com po-
nent which refers to the CPU com plex. From a for ward ing per spec tive, there
are two data plane com po
nents, viz. the NFE (Net
work For ward ing En gine ASIC) which
provide the front panel ports, and the ALE or ALE2 (Ap pli
cation Leaf En gine ASIC) de -
pend ing on the gener
at
ion of switch hardware, which pro vides up link con nectiv
ity to
the spines. The fol
lowing methods can be used to deter
mine the sta tus of the mod ules
in the switch.
320 Monitoring
GUI
To ac
cess module sta
tus for the NFE and the CPU complex, in the APIC GUI, nav
ig
ate to
Fab
ric > In
ven
tory > Pod-1 > Leaf-X > Chassis > Mod
ule > Supervi
sor mod
ules and the
sta
tus of the mod
ule is dis
played in the work pane.
To access mod ule sta

tus for the ALE/ALE2, in the APIC GUI, navig
ate to Fabric > In
ven
-
tory > Pod-1 > Leaf-X > Chas sis > Mod
ule > Line mod
ules and the status of the mod ule
is dis
REST API
The fol
low
ing REST API call(s) can be used to mon
it
or the state of the su
per
vi
sor and
the module.
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/supslot-1/sup
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/lcslot-1/lc
CLI
The show mod
ule com
mand can be used to ob
tain the sta
tus of the base mod
ule and
the up
link mod
ule.
Leaf-1# show module
Mod Ports Module-Type Model Status

--- ----- ----------------------------------- ------------------ ----------
1 48 1/10G Ethernet Module N9K-C9396PX active
GEM 12 40G Ethernet Expansion Module N9K-M12PQ ok
Mod Sw Hw
--- -------------- ------
1 11.1(0.152) 0.2050
Mod MAC-Address(es) Serial-Num

--- -------------------------------------- ----------
1 7c-69-f6-0f-87-71 to 7c-69-f6-0f-87-ad SAL17267Z9U
Monitoring 321
Mod Online Diag Status

--- ------------------
1 pass
SNMP
As men
er
lease 1.x, the fol
low
ing
SNMP ob jects are supported from an SNMP polling per spec
tive for mod ules.
See: http://www.cisco.
com/c/
en/
us/
td/
docs/
switches/
datacenter/aci/
apic/
sw/
1-x/
mib/list/
mib-support.
html
Switch Fan Status

The fol
lowing section de
scribes method
olo
gies to re
trieve the sta
tus of the fan trays on
the leaf switches.
GUI
To ac
cess fan sta
tus for the leaf switch, in the APIC GUI, nav
ig
ate to Fab
ric > In
ven
tory
> Pod-1 > Leaf-X > Chas
sis > Fan Tray and the sta
tus of the mod
ules is dis
played in the
work pane.
REST API
The follow
ing REST API call(s) and their child ob jects can be used to mon it
or the state
of the fans on a leaf switch (note that there are 3 slots on this par
tic
ul
ar switch).
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/ftslot-1
CLI
The fol
low
ing CLI's can be used to mon
it
or the state of the fans on a leaf switch.
Leaf-1# show environment fan

Fan:
322 Monitoring
------------------------------------------------------
Fan Model Hw Status
------------------------------------------------------
Fan1(sys_fan1) N9K-C9300-FAN1-B -- ok
Fan_in_PS1 -- -- unknown
Fan_in_PS2 -- -- ok
Fan Speed: Zone 1: 0x5f
Fan Air Filter : Absent
SNMP
As men
er
lease 1.x, the fol
low
ing
ported from an SNMP polling per
spec
tive for fan trays (http://
www. cisco.
com/ c/en/ us/
td/
docs/
switches/
datacenter/
aci/
apic/ sw/1-x/
mib/
list/
mib-support. html).
Power Supply Status

The fol
lowing sections de
scribe method
olo
gies to re
trieve the sta
tus of the power sup
-
plies on the leaf switches
GUI
To ac
cess power supply sta
tus for the leaf switch, in the APIC GUI, navate to Fab
ig ric >
In
ven
tory > Pod-1 > Leaf-X > Chas sis > Power Sup ply Units and the sta
tus of the mod -
ules is dis
REST API
The follow
ing REST API call(s) and their child ob jects can be used to mon it
or the state
of the fans on a leaf switch (note that there are 3 slots on this par
tic
ul
ar switch).
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/psuslot-1
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/ch/psuslot-2
Monitoring 323
CLI
The fol
low
ing CLI com
mands can be used to mon
it
or the state of the fans on a leaf
switch:
Leaf-1# show environment power

Power Supply:
Voltage: 12.0 Volts
Power Actual Total

Supply Model Output Capacity Status
(Watts ) (Watts )
------- ------------------- ----------- ----------- --------------
1 UCSC-PSU-650W 0 W 648 W shut
2 UCSC-PSU-650W 168 W 648 W ok
Actual Power
Module Model Draw Allocated Status
(Watts ) (Watts )
-------- ------------------- ----------- ----------- --------------
1 N9K-C9396PX 168 W 456 W Powered-Up
fan1 N9K-C9300-FAN1-B N/A N/A Powered-Up
N/A - Per module power not available
Power Usage Summary:
--------------------
Power Supply redundancy mode (configured) Non-Redundant(combined)
Power Supply redundancy mode (operational) Non-Redundant(combined)
Total Power Capacity (based on configured mode) 648 W
Total Power of all Inputs (cumulative) 648 W
Total Power Output (actual draw) 168 W
Total Power Allocated (budget) N/A
Total Power Available for additional modules N/A
SNMP
As men
er
lease 1.x, the fol
low ing
ported from an SNMP polling per spec
tive for power sup plies.
324 Monitoring
See: http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/
list/
mib-support.
html
LLDP Neighbor Status

The APIC pro
vides a sin
gle pane of glass to query and de
ter
mine all LLDP neigh
bors in a
fab
ric.
To ob
tain a list of LLDP neighbors on an in
ter
face, navate to Fab
ig ric > In
ven
tory >
Pod-1 > Leaf-X > Proto
cols > LLDP > Neigh
bors > eth x/y
A full list
ing of all LLDP neigh
bors on the in
ter
face can be ob
tained in the work pane.
In the above work ing on Neigh

flow click bors (in
stead of eth x/y) gives you a list of all
LLDP neighbors on the switch.
REST API
The fol
low
ing rest API call can be used to ob
tain the same in
for
ma
tion:
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/lldp/inst/if-[eth1/1]
CLI
Leaf-1# show lldp neighbors

Capability codes:
(R) Router, (B) Bridge, (T) Telephone, (C) DOCSIS Cable Device
(W) WLAN Access Point, (P) Repeater, (S) Station, (O) Other
Device ID Local Intf Hold-time Capability Port ID
apic2 Eth1/1 120 4c:4e:35:09:77:2f
apic3 Eth1/4 120 90:e2:ba:4b:fa:d4
5548-2 Eth1/7 120 B Eth1/3
Spine-1 Eth1/49 120 BR Eth4/1
Monitoring 325

Total entries displayed: 9
SNMP
As men tioned in the URL for the SNMP ref
er
ence guide for ACI release 1.x, the fol
low
ing
ported from an SNMP polling perspective for LLDP. See: http://
www. cisco.
com/ c/en/
us/td/docs/
switches/datacenter/aci/apic/sw/ 1-x/mib/
list/
mib-support. html
APIC provides a sin

gle pane of glass to query and de
ter
mine all CDP neigh
bors in a fab
-
ric. CDP Neighbor Status:
GUI
To obtain a list of CDP neighbors on an in
ter
face, navate to Fab
ig ric > In
ven
tory > Pod-
1 > Leaf-X > Pro to
cols > CDP > Neighbors > eth x/y
A full list
ing of all CDP neigh
bors on the in
ter
face can be ob
tained in the work pane.
In the above work ing on Neigh

flow click bors (in
stead of eth x/y) gives you a list of all
LLDP neighbors on the switch.
REST API
The fol
low
ing rest API call can be used to ob
tain the same in
for
ma
tion:
https://{{apic-ip}}/api/node/mo/topology/pod-1/node-101/sys/cdp/inst/if-[eth1/1]
CLI
Leaf-1# show cdp neighbors

Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute
Device-ID Local Intrfce Hldtme Capability Platform Port ID
326 Monitoring
Services-UCS-A(SSI15450J63)
Eth1/5 129 S I s UCS-FI-6248UP Eth1/17
5548-2(SSI154300VL)
Eth1/7 123 S I s N5K-C5548UP Eth1/3
SNMP
As men
er
lease 1.x, the fol
low
ing
ported from an SNMP polling perspec
tive for CDP. See: http://
www. cisco.
com/
c/en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/
list/
mib-support.
html
GOLD Diagnostic Results

GOLD is cov
ered in greater de
tail in the Hard
ware Re
place
ment sec
tion.
GOLD di ag
nostics pro
vide an easy and quick way for op
er
at
ions teams to con firm that
bootup and non-dis rup
tive tests that run dur
ing normal operat
ions have ex e
cuted
prop
erly, as well as the abil
ity to run on de
mand di
ag
nos
tics to isolate po
ten
tial hard
-
ware at fault.
GUI
To view GOLD Di
ag
nos
tic test re
sults in the GUI for the Su
per sors, click on Fab
vi ric >
In
ven
tory > Pod-1 > Leaf-1 > Chas
sis > Su
per
vi
sor Mod
ules > Slot-1. Then click trou
-
bleshoot
ing in the work pane.
To view the same for mod ules, click on Fabric > In

ventory > Pod-1 > Leaf-1 > Chas
sis >
Line Modules > Slot-x. Then click Trou bleshooting in the work pane.
Monitoring 327
CLI
Leaf-1# show diagnostic result module all

Current bootup diagnostic level: bypass
Module 1: 1/10G Ethernet Module (Active)
Test results: (. = Pass, F = Fail, I = Incomplete,
U = Untested, A = Abort, E = Error disabled)
1) bios-mem-----------------------> .
2) mgmtp-lb-----------------------> .
4) nsa-mem------------------------> .
6) fabp-prbs:
Port 1 2 3 4 5 6 7 8 9 10 11 12
-----------------------------------------
. . . . . . . . . . . .
22) cpu-cache----------------------> .
23) mem-health---------------------> .
24) ssd-acc------------------------> .
25) act2-acc-----------------------> .
26) ge-eeprom----------------------> .
29) usb-bus------------------------> .
30) cons-dev-----------------------> .
31) obfl-acc-----------------------> .
32) nvram-cksum--------------------> .
33) fpga-reg-chk-------------------> .
34) asic-scratch-------------------> .
40) rtc-test-----------------------> .
41) pcie-bus-----------------------> .
Monitoring 329
Proactive Monitoring Use Cases
Monitoring Workload Bandwidth

ACME would like to proac
tively mon
it
or con
nec
tions to servers to de
ter
mine whether
ad
e
quate band
width is avail
able to a given work
load. This en
ables them to an
swer
ques
tions about whether a server needs to be up
graded from 10G to 40G, or mul
ti
ple
in
ter
faces need to be bonded to pro vide more band width to the server. The fol
low
ing
will pro
vide an ex
ample of how sta
tis
tics poli
cies and thresholds can be used to alert
the op
er
at
ors when ad
di
tional band
width is re
quired to a server.
ACME's server team has de termined that they would like to mon itor link uti
lization
over a 5 minute pe riod, and when the av er
age uti
lization is above 80%, they would like
to raise a fault for the af fected inter
face. To accom plish these tasks re quires con fig
ur
-
ing a mon itoring policy with two distinct types of policies. First, ACME will con figure a
stats collec tion pol icy to collect the aver
age utilization of a link over the de sired 5
minute in ter val. Next they will con fig
ure a thresh old policy to gen erate a fault when
the link uti lization exceeds vario
us thresholds ac
cord ing to the follow ing table.
Cre
ate an in
ter
face mon
it
or
ing pol
icy.
2 In the Navigation pane, choose Monitoring Policies.
3 In the Work pane, choose Actions > Create Monitoring Policy.
4 In the Create Monitoring Policy dialog box, perform the following actions:
a. Expand the newly created monitoring policy and select Stats Collection
Policy.
b. Click on the edit button next to Monitoring Object, and select L1 Physical
Interface Configuration.
c. From the Monitoring Object drop-down list, choose L1 Physical Interface
Configuration.
d. Click the edit button next to Stats Type and select Ingress and Egress.
e. From the stats type drop-down list, choose Ingress.
f. Click +.
g. Select 5 minutes from the granularity column, and click on update.
330 Monitoring
We have now cre

ated a pol
icy which will mon
it
or as
so
ci
ated in
ter
faces and ingress
traf
fic rate at 5 minute in
ter
vals. Next we will con
fig
ure a thresh
old which will alert us
if the uti
liza
tion ex
ceeds 80% and as
sign an de
fault fault sever
ity to it.
1 Click + in the Config Thresholds column.
2 In the Thresholds for Collection 5 minute window, click +.
3 Click on Ingress Link utilization average value.
4 Enter 0 for the normal value and click the Rising radio button.
Note: In this ex
am
ple, we are in
ter
ested in an ab
solute per
cent
age, these poli
cies can
be fur
ther cus
tomized to pro
vide a nor
mal value, and look for de
vi
at
ions above or
below that nor
mal value.
In this sce
nario, 80% uti
liza
tion does not nec
es
sar
ily mean that the ap
pli
ca
tion per
for
-
mance is de graded, so we will flag this as a warn
ing. Ad
di
tional lev
els/sever
it
ies can
also be spec
if
ied if de
sired.
The set col umn spec if

ies the level at which the fault will be raised, and the reset col -
umn will spec ify the the level at which the fault will be cleared. For ex ample, we will be
raising a warn ing when the uti liza
tion goes above 80, and clear the warn ing when the
uti
liza
tion falls below 75. Re peat these steps for the egress statis
tics as well.
Fi
nally, we will as
so
ci
ate the newly created policy with an in
ter
face pol
icy group that
rep
resents the in
ter
faces we to monit
or with this pol
icy.
For our ex
am
ple, we will apply the pol
icy to the UCS-10G-PG
2 In the Work pane, choose Interface Policies > Policy Groups.

a. Select UCS-10G-PG.
b. From the Monitoring Policy drop down menu, select the newly created
monitoring policy.
EPG Level Statistics

The ap pli
ca
tion owner would like to be able to mon it
or net work-related in for
mation
for their appli
ca
tion, such as the ag
gre
gate amount of traffic to a spe
cific tier. As an ex-
ample, we will mon it
or the amount of traf
fic to the web tier of a given ap plica
tion. In
Monitoring 331
this ex
am
ple, the de
fault mon
it
or
ing poli
cies are ap
pro
pri
ate, and they are sim
ply ex
-
tract
ing them from the sys
tem to be con
sumed ex
ter
nally. This in
for
ma
tion is use
ful in
sce
nar
ios such as a new re
lease being pushed, and to make sure that no traf
fic anom
-
alies are cre
ated after the push.
This can be ac
com
plished by nav
ig
at
ing to the EPG, and se ing the Stats tab:
lect
EPG-level throughput statistics
Ad
di
tion
ally, this in
for
ma
tion can be gath
ered from the API:
http[s]://apic_ip/api/node/mo/uni/tn-mynewproject/ap-app1/epg-web-epg.xml?query-
target=self&rsp-subtree-include=stats
Monitoring 333
Reactive Monitoring
It is cru
cial that the ACME op
er
a
tional staff are able to react to any in
di
ca
tion of some
-
thing going wrong. If there is a no
ti
fi
ca
tion that some
thing has gone wrong, such as a
fault no
ti
fi
ca
tion, a low health score, or a ticket/re
port that end-user func
tion
al
ity has
been impacted, knowledge of the avail
able moni
tor
ing tools is im
por
tant for the iden
ti
fi
-
ca
tion and col
lec
tion of ev
i
dence. This evi
dence can then be used to iden tify and ana-
lyze the root cause of the prob
lem be
fore tak
ing cor
rec
tive ac
tion. For more in
for
ma
tion
re
gard
ing faults and health scores please refer to those spe
cific sec
tions within this book.
A deep dive into the processes of trou bleshooting is out of the scope of this book.
Please refer to "Troubleshooting Cisco Ap pli
ca
tion Cen tric Infra
struc
ture: An
a
lyt
i
cal
lem solv
prob ing ap
plied to the pol
icy dri
ven data center" avail
able at: http://
datacenter.
github.
io/
TenantTroubleshoot Policies
Within the APIC GUI, under each Ten ant you can find a Trou bleshoot Policy sec
tion.
This sec
tion will allow con fig
ur
a
tion of policies that are spe
cific to one ten
ant, and the
monit
or
ing of traf
fic and test connectiv
ity between end points.
As seen in the image above, the fol

low
ing trou
bleshoot
ing poli
cies can be con
fig
ured:
SPAN (Switched Port ANalyzer)Configuration of SPAN and ERSPAN sources

and destinations to be used in external monitoring of Tenant traffic flows
Endpoint-To-Endpoint TracerouteConfiguration of a path validation tool for
verifying validity of communications between Tenant endpoints in an ACI fabric
Atomic CountersConfiguration of a set of customizable counters to collect
and report on information between a definable set of objects. As shown in the
image below, a policy can be configured to collect statistics between EPs,
between EPGs, between EPGs and specific IPs and other special objects, such
as Any or External
traffic flows
FabricTroubleshoot Policies
For trou
bleshoot
ing within the en
tire fab
ric, there are the fol
low
ing tools and poli
cies:
334 Monitoring
SPAN (Switched Port Analyzer)Configuration of SPAN and ERSPAN sources

and destinations to be used in external monitoring of fabric traffic flows
On-demand DiagnosticsConfiguration of a policy for collection of diagnostic
information that can be executed at a point in time and which will return a set
of valuable output for investigation
Leaf Nodes TracerouteConfiguration of a path validation tool for verifying
validity of communications between ACI fabric nodes
Traffic MapAt-a-glance hotspot map of node-to-node traffic flow in an ACI
fabric

From ver
sion 1.1, APIC pro
vides a trou
bleshoot
ing graphic tool to find rel
e
vant faults
and sta
tis
tics, re
cent changes, and run con
nec
tiv
ity tests in a sim
ple man
ner.
It also gives the op

tion to gen
er
ate a re
port to save the re
sults so it can be used as a
ref
erence.
Other Tools
iPingA troubleshooting tool in the ACI fabric that can be used to verify
reachability of a device connected to the fabric utilizing the fabric as the
pervasive source
Audit LogsAudit logs are continually collected on all actions taken in an ACI
fabric and can give a quick indication of which user took which actions at what
time

At the end of this chap
ter, we are going to describe two sit
ua
tions where ACME ran
into is
sues and how they made use of the tools pre
vi
ously de
scribed.
1 No access to application: An end-user calls and reports that they can no longer
access a web application running within the fabric.
2 Users report that an application running in the fabric is slow or there is a report
of slowness to the web application running within the fabric.
Monitoring 335
Reactive Monitoring Tools
Switch Port Analyzer (SPAN)

SPAN is gen
er
ally used in two ways:
Proactively as part of a third party or offline analysis requirement.

Security tools (IDS/IPS, Data Loss Prevention)
Call Recording
Troubleshooting application and networking issues within the fabric.
It may be helpful to perform a cap ture of some partic

ul
ar traf
fic to see what is going on
within the stream of data. Look ing through traf fic flows will allow in vesti
gation of
packet and pro tocol level de
tails, such as traffic resets, misbehaving pro to
cols, im-
proper host re quests or re sponses, or node level com mu ni
cations. This will pro -
vide deeper insight into how de vices are using the net work than sim ple traf
fic flow and
fab
ric con
fig
ura
tion review.
Switched Port AN al
yzer, or SPAN, is a standard fea
ture that al
lows copy and repli ca
tion
of traf
fic to a net
work an al
yzer for fur
ther de
coding and in
vesti
ga
tion. It can be used to
copy traf fic from one or more ports, VLANs, or end point groups (EPGs).
The SPAN feature process is non-dis

rup
tive to any con
nected de
vices and is fa
cil
it
ated
in hard
ware, which pre
vents any unnec
es
sary CPU load.
SPAN ses sions can be con fig

ured to mon it
or traf
fic received by the source (ingress
traf
fic), traf
fic transmitted from the source (egress traf fic), or both. By default, SPAN
mon it
ors all traf
fic, but fil
ters can be con
fig
ured to mon itor only se
lected traf
fic.
Multinode SPAN
APIC traf fic mon it
or
ing policies can enforce SPAN at the ap pro
pri
ate places to copy
traf
fic from mem bers of each End Point Group wher ever they are con
nected. If a mem-
ber moves, APIC au to
mat ic
ally pushes the pol
icy to the new leaf switch. For example,
when a VMo tion event re lo
cates an Endpoint to a new leaf switch, the SPAN fea ture
config
ura
tion automatic
ally adjusts.
336 Monitoring
SPAN Guidelines and Restrictions
Use SPAN for troubleshooting. SPAN traffic competes with user traffic for
switch resources. To minimize the load, configure SPAN to copy only the
specific traffic that you want to analyze.
An l3extLIfP Layer 3 subinterface cannot be configured as a SPAN source. The
entire port must be used for monitoring traffic from external sources.
Tenant and access SPAN use the encapsulated remote extension of SPAN
(ERSPAN) type I, while fabric SPAN uses ERSPAN type II. For information
regarding ERSPAN headers, refer to the IETF Internet Draft at this URL: https:/
/
tools.ietf.org/
html/
draft-foschiano-erspan-00.
See the Verified Scalability Guide for Cisco ACI document for SPAN-related
limits, such as the maximum number of active SPAN sessions.
Configuring a SPAN Session

This pro
cedure shows how to con fig
ure a SPAN pol
icy to for
ward repli
cated source
pack
ets to a re
mote traf
fic an
al
yzer.
3 In the Navigation pane, choose Tenant_Name > Troubleshooting Policies >

SPAN > SPAN Destination Groups.
4 In the Work pane, choose Actions > Create SPAN Destination Group.
5 In the Create SPAN Destination Group dialog box, perform the following
actions:
a. In the Name field, enter a name for the SPAN destination group.
b. In the Destination EPG dropdowns, select the destination Tenant,
Application Profile, and EPG.
c. Enter the Destination IP.
d. Enter the Source IP Prefix.
e. Optionally, modify the other fields as needed.
f. Click OK.
g. If needed, add additional destinations.
h. Click Submit.
Monitoring 337
6 Under SPAN, right-click SPAN Source Groups and choose Create SPAN Source
Group.
7 In the Create SPAN Source Group dialog box, perform the following actions:
a. In the Name field, enter a name for the SPAN source group.
b. From the Destination Group drop-down list, choose the SPAN destination
group that you configured previously.
c. In the Create Sources table, click the + icon to open the Create Sources
dialog box.
d. In the Name field, enter a name for the source.
e. In the Direction field, choose the radio button based on whether you want to
replicate and forward packets that are incoming to the source, outgoing from
the source, or both incoming and outgoing.
f. From the Source EPG drop-down list, choose the EPG (identified by
Tenant/ApplicationProfile/EPG) whose packets will be replicated and
forwarded to the SPAN destination. Click OK to save the SPAN source.
g. Click Submit to save the SPAN source group.
Traceroute
Tracer
oute is a use
ful fea
ture in tra
di
tional net
working. In ACI this fea
ture is im
ple
-
mented tak
ing into ac
count the way the fabric works.
Tracer oute supports a va

ri
ety of modes, including endpoint-to-end point, and leaf-to-
leaf (tun
nel endpoint, or TEP to TEP). It dis
covers all paths across the fab ric, dis
cov
ers
point of exits for ex
ter
nal endpoints, and helps to detect if any path is blocked.
A tracer
oute that is ini
ti
ated from the tenant end points shows the de
fault gate
way as
an in
ter
me
di
ate hop that ap pears at the ingress leaf switch.
Note: If tracer
oute is done from the OS of a con nected server or VM, it will show the
hops for the leaves and spines as un known, and will keep recording the infor
mation
after the packet gets out of the fab
ric. For more pre
cise in
for
ma
tion, please use tracer
-
oute from the APIC (GUI or CLI)
338 Monitoring
Traceroute Guidelines and Restrictions
When the traceroute source or destination is an endpoint, the endpoint must

be dynamic and not static. Unlike a dynamic endpoint (fv:CEp), a static endpoint
(fv:StCEp) does not have a child object (fv:RsCEpToPathEp) that is required for
traceroute.
See the Verified Scalability Guide for Cisco ACI document for traceroute-related
limits.
Traceroute results will display the IP address of the remote node and interface
which it came it came in on. (Browse on the APIC GUI to Fabric | Inventory |
Fabric Management to view the IP address information for correlation.)
Traceroute cannot be used for endpoints that reside in an external EPG.
Performing a Traceroute Between Endpoints

Endpoint-to-Endpoint Traceroute Policies.
4 In the Work pane, choose Actions > Create Endpoint-to-Endpoint Traceroute

Policy.
5 In the Create Endpoint-to-Endpoint Traceroute Policy dialog box, perform the

following actions:
a. In the Name field, enter a name for the traceroute policy.
b. In the Source End Points table, click the + icon to edit the traceroute source.
c. From the Source MAC drop-down list, choose or enter the MAC address of
the source endpoint and click Update.
d. In the Destination End Points table, click the + icon to edit the traceroute
destination.
e. From the Destination MAC drop-down list, choose or enter the MAC address
of the destination endpoint and click Update.
f. In the State field, click the Start radio button.
g. Click Submit to launch the traceroute.
6 In the Navigation pane or the Traceroute Policies table, click the traceroute
policy. The traceroute policy is displayed in the Work pane.
Monitoring 339
7 In the Work pane, click the Operational tab, click the Source End Points tab,
and click the Results tab.
8 In the Traceroute Results table, verify the path or paths that were used in the trace.
a. More than one path might have been traversed from the source node to the
destination node.
b. For readability, increase the width of one or more columns, such as the Name
column.
Atomic Counters
Atomic Coun ters are use
ful for troubleshoot
ing connec
tiv
ity be
tween endpoints, EPGs,
or an ap
pli
ca
tion within the fab ric. A user re
port
ing appli
ca
tion may be ex
peri
enc
ing
slow
ness, or atomic coun
ters may be needed for mon
it
or
ing any traf
fic loss be
tween
two end points. One capa
bil
ity provided by atomic coun ters is the abil
ity to place a
trouble ticket into a proac
tive mon it
or
ing mode, for example when the prob lem is in
-
ter
mittent, and not neces
sarily hap
pening at the time the oper
at
or is ac
tively working
the ticket.
Atomic counters can help de tect packet loss in the fab

ric and allow the quick iso
la
tion of the
source of con
nectiv
ity is
sues. Atomic coun ters re
quire NTP to be en abled on the fabric.
Leaf-to-leaf (TEP to TEP) atomic coun

ters can pro
vide the fol
low
ing:
Counts of drops, ad
mits, and ex
cess pack
ets
Short-term data col
lec
tion such as the last 30 sec
onds, and long-term data col
-
lec
tion such as 5 min
utes, 15 min
utes, or more
A break
down of per-spine traf
fic (avail
able when the num
ber of TEPs, leaf or
VPC, is less than 64)
On
go
ing mon
it
or
ing
Leaf-to-leaf (TEP to TEP) atomic coun ters are cumula

tive and can
not be cleared. How -
ever, because 30 sec
ond atomic coun ters reset at 30 second in
ter
vals, they can be used
to iso
late in
ter
mit
tent or re
cur
ring problems.
Ten
ant atomic coun
ters can pro
vide the fol
low
ing:
Application-specific counters for traffic across the fabric, including drops,

admits, and excess packets
340 Monitoring
Modes include the following:

Endpoint to endpoint MAC address, or endpoint to endpoint IP address. Note
that a single target endpoint could have multiple IP addresses associated with it.
EPG to EPG with optional drill down
EPG to endpoint
EPG to * (any)
Endpoint to external IP address
***Atomic coun
ters track the amount pack
ets of be
tween the two end
points and
use this as a mea
sure
ment. They do not take into ac
count drops or error coun
ters in a
hard
ware level.***
Dropped pack ets are cal

cu
lated when there are less pack
ets re
ceived by the des
ti
na
tion
than trans
mit
ted by the source.
Ex
cess pack
ets are cal
cu
lated when there are more pack
ets re
ceived by the des
ti
na
tion
than trans
mit
ted by the source.
Configuring Atomic Counters

Atomic Counter Policy.
4 Choose a policy (traffic topology). Traffic can be measured between a

combination of endpoints, endpoint groups, external interfaces, and IP
addresses.
5 In the Work pane, choose Actions > Add Policy_Name Policy.
6 In the Add Policy dialog box, perform the following actions:

a. In the Name field, enter a name for the policy.
b. Choose or enter the identifying information for the traffic source. The
required identifying information differs depending on the type of source
(endpoint, endpoint group, external interface, or IP address).
c. Choose or enter the identifying information for the traffic destination.
Monitoring 341
d. Optional: (Optional) In the Filters table, click + to specify filtering of the

traffic to be counted. In the resulting Create Atomic Counter Filter dialog
box, specify filtering by the IP protocol number (TCP=6, for example) and by
source and destination IP port numbers.
e. Click Submit to save the atomic counter policy.
7 In the Navigation pane, under the selected topology, choose the new atomic
counter policy. The policy configuration is displayed in the Work pane.
8 In the Work pane, choose the Operational tab and choose the Traffic subtab to
view the atomic counter statistics.
Traffic Map
Low per
for
mance and con
ges
tion can be iden
ti
fied by use of Traf
fic maps.
Traf
fic maps make use of atomic counters to con
tin
uo
usly mon it
or and dis
play traf
fic
be
tween leaf switches to help with quick debug
ging and iso
la
tion of ap
pli
ca
tion con -
nec
tivity is
sues.
Configuring Traffic Map
2 In the Navigation pane, choose Troubleshooting Policies > Traffic Map.

342 Monitoring
3 Set the drop-down menu options to view the source Leaf to destination Leaf
traffic paths.
Last interval and cumulative
Sent, received, dropped and excess
All spines and a specific spine switch
The per
cent age is shown in rel
at
ive terms to all traf
fic by Source or re
ceived by
Des
ti
na
tion.
4 Clicking on a cell opens a table with all data for all trails and links.

When troubleshooting con
nectiv
ity between end points within the fabric, the Enhanced
Trou
bleshooting Wizard may be used to quickly iden tify con
nectiv
ity issues. The En
-
hanced Troubleshooting Wizard pro vides a sin
gle lo
ca
tion that in
cludes sev eral com
-
monly used tools and outputs re
quired for trou bleshooting end point con nectiv
ity.
1 In the menu bar, click the wrench icon to launch the enhanced troubleshooting
wizard.
2 In the Create SPAN Destination Group dialog box, perform the following
actions:
a. In the Name field, enter a name for the session.
b. Optional: If it is an external IP address, click the check box.
c. In the Source field, enter the MAC or IP address for the source endpoint, and
click search.
d. In the Destination field, enter the MAC or IP address for the destination
endpoint, and click search.
e. Select the Time Window duration for the debug. Optional: Generate a Report
to download the results.
f. Click Start.
3 In the next screen, you can click the following items for more information:
a. Show the faults in the path between the selected EPs, highlighting the
affected component.
b. Run traceroute between EPs.
c. Show relevant statistics for those EPs.
d. Show any related recent changes in the path between EPs.
Monitoring 343
e. Configure Atomic Counters between EPs.

f. Configure SPAN between EPs
g. Show the configured contracts between EPs.
IPing
IPing is used to test and val id
ate connec
tiv
ity within the from leaf node to endpoints
within the fabric, tak
ing into ac count the private net
work. IPing is a trou
bleshoot
ing
tool for net
work users sim il
ar to the PING com mand.
Using IPing
iping [ -V vrf ] [ -c count ] [ -i wait ] [ -p pattern ] [ -s packetsize ] [ -t timeout ] host
Syntax Description
Examples
pod1-leaf1# iping -V overlay-1 10.0.59.154
PING 10.0.59.154 (10.0.59.154): 56 data bytes

344 Monitoring

--- 10.0.59.154 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.23/0.245/0.256 ms
Audit Logs
At times it may be re quired to view changes which have taken place in the fab ric. An
out
age reported on a host or ap pli
ca
tion in the fab
ric may need to be tracked, or data
pulled for an audit re
quire
ment.
Audit logs are records of who made a change, when the change was made, and a de
-
scrip
tion of the ac
tion. Audit logs also record when users logon and lo
goff.
Audit logs can be found in sev

eral places within the GUI, fil
tered to show only those
events rel
e
vant to the cur
rent GUI con
text. Wher
ever a His
tory tab ap
pears in the GUI
Work pane, the audit log can be viewed. This pro ce
dure shows how to view ten ant
events as an ex
ample.
Viewing Audit Logs

Pro
ce
dure (ex
am
ple for view
ing audit log on a ten
ant)
2 In the Work pane, choose Common.
3 In the Navigation pane, choose Common.
4 In the Work pane, choose the History tab.
5 Under the History tab, choose the Events subtab to view the event log.
6 Under the History tab, choose the Audit Log subtab to view the audit log.
7 Double-click a log entry to view additional details about the event.

Monitoring 345
This chap ter will show some ex am

ples of how ACME's op er
at
ions teams
can use their ACI mon
it
or
ing tools to react to a few com
mon pos
si
ble sce
narios.
Note: these ex
am
ples as
sume that basic low-level in
ves
ti
ga
tion has been done and the
issue has been iso
lated to an issue with traf
fic flows across the fab
ric. Ca
bles and con
-
nec
tiv
ity have been verif
ied, hosts are up, VMs are run
ning, processes are run
ning,
mem ory and CPU uti
liza
tion has been checked, etc.
Loss of Connectivity to Endpoint

The ACME ap pli
ca
tion owner has re ported that two servers have lost con nec
tiv
ity.
These two End Points (EPs) be long to two dif
fer
ent End Point Groups (EPGs), within the
same sub net, bridge do main and Ten ant, and each EP is con nected to differ
ent leaf
switches. The bridge do main has unicast routing en
abled, there
fore the de
fault gate
way
for both EPs exist in the leaf switches.
The fol
low
ing steps trou
bleshoot this sit
ua
t
ion:
1 Check that the EPs have been learned by the leaf switches.
2 Verify that the required contracts are in place between the EPs.
Check that the EPs have been learned by the leaf switches

App_Profile_Name > Application EPGs > EPG_Name.
4 In the Work pane, choose the Operational tab and verify that the endpoint is
present.
5 Repeat this procedure for the destination EPG.
Verify the required contracts are in place between the EPs

346 Monitoring
Verify the required contracts are in place between the EPs

App_Profile_Name > Application EPGs > EPG_Name.
5 Choose the Contracts subtab.
6 Check for a relationship between the source EPG and destination EPG, noting
the direction of the arrows.
7 Click on the contract to verify the contents of the contract. This displays the
filters that are present within that contract.
8 Inspect the contents of each filter by examining the contract under the
Security Policies folder and verifying that each filter contains the appropriate
filter entries.
9 If the endpoints are discovered within each EPG and the contract relationships
look correct, examine the troubleshooting policies. A good starting place is the
Enhanced Troubleshooting Wizard.
10 Alternate techniques are available to validate communications between

endpoints within the fabric. One method could be to use endpoint-to-endpoint
traceroute to show if there are paths available between those endpoints.
a. Another option inside the fabric could be to utilize the iPing tool to verify
connectivity between the default gateway and the endpoints. Each leaf has an
SVI used as default gateway. If this test is successful, the connectivity
between endpoints and leaf switches is not the problem. To check the
connectivity to the remote end, use iPing from each leaf to the remote
endpoint using the default gateway as source.
b. In the event that these things seem valid, it then might be necessary to
use SPAN to verify where the traffic is entering and leaving the fabric, and
make sure frames have the right format.
Monitoring 347
Users Report that an Application Running in the Fabric is Slow

ACME test users re
port slow re
sponse of the web servers run
ning in the fab
ric. This
per
for
mance issue could be caused due to la tency, packet drops, or in
termittent con
-
nec
tiv
ity loss. In this case, the end-user is try
ing to ac
cess the web por tal from a test
VM. The VM and the web por
tal be
long to a dif
fer
ent EPGs in the same bridge do
main
and ten
ant.
First the op
er
at
ions staff should make sure end points are learned by the leaf switches
in a con
sis
tent way so that the EPs are vis
ib
le in the op
er
at
ional tab under the Ten
ant->
Ap
pli
ca
tion Pro
file-> EPGs-> Op
er
at
ional Tab
Once they ver

ify end
points have been learned, they can use EP-to-EP Tracer
oute to
show the path va
lid
ity. Atomic coun
ters can also be de
ployed to check if there are any
drops or ir
reg
ul
ar
it
ies be
tween the de
fined de
vices. Look
ing through the Traf
fic Map
can also show an at-a-glance view in the fab ric and high
light any possi
ble hotspots or
conges
tion in cer
tain el
e
ments of the fabric. Check specific coun
ters for CRC errors on
the in
ter
faces used to trans
mit EP specific traf
fic.
If everything in the fab

ric looks fine, SPAN may be used to ver
ify where the traf
fic is en
-
tering and leaving the fab ric and make sure frames have the right for mat. Cor rupted
frames could cause drops in dif fer
ent points, from the EPs on a OS level to the
switches.
349
Scripting
Scripting 351
Section Content
Leveraging Network Programmability
Reference to Object Model

Programmatic Interfaces
REST
Read Operations
Write Operations
Authentication
Filters
API Inspector
Development Techniques
POSTman
Installation
Collections
Build Login request
Make Query to APIC
Make Configuration Change in APIC
Use API Inspector for Query Guidance
Cobra SDK and Arya

Establish Session
Work with Objects
Cisco APIC REST to Python Adapter
352 Scripting
ACI Toolkit
ACI Toolkit Applications
Endpoint Tracker
ACI Lint
GitHub
Source Control
GitHub
"It's on github"
Scripting 353
Leveraging Network Programmability
The in
dus
trial rev
ol
u
tion mod
ern
ized the tech
niques used to man uf
ac
ture goods, going
from hand pro duc
tion meth
ods to mech an
ized manuf
ac
turing. This movement from
man
ual to au
to
mated op
er
at
ions changed human pro
duc
tiv
ity, al
low
ing peo
ple to free
themselves from repet
it
ive tasks that could be more eas
ily accomplished by a machine.
The as
soci
ated de
crease in costs, in
crease in speed and in creased quality al
lowed for
more work to be done for less money in less time, yield
ing a higher qual
ity prod
uct.
Pro
gram
ma
bil
ity promises to offer the same out
come for net
works as the in
dus
trial
rev
ol
u
tion did for goods.
The inevitable move to ward au tomation in the IT in dustry has pro vided peo ple and
businesses a faster way to achieve their de sired goals, a more cost-ef fec
tive way to
rapidly provi
sion in fra
structure in a timely fash ion ac cording to de mand, and
yielded more con sis
tency in the con fig
ured re sults. ACI is able to take ad vantage of all
of these bene
fits by com pletely expos
ing all of the na tive functional
ity in programma ble
ways, using com mon tools and lan guages to pro vide network en gi
neers, de vel
opers and
even novices an ap proach able path toward au toma tion. Though ACME is just get ting
started with true De vOps in their IT or ga
nization, they re al
ize that these ben e
fits will
allow them to keep up with the pace of busi ness.
Given the com pre hensive

ness of the pro gram mabil
ity features avail
able on ACI, every -
one can ben e
fit. ACME's net work en gi
neering and de sign teams can ben efit from the
quick time to pro vi
sion large con fig
ura
tions, and the con sis
tency provided by the abil -
ity to auto
mate all of the mov ing parts. Their op er
ations teams can uti lize the plethora
of infor
mation con tained within the APIC to stream line their processes, gather bet ter
met rics and cor re
late events more ac curately, yielding faster time to res olu
tion and
higher cus tomer sat is
fac
tion.
Scripting 355
ACI and Scripting
The goals for network programmabil

ity are clear, however the meth ods by which these
goals may be real
ized have been more dif fi
cult to grasp. Tra
di
tional net
work ing de
vices
provide out
put that is meant for vi
sual con sump tion by people, and config
ur
a
tions are
dri
ven using text input that is sim
pler for a per son to type, how ever these goals stand
in contrast to an au tomation-dri
ven ap proach. Ma chines are able to more eas ily
process data that is pro
vided in some struc tured form. Structured data that may not be
vi
su
ally ap
peal
ing can be rapidly parsed, and also can easily rep
re
sent the full de
tail
that a compre
hensive ob
ject-ori
ented con
fig
ur
a
tion model may repre
sent.
ACI uses an ad vanced ob ject model that rep resents network configur
a
tion with ap pli
-
ca
tion-based se mantics which can be con sumed and posted against using a well doc u-
mented REST API. In ad di
tion to provid
ing this in
ter
face into the object model, ACI also
provides a num ber of ac
cess meth ods to read and ma nip
ul
ate this data, at a vari
ety of
lev
els that will cater to the level of comfort the user has with program ming, all of which
use open stan dards and open source.
Reference to Object Model
Representation of the top levels of the Object Model
While a com
pre hen
sive overview of the Ob ject Model is outside of this book, from a
pro
gramma
bil
ity per
spec
tive it is im
por
tant to note that every as
pect of ACI func
tion
al
-
356 Scripting
ity is en
com
passed within the ob ject model. This means that all of the con
fig
ur
a
tion
that can be made on the fab
ric, can be made pro
grammat
ic
ally using the REST API. This
in
cludes inter
nal fabric networking, ex
ternal net
work ing, vir
tu
al
iza
tion in
te
gra
tion,
com pute in
te
gra
tion, and all other facets of the prod
uct.
This data is stored within the Man

agement Infor
mation Tree, with every piece of the
model represented as a pro
gram
matic ob
ject with proper
ties, iden
tity, and con
sis
tency
rules that are en
forced. This en
sures that the con
fig
ured state of the model will never
get out of hand with stale nodes or entries, and every as
pect can be in
spected, ma
nip
u-
lated, and made to cater for the user's needs.
Programmatic Interfaces
APIC is very flexib
le in terms of how it can accept config
ur
ation and pro vide admin
is
-
tra
tive and op er
able states, as well as ex
tending that configur
a
tion into sub or
di
nate
com ponents. There are two pri mary cate
gories of in
ter
faces that facil
it
ate these func-
tions: the northbound REST API and the south bound pro gram matic in
ter faces.
The north bound REST API is re sponsi

ble for ac
cepting con fig
ura
tion, as well as provid
-
ing ac cess to man age
ment func tions for the controller. This interface is a cru
cial com-
ponent for the GUI and CLI, and also pro vides a touch point for au toma tion tools, pro
-
vi
sion ing scripts and third party mon it
or
ing and man agement tools. The REST API is a
sin
gu lar entry point to the fabric for mak ing config
ura
tion changes, and as such is a
crit
ic
al aspect of the ar
chi
tec
ture for being able to pro vide a con sistent program matic
experience.
South bound in ter

faces on APIC allow for the de clarat
ive model of intent to be ex
tended
be
yond the fab ric, into subor
di
nate de vices. This is a key as
pect to the open ness of the
ACI fabric, in that policy can be programmed once via APIC and then pushed out to hy -
pervi
sors, L4-7 de vices and potentially more in the fu ture, with
out the need to indi
vid
-
u
ally con fig
ure those de vices. This south bound ex ten
sion is real
ized through two
meth ods: L4-7 De vice Pack ages and OpFlex.
The L4-7 de vice pack age in

ter face al
lows for ACI to apply policy to ex
ist
ing L4-7 de -
vices that do not have an im plicit knowledge of ACI policy. These de
vices can be from
any vendor, so long as the de vice has some form of in ter
face which is ac
cessi
ble via IP.
The actual imple
men ta
tion of de vice packages is done via Python scripts which run on
the APIC within a con tained ex ecu
tion envi
ronment, which can reach the de vice
through their na tive config
ura
tion inter
faces, be that REST, CLI, SOAP or oth ers. As a
Scripting 357
user makes changes to ser vice graphs or EPG policy, the de

vice pack
age will trans
late
the APIC pol
icy into API calls on the L4-7 de
vice.
OpFlex is designed to allow a data ex

change of a set of managed objects that is de
fined
as part of an in
for
ma
tional model. OpFlex it
self does not dic
tate the in
for
mation model,
and can be used with any tree-based ab
stract model in which each node in the tree has
a uni
ver
sal re
source iden
ti
fier (URI) as
so
ci
ated with it. The pro
to
col is de
signed to sup
-
port XML and JSON (as well as the bi
nary en
cod
ing used in some sce
nar
ios) and to use
stan
dard re
mote pro
ce
dure call (RPC) mech
an
isms such as JSON-RPC over TCP. In ACI,
OpFlex is cur
rently used to ex
tend pol
icy to the Ap
pli
ca
tion Vir
tual Switch as well as
ex
tend Group Based Pol
icy into Open
Stack.
REST
The Cisco APIC REST API is a program matic inter
face for Cisco APIC that uses REST ar-
chi
tec
ture. The API accepts and returns HTTP (not en abled by de
fault) or HTTPS mes -
sages that contain JavaScript Object Nota
tion (JSON) or Ex ten
si
ble Markup Lan guage
(XML) documents. You can use any pro gram ming lan guage to gener
ate the mes sages
and the JSON or XML doc um
ents that con tain the API meth ods or MO de scrip
tions.
The REST API is the in terface into the MIT and al lows manipula
tion of the object model
state. The same REST in terface is used by the Cisco APIC com mand-line in terface (CLI),
GUI, and SDK, so that when ever information is dis
played, it is read through the REST
API, and when con fig
ur
ation changes are made, they are writ ten through the REST API.
The REST API also pro vides an in terface through which other in for
mation can be re -
trieved, in
cluding sta
tis
tics, faults, and audit events, and it even pro vides a means of
subscrib
ing to push-based event no ti
fi
ca
tion, so that when a change oc curs in the MIT,
an event can be sent through a web socket.
Stan
dard REST meth ods are sup ported on the API, which in cludes POST, GET, and
DELETE op er
at
ions through HTTP. The POST and DELETE meth ods are idem po
tent,
meaning that there is no ad
di
tional ef
fect if they are called more than once with the
same input para
meters. The GET method is nul lipo
tent, mean ing that it can be called
zero or more times without making any changes (or that it is a read-only op er
at
ion).
Payloads to and from the REST in ter

face can be en cap
su lated through ei ther XML or
JSON en coding. In the case of XML, the encoding oper
at
ion is sim ple: the el
e
ment tag is
the name of the pack age and class, and any properties of that object are specif
ied as at
-
trib
utes of that el
e
ment. Con tain
ment is defined by creating child el
ements.
358 Scripting
For JSON, en
cod
ing re
quires de
f
in
i
t
ion of cer
tain en
ti
ties to re
flect the tree-based hi
-
er
ar
chy; how
ever, the de f
in
i
t
ion is re
peated at all lev
els of the tree, so it is fairly sim
ple
to im
ple
ment after it is ini
tially under
stood.
All objects are described as JSON dictionaries, in which the key is the name of the
package and class, and the value is another nested dictionary with two keys:
attribute and children.
The attribute key contains a further nested dictionary describing key-value pairs
that define attributes on the object.
The children key contains a list that defines all the child objects. The children in
this list are dictionaries containing any nested objects, which are defined as
described here.
Read Operations
After the ob
ject payloads are properly encoded as XML or JSON, they can be used in
cre
ate, read, update, or delete operat
ions on the REST API. The fol
low
ing di
ag
ram
shows the syntax for a read op
er
at
ion from the REST API.
REST syntax
Be
cause the REST API is HTTP based, defin ing the univer
sal re
source identi
fier (URI) to
ac
cess a cer
tain re
source type is im
portant. The first two sections of the request URI
sim
ply de
fine the pro
to
col and access de
tails of Cisco APIC. Next in the request URI is
eral string /api, in
the lit di
cat
ing that the API will be in voked. Gen erally, read op
era-
tions are for an object or class, as dis
cussed ear
lier, so the next part of the URI specif
ies
Scripting 359
whether the op
er
at
ion will be for an MO or class. The next com
po
nent de
fines ei
ther
the fully qual
if
ied Dn being queried for ob
ject-based queries, or the pack
age and class
name for class-based queries. The final manda tory part of the re
quest URI is the en
-
coding for
mat: ei
ther .xml or .json. This is the only method by which the pay
load for
mat
is de
fined (Cisco APIC ig
nores Con tent-Type and other head ers).
Write Operations
Cre
ate and up
date op
er
at
ions in the REST API are both im
ple
mented using the POST
method, so that if an ob ject does not already exist, it will be cre
ated, and if it does al
-
ready exist, it will be up
dated to re
flect any changes be tween its exist
ing state and de-
sired state.
Both create and up date op

er
ations can contain com
plex object hi
er
ar
chies, so that a
complete tree can be defined in a sin
gle command so long as all ob
jects are within the
same con text root and are under the 1MB limit for data pay
loads for the REST API. This
limit is in place to guar
an
tee per
for
mance and pro
tect the sys
tem under high load.
The context root helps define a method by which Cisco APIC dis trib
utes in
for
mation to
mul
ti
ple controllers and helps ensure con
sis
tency. For the most part, the con fig
ur
a
tion
should be trans parent to the user, though very large con fig
ur
ations may need to be
bro
ken into smaller pieces if they re
sult in a dis
trib
uted trans
ac
tion.
REST Payload
360 Scripting
Cre
ate and up
date op
er
at
ions use the same syn
tax as read op
er
at
ions, ex
cept that they
al
ways are targeted at an ob ject level, be
cause you can
not make changes to every ob-
ject of a spe
cific class (nor would you want to). The create or up
date op
er
at
ion should
tar
get a spe
cific managed ob eral string /mo in
ject, so the lit di
cates that the Dn of the
managed object will be pro
vided, followed next by the ac tual Dn. Fil
ter strings can be
applied to POST op er
at
ions; if you want to re
trieve the re
sults of your POST op er
at
ion
in the response, for example, you can pass the rsp-sub tree=mod i
fied query string to
in
di
cate that you want the re sponse to in
clude any objects that have been mod if
ied by
your POST op er
ation.
The payload of the POST op er

at
ion will con
tain the XML or JSON en
coded data rep
re
-
sent
ing the man aged ob
ject the de
fines the Cisco API com
mand body.
Authentication
REST API user name- and pass word-based au thenti
ca
tion uses a special sub
set of re
-
quest URIs, in ing aaaLo
clud gin, aaaL
ogout, and aaaRefresh as the Dn targets of a POST
op
er
at
ion. Their payloads contain a sim
ple XML or JSON pay load contain
ing the MO
repre
sen tion of an aaaUser ob
ta ject with the at tribute name and pwd defin ing the
username and pass word: for exam ple, <aaaUser name='admin' pwd='in sieme'/>. The
re
sponse to the POST op erat
ion will contain an au thenti
ca
tion token as both a Set-
Cookie header and an at tribute to the aaaLo gin object in the response named token,
for which the XPath is /im data/aaaLo gin/@to ken if the encoding is XML. Sub
sequent
oper ions on the REST API can use this token value as a cookie named APIC-cookie to
at
au
then ti
cate fu
ture re
quests.
Filters
The REST API sup

ports a wide range of flex
i
ble fil
ters, use
ful for nar
row
ing the scope of
your search to allow infor
mation to be located more quickly. The filters them selves are
ap
pended as query URI op tions, start
ing with a question mark (?) and concate
nated with
an amper
sand (&). Mul
ti
ple condi
tions can be joined together to form complex fil
ters.
Scripting 361
The fol
low
ing query fil
ters are avail
able:
Scripting 363
API Inspector
All op
er
at
ions that are per
formed in the GUI invoke REST calls to fetch and com mit the
in
for
mation being accessed. The API In
spector fur
ther sim
pli
fies the process of exam-
in
ing what is tak
ing place on the REST in
ter
face as the GUI is navig
ated by dis
play
ing in
real time the URIs and pay loads. When a new config
ur
a
tion is commit ted, the API In
-
spector dis
plays the re
sult
ing POST re
quests, and when infor
mation is dis
played on the
GUI, the GET re quest is dis
played.
To get started with the API In spec

tor, it can be ac
cessed from the Ac
count menu, vis
i-
ble at the top right of the Cisco APIC GUI. Click Wel come, <user
name> and then choose
the Show API In spector option
After the API In

spector is brought up, time stamps will appear along with the REST
method, URIs, and payloads. There may also be oc
ca
sional up
dates in the list as the GUI
re
freshes sub
scrip
tions to data being shown on the screen.
API Inspector
364 Scripting
From the out

put above it can see that the last logged item has a POST re quest with the
JSON pay
load contain ant named Cisco and some at
ing a ten trib
utes de
fined on that ob
-
ject:
url: http://
172.
16.
176.
176/
api/
node/
mo/
uni/
tn-Cisco.
json
{
"fvTenant": {
"attributes": {
"name": "Cisco",
"status": "created"
},
"children": [
{
"fvBD": {
"attributes": {
"mac": "00:22:BD:F8:19:FF",
"name": "CiscoBd",
"status": "created"
},
"children": [
{
"fvRsCtx": {
"attributes": {
"tnFvCtxName": "CiscoVrf",
"status": "created,modified"
},
"children": []
}
}
]
}
},
{
"fvCtx": {
"attributes": {
"name": "CiscoVrf",
"status": "created"
Scripting 365
},
"children": []
}
}
]
}
}
Scripting 367
Development Techniques
ACI has a num ber of methods for de

vel
op
ing code that can be used by engi
neers who
have vary
ing lev
els of com
fort with pro
gramming or in
ter
act
ing with pro
grammatic in
-
ter
faces.
The most basic and straight-for

ward tech
nique in
volves sim
ply tak
ing in
for
ma
tion
gleaned by the API in
spec
tor, Vi
sore, or by sav
ing XML/JSON directly from the GUI,
and using common freely avail
able tools, such as POST
man, to send this in
for
ma
tion
back to the REST API.
A step up from this method en ables users to use com mon terminol
ogy and well un der-
stood net work ing con
structs, cou pling these with the power and flex ib
il
ity of the ACI
pol
icy language and the pop ul
ar Python pro gram ming language to con figure ACI in a
programmatic fash ion. ACI Toolkit is a utility devel
oped in open-source that ex poses
the most com mon ACI build ing blocks, to en able users to rapidly cre
ate ten ants, appli
-
ca
tion pro files, EPGs and the as so
ciated con cepts to connect those to phys ic
al in
fra-
structure. The stream lined inter
face provided makes it very quick to adopt and al lows
users to begin to quickly de velop their appli
cations.
The most pow er

ful of the de
vel
op
ment tools avail able is the Cobra SDK. With a com -
plete rep
re
sen ta
tion of the ACI ob
ject model avail able, compre
hensive data val
id
a
tion,
and exten
sive sup port for query
ing and fil
ter
ing, Cobra en sures that the complete ACI
ex
peri
ence is avail
able to de
vel
op
ers and users alike.
Scripting 369
POSTman
POST
man is an open source ex ten
sion for the Chrome web browser, which pro vides
REST client func
tion
al
ity in an easy-to-use package. POST
man can be used to in
ter
act
with the APIC REST in
ter
face, to both send and re
ceive data which may rep
re
sent con
-
fig
ur
a
tion, ac
tions, pol
icy and op er
at
ional state data. For an indi
vid
ual un
fa
mil
iar with
the struc
ture of REST, it is very simple to uti
lize the API In
spec
tor to view what the un-
der
ly
ing calls being made to the GUI are for cer
tain op
er
at
ions, cap
ture those, and then
use POST
man to re
play those op
er
at
ions. Fur
ther
more POST
man al
lows for the re
-
quests to be mod
if
ied: GUI op
er
at
ions can be made once, at
trib
utes changed in the
cap
tured data and then sent back to the REST API to make the mod
if
i
ca
tions.
Installation
To get started with POST man, the first step is to down
load the plu
gin for the Chrome
web browser, which is available at www. getpostman.com. Once the plu gin is in
stalled,
it can be ac
cessed using the Chrome App launcher.
Ini
tially the user will be pre
sented with an inter
face that has two pri
mary sections: the
sidebar on the left and the request con
struc
tor on the right. Using the side
bar, the user
can switch be tween the history of REST re
quests sent by POST man, as well as Collec
-
tions of requests that contain common tasks.
Collections
A useful post to cre
ate in a col
lec
tion is a basic Login oper
at
ion. In order to do this, the
user should first click into the Collec
tions tab in the side
bar. Within the sidebar, a small
folder with a plus (+) sign will become vis ib
le, which should then be clicked, at which
point a popup will ap pear prompt ing the user to give a name to the col lec
tion. For this
example, the col
lec
tion can be named APIC, after which the Cre ate but
ton should be
clicked.
370 Scripting
Build Login request

Now a new re
quest can be built. In the re
quest con
struc
tor, where Enter re
quest URL
here is shown, the fol
low
ing re
quest URI should be en
tered, sub
sti
tut
ing APIC
IP
AD
-
DRESS with the IP of the APIC: https://<APIC-IP>/api/mo/aaaLogin.
xml
This re
quest URI will call the Login method in the REST API. Since a Login will re
quire
post
ing data, the HTTP method should be changed, which can be done by click
ing the
drop
down list to the right of the re
quest URL. By de
fault it will be a GET re
quest, but
POST will need to be se
lected from the drop down list.
With the POST method se

lected, it is now pos
si
ble to pro
vide the REST pay
load. Given
that the data will be sent via REST, the raw Re
quest body se
lec
tor should be picked.
Now the pay load for the re

quest can be entered, which will be the simple XML contain
-
ing the username and pass word that will be used for au
thenti
ca
tion. Note that the URL
is https, meaning that it will be en
crypted be
tween the web browser and the APIC, so
no data is being trans mit
ted in clear text. The fol
low
ing request body should be en -
tered, substi
tut
ing the correct username and pass word in place of USER NAME and
PASS WORD: <aaaUser name='USERNAME' pwd='PASSWORD'/>
With this re
quest built, it is now pos si
ble to Send the re quest, but since this will be a
commonly used method, the re quest should be added to a col lec
tion. This can be ac-
complished by click
ing the Add to col lec
tion button be neath the request body. Select
the APIC col
lec
tion from the exist
ing col lec
tion list, and change the Re quest name to
Login and then click Add to col
lection.
By adding the re
quest to a col
lec
tion it can later be quickly ac
cessed to es
tab
lish a login
ses
sion with APIC as needed.
After com plet

ing the above steps, the re
quest is ready to be sent. Click the Send but
-
ton in the request construc
tor, and the REST API will re turn the XML repre
sent
ing a
login ses
sion with the APIC. The follow
ing will be vis
ib
le in the POSTman GUI:
Scripting 371
Login request in POSTman
Make Query to APIC

The next re
quest that will be built is one that queries the APIC for a list of ten
ants on
the sys
tem. First click the Reset but
ton in the re
quest con
struc
tor, and pro ceed with
the same steps as above, ex cept that the re
quest URL will be shown as https://<APIC-
IP>/api/class/fvTenant.
xml, and the re
quest method will be changed to GET.
Click Add to col

lec
tion and place the re
quest into the APIC col
lec
tion, and for the
name enter Query APIC for tenants
Now upon click

ing Send, this re
quest will re
turn an XML encoded list of ten
ants in the
re
sponse body sec
tion of the construc
tor pane on the right.
Make Configuration Change in APIC

Mak
ing a con
fig
ur
a
tion change will use a POST re
quest sim
il
ar to log
ging in, how
ever
the re
quest URL and body will con
tain a dif
fer
ent set of in
for
ma
tion.
372 Scripting
For this ex
am
ple, a new ten
ant will be cre
ated in the fab
ric. Click the Reset but
ton in
the request con
struc
tor to clear out all ex
ist
ing re
quest fields, and use URL
https://<APIC-IP>/api/mo/uni.xml and change the method to POST.
In the re
quest pay
load, enter the fol
low
ing data: <fvTenant name="Cisco"/>
The request URL specif

ies that the tar get for this query will be the policy uni
verse,
which is where ten
ants live. With this target prop
erly scoped, the data rep
resent
ing the
ten
ant can be pro
vided in the payload, in this case cre
at
ing a ten
ant named Cisco.
Use API Inspector for Query Guidance

As dis
cussed in the In tro
duc
tion to Script
ing section, API in
spec
tor can be used as a
guide
line for build
ing custom REST requests. Fur
thering on the ex
ample in that sec
tion,
where the re quest URL is https://<APIC-IP>/api/node/mo/uni/tn-Cisco. json and
the pay
load is the fol
low
ing com
pacted ver
sion of JSON:
{"fvTenant": {"attributes": {"name": "Cisco", "status": "created"}, "children":

[{"fvBD": {"attributes": {"mac": "00:22:BD:F8:19:FF", "name": "CiscoBd", "status":
"created"}, "children": [{"fvRsCtx": {"attributes": {"tnFvCtxName": "CiscoVrf",
"status": "created,modified"}, "children": [] } } ] } }, {"fvCtx": {"attributes":
{"name": "CiscoVrf", "status": "created"}, "children": [] } } ] } }
It is possi
ble to modify the re
quest URI and pay load and substi
tute the tenant name
Cisco with an other tenant name, to cre
ate an en
tirely new ten
ant, with the same con
-
fig
ura
tion. The new re quest URL and JSON would be: https://<APIC-
IP>/api/node/mo/uni/tn-Acme.
json
{"fvTenant": {"attributes": {"name": "Acme", "status": "created"}, "children":

[{"fvBD": {"attributes": {"mac": "00:22:BD:F8:19:FF", "name": "AcmeBd", "status":
"created"}, "children": [{"fvRsCtx": {"attributes": {"tnFvCtxName": "AcmeVrf",
"status": "created,modified"}, "children": [] } } ] } }, {"fvCtx": {"attributes":
{"name": "AcmeVrf", "status": "created"}, "children": [] } } ] } }
Scripting 373
These val
ues can be placed into a POST re
quest in POST
man, and after es
tab
lish
ing a
Login ses
sion using the saved Login re
quest, the new ten
ant Acme can be cre
ated,
iden
ti
cal to the pre
vi
ously cre
ated Cisco ten
ant, with
out need
ing to man
ua
lly click
through the GUI or use other man
ual meth
ods.
Scripting 375
Cobra SDK and Arya
The com plete Cisco ACI Python SDK is named Cobra. It is a pure Python im ple
men ta
-
tion of the API that pro
vides na
tive bind
ings for all the REST func tions and also has a
complete copy of the ob
ject model so that data integrity can be ensured, as well as sup-
port
ing the com plete set of fea
tures and func tions available in ACI. Cobra pro vides
methods for performing lookups and queries and ob ject cre
ation, mod if
i
ca
tion, and
dele
tion that match the REST meth ods used by the GUI and those that can be found
using API In
spector. As a re
sult, pol
icy cre
ated in the GUI can be used as a pro
gram
-
ming template for rapid devel
op
ment.
The instal
la
tion process for Cobra is straight for
ward, and you can use stan dard Python
dis
tri
bu
t
ion utilit
ies. Cobra is dis
trib
uted on the APIC as an .egg file and can be in
stalled
using easy_in stall, and is also available on github at http://github.com/ datacenter/
cobra.
The com plete doc

um
enta
tion for the Cobra SDK is avail
able at http://
cobra.
readthedocs.
org/
en/
latest/
Establish Session
The first step in any code that uses Cobra is estab
lish
ing a login ses
sion. Cobra cur -
rently supports username- and password-based au thenti
ca
tion, as well as cer
tifi
cate-
based authenti
ca
tion. The ex
am
ple here uses username- and pass word-based au then-
ti
ca
tion.
import cobra.mit.access
import cobra.mit.session
apicUri = 'https://10.0.0.2'
apicUser = 'username'
apicPassword = 'password'
ls = cobra.mit.session.LoginSession(apicUri, apicUser, apicPassword)
md = cobra.mit.access.MoDirectory(ls)
md.login()
376 Scripting
This ex
am vides an MoDi
ple pro rec
tory ob
ject named md, which is logged into and au
-
then
ti
cated for Cisco APIC. If for some reason au
then
ti
ca
tion fails, Cobra will dis
play a
cobra.
mit.
request.
CommitEr ror excep
tion mes
sage. With the session logged in, you are
ready to pro
ceed.
Work with Objects

Use of the Cobra SDK to ma
nip
ul
ate the MIT gen
er
ally fol
lows this work
flow:
1 Identify the object to be manipulated.
2 Build a request to change attributes or add or remove children.
3 Commit the changes made to the object.
For ex
am ple, if you want to cre
ate a new ten
ant, you must first iden
tify where the ten
-
ant will be placed in the MIT, where in this case it will be a child of the policy Uni-
verse man aged ob Un
ject (poliMo):
import cobra.model.pol
polUniMo = cobra.model.pol.Uni('')
With the pol

Un
iMo ob
ject de
fined, you can cre
ate a ten ject as a child of pol
ant ob U-
niMo:
import cobra.model.fv
tenantMo = cobra.model.fv.Tenant(polUniMo, 'cisco')
All these op
er
at
ions have re
sulted only in the cre
ation of Python ob
jects. To apply the
con
fig
ur
ation, you must com mit it. You can do this using an object called a Con
fi
gRe-
quest. Con fi
gRe
quest acts as a con tainer for MO-based classes that fall into a sin
gle
con
text, and they can all be commit ted in a sin
gle atomic POST op
erat
ion.
import cobra.mit.request
config = cobra.mit.request.ConfigRequest()
config.addMo(tenantMo)
md.commit(config)
Scripting 377
The Con
fi
gRe
quest ob ated, then the ten
ject is cre antMo ob
ject is added to the re
quest,
and then you com
mit the con
fig
urtion through the MoDi
a rec
tory ob
ject.
For the pre

ced
ing ex ple, the first step builds a local copy of the pol
am Uni ob
ject. Be
-
cause it does not have any nam
ing prop
er
ties (re
flected by the empty dou
ble sin
gle
quota
tion marks), you dont need to look it up in the MIT to fig
ure out what the full Dn
for the ob ways known as uni.
ject is; it is al
If you wanted to post some thing deeper in the MIT, where the ob ject has nam ing prop
-
erties, you would need to per form a lookup for that ob ject. For ex
am ple, if you wanted
to post a config
uration to an exist
ing ten
ant, you could query for that ten ant and create
objects beneath it.
tenantMo = md.lookupByClass('fvTenant', propFilter='eq(fvTenant.name, "cisco")')

tenantMo = tenantMo[0] if tenantMo else None
The re ing ten

sult antMo ob ject will be of class cobra.
model.fv.
Tenant and will con tain
prop
er
ties such as .dn, .sta
tus, and .name, all describ
ing the ob self. The lookup
ject it -
By
Class() entry re
turns an array, be cause it can return more than one ob ject. In this
case, the com mand is spe cific and is fil ing on an fv
ter Tenant object with a partic
ul
ar
name. For a ten ant, the name at tribute is a spe
cial type of at
tribute called a naming at
-
tribute. The nam ing at
tribute is used to build the rel at
ive name, which must be unique
in its local namespace. As a re sult, you can be as sured that lookup ByClass on an fv-
Tenant ob ject with a fil
ter on the name al ways returns either an array of length 1 or
None, mean
ing that noth
ing was found.
To en
tirely avoid a lookup, you can build a Dn ob
ject and make an ob ject a child of that
Dn. This method works only in cases in which the par ent ob
ject al
ready exists.
topDn = cobra.mit.naming.Dn.fromString('uni/tn-cisco')
fvAp = cobra.model.fv.Ap(topMo, name='AppProfile')
These funda
men tal meth
ods for in
ter
acting with Cobra provide the build
ing blocks
nec
es
sary to cre
ate more complex workflows that can help au
to
mate net
work config
u-
ra
tion, per
form trou
bleshoot
ing, and man
age the net
work.
378 Scripting
Cisco APIC REST to Python Adapter

The process of build
ing a re
quest can be time con
sum
ing, be
cause you must rep
re
sent
the ob
ject data pay
load as Python code re
flect
ing the ob
ject changes that you want to
make. Because the Cobra SDK is di
rectly modeled on the Cisco ACI object model, you
should be able to gen
er
ate code directly from what resides in the ob
ject model. As ex -
pected, you can do this using a tool de
veloped by Cisco Advanced Ser vices. The tool is
the Cisco APIC REST to Python Adapter, known as Arya.
Sample REST to Python Adapter
The above fig

ure clearly shows how the input that might come from the API In
spec
tor,
Vi
sore, or even the output of a REST query and can then be quickly converted into
Cobra SDK code, toke
nized, and reused in more ad
vanced ways.
In
stal
la
tion of Arya is relat
ively simple, and the tool has few exter
nal de
penden
cies. To
in
stall Arya, you must have Python 2.7.5 and git in stalled. Use the fol
low
ing quick in
stal
-
la
tion steps to in
stall it and place it in your sys
tem Python.
git clone https://github.com/datacenter/ACI.git

cd ACI/arya
sudo python setup.py install
Scripting 379
After Arya has been in

stalled, you can take XML or JSON rep re
senting Cisco ACI mod
-
eled ob
jects and con
vert it to Python code quickly. For ex
am
ple, enter:
arya.py -f /home/palesiak/simpletenant.xml
The entry will yield the fol

low
ing Python code:
#!/usr/bin/env python
'''
Autogenerated code using arya.py
Original Object Document Input:
<fvTenant name='bob'/>
''' raise RuntimeError('Please review the auto generated code before ' +
'executing the output. Some placeholders will ' +
'need to be changed')
# list of packages that should be imported for this code to work
import cobra.mit.request
import cobra.model.fv
import cobra.model.pol
from cobra.internal.codec.xmlcodec import toXMLStr
# log into an APIC and create a directory object
ls = cobra.mit.session.LoginSession('https://1.1.1.1', 'admin', 'password')
md.login()
# the top level object on which operations will be made
topMo = cobra.model.pol.Uni('')
# build the request using cobra syntax
fvTenant = cobra.model.fv.Tenant(topMo, name='bob')
# commit the generated code to APIC
print toXMLStr(topMo)
c = cobra.mit.request.ConfigRequest()
c.addMo(topMo)
md.commit(c)
380 Scripting
The place
holder rais
ing a run
time error must first be re
moved be
fore this code can be
ex
e
cuted; it is pur
posely put in place to help en
sure that any other to
ke
nized val
ues
that must be up
dated are cor
rected. For ex
am
ple, the Cisco APIC IP ad
dress, which de
-
faults to 1.
1.
1.
1, should be up dated to re
flect the ac
tual Cisco APIC IP ad
dress. The same
ap
plies to the cre dentials and any other place hold
ers.
Note that if you pro

vide input XML or JSON that does not have a fully qual
if
ied hi
er
ar
-
chy, Arya may not be able to identify it through heuris
tics. In this case, a place
holder
will be pop ated with the text RE
ul PLACEME, which you will need to re place with the
cor
rect Dn. You can find this Dn by querying for the ob
ject in Vi
sore or in
spect
ing the
re
quest URI for the ob
ject shown in the API In
spector.
Scripting 381
ACI Toolkit
The com plete ACI ob ject model con

tains many en
ti
ties, which may be daunt ing for a
user being first in
tro
duced to net
work program
mabil
ity. The ac
it
oolkit makes avail
able
a sim
pli
fied sub
set of the model that can act as an in
tro
duc
tion to the con
cepts in ACI,
and give users a way to quickly bring up com mon tasks and work
flows. In ad
di
tion, a
number of ap
pli
ca
tions have been built on top of ACI toolkit.
The com
plete doc
um
en
ta
tion for ac
it
oolkit is avail
able at http://
datacenter.
github.
io/
acitoolkit/
ACI Toolkit Applications
Endpoint Tracker
The end point tracker appli

ca
tion cre
ates a sub scription to the end point class (fvCEp)
and popul
ates a MySQL data base with per ti
nent details about each endpoint present on
the fab
ric (for ex
ample servers, fire
walls, load bal
ancers, and other de vices). In
stalling
MySQL is out side the scope of this book, so we will as sume you have ac cess to create a
new data base on a MySQL server. The end point tracker ap pli
ca
tion has two pri mary
components that are both python scripts.
aci-endpoint-tracker.pyThis script creates the subscription to the endpoint

class and populates the MySQL database
aci-endpoint-tracker-gui.pyThis script creates a web interface that provides
a way to present the contents of the database to the operator. A sample is
shown below:
382 Scripting
To launch Endpoint Tracker run the following python scripts. The first script, aci-
endpoint-tracker.py, will actually connect to the APIC and populate the database. The
second script enables the content to be viewed in an understandable web UI.
user@linuxhost:~/acitoolkit/applications/endpointtracker$ ./aci-endpoint-tracker.py
MySQL IP address: 127.0.0.1
MySQL login username: root
MySQL Password:
user@linuxhost::~/acitoolkit/applications/endpointtracker$ python aci-endpoint-
tracker-gui.py
MySQL IP address: 127.0.0.1
MySQL login username: root
MySQL Password:
* Running on http://127.0.0.1:5000/
* Restarting with reloader
After run
ning those python scripts you can now bring up a browser and go the Web
UI. Using the ACI End
point Tracker is sim
ply a mat
ter of in
putting an IP or MAC ad
-
dress into the search field, and the table is fil
tered ac
cord
ingly. In the ex
am
ple below,
the IP address 192.168.
5.
20 has been input into the search field, and the matching re
-
sults are dis
played.
Scripting 383
One more interesting usage of the endpoint tracker applications is a series of

visualizations that can represent how various endpoints are mapped to other fabric
constructs including Tenants, Applications, and EPGs.
Some sam
ple screen
shots are shown below. These are rep
re
sen
ta
tions of where end
points are within the ACI fab
ric and how they re
late to or de
pend on other ob
jects in
the en
vi
ron
ment.
Pie chart view of endpoint distribution

384 Scripting
Tree view of endpoint relationships
Force diagram of endpoint location
ACI Lint
Scripting 385
ACI Lint
In com puter pro
gram ming, Lint is a term that refers to iden
ti
fy
ing discrepan
cies, or
simple debug tool for com
mon er rors. In the sense that ACI provides in
fra
struc
ture as
code, it is ap
propri
ate for ACI to also have a Lint appli
ca
tion. ACI Toolkit pro
vides just
that. ACI Lint is an appli
ca
tion that checks and noti
fies an op
erat
or of mis
con
fig
ur
a
tion
er
rors in two pri
mary ca
pac
it
ies:
Se
curity Is
sues - supports the abil
ity to tag EPGs as either secure or in
se
cure, and
then runs a val
id
a
tion that con
tracts are not used to cross se
cu
rity boundaries.
Con
fig
u
ra
tion Is
sues - checks for com
mon con
fig
ur
a
tion er
rors and re
ports them to
the user.
A sam
ple out
put is pro
vided here for ref
er
ence:
user@linuxhost:~/acitoolkit/applications/lint$ ./acilint.py
Getting configuration from APIC....
Processing configuration....
Critical 001: EPG 'default' in tenant 'infra' app 'access' is not assigned security
clearance
Critical 001: EPG 'x' in tenant 'common' app 'default' is not assigned security
clearance
Warning 001: Tenant 'Cisco' has no Application Profile.
Warning 001: Tenant 'Books' has no Application Profile.
Warning 001: Tenant '3tierapp' has no Application Profile.
Warning 001: Tenant 'mgmt' has no Application Profile.
Warning 002: Tenant 'Books' has no Context.
Warning 002: Tenant '3tierapp' has no Context.
Warning 004: Context 'oob' in Tenant 'mgmt' has no BridgeDomains.
Warning 005: BridgeDomain 'CiscoBd' in Tenant 'Cisco' has no EPGs.
Warning 005: BridgeDomain 'inb' in Tenant 'mgmt' has no EPGs.
Warning 006: Contract 'default' in Tenant 'common' is not provided at all.
Warning 006: Contract 'WebServers' in Tenant 'Acme' is not provided at all.
Warning 006: Contract 'External' in Tenant 'Acme' is not provided at all.
Warning 007: Contract 'default' in Tenant 'common' is not consumed at all.
Warning 007: Contract 'WebServers' in Tenant 'Acme' is not consumed at all.
Warning 007: Contract 'External' in Tenant 'Acme' is not consumed at all.
Warning 007: Contract 'outside-to-web' in Tenant 'roberbur' is not consumed at all.
386 Scripting
While the ACI Toolkit pro

vides some use
ful tools for an op
er
at
or to im
me
di
ately use,
the real value is in the abil
ity to take these ex
am
ples as a start
ing point, and mod
ify or
ex
tend these sam
ples to suit your par
tic
ul
ar needs. Give it a try! Be sure to share your
work back with the com
mu
nity!
Scripting 387
GitHub
Source Control
Open source soft
ware has been a pop
ul
ar move
ment in IT, and has been the mo
ti
va
tion
be
hind many suc
cess
ful pro
jects, in
clud
ing con
sumer soft
ware, web servers, data
bases
and even en
tire op
er
at
ing sys
tems. One of the key as
pects to the suc
cess of open
source is the abil
ity for many devel
op
ers around the globe to col
lab
or
ate to
gether on a
sin
gle project. Pre
vi
ous tools like Con
current Ver
sion Control (CVS), and Subver
sion
(SVN) were used to allow many de
vel
op
ers to work to
gether, with a cen
tral server
maintain
ing a com mon data base of source code. While these tools have and con tinue to
work well, there has been a slow mi gration away from those server-based tools to de -
cen
tral
ized util
it
ies, with the fore
most being Git. Git was cre ated by Linus Torvalds, the
au
thor of the pop ul
ar open-source op erat
ing sys
tem Linux. Git has a num ber of ad
van-
tages over most other source con trol tools: com plete local reposit
ory copies, dis
trib
-
uted ar
chitec
ture, and more ef fi
cient sup port for branches.
GitHub
GitHub is a host ing plat
form based around git, which pro vides both free and paid host -
ing services, that allow for in di
vidua
ls to collab
o
rate with over eight-mil lion other
GitHub users on pro jects to
gether. Aside from being a wrap per around git, GitHub also
provides techniques for track ing issues, se
curing ac
cess to projects, and built-in pro-
ject docum
en ta
tion. The com bi
nation of all of these fea
tures has made GitHub a very
com mon place for mem bers of the com munity to share code with one an other, build on
each other's work, and con tribute their efforts back into larger projects.
What is stored on GitHub is usu ally source code, not lim ited to any spe cific language,
how ever the git pro to
col it
self supports storage and ver sion con trol of any file type, so
its not un common for users to store doc um
en ta
tion or other con stantly chang ing files
in git. The pri mary ad vantage is that the version control pro vided by git al
lows a user to
revert a file back to any pre vi
ously stored ver sion, or alter
nately move for ward to a
newer ver sion. Git also main tains an audit of changes that have been made to files and
even has ad vanced sup port for branch ing ver
sions of files to allow mul ti
ple concur rent
mod if
i
ca
tions to a file to take place, and allow for them to be merged after work ef forts
have com pleted.
388 Scripting
"It's on github"
A com
mon phrase in mod
ern IT jar
gon is, Its on github, and for users fa
mil
iar with
GitHub, this is an in
vi
ta
tion to down
load, mod
ify and contribute to the pro
ject, how-
ever for those who have not had an in tro
duc
tion it can seem like a com plex topic.
GitHub is ac
tu
ally a very sim
ple tool to use, and the sim
plest way to begin to take ad
-
van
tage of the in
for
ma
tion stored on GitHub is to sim
ply ac
cess a pro
jects main page
and look for the Down
load ZIP but
ton at the bot
tom right of any pro
ject's main page.
The re
sult
ing down
loaded file will con
tain the lat
est ver
sion of the files in the pro
ject.
What a user does with these files will greatly de
pend on what the con
tents are, how
-
ever one of the most highly en couraged be hav
iors on GitHub is to pro
vide clear and
ob
vi
ous doc
umenta
tion for a pro
ject, so if a new user ac
cesses the front page of a pro
-
ject on Git, they will typic
ally be able to find in
struc
tions on how to down
load and in
-
stall the pro
ject, right on the first page they see.
For users look ing to contribute back to a pro

ject, the next step would be to sign up for
an account on GitHub, and down load a graphic
al-based client to provide a sim pler in
-
ter
face to the com mand line-based git tool. GitHub itself has a graph
ic
al client with the
Windows ver sion available at http://windows. github.com and the Mac ver sions at
http://mac.github.com. Other com mon source con trol tools in
clude Source Tree from
At
lass
ian, avail
able at http://sourcetreeapp.com.
Once a user has an ac count and a github client, they can Fork, or split off a pro ject
that is available into their own pri vate repos it
ory, make changes and com mit those
back to their pri vate branch. If those changes work, and the user wishes to con tribute
them back into the orig in
al pro
ject, it is possi
ble to sub mit a Pull re
quest, which es-
sentially means that the user is propos ing their efforts should be pulled back into the
origin
al pro
ject. The process can be that sim ple, though many more ad vanced projects
have stan dards and rules for con tributing to those pro jects that put in place re
quire-
ments around how work is com mit ted back into the pro jects, which may require some
read ing be
fore attempting to contribute.
389
Hardware Expansion
and Replacement
Hardware Expansion and Replacement 391
Section Content
Expanding and Shrinking the Fabric

Switches
Add Connected Switch
Pre-Provision Switch Before Connection
Decommission Existing Switch
APICs
Add New APIC
Decommission Existing APIC
Hardware Diagnostics and Replacement

Identify Hardware Failure
Resolve Leaf Hardware Failure
Resolve APIC Hardware Failure
Diagnose Equipment Failures
Expanding and Shrinking the Fabric
ACME may de cide to ex

pand their ACI fab
ric as their data cen
ter grows, which means
adding new leaf and spine switches, and possi
bly APICs. Gener
ally, spine switches are
added for more through
put, and leaf switches are added for more ac
cess ports. APICs
are added as the num ber of poli
cies and endpoints in crease. Ad
dition
ally, some
switches or APICs may need to be de
commis
sioned. Also, there may be times when you
need to re
place failed hard
ware, which is dis
cussed in the Hard
ware Re
place
ments
chap
ter.
This sec
tion will walk through the op er
at
ions of adding and re
mov
ing switches and
APICs in your exist
ing ACI fab
ric. This is done the same way for both spine and leaf
switches. Adding APICs will also be cov
ered.
Switches
There are two ways switches can be added to ACME's ex ist
ing fab
ric: by discov er
ing the
switches automat
ic
ally in the APIC after they have been ca bled to the fab ric, or by pre-
pro
vi
sioning the switches by adding their se ri
al numbers and later con nect ing them
phys
ic
ally to the fabric when the switches ar rive. Both meth ods have the same out -
come: an ex panded fabric in the mat
ter of minutes. This section will also cover de com-
mis
sioning switches.
Add Connected Switch

To add a switch that has al
ready been at
tached to the fab
ric go through the fol
low
ing
steps in the APIC GUI:
1 In the case of a leaf switch, cable it to all of the spine switches. In the case of a
spine switch, cable it to all the leaf switches. Ideally, a best-practice ACI fabric
is connected in a full mesh topology with every leaf cabled to every spine. All
devices should connect to the leaf switches, leaves should never connect to
other leaves, and spines should never connect to other spines.
2 On the APIC click on Fabric at the top of the screen.
3 Click on Fabric Membership in the left navigation pane.

394 Hardware Expansion and Replacement
4 When the new switch appears, you'll see a node with a serial number but no
Node ID or Node Name configured. Double click the switch and assign a Node
ID and a Node Name. As a best practice, number leaf nodes starting with 101,
and spine nodes with 201. Lower numbers are reserved for APICs.
5 Optionally, add a Rack Name name. This is commonly used to identify the
physical location of the switch in the data center.
6 Click Submit.
7 Repeat this process for all new switches connected to the fabric.
Pre-Provision Switch Before Connection

Pre-pro
vi
sion
ing a switch is a handy op
er
at
ionally proac
tive step to get the switch reg
-
is
tered be
fore it even ar
rives to your data center. You will need to know the se
ri
al num
-
ber of the switch you will re ceive to pre-pro vision. The fol
lowing steps walk you
through switch pre-pro vi
sion
ing for both leaves and spines:
1 On the menu bar, choose Fabric.
2 In the Navigation pane, choose Fabric Membership.
3 In the Work pane, choose Actions > Create Add Fabric Node Member.
4 In the Create Add Fabric Node Member dialog box, perform the following
actions:
a. In the pop-up window, enter the serial number of the switch that will be
arriving.
b. Assign a Node ID and a Switch Name. As a best practice, number leaf nodes
starting with 101, and spine nodes with 201. Lower numbers are reserved for
APICs.
5 Click Submit.
Note: Repeat this process for all switches you wish to pre-provision.
The new entry in the Fab ric Mem bership window will show Un supported in the Role
col
umn until the switch is ac
tu
ally con nected to the fab
ric, but the switch will im
me
di
-
ately be
come a mem ber of the fabric once it ar
rives and is ca
bled.
To be proac tive, you can also pre-pro vi

sion fab
ric poli
cies. Fab
ric poli
cies are covered
in the Fabric Connectiv
ity chap
ter. For more infor
mation on pre-pro vi
sioning poli
cies,
refer to the fol
lowing white paper:
http://
www.
cisco.
com/
c/
en/
us/
solutions/
collateral/
application-centric-infrastructure/
white-paper-c11-731960.
html#_
Toc405844675
Decommission Existing Switch

De
com
mis
sion
ing a switch can ei
ther re
move it from the fab
ric en
tirely, or re
move a
switch tem
porar
ily to per
form main
te
nance. En
sure you do not have any de
vices con
-
nected, as the switch will not for
ward traffic after de
com
mis
sioning. There are two
types of switch de
com
mission
ing: Reg
ul
ar, and Remove from Controller.
Reg
u
lar de
com
mis
sion
ing can be used for main
te
nance and es
sen
tially si
lences the switch
from re
port
ing faults and send
ing SNMP in
for
ma
tion as a tem
po
rary so
lu
tion, while keep
-
ing the switch's node ID and fab
ric mem
ber
ship. The switch will show up under the Dis
-
abled In
ter
faces and De
com
mis
sioned Switches folder in the left nav
i
ga
tion pane.
The Remove from Con troller op

tion will com
pletely remove the switch from the ACI fab ric
and all APICs. The switch will no longer show up in the fab ric mem ber
ship as a reg
is
tered
node and the infra
struc
ture VTEP IP ad dresses it was as
signed will be re
moved.
Per
form the fol
low
ing steps from the APIC GUI to de
com
mis
sion a switch from the ACI
fab
ric:
1 On the menu bar, choose Fabric.
2 In the Navigation pane, choose Inventory > Pod 1.
3 Click the switch to decommission in the Navigation pane.

a. Click the General tab.
b. Chose the Actions > Decommission.
c. In the pop-up, choose either Regular or Remove from Controller.
4 Click Submit.
APICs
Add New APIC
Be fore making any changes to an APIC clus ter, en

sure each APIC in the clus ter is fully
fit and change the cluster size to re
flect the new con troller you are adding to the clus -
ter. Perform the fol
low
ing steps to verify clus
ter health:
1 On the menu bar, choose System Controllers.
2 In the Navigation pane, choose Controllers.

a. Expand the first APIC in the folder.
b. Click the Cluster folder.
c. Verify every controller shows Fully Fit under the Heath State column.
If any of the APICs are not fully fit, refer to the fol
low
ing trou
bleshoot
ing guide:
http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
troubleshooting/
b_
APIC_
Troubleshooting.
html
Per
form the fol
low
ing steps to change the APIC clus
ter size:
1 On the menu bar, choose System > Controllers.
2 In the Navigation pane, choose Controllers > APIC_Name > Cluster.
3 In the Work pane, choose Actions > Change Cluster Size.

a. Change the Target Cluster Administrative Size to reflect the new APIC(s)
being added.
Note: A cluster size of two is not permitted as that does not allow for quorum
amongst APICs.
4 Click Submit.
Per
form the fol
low
ing steps to add a new APIC to the clus
ter:
1 Install and stage the APIC by connecting it to the fabric by following the
hardware installation guide:
http://www. cisco.
com/c/en/
us/support/cloud-systems-management/
application-policy-infrastructure-controller-apic/
products-installation-
guides-list.
html

a. The APIC controllers are added one by one and displayed in the sequential order
starting with N + 1 and continuing until the target cluster size is achieved.
b. Verify that the APIC controllers are in operational state, and the health state of
each controller is Fully Fit from the Cluster folder under the new controller.
Note: It will take sev

eral min
utes for the APICs to syn
chro
nize and join the new APIC to
the clus
ter. Fab
ric op
er
at
ion will con
tinue nor
mally.
Decommission Existing APIC

When de
com
mis
sion
ing APICs, they must be de
com
mis
sioned se
quen
tially in re
verse
order. For ex
am
ple, APIC5 must be de
com
mis
sioned be
fore APIC4. Again, be
fore mak
-
ing any changes to an APIC clus
ter, en
sure each APIC in the clus
ter is fully fit with the
ex
cep
tion of the faulty APIC being de
com
mis
sioned. You can
not de
com
mis
sion a pow
-
ered on fully fit APIC.
Per
form the fol
low
ing steps to de
com
mis
sion an APIC that needs to be re
moved from
the fab
ric:

Note: Select an APIC that is NOT being decommissioned.
3 In the Work pane, choose Actions > Change Cluster Size.

a. Change the Target Cluster Administrative Size to reflect the new APIC(s)
being added.
Note: A cluster size of two is not permitted as that does not allow for quorum

amongst APICs.
4 Click Submit.

Note: In the main pane, click the APIC to be decommissioned.
6 In the Work pane, choose Actions > Actions > Decommission.

a. Verify the APIC no longer appears in the Cluster folder under any of the
remaining APICs.
Hardware Diagnostics and Replacement
The Cisco Ap
pli
ca
tion Cen
tric In
fra
struc
ture (ACI) fabric employs a com
bi
nation of key
soft
ware and hard
ware fea
tures that are specific
ally designed to re
duce the mean time
be
tween fail
ures (MTBF) and the mean time to re
pair (MTTR). Re
gard
ing hard
ware,
there are sev
eral hot-swap
pable components on both the leaf and spine switches in ad
-
di
tion to a few com po
nents that are fixed on the chas
sis. If ACME ever ex pe
ri
ences
some sort of power surge or sees a com
po
nent of their switches go bad, the hot-swap
-
pable com
po
nents en
able them to re
place failed hard
ware quickly and non-dis
rup
-
tively.
Ex
am
ples of hot-swap
pable com
po
nents on both the leaves and spines in
clude:
Power supplies
Fan trays
Ex
am
ples of hot-swap
pable com
po
nents on the spines in
clude:
Power supplies
Fan trays
Supervisors
System Controller cards
Linecard modules
Despite sig
nif
ic
ant ad vances in the above com ponents that re
duce the MTBF, there is
al
ways the pos si
bil
ity of a fail
ure on a leaf switch either in switch hard
ware or software,
or a com bi
na
tion of the two that ne cessi
tates a leaf re
place
ment. In such an event, the
state
less nature of the ACI fab ric provides sig nif
ic
ant ad
vantages to ad
minis
tra
tors
from an op er
at
ions stand point.
Identify Hardware Failure

When a hardware fail
ure oc
curs in the fabric, faults are raised in the sys
tem dash
board
and are pre
sented to the admin is
tra
tor. For cases where there is a com po
nent level
fail
ure with re
dundant com
po
nents pre
sent in the sys
tem, sys
log mes
sages and SNMP
traps are gener
ated.
Ex
am
ples of hard
ware events that gen
er
ate sys
log mes
sages and SNMP traps in
clude:
Linecard failure on a spine switch

Supervisor failure on a spine switch
System controller failure on a spine switch
Power supply or fan failures on a leaf or a spine switch
While Cisco Ap
pli
ca
tion Policy In
fra
struc
ture Controller (APIC) is a cen
tral point of
manage
ment for the entire fab
ric, op
er
at
ions teams can leverage their exist
ing NMS
tools. Log
ging mes
sages can be sent to sys
log servers, such as Splunk, or SNMP mes
-
sages can be sent to NMS sys tems, such as ZenOSS, to pro vide alerting. The leaf and
spine switches in the ACI fabric also sup port tra
di
tional methods of de tect
ing fail
ures,
such as SNMP polling at a set in ter
val. If re
sponses are not received from the switch in
a cer
tain time
frame, there is a possi
bil
ity that the hardware has failed.
However, while the leaf and spine switches re port SNMP and Sys log messages for com -
po
nent level fail
ures, the APICs them selves do not have the abil ity to gener
ate alerts
using SNMP or sys log. For ex
ample a power sup ply fail
ure on the APIC will not generate
an SNMP or sys log mes sage and must be mon it
ored and re mediated using the APIC
dash
board.
Resolve Leaf Hardware Failure

As an ex
am ple of a leaf fail
ure, a Nexus 9396 leaf switch that is a part of the fab
ric is un
-
reachable, per
haps due to a hard ware fail
ure on the uplink mod ules. You can use the
GUI to deter
mine the node health to con firm that the leaf has failed.
To view the node health score:
2 In the Navigation pane, choose Pod 1.

Note: The pod health displays in the Work pane and is zero.
After con
firm
ing that the leaf node has failed, you want to remove the failed switch and
pro
vi
sion a new switch as part of the fab
ric. The first step in re
plac
ing the failed switch
is to get the failed switch's unique ID (node ID). Each node is as signed an ID in the fab-
ric, which is the refer
ence ob ject that al
lows a replace
ment switch with a new se ri
al
num ber to in
herit the same state less con
fig
ur
a
tion that was as
signed to the old node.
To view the fab

ric node IDs using the GUI:
You can also use a sin

gle REST API call to pe
ri
od
ic
ally poll for a full list of nodes that are
at or below a cer
tain health level, as shown in the fol
low
ing ex
am
ple:
{{protocol}}://{{apic}}/api/class/topSystem.xml?rsp-subtree-include=health&rsp-
subtree-filter=le(healthInst.cur,"0")
In the case of a tradi

tional op
er
ations model where each switch was man aged as an in
-
depen
dent entity, the fol
lowing high-level pro
ce
dure re
places the switch:
1 Stand up the replacement switch.
2 Load the correct version of code.
3 Attempt to obtain the latest version of configurations from a configuration

repository server.
4 Stage the device with the right configuration file and eliminate any errors. For
example, update the AAA, NTP, and syslog servers and the ACLs that are
associated with each of them.
5 Copy the old configuration over to the switch.
6 Bring up links one by one and verify if data traffic is flowing correctly.
In an ACI fab
ric, you can take ad
van
tage of the state
less na
ture of the hard
ware to in
-
stan
ti
ate the log
ic
al con
fig
ur
a
tion pro
files. Re plac
ing the node is as sim
ple as de
com
-
mis
sioning the switch and recom mis
sioning it.
To de
com
mis
sion and recom
mis
sion a switch:
2 In the Navigation pane, expand Pod 1.
3 Right click the failed node and choose Decommission.

4 Replace the failed leaf switch with the new leaf switch.
7 The new leaf appears with a node ID of 0 and an IP address of 0.0.0.0.
8 In the Work pane, click on the new leaf.
9 Choose Actions > Commission Switch.
10 When prompted for the node ID, enter the old node's ID. In most cases, you
can also reuse the same leaf name.
11 Click Update.
If the new switch is not op

er
at
ional, the new switch's name and node ID are dif
fer
ent
from the name and ID that you en
tered. You can get the name and ID by view
ing the
unreach
able nodes.
To view the un
reach
able nodes:
2 In the Navigation pane, choose Unreachable Nodes.
3 Find the new switch and record its name and node ID.
4 Repeat the "To decommission and recommission a switch" procedure, starting

with step 5. When prompted for the name and node ID, enter the information
that you recorded in this procedure.
When the new leaf switch is com missioned suc cess

fully, the APIC au
to
mat
ic
ally loads
the cor
rect ver
sion of the firmware into the leaf.
To view which ver

sion of the firmware that the APIC will load:
2 In the Navigation pane, choose Fabric Node Firmware > Firmware Groups > All.
Note: In the Work pane, you can see the target firmware version, which
is automatically set to the latest firmware version.
In ad
di
tion, by lever
ag
ing the state
less object mod eling that re
places the tra
di
tional run
-
ning config
u
ra
tion on a device, APIC automati
cally loads the correct running config
u
ra
-
tion onto the de
vice, such as AAA, sys
log, SNMP, NTP, ACLs, bridge do
mains, and EPGs.
In the event that the re

place
ment switch runs stand
alone NX-OS soft
ware in
stead
of ACI switch soft
ware, you might need to copy the ACI switch soft
ware image to the
switch in ques
tion.
To copy the ACI switch soft

ware image to the switch:
1 Connect to the switch console.
2 Set the IP address on the mgmt0 interface to allow connectivity between the
switch and the APIC.
3 Enable SCP services:
# feature scp-server
4 Copy the firmware image from APIC to the switch:
# scp -r /firmware/fwrepos/fwrepo/switch_image_name
admin@switch_ip_address:switch_image_name
5 For dual supervisor systems, ensure that images are copied to the standby
supervisor in case of a full chassis replacement by using the command:
# copy bootflash:aci_image bootflash://sup-standby/
6 Configure the switch not to boot from Cisco NX-OS.
switch(config)# no boot nxos
7 Save the configuration.
switch(config)# copy running-config startup-config

8 Boot the active and standby supervisor modules with the ACI image.
switch(config)# boot aci bootflash:aci-image-name
9 Verify the integrity of the file by displaying the MD5 checksum.
switch(config)# show file bootflash:aci-image-name md5sum
10 Reload the switch.
switch(config)# reload
11 Log in to the switch as an administrator.
Login: admin
12 Verify whether you must install certificates for your device.
admin@apic1:aci> openssl asn1parse /securedata/ssl/server.crt
13 Look for PRINTABLESTRING in the command output. If "Cisco Manufacturing

CA" is listed, the correct certificates are installed. If something else is listed,
contact TAC to generate and install the correct certificates for your device.
Once you have con firmed the that cer

tifi
cate is in
stalled and the switch is in ACI mode,
the switch should appear as an un
managed fab ric node when con nected to the fabric.
Resolve APIC Hardware Failure

In this ex
am
ple, you must iden
tify and re
me
di
ate a hard
ware fail
ure on one of the
APICs in your APIC clus
ter.
From the GUI of an op

er
at
ional APIC,
1 On the menu bar choose System > Controllers.

2 In the Navigation pane, choose Controllers > apic_name > Cluster.

Note: In the Work pane, you should see the failed APIC in the "Unavailable"
operational state.
3 Record the fabric name, target size, node ID of the failed APIC, and the TEP
address space. This information is also available through the acidiag avread
command on APICs CLI.
4 In the Work pane, click the failed APIC to select it.
5 Choose Actions > Decommission. The APIC changes to an "Out of Service"

admin state.
Decommissioning a Failed APIC
6 Remove the failed APIC from your rack and install the replacement. The new
APIC should boot to the initial setup script.
7 Proceed through the setup script and enter the values of the failed APIC that
you recorded in step 3. Failure to configure the APIC with the same settings
could result in the fabric entering a partially diverged state.
8 Once the new APIC finishes booting, in the Navigation pane, choose Controllers
> apic_name > Cluster. You can choose any APIC.
9 In the Work pane, click the new APIC to select it.

10 Choose Actions > Commission.
Recommissioning an APIC
11 The new APIC will receive an IP address, which will be reflected in the APIC GUI. It
might take 5 to 10 minutes for this to occur. The new APIC might also cycle
between the Available and Unavailable operational states before becoming Fully Fit.
Waiting for Cluster Convergence
12 On the command line of the new APIC, you can verify that it has joined the
fabric by logging in using the credentials that are configured for the rest of the
fabric.
Diagnose Equipment Failures

The ACI fab
ric pro
vides bootup, run
time and on-demand di ag
nos
tics to help as
sess the
hard
ware health of sev
eral sub-sys
tems on each leaf and spine switch.
1 Boot-up tests run when switch, card boots up. These are typically ONLY
disruptive tests. Comes with default set of tests that can be modified. Deployed
via selectors.
2 Health (aka On-going) tests run periodically. Can only run non-disruptive tests.
Comes with default set of tests that can be modified and are deployed via
selectors
3 On-Demand Tests are to be run on specific ports or cards for troubleshooting,

there are no defaults, and they can be disruptive.
By de
fault, tests are log
ic
ally grouped into col
lec
tions.
To look at the default di

ag
nostic poli
cies, click fab
ric > fab
ric poli
cies > Mon
it
or
ing poli
-
cies > de
fault > di
ag
nostics pol
icy
In the work pane se lect the fab

ric el
e
ment that you would like to view the di
ag
nos
tic
mon it
or
ing pol
icy for.
Viewing Diagnostic Monitoring Policies in the GUI

Test re
sults are view
able by click
ing on
Fab
ric > in
ven
tory > Pod-1 > Leaf-xx or Spine-xx > Chas
sis > Su
per
vi
sor mod
ules > Slot-1
AND
Fab
ric > in
ven
tory > Pod-1 > Leaf-xx or Spine-xx > Chas
sis > Line mod
ules > Slot-1
Once there, in the work pane se

lect the Trou
bleshoot
ing tab to view GOLD di
ag
nos
tic
re
sults for the su
per
vi
sor.
Viewing GOLD Diagnostics Information in the GUI
In a mod ul
ar chassis-based system such as the Cisco Nexus 9500 se ries switch, di
ag
-
nostic re
sults are avail
able for all the su
per
vi
sor, mod
ules, sys
tem controller and the
fab
ric mod
ules in the sys
tem.
Create new on-demand diagnostic test

As a part of op
er
at
ional pro
ce
dures, it may be nec
es
sary to val
id
ate a sys
tem is healthy
by run
ning non-dis
rup
tive tests on the sys
tem.
In order to do this, the APIC GUI al

lows cre
ation of an On-de
mand di
ag
nos
tic test.
To do this, nav
ig
ate to Fab
ric > Fab
ric Poli
cies > Trou
bleshoot Poli
cies > On de
mand di
-
ag
nos
tics
Once there, right click on the test or test set you would like to run on de
mand, and
click on Cre
ate.
In the ex
ample below, an op er
ator is cre
at
ing a fab
ric-port on-de
mand diag pol
icy that
al
lows the abil
ity to test the leaf uplink ports going to the spine.
Creating a On-Demand Diag Policy in the GUI
Once you click cre

ate, en
sure you check the box against "in
clude dis
rup
tive tests" if this
is the in
tent.
Enter a name for the test, se lect the admin state, under the di agnostic tests, se
lect a
test con
fig (de
fault is "no tests") and se
lect the fab
ric ports that the test would need to
be run on.
In this case, the op

er
at
or se
lects the op
tions for a non-dis
rup
tive test to be done on
leaf-2, fab
ric-port 1/97.
Creating a Fabric Port On-Demand Diag Policy in the GUI
After ver
if
y
ing the pa
ra
me
ters above, click "SUB
MIT" to sub
mit the pol
icy.
Once sub mit

ted, you can kick off the di
ag
nos
tic test by click
ing on the test it
self and
click
ing sub
mit.
Note that APIC dis plays a warning mes

sage in cases where a non-dis
rup
tive tests are
se
lected to the ef
fect of below.
GUI Warning for Executing a Disruptive Diag Policy
Once you con firm the test run, the di

agnostic re
sults can be ob
tained from the same
lo
ca
tion as the lo
ca
tion for di
ag
nos
tics that are run at bootup or are on
going.
Fab
ric > In
ven
tory > Pod-1 > Leaf-2 > Line-mod
ules > Slot-1 > Fab
ric ports > 1/97
Viewing On-Demand Diag Policy Test Results in the GUI
Note the on-de

mand re
sults in the right hand cor
ner after the test has com
pleted its
run.
413
Appendix
Appendix 415
Classes
The Appli
ca
tion Policy Infrastruc
ture Con troller (APIC) classes are crucial from an op-
er
at
ional perspective to un derstand how sys tem events and faults re late to objects
within the object model. Each event and/or fault in the sys tem is a unique object that
can be accessed for con figura
tion, health, fault, and/or sta
tis
tics.
All the physic

al and log ic
al components that com prise the Ap
pli
ca
tion Cen tric In
fra
-
structure fab
ric are represented in a hi
er
ar
chi
cal man age
ment infor
mation tree (MIT).
Each node in the tree rep re
sents a man
aged ob ject (MO) or group of objects that con-
tains its ad
minis
tra
tive state and its op
er
at
ional state.
The APIC REST API is a pro

grammatic in
ter
face to the APIC that uses a REST ar
chi
tec
-
ture. The API ac
cepts and returns HTTP or HTTPS mes sages that contain JSON or
XML doc um
ents. You can use any pro gramming language to gener
ate the messages,
and the JSON or XML doc
um
ents that con
tain the API meth
ods or man
aged ob
ject (MO)
de
scrip
tions.
You can in
voke an API func
tion by send
ing an HTTP/1.1 or HTTPS POST, GET, or
DELETE mes sage to the APIC. The HTML body of the POST mes sage con
tains a JSON or
XML data struc
ture that describes an MO or an API method. The HTML body of the re -
sponse message contains a JSON or XML structure that con
tains re
quested data, con
-
fir
ma
tion of a re
quested ac
tion, or error in
for
ma
tion.
The fol
lowing section is a rep
re
sen
tation of useful classes for es
tab
lish
ing a foun dation
for monit
or
ing and man agement of the fabric. The list below is a sub
set of the full list of
the avail
able classes.
To access the com plete list of classes, point to the APIC and ref
er
ence
the doc/html di
rec
tory at the end of the URL:
https://apic_ip_address/doc/html/
416 Appendix
Fabric Monitoring
topSystem
Name: top:Sys
tem
Descrip
tion: Pro
vides a list of all the de
vices within the fab
ric, in
clud
ing con
trollers,
leafs and spines.
Usage: The top

Sys
tem class can be used to de rive ob
ject proper
ties in
clud
ing
inb/oob man
age
ment de
tails, cur
rent time, sys
tem up
time and cur rent state.
topSystem REST :: https://172.16.96.2/api/node/class/topSystem.json
fabricNode
Name: fab
ric:Node
De
scription: Provides a list of all the nodes that are part of the fab
ric, in
clud
ing
con
trollers, leafs and spines.
Usage: The fabric

N
ode class can be used to de
rive object prop
erties in
clud
ing
node se
ri
al num
bers, as
signed node ids, node model num bers and device roles.
fabricNode REST :: https://172.16.96.2/api/node/class/fabricNode.json
faultInst
Name: fault:Inst
Descrip
tion: Contains de
tailed in
for
ma
tion of the fault. This ob
ject is at
tached as a
child of the ob
ject on which the fault condi
tion occurred. One instance ob ject is cre
-
ated for each fault con
di
tion of the par
ent ob
ject. A fault in
stance ob
ject is iden
ti
fied by
a fault code.
Usage: The fault

Inst class can be used to de
rive all faults as
so
ci
ated with the
fab
ric, ten
ant or in
di
vid
ual man
aged ob
jects within the APIC.
Appendix 417
faultInst REST :: https://172.16.96.2/api/node/class/faultInst.json
fabricHealthTotal
Name: fab
ric:Health
To
tal
De
scrip
tion: The fab
ric total health score in
stance.
Usage: The fab

ricHealth
To
tal class can be used to de
rive the over
all sys
tem
health.
fabricHealthTotal REST :: https://172.16.96.2/api/node/class/fabricHealthTotal.json
fvCEp
Name: fv:CEp
De
scrip
tion: A client end
point at
tach
ing to the net
work.
Usage: The fvCEp class can be used to de

rive a list of end points at
tached to the
fab
ric and the as
so
ci
ated ip/mac address and encapsu
lation for each object.
fvCEp REST :: https://172.16.96.2/api/node/class/fvCEp.json
fvRsCEpToPathEp
Name: fv:RsCEp
ToPathEp
De
scrip
tion: This is an in
ter
nal ob
ject that pro
vides a re
la
tion to a path end
point.
Usage: The fvRsCEpToPathEp class can be used to de rive path fab

ric de
tails
such as the node and port as well as the ten
ant de
tails such as the ten
ant name, ap pli
-
ca
tion pro
file and end point group.
fvRsCEpToPathEp REST :: https://172.16.96.2/api/node/class/fvRsCEpToPathEp.json
eqptFabP
418 Appendix
eqptFabP
Name: eqpt:FabP
De
scrip
tion: Fab
ric port, the fab
ric fac
ing ex
ter
nal IO port.
Usage: The eqptFabP class can be used to de rive a list of fab

ric port and the as
-
so
ci
ated de
tails such as the line card and chas
sis placement.
eqptFabP REST :: https://172.16.96.2/api/node/class/eqptFabP.json
eqptLeafP
Name: eqpt:LeafP
De
scrip
tion: Fab
ric port, the non-fab
ric fac
ing ex
ter
nal leaf IO port.
Usage: The eqptFabP class can be used to de rive a list of non-fab

ric port and
the as
so
ci
ated de
tails such as the line card and chas
sis placement.
eqptLeafP REST :: https://172.16.96.2/api/node/class/eqptLeafP.json
eqptCh
Name: eqpt:ChA
De
scrip
tion: The hard
ware chas
sis con
tainer.
Usage: The eqptCh class can be used to de rive a chas

sis list and the as
so
ci
ated
de
tails such as the op
er
at
ional state, se
ri
al num
ber and model num ber.
eqptCh REST :: https://172.16.96.2/api/node/class/eqptCh.json
eqptLC
Name: eqpt:LCA
Appendix 419
De
scrip
tion: The line card (IO card), con
tain
ing IO ports.
Usage: The eqptLC class can be used to de

rive a list of line cards de
ployed
within the fab
ric and the as
so
ci
ated de
tails such as the re
dun
dancy state, model, se
ri
al
numbers and the num ber of ports.
eqptLC REST :: https://172.16.96.2/api/node/class/eqptLC.json
eqptFt
Name: eqpt:Ft
De
scrip
tion: The in
ven
to
ried fan tray.
Usage: The eqptFt class can be used to de

rive a list of fan trays and the as
so
ci
ated
de
tails such as the op
er
a
tional sta
tus, model num
ber, ser
ial num ber and hardware version.
eqptFt REST :: https://172.16.96.2/api/node/class/eqptFt.json
eqptPsu
Name: eqpt:Psu
De
scrip
tion: The power sup
ply unit.
Usage: The eqptFt class can be used to de

rive a list of power supplies within the
fab
ric and the as
so
ci
ated de
tails such as the model num ber, se
ri
al num
ber, oper
at
ional
sta
tus, and the volt
age source.
eqptPsu REST :: https://172.16.96.2/api/node/class/eqptPsu.json
eqptSupC
Name: eqpt:SupC
De
scrip
tion: The su
per
vi
sor card, which con
tains the CPU run
ning con
trol plane.
420 Appendix
Usage: The eqptFt class can be used to derive a list of su

per
vi
sor cards de -
ployed within the fab
ric and the as
so
ci
ated de
tails such as the model num ber, se
ri
al
num
ber, op
er
at
ional sta
tus and re
dun
dancy state.
eqptSupC REST :: https://172.16.96.2/api/node/class/eqptSupC.json
ethpmPhysIf
Name: ethpm:PhysIf
De
scrip
tion: The phys
ic
al in
ter
face in
for
ma
tion holder.
Usage: The ethpmPhysIf class can be used to derive a list of phys

ic
al in
terfaces
in the fab
ric and the as
so
ci
ated details such as a the speed, duplex, oper
ational status,
and usage state.
ethpmPhysIf REST :: https://172.16.96.2/api/node/class/ethpmPhysIf.json
dbgAcTrail
Name: dbg:Ac
Trail
De
scrip
tion: The atomic counter trail.
Usage: The dbgAcTrail class can be used to derive a list of the atomic counters
de
ric and the asso
ci
ated de
tails such as dropped packet sta tis
tics
and packet counts.
dbgAcTrail REST :: https://172.16.96.2/api/node/class/dbgAcTrail.json
dbgEpgToEpgRslt
Name: dbg:Epg
ToEp
gRslt
De
scrip
tion: The end
point group to end
point group atomic counter, on-de
mand,
entry.
Appendix 421
Usage: The dbgEpg

ToEp
gR
sIt class can be used to de
rive a list of the EPG to
EPG atomic coun ters deployed within the fab
ric, and the as
so
ci
ated de
tails such as
dropped packet sta
tis
tics and packet counts.
dbgEpgToEpgRsIt REST :: https://172.16.96.2/api/node/class/dbgEpgToEpgRslt.json
dbgEpToEpRslt
Name: dbg:Ep
ToEpRslt
De
scrip
tion: The end
point to end
point atomic counter, On-de
mand, Entry.
Usage: The dbgEp

ToEpT
sIt class can be used to de
rive a list of the end
point to
end
point atomic coun ters de
ric and the as
so
ci
ated de
tails such as
dropped packet sta
tis
tics and packet counts.
dbgEpToEpTsIt REST :: https://172.16.96.2/api/node/class/dbgEpToEpRslt.json
VMM Monitoring
compVm
Name: comp:Vm
De
scrip
tion: The Vir
tual ma
chine ob
ject.
Usage: The com pVm class can be used to de rive a list of vir
tual ma
chines de
-
ric and the as
so
ci
ated de
tails such as the name and state.
compVm REST :: https://172.16.96.2/api/node/class/compVm.json
compHv
Name: comp:Hv
De
scrip
tion: An ob
ject rep
re
sent
ing the com
pute hy
per
vi
sor.
422 Appendix
Usage: The com

pVm class can be used to de
rive a list of com
pute hy
per
vi
sor
de
ric and the as
so
ci
ated de
tails such as the name and sta
tus.
compHv REST :: https://172.16.96.2/api/node/class/compHv.json
fvRsVm
Name: fv:RsVm
Descrip
tion: A re
la
tion to a vir
tual ma
chine con
nected to a hy
per
vi
sor. This is an in
-
ter
nal object.
Usage: The fvRsVm class can be used to de

rive the re
la
tion
ship of the vir
tual
ma
chines con
nected to the hy
per
vi
sor.
vRsVm REST :: https://172.16.96.2/api/node/class/fvRsVm.json
fvRsHyper
Name: fv:RsHy
per
De
scrip
tion: A re
la
tion to the hy
per
vi
sor that con
trols and mon
it
ors the APIC VMs.
This is an in
ter
nal ob
ject.
Usage: The fvR

sHy
per class can be used to de
rive the re
la
tion
ship of the hy
per
-
vi
sor that con
trols and mon
it
ors the APIC VMs.
fvRsHyper REST :: https://172.16.96.2/api/node/class/fvRsHyper.json
vmmCtrlrP
Name: vmm:CtrlrP
De
scrip
tion: The VMM con troller pro
file, which specif
ies how to con
nect to a sin
gle
VM man age
ment con
troller that is part of contain
ing policy enforce
ment domain. For
ex
am
ple, the VMM controller profile could be a policy to con nect a VMware vCen ter
that is part a VMM do
main.
Appendix 423
Usage: The vmm

C
trlrP class can be used to de
rive the ip ad
dress and the dat
a-
cen
ter name of the con
nected VM do
main.
vmmCtrlrP REST :: https://172.16.96.2/api/node/class/vmmCtrlrP.json
Layer 4 to Layer 7 Monitoring
vnsAbsGraph
Name: vns
Ab
s
Graph
Descrip
tion: The abstract graph is made up of abstract nodes and used to de fine
the traf
fic flow through a ser vice func
tion such as load bal
anc
ing, SSL offload, or fire
-
wall. Abstract nodes are com prised of ser
vice nodes such as a ser vice node bal ancer
(SLB) or firewall (FW), ab
stract term nodes (the nodes that are con nected to end point
groups), and con nec
tions.
Usage: The class vns

Ab
s
Graph can be used to de
rive a list of ser
vice graph tem
-
plates con
fig
ured on the APIC, along with their prop
er
ties.
vnsAbsGraph REST :: https://172.16.96.2/api/node/class/vnsAbsGraph.json
vnsLDevVip
Name: vnsLDevVip
Descrip
tion: An L4-L7 de vice clus
ter, which is rep
re
sented by a sin
gle vir
tual IP
(VIP). The con
fig
ur
a
tion is pushed down to the VIP ad dress.
Usage: The class vnsLDevVip can be used to de

rive all the VIPs con
fig
ured for
the log
ic
al de
vice clus
ters in the fab
ric
vnsLDevVip REST :: https://172.16.96.2/api/node/class/vnsLDevVip.json

424 Appendix
vnsCDev
Name: vn
sCDev
De
scrip
tion: The in
di
vid
ual ser
vice de
vice, which is used to de
fine a con
crete l4-l7
ser
vice de
vice.
Usage: The class vnsCDev can be used to de

rive a list of con
crete de
vices con
-
fig
ured as part of the L4-7 ser
vice in
te
gra
tion
vnsCDev REST :: https://172.16.96.2/api/node/class/vnsCDev.json
vnsLif
Name: vnsLif
De
scrip
tion: The log
ic
al in
ter
face, which is as
so
ci
ated with a set of con
crete in
ter
-
faces from the L4-L7 de
vice clus
ter.
Usage: The class vnsLif can be used to de

rive the con
nec
tion be
tween a ser
vice
graph and de
vice in
ter
faces.
vnsLif REST :: https://172.16.96.2/api/node/class/vnsLIf.json
vnsLDevCtx
Name: vnsLDe
vCtx
De scrip
tion: A device clus
ter context, which points to the de vice clus
ter used to
pick a spe cific de
vice based on con tract, sub
ject, and func
tion label or names. To spec-
ify a wild card, set the name to Any.
Usage: The class vnsLDe

vCtx can be used to de
rive the node and con
tract
name.
nsLDevCtx REST :: https://172.16.96.2/api/node/class/vnsLDevCtx.json
vnsRsLDevCtxToLDev
Appendix 425
vnsRsLDevCtxToLDev
Name: vn
sRsLDe
vC
tx
ToLDev
De
scrip
tion: A source re
la
tion to the ab
strac
tion of a ser
vice de
vice clus
ter or of a
proxy ob
ject for a log
ic
al de
vice clus
ter in the ten
ant.
Usage: The class vn

sRsLDe
vC
tx
ToLDev can be used to de
rive the re
la
tion
ship
be
tween vnsLDe
vCtx and vnsLDev.
vnsRsLDevCtxToLDev REST :: https://172.16.96.2/api/node/class/vnsRsLDevCtxToLDev.json
Statistics
compHostStats1h
Name: comp:Host
Stat
s1h
De
scrip
tion: A class that repre
sents the most cur
rent sta
tis
tics for host in a 1 hour
sam
pling in
ter
val. This class up
dates every 15 min
utes.
Usage: The compHostStat

s1h class can be used to de
rive the sta
tis
tics as
so
ci
-
ated with the com
pute hy
per
vi
sor.
compHostStats1h REST :: https://172.16.96.2/api/node/class/compHostStats1h.json
compRcvdErrPkts1h
Name: comp:RcvdEr
rP
kt
s
1h
De
scrip
tion: A class that rep
re
sents the most cur rent statis
tics for re
ceived error
pack
ets in a 1 hour sam
pling in
ter
val. This class up
dates every 15 min utes.
Usage: The com pRcvdErrPkt

s
1h class can be used to de
rive the most cur
rent
sta
tis
tics for re
ceived error pack
ets.
compRcvdErrPkts1h REST :: https://172.16.96.2/api/node/class/compRcvdErrPkts1h.json

426 Appendix
compTrnsmtdErrPkts1h
Name: comp:Trnsmt
dEr
rP
kt
s
1h
De
scrip
tion: A class that rep
re
sents the most cur
rent sta
tis
tics for trans
mit
ted
error pack
ets in a 1 hour sam
pling in
ter
val. This class up
dates every 15 min
utes.
Usage: The compTrnsmt dEr

rP
kt
s
1h class can be used to de
rive the most cur
-
rent sta
tis
tics for trans
mit
ted error pack
ets.
compTrnsmtdErrPkts1h REST ::
https://172.16.96.2/api/node/class/compTrnsmtdErrPkts1h.json
Authentication, Authorization, and Accounting
aaaModLR
Name: aaa:ModLR
De
scrip
tion: The AAA audit log record. A log record is au
to
mat
ic
ally gen
er
ated
whenever a user mod
if
ies an ob
ject.
Usage: The aaaModLR class can be used to de

rive a fab
ric based audit log for all
changes and events.
aaaModLR REST :: https://172.16.96.2/api/node/class/aaaModLR.json
aaaUser
Name: aaa:User
De
scrip
tion: A lo
cally-au
then
ti
cated user ac
count.
Usage: The aaaUser class can be used to de

rive a list of user ac
counts de
ployed
within the fab
ric.
aaaUser REST :: https://172.16.96.2/api/node/class/aaaUser.json

Appendix 427
aaaRemoteUser
Name: aaa:Re
mo
teUser
De
scrip
tion: A re
mote user login ac
count.
Usage: The aaaUser class can be used to de

rive a list of re
mote user ac
counts
de
ric.
aaaRemoteUser REST :: https://172.16.96.2/api/node/class/aaaRemoteUser.json
Fabric Capacity
Policy TCAM
Name: eqpt
ca
pac
it
y
Po
lEn
try5min
Description: Pol
icy CAM entry sta
tis
tics. A class that repre
sents the most current
sta
tis
tics for pol
icy entry in a 5 minute sam pling in
terval. This class up
dates every 10
seconds.
Usage: The eqpt ca

pacit
y
Po
lEn
try5min class can be used to de
rive the cur
rent
value as
so
ci
ated with the Pol
icy TCAM usage.
eqptcapacityPolEntry5min REST ::
http://172.16.96.2/api/class/eqptcapacityPolEntry5min.json
Prefix TCAM
Name: eqpt
ca
pac
ityL3En
try5min
Description: Lay
er3 entry sta
tis
tics. A class that repre
sents the most cur
rent sta
tis
-
tics for lay
er3 entry in a 5 minute sam pling inter
val. This class up
dates every 10 sec-
onds.
Usage: The eqptca

pacityL3En
try5min class can be used to de
rive the cur
rent
value as
so
ci
ated with the Pre
fix TCAM usage.
428 Appendix
eqptcapacityL3Entry5min REST ::
https://172.16.96.2/api/class/eqptcapacityL3Entry5min.json
SNMP/SYSLOG
SNMP Trap Destination
Name: sn
mpTrapDest
De
scrip
tion: A des
ti
na
tion to which traps and in
forms are sent.
Usage: The snmpTrapDest class can be used to de

rive the cur
rent list of snmp
trap des
ti
na
tions im
ple
mented within the fab
ric.
snmpTrapDest REST :: https://172.16.96.2/api/node/class/snmpTrapDest.json
Prefix TCAM
Name: sys
lo
gRe
mot
eDest
Descrip
tion: The sys
log re
mote desti
na
tion host en ables you to spec
ify sys
log
servers to which mes
sages from the APIC and fabric nodes should be for
warded.
Usage: The syslo

gRemot
eDest class can be used to de
rive the cur
rent list of
sys
log re
mote des
ti
na
tions im
ple
mented within the fab
ric.
syslogRemoteDest REST :: https://172.16.96.2/api/node/class/syslogRemoteDest.json
Use Cases
The class fault
Inst used in Use Case #1 and Use Case #2 below can be re
placed with
any of the managed object classses dis
cussed above or spec
if
ied within the APIC doc u-
men tion. The Cisco APIC Com
ta mand-Line Inter
face User Guide may also be helpful for
un
der
stand
ing the fol
low
ing sec
tions - see: http://
www.
cisco.
com/
c/en/us/
td/
docs/
switches/
datacenter/aci/
apic/
sw/1-x/
cli/
b_
APIC_
CLI_
User_Guide.
html
Appendix 429
Case 1: Creating an application script to retrieve the current list of

faults in the fabric.
ic
al for en
vi
ron
min
is
tra
tor wishes to
ob
tain the list of cur
rent faults in the fab
ric. The user has the op
tion of col
lect
ing the
re
sults via CLI, Vi
sore, POST MAN and/or Cobra. Please refer to the sec
tion above for
ap
pli
ca
tion specific ac
cess and ex
plan
tions.
From a CLI per

spec
tive, use the fol
low
ing com
mand to per
form the query:
admin@apic1:~> moquery -c faultInst
From a Vi
sore per
spec
tive, use the fol
low
ing pa
ra
me
ters to per
form the query.:
Class or DN :: faultInst
Property :: n/a
Op :: n/a
Value :: n/a
From a POST
MAN per
spec
tive, user the fol
low
ing REST GET to per
form the query:
GET http://<your apic ip address>/api/node/class/faultInst.xml
From a Cobra per

spec
tive, use the fol
low
ing class query:
# Class Query
classQuery= ClassQuery('faultInst')
for fault in md.query(classQuery):
print fault.name
Sam
ple Cobra script to cap
ture faults within the fab
ric:
from cobra.mit.session import LoginSession
430 Appendix
from cobra.mit.request import ClassQuery

ls = cobra.mit.session.LoginSession('https://'<your apic ip address>, <username>,
<password>, secure=False)
md.login()
# Class Query
print fault.name

faults in the fabric that have been caused by a failed configuration.
This use case may be typ ic
al for envi
ronments where an ACI ad min
is
tra
tor wishes to
cob
tain the list of current faults in the fabric. The user has an op
tion of col
lect
ing the
re
sults via CLI, Vi
sore, POST MAN and/or Cobra. Please refer to the sec tion above for
ap
pli
ca
tion specific ac
cess and ex plantions.
From a CLI per

spec
tive, use the fol
low
ing com
mand to per
form the query:
admin@apic1:~> moquery -c faultInst -f 'fv.faultInst.cause=="config-failure"'
From a Vi
sore per
spec
tive, use the fol
low
ing pa
ra
me
ters to per
form the query:
Class or DN :: faultInst
Property :: cause
Op :: ==
Value :: config-failure
From a POST
MAN per
spec
tive, use the fol
low
ing REST GET to per
form the query:
GET http://<your apic ip address>/api/node/class/faultInst.xml?query-target-

filter=and(eq(faultInst.cause,"config-failure"))
Appendix 431
From a Cobra per

spec
tive, use the fol
low
ing class query:
# Class Query
classQuery.propFilter = 'wcard(faultInst.cause, "{0}")'.format('config-failure')

print fault.name
Cobra Script to cap

ture faults ca
sued by con
fig
ur
a
tion fail
ures.
md.login()
# Class Query
print fault.name
Case 3: Creating an application script to retrieve the properties for a

specific managed object, DN
This use case may be typ ic
al for en
vi
ron
ments where an ACI admin
is
tra
tor wishes to
ob
tain the prop
er
ties of the tenant name Common. The user has an op
tion of col
lect
ing
the re
sults via CLI, Vi
sore, POSTMAN and/or Cobra. Please refer to the sec
tion above
for ap
pli
ca
tion spe
cific ac
cess and ex
plan
tions.
From a CLI per

spec
tive, use the fol
low
ing com
mand to per
form the query:
admin@apic1:~> moquery -d uni/tn-common

432 Appendix
From a Vi
sore per
spec
tive, use the fol
low
ing pa
ra
me
ters to per
form the query:
Class or DN :: uni/tn-common
Property :: n/a
Op :: n/a
Value :: n/a
From a POST
MAN per
spec
tive, use the fol
low
ing REST GET to per
form the query:
GET http://<your apic ip address>/api/node/mo/uni/tn-common.xml?query-target=self
From a Cobra per

spec
tive, use the fol
low
ing class query:
# DN Query
dnQuery= DnQuery('uni/tn-common')
for results in md.query(dnQuery):
print results.dn
Cobra Script to cap

ture faults ca
sued by con
fig
ur
a
tion fail
ures.
from cobra.mit.request import DnQuery
md.login()
# DN Query
dnQuery= DnQuery('uni/tn-common')
for results in md.query(dnQuery):
print results.dn
Appendix 433

endpoints (mac-addresses) attached to the fabric
ic
al for en
vi
ron
min
is
tra
tor wishes to
cre
ate an ap
pli
ca
tion script to cap
ture the list of cur
rent end
points at
tached to the fab
-
ric along with the node de
tails per
tain
ing to each end
point.
Cobra Script to cap

ture faults ca
sued by con
fig
ur
a
tion fail
ures.
from cobra.mit.access import MoDirectory
lls = cobra.mit.session.LoginSession('https://'<your apic ip address>, <username>,
md = MoDirectory(ls)
md.login()
q = ClassQuery('fvCEp')
q.subtree = 'children'
q.subtreeClassFilter = 'fvRsCEpToPathEp'
mos = md.query(q)
for mo in mos:
for child in mo.rscEpToPathEp:
print child.dn
Appendix 435
Package Decoder
There are sev

eral ab
brevi
at
ions used in the names of classes in the ACI ob
ject model.
Here are some descrip
tions of commonly used ab
bre
vi
at
ions, which may help when de-
ci
pher
ing what class ob
jects are when using them with REST calls.
Package Decoder
Aaa: au
then
ti
ca
tion, au
tho
riza
tion, ac
count
ing
ac: atomic coun

ters
actrl: ac
cess con
trol
ac
trl
cap: ac
cess con
trol ca
pa
bil
ity
adcom: ap
pli
ance di
rec
tor com
mu
ni
ca
tion
aib: ad
ja
cency in
for
ma
tion base
arp: ad
dress res
o
lu
tion pro
to
col
bgp: bor
der gate
way pro
to
col
call
home: Cisco smart call home ser
vices
cap: ca
pa
bil
ity
cdp: Cisco dis

cov
ery pro
to
col
cnw: node clus

ter
comm: com
mu
ni
ca
tion pol
icy
comp: com
pute
436 Appendix
com
pat: com
pat
ib
il
ity
con
di
tion: health pol
icy
con
fig: con
fig
ur
a
tion pol
icy
coop: Coun
cil of Or
ac
les pro
to
col
copp: con
trol plane polic
ing pol
icy: con
tains set of rules de
scrib
ing po
licer rates
ctrlr: con
troller
ctx: con
text
date
time: date/time pol
icy
dbg: debug
dbgac: debug atomic coun

ters
dbg
exp: debug ex
port pol
icy
dhcp: dy
namic host con
fig
ur
a
tion pro
to
col
dhcptlv: dy
namic host con
fig
ur
a
tion pro
to
col type length value
dhcptlvpol: dy
namic host con
fig
ur
a
tion pro
to
col type length value pol
icy
dns: do
main name ser
vice
draw: graph vi
su
al
iza
tion for GUI
epm: end
point man
ager
eqpt: equip
ment
eqpt
cap: equip
ment ca
pa
bil
ity
eqpt
ca
pac
ity: equip
ment ca
pac
ity
Appendix 437
eqpt
diag: equip
ment di
ag
nos
tics
eqpt
di
agp: equip
ment di
ag
nos
tics pol
icy
ethpm: eth
er
net pol
icy man
ager
event: event pol

icy
extnw: ex
ter
nal net
work
fab
ric: fab
ric
fault: fault pol

icy, coun
ters
file: file path, con

fig im
port/ex
port pol
icy
firmware: firmware
fm
cast: fab
ric mul
ti
cast
fsm: fi
nite state ma
chine
fv: fab
ric vir
tu
al
iza
tion
fvns: fab
ric vir
tu
al
iza
tion name
space
fv
topo: fab
ric vir
tu
al
iza
tion topol
ogy
geo: ge
olo
ca
tion
glean: glean ad
ja
cency
ha: high avail

abil
ity
health: health score
hvs: hy
per
vi
sors vir
tual switch
icmp: in
ter
net con
trol pro
to
col
438 Appendix
icm
pv4: in
ter
net con
trol pro
to
col ver
sion 4
icm
pv6: in
ter
net con
trol pro
to
col ver
sion 6
ident: iden
tity
igmp: in
ter
net group man
age
ment pro
to
col
igmp
snoop: in
ter
net group man
age
ment pro
to
col snoop
ing
im: in
ter
face man
ager mod
ule
im
gin
stall: image in
stall
infra: in
fra
struc
ture
ip: in
ter
net pro
to
col
ipv4: in
ter
net pro
to
col ver
sion 4
ipv6: in
ter
net pro
to
col ver
sion 6
isis: in
ter
me
di
ate sys
tem to in
ter
me
di
ate sys
tem
isistlv: in
ter
me
di
ate sys
tem to in
ter
me
di
ate sys
tem type length value
l1: layer 1
l1cap: layer 1 ca
pa
bil
ity
l2: layer 2
l2cap: layer 2 ca
pa
bil
ity
l2ext: layer 2 ex
ter
nal
l3: layer 3
l3cap: layer 3 ca
pa
bil
ity
Appendix 439
l3ext: layer 3 ex
ter
nal
l3vm: Layer 3 Vir

tual Ma
chine
lacp: link ag
gre
ga
tion pro
to
col
lbp: load bal

anc
ing pol
icy
leqpt: loose equip

ment (un
man
aged nodes, not in the fab
ric)
lldp: link layer dis

cov
ery pro
to
col
lldptlv: link layer dis

cov
ery pro
to
col type length value
lldptlvpol: link layer dis

cov
ery pro
to
col type length value pol
icy
maint: main
te
nance
mcast: mul
ti
cast
mcp: mas
ter con
trol proces
sor
mem
ory: mem
ory sta
tis
tics
mgmt: man
age
ment
mo: man
aged ob
ject
mock: mock (ob

jects used on the sim
ul
a
tor mostly for show
ing stats/faults/etc)
mon: mon
it
or
ing
mon
i
tor: mon
it
or (SPAN)
nam
ing: ab
stract for ob
jects with names
nd: neigh
bor dis
cov
ery
nw: net
work
440 Appendix
oam: eth
er
net op
er
at
ions, ad
min
is
tra
tions and man
age
ment
ob
server: ob
server for sta
tis
tics, fault, state, health, logs/his
tory
opflex: OpFlex
os: op
er
at
ing sys
tem
ospf: open short

est path first
pc: port chan

nel
pcons: **gen
er
ated and used by in
ter
nal processes**
phys: phys
ic
al do
main pro
file
ping: ping ex
e
cu
tion and re
sults
pki: pub
lic key in
fra
struc
ture
pol: pol
icy de
f
in
i
t
ion
po
licer: traf
fic polic
ing (rate lim
it
ing)
pool: ob
ject pool
pres: **gen
er
ated and used by in
ter
nal processes**
proc: sys
tem load, cpu, and mem
ory uti
liza
tion sta
tis
tics
psu: power sup

ply unit pol
icy
qos: qual
ity of ser
vice pol
icy
qosm: qos sta

tis
tics
qosp: qos/ 802.1p
rbqm: de
bug
ging
Appendix 441
regress: re
gres
sion
reln: **gen
er
ated and used by in
ter
nal processes**
repl: **gen
er
ated and used by in
ter
nal processes**
res: **gen
er
ated and used by in
ter
nal processes**
rib: rout
ing in
for
ma
tion base
rmon: re
mote net
work mon
it
or
ing/ in
ter
face stats/coun
ters
rpm: route pol

icy map
rtcom: route con

trol com
mu
nity list
rtc
trl: route con
trol
rtextcom: router ex
tended com
mu
nity
rtflt: route fil

ter
rtleak: route leak
rtmap: RPM route map
rtpfx: route pre

fix list
rtreg
com: route reg
ul
ar com
mu
nity list
rtsum: route sum

ma
riza
tion ad
dress/pol
icy
satm: satel
lite man
ager
snmp: sim
ple net
work man
age
ment pro
to
col
span: switched port an

al
yzer
stats: sta
tis
tics col
lec
tion poli
cies
442 Appendix
stat
store: sta
tis
tics data hold
ers
storm
c
trl: storm con
trol (traf
fic sup
pres
sion) pol
icy
stp: span
ning tree pro
to
col de
f
in
i
t
ions and pol
icy
sts: Ser
vice Tag Switch
ing (used for ser
vices in
ser
tion)
svc
core: core pol
icy
svi: switched vir

tual in
ter
face/ routed VLAN in
ter
face
syn
thetic: syn
thetic ob
jects (for test
ing)
sys
de
bug: sys
tem debug
sys
file: sys
tem files
syshist: sys
tem cards reset records/his
tory
sys
log: sys
log pol
icy
sys
mgr: sys
tem man
ager (firmware, su
per
vi
sor, sys
tem states, etc)
sys
m
grp: con
tainer for cores pol
icy & ab
stract class for all qos pol
icy de
f
in
i
t
ions
tag: alias (use de

scrip
tive name for dn), tags (group mul
ti
ple ob
jects by a de
scrip
tive
name)
task: task ex
e
cu
tion, in
stance, and re
sult
test: ab
stract class for test rule, sub
ject, and re
sult
testin
fralab: test in
fra
struc
ture
tlv: type, length, value sys

tem struc
tures
top: sys
tem task man
ager for proces
sor ac
tiv
ity
Appendix 443
topoc
trl: topol
ogy con
trol pol
icy (shard
ing, fab
ric LB, fab
ric VxLan, etc)
tracer
oute: tracer
oute ex
e
cu
tion and re
sults
tracer
outep: tracer
oute end points
trig: trig
ger
ing pol
icy
tun
nel: tun
nel
ing
uribv4: ipv4 uni

cast rout
ing in
for
ma
tion base en
tity
vlan: vlan in
stances
vlan
mgr: vlan man
ager con
trol plane
vmm: vir
tual macine man
ager (con
troller, vmm pol
icy and de
f
in
i
t
ions)
vns: vir
tual net
work ser
vice (L4-L7 pol
icy and de
f
in
i
t
ions)
vpc: vir
tual port chan
nel (vpc pol
icy and de
f
in
i
t
ions)
vsvc: ser
vice la
bels (provider/con
sumer)
vtap: trans
lated ad
dress of ex
ter
nal node (NATed IP of ser
vice node)
vxlan: Vir
tu
ally ex
ten
si
ble LAN de
f
in
i
t
ions
vz: vir
tual zones (for
mer name of the pol
icy con
trols) i.e. Con
tracts
Model Naming schemes

Rs: Re
la
tionship source
Rt: Re
la
tion
ship tar
get
Ag: Ag
gregated stats
BrCP: Bi nary Contract Pro
file
Appendix 445
Acronyms and Definitions
Overview
This sec
tion is de
signed to pro
vide a high level de
scrip
tion of terms and con
cepts that
get brought up in this book. While ACI does not change how pack
ets are trans
mit
ted on
a wire, there are some new terms and con
cepts em
ployed, and un
der
stand
ing those
new terms and concepts will help those work
ing on ACI com muni
cate with one an
other
about the con
structs used in ACI to trans
mit those bits. As
so
ci
ated new acronyms are
also pro
vided.
This is not meant to be an exhaustive list nor a com

pletely de
tailed dic
tio
nary of all of
the terms and con cepts, only the key ones that may not be a part of the com mon ver -
nacul
ar, or which would be rel
e
vant to the troubleshoot
ing ex
er
cises that were covered
in the troubleshoot
ing sce
nar
ios dis
cussed.
A
AAA: acronym for Au
then
ti
ca
tion, Au
tho
riza
tion, and Ac
count
ing.
ACI Exter
nal Con
nectiv
ity: Any con
nectiv
ity to and from the fabric that uses an ex
ter
-
nal routed or switched inter
medi
ary sys
tem, where end points fall out
side of the man-
aged scope of the fab
ric.
ACID trans actions: ACID is an acronym for Atom ic

ity, Con
sis
tency, Isola
tion, Dura
bil
ity
properties of transac
tions that en
sure con
sis
tency in database trans actions. Trans
ac-
tions to APIC de vices in an ACI clus
ter are considered ACID, to en sure that data base
consis
tency is main tained. This means that if one part of a transaction fails the entire
transac
tion fails.
AEP: Attach
able Entity Pro
file this is a config
ur
a
tion pro
file of the in
ter
face that gets
ap
plied when an en tity at
taches to the fab ric. An AEP rep re
sents a group of ex ter
nal
en
ti
ties with sim
il
ar in
fra
structure pol icy re
quirements. AEPs are also the mech an
ism
that ties the phys
ic
al port to the do
main (phys
ic
al or vir
tual) to a switch pol
icy.
ALE: Ap
pli
ca
tion Leaf En
gine, an ASIC on a leaf switch.
446 Appendix
APIC: Ap
pli
ca
tion Pol
icy In
fra
struc
ture Con
troller is a cen
tral
ized pol
icy man
age
ment
con
troller clus
ter. The APIC con
fig
ures the in
tended state of the pol
icy to the fab
ric.
API: Ap
pli
ca
tion Pro
gram
ming In
ter
face used for pro
gram
ma
ble ex
ten
si
bil
ity.
Ap
pli
ca
tion Pro
file: Term used to ref
er
ence an ap
pli
ca
tion pro
file-man
aged ob
ject ref
-
er
ence that mod
els the log
ic
al com
po
nents of an ap
pli
ca
tion and how those com
po
-
nents com muni
cate. The AP is the key object used to rep
re
sent an appli
ca
tion and is
also the an
chor point for the au
to
mated in
frastruc
ture man
agement in an ACI fabric.
ASE: Ap
pli
ca
tion Spine En
gine, an ASIC on a Spine switch.
B
BGP: Bor der Gate way Proto
col, on the ACI fabric Multi-Proto
col BGP is used to dis
trib
-
ute reachab
il
ity in
for
mation within the fab
ric, and In
ternal BGP is used to peer the fab-
ric with ex
ternal Layer 3 devices.
Bridge Do main: An ACI con struct that de fines Layer 2 for

warding behaviors (Broadcast,
ARP flooding, etc.) for each unique Layer 2 for warding domain. Bridge Do mains are also a
con
tainer for IP sub nets and are where fab ric Layer 3 gateway func tion
al
ity is con
fig
-
ured. BDs can em ulate the be
havior of a tra
di
tional VLAN but are not con strained by for-
warding scale limi
tations. In the ACI ob ject model, a BD is a child of a Pri
vate Layer 3 or
con
text.
C
CLOS fab
ric: A multi-tier non
block
ing leaf-spine ar
chi
tec
ture net
work.
Cluster: Set of de
vices that work to
gether as a sin
gle sys
tem to pro
vide an iden
ti
cal or
sim
il
ar set of func
tions.
Contracts: A log
ic
al container for the sub jects which relate to the fil
ters that gov
ern
the rules for communica
tion between end point groups. ACI works on a white list policy
model. With out a contract, the de
fault for
ward ing pol
icy is to not allow any com mu
ni
-
ca
tion between EPGs, but com muni
ca
tion within an EPG is al lowed.
Appendix 447
Con
text: A Layer 3 for
ward
ing do
main, equiv
al
ent to a VRF, and in ACI ver
nac
ul
ar a Pri
-
vate Layer 3.
D
DLB: Dy
namic Load Bal
anc
ing a net
work traf
fic load-bal
anc
ing mech
an
ism in the ACI
fab
ric based on flowlet switch
ing.
DME: Data Man

age
ment En
gine, a ser
vice that runs on the APIC that man
ages data for
the data model.
dMIT: dis
trib
uted Man age
ment In for
ma
tion Tree, a repre
senta
tion of the ACI object
model with the root of the tree at the top and the leaves of the tree at the bot
tom. The
tree con
tains all as
pects of the ob
ject model that rep
re
sent an ACI fab
ric.
Dn: Dis tin

guished name a fully qualif
ied name that rep resents a specific object within
the ACI man age
ment infor
mation tree as well as the spe cific lo
ca
tion in
forma tion in the
tree. It is made up of a concate
nation of all of the rel
at
ive names from it self back to the
root of the tree. As an ex ample, if pol
icy object of type Ap pli
ca
tion Profile is created,
named com merce work space within a Ten ant named Prod, the dn would be ex pressed
as uni/tn-Prod/ap-com mercework space.
E
EP: Endpoint - Any logic
al or phys ic
al de
vice connected directly or in
di
rectly to a port
on a leaf switch that is not a fabric facing port. Endpoints have specific proper
ties like
an address, lo
ca
tion, or poten
tially some other at tribute, which is used to iden tify the
endpoint. Exam
ples in
clude virtual-ma chines, servers, stor
age de
vices, etc.
EPG: End Point Group. A col

lec
tion of end
points that can be grouped based on com-
mon re
quire
ments for a com
mon policy. End
point groups can be dy
namic or sta
tic.
F
Fault: When a failure oc curs or an alarm is raised, the sys tem creates a fault-managed
object for the fault. A fault contains the conditions, in
for
mation about the op er
at
ional
state of the af
fected ob ject and poten
tial res
ol
utions for the prob
lem.
448 Appendix
Fab
ric: The col
lec
tive end
points as
so
ci
ated with an ACI so
lu
tion (Leaf, Spine and Vir
tual
Switches plus APICs)
FCAPS: The ISO model de fines network management tasks. FCAPS is n acronym for
fault, con
fig
ur
a
tion, ac
count
ing, per
for
mance, se
cu
rity, the man
age
ment cat
e
gories
Fil
ters: Fil
ters de
fine the rules out
lin
ing the Layer 2 to layer 4 fields that will be
matched by a con
tract.
Flowlet switch
ing: An op
ti
mized, mul
ti
path, load-bal
anc
ing method
ol
ogy based on re
-
search from MIT in 2004. Flowlet Switch
ing is a way to use TCPs own bursty na
ture to
more ef fi
ciently for
ward TCP flows by dy namic
ally split
ting flows into flowlets, and
split
ting traf
fic across mul
ti
ple par
al
lel paths without re
quiring packet re
order
ing.
G
GUI: Graph
ic
al User In
ter
face.
H
HTML: Hy per
Text Markup Lan
guage, a markup lan
guage that fo
cuses on the for
mat
-
ting of web pages.
Hy
pervi
sor: Soft
ware that abstracts the hardware on a host ma
chine and al
lows the
host machine to run mul
ti
ple vir
tual ma
chines.
Hyper
vi
sor in
te
gration: Ex
ten
sion of ACI Fab
ric connectiv
ity to a virtual machine man-
ager to pro
vide the APIC with a mech an
ism for vir
tual machine vis ib
il
ity and pol
icy en
-
force
ment.
I
IFM: In
tra-Fab
ric Mes
sages, Used for com
mu
ni
ca
tion be
tween dif
fer
ent de
vices on the
ACI fab
ric.
Appendix 449
In
band Man
age
ment (INB): In
band Man
age
ment. Con
nec
tiv
ity using an in
band man
-
age
ment con
fig
ur
a
tion. This uses a front panel (data plane) port of a leaf switch for ex
-
ter
nal man
age
ment con
nec
tiv
ity for the fab
ric and APICs.
IS-IS: Link local rout

ing pro
to
col lever
aged by the fab
ric for in
fra
struc
ture topol
ogy.
Loop
back and VTEP ad
dresses are in
ter
nally ad
ver
tised over IS-IS. IS-IS an
nounces the
cre
ation of tun
nels from leaf nodes to all other nodes in fab
ric.
J
JSON: JavaScript Ob
ject No
ta
tion, a data en
cap
su
la
tion for
mat that uses human read
-
able text to en
cap
su
late data ob
jects in at
tribute and value pairs.
L
Layer 2 Out (l2out): Layer 2 con
nec
tiv
ity to an ex
ter
nal net
work that ex
ists out
side of
the ACI fab
ric.
Layer 3 Out (l3out): Layer 3 con

nec
tiv
ity to an ex
ter
nal net
work that ex
ists out
side of
the ACI fab
ric.
L4-L7 Ser vice In ser

tion: The insertion and stitching of VLANs/Layer 3 con structs of
vir
tual or phys ic
al service ap pli
ances (Firewall, IDS/IPS, Load Bal ancers, DLP,
etc....) into the flow of traffic. Ser
vice nodes op er
ate between Lay ers 4 and Layer 7 of
the OSI model, where as net working el
e
ments (i.e. the fab
ric) op
er
ate at lay
ers 1-3).
La
bels: Used for clas
si
fy
ing which ob
jects can and can
not com
mu
ni
cate with each
other.
Leaf: Network node in fab ric pro

vid
ing host and border con
nec
tiv
ity. Leafs con
nect
only to hosts and spines. Leafs never con
nect to each other.
M
MO: Man aged Object every con
fig
urable com
po
nent of the ACI pol
icy model man
-
aged in the MIT is called a MO.
450 Appendix
Model: A model is a con

cept which rep
re
sents en
ti
ties and the re
la
tion
ships that exist
be
tween them.
Multi-tier Ap
pli
ca
tion: Clientserver ar
chi
tec
ture in which pre
sen
ta
tion, ap
pli
ca
tion
logic, and data
base man
age
ment func
tions re
quire phys
ic
al or log
ic
al sep
ar
a
tion and
re
quire net
work
ing func
tions to com
mu
ni
cate with the other tiers for ap
pli
ca
tion func
-
tion
al
ity.
O
Ob
ject Model: A col
lec
tion of ob
jects and classes are used to ex
am
ine and ma
nip
ul
ate
the con
fig
ur
a
tion and running state of the sys
tem that is ex
pos
ing that ob
ject model. In
ACI the ob
ject model is rep
re
sented as a tree known as the distrib
uted man age
ment in-
for
ma
tion tree (dMIT).
Out-of-Band manage
ment (OOB man age
ment): Exter
nal connec
tiv
ity using a spe
cific
out-of-band man
age
ment in
ter
face on every switch and APIC.
P
Port Chan nel: A Port link aggre
ga
tion tech
nol
ogy that binds mul
ti
ple phys
ic
al in
ter
-
faces into a sin
gle log
ic
al in
ter
face and pro
vides more aggre
gate bandwidth and link
fail
ure re
cov
ery with
out a topology change.
R
RBAC: Role Based Ac cess Control, which is a method of man ag
ing se
cure ac
cess to in
-
fra
struc
ture by as
sign
ing roles to users, then using those roles in the process of grant -
ing or deny
ing ac
cess to de
vices, objects and priv
il
ege lev
els.
Rep re
senta
tional State Trans fer (REST): a stateless proto
col usually run over HTTP
that al
lows a client to ac
cess a server-side or cloud-based API with out having to write a
local client for the host ac
cess ing the API. The lo
cation that the client ac
cesses usually
defines the data the client is trying to ac
cess from the service. Data is usu
ally ac
cessed
and returned in either XML or JSON for mat.
REST
ful: An API that uses REST, or Rep
re
sen
ta
tional State Trans
fer.
Appendix 451
Rn: Rel
at
ive name, a name of a spe
cific ob
ject within the ACI man
age
ment in
for
ma
tion
tree that is not fully qual
if
ied. A Rn is sig
nif
ic
ant to the in
di
vid
ual ob
ject, but with
out
con
text, its not very use
ful in nav
ig
a
tion. A Rn would need to be con
cate
nated with all
the rel
ative names from it
self back up to the root to make a distin
guished name, which
then be comes useful for navig
a
tion. As an ex
ample, if an Ap
pli
ca
tion Pro
file ob
ject is
cre
ated named com
merce
work
space, the Rn would be ap-com
merce
work
space be
-
cause Ap
pli
ca
tion Pro
file rel
at
ive names are all pref
aced with the let
ters ap-. See also
the Dn de
f
in
i
t
ion.
S
Ser
vice graph: A mech
an
ism within ACI that au
to
mates redi
rec
tion of traf
fic and VLAN
stitching based on de fined para
me ters. Any services that are re
quired are treated as a
service graph that is instan
ti
ated on the ACI fab ric from the APIC. Ser
vice graphs iden-
tify the set of net
work or ser vice functions that are needed by the appli
ca
tion, and rep
-
resent each function as a node.
Spine: Net work node in fab ric car

ry
ing ag
gregate host traf
fic from leafs, con
nected
only to leafs in the fab
ric and no other de
vice types.
Spine/Leaf topol ogy: A clos-based fabric topol

ogy in which spine nodes con
nect to
leaf nodes, leaf nodes con
nect to hosts and exter
nal net
works.
Sub
nets: Con tained by a bridge domain or an EPG, a sub
net de
fines the IP ad
dress
range that can be used within the bridge do
main.
Subjects: Con
tained by con
tracts and cre
ate the re
la
tion
ship be
tween fil
ters and con
-
tracts.
Su
per
vi
sor: Switch mod
ule that pro
vides the con
trol plane for the 95xx switches.
T
Ten
ants: The log
ic
al con
tainer to group all poli
cies for ap
pli
ca
tion poli
cies.
452 Appendix
V
Vir
tu
al
iza
tion: Tech
nol
ogy used to ab
stract hard
ware re
sources into vir
tual rep
re
sen
-
ta
tions and al
low
ing soft
ware con
fig
ura
bil
ity.
vPC: vir
tual Port Chan
nel, in which a port chan
nel is cre
ated for link ag
gre
ga
tion, but is
spread across no more or less than 2 phys
ic
al switches.
VRF: Vir
tual Rout
ing and For
ward
ing - A Layer 3 name
space iso
la
tion method
ol
ogy to
allow for mul
ti
ple con
texts to be de
ployed on a sin
gle de
vice or in
fra
struc
ture.
VXLAN: VXLAN is a Layer 2 over

lay scheme trans
ported across a Layer 3 net
work. A 24-
bit VXLAN seg
ment ID (SID) or VXLAN net
work iden
ti
fier (VNID) is in
cluded in the en
-
capsulation to pro
vide up to 16 million VXLAN seg ments for traf
fic iso
la
tion or seg
men-
ta
tion. Each seg ment rep resents a unique Layer 2 broad cast do main. An ACI VXLAN
header is used to iden tify the policy at
trib
utes if the ap
pli
ca
tion end point within the
fabric, and every packet car ries these pol
icy at
trib
utes.
X
XML: eXten
si
ble Markup Language, a markup lan guage that focuses on en
cod
ing data
for doc
um
ents rather than the for
mat
ting of the data for those doc
um
ents.
Appendix 453
Reference Material
Top
ics that are outside of the scope of this op er
at
ions guide may be docum
ented in
other places. This section in
cludes links to other help
ful ref
er
ence doc
um
en
ta
tion for
fur
ther read
ing and view
ing.
ACI Install and Upgrade Guides

http://
www.
cisco.
com/
c/
en/
us/
support/
application-
policy-infrastructure-controller-apic/
products-installation-guides-list.
html
ACI Get
ting Started - Fab
ric Ini
tial
iza
tion
http://www. cisco.
com/
c/en/us/support/
cloud-systems-management/ application-
products-installation-and-configuration-
guides-list.
html
ACI De
sign Guide
http://www.cisco.
com/ c/en/
us/solutions/
collateral/
application-centric-infrastructure/white-paper-c11-731960.html#_Toc405844675
ACI Trou
bleshoot
ing Guides
http://www. cisco.
com/
c/en/us/support/
products-troubleshooting-guides-list.
html
https://
datacenter.
github.
io/
ACI White Pa
pers
http://www. cisco.
com/
c/
en/us/
solutions/ data-center-virtualization/
application-
centric-infrastructure/
white-paper-listing.
html
454 Appendix
ACI Case Stud

ies
www. cisco.
com/
c/en/
us/
solutions/
infrastructure/
customer-case-study-listing.
html
ACI Demos, Pre

sen
ta
tions and Train
ings
www. cisco.
com/c/
en/us/
solutions/ data-center-virtualization/
infrastructure/
sales-resources-list.
html
ACI Ecosys
tem Com
pata
bil
ity List
http://
www.
cisco.
com/
c/
en/
us/
solutions/
collateral/
application-centric-infrastructure/
solution-overview-c22-732445.
html
ACI Part
ners and Cus
tomers Pre
sen
ta
tions
http://www. cisco.
com/
c/
en/us/solutions/data-center-virtualization/
application-
presentations-listings.
html
ACI So
lu
tions Overview
http://www. cisco.
com/
c/en/us/
solutions/data-center-virtualization/
application-
solution-overview-listing.
html
ACI Toolkit
http://
datacenter.
github.
io/
acitoolkit/
ACI Com
pata
bil
ity Tool
http://
www.
cisco.
com/
web/
techdoc/
aci/
acimatrix/
matrix.
html
AVS Con
fig
u
ra
tion and Scal
a
bil
ity Guides
http://
www. cisco.
com/c/
en/us/support/switches/ application-virtual-switch/
products-installation-and-configuration-guides-list.
html
Appendix 455
AVS Topolo
gies and So
lu
tion Guide
http://
www.cisco.
com/ c/
en/us/ support/
switches/
application-virtual-switch/
products-technical-reference-list.
html
APIC Com
mand-Line In
ter
face User Guide
http://
www.
cisco.
com/
c/
en/
us/
support/
application-
products-command-reference-list.
html
APIC Layer 4 to Layer 7 Ser

vices De
ploy
ment Guide
http://
www.
cisco.
com/
c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
L4-L7_
Services_
Deployment/
guide/
b_
L4L7_
Deploy.
html
Cobra Docs
http://
cobra.
readthedocs.
org/
en/
latest/
Cobra GitHub
http://
github.
com/
datacenter/
cobra
Con
nect
ing ACI to Out
side Layer 2 and 3 Net
works
http://www.cisco.
com/ c/en/
us/solutions/
collateral/
application-centric-infrastructure/white-paper-c07-732033. html
Fab
ric Con
nec
tiv
ity Video
https://
www.
youtube.
com/
watch?
v=_
iQvoC9zQ_
A
Nexus CLI to Cisco APIC Map

ping
http://www. cisco.
com/
c/en/us/support/
products-configuration-examples-list.
html
456 Appendix
POST
man
getpostman.
http://www. com
Sup
ported SNMP MIB List
http://www.cisco.
com/c/
en/
us/
td/
docs/
switches/
datacenter/
aci/
apic/
sw/
1-x/
mib/list/
mib-support.
html

Cisco OperatingApplicationCentricInfrastructure

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Cisco OperatingApplicationCentricInfrastructure

Uploaded by

Copyright:

Available Formats

Operating Cisco

Alejandra Sanchez, Andres Vega, Arvind Chari, Carly Stoughton,

Cisco Press logo is a trade

Warning and Disclaimer

This book repre

Book Sprint Facilitation

Book Writing Methodology

The Book Sprint (booksprints.

Hardware and Software Included in the Book

The Cisco Appli

For the pur

Cisco Application Policy Infrastructure Controller (APIC)

This book was writ

The Story of ACME Inc.

ACME Inc. is a multi na

One of the largest chal lenges ACME has his tor

While ACME Inc. is a fic ti

The Why, Who, What, When and How

Within ACI, ACME is look ing to simplify how it oper

ACME's Stor age Team is pri mar ily fo

In ACI there are some el ements of the tra di

tion of many dif

How does a networking person go about configuring the network forwarding?

When "how" in volves mak ing a change to the con fig

As a baseline, most orga

Emergency changes are by de f

Normal changes are those that go through the reg ul

The items mentioned above are not in

A more in-depth dis

ACI addresses IT requirements from across the organization

Configuring Management Protocols

Role-Based Access Control

Import and Export Policies

There are a number of fun

Single point of management controller-based architecture

The sin gle point of man age

This con troller-based ar chi

The de sired state model for con

Configuring Management Protocols

Cisco Discovery Protocol

1 On the menu bar, choose Fabric > Access Policies.

Your CDP pol

Link Layer Discovery Protocol

1 On the menu bar, choose Fabric > Access Policies.

a. In the Name field, enter "LLDP-TX-ON-RX-ON".

Your LLDP pol

Time Synchronization and NTP

Out-of-Band Management NTP

1 On the menu bar, choose Fabric > Fabric Policies.

2 In the Navigation pane, choose Pod Policies > Policies.

Your NTP pol

In-Band Management NTP

Once you have es tablished a pol

Verifying NTP Operation

1 On the menu bar, choose Fabric > Fabric Policies.

3 In the Work pane, verify the details of the server.

Verifying that the NTP Policy Deployed to Each Node

admin@apic1:~> attach <Tab> <Tab>

admin@apic1:~> attach node_name

4 View the NTP peer status:

leaf-1# show ntp peer-status

A reach able NTP server has its IP ad

5 Repeat steps 3 and 4 for each node on the fabric.

Domain Name Services (DNS)