Cisco Application Centric

Infrastructure
Agenda

• Introduction
• Overview and Features
• Stateless Hardware
• Overlays
• Forwarding

• Use Cases
industry trends

• Cloud services
- be it Amazon Web services, Microsoft Azure cloud, digital ocean
• big date
- Adobe, MapR (Map and Reduce), Mongo database.
• Automation tools.
- Ansible, chef, puppet, GitHub, Jenkins and continuous integration.
• SDN
Industry trends

• Cloud services
- be it Amazon Web services, Microsoft Azure cloud, digital ocean
• big date
- Adobe, MapR (Map and Reduce), Mongo database.
• Automation tools.
- Ansible, chef, puppet, GitHub, Jenkins and continuous integration.
• SDN
Software Defined Networking

• peoples describe this in different ways

• control plane and data plane are now being controlled by some sort of
centralized controller
- OpenFlow
• network virtualization functions
- Nuage, PLUM, Midokura
• pure programmability
- Arista, Cumulus
Software Overlay
• run a layer 3 routed, non-blocking ECMP fabric or “CLOS fabric” as our underlying infrastructure
• on top of that we place multiple “virtual networks”
Application Centric Infrastructure
• Is a network fabric for datacenters.
• Leaf/Spine Topology
• Uses VXLAN and Tunnel Endpoints as
an underlay
• All configuration is programed, provisioned
and initially controlled from the controller and
pushed to the network switches
• Control plane and data place are
separate
• APICs form a cluster for distributed
Computing
What is ACI?
• Behaves like a Switch (Bridge Domain)
• Behaves like a Router (Unicast Routing)
• Utilizes VRFs (VRF)
• Utilizes VLANs (EPGs and SVIs)
• Utilizes VXLANs (Overlay)
• Behaves like an Orchestrator
• Configures Hypervisors/Controllers
• Configures L4-L7 devices

• Open North and South-bound API

• Automation
 What is ACI?

Application Network Profile

Tenant Policies Legacy

F/W
EPG WEB
L/B
EPGAPP EPG DB
Network

Fabric Policies

Access Policies
What is ACI?
Application Network Profile

• everything is based on the application, but we need to map that to network

constructs
• ANP introduce stateless definition of the application requirement
- Application Tiers
- Connectivity Policies
- Layer 4-7 Services
• network profile is fully abstracted from the infrastructure
- removes all dependencies of the infrastructure
- portable across different data center fabrics
 Applications Policy Model and instantiation

• application policy model and requirements defined based on the “network profile”
• Then, based on the Deployment model, APIC pushes and provision this down to
the fabric infrastructure
• all forwarding in the fabric is managed through this “application network profile”
- IP addresses are fully portable anywhere within the fabric
- Security and forwarding are fully decoupled from any physical or virtual network
attributes
- devices autonomously update the state of the network based on the configuration
policy requirements
ACI Fabric
What are we solving?
Overloaded Network Constructs
Basic Network
SLAs L4-7 Services
Policy

Subnet Subnet Subnet

VLAN VLAN VLAN

Network constructs are overloaded with unintended functionality.

Application Language Barriers
Developers Infrastructure Teams

Application VLANs
Tiers
Subnets

Provider / Protocols
Consumer
Ports
Relationships

Developer and infrastructure teams must translate between disparate languages.

What is an application to the network?

• collection of all the applications end-points

• layer 2 through L7 network policies
• The Relation between these Endpoint and
Their Policies
o so the idea of what we want to get to, is to
build teamwork to create a logical,
abstracted, stateless model that supports
the application
Applying policy to endpoints

1) endpoint attaches to the fabric.

2) the APIC detects endpoint and learns its source EPG.
3) APIC pushes the required policy down to the leaf switch
Policies
• Can be subdivided into two main categories:
• Access Policies = Define how a switch or switchport is configured. Specifically Ethernet
and link layer properties such as LLDP, LACP, CDP, speed/duplex, etc.
• Tenant Policies = Govern traditional networking. This is where Application connectivity is
defined.
• Both work in tandem to define where and how endpoints or applications are
connected

Application Network Profile

EPG DB
F/W L/B
Legacy EPG EPGA
Tenant Policies Network
WEB PP

Access Policies
 Access Policies

Application Network Profile

Legacy EPG WEB EPGAPP EPG DB

F/W L/B
Network

Access Policies
Access Policies

?
Access Policies Application Network Profile
EPG DB
Legacy
Network
F/W EPG
WEB L/B EPGA
PP

• Consist of named selectors and profiles for the:

• Switches where a device is connected
• Interface on that switch where the device is connected
• L1 and L2 configuration for that interface such as:
• CDP, LLDP, LACP
• Attachable Access Entity Profile(AAEP) to tie the switch and interface to a set of VLANs
and the Domain used to reference the set under the Tenant. Represents a group of
external entities with similar infrastructure policy requirements.
• VLAN Pool to describe the group of possible VLANs the device will possibly use at
some point
• A Domain to tie the VLANs and switch/interface together as well as give the Tenant
something to reference and validate the configuration is correct.
Tenant Policies

Application Network Profile

EPG DB
Tenant Policies Legacy
Network
F/W
EPG
WEB L/B EPGA
PP
Tenant Policies
• Govern traditional networking configuration
• What VLAN goes on what interface as trunk or access
• Creates SVIs and VRFs
• Creates router configuration (OSPF, EIGRP, Static, etc.)

Application Network Profile

EPG DB
F/W L/B
Legacy EPG EPGA
Tenant Policies Network
WEB PP
Tenant Policies
• Logical container for set of policies
• Main Components:
• Application Profiles = Container of similar applications that are somehow related
• Application Profile has any number of Endpoint Groups (EPGs) inside
• Networking = Container for Network Infrastructure related items
• Bridge Domains
• VRFs
• External Bridged Networks
• External Routed Networks
• Security Policies
• Contain the Contracts used between EPGs to enable communication
 Tenant Model

Tenant

Outside Application Bridge

Profile VRF Contract Filter
Network Domain

Subnet
Subject
Endpoint
Group
 Tenant

VRF
VRF
• Layer 3 forwarding domain.
• Nothing fancy here, contains all routes for the particular VRF
• Routes will usually point to the local leaf SVI VLAN or via the overlay-1 VRF to a
destination leaf VTEP
• VRF scope is where communication policy is enforced.
Comprehensive look

VRF
 Tenant

Bridge Domain
Bridge VRF
• Ties to a VRF Domain

• Defines L2 forwarding characteristics and boundaries.

• L2 Unknown Unicast (Flood | Hardware Proxy)
• Forwarding for unknown L2 destinations
• L3 Unknown multicast(Flood | Optimized Flood)
• Multi-Destination Flooding(Flood in BD | Drop | Flood in Encapsulation)
• Multicast-Frame/MAC
• ARP Flooding(On | Off)

• Similar to a VLAN but not tied to a single VLAN

• Unicast Routing
• Subnets
Comprehensive look
Tenant

VRF

BD1

BD2

BD3
 Tenant

Subnets
Bridge VRF
Domain
• HSRP Evolved
Subnet

• Subnet under the BD creates an SVI only on the switches where there is an
endpoint that needs it.
• Known as a distributed default gateway
• Gateway inside the fabric is good, flood is always answered in a single hop.
• This SVI can be advertised externally through a routing protocol
Comprehensive look
Tenant

VRF

BD1
10.0.1.1/24

BD2
10.0.2.1/24

BD3
10.0.3.1/24
Distributed Gateway

10.0.1.1/24 10.0.1.1/24 10.0.2.1/24 10.0.1.1/24

• EP Move: SVI will be removed from the original leaf and programmed on the new leaf/location
• Gateway is always one hop away. Decouple identity and location
Unicast Routing
• Enables Routing
• Route between all BDs inside a VRF without configuring a routing protocol
• The subnet configured under the BD will be the SVI and Default Gateway for endpoints
• SVI is only programmed on the switches that have endpoints in that BD/EPG

• Traffic from inside a BD will hit the Distributed default gateway MAC and the
fabric will handle routing to the destination BD
Application Profile & Endpoint Groups
• Endpoint Groups are used to group similar endpoints connected to the fabric.
This is where policy is defined.
• An Application Profile(AP) is a logical container for Endpoint Groups (EPGs)
• An AP should logically group related EPGs, such as the 3-tierd Application
example:
• Application Profile “My Web App” Tenant

• Website – EPG
• Application – EPG
• DB – EPG Application Bridge
Profile Domain
Comprehensive look
Application Profile: My-Web-App

EPG1 – Web-Servers

Drupal
IIS
Server

Apache
Server
VLANS Tenant
Tenant
Redsox
Yankees
• In ACI, what is define as the encapsulation VLAN is
used as an identification for classifying traffic into End
Point Groups ( EPGs )
• The definition this classification of packets into an
EPG is done via static bindings or dynamic bindings
5
associated to VMM domains 1 3

2 4 6
• Once a packet has been identified as pertaining to an
EPG, it is tagged into specific, locally significant
VLANs or and globally unique VXLANs inside the leaf
node to identify for fabric policy enforcement
• Known as encapsulation normalization vlan100
vlan100
Comprehensive look
Tenant

AP VRF

EPG1 BD1
10.0.1.1/24

EPG2 BD2
10.0.2.1/24

EPG3 BD3
10.0.3.1/24
Security Policies
• ACI is whitelist based network
• Use contracts to define policy for which EPGs can talk to which other EPGs and
external EPGs
• Contracts are built with the following objects:
• Contract - Name
• Subject – Direction and Options
• Filter – Name and groups of filter entries
• Filter Entry – Specific protocol and ports and in which direction
Contracts
• One EPG is Providing the other is Consuming
• Think client/server relationship. One EPG is a server providing a service the client is
consuming the service
• Bi-Directional Communication is allowed by default
• Once again, do not confuse bi-directional communication with a provider/consumer role
• Pro-Tip: Only the client/consumer is allowed to initiate communications
Tenant

Contract Filter

Subject
ACI Provider/Consumer

HTTP Contract
HTTP Subject
Web-Client HTTP Filter Web-Server
EPG Consume 80 Provide 80 EPG
Source X
Dest 80

Sport = X Dport = 80 Sport = 80 Dport = X

Sport = X Dport = 80
 Comprehensive look with Contracts
Tenant

AP VRF

Consume
EPG1 BD1
10.0.1.1/24

EPG2 BD2
ICMP 10.0.2.1/24

EPG3 BD3
Provide
10.0.3.1/24
 VRF

10.0.1.1/24 BD1 BD3 10.0.3.1/24

VLAN 10 VLAN 20

EPG1 ICMP ICMP Contract Contract EPG3

What can one do with ACI?
Monitoring
• ACI offers a slew of monitoring and troubleshooting tools
• Event and Audit logs at numerous levels
• Ongoing as well as on-demand counters
• Graphs for statistics at numerous levels (vm, port, PC, vPC, BD, EPG, VRF)

• Troubleshooting Wizard for end to end traffic between two endpoints

• Shows counters, Contracts, traceroute, Topology
• Endpoint Tracker
• History, per endpoint, of all moves
• Capacity Dashboard
• Shows usage of different policies and scale
Stats - Port
Policy upgrade
• Ability to upgrade all switches and controllers in the fabric from one place, with a
single click
• Requires the upload of the new controller and switch image
• Then, create a firmware group
• Finally, Create Maintenance groups as needed to define which switches get
upgrade at what time
• Controllers are upgraded through a different “Controller Firmware” Policy
• Controllers are kicked off at the same time (sort of like a single maintenance group) and
upgrade sequentially.
Maintenance Group Logic - Safest
Group1 Group2

Group3 Group5
Group4 Group6
What is a fault ?
• Faults, events and audit logs are essential tools for monitoring the administrative and
operational state of an ACI fabric as well as troubleshooting current and past issues
• They are the first thing to check when something is not behaving as expected!
Fault
Tenant

AP VRF

EPG1 BD1
10.0.1.1/24

EPG2 BD2
10.0.2.1/24

EPG3 BD3
10.0.3.1/24
How does ACI work?
How Does it All Work?
 What is ACI?
• Interaction from a user through an Application Program Interface (API) creates
or modifies the objects in the model with the end goal of a policy to allocate or
configure resources.
• This interaction is done through
Data Management Engines (DMEs)
communicating with each other.

APIC PM/PE NXOS

Conf t
NGINX/
API
Logical Resolved Concrete Hardware Int e1/25
Switchport mode access
Switchport access vlan 3
No shut
Types of Objects
• Logical, resolved, and concrete
• Logical = configured in the GUI by the user
• Resolved = created by the APIC as a unit/object to communicate and pass information
to the switches
• Concrete = objects used by the switches to program hardware

Logical Resolved Concrete Hardware

Flow
• Process flow
• Sequential
• Use to your advantage

APIC Switch

APIC PM/
PE NXOS

NGINX/ Logical Resolved Concrete Hardware

API
Flow
Logical MO Concrete

Resolved Concrete

fvTenant
fvAp
fvEpP
fvAEPg
fvCtxDef vlanCktEp
fvCtx
fvBDDef l3Ctx
fvBD
l2BD

APIC SW
NGINX Policy Policy NXOS Hardware
Manager Element
Stateless Hardware
• Just like UCSB/UCSM…just applied to networking!
• Service Profiles allow a blade to fail and to be redeployed immediately.
• Templates and Policies abstract configuration from hardware. Reusability!
• Application Profile is equivalent to Service Profile
Service Profile

Network
Storage
Server
Overlays and Tunnels
• When first discovering the fabric, each switch that is registered is dynamically
assigned an IP address out of the Tunnel End Point (TEP) range specified
during the APIC setup script.
• The TEP range defines the Overlay-1 VRF.
• The IP address every switch receives is known as a virtual TEP and is used to
build tunnels between the leafs and spines
• Overlay-1 VRF contains /32 routes to each VTEP, VPC Virtual IP, APIC and
Spine Proxy IP
Overlays and Tunnels
10.0.64.1 10.0.64.2
• Infra-VLAN=3967
• TEP-
Pool:10.0.0.0/16
• Multicast Range: 10.0.128.1 10.0.128.2 10.0.128.3
255.1.1.1
• Admin Password:
ciscoLive16!

10.0.0.1 10.0.0.2 10.0.0.3

Forwarding
• The most important thing any router or switch can do
• ACI does it too
• Uses a fancy mix of IS-IS, enhanced VXLAN encapsulation, special VLAN
translation and a splash of policy

Policy
VLAN
xlate

IS-IS

iVXLAN
 Tunnel
IP-A IP-B
MAC-A MAC-B

Payload L3i L2i iVXLAN L3o L2o

DIP=IP-B DMAC=MAC-B DIP=TEP-3 DMAC=TEP3

SMAC=MAC-A SMAC=TEP1
SIP=IP-A SIP=TEP1
Forwarding and Learning
• Acts as a regular switch, learns and forwards based on MACs
• Also capable of learning IP addresses for a comprehensive endpoint

• Leafs learn remote endpoints as well for quicker lookup and directed forwarding
to a destination leaf.
• Not just an outgoing port

• Spines have a global (fabric wide) database of all endpoints and can forward to
any destination if needed
• BD settings determine learning and forwarding behavior
 MAC IP VTEP
MAC-A IP-A VTEP-1
MAC-B IP-B VTEP-2
MAC-C IP-C VTEP-3

• Spine looks up endpoint in global database/COOP and forward to leaf VTEP. If not found, packet is dropped.
• Optimization to traditional networking to cut down on unnecessary flooding.
L2 Unknown Unicast: Flood

• Uses multicast tree rooted in the spine for a specific BD(illustrated in red computers) all leafs
that have the BD are part of the multicast tree
• Imitates traditional networks, helpful for integrating an external gateway for migration
 VLAN 10 VLAN 10 VLAN 20

• Option One for dealing with some flooded traffic. The most traditional. Flood everywhere, every
encapsulation in the BD
 VLAN 10 VLAN 10 VLAN 20

• Option Two: Disallow floods entirely

 VLAN 10 VLAN 10 VLAN 20

• Option Three: Only allow the flood to propagate inside its own encapsulation, not the BD
Unicast Routing/DirectedARP. InspectARP frame for Destination IP and unicast to that leaf/Endpoint
Standard, TraditionalARP Flooding
Conversational Learning
MAC IP Interface
MAC-A IP-A Tunnel1
MAC-B IP-B Tunnel3

MAC IP Interface MAC IP Interface

MAC-A IP-A 1/15 MAC-B IP-B 1/25
MAC-B IP-B Tunnel13 MAC-A IP-A Tunnel31

IP-A IP-B
MAC-A MAC-B
Connecting to External Switches
• Just like other switches can be trunked together, ACI can trunk to any existing
switch in your datacenter
• The benefit is that ACI allows you to decide where to apply policy and where the
external endpoints are classified/learned
• ACI offers two options to connect to external switches:
• Extending the EPG outside of the fabric
• Extending the BD outside of the fabric
External Switches / Legacy Network

Gateway

Gateway

• Gateway can start outside of the fabric for migration purposes. Services on the Fabric will send their
traffic and floods outside
• Gateway can then be migrated into the fabric. External services can flood into the fabric
Connecting to External Routing Instances
• ACI can participate in routing as well, via static or dynamic protocols.
• Advertising subnets and learning external subnets just like any other router
• This is done through an External Routed Network in ACI
• The benefit is that policy can be applied at a subnet/prefix level toward a specific
EPG
• Known as a prefix-based EPG
External Routing Instances / Legacy Network

• ACI and External Legacy network will exchange routing tables

Connect to Servers
• ACI can accept any sort of server connected to a leaf, just like a traditional
switch can take any physical connection from an endpoint.
• ACI can seamlessly integrate with existing hypervisor environments
• The APIC will communicate to the hypervisor controller and create a virtual switch,
dynamically assign VLANs and create portgroups/networks for the VMs
• ACI will dynamically configure the interfaces with the appropriate VLANs
Servers, Hypervisors, FEX
What are Domains and why I need them?
• Domains tie together the Access Policy model to the Tenant/EPG model.
• When a domain is associated VLANs and interfaces are associated to an EPG

• Static Paths and Static VLAN pools work together with Domains to properly
program interfaces
• Imperative to have domains associated to EPGs when mixing VMM dynamic
domains and any other Domains
Static vs Dynamic Configuration
• Static implies manually configuring which interfaces have which VLANs from the
pool defined under access policies
• Used with a physical domain and a static VLAN pool
• Static configuration is done under the EPG by associating the physical domain and
creating a static path to a port and specifying a VLAN
• Dynamic implies that the VLAN is allocated automatically, randomly from the
pool
• Used with a VMM domain and a dynamic VLAN pool.
• Associating the VMM domain to the EPG creates a port-group/network in the VM
environment and based on CDP/LLDP adjacencies that are reported, VLANs are
programmed on the interface.
Static Deployment
• Compared to dynamic deployment, physical workloads are defined statically
• A Physical domain is needed on the EPG
• The second requirement is to configure a static path
• A static path specifies an interface on a switch, a port-channel on a switch, or a
vPC interface between a pair of switches as well as the VLAN that the end
device will be communicating on
• This VLAN can be:
• tagged
• untagged (access/native)
• 802.1p (still access/native but with QoS at MAC layer)
Static Deployment
Dynamic Configuration
• Used for VMM Domain Integration
• ACI and the controller exchange information such as
• Number and name of Hypervisors
• vmnic adjacencies to the leaf ports
• Requires CDP or LLDP
• VMs added to port-groups
• VMM domain associated to an EPG programs a port-group on the Controller

• With the goal of dynamically programming VLANs on the leaf interfaces.

Cisco ACI Hypervisor Integration
 VMM
Controller
Objects
Hypervisor VM
APIC

Hypervisor NIC

EPG
Adjacency VM NIC

Leaf Interface

ACI
VMM Domain
VM Portgroup VM
Hypervisor
Route Leaking and Inter-Tenant Communications
• In ACI, it is possible to have inter-VRF or inter-tenant communications
• This is accomplished by route leaking from one VRF to another using route-
maps and prefix-lists in the fabric
• Route leaking is enabled by a contract applied to an EPG where one EPG is
providing, the other EPG is in another Tenant or VRF and consuming.
Inter-Tenant Inter-VRF
Connecting ACI to Existing L4-L7 Service
Appliances Device Model

• Connecting to a Service appliance can be

accomplished in several ways:
1. Manual configuration of bridge domains and
EPGs (static)
• Someone needs to configure the device APIC Scripting Interface
2. Using the service graph feature of ACI in an
unmanaged mode / network-only stitching Device-Specific Python Scripts
mode (dynamic)
• Someone needs to configure the device Device Interface: REST or CLI
3. Using the service graph feature of ACI with a
device package to dynamically configure the
service appliance as well as the network

Service Device
Cisco ACI Service Insertion
Extending ACI Policy Model to L4-L7 Services
Application Centric Infrastructure Building Blocks
Physical + Virtual
Traditional
3-Tier FW
Application ADC WEB ACC APP DB

APPLICATION
NETWORK PROFILE

CONTROLLER POLICY MODEL NEXUS 9300 AND 9500 F5 BIG-IP

Policy Model Extended to L4-L7

Building blocks of ACI

Application: 3 tier application (WEB-APP-DB)  This may use ADC, FW services

End point Group (EPG): Grouping of application Components
Policy model: Define QOS, Security, Network, L4-L7 etc. to be applied to EPG
 ACI L4-L7 Service Automation thru Device Package
F5 Device Package

Device Package contains

Policy
Configuration Model (XML File) Engine APIC provides extendable policy model through
Device Package
Python Scripts
Device Package contains XML file defining
APIC– Policy Manager Device Configuration Model
Configuration Model (XML File)
Provider Administrator can upload a Device
Package

Device scripts translatesAPIC API callouts to

Script Engine device specific callouts
APIC Script Interface
Python Scripts
APIC Script Interface

BIG-IP
Deploy F5 iWorkflow Dynamic Device Package in ACI

F5 Synthesis Fabric
ACI Fabric Virtual Edition Appliance Chassis