Verifying Endpoint Connectivity

Verifying Endpoint Connectivity

Cisco ACI endpoint learning behavior is important to understand, so you can successfully troubleshoot issues in the fabric. First, you should know how to verify the endpoint connectivity in the fabric and obtain relevant endpoint information.

The endpoints in Cisco ACI are devices that are connected to the ACI fabric directly or indirectly. Endpoint examples include servers, virtual machines, storage, client computers, and so on. During ACI operation or troubleshooting, typical queries regarding endpoint connectivity include:

• What are the current endpoints in the fabric?
• Where is a specific endpoint?
• What was connected to the network last Thursday between 3:30 and 4:00?
• What are all the endpoints belonging to a given tenant?
• What endpoints are on this subnet?
• What is the history of a given endpoint (movement, and so on)?

You can provide answers to these questions while understanding the endpoint learning and behavior in the fabric, and using ACI tools that provide endpoint information.

Endpoints in Cisco ACI

Cisco ACI replaced the MAC address table and ARP table with a single table called the endpoint table. This change implies that Cisco ACI learns that information in a different way than in a traditional network. Cisco ACI learns MAC and IP addresses in hardware by looking at the packet source MAC address and source IP address in the data plane, instead of relying on ARP to obtain a next-hop MAC address for IP addresses.

This approach reduces the amount of resources that are needed to process and generate ARP traffic. It also allows detection of IP address and MAC address movement without the need to wait for Gratuitous Address Resolution Protocol (GARP) if some traffic is sent from the new host. Although Cisco ACI uses the endpoint table instead of the MAC address and ARP tables, it still uses the RIB and the ARP table (for Layer 3 Outside).

In a traditional network, three tables are used to maintain the network addresses of external devices: MAC address table for Layer 2 Forwarding, Routing Information Base (RIB) for Layer 3 forwarding, and ARP table for the combination of IP addresses and MAC addresses.

Cisco ACI maintains this information in a different way, and uses endpoints to forward traffic. An endpoint consists of one MAC address and zero or more IP addresses (endpoint can be Layer 2 or Layer 2 + Layer 3 information), while each endpoint represents a single networking device.

Forwarding table lookup order in Cisco ACI follows this order:

• Endpoint table (can be displayed using the show endpoint command)
• RIB (can be displayed using the show ip route command)

Although Cisco ACI mainly uses the endpoint table instead of the MAC address and ARP tables, it still uses the RIB and the ARP table. This capability is especially for L3Out communication, because the maximum number of IP addresses on a single endpoint (one MAC address) is limited, and there can be a huge number of IP addresses behind a single next-hop MAC address (external router) on a L3Out connection.

Local Endpoints and Remote Endpoints

A leaf switch has two types of endpoints: local endpoints and remote endpoints. Local endpoint information is stored in the endpoint table on each leaf. Remote endpoint is stored in the endpoint table on each leaf only when a conversation to the endpoint is happening (conversational learning).

Therefore, local endpoints for a specific leaf switch reside directly on that leaf switch (for example, directly attached), whereas remote endpoints for that leaf switch reside on other leaf endpoints. Although both local and remote endpoints are learned from the data plane, remote endpoints are merely cached locally to each leaf.

Local endpoints are the main source of endpoint information for the entire Cisco ACI fabric. Each leaf is responsible for reporting its local endpoints to the Council Of Oracle Protocol (COOP) database, located on each spine switch, which implies that all endpoint information in the Cisco ACI fabric is stored in the spine COOP database. Because this database is accessible, each leaf does not need to know about all the remote endpoints to forward packets to the remote leaf endpoints. Instead, a leaf can forward packets to spine switches, even if the leaf does not know about a particular remote endpoint. This forwarding behavior is called spine proxy.

Local Endpoint Learning

Cisco ACI learns the MAC (and IP) address as a local endpoint when a packet comes into a Cisco ACI leaf switch from its front-panel ports. Front-panel ports are southbound ports from the perspective of Cisco ACI and do not face spine switches.

A Cisco ACI leaf switch follows these steps to learn a local endpoint MAC address and IP address:

1. Leaf receives a packet with a source MAC Address (MAC A) and source IP Address (IP A).
2. Leaf learns MAC A as a local endpoint.
3. Leaf learns IP A tied to MAC A if the packet is an ARP packet.
4. Leaf learns IP A tied to MAC A if the packet is routed.

If the packet is switched and not an ARP packet, the Cisco ACI leaf never learns the IP address but only the MAC address. This behavior is the same as traditional MAC address learning behavior on a traditional switch.

Remote Endpoint Learning

Cisco ACI learns a MAC or IP address as a remote endpoint when a packet comes into a Cisco ACI leaf switch from another leaf switch through a spine switch. When a packet is sent from one leaf to another leaf, Cisco ACI encapsulates the original packet with an outer header representing the source and destination leaf Tunnel Endpoint (TEP) and the Virtual Extensible LAN (VXLAN) header, which contains the bridge domain or VRF information of the original packet.