Virtual & Physical Network Connectivity
Virtual & Physical Network Connectivity
Logical Router VLAN LIF
The logical router’s LIFs may connect to VXLAN backed dvPortgroups (logical switches) or VLAN backed vPortgroups. Any logical router may have a direct Layer 3 connection to the physical world if the following condition is met: It is a distributed logical router and all ESXi hosts running the instance of the logical router have access to the same VLAN backed dvPortgroup.
If any of the ESXi hosts in the transport zone did not have access to the VLAN backed dvPortgroup, there would be no way to configure the VLAN LIF for the logical router copy in that ESXi host.
vMAC is the same across all ESXi hosts and the pMAC is different in each ESXi host, while the LIF IP remains the same in all ESXi hosts running the same copy of the logical router. The vMAC is seen by the physical switches when the logical router has VLAN LIFs.
Let’s do some packet walks to see what happens when the ESXi hosts running a copy of the same logical router send and respond to ARP requests over VLAN LIFs.
Above figure shows a logical router named Physical Access that has a VLAN LIF in dvPortgroup Physical-PG and the following configurations:
- Virtual machines L3-T-VM1 and L3-T-VM2 and logical router Physical Access are connected to the same logical switch.
- Virtual machines L3-T-VM1 and L3-T-VM2 have an ARP entry for their default gateway, which is the IP of logical router’s Physical Access VXLAN LIF.
- Virtual machine L3-T-VM1 is running in ESXi host COM-B1-ESXi01.
- Virtual machine L3-T-VM2 is running in ESXi host COM-B1-ESXi02.
- dvPortgroup Physical-PG is in VLAN 10.
- Physical server L3-Server-P1 is connected to a physical switch in VLAN 10.
- Physical server L3-Server-P1’s default gateway IP is the IP of the logical router’s Physical Router VLAN LIF.
Step 1. Virtual machine L3-T-VM1 sends traffic destined for L3-Server-P1.The destination MAC is L3-T-VM1’s default gateway, the vMAC.
Step 2. Physical Access, in COM-B1-ESXi01, receives the frame, does a routing table lookup, and determines the egress interface to be the VLAN LIF.
Step 3. Physical Access sends an ARP request out of VLAN LIF with a source MAC of the vMAC, 00:50:56:56:44:52.
Step 4. The frame is received by the physical switch L2-SWITCH, which learns MAC 02:50:56:56:44:52 is coming from port 1/1.L2-SWITCH adds the entry to its MAC table.
Step 5. L2-SWITCH forwards the frame out of port 1/3 as shown in below figure
Step 6. Now virtual machine L3-T-VM2 sends traffic destined for L3-Server-P1, JUST after L3-T-VM1 had sent its packet.The destination MAC is L3-T-VM2’s default gateway, the vMAC.
Step 7. Physical Access, in COM-B1-ESXi02, receives the frame, does a routing table lookup, and determines the egress interface to be the VLAN LIF.
Step 8. Physical Access sends an ARP request out of VLAN LIF with a source MAC of the vMAC, 02:50:56:56:44:52.This is the same MAC from step 3.
Step 9. The frame is received by L2-SWITCH, which learns MAC 02:50:56:56:44:52 is coming from port 1/2.
Do you see the problem? L2-SWITCH sees the same vMAC bouncing back and forth between ports 1/1 and 1/2 and this is called a MAC flap, and it is BAD. MAC flaps can cause Layer 2 switches to start flooding frames. In this packet walk we just concluded, only L2-SWITCH experienced the MAC flap, but in practice, the MAC flap may be experienced by many of the physical switches in the same Layer 2 domain, including the Top of Rack switches.
The Designated Instance is the ESXi host, randomly selected by the NSX Controller responsible for the logical router, that responds to ARP requests for the VLAN LIF IP and that sends out ARP requests over the VLAN LIF. No other ESXi host running a copy of the logical router sends an ARP request over the VLAN LIF nor responds to an ARP request for the IP of the VLAN LIF. By assigning the Designated Instance to send ARP requests over the VLAN LIF and to respond to ARP requests for the IP of the VLAN LIF, the vMAC will only be visible to physical switches over the port leading back to the Designated Instance. There is a single Designated Instance per VLAN LIF.
If the Designated Instance becomes unavailable, the NSX Controller selects a new Designated Instance.
Below Figure Will show you the traffic flow:
- Virtual machine L3-T-VM2 sends a frame for L3-Server-P2 with a destination MAC of the vMAC.
- Logical router Physical Access in COM-B1-ESXi02, the non-Designated Instance, gets the frame, does routing, and determines that the destination IP is directly connected in the VLAN LIF interface’s subnet.
- DLR Physical Access in COM-B1-ESXi02 sends a request to the Designated Instance, via DI IP, to send out an ARP request for L3-Server-P2’s MAC address.
- The Designated Instance sends the ARP request.The ARP request source MAC is the vMAC, 00:50:56:56:44:52.
- L3-Server-P2 receives the ARP request and sends back an ARP reply as shown in below figure
- The Designated Instance forwards the ARP reply information to DLR Physical Access in COM-B1-ESXi02.
- The DLR Physical Access in COM-B1-ESXi02 adds the entry in its ARP table and forwards the packet from L3-Server-P2.The source MAC address is COM-B1-ESXi02’s pMAC, 00:73:21:02:45:B0.
- L3-Server-P2 receives the packet from L3-T-VM1
- L3-Server-P2 wants to reply to L3-T-VM1, but it does not have the MAC address of the VLAN LIF, so it sends out an ARP request.
- Physical Access in COM-B1-ESXi02 receives the ARP request and ignores it because it is not the Designated Instance.
- Physical Access in COM-B1-ESXi01 receives the ARP request and responds, because it is the Designated Instance, with an ARP reply with the vMAC.
- L3-Server-P2 receives the ARP reply, adds the vMAC to the ARP table, and sends the packet to L3-T-VM1 inside a frame with a destination MAC address of 00:50:56:56:44:52.
- The Designated Instance receives the frame, does routing, and forwards the packet out of the VXLAN LIF.
- The logical switch delivers the frame to L3-T-VM1
In our packet walks we assumed that the logical router was the default gateway for the physical servers communicating with the VMs.
NSX Edge Gateway
The second method to provide IP connectivity between the virtual and physical networks is via the NSX Edge Gateway with at least one Uplink interface. When providing IP connectivity between the virtual and the physical networks, the NSX Edge Gateway is usually referred to as a Perimeter Edge.
Using a logical router with a VLAN LIF to connect to the physical world using a Perimeter Edge gateway has the advantage that you get all ESXi hosts sending traffic directly to the physical world. However, there are potential disadvantages of using the VLAN LIF over the Perimeter Edge. One of them is that the VLAN LIF’s VLAN might have to be stretched across multiple physical switches, thus necessitating STP. The other disadvantage is that the Designated Instance is randomly selected by the NSX Controller, thus making it hard to determine what the ingress point will be for traffic coming from the physical world. VMware recommends the use of a Perimeter Edge over using the logical router with a VLAN LIF.
Below figure hows a Perimeter Edge providing IP connectivity between the virtual and physical network. The process of enabling the NSX Edge Gateway to provide IP connectivity with the physical network is as simple as connecting one of the Edge’s Uplinks to a VLAN dvPortgroup , Remember that this VLAN is called the Edge VLAN. The VLAN portgroup could be standard or distributed. The portgroup should be present on all the ESXi hosts where the Perimeter Edge may run, which is typically just the hosts in the Edge Cluster.
Let’s do a quick packet walk to show traffic flowing between virtual and physical entities using a Perimeter Edge. Below figure shows the network we use for the packet walk, with a user opening a web page in the virtual machine ServerAPP02. We assume the logical router Monkey Island, the Perimeter Edge zambo, and the physical router all have full route visibility.
- The user sends a request to get a page from ServerAPP02.
- The traffic arrives at Router, which does a routing table lookup for ServerApp02’s IP.
- Router forwards the traffic to Hook’s Uplink interface.
- Zambo receives the traffic, does a routing table lookup, and routes the traffic toward Monkey Island’s Uplink interface through logical switch Layer 3 Transit.
- Monkey Island receives the frame, does a routing table lookup, and concludes ServerAPP02 is directly connected over the Internal interface APP.It should be clear that it is the logical router in the same host as the Perimeter Edge that receives the traffic.
- Monkey Island sends the traffic to ServerAPP02 through logical switch APP, as shown in below figure
Logical switch APP takes care of delivering the frame to ServerAPP02, wherever it might be running.
- ServerApp02 sends the page to the user by sending a frame to its default gateway, Monkey Island.
- Monkey Island receives the frame, does a routing table lookup, and concludes the next hop is zambo.
- Monkey Island sends the traffic to zambo over logical switch Physical-Virtual Boundary.Logical switch Layer 3 Transit takes care of delivering the frame to zambo, wherever it might be running.
- zambo receives the frame, does a routing table lookup, and determines the next hop is Router.
- zambo forwards the traffic to Router, over the EDG_A1-External_Access Portgroup in VLAN 25.
- Router forwards the traffic to the user over the physical network,