NSX Layer 2 Extention
NSX Layer 2 Extention
Layer 2 VPN is used to extend the Layer 2 Broadcast domain, and can be used for following:
- Extend a Layer 2 between a remote office and the main office.
- Extend a Layer 2 between virtual machines in different data centers.
- Extend a Layer 2 between a private and a public cloud.
The NSX Edge supports Layer 2 VPN over Secure Sockets Layer (SSL), port TCP 443. If the NSX Edge is extending a VLAN, the VLAN must be configured in a distributed portgroup.
Below figure shows logical view of multiple pairs of NSX Edges creating a Layer 2 VPN between VXLANs, VLANs, and a VXLAN and a VLAN.
The NSX Edge can also be used to extend a VLAN between a private cloud and vCloud Hybrid Services, vCloud Air as shown below:
The NSX Edge supports Layer 2 VPN in a point-to-point deployment, and it must be with another NSX Edge in a server-client relationship
The NSX Edge can be used to extend a VLAN or VXLAN between two data centers even if the NSX Edges are managed by different NSX Managers or one of the Edges is a Standalone Edge.
The encryption algorithms supported by the NSX Edge for Layer 2 VPN. The Layer 2 VPN Server dictates the encryption algorithm upon tunnel negotiations with the Layer 2 VPN Client.
An NSX Edge can only support being either a client or a server at one time, but not both, and requires a Trunk interface with sub interfaces. A pair of Edges can only do a Layer 2 VPN for 200 pairs of Ethernet domains, as shown in below figure. The path from the Layer 2 VPN Server and the Layer 2 VPN Client must have an MTU of 1600 or higher.
Layer 2 VPN Traffic Flow:
To understand the packet walk, let’s suppose the first ping sent from Virtual Machine ServerApp02 to ServerApp01. To understand we have to assume the following:
- NSX Edge L2-VPN-S is running in EDG-A1-ESXi02.
- NSX Edge L2-VPN-T is running in COM-B1-ESXi01.
- ServerApp01 does not have an ARP entry for ServerApp02.
- ServerApp02 does not have an ARP entry for ServerApp01.
- Replication Mode for all logical switches is Hybrid or Unicast.
- The NSX Controllers do not have an ARP entry for ServerApp01 or ServerApp02.
- Virtual Machine ServerApp01 is running in COM-B1-ESXi02.
- Virtual Machine ServerApp02 is running in COM-A1-ESXi02.
Step 1. ServerApp02 wants to send a ping to ServerApp01 but does not have an ARP entry for it.
Step 2. ServerApp02 sends an ARP request.
Step 3. Logical switch Layer 2 VPN-X receives the request and processes it.
Step 4. Logical switch Layer 2 VPN-X replicates the ARP request, as shown in Figure below
Step 5. The ARP request is received by Edge L2-VPN-S’ Internal interface, Layer 2 VPN-X.The local copy of Layer 2 VPN-X in ESXi host EDG-A1-ESXi02 learns the MAC address of ServerApp02.
Step 6. L2-VPN-S takes the ARP request, puts it in a Layer 2 VPN tunnel, and sends it to Edge L2-VPN-T.The tunnel goes out of the Uplink interface External- VPN X, toward the physical network.
Step 7. Edge L2-VPN-T receives the tunnel traffic, over its Uplink Interface External-VPN-Y, validates it, and decapsulates it.
Step 8. Edge L2-VPN-T takes the ARP request and sends it to logical switch Layer 2 VPN-Y.The traffic is sent out over the Internal interface, Internal-Y.
Step 9. Logical switch Layer 2 VPN-Y learns the MAC address of ServerApp02 and tells the NSX Controller.If this had been a distributed portgroup instead of a logical switch, no MAC learning would’ve taken place.
Step 10. Logical switch Layer 2 VPN-Y receives the ARP request and processes it.
Step 11. Logical switch Layer 2 VPN-Y replicates the ARP request. The replication happens over VXLAN overlays.
Step 12. ServerApp01 receives the ARP request and processes it.The local copy of Layer 2 VPN-Y in ESXi host COM-A1-ESXi02 learns the MAC address of ServerApp02.
Step 13. ServerApp01 sends a unicast ARP reply to ServerApp02.
Step 14. Logical switch Layer 2 VPN – Y receives the ARP reply, processes it, and sends it over to COM-B1-ESXi01 over a unicast VXLAN overlay.The logical switch learned the MAC address of ServerApp02 in step 12.
Step 15. Logical switch Layer 2 VPN-Y in COM-B1-ESXi01 receives the ARP reply, processes it, and sends it to Edge L2-VPN-T’s interface Internal-Y
- The logical switch learns ServerApp01’s MAC address.
- The logical switch learned ServerApp02’s MAC address in step 10.
Step 16. Edge L2-VPN-T receives the ARP reply, puts it in a Layer 2 VPN tunnel, and sends it to Edge L2-VPN-S.The tunnel goes out of the Uplink interface External-Y, toward the physical network.
Step 17. Edge L2-VPN-S receives the tunnel traffic over its Uplink Interface External-VPN-X, validates it, and decapsulates it.
Step 18. Edge L2-VPN-S takes the ARP reply and sends it to logical switch Layer 2 VPN-X.The traffic is sent out over the Internal interface, Internal-VPN-X.
Step 19. Logical switch Layer 2 VPN-X learns the MAC address of ServerApp01 and tells the NSX Controller.
Step 20. Logical switch Layer 2 VPN-X receives the ARP reply and processes it.
Step 21. Logical switch Layer 2 VPN-X forwards the ARP request in a VXLAN overlay unicast to COM-B1-ESXi02.
Step 22. ServerApp02 receives the ARP reply, adds the entry to its ARP table, and sends the first ping to ServerApp01
Step 23. ServerApp01 receives the ping, but it doesn’t have an ARP entry in its ARP table for ServerApp02, so it sends out an ARP request for ServerApp02’s MAC.
Steps 1 thru 22 are repeated without the MAC learning by the logical switches.
Layer 2 Bridging
With Layer 2 VPNs we can extend Ethernet broadcast domains, whether VXLAN, VLANs, or both, over untrusted mediums.
NSX does not provide an option to bridge between a virtual workload in a VXLAN and a physical workload in a VLAN. NSX does not support connecting a physical workload directly to a VXLAN (only VMs can be in an NSX VNI); therefore there is no option to bridge two VXLANs.
In NSX, the bridging of virtual and physical workloads is called Layer 2 Bridging. Layer 2 Bridging is done by the DLR, and it only supports bridging between VXLANs and VLANs. The ULR does not support Layer 2 Bridging. Table below shows the type of Layer 2 extensions supported by the NSX Edge and the DLR.
Layer 2 Bridging is done by selecting a logical switch and a distributed portgroup and telling a DLR to connect to each. These connections are LIFs but with connections to dvPorts in the vDS called sinkports. sinkports are special vDS ports that get a copy of all BUMs in the VLAN.
The function of Layer 2 Bridging is done in kernel, but it is not distributed. One of the ESXi hosts that has the DLR is selected to do all the bridging. That ESXi host is referred to as the Bridge Instance. The Bridge Instance is the only ESXi host that has the sinkport connections doing the bridging.
DLR doing bridging between a VM in logical switch Layer 2 Bridge and a physical server in VLAN 200.
A single DLR can have multiple bridges as long as it always involves different pairs of VXLAN-VLAN LIFs; however, a DLR always has a single Bridge Instance.
A single DLR doing Layer 2 Bridging for two different VXLAN-VLAN pairs. Traffic from one bridge pair is not seen by the other unless it is routed.
Layer 2 Bridging Traffic Flow:
Below Figure shows two clusters, a VM and a physical NFS server. Each cluster has its own vDS, and the NFS server is in the same subnet (10.10.11.0/24) as the virtual machine (ServerApp02). We do a packet walk of the first ping ServerApp02 sends to the physical NFS server. EDG-ESXi02 is the Bridge Instance.
Step 1. Virtual machine ServerApp02 wants to ping the NFS server but does not have an ARP entry for the NFS sever in its ARP table.
Step 2. ServerApp02 sends an ARP request.
Step 3. The ARP request is received by logical switch X-Layer2 Bridge in ESXi host EDG-ESXi02.The ARP request was replicated.
Step 4. Logical switch X-Layer2 Bridge in ESXi host EDG-ESXi02, which is the Bridge Instance, broadcasts the frame locally, and a copy is received by the sinkport connected to the LIF of DLR Layer 2 Bridge.
Step 5. Layer 2 Bridge learns the MAC address of ServerApp02 and tells the NSX Controller.
Step 6. Layer 2 Bridge forwards the ARP request out of the sinkport connected to the bridged VLAN LIF.
Step 7. The frame is received by Edge vDS EDG_A1's EDG_A1-L2-Bridge dvPortgroup in VLAN 10 in EDG-ESXi02.vDS EDG_A1 does not learn the MAC address of ServerApp02.
Step 8. vDS EDG_A1 sends the frame out to the physical network, over VLAN 10, where every physical switch in the VLAN learns the MAC address of ServerApp02, as shown in Figure below
Bridged traffic is pinned to a single dvUplink in vDS EDG_A1.The ARP request is a broadcast that will be processed by all switches in VLAN 10, including the local copy vDS EDG_A1 running in ESXi host EDG-ESXi02.
Step 9. The physical NFS server receives the ARP request and sends back a unicast ARP reply to ServerApp02.
Step 10. The ARP reply is received by the dvUplink of vDS EDG_A1 in EDG-ESXi02 to which the bridge traffic is pinned.
Step 11. vDS EDG_vDS forwards the frame to all of its interfaces in VLAN 10, including the sinkport connected to the bridged VLAN LIF of DLR Layer 2 Bridge.
Step 12. Layer 2 Bridge learns the MAC address of the physical NFS server and tells the NSX Controller.
Step 13. Layer 2 Bridge sends the ARP reply out of the sinkport connected to the bridged VXLAN LIF.
Step 14. Logical switch X-Layer 2 Bridge in ESXi host EDG-ESXi02 gets the ARP reply and learns the MAC address of the NFS server.
The logical switch will not tell the NSX Controller about it. This is where the close working relationship between the logical switch and the DLR comes into play.
Step 15. The ARP reply is forwarded to ServerApp02.
Step 16. ServerApp02 adds the ARP entry in its ARP table and sends a ping to the physical NFS server.If the physical NFS server sends an ARP request for ServerApp02 before it can respond to the ping, the steps are similar to steps 1 through 15.