QOS on Nexus 5500 Series Switches
QOS on Nexus 5500 Series Switches
Nexus 5500 QOS provides the per-system class-based traffic control. These feature are:
- Lossless Ethernet: Priority flow control
- Traffic Protection: Bandwidth management
- Configuration Signaling to Endpoints: Part of DCBX.
Here we are not much going to discuss about the Nexus 5500 Hardware Architecture but yes we will discuss how Nexus 5500 manages the QOS and VOQ concepts.
Each Nexus 5500 is composed of following components:
- The Ingress unified port-controller (UPC)
- The Crossbar fabric
- The Egress UPC
Each UPC manages the traffic for group of eight 1/10GE ports. These UPC provides Traffic buffering, arbitration to cross crossbar fabric etc.
In Nexus 5500, all packets are managed by UPC and are never managed by Control Plane CPU. All classification, Marking, Queuing, Policing are done in hardware either on Ingress UPC, Egress UPC or On Crossbar fabric.
Nexus 5500 supports following types of QOS policies:
- qos: It defines (MQC) objects which is used for marking and policing
- network-qos: It defines the various characteristics of network-wide QoS properties (such as used in data center bridging [DCB] networks) and it should be applied consistently on all switches participating in the network
- queuing: it defines MQC objects which is used for queuing and scheduling, and in addition to a limited set of marking objects
Here qos Policies are applied on Ingress interface or crossbar fabric and network-qos policies are applied only on crossbar fabric and queuing policies are applied on ingress , egress interface and crossbar fabric.
Virtual Output Queue:
Whenever any nexus 5500 receives the packet and destination interfaces is congested then the nexus 5500 queues the packet at ingress port. In ingress the ingress port has very less buffer say 640KB which is not good enough in traffic congestion in that case the VOQ plays very important role in queuing.
Total queue size available to egress ports is equal to total number of ingress ports multiply queue depth per port.
Let’s say if you have 10 ingress ports and each has 640KB buffer than the instead on relaying the buffer size available to egress , these ten ingress ports increases the buffer space for uplink by 10*640KB = 6.4MB of buffer space to the congested port.
Let’s say there are three ingress ports sending packets to single egress port which is congested. Packets are queued at ingress port buffer until egress queues are somehow empty to process next packet.
Each Nexus 5500 ingress port has set of eight VOQ for each egress ports – one for each class of service. Let’s say if server is connected to Eth1/1 port and there are three 1 egress port 1/46, eth1/47, eth1/48 then three VOQ of 8 queue for eth1/1 ingress port will be used for three egress ports. So in total NX-OS supports 1024 ingress VOQ on each and every ingress ports ( 8Classes * 128 Destination) of switch model is 5596 then 96 ports + 2 optional layer 3 daughter cards each having 16 connection to cross bar fabric so 96+16+16 = 128.
Central arbiter manages the queues on each port which further provides the access to cross the crossbar fabric.
Here Queue 0 on egress port is congested while other 7 queue is still not congested then in this case the central arbiter will notify the ingress port to queue the packet for Queue 0 till congestion is cleared on egress while packet on rest queue will keep on flowing as it is.
The show queuing interface ethernet x/y command is used to investigate packet drops on a Nexus 5500. Below example shows how to look for ingress packet drops on a Nexus 5500.
In above example, packets are discarded on ingress rather in egress.
QOS Groups & System Class:
In IOS the policy-maps are attached to interface however in NX-OS these policy-maps can be attached to system also by system-qos, that means if you are using the system-qos, you are targeting the cross-bar fabric.
Service-policy can be used to associate the policy map with system-qos target. System-Qos policies are used to define system-class, class of traffic across entire switch and their attributes.
On Nexus 5500 switch, a system –class is uniquely identified by a qos-group value. There are total six system class where class 0 is default and rest 1-5 are user defined configurable class and if we are using FCOE then class 1 will be used for FCOE and then class-2 to 5 will be used defined class.
The default system class are defined below:
- Drop system class: By default, the NXOS software classifies all unicast and multicast Ethernet traffic into the default drop system class. This default drop system class is identified by QoS group 0. As soon as the system starts up this class (the class is named class-default in the CLI) is automatically created when the system starts up. This class is also the default for all traffic (class-default), meaning any matched traffic ends up in QoS group 0.
- FCoE system class: All Fibre Channel and FCoE control and data traffic is automatically classified into the FCoE system class, which provides no-drop service. This class is created automatically when the FCoE feature is enabled (the class is named class-fcoe in the CLI). Like the Drop system class, you can neither delete this class nor modify the CoS value associated with this class. This class is always identified by QoS group 1.
- Internetwork Control: This class is inherent in the system and cannot be deleted. It is responsible for things like Simple Network Management Protocol (SNMP), routing protocol traffic, and so on. Internetwork Control traffic uses CoS 6.
- Network Control: This class is also inherent in the system and cannot be deleted. It is used for things like communicating to the Nexus 2000 Fabric Extender. Network Control class traffic is marked as CoS 6.
The QoS Groups are mostly used as internal label and is used to identify the traffic so that it can be managed correctly in Nexus 5500 QOS system. The attributes that QoS Groups used to identify the traffic is MTU, BW, CoS value etc.
In working at first Class-map is used to match the traffic pattern and then is these matched traffic is associated to QoS group by policy map. Now the characteristics of each QOS group is configured by network-qos policy maps.
Nexus 5500 QOS configuration & Design Steps:
In Nexus 5500 QOS is enabled by default and there are following steps used to configure QOS in Datacenter switch or aggregation switch.
- Configure ingress QoS models, including the following:
- Trust models
- Classification and marking models
- Ingress policing models
- Configure egress/VOQ queuing model.
- Configure QoS to support optional designs.
Each of these configuration steps is discussed in the following sections.
Ingress QOS model
The ingress QOS model has three steps:
- Classification and marking
- Policing (Optional)
Here we will discuss about the trusted server models where DC ports are trusted and untrusted Server Model where Ports are not trusted.
Trusted Server Model: In this all ethernet interface are by default trusted which means the CoS and DHCP marking present in packet are preserved unless marking policies are configured to overwrite.
Untrusted Server Model: In this model , when server ports are not trusted or server is not trusted then QOS marking is reset to 0 and is put to default class and its QoS group and DSCP is set to 0.
These default class 0 is by default created and we don’t need to create the class-maps.
Now we only need to configure policy-map that uses class-default to reset its DSCP to 0 and assign to QoS group 0.
With the qos type policy map, it is not possible to re-mark the CoS value, which is used for internal queuing. Setting/resetting CoS values is done with a network-qos type policy map.
Now once policy-map is configured, let’s see how QoS policy map is applied to a target and it can be done by associating to system qos that means it will be attached to all interface of Switch.
Classification and marking
Classification is done on ingress port of Nexus 5500 switch but at least CoS is used to map Eight VOQs per port.
Following is the eight CoS values used in Nexus 5500 system.
Let’s understand with the example that a server is running a web application and provide traffic with DSCP value AF22 and COS value 2. Here we can use ACL because all traffic is from same server for traffic classify and then map to QOS group.
The next step is to assign matched traffic to the desired QoS group. There is no hard-and-fast rule as to which QoS group traffic types get assigned to, as long as you remember that you are limited to five customizable groups (groups 1–5).
Once the qos type policies have been configured, the next step is to attach them to an interface.
Now that the classification and marking portions of the configuration are finished, the next steps involve enabling the QoS group to which this traffic is added. If this step is missed, the queue for this class is not enabled and QoS does not work. It is necessary to first activate and configure the new QoS group before it can be used.
Once the policy map for the QoS group is attached to the system class, it is activated. In this example, the policy map does nothing more than activate the class and set the CoS value, but network-qos policy maps are also used to set a host of other features including setting the MTU, ingress buffer size, and no-drop behavior.
It may be the situation where we have police the traffic at ingress level. In nexus 5500 platform, remarking is not possible and here if policer detects the violation it drops the packets.
The steps required to configure ingress policing are as follows:
- Configure the qostype class map.
- Configure the qostype policy map with the police option set.
- Attach the qostype policy map to an ingress interface.
Let’s understand how ingress policer works:
In this example, traffic is policed down to 60 percent of the available bandwidth with a committed burst rate of 15 MB. Note that the conform action in this example is to transmit and the violate action is to drop.
Modifying Ingress Buffer Size:
By default, class-default is allocated to entire ingress buffer available (470KB). But when we create the new QOS group the buffer required for this new group is carried away from default class and the amount of buffer that is left for default class is by following equation:
470KB – Σ [28.6KB + B] * N
B = Ingress buffer size of each traffic class configured (see below Table)
N = The number of QoS groups in the system
Below example show how to adjust the ingress buffer size for two classes. Here Transnational Data class is given a system-wide ingress buffer size of 85000 bytes and the Real-Time class is given 30000 bytes.
Egress Queuing Model:
Nexus 5500 supports both four class system model and eight class system model.
Here we will discuss eight class system model for Nexus 5500.
Although the Nexus 5500 only offers up to six custom queues (including the FCoE and default classes), it is still possible to map the eight-class reference model to fit your QoS design reasonably well.
As before, the eight-class model design is configured in four steps, as follows:
- Configure the qostype class and policy maps.
- Configure the network-qostype class and policy maps.
- Configure the queuingtype class and policy maps.
- Apply all the service policies to the system QoS class.
Below configuration explains the configuration of the qos type class and policy maps. Note that in this example five class maps are defined because the FCoE and default classes are hard-coded in the Nexus operating system.
The next step is to configure the network-qos class and policy maps. Because there are only four customizable QoS groups to work with, the network-qos class map must consolidate two of these classes. This is most easily done by consolidating the Interactive Video and Streaming Video classes into one class called simply Video
The next step is to configure the queuing policies for each traffic class.
The final configuration step is to attach these three policy maps to the system, as demonstrated in below Example this activates the policies for all interfaces on the Nexus 5500.