EMAIL SUPPORT

dclessons@dclessons.com

LOCATION

NZ

QOS on Nexus 5500 Series Switches

QOS on Nexus 5500 Series Switches

Posted on Jan 24, 2020 (0)

QOS on Nexus 5500 Series Switches

Nexus 5500 QOS provides the per-system class-based traffic control. These feature are:

  • Lossless Ethernet: Priority flow control
  • Traffic Protection: Bandwidth management
  • Configuration Signaling to Endpoints: Part of DCBX.

Here we are not much going to discuss about the Nexus 5500 Hardware Architecture but yes we will discuss how Nexus 5500 manages the QOS and VOQ concepts.

Each Nexus 5500 is composed of following components:

  • The Ingress unified port-controller (UPC)
  • The Crossbar fabric
  • The Egress UPC

Each UPC manages the traffic for group of eight 1/10GE ports. These UPC provides Traffic buffering, arbitration to cross crossbar fabric etc.

In Nexus 5500, all packets are managed by UPC and are never managed by Control Plane CPU. All classification, Marking, Queuing, Policing are done in hardware either on Ingress UPC, Egress UPC or On Crossbar fabric.

Nexus 5500 supports following types of QOS policies:

  • qos: It defines (MQC) objects which is used for marking and policing
  • network-qos: It defines the various characteristics of network-wide QoS properties (such as used in data center bridging [DCB] networks) and it should be applied consistently on all switches participating in the network
  • queuing: it defines MQC objects which is used for queuing and scheduling, and in addition to a limited set of marking objects

Here qos Policies are applied on Ingress interface or crossbar fabric and network-qos policies are applied only on crossbar fabric and queuing policies are applied on ingress , egress interface and crossbar fabric.

Virtual Output Queue:

Whenever any nexus 5500 receives the packet and destination interfaces is congested then the nexus 5500 queues the packet at ingress port. In ingress the ingress port has very less buffer say 640KB which is not good enough in traffic congestion in that case the VOQ plays very important role in queuing.

Total queue size available to egress ports is equal to total number of ingress ports multiply queue depth per port.

Let’s say if you have 10 ingress ports and each has 640KB buffer than the instead on relaying the buffer size available to egress , these ten ingress ports increases the buffer space  for uplink by 10*640KB = 6.4MB of buffer space to the congested port.

Let’s say there are three ingress ports sending packets to single egress port which is congested. Packets are queued at ingress port buffer until egress queues are somehow empty to process next packet.

Each Nexus 5500 ingress port has set of eight VOQ for each egress ports – one for each class of service. Let’s say if server is connected to Eth1/1 port and there are three 1 egress port 1/46, eth1/47, eth1/48 then three VOQ of 8 queue for eth1/1 ingress port will be used for three egress ports. So in total NX-OS supports 1024 ingress VOQ on each and every ingress ports ( 8Classes * 128 Destination) of switch model is 5596 then 96 ports + 2 optional layer 3 daughter cards each having 16 connection to cross bar fabric so 96+16+16 = 128.

Central arbiter manages the queues on each port which further provides the access to cross the crossbar fabric.



Here Queue 0 on egress port is congested while other 7 queue is still not congested then in this case the central arbiter will notify the ingress port to queue the packet for Queue 0 till congestion is cleared on egress while packet on rest queue will keep on flowing as it is.

The show queuing interface ethernet x/y command is used to investigate packet drops on a Nexus 5500. Below example shows how to look for ingress packet drops on a Nexus 5500.

N5K# show queuing interface ethernet 1/11
Interface Ethernet1/39 TX Queuing
qos-group         sched-type        oper-bandwidth
0                      WRR                      50
1                      WRR                      50
Interface Ethernet1/22 RX Queuing
qos-group 0
q-size: 243200, HW MTU: 1600 (1500 configured)
drop-type: drop, xon: 0, xoff: 1520
Statistics:
Pkts received over the port : 85257
Ucast pkts sent to the cross-bar : 930
Mcast pkts sent to the cross-bar : 84327
Ucast pkts received from the cross-bar : 249
Pkts sent to the port : 133878
Pkts discarded on ingress : 543
Per-priority-pause status : Rx (Inactive), Tx

In above example, packets are discarded on ingress rather in egress.

QOS Groups & System Class:

In IOS the policy-maps are attached to interface however in NX-OS these policy-maps can be attached to system also by system-qos, that means if you are using the system-qos, you are targeting the cross-bar fabric.

Service-policy can be used to associate the policy map with system-qos target. System-Qos policies are used to define system-class, class of traffic across entire switch and their attributes.

On Nexus 5500 switch, a system –class is uniquely identified by a qos-group value. There are total six system class where class 0 is default and rest 1-5 are user defined configurable class and if we are using FCOE then class 1 will be used for FCOE and then class-2 to 5 will be used defined class.

The default system class are defined below:

  • Drop system class: By default, the NXOS software classifies all unicast and multicast Ethernet traffic into the default drop system class. This default drop system class is identified by QoS group 0. As soon as the system starts up this class (the class is named class-default in the CLI) is automatically created when the system starts up. This class is also the default for all traffic (class-default), meaning any matched traffic ends up in QoS group 0.
  • FCoE system class: All Fibre Channel and FCoE control and data traffic is automatically classified into the FCoE system class, which provides no-drop service. This class is created automatically when the FCoE feature is enabled (the class is named class-fcoe in the CLI). Like the Drop system class, you can neither delete this class nor modify the CoS value associated with this class. This class is always identified by QoS group 1.
  • Internetwork Control: This class is inherent in the system and cannot be deleted. It is responsible for things like Simple Network Management Protocol (SNMP), routing protocol traffic, and so on. Internetwork Control traffic uses CoS 6.
  • Network Control: This class is also inherent in the system and cannot be deleted. It is used for things like communicating to the Nexus 2000 Fabric Extender. Network Control class traffic is marked as CoS 6.

The QoS Groups are mostly used as internal label and is used to identify the traffic so that it can be managed correctly in Nexus 5500 QOS system. The attributes that QoS Groups used to identify the traffic is MTU, BW, CoS value etc.

In working at first Class-map is used to match the traffic pattern and then is these matched traffic is associated to QoS group by policy map. Now the characteristics of each QOS group is configured by network-qos policy maps.

Nexus 5500 QOS configuration & Design Steps:

In Nexus 5500 QOS is enabled by default and there are following steps used to configure QOS in Datacenter switch or aggregation switch.

  1. Configure ingress QoS models, including the following:
  2. Trust models
  3. Classification and marking models
  4. Ingress policing models
  5. Configure egress/VOQ queuing model.
  6. Configure QoS to support optional designs.

Each of these configuration steps is discussed in the following sections.

Ingress QOS model

The ingress QOS model has three steps:

  • Trust
  • Classification and marking
  • Policing (Optional)

Trust:

Here we will discuss about the trusted server models where DC ports are trusted and untrusted Server Model where Ports are not trusted.

Trusted Server Model: In this all ethernet interface are by default trusted which means the CoS and DHCP marking present in packet are preserved unless marking policies are configured to overwrite.

Untrusted Server Model: In this model , when server ports are not trusted or server is not trusted then QOS marking is reset  to 0 and is put to default class and its QoS group and DSCP is set to 0.

These default class 0 is by default created and we don’t need to create the class-maps.

Now we only need to configure policy-map that uses class-default to reset its DSCP to 0 and assign to QoS group 0.

N5K(config)# policy-map type qos NO-TRUST
N5K(config-pmap-qos)# class type qos class-default
N5K(config-pmap-c-qos)# set dscp 0

With the qos type policy map, it is not possible to re-mark the CoS value, which is used for internal queuing. Setting/resetting CoS values is done with a network-qos type policy map.

Now once policy-map is configured, let’s see how QoS policy map is applied to a target and it can be done by associating  to system qos that means it will be attached to all interface of Switch.

N5K(config)# system qos
! This command enables system qos (global) configuration mode
N5K(config-sys-qos)# service-policy type qos input NO-TRUST
! Applies the NO-TRUST policy globally
Verification:
N5K# show policy-map NO-TRUST
Type qos policy-maps
====================
policy-map type qos NO-TRUST
class type qos class-default
set qos-group 0
set dscp 0
! Note how the QoS group is set to zero by the default class map

Classification and marking

Classification is done on ingress port of Nexus 5500 switch but at least CoS is used to map Eight VOQs per port.

Following is the eight CoS values used in Nexus 5500 system.

Let’s understand with the example that a server is running a web application and provide traffic with DSCP value AF22 and COS value 2. Here we can use ACL because all traffic is from same server for traffic classify and then map to QOS group.

N5K(config)# ip access-list WEB-SERVER
N5K(config-acl)# permit ip 10.1.1.0/24 any
! Define an ACL to match traffic
N5K(config)# class-map type qos WEB-CLASS
N5K(config-cmap-qos)# match access-group name WEB-SERVER
! Use the ACL as the match criteria

The next step is to assign matched traffic to the desired QoS group. There is no hard-and-fast rule as to which QoS group traffic types get assigned to, as long as you remember that you are limited to five customizable groups (groups 1–5).

N5K(config)# policy-map type qos WEB-POLICY
N5K(config-pmap-qos)# class type qos WEB-CLASS
N5K(config-pmap-c-qos)# set qos-group 2
! Map all matched traffic to QoS group 2
N5K(config-pmap-c-qos)# set dscp af22
! Set the DSCP value to AF22
Verification:
N5K# show class-map type qos
Type qos class-maps
===================
class-map type qos match-all WEB-CLASS
match access-group name WEB-SERVER
class-map type qos match-any class-fcoe
match cos 3
class-map type qos match-any class-default
match any
N5K# show policy-map type qos
Type qos policy-maps
====================
policy-map type qos WEB-POLICY
class type qos WEB-CLASS
set qos-group 2
set dscp af22
class type qos class-default
set qos-group 0
policy-map type qos fcoe-default-in-policy
class type qos class-fcoe
set qos-group 1
class type qos class-default
set qos-group 0

Once the qos type policies have been configured, the next step is to attach them to an interface.

N5K(config)# interface ethernet 1/10-15
N5K(config-if-range)# service-policy type qos input WEB-POLICY
! Attaches the policy map to the range of interface

Now that the classification and marking portions of the configuration are finished, the next steps involve enabling the QoS group to which this traffic is added. If this step is missed, the queue for this class is not enabled and QoS does not work. It is necessary to first activate and configure the new QoS group before it can be used.

! Step 1 – configure the network-qos class map
N5K(config)# class-map type network-qos CLASS-2
N5K(config-cmap-nq)# match qos-group 2
! Match on all traffic mapped to QoS group 2
! Step 2 – configure the network-qos policy map
N5K(config)# policy-map type network-qos NQ-POLICY
N5K(config-pmap-nq)# class type network-qos CLASS-2
! Enable the class – essentially turning on the queue
N5K(config-pmap-nq)# set cos 2
! Manually set the cos value for this class
! Step 3 – attach the policy map to the system class
N5K(config-pmap-nq-c)# system qos
N5K(config-sys-qos)# service-policy type network-qos NQ-POLICY

Once the policy map for the QoS group is attached to the system class, it is activated. In this example, the policy map does nothing more than activate the class and set the CoS value, but network-qos policy maps are also used to set a host of other features including setting the MTU, ingress buffer size, and no-drop behavior.

Policing:

It may be the situation where we have police the traffic at ingress level. In nexus 5500 platform, remarking is not possible and here if policer detects the violation it drops the packets.

The steps required to configure ingress policing are as follows:

  1. Configure the qostype class map.
  2. Configure the qostype policy map with the police option set.
  3. Attach the qostype policy map to an ingress interface.

Let’s understand how ingress policer works:

N5K(config)# policy-map type qos POLICE-WEB
N5K(config-pmap-qos)# class class-default
! Match traffic in the default-class
N5K(config-pmap-c-qos)# police cir percent 60 bc 15 mbytes conform transmit violate drop
! Police traffic in this class to 50% of bandwidth
N5K(config)# interface ethernet 1/10
N5K(config-if)# service-policy type qos input POLICE-MAP
! Attach the policy map to an interface

In this example, traffic is policed down to 60 percent of the available bandwidth with a committed burst rate of 15 MB. Note that the conform action in this example is to transmit and the violate action is to drop.

Modifying Ingress Buffer Size:

By default, class-default is allocated to entire ingress buffer available (470KB). But when we create the new QOS group the buffer required for this new group is carried away from default class and the amount of buffer that is left for default class is by following equation:

470KB – Σ [28.6KB + B] * N

Where:

B = Ingress buffer size of each traffic class configured (see below Table)

N = The number of QoS groups in the system

Below example show how to adjust the ingress buffer size for two classes. Here Transnational Data class is given a system-wide ingress buffer size of 85000 bytes and the Real-Time class is given 30000 bytes.

! Step 1 – Define a QoS class map
N5K(config)# class-map type qos TRANSACTIONAL-DATA
N5K(config-cmap-qos)# match af21
! Transactional Data is matched with AF21
N5K(config-cmap-qos)# class-map type qos REALTIME
N5K(config-cmap-qos)# match dscp ef
! Realtime traffic is matched with EF
! Step 2 – Define a QoS Policy Map
N5K(config)# policy-map type qos POLICY-QOS
N5K(config-pmap-qos)# class type qos TRANSACTIONAL-DATA
N5K(config-pmap-c-qos)# set qos-group 2
! Set transactional data to QoS group 2
N5K(config-pmap-c-qos)# class type qos REALTIME
N5K(config-pmap-c-qos)# set qos-group 5
! Map Realtime traffic to QoS group 5
! Step 3 – Apply the QoS policy map to the system target
N5K(config)# system qos
N5K(config-sys-qos)# service-policy type qos input POLICY-QOS
! Attaches the policy map to the system
! Step 4 – Define a network-qos Class Map
N5K(config)# class-map type network-qos TRANSACTIONAL-DATA
N5K(config-cmap-nq)# match qos-group 2
! Match traffic belonging to QoS group 2 (transactional data)
N5K(config-cmap-nq)# class-map type network-qos REALTIME
N5K(config-cmap-nq)# match qos-group 5
! Match traffic belonging to QoS group 5 (real-time traffic)
! Step 5 – Define a network-qos policy map
N5K(config)# policy-map type network-qos POLICY-NQ
N5K(config-pmap-nq)# class type network-qos TRANSACTIONAL-DATA
N5K(config-pmap-nq-c) queue-limit 85000 bytes
! This is an arbitrary value, for demonstration purposes only
N5K(config-pmap-nq-c)# class type network-qos REALTIME
N5K(config-pmap-nq-c) queue-limit 30000 bytes
! Lower the queue-limit for real-time traffic
! Step 6 – Apply the network-qos policy map to the system target
N5K(config)# system qos
N5K(config-sys-qos)# service-policy type network-qos POLICY-NQ
! Applies the policy map system-wide (to all interfaces)

Egress Queuing Model:

Nexus 5500 supports both four class system model and eight class system model.

Here we will discuss eight class system model for Nexus 5500.

Although the Nexus 5500 only offers up to six custom queues (including the FCoE and default classes), it is still possible to map the eight-class reference model to fit your QoS design reasonably well.

As before, the eight-class model design is configured in four steps, as follows:

  1. Configure the qostype class and policy maps.
  2. Configure the network-qostype class and policy maps.
  3. Configure the queuingtype class and policy maps.
  4. Apply all the service policies to the system QoS class.

Below configuration explains the configuration of the qos type class and policy maps. Note that in this example five class maps are defined because the FCoE and default classes are hard-coded in the Nexus operating system.

! Configure the qos class maps
N5K(config-cmap-qos)# class-map type qos match-any VOICE
N5K(config-cmap-qos)# match dscp ef
! Match voice traffic with DSCP EF
N5K(config-cmap-qos)# class-map type qos match-any SIGNALING
N5K(config-cmap-qos)# match dscp cs3
! Match signaling traffic with DSCP CS3
N5K(config-cmap-qos)# class-map type qos match-any INTERACTIVE-VIDEO
N5K(config-cmap-qos)# match dscp af41 af42 af43
! Match interactive video traffic with DSCP AF4
N5K(config-cmap-qos)# class-map type qos match-any STREAMING-VIDEO
N5K(config-cmap-qos)# match dscp af31 af32 af33 ! Match streaming video traffic with DSCP AF3
N5K(config-cmap-qos)# class-map type qos match-any TRANSACTINOAL-DATA
N5K(config-cmap-qos)# match dscp af21 af22 af23
! Match transactional data with DSCP AF2
! Configure the qos policy map
N5K(config)# policy-map type qos 8-CLASS-QOS-POLICY
N5K(config-pmap-qos)# class VOICE
N5K(config-pmap-c-qos)# set qos-group 5
! Map voice traffic to QoS group 5
N5K(config-pmap-c-qos)# class INTERACTIVE-VIDEO
N5K(config-pmap-c-qos)# set qos-group 4
! Map interactive video traffic to QoS group 4
N5K(config-pmap-c-qos)# class STREAMING-VIDEO
N5K(config-pmap-c-qos)# set qos-group 4
! Map streaming video to QoS group 4 (same as INTERACTIVE-VIDEO)
N5K(config-pmap-c-qos)# class SIGNALING
N5K(config-pmap-c-qos)# set qos-group 3
! Map signaling traffic to QoS group 3
N5K(config-pmap-c-qos)# class TRANSACTIONAL-DATA
N5K(config-pmap-c-qos)# set qos-group 2
! Map transactional data to QoS group 2
N5K(config-pmap-c-qos)# class class-fcoe
N5K(config-pmap-c-qos)# set qos-group 1
! Map FCoE to QoS group 1

The next step is to configure the network-qos class and policy maps. Because there are only four customizable QoS groups to work with, the network-qos class map must consolidate two of these classes. This is most easily done by consolidating the Interactive Video and Streaming Video classes into one class called simply Video

! Configure the network-qos class maps
N5K(config)# class-map type network-qos VOICE
N5K(config-cmap-nq)# match qos-group 5
! Matches traffic in QoS group 5
N5K(config)# class-map type network-qos VIDEO
N5K(config-cmap-nq)# match qos-group 4
! Matches traffic in QoS group 4
N5K(config)# class-map type network-qos SIGNALING
N5K(config-cmap-nq)# match qos-group 3
! Matches traffic in QoS group 3
! This is the consolidate class for the two video classes
N5K(config)# class-map type network-qos TRANSACTIONAL-DATA
N5K(config-cmap-nq)# match qos-group 2
! Matches traffic in QoS group 2
! Configure the network-qos policy map
N5K(config)# policy-map type network-qos 8-CLASS-NQ-POLICY
N5K(config-pmap-nq)# class type network-qos VOICE
! Activates the Voice network-qos class
N5K(config-pmap-nq-c)# class type network-qos SIGNALING
! Activates the Signaling network-qos class
N5K(config-pmap-nq-c)# class type network-qos VIDEO
! Activates the Video network-qos class
N5K(config-pmap-nq-c)# class type network-qos TRANSACTIONAL-DATA
! Activates the Transactional Data class
N5K(config-pmap-nq-c)# class type network-qos class-fcoe
N5K(config-pmap-nq-c)# pause no-drop
N5K(config-pmap-nq-c)# mtu 2158
! Activates the FCoE class and establishes the class as no-drop

The next step is to configure the queuing policies for each traffic class.

! Configure the queuing class maps
N5K(config-cmap-que)# class-map type queuing VOICE
N5K(config-cmap-que)# match qos-group 5
! Matches traffic in QoS group 5
N5K(config-cmap-que)# class-map type queuing VIDEO
N5K(config-cmap-que)# match qos-group 4
! Matches traffic in QoS group 4
N5K(config-cmap-que)# class-map type queuing SIGNALING
N5K(config-cmap-que)# match qos-group 3
! Matches traffic in QoS group 3
N5K(config-cmap-que)# class-map type queuing TRANSACTIONAL-DATA
N5K(config-cmap-que)# match qos-group 2
! Matches traffic in QoS group 2
! Configure the queuing policy map
N5K(config)# policy-map type queuing 8-CLASS-GLOBAL-QUEUING-POLICY
N5K(config-pmap-que)# class type queuing VOICE
N5K(config-pmap-c-que)# priority
! Gives voice the priority queue
N5K(config-pmap-c-que)# class type queuing SIGNALING
N5K(config-pmap-c-que)# bandwidth percent 10
! Signaling is given 10% of available BW
N5K(config-pmap-c-que)# class type queuing VIDEO
N5K(config-pmap-c-que)# bandwidth percent 25
! Video is given 25% of available BW
N5K(config-pmap-c-que)# class type queuing TRANSACTIONAL-DATA
N5K(config-pmap-c-que)# bandwidth percent 20
! Transactional Data is given 20% of available BW
N5K(config-pmap-c-que)# class type queuing class-fcoe
N5K(config-pmap-c-que)# bandwidth percent 20
! FCoE is given 20% of available BW
N5K(config-pmap-c-que)# class type queuing class-default
N5K(config-pmap-c-que)# bandwidth percent 25
! The default class is given 25% of available BW

The final configuration step is to attach these three policy maps to the system, as demonstrated in below Example this activates the policies for all interfaces on the Nexus 5500.

N5K(config)# system qos
N5K(config -sys-qos)# service-policy type qos input 8-CLASS-QOS-POLICY
! Attach the qos policy to the system class
N5K(config -sys-qos)# service-policy type queuing output 8-CLASS-GLOBAL-QUEUING-
POLICY
! Attach the queuing policy in the output direction
N5K(config -sys-qos)# service-policy type network-qos 8-CLASS-NQ-POLICY
! Attaches the network-qos policy to the system


Comment

    You are will be the first.

LEAVE A COMMENT

Please login here to comment.