So we have been working diligently to resolve an issue in relation to a vendor deployed Cisco Telepresence Show and Share setup. We ended up with through the troubleshooting process with two requirements. Basically what we were seeing was if the access port attached to the receiver was configured or negotiated at 100Mbps as opposed to Gigabit that the video stream would not play with any measure of predictability.
When everyone would go home or be out for lunch, the stream would play okay, but when the switch became even slightly busy, the video would most of the time not even provide a single key-frame and hence no video. Keep in mind that this was a 768kbps stream, so it was insane to think this could actually be the switch causing the issue.
With Auto-QoS defaults, we were dropping around 500 packets in 10 seconds. We actually forced this work by modifying marking the multicast traffic generated at the server on the server’s access switch and by modifying the output buffers for QoS on the client attached 2960S Access Switch. This required giving 70% of the buffer to the priority Queue, making sure CS5 marked traffic was getting inserted into the Priority Queue, and setting the thresholds for the priority queue to allow oversubsciption at 3200%.
auto-qos only
2960S-Switch# show mls qos interface GigabitEthernet1/0/40 statistics
GigabitEthernet1/0/40 (All statistics are in packets)
dscp: incoming
-------------------------------
...
dscp: outgoing
-------------------------------
0 - 4 : 671 0 0 0 0
.. .. ... ... ... ... ...
40 - 44 : 889 0 0 0 0
45 - 49 : 0 0 0 161 0
.. .. ... ... ... ... ...
cos: incoming
-------------------------------
...
cos: outgoing
-------------------------------
0 - 4 : 698 7 0 0 0
5 - 7 : 891 164 25
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 891
queue 1: 0 373 272
queue 2: 0 0 698
queue 3: 0 0 7
output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 485
queue 1: 0 0 0
queue 2: 0 0 9
queue 3: 0 0 0
...
mls qos
mls qos map cos-dscp 0 8 16 24 32 46 48 56
!
! #############################################################################
! ### QUEUE 1 - REAL TIME QUEUE
mls qos srr-queue output cos-map queue 1 threshold 3 4
! CS4
mls qos srr-queue output dscp-map queue 1 threshold 3 32 33
! -----------------------------------------------------------------------------
mls qos srr-queue output cos-map queue 1 threshold 3 5
! CS5 VA EF
mls qos srr-queue output dscp-map queue 1 threshold 3 40 41 42 43 44 45 46 47
! -----------------------------------------------------------------------------
! auto-qos: queue-set output 1 threshold 1 100 100 50 200
mls qos queue-set output 1 threshold 1 100 100 50 3200
!
! #############################################################################
! ### QUEUE 2 - CRITICAL QUEUE
mls qos srr-queue output cos-map queue 2 threshold 1 2
! CS2 AF21 AF22 AF23
mls qos srr-queue output dscp-map queue 2 threshold 1 16 17 18 19 20 21 22 23
! AF31 AF32 AF33
mls qos srr-queue output dscp-map queue 2 threshold 1 26 27 28 29 30 31
! AF41 AF42 AF43
mls qos srr-queue output dscp-map queue 2 threshold 1 34 35 36 37 38 39
! -----------------------------------------------------------------------------
mls qos srr-queue output cos-map queue 2 threshold 2 3
! CS3 - CALL SIGNALING
mls qos srr-queue output dscp-map queue 2 threshold 2 24
! -----------------------------------------------------------------------------
mls qos srr-queue output cos-map queue 2 threshold 3 6
! CS6 - INTERNETWORK CTRL
mls qos srr-queue output dscp-map queue 2 threshold 3 48 49 50 51 52 53 54 55
! -----------------------------------------------------------------------------
mls qos srr-queue output cos-map queue 2 threshold 3 7
! CS7 - NETWORK CONTROL
mls qos srr-queue output dscp-map queue 2 threshold 3 56 57 58 59 60 61 62 63
! -----------------------------------------------------------------------------
mls qos queue-set output 1 threshold 2 125 125 100 400
!
! #############################################################################
! ### QUEUE 3 - BEST EFFORT QUEUE
mls qos srr-queue output cos-map queue 3 threshold 3 0
mls qos srr-queue output dscp-map queue 3 threshold 3 0 1 2 3 4 5 6 7
! -----------------------------------------------------------------------------
mls qos queue-set output 1 threshold 3 100 100 100 400
!
! #############################################################################
! ### QUEUE 4 - BULK/SCAVENGER QUEUE
mls qos srr-queue output dscp-map queue 4 threshold 1 8 9 11 13 15
! -----------------------------------------------------------------------------
mls qos srr-queue output dscp-map queue 4 threshold 2 10 12 14
! -----------------------------------------------------------------------------
mls qos srr-queue output cos-map queue 4 threshold 3 1
! -----------------------------------------------------------------------------
mls qos queue-set output 2 threshold 4 60 150 50 200
!
! #############################################################################
! auto-qos: mls qos queue=set output 2 buffers 15 25 40 20
mls qos queue-set output 2 buffers 70 10 10 10
2960S-Switch# show mls qos interface GigabitEthernet1/0/40 statistics
GigabitEthernet1/0/40 (All statistics are in packets)
dscp: incoming
-------------------------------
...
dscp: outgoing
-------------------------------
0 - 4 : 952 0 0 0 0
.. .. ... ... ... ... ...
40 - 44 : 941 0 0 0 0
45 - 49 : 0 0 0 102 0
.. .. ... ... ... ... ...
cos: incoming
-------------------------------
...
cos: outgoing
-------------------------------
0 - 4 : 898 54 0 0 0
5 - 7 : 852 101 89
output queues enqueued:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 852
queue 1: 0 101 89
queue 2: 0 0 898
queue 3: 0 0 54
output queues dropped:
queue: threshold1 threshold2 threshold3
-----------------------------------------------
queue 0: 0 0 8
queue 1: 0 0 0
queue 2: 0 0 385
queue 3: 0 0 25
...
It seemed insane that we had to provide this level of buffer assignment on the priority queue to enable delivery of a 768kbps multicast stream. We grabbed a few packet captures and what they showed was that when you zoom in close enough, there was just too much traffic showing up at the receiver to make it down the interface without exceeding the buffers. For example, in the following picture you can see a graph of a key-frame (I Frame) just after 13:19:48.049 and 13:19:48.449. The scale is now horizontally 1 pixel = 1 millisecond of time which makes the vertical scale in Kbps. Keep in mind this is on a 100 Mbps link. You’ll quickly notice that somehow at an instance within the key-frame receipt the interface appears to be exceeding 100 Mbps.
This isn’t actually possible, but its very telling of a problem. Wireshark is showing this impossibility because there is a buffer on our NIC and depending on when WINSOCK is notified of the arriving packet, time can get skewed and mislead wireshark as to the actual time the packet arrived resulting in miscalculated interface peak in that time slot.
Back to the issue at hand. The problem is the NIC on the server is connected to a switch at Gigabit, while the NIC on the receiver is connected at 100 Mbps. The Windows Media Server has a buffer that it stores packets in before dropping it into the WINSOCK stack for delivery. If you are using the Unicast Data Sink, Microsoft provides the ability to not buffer the packet at the server before delivery. (page 17) This spreads out the packets generated by the steam considerably in time. This to my knowledge is not modifiable for multicast. Since multicast rides UDP and UDP doesn’t provide a packet recovery mechanism by itself, all of these packets show up at the same time and the queue is exceeded on the switchport resulting in packet loss.
However, Microsoft does have a (rather old) document with an insinuating reference in it. According to the Optimizing Windows Media Services Document provided by Microsoft, in order to provide Multicast Windows Media Services 9 recommends:
Use a 1 Gbps network adapter to maximize your server capacity, except in multicast scenarios.
Considering at the time the referenced document was written 10-Gigabit Ethernet wasn’t considerably available, we made the assumption they were expecting you to utilize 100BaseTX interfaces for multicast transmission. After weeks of trying to resolve this issue, configuring the switch the Windows Media Server was plugged into to 100/Full fixed it.
After speaking with Cisco TAC and reading various blogs and threads on CCO, We have come to the conclusion that the 2960S/3560 provides inadequate buffers in the ASICs. When attached to a 3750, 4500, or 6500 series Catalyst Switch this issue does not present itself at least until the WMS Multicast stream approaches 6 Mbps.
We also noticed, Microsoft has pulled support for MS-MSB in Windows Media Services on Server 2012. The entire framework has actually moved to IIS Smooth Streaming. So no more Multicast from WMS. This is a damned shame.
Notes/Links for Reference:
- Microsoft Windows Media Services Multicast Whitepaper for IPTV
- Optimizing Windows Media Services
- Enterprise Medianet Quality of Service Design 4.0—Overview
- How To Enable Type of Service (TOS) for Windows Media Services 9 Series
EnableToS-WMS.reg Windows Registry Editor Version 5.00 [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters] "DisableUserTOSSetting"=dword:00000000
C:\Windows\System32\Windows Media\Server\ServerNamespace.xml <document WMSNameSpaceVersion="2.0"> <node name="Publishing Points" opcode="create" > <node name="Channel2" opcode="create" > <node name="Broadcast Data Sink" opcode="create" > <node name="Object Store" opcode="create" > <node name="WMS Multicast Data Writer" opcode="create" > <node name="Properties" opcode="create" > <node name="SubCategory" opcode="create" type="string" value="Multicast" /> <node name="WMSSystemPlugin" opcode="create" type="int32" value="0x1" /> <node name="MMCMoniker" opcode="create" type="string" value="CLSID:{33C065A0-E698-46ae-A135-D307295342B9},CLSID:{7387E84F-B98D-499d-91DA-72211F778F4D}" /> <node name="ASPMoniker" opcode="create" type="string" value="MulticastSink.asp" /> <node name="Multicast TTL" opcode="create" type="int32" value="0x20" /> <node name="Local IP Address" opcode="create" type="string" value="10.19.1.23" /> <node name="Multicast Group" opcode="create" type="string" value="232.50.49.2" /> <node name="Protocols" opcode="create" type="string" value="MSB" /> <node name="Multicast Port" opcode="create" type="int32" value="0x7f71" /> <node name="UseMulticastScope" opcode="create" type="int32" value="0xffffffff" /> <node name="Multicast Scope" opcode="create" type="string" value="" /> <node name="Lease Request ID" opcode="create" type="string" value="" /> <node name="Lease Start Time" opcode="create" type="int64" value="0x0" /> <node name="Lease Stop Time" opcode="create" type="int64" value="0x0" /> <node name="Multicast Lease Server" opcode="create" type="string" value="" /> <node name="Logging URL" opcode="create" type="string" value="" /> <node name="Custom Unicast Rollover URL" opcode="create" type="string" value="mms://tandberg-16d163.cisco.com/TCSmulticastTemplate" /> <node name="Unicast Rollover URL Options" opcode="create" type="int32" value="0x1" /> <node name="PacketSize" opcode="create" type="int32" value="0xffe3" /> </node> <!-- Properties --> </node> <!-- WMS Multicast Data Writer --> </node> </node> <!-- Broadcast Data Sink --> <node name="Limits" opcode="create" > <node name="FECPacketSpan" opcode="create" type="int32" value="0x3" /> </node> <!-- Limits --> <node name="Enable QOS" opcode="create" type="boolean" value="true" /> <!-- #### Sets QoS IPP=5 (DSCP=CS5) Flag ### --> <node name="QOS Setting" opcode="create" type="int32" value="0x40" /> <node name="Buffer Setting" opcode="create" type="int32" value="0x1" /> <node name="Allow Player Side Disk Caching" opcode="create" type="boolean" value="true" /> <node name="Enable FEC" opcode="create" type="boolean" value="true" /> </node> <!-- Channel2 --> <node name="Channel1" opcode="create" > <node name="Broadcast Data Sink" opcode="create" > <node name="Object Store" opcode="create" > <node name="WMS Multicast Data Writer" opcode="create" > <node name="Properties" opcode="create" > <node name="SubCategory" opcode="create" type="string" value="Multicast" /> <node name="WMSSystemPlugin" opcode="create" type="int32" value="0x1" /> <node name="MMCMoniker" opcode="create" type="string" value="CLSID:{33C065A0-E698-46ae-A135-D307295342B9},CLSID:{7387E84F-B98D-499d-91DA-72211F778F4D}" /> <node name="ASPMoniker" opcode="create" type="string" value="MulticastSink.asp" /> <node name="Multicast TTL" opcode="create" type="int32" value="0x20" /> <node name="Local IP Address" opcode="create" type="string" value="10.19.1.23" /> <node name="Multicast Group" opcode="create" type="string" value="232.50.49.101" /> <node name="Protocols" opcode="create" type="string" value="MSB" /> <node name="Multicast Port" opcode="create" type="int32" value="0x2deb" /> <node name="UseMulticastScope" opcode="create" type="int32" value="0xffffffff" /> <node name="Multicast Scope" opcode="create" type="string" value="" /> <node name="Lease Request ID" opcode="create" type="string" value="" /> <node name="Lease Start Time" opcode="create" type="int64" value="0x0" /> <node name="Lease Stop Time" opcode="create" type="int64" value="0x0" /> <node name="Multicast Lease Server" opcode="create" type="string" value="" /> <node name="Logging URL" opcode="create" type="string" value="" /> <node name="Custom Unicast Rollover URL" opcode="create" type="string" value="mms://tandberg-16d163.cisco.com/TCSmulticastTemplate" /> <node name="Unicast Rollover URL Options" opcode="create" type="int32" value="0x1" /> <!-- Default Window Packet Size 0xffe3 (65507 Bytes) --> <node name="PacketSize" opcode="create" type="int32" value="0xd0" /> </node> <!-- Properties --> </node> <!-- WMS Multicast Data Writer --> </node> <!-- Object Store --> </node> <!-- Broadcast Data Sink --> <node name="Limits" opcode="create" > <node name="FECPacketSpan" opcode="create" type="int32" value="0x3" /> </node> <!-- Limits --> <node name="Enable QOS" opcode="create" type="boolean" value="true" /> <!-- #### Sets QoS IPP=5 (DSCP=CS5) Flag ### --> <node name="QOS Setting" opcode="create" type="int32" value="0x40" /> <node name="Buffer Setting" opcode="create" type="int32" value="0x2" /> <node name="Allow Player Side Disk Caching" opcode="create" type="boolean" value="true" /> <node name="Enable FEC" opcode="create" type="boolean" value="true" /> </node> <!-- Channel1 --> </node> <!-- Publishing Points --> <node name="Unicast Data Sink" opcode="create" > <node name="Object Store" opcode="create" > <node name="WMS Unicast Data Writer" opcode="create" > <node name="Properties" opcode="create" > <node name="OutstandingBytesQuota" opcode="create" type="int32" value="0x10000" /> <!-- #### Try to push UDP packets every 100ms, default is 500ms #### --> <node name="UDPBurstLimitMs" opcode="create" type="int32" value="0x64" /> <!-- #### Adjusts TCP Packet Burstiness, default is 1000ms #### --> <node name="TCPBurstLimitMs" opcode="create" type="int32" value="0x0" /> <!-- #### Sets Type of Service Field in IPv4 Header to IPP=5 #### --> <node name="IP TOS" opcode="create" type="int32" value="0xa0" /> <!-- #### NetWritePacketTolerance: #### This value sets the tolerance between the system time and the network send time of the ASF packet. This value appears in 100-nanosecond units. By default, this value is 1000000 100-nanosecond units (100ms). --> <!-- #### NetWritePacketListSleepDelay: #### This value sets the delay that is subtracted from a packet when the Windows Media Services service determines that the network send time of the ASF packet will be outside the tolerance. This value appears in 100-nanosecond units. By default, this value is 250000 100-nanoseconds (25ms). This value _MUST_ be lower than NetWritePacketTolerance --> <!-- For further information see: https://support.microsoft.com/en-us/kb/828566 --> <node name="NetWritePacketTolerance" opcode="create" type="int32" value="0x186A0" /> <node name="NetWritePacketListSleepDelay" opcode="create" type="int32" value="0xC350" /> </node> <!-- Properties --> </node> <!-- WMS Unicast Data Writer --> </node> <!-- Object Store --> </node> <!-- Unicast Data Sink --> </document> - Thread on CCO – Ingess vs Egress Speed Difference and High Output Queue Drops
- CSCte64832 – FTP performance problem with 100Mbps connection
- Catalyst 3750 QoS Examples (Same architecture.. larger buffer in ASIC)
Note: 2960-S does not provide modifiable ingress queuing. - 2960 buffers (drops+packet loss due to micro-bursts)
- Microsoft KB 828566: Windows Media Services 9 Series network send behavior may lead to an unwanted client experience.
- Microsoft KB 940174: The video occasionally is rebuffered, or frames are dropped when you try to stream a video through a multicast from Windows Media Services to Windows Media Player clients.
protocolSyntax 
Reblogged this on 49514262200010.