Jumbo Frames Demystified

The Promise and Peril of Jumbo Frames

We sit at the intersection of two trends:

1. Most home networking gear, including routers, has safely transitioned to gigabit ethernet.
2. The generation, storage, and transmission of large high definition video files is becoming commonplace.

If that sounds like you, or someone you know, there's one tweak you should know about that can potentally improve your local network throughput quite a bit -- enabling Jumbo Frames.
The typical UDP packet looks something like this:

But the default size of that data payload was established years ago. In the context of gigabit ethernet and the amount of data we transfer today, it does seem a bit.. anemic.

The original 1,518-byte MTU for Ethernet was chosen because of the high error rates and low speed of communications. If a corrupted packet is sent, only 1,518 bytes must be re-sent to correct the error. However, each frame requires that the network hardware and software process it. If the frame size is increased, the same amount of data can be transferred with less effort. This reduces CPU utilization (mostly due to interrupt reduction) and increases throughput by allowing the system to concentrate on the data in the frames, instead of the frames around the data.

I use my beloved energy efficient home theater PC as an always-on media server, and I'm constantly transferring gigabytes of video, music, and photos to it. Let's try enabling jumbo frames for my little network.
The first thing you'll need to do is update your network hardware drivers to the latest versions. I learned this the hard way, but if you want to play with advanced networking features like Jumbo Frames, you need the latest and greatest network hardware drivers. What was included with the OS is unlikely to cut it. Check on the network chipset manufacturer's website.
Once you've got those drivers up to date, look for the Jumbo Frames setting in the advanced properties of the network card. Here's what it looks like on two different ethernet chipsets:
 
That's my computer, and the HTPC, respectively. I was a little disturbed to notice that neither driver recognizes exactly the same data payload size. It's named "Jumbo Frame" with 2KB - 9KB settings in 1KB increments on the Realtek, and "Jumbo Packet" with 4088 or 9014 settings on the Marvell. I know that technically, for jumbo frames to work, all the networking devices on the subnet have to agree on the data payload size. I couldn't tell quite what to do, so I set them as you see above.
(I didn't change anything on my router / switch, which at the moment is the D-Link DGL-4500; note that most gigabit switches support jumbo frames, but you should always verify with the manufacturer's website to be sure.)
I then ran a few tests to see if there was any difference. I started with a simple file copy.
Default network settings

Jumbo Frames enabled

My file copy went from 47.6 MB/sec to 60.0 MB/sec. Not too shabby! But this is a very ad hoc sort of testing. Let's see what the PassMark Network Benchmark has to say.
Default network settings

Jumbo Frames enabled

This confirms what I saw with the file copy. With jumbo frames enabled, we go from 390,638 kilobits/sec to 477,927 kilobits/sec average. A solid 20% improvement.
Now, jumbo frames aren't a silver bullet. There's a reason jumbo frames are never enabled by default: some networking equipment can't deal with the non-standard frame sizes. Like all deviations from default settings, it is absolutely possible to make your networking worse by enabling jumbo frames, so proceed with caution. This SmallNetBuilder article outlines some of the pitfalls:

1) For a large frame to be transmitted intact from end to end, every component on the path must support that frame size. The switch(es), router(s), and NIC(s) from one end to the other must all support the same size of jumbo frame transmission for a successful jumbo frame communication session.
2) Switches that don't support jumbo frames will drop jumbo frames.
In the event that both ends agree to jumbo frame transmission, there still needs to be end-to-end support for jumbo frames, meaning all the switches and routers must be jumbo frame enabled. At Layer 2, not all gigabit switches support jumbo frames. Those that do will forward the jumbo frames. Those that don't will drop the frames.
3) For a jumbo packet to pass through a router, both the ingress and egress interfaces must support the larger packet size. Otherwise, the packets will be dropped or fragmented.
If the size of the data payload can't be negotiated (this is known as PMTUD, packet MTU discovery) due to firewalls, the data will be dropped with no warning, or "blackholed". And if the MTU isn't supported, the data will have to be fragmented to a supported size and retransmitted, reducing throughput.

In addition to these issues, large packets can also hurt latency for gaming and voice-over-IP applications. Bigger isn't always better.
Still, if you regularly transfer large files, jumbo frames are definitely worth looking into. My tests showed a solid 20% gain in throughput, and for the type of activity on my little network, I can't think of any downside.


 Comments Worth Noting:

i have been building networks for broadcasters for over a decade - who always wanted bigger / faster / more type networks.
jumbo frames are great in theory, but the pain level can be very high.
A core network switch can be brought to its knees when 9 Kbyte frames have to be fragmented to run out a lower MTU interface.
Many devices dont implement PMTU correctly, or just ignore responses - video codecs seem particularly prone to this.
and wasnt there a discussion a few newsletters ago about dont try to optimise things too much? If you need 20% more network performance, but you are only operating at maybe 40% load, then you need a faster machine or a better NIC card.
And there have been something like 5 definitions of jumbo just in the cisco product line. Also telecomms manufacturers idea of jumbo often have frames with 4 Kbytes, not 9 Kbytes.....
And just to set the record straight - the reason for the 1514 bytes frame limit in GigE and 10G ethernet is backward compatibility.
Just about every network has some 10/100 (or 10 only) equipment still, and the 1514 limit has been built into other standards such as 802.11 wireless LAN.
the old saying is that God would have struggled to make the world in 7 days if he started with an installed base...

------------------------------------------------------------------------

Just a couple things to point out.
File transfer is typically done using TCP, not UDP. TCP has more overhead than UDP.
I'm curious why we see a sawtooth pattern in the un-jumbo framed graph. Is that TCP Vegas doing its thing?
I'm glad you've gone ahead and tried this out. Jumbo frames wouldn't exist if they didn't have a purpose, but with all the different kinds of traffic I think 1500 MTU is a good choice.
One with jumbo frames that you touched on, but didn't adequately explain, is that most consumer switches use the store-and-forward method of switching packets. This means that your switch must receive the whole packet before it can send it along, it can't be doing anything else because packets can't be multiplexed. This can cause unacceptable latency (you have 2 computers, not a big deal, but between several machines all trying to send data, you can end up with some seriously delayed packets).
I just would have liked to see more reasons not to do this that it's not a supported standard and doesn't work with a lot of hardware. There are other reasons this has not become the default.

----------------------------------------------------------------------

@Bob from what I have seen IPv6 is potentially a bigger problem than IPv4, because where an IPv4 router may see that the packet is too large and fragment it, IPv6 leaves it to the end devices.

---------------------------------------------------------------------------------


Jumbo frames are great. I work on VMware ESX networking, and I will point out what may not be obvious to everyone. In a virtualized environment (hosted or hypervisor) jumbo frames make an even bigger difference, since you are doing more work per packet to begin with. That's why we added jumbo frame support since ESX 3.5 shipped.
My experience is that any recent machine can easily push full 1Gbit line rate (on native, and for that matter ESX VMs). Setting Jumbo Frames will save you on CPU though, which will allow you to run more VMs or otherwise use that power. And while Jumbo Frames are nice- they get you from 1.5k packets to 9, TCP Segmentation Offloading (TSO) is much better, since you push down entire 64k (or sometimes up to 256k) packets, and an engine on the NIC itself automatically handles dicing them into 1.5K packets. Most good nics support this- Intel, Broadcom, etc. On the other side, the reverse is LRO, or RSS, but this is more complicated and less common. Plus with TSO, you don't have to worry about MTU.
The other thing I would mention is- for the love of god, don't run networking benchmarks by doing a file copy. With 1GBit networks, you are limited by your disk speed! Run a proper tool such as iperf (brain dead simple) or netperf, which just blasts data. Even if your hard drive could reach 1Gbit speeds, you would be wasting cycles, so your networking performance would be worse. You always want to look at these things in isolation.

--------------------------------------------------------------------------------------------

The reason that all these people are seeing performance improvements using Jumbo Frame on Windows is because Windows networking stack sucks. Windows is really stupid and often will not let a single tcp stream reach the full capacity of the NIC. I.e. you run 1 TCP stream and measure 400Mbits, but if you ran 3 in parallel you would hit 940Mbits (~Line rate). This is even more annoying with 10G, since you need like 18 streams to reach the peak performance. Linux doesn't have these problems, and will give you its best possible performance on a single stream. I can only imagine Window's behavior is the result of some misguided attempt at ensuring fairness between connections by making sure that even if there is only one connection, it never uses the full capacity.

--------------------------------------------------------------------------------------------

If you simply enable jumbo frames on your NIC, every connection to any Internet destination (which don't support jumbos) will need to undergo PMTU discovery, PMTU blackhole detection, router fragmentation, or other time-consuming / performance-sapping hacks. This might explain why people complain about latency issues with gaming. These people are also seeing slightly slower peformance with all Internet activity.
*nix, as/400/, mainframes, and other operating systems let you set the frame size on a per route basis. E.g.,
route add -net 0.0.0.0 gw 192.168.0.1 mss 1460
This tells the OS to use jumbo frames only on the local LAN, and to assume a normal packet size everywhere else.
Alas, Windows has no such ability. One solution on Windows is to use two NICs attached to the same network. Have one NIC configured with normal frames and the default route. Have the second NIC configured for jumbos with no default route.

---------------------------------------------------------------------------------------

I participated in the IEEE 802.3 committee for a while. IEEE never standardized a larger frame size for two reasons that I know of:
1. The end stations can negotiate the frame size, but there was no backwards-compatible way to ensure that all L2 bridges between them can handle it. Even if you send a jumbo frame successfully, you can still run into a problem later if the network topology changes and your packets begin taking a different path through the network.
2. The CRC32 at the end of the packet becomes weaker after around 4 KBytes of data. It can no longer guarantee that single bit errors will be caught, and the multibit error detection becomes weaker as well.
One is free to enable it, and it does improve the performance, but the situation is unlikely to ever get better in terms of standard interoperability. It will always be an option to be enabled manually.
Also a number of years ago,. jumbo frames provided a much bigger boost. Going from 1.5K to 9K regularly doubled performance or more. What has happened since is smarter ethernet NICs: they routinely coalesce interrupts, steer packets from the same flow to the same CPU, and sometimes even reassemble the payload of the 1.5K frames back into larger units. The resistance to standardizing jumbo frames resulted in increased innovation elsewhere to compensate.

-----------------------------------------------------------------------------

@Timothy Layer 2 ethernet switches will just drop packets they cannot handle. It is not just if they don't handle jumbo frames: they can drop a normal size packet if their internal queues are full, or if rate limiting has been configured, or if the switch hit some other internal condition which the ASIC designer didn't bother resolving. They just drop the packet and expect the sender to retransmit. There is no mechanism for an L2 device to send back a notification that it has dropped the packet. A managed L2 switch will have some counters so you can post-mortem analyze what is wrong with your network.
Layer 3 routers will drop packets for more reasons, in addition to queue congestion. For example when the packet is too big and the don't fragment bit is set, an ICMP message is sent back (this is how path MTU discovery works). Similarly routers send back ICMP messages if they have no route to the destination.
Even the ICMP is just a best effort notification. Routers routinely limit the rate of ICMP messages they will send, to avoid having a flood of ICMP messages make a bad network problem worse. ICMP messages can also be dropped on their way back to the originator. So the best the sender can expect is it _might_ get notified of an L3 problem, sometime.

--------------------------------------------------------------------------


Disclamier: Credit to original Poster
http://www.codinghorror.com/blog/2009/03/the-promise-and-peril-of-jumbo-frames.html

Configuring LLQ and LFI on a Router

LFI

Link fragmentation and interleaving, why do we need it cisco recommends that for links equal to or lower than 768 kbps needs fragmentation and interleaving for voice packets, links slower than 768 kbps consume higher serialization delay when putting a voice packet, which is inherently large, on the wire. this is recommended for higher sped links.


#access-list 102 permit udp any nay range 16384 32767
#access-list 103 permit tcp any eq 1720 any
#access-list 103 permit tcp any any eq 1720

#class-map match-all VOICE-SIGNALING
#match access-group 103
#class-map match-all VOICE-TRAFFIC
#match access-group 102

#policy-map VOICE-POLICY
#class VOICE-TRAFFIC
#priority 48
#class VOICE-SIGNALING
#bandwidth 8
#class class-default
#fair-queue

#interface multilink1
#ip address 172.22.130.1 255.255.255.252
#ip tcp header-compression iphc-format
#ip rtp header-compression iphc-format

#serivce policy output VOICE-POLICY

#ppp multilink
#ppp multilink fragment-delay
#ppp multilink interleave
#multilink-group 1

#interface serial 0/0
#multilink-group 1

Config description,



#access-list 102 permit udp any nay range 16384 32767
#access-list 103 permit tcp any eq 1720 any
#access-list 103 permit tcp any any eq 1720

in the first statement 102 we are matching UDP ports 16384 to 32767  which are responsible for carrying voice RTP payload.
in the second & third statement we are matching TCP voice signaling port 1720

#class-map match-all VOICE-SIGNALING
#match access-group 103
#class-map match-all VOICE-TRAFFIC
#match access-group 102


Relevant class maps, against configured ACL's


#policy-map VOICE-POLICY
#class VOICE-TRAFFIC
#priority 48
#class VOICE-SIGNALING
#bandwidth 8
#class class-default
#fair-queue

Policy maps are configured with relevant policies, here we have configured 48 kbps of priority bandwidth for voice traffic which is merely a single call so you should configure it according to your situation, one thing very important here the VOICE-TRAFFIC will get 48 kbps of priority bandwidth traffic exceding that will be policed, on the contrary the bandwidth command reserves minimum 8 kbps of bandwidth for VOICE-SIGNALING and it can exceed its threshold of 8 kbps whereas priority cant.

Every thing else will be treated by Weighted fair queuing (fair-queue) which penalizes high talkers i.e. sessions consuming higher bandwidth as compared to low talkers sessions consuming lower bandwidth



#interface multilink1
#ip address 172.22.130.1 255.255.255.252
#ip tcp header-compression iphc-format
#ip rtp header-compression iphc-format

We have to create interface multilink1 to enable LFI.
Then the header compressions will compress the headers accordingly.

#serivce policy output VOICE-POLICY

applies the LLQ to this interface, to be used along with LFI.


#ppp multilink
#ppp multilink fragment-delay 10
#ppp multilink interleave
#multilink-group 1

#interface serial 0/0
#multilink-group 1

the first statement enables multilink.

#ppp multilink fragment-delay 10, statement enables delay no more than 10 ms, on a 56 k link a 1500 Bytes packet takes 215 mSec to be put on the wire which is to much, we need delay something b/w 150-200 msec so fragmentation will definately help.

#ppp multilink interleave, will enable interleaving so the packets that were chip choped by fragmentation are treated voice packets are sent first always.

Then the interleaving and fragmentation is applied on the interface

Configuring LLQ Quality of service

Configuring LLQ on a switch access port.


#interface fastethernet 0/1
#wrr-queue cos-map 1 0 1
#wrr-queue cos-map 2 2 3
#wrr-queue cos-map 3 4 6 7
#wrr-queue cos-map 4 5
#priority-queue out

#mls qos trust device cisco-phone
#mls cos trust cos
#switchport voice vlan 100
#switchport access vlan 10

#switchport priority extend cos 0
#mls qos map cos-dscp 0 8 16 24 34 46 48 56
#mls qos bandwidth 10 20 70 0(0 or 1)

Description and usage of each command is as follows


In this Configuration im going to configure low latency queuing on a cisco switch, applicable to access port connecting to the IP phone daisy chained to a PC.


#interface fastethernet 0/1
#wrr-queue cos-map 1 0 1
#wrr-queue cos-map 2 2 3
#wrr-queue cos-map 3 4 6 7
#wrr-queue cos-map 4 5

In the above config we have configured four queues, 1 0 1, first digit is the queue number and preceding 0 and 1 are cos values reserved for this queue similarly queue 2 3 4 are configured, queue 4 is the main queue for which cos value 5 is reserved, IP phone marks all its traffic with cos value 5  


#priority-queue out

very important command this sets queue 4 to be prioritized in case of bottleneck, PQ traffic is out first.

#mls qos trust device cisco-phone
#mls cos trust cos

trust the cos vlaue only if cisco IP phone is attached, which is not good if you have phones from other vendors,switches utilize CDP to detect if their is a cisco IP phone attached.

#switchport voice vlan 100
#switchport access vlan 10

#switchport priority extend cos 0

Mark any packet from PC with cos 0

#mls qos map cos-dscp 0 8 16 24 34 46 48 56

Switch will mark the DSCP of packets according to the above map, if a layer 3 device encounters the packet and is taking decisions based on layer 3 DSCP markings. the values are mapped against 0 1 2 3 4 5 6 7 cos markings.

#mls qos bandwidth 10 20 70 0(0 or 1)

Sets the bandwidth for each queue, remember queue 4 is PQ so it will be serviced first and prioritized so you can set a 0 or 1 for this queue,0 preferable.

Configuring Cisco Extension Mobility in CUCM 7

Configuration Checklist for Cisco Extension Mobility 

https://[CUCM_IP]/Help/en_US/ccm/wwhelp/wwhimpl/common/html/frameset.htm

Perform the procedures in the order shown in Table 8-1 to configure Cisco Extension Mobility.

Summary steps in Table 8-1 point out the major tasks that are required to configure Cisco Extension Mobility in Cisco Unified Communications Manager Administration. For a complete set of instructions, be sure to follow the procedure that is listed in the Related Procedures and Topics. 

Table 8-1 Configuration Checklist for Cisco Extension Mobility
	Configuration Steps	Related Procedures and Topics
Step 1:	Using Cisco Unified Serviceability, choose Tools > Service Activation to activate the Cisco Extension Mobility service. Note : To disable the extension mobility service on any node, you must first deactivate the service for that node in Service Activation. Note : When a change in activation or deactivation of the Cisco Extension Mobility service occurs, on any node, the database tables get updated with information that is required to build the service URLs. The database tables also get updated when the extension mobility service parameters get modified. The EMApp service handles the change notification.	For information on service activation, refer to the Cisco Unified Serviceability Administration Guide.
Step 2:	Create the Cisco Extension Mobility Service. Summary steps include Choose Device > Device Settings > Phone Services. Enter the service name (such as, Extension Mobility Service or EM). Enter the following URL: http://:8080/emapp/ EMAppServlet?device=#DEVICENAME# Note : If you should enter the URL incorrectly and subscribe the wrong service to the phones, you can correct the URL, save it, and press Update Subscriptions or correct the URL and resubscribe each phone to which the wrong service was subscribed. Select values for Service Category and Service Type. Enter a value for Service Vendor (Java MIDlet services only). (wrong). Select XML. Click Save. Enable Check Box(Must) Note : For Java MIDlet services, the service name and service vendor must exactly match the values that are defined in the Java Application Descriptor (JAD) file.	Adding the Cisco Extension Mobility Service.
Step 3:	Configure administration parameters.	Setting the Service Parameters
Step 4:	Create a default device profile for each phone type that you want to support Cisco Extension Mobility.	Creating a Default Device Profile for Each Cisco Unified IP Phone Type
Step 5:	Create the user device profile for a user. Summary steps include Choose Device > Device Settings >Device Profile and click Add New. Enter the Device Type. Enter the Device Profile Name, choose the phone button template, and click Save. Enter the directory numbers (DNs) and required information and click Save. Repeat for all DNs. To enable intercom lines for this device profile, configure intercom directory numbers (DNs) for this device profile. You configure an intercom DN in the Intercom Directory Number Configuration window, which you can also access by choosing Call Routing > Intercom > Intercom Directory Number. You must designate a Default Activated Device in the Intercom Directory Number Settings pane for an intercom DN to be active.	Creating the Device Profile for a User Intercom Directory Number Configuration, Cisco Unified Communications Manager Administration Guide
Step 6:	Associate a user device profile to a user. Summary steps include Choose User Management > End User and click Add New; enter user information. In Available Profiles, choose the service that you created in Step 2 and click the down arrow; this places the service that you chose in the Controlled Profiles box. Click Save.	Associating a User Device Profile to a User
Step 7:	Configure and subscribe Cisco Unified IP Phone and user device profile to Cisco Extension Mobility. Summary steps include Subscribe the phone and the user device profile to Cisco Extension Mobility. Choose Device > Phone and click Add New. On the Phone Configuration window, in Extension Information, check Enable Extension Mobility. In the Log Out Profile drop-down list box, choose Use Current Device Settings or a specific configured profile and click Save. To subscribe Cisco Extension Mobility to the Cisco Unified IP Phone, go to the Related Links drop-down list box in the upper, right corner of the window and choose Subscribe/Unsubscribe Services; then, click Go.	Cisco Unified IP Phone Configuration, Cisco Unified Communications Manager Administration Guide Subscribing Cisco Unified IP Phones to Cisco Extension Mobility Finding an Actively Logged-In Device, Cisco Unified Communications Manager Administration Guide

No magic file found in flash -- Booting in ROMMON mode

If you have a router that goes directly in ROMMON mode and is returned with an error "No magic file found in flash"

Couple of solutions,

If you get a message saying "no magic number found" then chances are your EEPROM has gotten corrupted. You probably will have to replace the box or call Cisco TAC. This is from Cisco's CCO:

"No magic number found in RP1 EEPROM
Explanation The structure containing the system MAC address needed by the OIR facility cannot be found in the RP EEPROM. The RP must be returned for reprogramming.

Recommended Action Return the RP for reprogramming and call your technical support representative for assistance. "


You can try the following methods to recover your box.

(1)Check the confreg value and make sure it correct.
show config
Configuration register is 0x2102.

(2)Try booting from ROMMON mode only
u can try following commands
IP_ADDRESS=1.1.1.1
IP_SUBNET_MASK=255.0.0.0
GATWAY=1.1.1.2 (UR MACHINES IP ADDRESS)
TFTP_SERVER=1.1.1.2
TFTP_FILE= bootable image file
tftpdnld

(3) XModem
It can be done using Xmodem through the console connection. It takes a LONG time compared to other methods, but you can speed it up a little bit by changing the baud rate on the console from 9600 to 19200 or whatever is the fastest. This can be done in ROMMON. Don't forget to also change the baud on your serial connection on your laptop. Some of the particulars of how to do this depend on what program you use--hyperterm, teraterm, etc.