Home > Cisco, Microsoft, Network > How To Configure Jumbo Frames

How To Configure Jumbo Frames

I have found there is a lot of confusion out there, as to how jumbo frames work, and in what circumstances they should be used.  Hopefully this post will be useful to others to avoid some of the pitfalls I have run into.

On your standard Ethernet network, the default frame size is 1500 bytes.  This can be notated in a number of different ways, however, since the actual amount of data sent is normally 1514 bytes which includes the source mac address (6 bytes), dest mac address (6 bytes), and protocol id (2 bytes), of the data in the frame (in most cases these days IP).  If you further include the 7 byte preamble, the 1 byte Start-of-Frame-Delimiter, and the 4 bytes of CRC32 at the end, you get 1526 (not to mention the 12 bytes of interframe-gap after transmission).  I have also seen it notated as 1518 bytes (1500 + 6 bytes src mac, 6 bytes dest mac, 2 bytes protocol id, and 4 bytes CRC 32).  When you run ifconfig on most operating systems it will show you 1500 bytes.  Within this “1500 byte frame” the usable space is further cut down to 1480 bytes by 20 bytes of IP header, and then if you are using TCP the data payload is cut down to 1460 bytes by another 20 bytes of TCP header.

Phew!  Was that enough numbers?  Now that we have that confusion out of the way, the question is why would you run something larger than 1500 bytes?  Well, back when the standard was set at 1500 byte frames we were not dealing with as large amounts of data as we are today.  Data transmission speeds were slower (i.e. 10 megabit) and the chances of encountering an error during the time it took to send a 1500 byte frame were much higher than sending a 1500 byte frame at gigabit speeds.  Keeping frames smaller meant you had to re-transmit less data when an error occurred.

As 100 megabit and then gigabit came out, the standards groups kept with the 1500 byte limit for backwards compatability.  This is great, except that when transferring large blocks of data (say during backups) this is less efficient than it could be since for every 1460 bytes of TCP payload data, you have 78 bytes worth of “time on the wire” that is used up.  You also force the computers at both ends of the connection to calculate a CRC-32 value for every 1460 bytes of data which is processor intensive (I am assuming you are not running a TOE), and also send ACK’s back etc…

So for certain applications you might want to consider running larger than 1500 byte mtu’s.  I recommend them only in the following circumstances:

  • You have an all gigabit network for the VLAN/network in question
  • All switches in your switching path can be configured to be “jumbo frame clean” up to the size frames you choose to use
  • You are sure all your host operating systems and NIC drivers will support Jumbo Frames at the size of frame you decide to use (this can vary a lot!) – note that I was once told that DataDomain boxes support jumbo frames, but they are not recommended because it causes a performance impact (this does not make sense to me, but it is what they said)
  • You have a real reason to need to run them (i.e. a backup VLAN or NFS VLAN)

Note that getting jumbo frames to work can be a real pain and in many cases is probably not worth it.  When you make the switch on a given subnet over to jumbo frames, you must simultaneously change *every* host on that network/VLAN or you will have weird problems where some hosts can talk to each other and others have difficulty for some types of connections.  TCP connections might work, but UDP connections fail intermittently (i.e. you can mount an nfs mount but when transferring files it fails).

As a side note, the reason that TCP will work on a MTU size mismatched network is that in the TCP 3 way handshake, a MSS (Maximum Segment Size) is specified by both ends of the connection and the smaller of the two is agreed upon.  Running different hosts on the same subnet with differing MTU sizes is *not* recommended.

So what size MTU should you choose?  Since there is no standard, different vendors have chosen to implement different maximum supported sizes.  I have seen drivers support all sorts of different mtu’s…  Some allow any mtu up to a max, while others support only several “canned” values (i.e. some Windows drivers).  From some research I have read, CRC-32 becomes ineffective above about 11,000 bytes.  I have found that every manufacturer I have used (that supports jumbo frames at all) will support 9000 byte MTU’s, therefore I highly recommend using 9000 byte MTU’s (this is also what is standardized on for Internet 2 traffic).

It is extremely important that your entire VLAN/network you are implementing this on be “Jumbo Frame Clean” for the size MTU you are using.  What I mean by this is that every switch in the path will allow at least 9000 byte mtu packets through.  Different switches have differing commands to make them pass jumbo frames.  In my Cisco 6500 series platforms I set the mtu on a per-port basis (depending on your Supervisor and OS version you might only have 9216 as an option which is fine as long as it is larger than your chosen mtu).  On my 3750’s (and similar stacking blade enclosure switches) I have to set it globally on the switch and reboot the switch to take effect.

Operating System Settings

To actually set jumbo frames in Windows you go to the device manager, select your NIC driver, go to the Advanced tab, and find the Jumbo frame setting.  You must either choose the correct MTU size from a drop-down list provided (in some drivers), or enter the correct frame size it into the box (if applicable).  You might need to verifiy with the manufacturer what number they are expecting here (does it include frame headers and footers?)

Here is an example of a NVIDIA NIC:

NVIDIA Jumbo Frame Settings in Windows

NVIDIA Jumbo Frame Settings in Windows

And here is an example of an Intel NIC.  Note that NVIDA lists it as 9000 bytes and Intel lists 9014, but my testing has shown these settings to be equivalent!  Also note that none of the other options align…  (except for 1500 bytes)

Intel Jumbo Frame Settings in Windows

Intel Jumbo Frame Settings in Windows

To set jumbo frames on Solaris, you must first figure out how to configure  your specific driver for jumbo frames, and then use ifconfig to set the interface to the specific frame size you decide on.

Testing

To test to make sure jumbo frames are working, you can use the following ping command from your Cisco switch (this is assuming that you have a layer 3 interface in the jumbo frame vlan and that L3 virtual interface is setup for your jumbo mtu size).

switch#show run int vlan 99               
Building configuration…

Current configuration : 190 bytes
!
interface Vlan99
 description backup
 mtu 9000
 ip address 10.10.9.1 255.255.255.0
 ip broadcast-address 10.10.9.255
end

switch#

switch#ping ip 10.10.9.19 size 9000 df-bit

Type escape sequence to abort.
Sending 5, 9000-byte ICMP Echos to 10.10.9.19, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
switch#

You can also generate jumbo frames from Solaris boxes, but Solaris does not seem to support setting the “DF” (do not fragment) bit which is a shame since you might never know that the machine you are on fragmented the packet before it sent it, hence not actually testing jumbo frames…

bash-3.00# ping 10.10.9.1 9000
10.10.9.1 is alive

Windows boxes can send jumbo frame pings as well, but you must subtract out the 20 bytes of IP header and 8 bytes of ICMP header that it inserts in the size you specify.  They *do* support setting the DF bit.

C:\>ping 10.10.9.1 -f -l 8972

Pinging 10.10.9.1 with 8972 bytes of data:

Reply from 10.10.9.1: bytes=8972 time=2ms TTL=255
Reply from 10.10.9.1: bytes=8972 time=2ms TTL=255
Reply from 10.10.9.1: bytes=8972 time=2ms TTL=255
Reply from 10.10.9.1: bytes=8972 time=2ms TTL=255

Ping statistics for 10.10.9.1:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 2ms, Maximum = 2ms, Average = 2ms

C:\>

There are a lot more details to everything we went through to implement jumbo frames on our backup networks, but this has hopefully given you the high level overview and a few key tips.

-Eric

Categories: Cisco, Microsoft, Network Tags:
  1. John
    July 2nd, 2009 at 21:58 | #1

    Terrific article. I you have almost the exact same environment as I do, Solaris and a Windows/NVidia NIC. I’ve connected my two machines directly because my switch does not support jumbo frames, and your jumbo ping commands show I have it working correctly.

    Now, for some reason, I am still getting only 50% utilization of the network when transferring files, and yes (everyone always asks) I have plenty of disk bandwidth as both machines are using 5-disk Raid5s.

    I’ve tried FTP, CIFS and Samba and my speeds are not what I would expect.

    Thanks!

  2. July 3rd, 2009 at 07:51 | #2

    Hmm, how are you measuring utilization? Are you using a graphing program that monitors interfaces via SNMP, or just calculating throughput based on file size and transmission time?

    I have yet to actually find a machine that can fully saturate a 1 gigabit network interface (I am sure they exist…). I wonder if it is running in to bus speed limitations between the NIC and the PCI controller or something…

    Note that of course there is framing and IP overhead in the transfers so you will never see 100% or the rated link speed, however, 50% seems quite low.

    -Eric

    P.S. Glad you found my post interesting! It was sure a frustration for quite some time…

  3. Ulf
    July 7th, 2009 at 12:10 | #3

    In a home network environment, what is your recommendation if I want to use parallel jumbo/non-jumbo subnets? What would be the best solution to have subnet A (jumbo) and B (nonjumbo) talk to each other? Obviously you need some sort of router, or L3-switch (expensive).

    I guess the best solution would be to get some jumbo-capable, dual-subnet-capable, router that also acts as an Internet router.

  4. Ulf
    July 7th, 2009 at 12:15 | #4

    .. furthermore: is it necessary to divide the subnets A and B using different VLANs, or does simply using different subnets suffice? Simply, can jumbo-capable switches send some packet 1 on subnet A using frame size 9000, followed by some packet 1 on subnet B using frame size 1500 on the SAME physical switch port?

  5. July 7th, 2009 at 13:50 | #5

    Hmm, I am not sure I would recommend jumbo frames in most home network environments. 😉

    You could just run *everything* in jumbo frame mode (assuming you could get all your devices to support it, printers might suck).

    You are correct though, you would need a router for sure if you wanted IP connectivity between the two subnets. Though in many cases, if you just want a jumbo frames subnet for NFS purposes or as a backup subnet, does it really need routed connectivity to the Internet if all the hosts attached to it are dual NIC’ed?

    Note that if you have mis-matched frame sizes on a subnet, TCP will still work fine since during the 3 way handshake there is an MSS (max segment size) negotiated that will default to the lower of the two devices in the conversation. UDP on the other hand will work fine till someone tries to send an over-sized frame and then it’s game over.

    -Eric

  6. July 7th, 2009 at 13:52 | #6

    Uhh, that is an interesting question. I have never tried it. 😉

    I am not sure of any OS’s that support such an overlay. I don’t think Cisco routers even do, and most hosts have a less robust IP stack than Cisco gear…

    From the switches standpoint I don’t think it cares. You are basically just telling it what the max size frame it will allow is before it treats it as an errored frame. As long as your switching path is “jumbo frame clean” up to the frame size you are using you should be good to go.

    -Eric

  7. Tony
    April 1st, 2010 at 15:18 | #7

    Im a little confused when you say, every device must be on jumbo frames or problems might arise.

    All my Macs are 2009-2010 and support Jumbo Frames
    I have a Smart Gigabit switch and another unmanaged gigabit switch

    My server is setup with aggregate link and is using jumbo frames.

    However, I do have a PS3 (wired) and a couple network printers (10/100) on the network.

    Am I correct in assuming that I cannot really use jumbo frames everywhere because of the PS3 and network printers that may not support it?

    Thanks in advance for the clarification.

    Tony

  8. April 1st, 2010 at 23:10 | #8

    Yes, all devices on the same subnet (network) that you will be using *MUST* have the same MTU setting or else connections will fail (potentially intermittently as small frames will make it through and larger ones will not).

    Unless you have a really compelling reason to use jumbo frames in a situation like that, I would recommend sticking to 1500 byte mtu. It is unclear to me exactly how much uplift the larger frame size provides with todays multi-core processors and TCP/IP offload engines in the network cards. I feel they are best relegated to dedicated backup networks and storage (iSCSI) networks.

    -Eric

  9. Slee
    April 6th, 2010 at 06:17 | #9

    If I have all my 10/100 network devices connected to a 10/100 router and all my gigabit devices connected to a single, jumbo capable switch, then I should be OK with jumbo frames?

    That is, will the router negotiate the frame size for the slower devices connected to it?

  10. April 6th, 2010 at 10:13 | #10

    No, there is no auto-negotiation whatsoever for frame sizes. The 10/100 switch (router) will likely just drop any frame over 1500 bytes that came into it from the switch that supported jumbo frames. 🙁

    You could create two separate networks (one jumbo frame enabled, and one not), but they would need separate IP subnets and then you may need routing between them depending on what you are trying to do.

    -Eric

  11. Bane
    July 21st, 2010 at 13:13 | #11

    @John
    Hi John,
    I have same situation, 50% utilization on Gbit/s link. Did you solved this? I tried everything but without sucess.

  12. Sycane
    August 3rd, 2010 at 02:07 | #12

    Interesting article. I’m currently putting together a Backup network for a mix of Solaris and Windows and came across your page while searching for Jumbo Frames and Cisco.

    I have a backup server with an aggregated link (4x 1Gb) to a Backup VLAN on the Cisco 3750. The Windows hosts will support MTU 9000 as will the Solaris 10 kit, without wanting to make physical changes to the Solaris 9 servers I still need access to the Backup VLAN to perform backup/restore operations. Can I create another VLAN on the 3750 and allow routing between the two VLANs to achieve this? I suspect not.. so how could this be architected?

    Thanks

  13. August 3rd, 2010 at 08:23 | #13

    Yes, you should be able to create another VLAN for the Solaris 9 hosts and route between them no problem. It will just not be 9000 byte MTU’s (only 1500 will be supported). You will probably need to set the layer 3 routing interface of the 3750 to be 9000 byte MTU in the Jumbo frame VLAN. Also, if your using TCP as a protocol it will auto-negotiate to 1500 byte MTU (through the MSS – Maximum Segment Size parameter). UDP applications (like NFS) that are configured to use jumbo frames could pose an issue forcing the 3750 to try to fragment them as they are routed between the two subnets.

    Note though with multi-homed servers in a bunch of VLAN’s traffic might not go in/out the interface you are planning on. If you have the Solaris 9 host in some primary subnet, plus also in this special 1500 byte backup VLAN and you are targeting your backup server in the 9000 byte backup VLAN, the Solaris 9 host will send traffic out whichever interface has the default gateway assigned to it (i.e. the primary interface) rather than the intended backup interface, for any traffic that is not destined for a directly connected subnet. Adding some custom route into the routing table can perhaps help you fix this.

    Also, I should point out to you that bonding 4x 1 gig interfaces to the backup server may not work out as intended as generally the hashing algorithms in use on either end of the bonded links will only allow a single flow to transit one of the four links (so the four interfaces do no good unless you have a bunch of different packet flows simultaneously and even then it is random which interface they hash to and not based on load).

    Good luck!

    -Eric

  14. Ola Andreas
    December 14th, 2010 at 08:39 | #14

    Hi!

    I strongly advice You to use the max setting for the switch when setting the jumboframe size.. often that is 9216. that way You can be sure that jumboframes from servers and simmilar do not get blocked/rejected.. gives a huge perf.drop.
    example try running 9000 size on the switch,, run jumboframes and iscsi from vmware and the jumbofremse from the servers will be ca 9007-9008size,, these will then not get trough the switch…. sometimes setting a switch to frame size 9000 actually is 9000and something, and all is well.
    sometimes on another switch 9000 will be 9000 exact, and You will have pain…

  15. Richard
    July 29th, 2011 at 00:41 | #15

    There is a way to force the DF bit on Solaris – use traceroute (with the -F flag) instead of ping:

    traceroute -F target pktsize

    I only just found this out myself…

    Cheers

    Richard

  16. January 6th, 2012 at 00:10 | #16

    hi,

    at our workplace we have just installed a second 3750g in a stack configuration and would like to investigate and test jumbo frames in a our server environment which consists of Windows, Linux, vCenter and our SAN running Storage Server 2003.
    Our 3750g’s are both running ip ip services image version 12.2 (55) SE4
    We have a vlan 50 and have tried the mtu 9000 and it is not being accepted, we suspect this doesnt work as jumbo frames cannot be enabled on a per vlan basis?

    any help much appreciated!

  17. January 6th, 2012 at 00:18 | #17

    If my memory serves me correctly, on the 3750 platforms you must set a global setting in order to enable jumbo frames, and then reboot the switch. It will allow jumbo frames on all VLAN’s (which does not hurt the VLAN’s you are just running 1500 byte frames on).

    I forget what the setting is off the top of my head.

    Good luck!

    Here is a show command off a blade enclosure chassis that is similar in design to the 3750’s:

    DVA01-SW1#show system mtu

    System MTU size is 1500 bytes
    System Jumbo MTU size is 9000 bytes
    System Alternate MTU size is 1500 bytes
    Routing MTU size is 1500 bytes
    DVA01-SW1#

    You can see I am capable of 9000 byte MTU’s.

    -Eric

  1. September 9th, 2009 at 08:02 | #1