I have found there is a lot of confusion out there, as to how jumbo frames work, and in what circumstances they should be used. Hopefully this post will be useful to others to avoid some of the pitfalls I have run into.
On your standard Ethernet network, the default frame size is 1500 bytes. This can be notated in a number of different ways, however, since the actual amount of data sent is normally 1514 bytes which includes the source mac address (6 bytes), dest mac address (6 bytes), and protocol id (2 bytes), of the data in the frame (in most cases these days IP). If you further include the 7 byte preamble, the 1 byte Start-of-Frame-Delimiter, and the 4 bytes of CRC32 at the end, you get 1526 (not to mention the 12 bytes of interframe-gap after transmission). I have also seen it notated as 1518 bytes (1500 + 6 bytes src mac, 6 bytes dest mac, 2 bytes protocol id, and 4 bytes CRC 32). When you run ifconfig on most operating systems it will show you 1500 bytes. Within this “1500 byte frame” the usable space is further cut down to 1480 bytes by 20 bytes of IP header, and then if you are using TCP the data payload is cut down to 1460 bytes by another 20 bytes of TCP header.
Phew! Was that enough numbers? Now that we have that confusion out of the way, the question is why would you run something larger than 1500 bytes? Well, back when the standard was set at 1500 byte frames we were not dealing with as large amounts of data as we are today. Data transmission speeds were slower (i.e. 10 megabit) and the chances of encountering an error during the time it took to send a 1500 byte frame were much higher than sending a 1500 byte frame at gigabit speeds. Keeping frames smaller meant you had to re-transmit less data when an error occurred.
As 100 megabit and then gigabit came out, the standards groups kept with the 1500 byte limit for backwards compatability. This is great, except that when transferring large blocks of data (say during backups) this is less efficient than it could be since for every 1460 bytes of TCP payload data, you have 78 bytes worth of “time on the wire” that is used up. You also force the computers at both ends of the connection to calculate a CRC-32 value for every 1460 bytes of data which is processor intensive (I am assuming you are not running a TOE), and also send ACK’s back etc…
So for certain applications you might want to consider running larger than 1500 byte mtu’s. I recommend them only in the following circumstances:
- You have an all gigabit network for the VLAN/network in question
- All switches in your switching path can be configured to be “jumbo frame clean” up to the size frames you choose to use
- You are sure all your host operating systems and NIC drivers will support Jumbo Frames at the size of frame you decide to use (this can vary a lot!) – note that I was once told that DataDomain boxes support jumbo frames, but they are not recommended because it causes a performance impact (this does not make sense to me, but it is what they said)
- You have a real reason to need to run them (i.e. a backup VLAN or NFS VLAN)
Note that getting jumbo frames to work can be a real pain and in many cases is probably not worth it. When you make the switch on a given subnet over to jumbo frames, you must simultaneously change *every* host on that network/VLAN or you will have weird problems where some hosts can talk to each other and others have difficulty for some types of connections. TCP connections might work, but UDP connections fail intermittently (i.e. you can mount an nfs mount but when transferring files it fails).
As a side note, the reason that TCP will work on a MTU size mismatched network is that in the TCP 3 way handshake, a MSS (Maximum Segment Size) is specified by both ends of the connection and the smaller of the two is agreed upon. Running different hosts on the same subnet with differing MTU sizes is *not* recommended.
So what size MTU should you choose? Since there is no standard, different vendors have chosen to implement different maximum supported sizes. I have seen drivers support all sorts of different mtu’s… Some allow any mtu up to a max, while others support only several “canned” values (i.e. some Windows drivers). From some research I have read, CRC-32 becomes ineffective above about 11,000 bytes. I have found that every manufacturer I have used (that supports jumbo frames at all) will support 9000 byte MTU’s, therefore I highly recommend using 9000 byte MTU’s (this is also what is standardized on for Internet 2 traffic).
It is extremely important that your entire VLAN/network you are implementing this on be “Jumbo Frame Clean” for the size MTU you are using. What I mean by this is that every switch in the path will allow at least 9000 byte mtu packets through. Different switches have differing commands to make them pass jumbo frames. In my Cisco 6500 series platforms I set the mtu on a per-port basis (depending on your Supervisor and OS version you might only have 9216 as an option which is fine as long as it is larger than your chosen mtu). On my 3750’s (and similar stacking blade enclosure switches) I have to set it globally on the switch and reboot the switch to take effect.
Operating System Settings
To actually set jumbo frames in Windows you go to the device manager, select your NIC driver, go to the Advanced tab, and find the Jumbo frame setting. You must either choose the correct MTU size from a drop-down list provided (in some drivers), or enter the correct frame size it into the box (if applicable). You might need to verifiy with the manufacturer what number they are expecting here (does it include frame headers and footers?)
Here is an example of a NVIDIA NIC:
NVIDIA Jumbo Frame Settings in Windows
And here is an example of an Intel NIC. Note that NVIDA lists it as 9000 bytes and Intel lists 9014, but my testing has shown these settings to be equivalent! Also note that none of the other options align… (except for 1500 bytes)
Intel Jumbo Frame Settings in Windows
To set jumbo frames on Solaris, you must first figure out how to configure your specific driver for jumbo frames, and then use ifconfig to set the interface to the specific frame size you decide on.
Testing
To test to make sure jumbo frames are working, you can use the following ping command from your Cisco switch (this is assuming that you have a layer 3 interface in the jumbo frame vlan and that L3 virtual interface is setup for your jumbo mtu size).
switch#show run int vlan 99
Building configuration…
Current configuration : 190 bytes
!
interface Vlan99
description backup
mtu 9000
ip address 10.10.9.1 255.255.255.0
ip broadcast-address 10.10.9.255
end
switch#
switch#ping ip 10.10.9.19 size 9000 df-bit
Type escape sequence to abort.
Sending 5, 9000-byte ICMP Echos to 10.10.9.19, timeout is 2 seconds:
Packet sent with the DF bit set
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms
switch#
You can also generate jumbo frames from Solaris boxes, but Solaris does not seem to support setting the “DF” (do not fragment) bit which is a shame since you might never know that the machine you are on fragmented the packet before it sent it, hence not actually testing jumbo frames…
bash-3.00# ping 10.10.9.1 9000
10.10.9.1 is alive
Windows boxes can send jumbo frame pings as well, but you must subtract out the 20 bytes of IP header and 8 bytes of ICMP header that it inserts in the size you specify. They *do* support setting the DF bit.
C:\>ping 10.10.9.1 -f -l 8972
Pinging 10.10.9.1 with 8972 bytes of data:
Reply from 10.10.9.1: bytes=8972 time=2ms TTL=255
Reply from 10.10.9.1: bytes=8972 time=2ms TTL=255
Reply from 10.10.9.1: bytes=8972 time=2ms TTL=255
Reply from 10.10.9.1: bytes=8972 time=2ms TTL=255
Ping statistics for 10.10.9.1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 2ms, Maximum = 2ms, Average = 2ms
C:\>
There are a lot more details to everything we went through to implement jumbo frames on our backup networks, but this has hopefully given you the high level overview and a few key tips.
-Eric
