Archive for March, 2009

Cisco PIX/ASA ASDM Troubles with Java

March 31st, 2009 No comments

Over the past few months I have been annoyed by ASDM bug CSCsv12681 which causes pretty much all versions of ASDM to fail on any machine that gets upgraded to a recent Java version:

Error while loading ASDM: “Unconnected sockets not implemented”

While loading ASDM, a dialog is displayed that says:
“ASDM cannot be loaded. Click OK to exit ASDM.
Unconnected sockets not implemented”

This occurs when using Java 6 Update 10 or later.

ASDM version 5.0 or later running on ASA, PIX or FWSM and using Java 6
Update 10 or later.

Use Java 6 Update 7.

I have a couple Pix devices running 8.0(4) code that I have not been able to manage with ASDM 6.1.3 since I re-built my laptop with Windows Vista and the latest JVM.  Today I decided to fix this by upgrading to ASDM interim release 6.1(5)51.  After the upgrade, I ran into a different issue where the software would load (progress bar would finish), but it would get stuck trying to open the main window and just sit there spinning.

I went to Cisco’s download page for the pix and looked for even newer interim releases (the 6.1(5)51 version had been listed on the main download page) and discovered bug CSCsw43498:

ASDM is not working with Java 1.6.0_11 and Vista OS. 


ASDM on Vista OS with Java 1.6.0_11 unable to load


Windows Vista OS and Java 1.6.0_10/Java 1.6.0_11

1. Right-click on the Cisco ASDM Launcher shortcut.
2. Select “Property”
3. Click on “Compatibility”
4. Click on the box “Disable Visual Theme”
5. Restart ASDM

Alternatively, users can downgrade to Java version 1.6.0_07 (download from

I opted for the workaround rather than running an even more bleeding edge interim release 6.1(5)57.  Note that it did not work the first time, so I set it “for all users” rather than just for myself and then it seemed to work.

ASDM Vista Compatibility Mode

ASDM Vista Compatibility Mode


UPDATE: 5/30/09

Cisco has release ASDM 6.2 which I believe fixes all the bugs listed above and it works with PIX/ASA code 8.0.2, 8.0.2, 8.0.4 and 8.2.  Check out the release notes for the full compatability guide.  This release corresponds with version 8.2 of the ASA code which has a number of new exciting features including the AnyConnect Essentials license pack which supports 64 bit Windows.

Categories: Cisco, Network Tags:

Cisco 3750 Delayed Power On

March 31st, 2009 7 comments

Last week I un-racked a Cisco 3750 (WS-C3750G-24TS) from our datacenter and brought it over to our office.  This device had been running reliably at the datacenter for over a year.  When I went to power it back up at the office, the device acted DOA.  No lights, no fans, nothing.

I tested the power cord on another 3750 switch to make sure the power was good and it worked fine.  So I plugged it back into the dead 3750 and tried re-seating the cord, etc, to no avail.  At this point I picked up the phone and called Cisco to get an RMA issued.  About half way through my conversation with the call-taker, I looked down on my desk and realized the switch had booted.  I continued with the Cisco case as the last thing I needed was an intermittent switch.

I then tried un-plugging the switch and plugging it back in several times, and it would boot every time.  Thinking it might be a capacitor charge issue, I un-plugged it while I went to lunch, and then re-connected it when I got back and it took several minutes again to power on.  I have since reproduced this a third time, in which it took over five minutes to boot!  I actually have it on video just in case I need to prove it to Cisco.

This was such a weird failure mechanism that I figured I should share in the hopes that someone else might find it a useful data point.


Categories: Cisco, Network Tags:

Sawtooth looking graphs from Cisco SNMP queries

March 28th, 2009 4 comments

I have been annoyed for quite some time by very regular pattern’s of “Spikiness ” that I run into on certain network graphs and wanted to share my findings to see if anyone else can explain this better than I can.

Below is a graph of what should be extremely consistent traffic.  This graph is created by polling a SNMP OID once every thirty seconds.  It is over a one hour time period.

Spiky Traffic

Spiky Traffic

My best theory is that internally to the Cisco device I am polling (in this case a Cisco ASR) the SNMP OID counters are only updated once every “x” amount of time, where “x” is maybe 10 seconds.  Since I poll every 30 seconds I most often get three “updates” in between the time I poll the OID, but once every five minutes for some reason I only get two updates worth of data (20 seconds).  Then in the next interval I get forty seconds worth of data.

If you have a better explanation than I do, please post a comment or email me and I will update this post with the answer!


Categories: Cisco, Network Tags:

Review of Cradlepoint CTR500 EVDO Router

March 25th, 2009 1 comment

This evening I got an opportunity to test out a Cradlepoint CTR500 EVDO router that a friend of mine just got.  This device can accept an ExpressCard or USB data card (or both at once!) from a long list of different providers.  I tried it out with two Verizon EVDO cards, a Verizon (Novatel) V740 ExpressCard, and a USB727 card in the USB port.

Cradlepoint CTR500

Cradlepoint CTR500

I am very impressed.  My friend said it was incredibly simple to setup.  I only had limited time to run tests, but everything I checked out worked well.  You can check out their website for the full details on features, but here are the most interesting to me:

  • This model of the device is AC powered with a 5v adaptor and also comes with a 12v adaptor for your car.
  • It has a built in wireless access point to share Internet connectivity out to other PC’s.
  • There is a physical switch on the side to turn off the wireless access point.
  • It has an Ethernet jack on it that can be used to directly attach your PC, or a network switch if you want to connect multiple PC’s to provide them Internet.  The Ethernet jack can also be used to hook to some other upstream ISP and share that connection out via the built in wireless access point.
  • There is a lock switch to keep the ExpressCard from sliding out of the device.
  • There are a number of LED’s on the top to indicate status on each of the connection methodologies.
  • There is an antenna port so that you can connect an external WiFi antenna (802.11b/g) – by default it uses an internal antenna

The web interface to the device was nice and snappy.  I did not dive deep into the features, but it looked pretty extensive (with really useful features like custom DHCP settings, rather than stupid consumer features).

We had it setup to use both the ExpressCard and USB modem simultaneously, which is interesting, as it must choose which connection to route traffic out on a “per flow basis” (i.e. a TCP session or UDP exchange).  This means some connections you make might get NAT’d out from one IP address, and others will come from a different IP.  This could be problematic with some applications.

The average ping time to my favorite IP address was 150ms, which is about the same as my internal EVDO card.  All of the tests I ran were with an Ethernet cable directly plugged into my laptop from the CTR500.  I did not actually test the wireless access point in it since I was mostly interested with it’s EVDO performance and not the latency added by 802.11.

I ran a number of speedtests, and while the numbers were not very impressive, they were the same as the Verizon EVDO card built into my Dell D630 laptop (which I also tested in the same location).  The Verizon 1900mhz signal is fairly weak at my house where I ran the test (one bar according to my internal card).  I did not have the right adaptor to be able to plug my 3 watt cellular amplifier into the EVDO cards which would have probably sped it up.

Cradlepoint CTR500 EVDO Router Integra Speedtest

Cradlepoint CTR500 EVDO Router Integra Speedtest

I have to say, thus far I am extremely impressed with the device.  The features are great, the fit and finish is excellent, it is fast and easy to use, and the price is right!  The worst things I can say about it are that some apps might get confused with the dual card configuration (but this is not the routers fault, they implemented it the only way you could accomplish this feature), and the device did get somewhat warm during testing with two EVDO cards sucking power.

I am not sure that I personally have a need for this device today, but I know a number of people who could use one!


Categories: Network, Telecom, Wireless Tags:

Cisco Router/Switch Standard Configuration Checklist

March 25th, 2009 2 comments

Every time I go to configure a new Cisco router/switch, I refer to my trusty text file of the “standard” items I need to setup in order to easily manage/troubleshoot the device in the future, as well as to properly secure it.  I have posted the annotated version of that checklist here in the hopes that others find it useful as well!

As always, if you have any suggestions for items to add to my list feel free to email me!  I am sure there are cool tips/tricks I am missing.

  1. I first have to figure out how to “get in” to the device.  Often if I am re-configuring an old device I must first follow the password-recovery process as the password may be lost.
  2. Once I am logged into the device I must determine how much RAM and flash memory the device has so that I can decide on the best IOS version to run.
  3. I go to and figure out what IOS rev makes the most sense based on what features I need, how much RAM/flash I have to work with, what I am licensed for, and what level of reliability is required (i.e. can I only run “General Deployment” code, or am I willing to run a “technology” train because it has some new feature I need).
  4. If the device is a stackable switch make sure it is all stacked together the way you want it stacked with the correct device the command switch (check your priority settings).  Note that even a single switch by itself could think it was the 3rd switch in a stack (if previously a stack member) and mess up all your interface numbers which is painful to fix later!
  5. If configuring certain models of switch, you need to set the system MTU (which persists outside the configuration file) and reboot the switch for it to take effect.  The default is 1500, so unless you are planning on using jumbo frames (i.e. 9000 bytes hopefully), you don’t need to worry about this.
  6. I make sure I have IP connectivity to the device from my laptop so that I can download firmware to it.  Often times this means I will tftp the config from the device to my laptop to prove it works (and saving the old config may be valuable depending on the situation).
  7. Since I generally like to have as absolutely clean of a device as possible, I will often reformat the flash device so that I can be guaranteed to start from scratch.  Note that this can be dangerous if the device crashed/lost power before you got a new image on it.  You would then have to connect to the console and zmodem download the OS at 9600 baud (not fun!)
  8. Once the format is complete I tftp the new IOS version onto the device.
  9. I do a “write erase” to make sure there is no configuration left in the device (I want it to start from the new defaults as set by whatever version of IOS I am loading).
  10. I will often times check to see what the bootvar is set to, but really at this point it does not matter since there is only one image on the flash disk so it will choose that even if something else is specified.
  11. I issue the reload command and choose *not* to save the running config (since I just zeroed out the startup-config).
  12. When the device boots I say no to the setup prompts and I tell it to kill autoinstall if that is trying to run.
  13. I check all the startup messages to make sure all the hardware is recognized and initialized properly.
  14. At this point I jump into enable mode and do a “show run” to verify all the interfaces I expect are available and to see what new defaults or settings Cisco threw in on this build (they often have default settings show up in the config in order to call your attention to a new feature, or something that has changed).
  15. My first order of business is normally to get the device remotely accessible, as using the console is slow, and I am normally not in a very comfortable environment (i.e. cold, noisy datacenter with poor ergonomics).  To accomplish this I will often just enable telnet (if I am connecting across a secure network), and then later enable SSH.  To make telnet work you need an “enable secret” (well you could use an “enable password”, but that’s old and less secure so just don’t bother).  You will also need to set a password on the vty’s for telnet.  “line vty 0 15” “password <blah>”.
  16. Before you can actually login to the device you must configure an IP on at least one interface, and un-shut it.  You may also need a default route (or some kind of static route) depending on if you are connecting from the local subnet or not.
  17. Once into the device, it is time for a ton of “housekeeping”.  First off, set the “hostname” and “ip domain name <domainname>”.  Note that both of these are required for generating RSA keys later so you can enable SSH.
  18. Setup your name servers with “ip name-server <server>” commands.
  19. Make sure your “boot system <x>” parameter (also known as bootvars) is set to nothing (if you only have one image on the flash device it will boot off that image).  I generally only use this command when upgrading to a new IOS version and I want to keep the old one around for fallback purposes.  I believe in keeping my config’s as simple as possible.  I only configure something if it provides value.
  20. Make sure “service password-encryption” is turned on.  This at least obfuscates some passwords that are normally stored in clear text.
  21. Depending on how you want log entries to look, I will often adjust the “service timestamps” command to have it log in localtime.  If you are just dealing with a small local network in one timezone it is often easier to troubleshoot when not having to think about GMT.  Users generally report issues in local time.  😉
  22. Set the timezone using “clock timezone PST -8” (or whatever is appropriate for you).  You then must further tell it to do daylight savings time (if your locale observes it).  I use “clock summer-time PDT recurring” myself.  The thing that sucks about this command is that I suspect the dates it changes the time forward and back on change depending on how new your IOS is since the US keeps mucking with them.  You can also manually tell it when to make these time changes with this command.
  23. Setup at least one ntp server with the “ntp server <ip>” command so that your device always has the correct time.  Having the correct time is really important for correlating log file entries when troubleshooting.
  24. Turn off CDP on any interfaces that are hooked to “untrusted” networks (i.e. if they are connected to an ISP turn it off).  Issue the “no cdp enable” command on these interfaces.
  25. Setup logging as appropriate for your environment.  If you don’t have an external syslog server (or even if you do), you may want to use something like “logging buffered notifications” (or some other logging level) so that you can at least do a “show log” on the device and see the last “x” number of messages.
  26. I will generally setup my devices with a local username and password (which I can use for SSH, console, telnet, etc…) and then get rid of the enable password, enable secret, and line passwords.  To accomplish this I use the following commands: “username <username> privilege 15 password <password>”, “aaa new-model”, “aaa authentication login default local”, “aaa authorization exec default local”.  Make sure to test your access to the device before you drop out of enable mode or disconnect as you could lock yourself out!
  27. Now that you have configured a local user account, and earlier on you set a hostname and domain name, you can now generate RSA keys so that you can SSH to the device.  Issue the “crypto key generate rsa modulus 1024” command to accomplish this.  Note that you can use various modulus values.  The default is 512 but I generally step it up a bit.
  28. Turn off “ip http-server” unless you use it.  Otherwise it is just a security issue (I think the previous step may actually not password protect the http server by default so be careful of this).
  29. On all your terminal lines (console, aux, and vty’s) use the “transport preferred none” command to keep the stupid router from trying to telnet every time you fumble finger a command!  I prefer this method of disabling this annoying functionality over the “no ip domain-lookup” route since doing dns queries from your router is sometimes convenient (and necessary!).
  30. You can also use the “logging synchronous” on the terminal lines in order to have the device re-print the command you are in the process of typing after it rudely spews a log message to your console.
  31. To make your router/switch only accessible via ssh (since telnet is insecure right?) issue the “transport input ssh” command on the vtys.
  32. For security purposes, set timeout’s on the terminal lines so that users get kicked out after inactivity.
  33. Setup a loopback IP address for device management.  This allows you to connect (and stay connected) to the device when interfaces go up and down.  You will likely want to make this the source IP for syslog messages and snmp traps as to not confuse your NMS (if you have one).
  34. If you are configuring a switch, setup your VLANs (name them etc…), also, make sure to set the root bridge priorities (you don’t want random switches becoming the root bridge).
  35. On switches, you may want to set global settings for portfast, bpduguard, and rootguard to apply to all ports.
  36. If setting up a switch, you should configure (or disable using transparent mode) VTP explicitly, otherwise someone could add another VTP switch to the network as a server and magically your switch will become part of that VTP domain.  This can impact your VLAN settings for all your switchports!
  37. Ok, now that we have all that drugery out of the way, go ahead and configure all your interfaces with IP addresses as appropriate.  Please make sure while you are at it to use the “description” command on each of them to describe what is attached.  If it is an ISP or WAN circuit make sure to provide the circuit ID for ease of troubleshooting.
  38. If your device is a layer three switch and you want it to do routing, make sure to set “ip routing”.  This command is frequently overlooked and is very frustrating to troubleshoot why the device won’t route packets!  For some reason (probably with good intentions to keep routing loops from happening or for security reasons) Cisco keeps this one turned off by default.
  39. Make sure all your speed and duplex settings are set properly for each port.
  40. If on a switch, make sure all ports are set explicitly to access mode if that is what you intend them to be used for.  By default a negotiation will take place and a port you did not configure can become a “trunk” port, providing access to VLAN’s  you were not intending to make available.
  41. Add in whatever default routes or routing protocols you need to run in order to have connectivity to the rest of the network.  Remove any un-needed temporary static routes you had put in place earlier.  Make sure to disable routing protocols on interfaces where they could pose a security risk.
  42. Setup an access list to control remote management access to your router based on IP address.
  43. Setup an snmp community on the device if you plan to use it.  I generally recommend everyone setup snmp read access so they can poll their devices with network graphing software.  Some environment use snmp traps for alerting.  You can also control access to snmp with an access list.  Make sure to set the snmp “location” and “contact” settings.
  44. Once the device is up and operational, make sure to check the interface statistics to ensure you are not getting errors.  The most common mis-configurations are with speed/duplex settings on Ethernet, and when talking about T-1’s the most common issue is getting the clock sources correct.
  45. Shutdown any un-needed interfaces.

See, wasn’t that an easy 45 step simple process?  😉


Categories: Cisco, Network Tags:

Switching Providers With Cisco CDMA 3G HWIC

March 24th, 2009 1 comment

A while back I ran across info on Cisco’s site about their new HWIC-3G-CDMA and HWIC-3G-GSM cards that allow you to connect a Cisco router to the Internet through 3G cell phone networks!  I am glad to see this since I have on occasion wanted a *reliable* way to share my broadband card connection with a small group of computers (i.e. at a tradeshow, etc…).  I have shared broadband cards out through my laptop before but it is a pain, and my general feeling is that the consumer grade “routers” that you can buy now are generally poor quality.

This last week I noticed they have seperate HWIC-3G-CDMA-V and HWIC-3G-CDMA-S versions of the CDMA card that are specific to Verizon and Sprint.  I find this annoying, as if I am going to spend $850 (MSRP) on one of these cards I don’t want to be vendor locked to Sprint or Verizon.  From a chipset standpoint, these cards are using exactly the same technology.

This is particularly problematic as wireless service is so location dependant.  Say you are deploying a bunch of remote site routers across the country and using a 3G card for backup connectivity.  You can’t have any idea at the time you order/ship the devices as to which wireless provider will have the strongest signal available in a given area (i.e. Verizon might have EVDO Rev. A in an area that Sprint only has 1xRTT).

Furthermore, you might decide for cost reasons that you want to move from one provider to another, but the cost of buying new WIC’s makes that untenable.  Can you imagine if you had to buy a different T-1 WIC for Verizon vs. Qwest?

To find out if this was a permanent thing, or if they could be re-flashed over to the other provider, I asked one of my Cisco engineers.  Below is their response which I thought was worth sharing:

Short answer “No”. There is a lot of carrier specific information that gets bundled into the firmware for these modems and the only option is to swap the modems out. The carriers and probably the modem vendors have tools that are able to change the modem parameters to work from one carrier to the other, but we do not have that expertise.

I also asked if the cards required special data plans, or if the “standard” $59.99 plans would work.

In terms of pricing – the SPs have a special pricing plan for these modems that get plugged into Enterprise class networks. Information can be obtained on each of their websites.”

I have heard from reliable sources though that with Verizon you can put them on normal $59.99 data plans and they work just fine.  You just do an ESN swap to get that device activated.  Naturally, they won’t provide any kind of support for your device, but it does work.  Also note that on Verizon if you go over 5 gigs of transfer in a month they will throttle you to 200Kbps.

I am *sure* that with the right equipment you could “make it happen”.  Maybe some day we can hope Cisco releases a tool that let’s you do it.  So I wonder when the WiMax version of the card is available?  That would be cool since I do live in Portland, OR (one of the WiMax test markets).


Categories: Cisco, Network, Telecom, Wireless Tags:

Lack of Telnet Client in Windows Vista

March 23rd, 2009 1 comment

For the most part I will try to limit the number of “rants” on this Blog, but I have one I feel the need to share today (and I will even throw in a bit of helpful troublshooting advice).

Who in the world decided that removing the telnet *client* (not SERVER) from Windows Vista was a good idea?  As a network engineer I want to hunt that guy/gal down and give them a piece of my mind!  For crying out loud it is a 202KB executable file!

I could fully understand if we were talking about a *service* that had to load and run at boot that could represent a remote compromise opportunity, but no, we are talking about an extremely simple application that can simply open up a TCP connection to a remote host.

Windows Telnet Size

Windows Telnet Size

The worst part is that to install the Telnet client after the fact requires a number of clicks to drill down to wherever they hid the “Add/Remove Programs” control panel and then once you find and check the telnet client it takes like 20 minutes to install this 202KB file!

The reason this frustrates me so much is that if I am on a machine and I need to prove that I have TCP connectivity to something, the best way to do this is to simply type “telnet <ip or host> <port>” and hit enter.  I will immediately know if the remote host sent back a RST (connection denied), or if it connected (command terminal often will just be blank or the server could prompt something), or if it times out then the packets are getting lost somewhere.

Telnet Plunger Port 22

Telnet Plunger Port 22

Here you can see the result of the command above.  The connection opened successfully (as evidenced by the prompt clearing), and in this case SSH prompted the client with it’s version number.  You can press “ctrl-]” to drop to the telnet shell and enter “quit” to exit.

Telnet SSH Response

Telnet SSH Response

I use this extremely frequently when there is an access control list or firewall in the way that does not allow ICMP packets, or if I need to check if a port is open.  Also, I use this to prove “it’s not the network” when users complain of application issues and blame the network.  If I can open a TCP connection to their application server then that proves that round trip data flows between the devices are working.


Categories: Microsoft, Network Tags:

Configuring a Cisco Router For Secure dyndns Dynamic DNS

March 22nd, 2009 2 comments

While I normally dislike using dynamic IP addresses on cisco devices, they are sometimes a necessary evil.  I finally got around to setting up a Cisco router at home this weekend, and decided that I needed to setup Dynamic DNS so I could get into it remotely, even though it is on a dynamic IP address on my Verizon FiOS connection.  In doing the research I came across this posting which not only explains how to configure IOS to use the dyndns service, but it also shows how to do it securely using https.

To be honest, I went ahead and used the http (insecure) method as I did not want to mess with certificates, but in many situations you may need the security provided by https.  I am not really worried about anybody hijacking the dyndns for my home IP address.  😉

There is also a Cisco page on how to setup ddns, but it is really confusing and for my purposes, the only useful parts were all the way at the end of the page.  I believe most of this page deals with getting the Cisco router to update DNS on behalf of clients.  In our case all we want is for the router to update the DNS for a single IP address of it’s outside interface.

One very important learning I got out of this process was on how to escape a question mark (?) when you are trying to insert it into a text string on a Cisco device.  Normally IOS treats a question mark as a request for help.  To escape it you must type ctrl-v right before pressing ?.  I never knew that before.  I had tried to insert ? into description fields before but always gave up and chose to use other punctuation to get my point across.  😉

UPDATE: After running this config for a day I got a nastygram from dyndns that I was updating too frequently.  I had to add the minimum and maximum lines to my config so that it only updates once every 28 days if your IP does not change:

interface FastEthernet0/0
 ip ddns update hostname
 ip ddns update dyndns

ip ddns update method dyndns
 interval maximum 28 0 0 0
 interval minimum 28 0 0 0


Categories: Cisco, Network Tags:

Uploading software and config files with tftp

March 21st, 2009 No comments

When working with Cisco (and other) network gear, one of the common tasks I perform is copying configuration files or system images to/from a router/switch/firewall using tftp.  In many cases I am transfering these to my laptop using a tftp server running on my laptop and the tftp client built into the network device.

Many years ago someone told me about this tftp server called PumpKIN and I have been using it ever since.  The entire program comes in a 144k download and it installs in about half a second.  Works like a champ and it is just so dirt simple, it has never failed me.



The one thing to note is that if you accidentally un-check the box in the bottom right corner, it will stop responding to requests (or if you launch multiple copies of the server, only one can be bound to the socket at a time).

Here is a screenshot of the options window.  I always set the settings as shown below so that I don’t have to approve each time a device needs to “get” a file from the server (many devices send multiple requests when downloading an image to verify it’s existance and such before download).  I also set the directory to c:\tftp such that it is easy to find and get access to.

PumpKIN Options

PumpKIN Options

I should note that after this latest install in Windows Vista I had to open udp port 69 in the Windows Firewall to get PumpKIN to work.


Categories: Cisco, Network Tags:

How To Configure Jumbo Frames

March 18th, 2009 17 comments

I have found there is a lot of confusion out there, as to how jumbo frames work, and in what circumstances they should be used.  Hopefully this post will be useful to others to avoid some of the pitfalls I have run into.

On your standard Ethernet network, the default frame size is 1500 bytes.  This can be notated in a number of different ways, however, since the actual amount of data sent is normally 1514 bytes which includes the source mac address (6 bytes), dest mac address (6 bytes), and protocol id (2 bytes), of the data in the frame (in most cases these days IP).  If you further include the 7 byte preamble, the 1 byte Start-of-Frame-Delimiter, and the 4 bytes of CRC32 at the end, you get 1526 (not to mention the 12 bytes of interframe-gap after transmission).  I have also seen it notated as 1518 bytes (1500 + 6 bytes src mac, 6 bytes dest mac, 2 bytes protocol id, and 4 bytes CRC 32).  When you run ifconfig on most operating systems it will show you 1500 bytes.  Within this “1500 byte frame” the usable space is further cut down to 1480 bytes by 20 bytes of IP header, and then if you are using TCP the data payload is cut down to 1460 bytes by another 20 bytes of TCP header.

Phew!  Was that enough numbers?  Now that we have that confusion out of the way, the question is why would you run something larger than 1500 bytes?  Well, back when the standard was set at 1500 byte frames we were not dealing with as large amounts of data as we are today.  Data transmission speeds were slower (i.e. 10 megabit) and the chances of encountering an error during the time it took to send a 1500 byte frame were much higher than sending a 1500 byte frame at gigabit speeds.  Keeping frames smaller meant you had to re-transmit less data when an error occurred.

As 100 megabit and then gigabit came out, the standards groups kept with the 1500 byte limit for backwards compatability.  This is great, except that when transferring large blocks of data (say during backups) this is less efficient than it could be since for every 1460 bytes of TCP payload data, you have 78 bytes worth of “time on the wire” that is used up.  You also force the computers at both ends of the connection to calculate a CRC-32 value for every 1460 bytes of data which is processor intensive (I am assuming you are not running a TOE), and also send ACK’s back etc…

So for certain applications you might want to consider running larger than 1500 byte mtu’s.  I recommend them only in the following circumstances:

  • You have an all gigabit network for the VLAN/network in question
  • All switches in your switching path can be configured to be “jumbo frame clean” up to the size frames you choose to use
  • You are sure all your host operating systems and NIC drivers will support Jumbo Frames at the size of frame you decide to use (this can vary a lot!) – note that I was once told that DataDomain boxes support jumbo frames, but they are not recommended because it causes a performance impact (this does not make sense to me, but it is what they said)
  • You have a real reason to need to run them (i.e. a backup VLAN or NFS VLAN)

Note that getting jumbo frames to work can be a real pain and in many cases is probably not worth it.  When you make the switch on a given subnet over to jumbo frames, you must simultaneously change *every* host on that network/VLAN or you will have weird problems where some hosts can talk to each other and others have difficulty for some types of connections.  TCP connections might work, but UDP connections fail intermittently (i.e. you can mount an nfs mount but when transferring files it fails).

As a side note, the reason that TCP will work on a MTU size mismatched network is that in the TCP 3 way handshake, a MSS (Maximum Segment Size) is specified by both ends of the connection and the smaller of the two is agreed upon.  Running different hosts on the same subnet with differing MTU sizes is *not* recommended.

So what size MTU should you choose?  Since there is no standard, different vendors have chosen to implement different maximum supported sizes.  I have seen drivers support all sorts of different mtu’s…  Some allow any mtu up to a max, while others support only several “canned” values (i.e. some Windows drivers).  From some research I have read, CRC-32 becomes ineffective above about 11,000 bytes.  I have found that every manufacturer I have used (that supports jumbo frames at all) will support 9000 byte MTU’s, therefore I highly recommend using 9000 byte MTU’s (this is also what is standardized on for Internet 2 traffic).

It is extremely important that your entire VLAN/network you are implementing this on be “Jumbo Frame Clean” for the size MTU you are using.  What I mean by this is that every switch in the path will allow at least 9000 byte mtu packets through.  Different switches have differing commands to make them pass jumbo frames.  In my Cisco 6500 series platforms I set the mtu on a per-port basis (depending on your Supervisor and OS version you might only have 9216 as an option which is fine as long as it is larger than your chosen mtu).  On my 3750’s (and similar stacking blade enclosure switches) I have to set it globally on the switch and reboot the switch to take effect.

Operating System Settings

To actually set jumbo frames in Windows you go to the device manager, select your NIC driver, go to the Advanced tab, and find the Jumbo frame setting.  You must either choose the correct MTU size from a drop-down list provided (in some drivers), or enter the correct frame size it into the box (if applicable).  You might need to verifiy with the manufacturer what number they are expecting here (does it include frame headers and footers?)

Here is an example of a NVIDIA NIC:

NVIDIA Jumbo Frame Settings in Windows

NVIDIA Jumbo Frame Settings in Windows

And here is an example of an Intel NIC.  Note that NVIDA lists it as 9000 bytes and Intel lists 9014, but my testing has shown these settings to be equivalent!  Also note that none of the other options align…  (except for 1500 bytes)

Intel Jumbo Frame Settings in Windows

Intel Jumbo Frame Settings in Windows

To set jumbo frames on Solaris, you must first figure out how to configure  your specific driver for jumbo frames, and then use ifconfig to set the interface to the specific frame size you decide on.


To test to make sure jumbo frames are working, you can use the following ping command from your Cisco switch (this is assuming that you have a layer 3 interface in the jumbo frame vlan and that L3 virtual interface is setup for your jumbo mtu size).

switch#show run int vlan 99               
Building configuration…

Current configuration : 190 bytes
interface Vlan99
 description backup
 mtu 9000
 ip address
 ip broadcast-address


switch#ping ip size 9000 df-bit

Type escape sequence to abort.
Sending 5, 9000-byte ICMP Echos to, timeout is 2 seconds:
Packet sent with the DF bit set
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms

You can also generate jumbo frames from Solaris boxes, but Solaris does not seem to support setting the “DF” (do not fragment) bit which is a shame since you might never know that the machine you are on fragmented the packet before it sent it, hence not actually testing jumbo frames…

bash-3.00# ping 9000 is alive

Windows boxes can send jumbo frame pings as well, but you must subtract out the 20 bytes of IP header and 8 bytes of ICMP header that it inserts in the size you specify.  They *do* support setting the DF bit.

C:\>ping -f -l 8972

Pinging with 8972 bytes of data:

Reply from bytes=8972 time=2ms TTL=255
Reply from bytes=8972 time=2ms TTL=255
Reply from bytes=8972 time=2ms TTL=255
Reply from bytes=8972 time=2ms TTL=255

Ping statistics for
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 2ms, Maximum = 2ms, Average = 2ms


There are a lot more details to everything we went through to implement jumbo frames on our backup networks, but this has hopefully given you the high level overview and a few key tips.


Categories: Cisco, Microsoft, Network Tags: