bitplumber.net

As a Network Engineer I am frequently called upon to assist in troubleshooting connectivity and integration issues with our customers by capturing packets and providing analysis.  These clients make SOAP calls to our system across the Internet (or private networks) using SSL (TLS) encryption.  The encryption makes troubleshooting application issues and response time issues very difficult when everything inside the TCP stream is encrypted garbage.

If your system uses SSL offloading capabilities in a load balancer, you can sniff the unencrypted packets between the load balancer and the web server, however, this will not let you see Internet caused TCP flow issues (retransmissions, out of order packets, etc…), and very often (depending on your architecture) the load balancer will change the source IP of the connection to it’s own which makes tracking down the interesting flow very difficult.

I learned a new trick this week that allows me to give Wireshark my private SSL keys (the ones loaded on the web server or load balancer), which will allow it to decrypt the SSL encryption and show me the payload in clear-text.

This is actually written up pretty well on the Wireshark support wiki so I won’t spend much time rehashing that here, however, I do have a few notes:

• You need to get a copy of your rsa private key.  Often times this will mean exporting the certificate from the keystore (on Windows boxes, etc…), or it will mean grabbing the .key file from wherever it is stored on the file system in linux.
• You will most likely need to use openssl (or some other tool) to extract just the private key from whatever format you have available (you don’t care about the public key, the signature, or the certificate chain).
• In my case I had a .key file of just the private key, however, it was encrypted with a password and I needed to use openssl to convert it to an unencrypted form that Wireshark can use.
• As described in the Wireshark documentation, you must specificy which private key files to use for which server IPs.  This is specified in the Wireshark SSL preferences.
• To actually decode an SSL stream, find a packet that is part of an SSL exchange, right click on it, and select “Follow SSL Stream”.  This is effectively the same as “Follow TCP Stream”, except it does the decryption.
• If it does not seem to be working, in the Wireshark SSL preferences, turn on logging to a text file and try again.  Note that this is quite verbose so once you get your issues worked out I would not leave it on.
• Note that you must capture the entire SSL key exchange process, otherwise it will not be able to decrypt the payload.  I have seen flows where Wireshark would only decrypt the data in one direction or another.

So remember, it is incredibly important to protect your SSL keys!  All too often they are left laying around on numerous web servers that may be directly exposed to the Internet.  Make sure your keys are backed up, on a flash drive, in a safe, somewhere offline, and then mark all the keys on Windows servers as non-exportable.  On other platforms make sure the keys are at least encrypted with a password themselves.  Centralizing your keys on a load balancer (rather than a bunch of web servers) can decrease your risk exposure.

Update:  There is actually a good reference on this on Novells web site as well.  Apparently Wireshark will not decrypt the DHE protocol so if the browser and server are using that it will not work.  I am not sure if newer versions of Wireshark do already or will handle DHE.

-Eric