bitplumber.net

When working with telecom circuits that are slow and “expensive” (relative to lan circuits), the question frequently comes up “What is using up all of our bandwidth?”.  Many times this is asked because an over-subscribed WAN or Internet circuit is inducing latency/packet drops in mission critical applications such as Citrix or VoIP.  In other cases a company may be paying for a “burstable” Internet connection whereby they are paying for a floor of 10 megabits, but they can utilize up to 30 megabits and just be billed for the overage (at the 95th percentile generally).

So how do you tell which user/server/application is chewing up your Internet or WAN circuits?  Well Cisco has implemented a technology called “netflow” that allows your router to keep statistics on each TCP or UDP “flow” and then periodically shove that data into a logging packet and ship it off to some external server.  On this server you can run one of a variety of different software packages to analyze the data and understand what is using up your network bandwidth.

The question is, what software package should you utilize?  I have not gone and evaluated all of the available options, but I do have experience with a couple of them.  I have used Scrutinizer from Plixer in the past and not been very impressed.  Part of it may have been that the machine it was running on was not very fast, but I just did not like the interface or capabilities much.

More recently I have downloaded and run NetFlow Analyzer from ManageEngine and I have been very impressed!  It is free for only two interfaces and they have an easy-to download and install demo that will run unlimited interfaces for 30 days.  It runs on Linux or Windows (I tried the Linux version) and is is dirt simple to install and configure.  There really is nothing of note to configure on the server itself, you just need to point your router at the server’s IP and it will automatically start generating graphs for you.

I should also mention that Paessler has some kind of netflow capabilities (in PRTG), but I have not checked it out.  I note it here since I use their snmp monitoring software extensively and I have been happy with it.

To get your router to send NetFlow data to a collector, you need to set a couple of basic settings (including which version of NetFlow to use and where to send the packets), and then enable sending flows for traffic on all interfaces.  Note that it used to be you could only collect netflow data upon ingress to an interface and so in order to collect data on bi-directional traffic you needed to enable it on every single router interface in order to see the traffic in the opposite direction.  This was done with the “ip route-cache flow” command on each interface.

Now “ip route-cache flow” has been replaced with ”ip flow ingress” and you can also issue ”ip flow egress” command if you were to not wanting to monitor all router interfaces.  I have just stuck with issuing ”ip flow ingress” on all my interfaces since I wanted to see all traffic anyway (and I am not quite sure what would happen if you issue both commands on two interfaces and then had traffic flow between them, it might double count those flows).

Here are the exact commands I used on plunger to ship data to Netflow Analyzer 7:

plunger#conf t

Enter configuration commands, one per line.  End with CNTL/Z.

plunger(config)#ip flow-cache timeout active 1

plunger(config)#ip flow-export version 5

plunger(config)#ip flow-export destination x.x.x.x 9996

plunger(config)#int fastEthernet 0/0

plunger(config-if)#ip flow ingress

plunger(config-if)#int fastEthernet 0/1

plunger(config-if)#ip flow ingress

plunger(config-if)#end

plunger#write mem

Building configuration…

[OK]

plunger#exit

Happy NetFlowing!

-Eric

eprosenx Cisco, Network

After waiting for several weeks for Cisco to fufill my license code order (for ASA-AC-E-5510) I finally got the code in an email today!  Using the product authorization key to generate an activation key for my specific device was easy on the Cisco licensing web site.

I used the “activation key” command to plug it into my ASA 5510 and now I have the “AnyConnect Essentials” feature enabled when I do a “show ver”.

I never rebooted the device to activate the new license, however, I have not tested more than one user logged into it so I have no proof it works as of yet.

My ASA 5510 has been running the 8.2 code now for 26 days as my production corporate firewall without a hitch so I give it the thumbs up.  I did have one hiccup while rebooting the box after uploading the 8.2 code, but I actually think that 8.0.4 crashed when I asked it to reboot, rather than the 8.2 code failing to come up properly (I was not on the console however when I did this so I really have no proof, I ended up power-cycling the box).

I do have to gripe that the AnyConnect client has had issues on my Windows Vista laptop a number of times, though According to Cisco this may be due to Windows bugs relating to sleeping my laptop (which I do multiple times a day).  I get the dreaded “The vpn client driver has encountered an error” message.

Perhaps one other thing worth noting is that 8.2 created a new “coredumpinfo” folder on the internal flash file system with a file in it called coredump.cfg.  This file seems to somehow update it’s timestamp every time you do a show run and so It messes up my RANCID process which grabs the config and file system directory listings every 30 minutes and diff’s them for me.  This causes RANCID to email me every half hour with useless data that this file changed.

P.S.  The AnyConnect Essentials license key for my ASA 5510 was only $108 from my CDW rep including shipping (which was email btw…)

-Eric

UPDATE 6/24/09:

I forgot to mention that I am running this 5510 with only the stock 256 megs of RAM without issue.  There is reference in the release notes of possibly needing more RAM on that platform for 8.2 depending on what you are doing.  My RAM utilization actually went down between 8.0.4 and 8.2, though I also made some config changes around the same time so YMMV.

Also, there is some reference in the ASDM GUI about needing to reboot after applying a new activation key so I may need to do that…  Still have not tested it yet since my Vista laptop is being dumb.

eprosenx Cisco, Network

So I have burned *way* too many hours on and off over the last couple weeks trying to get an Ultra 25 Sparc box I inherited working.  This box came to me with the video card not in the machine and some comment about it not working.

After putting the video card back in the box, when I booted the box it would not give any video output to the monitor.  I hooked into the serial console (9600-8-N-1 of course) and it appeared to be hanging with the last output being: Probing I/O buses

reset reason: 0000.0000.0000.0004
@(#)OBP 4.25.9 2007/08/23 14:17 Sun Ultra 25 Workstation
Clearing TLBs
Power-On Reset
Membase: 0000.0000.0000.0000
MemSize: 0000.0000.0004.0000
Init CPU arrays Done
Init E$ tags Done
Setup TLB (small-footprint mode) Done
MMUs ON
Init Fire JBUS Control Register… 
Find dropin, Copying Done, Size 0000.0000.0000.7260
PC = 0000.07ff.f000.6178
PC = 0000.0000.0000.6228
Find dropin, Copying Done, Size 0000.0000.0001.1440
Diagnostic console initialized
Configuring system memory & CPU(s)

CPU 0 Memory Configuration: Valid
CPU 0 Bank 0 1024 MB Bank 1 <empty> Bank 2 1024 MB Bank 3 <empty>

reset reason: 0000.0000.0000.0005
@(#)OBP 4.25.9 2007/08/23 14:17 Sun Ultra 25 Workstation
Clearing TLBs
Loading Configuration

Membase: 0000.0002.0000.0000
MemSize: 0000.0000.4000.0000
Init CPU arrays Done
Init E$ tags Done
Setup TLB Done
MMUs ON
Init Fire JBUS Control Register… 
Block Scrubbing Done
Find dropin, Copying Done, Size 0000.0000.0000.7260
PC = 0000.07ff.f000.6178
PC = 0000.0000.0000.6228
Find dropin, (copied), Decompressing Done, Size 0000.0000.0006.4530
Diagnostic console initialized
System Reset: CPU Reset (SPOR)
Probing system devices
jbus at 0,0 SUNW,UltraSPARC-IIIi (1336 MHz @ 8:1, 1 MB) memory-controller
jbus at 1,0 Nothing there
jbus at 1c,0 Nothing there
jbus at 1d,0 Nothing there
jbus at 1e,0 pci
jbus at 1f,0 pci
Loading Support Packages: kbd-translator obp-tftp SUNW,i2c-ram-device SUNW,fru-device SUNW,asr
Loading onboard drivers: ebus i2c i2c i2c ppm
/ebus@1f,464000: flashprom rtc serial serial env-monitor i2c power
/ebus@1f,464000/i2c@3,80: gpio temperature temperature temperature front-io-fru-prom sas-backplane-fru-prom dimm-spd psu-fru-prom hardware-monitor
/i2c@1f,520000: dimm-spd dimm-spd dimm-spd dimm-spd
/i2c@1f,530000: motherboard-fru-prom gpio clock-generator
/i2c@1f,462020: nvram idprom
Probing memory
CPU 0 Bank 0 base          0 size 1024 MB
CPU 0 Bank 2 base  200000000 size 1024 MB
Probing I/O buses

Based on the fact that it stopped working at “Probing I/O buses” and there was the possibility of an issue with the video card, I tried removing the card and booting headless.  In this configuration the system came up fine, with access from the serial port.

I eventually discovered that the issue was an impacted pin in the external dongle that splits the high density dual-dvi port in to two separate DVI ports.  The important note here for anybody searching for this issue is that when you have a video card in the machine, the last thing you will see on the serial console is Probing I/O buses since once it finds the video card, all future output is redirected to the video console.  So if you don’t get any output on the screen make sure to double check your video dongle, cables, and monitor!

Also, another unexpected behavior I ran into while troubleshooting- If I left the USB keyboard hooked to the machine while booting, it will assign that as the input device, and it won’t accept input on the serial console, even though that is where all output is going!  It is very odd typing on a keyboard and having your output go to a serial console…

-Eric

eprosenx Sun

At each organization I am exposed to, it is interesting to see the various naming schemes that have been employed over time.  I most often find a hodgepodge of different naming standards that have been poorly followed.  Well thought out naming standards will make a huge difference in the ease of maintaining your environment.

So how should you come up with a device naming standard?  I won’t profess to give you a one-size-fits-all solution, but instead I will outline a number of the pitfalls to device naming that I have run into in order to help you devise your own convention.

Uses for a name

In IT, device names serve three primary roles:

• They are a unique identifier used to define a device (note that a MAC address or serial number could be used as a unique ID, though it provides no other information about the device and is difficult for humans to work with).
• When entered into DNS they provide an easy way to connect to a given device by typing in it’s name from scratch, or device names may be selected from a list in a program such as a SSH program.
• When you see a device name in a log, or on a document it’s name should be obvious what the device in question is and convey to you critical information about the device.

Naming goals

• Names should be as short as possible, easy to type and read, but with enough information to be unique and descriptive.
• Make things as intuative as possible.  If you have an IT contractor working in your environment it should be pretty obvious to them what various servers do based soley on the machine names.
• Your naming system should be flexible enough to allow for growth.

Naming structure

• Generally you should start the name with the most significant identifier, and work your way through to the least significant identifier.   This makes sorting useful.
• Think about how long should each field in the name be.  It needs to be long enough to hold unique entries for as many items of that type as will likely be utilized using the characterset defined for that field (i.e. if you have a two digit alpha field for site code, you can have a max of 676 sites, though if you want them to be intuative you probably don’t want to use the XZ designator) – a numeric only field has less options, 0-9 only yields 10 possibilities per digit.
• Within a name you might choose to include delimiters between fields in order to seperate them, or just for stylistic reasons.  This makes names longer to type (and sometimes to long to fit in documentation, etc…), but they are often worthwhile from a readability standpoint.  PRF5A is a lot harder to read than PR-F5-A.  Most special characters are banned from device names, though dash “-” seems pretty well supported.
• You can only have one variable length field in a name, unless you are using delimeters, or adjacent fields are obviously seperate since some are alpha only, and others are numeric only.
• Note that not everything needs to have names of the same length – It is ok to name one server PDXFILE1 and another PDXSAN1.
• Not everything needs to follow exactly the same nomenclature – routers and network hardware can follow one standard, while servers may follow another.  THIS IS OK!  As long as they don’t conflict…

Know your organization

• Think about how your company will grow.  Might you ever have more than one VMWare server?
• Unless there is no way your business will ever have more than one site (what if you were acquired) I highly recommend your names start with a site code (more on this below).
• Not everybody has the same needs!  You don’t have to force the same scheme on every organziation!  A small manufacturering company has different needs from a global multinational.  You can get away with much simpler names in a small company than in a huge multinational corporation.

Who is your audience?

• Names should be descriptive to your audience,  Who is your audience?  Users?  IT staff?
• In an optimal world, machine names should not be seen by users.  In end-user facing situations I recommend using CNAME’s wherever possible to alias “service names” to “server names”. (i.e. webmail.bitplumber.net could be CNAME’d to pdxmail1.bitplumber.net.  Note that this often falls down in Windows since in Outlook for instance it insists on showing the user the *real* servername…  The same goes for file server names.
• Internet facing services should never have users seeing the machine names.  They are likely connecting to a firwall and or load balancer first anyway so this is easy to hide.

High-level recommendations

• Don’t name things non-sensical names, this is not 1990 (yeah, I know I broke this rule when naming plunger.bitplumber.net)
• Avoid putting un-necessary junk in server names – I don’t really care what the model number of server is (in most cases), or even if it is a VMWare guest server or a physical server (this matters less and less as time goes on).
• Don’t put a version number of software in the name as you will likely upgrade it! (I have seen servers named Win2k that are running Windows 2003 Server)
• If the server might end up running multiple applications don’t put the name of one piece of software in the name, call it an application server or something…  (I have seen a server named backupexec that was running netbackup…)
• In a software development shop (or even a non-software shop), you will likely have multiple copies of similar environments for testing purposes.  PRODUCTION, QA, DEVELOPMENT, STAGING , etc…  This is a good thing to include in the name as you typically have similar server names in each and you don’t want to inadvertantly make a change in Production when you intended to make it in QA.
• Usually it makes sense to name services with a number on the end as you might have multiple servers performing the same function, or even if you only have a single server in that function you might move to another physical server later which you designate with a different number on the end.
  Many environments put two numbers on the end of servers, but how often do you really have more than 9 servers of the same type at one site?  It may be ok for some servers to have a single digit number on the end, while others have two digits.

Site codes

In most organizations I recommend the use of site codes as even single-site companies often end up with remote sales offices, disaster recovery datacenters, etc…

The goal with site codes is to choose a identifier that people both from the site in question, and others far away can easily identify as being related to a given location.  I have often struggled with this as there is no standard, and lots of potential for confusion and overlap.

You must decide how long you want your site codes to be.  I know Intel used to use two digit codes.  Many organizations choose three digit codes which conveniently enough corresponds with airport codes.

There are  a couple issues with airport codes however:

• Some airport codes are not obvious which city they are in
• You often times will have multiple sites within the serving area of a single airport

Note that not all site names have to be the same length (depending on your name structure).  At the last company I worked for I gave the large headquarters site in each region a three digit code, and then the smaller satellite sites got five character codes that began with the three digit region in which they were located.  i.e. PDX was the headquarters site and PDXPC was the Pacific Center satellite site.

A few other notes

Two situations to consider: Naming a device after a department, but that department moves elsewhere physically, but the device stays…  Or, naming a device after a building, but the company moves to another facility along with the device, and keeps the name.  Sometimes you must make a decision as to what a device will stay sticky with, the company/department, or the physical facility.

What is the timespan that your naming scheme must be good for?  I doubt a single site company is going to become a multinational overnight…  Your average IT device lasts 3-7 years so your naming scheme can easily change at replacement time to handle growth.

You might need to consider naming of devices with multiple network interfaces, each with different IP’s.

• Windows is dumb and by default wants to register every interface with the same thing in DNS.  This can lead to issues if all networks are not directly reachable by all hosts accessing the device.
• Solaris is interesting in that it wants each interface named differently.  In this case I recommend making the main server name map to the “primary” interface (i.e. probably the one you set the default gateway on) and then use <hostname>-xx for additional interfaces where -xx is something like -bk for backups, etc…
• Routers should have different forward and reverse names for each interface, plus forward and reverse names for a loopback IP.  (i.e. fa0-0.plunger.bitplumber.net and fa0-1.plunger.bitplumber.net and just plain plunger.bitplumber.net for the loopback IP)

In one environment I have worked in we name all of our iLO, ilom’s, DRAC’s, etc…  <hostname>-SC (sc = service controller).  This makes it easy to go login to one in an emergency.  Just don’t accidentally cross the DNS entries or else you might power cycle the wrong box!

You must be careful not use special characters in device names.  Note that different devices and directory systems may have different “special characters”.  Think about Windows names, Unix names, router names, DNS names, WINS names, etc…  Each different type of name has different restrictions on what characters and symbols are allowed, and what the minimum and maximum lengths are.  Some names could be case sensitive, but most are not.

I personally find uppercase names easier to read in documentation and on screen, but that is in many cases a matter of personal preference, and in others may be enforced by the system in/on which the name is set (i.e. DNS).

IP addressing in relation to names

This is a topic worthy of another complete blog post, but I will point out just a couple of key recommendations here.

Since private ip address space is “free” and “plentiful” I generally build my subnets with plenty of IP space so that I can space machines widely and align their last number with their server number.  Most often I will use /23 subnets for servers and clients which gives me 512 IP’s (minus a few for network, broadcast, and default gateway).  As an example, you could have a server called PDXESX1 with an IP of 10.111.2.21 and another called PDXESX2 with IP 10.111.2.22, PDXESX3 as 10.111.2.23, etc…

On a somewhat unrelated note, in my oppinion the default gateway should always be the lowest usable IP in the range because it is intuative for anyone that follows after you.  Along these same lines, I am a fan of always making my DNS servers .11 and .12 in a given subnet (or .11 in one subnet and .11 in another subnbet).

Is this the right time to change?

Is change really needed?  Or is it simply change for change sakes?

The natural tendency for each new “owner” of a network is to want to do things their way with a naming standard that makes sense to them.  Don’t keep changing your naming schemes!  Even if the existing one is not perfect, it may be better overall just to leave it as is!

You generally don’t want to avoid changing a machines name after it has been set – the name gets referenced all over the place, and unless your process to change it is perfect, it will get missed somewhere and cause confusion down the road…  Think about all of the places you might have to change the name:

• On the machine itself (hostname, hosts files, application configurations…)
• In your ip address spreadsheets
• In your inventory system
• In DNS entries (including CNAME’s that reference the host name)
• On the labels stuck to the machine physically
• Your labels in the network switch (and supporting documentation)
• Labels on the cables attached to the server – network, power, etc…
• In your monitoring software
• On your kvm switch
• In description fields on your remote power cycle device (PDU’s) 
• On your network diagrams and documentation

Final thoughts

While this may be a bit overwhelming, it is crucial to consider all of these aspects ahead of time in order to avoid needing to change your standard down the road.  I hope this has given you an overview of many of the pitfalls of naming I have run into during my career such that you can avoid the same mistakes!

As always, if you have any additional comments, feel free to post them here, or shoot me an email and I may include them in a future post.

-Eric

eprosenx Uncategorized

As a follow-up to a previous post, I am happy to report that Cisco has finally posted the bits to the ASA 8.2 code online for download.  I have been looking forward to this, as this release includes a new license model for the AnyConnect VPN client called “Cisco AnyConnect Essentials”.

While I still can’t find any written reference (on the Cisco price list or elsewhere) for how much the AnyConnect VPN client is going to cost, I have confirmed that the previous rumor of it being “next to free” is indeed true.  Cisco is only charging $150 for the AnyConnect VPN Essentials license on a 5510 which will give you up to 250 simultaneous users!  (that is about as close-too-free as Cisco gets)

This is the answer you are looking for to deal with 64 bit client support!  A coworker of mine even told me today that the AnyConnect client works in his Windows 7 Beta 2 machine (which surprised me, I suspect under-the-hood the Windows 7 networking stack is very similar to Windows Vista).

The part number you need for an ASA 5510 is ASA-AC-E-5510=.  If you need the part numbers for other models check out the release announcement.

There is some reference in the release notes to possibly needing more ram in the ASA 5510 platforms (I am not yet sure if this will impact me, I am not doing a ton of stuff on my ASA 5510 but yet I run near 80% RAM utilization on version 8.0.4).  It is worth noting that there is annoying footnote that says the 256 -> 512 meg of RAM upgrade won’t be available till June…

Also, I have been told that the Botnet detection feature will be $460 a year.  This is part number ASA5510-BOT-1YR= for the ASA 5510.

I will write up another post once I install the 8.2 code somewhere.

-Eric

UPDATE: 5/18/09

I am getting conflicting information from my VAR than I got directly from Cisco.  They say MSRP is $350 right now and it won’t be available till late this month or early June.  CDW has it posted for $232.99 without any special pricing discounts you may have with them.  Availability says to call…

UPDATE: 5/29/09

The CDW site now shows that the ASA-AC-E-5510 part is $101.99.  It still says availability is “call”…

And for those of you looking for the part numbers you need to purchase the AnyConnect Essentials for your model of ASA, here they are:

• AnyConnect Essentials VPN License - ASA 5505 (25 Prs) - ASA-AC-E-5505=
• AnyConnect Essentials VPN License – ASA 5510 (250 Prs) – ASA-AC-E-5510=
• AnyConnect Essentials VPN License – ASA 5520 (750 Prs) – ASA-AC-E-5520=
• AnyConnect Essentials VPN License – ASA 5540 (2500 Prs) – ASA-AC-E-5540=
• AnyConnect Essentials VPN License – ASA 5550 (5000 Prs) – ASA-AC-E-5550=
• AnyConnect Essentials VPN License – ASA 5580 (10K Prs) – ASA-AC-E-5580=

eprosenx Cisco, Network

For over a year now our team of oncall engineers has been tortured by an error generated periodically by our racks of Sun X4100 and X4200 servers.  These alerts come from the integrated ILOMs which we have set to syslog to our EM7 monitoring platform.  Usually about once a week one of our many servers will report something along the lines of the following error:

FIRST REPORTED: 2009-04-29 14:50:33

LAST REPORTED: 2009-04-29 14:50:34

 

SEVERITY: CRITICAL

OCCURRENCES: 2

SOURCE: Syslog

ORGANIZATION: Management

DEVICE: prsun1-sc

 

Full message text for most recent occurrence:

<130>logmgr: ID = 343 : Wed Apr 29 14:52:39 2009 : IPMI : Log : critical : ID =   7f : 04/29/2009 : 14:52:39 : Voltage : mb.v_+12v : Lower Non-critical going high : reading 12.16 > threshold 10.96 Volts

 

This event has not been acknowledged

 

Sent by notification policy: Major/Critical Events

 

The EM7 has received a CRITICAL syslog notification from this server.

If you go look at the event log on the ILOM it looks more like this:

04/29/2009 : 14:52:39 : Voltage : mb.v_+12v : Lower Non-critical going high : reading 12.16 > threshold 10.96 Volts
04/29/2009 : 14:52:38 : Voltage : mb.v_+12v : Lower Non-critical going low : reading 7.37 < threshold 10.96 Volts

Looking at the event log another server with the same type of issue, the error is for a different sensor, but yet it has the same behavior:

02/21/2009 : 06:25:01 : Voltage : p1.v_vddio : Lower Non-critical going high : reading 1.85 > threshold 1.60 Volts
02/21/2009 : 06:24:55 : Voltage : p1.v_vddio : Lower Non-critical going low : reading 0.97 < threshold 1.60 Volts

I should note that these errors *never* seem to turn out to be anything but noise…  We all just acknowledge the alarm and go back to bed.

This week I finally got annoyed enough to go look further into this issue as I do participate in the on-call rotation which covers these systems (even though I don’t *own* these systems).

After doing some digging, I found the following obscure note in the release notes for some firmware update bundle which includes ILOM firmware:

ILOM Service Processor firmware 2.0.2.5
  * Fixed the bug of lower non-critical voltage sense issue.

So I have gone ahead and upgraded a couple of my servers thus far.  Hopefully this will resolve the issue!

I have to get in a couple of jabs at Sun here since I burned an entire day today messing with their servers:

• When you upload the ILOM firmware (which includes a system BIOS upgrade also)  your server may get powered off during the upgrade without any warning.
• When you upgrade to a 2.0 BIOS from a 1.x version, you have to manually clear the CMOS according to their release notes (the update utility seriously could not do this for us?)
• And my personal favorite, their documentation makes some obscure reference to some bug you might run into and so they tell you that you must upload the new firmware *twice* in order to ensure it applied properly.  Mind you they don’t tell you what the problem you might run into is, and they give you no way to tell if the person that upgraded the firmware for you previously did the double firmware update properly.
• After the ILOM firmware and system BIOS updates I did today, the servers somehow managed to change the device ID’s (or something) on the onboard NVIDIA NICs in such a way that Windows recognized them as new NIC’s (5 and 6).  This caused them to loose all IP settings and I had to log in through the ILOM and reset them.  This happend on the two servers I upgraded.
• To upgrade the RAID card firmware/BIOS you must boot the server from a CD that runs DOS.  Note that on a Dell box you drop in the Openmanage CD, it scans your system to determine what needs updating to get you to a “known good set” of drivers, and you click the go button.  It takes care of all Firmware/Drivers/Software for you.
• The LSI software for Windows to monitor the built in RAID card is a joke.  It looks like an intern wrote it.
• At least Sun does provide a streamlined Windows driver installer package, this did work well.

Overall, I am not completely thrilled with Sun’s x86 hardware lines, though I suppose things may be better if you are a Solaris-on-x86 shop.

-Eric

UPDATE 5/13/09

I got another voltage error on one of my fully updated servers.  I have called Sun and opened another case on this, though so far Tier 1 and Tier 2 techs do not seem to have any ideas as to what is causing this issue.  I sent them a bunch of output from the ipmi tool that they are looking through.

ID = 1 : 05/10/2009 : 23:58:42 : Voltage : p1.v_vtt : Upper Non-critical going high : reading 1.79 > threshold 1.00 Volts

I should also note that after the firmware updates, one of the machines is now reporting ECC errors.  This makes me wonder if the previous firmware was not properly reporting them.  We have had almost zero RAM problems with our dozens of Sun x86 servers which makes me worry that they are just hiding their problems.  I must say the server handled the failure gracefully.  It was getting dual bit (uncorrectable) ECC errors and so upon boot it disabled the two (of four) offending DIMMS.  Very nice.

Also, I would like to take a moment to comment on Sun’s build quality in the x4100 and x4200 servers.  I opened a couple of them up today for the first time and I must say, I am *very* impressed with the physical build quality.  Sun has some very talented hardware engineers (almost over-built I would say).  The servers are made from some heavy gauge metal among other things.

So while I have changed my mind a bit on Sun’s build quality, they are certainly lacking some of the finer touches needed for x86 servers.  Their out of band management controllers (previously ALOM’s, now iLOM’s) have been quite the fiasco for us.  They also are a royal pain to bring all the different firmwares/drivers up to “known good sets”.  Dell has quite a nice tool for this.

One of the tech’s also did mention that there was a firmware update for the power supplies to keep them from powering the machine off in the event of a momentary power loss (like as a UPS kicks in).  Apparently they are programmed to power down after 20ms of lost power.  They should be able to run for over 100ms even after power is lost.

eprosenx Uncategorized

I discovered last night that Verizon Business (aka UUNET, MCI, alter.net, AS701) and Verizon proper (i.e. the Local Exchange Carrier here in Portland, AS19262) don’t appear to peer here.  That is a major shame since I am on Verizon FiOS and I can’t even access other businesses that use Verizon Business as their ISP here in Portland without bouncing of Seattle.

Check out this traceroute from my router on my FiOS connection to SilverStar Telecom who uses Verizon Business as one upstream:

plunger#traceroute www.silverstartelecom.com

Type escape sequence to abort.
Tracing the route to www.silverstartelecom.com (12.111.189.3)

  1 L100.PTLDOR-VFTTP-01.verizon-gni.net (72.87.39.1) 4 msec 4 msec 4 msec
  2 P2-3.PTLDOR-LCR-01.verizon-gni.net (130.81.32.164) 4 msec 4 msec 4 msec
  3 so-7-3-0-0.SEA01-BB-RTR1.verizon-gni.net (130.81.28.160) 8 msec 8 msec 8 msec
  4 0.so-7-1-0.XT1.SEA7.ALTER.NET (152.63.105.57) 8 msec 8 msec 8 msec
  5 0.so-6-2-0.XT1.POR3.ALTER.NET (152.63.105.233) 12 msec 16 msec 12 msec
  6 POS6-0-0.GW9.POR3.ALTER.NET (152.63.104.249) 12 msec 16 msec 12 msec
  7 IT-S-Star-gw.customer.alter.net (157.130.177.118) 12 msec 16 msec 12 msec
  8 sst-pit-6509-gi25-2-gsr12-gi60.silverstartelecom.com (66.206.80.21) 12 msec 16 msec 12 msec
  9 www.silverstartelecom.com (12.111.189.3) 12 msec 16 msec 16 msec
plunger#

What a bummer.  I hope they rectify this situation soon!

-Eric

eprosenx Uncategorized

When evaluating a hosting provider, colocation facility, or an ISP, one of the most important aspects is “How well peered are they?”  In this day and age you certainly want to go with an organization that has redundant connections.  In general, the more entities your partner is directly connected to, the less impact individual failures will have, and the lower your latencies for connectivity will be.

The best way to quickly determine who a given provider is peered with is by looking at BGP routing tables as seen by other networks in the world.  We are very fortunate that the Route Views Project is available, which is based out of the University of Oregon (I feel dirty now linking to U of O since I am a Beaver after all).

The route views project maintains a number of routers that are peered with routers from numerous different backbones.  These peering sessions exist not for the purpose of routing packets, but instead so that people can login to a route-views router and see what other networks think the best route is to someplace, and also so that the folks from the route views project can log data in order to allow various analytics later down the road.

Let’s say you are interested in determining the upstream peers for SilverStar Telecom (an ISP located in Portland with their routing core in the Pittock building).  You must first determine an IP address that resides within their network.  For the sake of this example we will do a dns lookup on www.silverstartelecom.com which resolves to 12.111.189.3.

Once you have an IP you wish to look up, telnet to route-views.routeviews.org and login as username “rviews”:

                    Oregon Exchange BGP Route Viewer
          route-views.oregon-ix.net / route-views.routeviews.org

 route views data is archived on http://archive.routeviews.org

 This hardware is part of a grant from Cisco Systems.
 Please contact help@routeviews.org if you have questions or
 comments about this service, its use, or if you might be able to
 contribute your view.

 This router has views of the full routing tables from several ASes.
 The list of ASes is documented under “Current Participants” on
 http://www.routeviews.org/.

                          **************

 route-views.routeviews.org is now using AAA for logins.  Login with
 username “rviews”.  See http://routeviews.org/aaa.html

 **********************************************************************
User Access Verification

Username: rviews
route-views.oregon-ix.net>

Issue the “show ip bgp 12.111.189.3″ command

route-views.oregon-ix.net>show ip bgp 12.111.189.3
BGP routing table entry for 12.111.189.0/24, version 17338865
Paths: (33 available, best #22, table Default-IP-Routing-Table)
  Not advertised to any peer
  7660 2516 3356 32869
    203.181.248.168 from 203.181.248.168 (203.181.248.168)
      Origin IGP, localpref 100, valid, external
      Community: 2516:1030
  3549 1239 32869
    208.51.134.254 from 208.51.134.254 (208.178.61.33)
      Origin IGP, metric 0, localpref 100, valid, external
  3582 3701 32869
    128.223.253.8 from 128.223.253.8 (128.223.253.8)
      Origin IGP, localpref 100, valid, external
      Community: 3582:466 3701:392
  701 32869
    157.130.10.233 from 157.130.10.233 (137.39.3.60)
      Origin IGP, localpref 100, valid, external
  3333 3356 32869
    193.0.0.56 from 193.0.0.56 (193.0.0.56)
      Origin IGP, localpref 100, valid, external
  7500 2497 701 32869
    202.249.2.86 from 202.249.2.86 (203.178.133.115)
      Origin IGP, localpref 100, valid, external
  3277 3267 9002 3356 32869
    194.85.4.55 from 194.85.4.55 (194.85.4.16)
      Origin IGP, localpref 100, valid, external
      Community: 3277:3267 3277:65321 3277:65323
  2828 7018 32869
    65.106.7.139 from 65.106.7.139 (66.239.189.139)
      Origin IGP, metric 3, localpref 100, valid, external
  2914 7018 32869
    129.250.0.11 from 129.250.0.11 (129.250.0.51)
      Origin IGP, metric 5, localpref 100, valid, external
      Community: 2914:420 2914:2000 2914:3000 65504:7018
  2914 7018 32869
    129.250.0.171 from 129.250.0.171 (129.250.0.79)
      Origin IGP, metric 1, localpref 100, valid, external
      Community: 2914:420 2914:2000 2914:3000 65504:7018
  852 174 7018 32869
    154.11.98.225 from 154.11.98.225 (154.11.98.225)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 852:180
  852 174 7018 32869
    154.11.11.113 from 154.11.11.113 (154.11.11.113)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 852:180
  12956 1239 32869
    213.140.32.146 from 213.140.32.146 (213.140.32.146)
      Origin IGP, localpref 100, valid, external
      Community: 1239:100 1239:123 1239:999 1239:1000 1239:1010 12956:321 12956:
4003 12956:4030 12956:4300 12956:18500 12956:28430 12956:28431
  3582 3701 32869
    128.223.253.9 from 128.223.253.9 (128.223.253.9)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 3582:466 3701:392
  8075 3356 32869
    207.46.32.34 from 207.46.32.34 (207.46.32.34)
      Origin IGP, localpref 100, valid, external
  286 3549 1239 32869
    134.222.87.1 from 134.222.87.1 (134.222.86.1)
      Origin IGP, localpref 100, valid, external
      Community: 286:18 286:19 286:29 286:888 286:900 286:3001 3549:2355 3549:30
840
  16150 3549 1239 32869
    217.75.96.60 from 217.75.96.60 (217.75.96.60)
      Origin IGP, metric 0, localpref 100, valid, external
      Community: 3549:2773 3549:31208 16150:63392 16150:65321 16150:65326
  2905 701 32869
    196.7.106.245 from 196.7.106.245 (196.7.106.245)
      Origin IGP, metric 0, localpref 100, valid, external
  3561 701 32869
    206.24.210.102 from 206.24.210.102 (206.24.210.102)
      Origin IGP, localpref 100, valid, external
  3257 3356 3356 3356 32869
    89.149.178.10 from 89.149.178.10 (213.200.87.91)
      Origin IGP, metric 10, localpref 100, valid, external
      Community: 3257:8091 3257:30042 3257:50001 3257:54900 3257:54901
  4826 3356 32869
    114.31.199.1 from 114.31.199.1 (114.31.199.1)
      Origin IGP, localpref 100, valid, external
  3356 32869
    4.69.184.193 from 4.69.184.193 (4.68.3.50)
      Origin IGP, metric 0, localpref 100, valid, external, best
      Community: 3356:3 3356:22 3356:90 3356:123 3356:575 3356:2012 65002:0
  6079 3356 32869
    207.172.6.20 from 207.172.6.20 (207.172.6.20)
      Origin IGP, metric 0, localpref 100, valid, external
  6079 3356 32869
    207.172.6.1 from 207.172.6.1 (207.172.6.1)
      Origin IGP, metric 0, localpref 100, valid, external
  812 6461 701 32869
    64.71.255.61 from 64.71.255.61 (64.71.255.61)
      Origin IGP, localpref 100, valid, external
  6939 3549 1239 32869
    216.218.252.164 from 216.218.252.164 (216.218.252.164)
      Origin IGP, localpref 100, valid, external
  1668 7018 32869
    66.185.128.48 from 66.185.128.48 (66.185.128.50)
      Origin IGP, metric 511, localpref 100, valid, external
  6539 3561 1239 32869
    66.59.190.221 from 66.59.190.221 (66.59.190.221)
      Origin IGP, localpref 100, valid, external
  1221 4637 3356 3356 3356 32869
    203.62.252.186 from 203.62.252.186 (203.62.252.186)
      Origin IGP, localpref 100, valid, external
  6453 1239 32869
    195.219.96.239 from 195.219.96.239 (195.219.96.239)
      Origin IGP, localpref 100, valid, external
  7018 32869
    12.0.1.63 from 12.0.1.63 (12.0.1.63)
      Origin IGP, localpref 100, valid, external
      Community: 7018:2000
  6453 1239 32869
    207.45.223.244 from 207.45.223.244 (66.110.0.124)
      Origin IGP, localpref 100, valid, external
  2497 701 32869
    202.232.0.2 from 202.232.0.2 (202.232.0.2)
      Origin IGP, localpref 100, valid, external
route-views.oregon-ix.net>

This will give you an extensive list of routes which you can use to reach SilverStar.  On the first line of the output you can see that 12.111.189.0/24 is the most specific route in the BGP table that matches 12.111.189.3.  Below that line, are a number of entries, each starting with a list of AS numbers on the least-indented line.  Let’s use the 4th item as an example.  It simply contains 701 32869.

If you look up the rightmost ASN (which is the originating ASN for this prefix), you will see that it is registered to SilverStar Telecom (as you might expect).  To look this up you can go to www.arin.net and enter AS32869 into the whois search box.

Now lets take a look at the AS number directly to the left of 32869 which in this case is also the first entry in the list, 701.  By virtue of being adjacent in the list, this means that SilverStar telecom advertised 12.111.189.0/24 to AS 701.  Furthermore, since 701 is the first leftmost entry in the list, it tells us that AS 701 peers directly with the route-views router.  If you look up AS 701 you will see it is registered to MCI (aka Verizon Business).  So Verizon Business is one of SilverStar Telecom’s upstream providers.

Let’s move on and take a look at the third entry in the list, 3582 3701 32869.  If we translate those entries to entity names by using whois, we can see it equates to University of Oregon -> NERO Net -> SilverStar Telecom.  In this case, SilverStar peers directly with NERO (presumably across NWAX).  Granted I am certain NERO does not provide “transit” for SilverStar, but it is notable in that SilverStar makes the effort to connect with others locally.

Now to speed up this process a bit, all we really care about is what AS number is just to the left of SilverStar’s ASN (32869) in each entry (that we have not already looked up and recorded.  Using this method I have generated the following list:

• 3356 – Level 3 Communications
• 1239 – Sprint
• 701 – MCI (Verizon Business)
• 7018 - ATT
• 3701 – NERO (Network for Education and Research in Oregon)

I must say, that is pretty impressive connectivity for Portland.  Verizon Business and ATT both actually have routing cores in Portland.  Sprint and Level 3 don’t and so you have to terminate circuits on routers in Seattle (or California).

That is all there is to it.  You simply login to the route views router and see what other routers think their best pathshould be to the network in question.  It is worth noting however that this is certainly not a 100% full view of the world.  It is very likely that SilverStar peers directly with other organizations (for non-transit traffic) but that we have no visibility into that since none of the downstream routers from that peering share their view of the world with the route views project.

For the most part however, the route views project has visibility into enough sites to see which major backbones a given ISP is attached to.  One other caviot to add however is that this will only give you an idea of how traffic gets *to* SilverStar Telecom, and not what outbound routes from SilverStars network packets will take.  It is possible that SilverStar is also hooked to another ISP (like XO communications) but that for some reason they don’t advertise 12.111.189.0/24 out that connection, or they use some metric to make it the least preferred route.  SilverStar may still route traffic out the XO connection even though no traffic comes in that way (I know for a fact though that SilverStar is not hooked to XO).

So go ahead and check out who your ISP is peered with!  You may be plesantly surprised (or disappointed).  This is a great way to double check what the sales droids tell you.  I have seen cases where ISP’s continue to maintain even a single T-1 to a provider in order to say that they are connected to them, while in reality they don’t route any traffic with them.  (or more likely, they have IP address space that belongs to that provider that they don’t want to have to re-number)

-Eric

eprosenx Network, Telecom

For quite some time now I have been annoyed at Cisco for not releasing a 64 bit edition of their IPSec VPN client.  As far as I am concerned, their plan has been to force everyone over to the AnyConnect VPN client (SSL VPN) which does support 64 bit clients (i.e. Windows Vista 64 bit).

Oh, and by-the-way the AnyConnect Premium client costs $1,250 (MSRP) for a 10 concurrent user license on a 5510, where as the IPSEC client is FREE for unlimited users.

On a recent trip to Costco I was amazed at what percentage of the systems now being sold were coming with 64bit Windows Vista.  There are becoming more-and-more home users that can’t VPN in anymore with the IPSEC client.

While I am still unhappy with Cisco for artificially forcing people over to AnyConnect client, there is some amount of relief in sight to this issue.

Cisco just announced this week at RSA some new features that will be included in the ASA 8.2 code.  One of which is a new AnyConnect license level “AnyConnect Essentials” which according to my sources will be “Almost Free” (you can decide what that means for yourself).  This license will provide basic VPN  access, but not include the clientless web portal stuff or Cisco Secure Desktop (basically the stuff that I would rather not have to support anyway).

The other cool feature I am looking forward to is a new Botnet detection capability.  Basically the ASA will periodically download signature files from Cisco that tell it what traffic to look for.  If the ASA observes internal machines connecting out to known Botnet controllers it will be able to report on them.  There will be a yearly fee to enable this, so I am curious to see what they charge.

No word yet on availability dates for 8.2 or official pricing.  You can find more details on what is in 8.2 here.

-Eric

eprosenx Cisco, Network

Until recently I never really realized how significant a role Oregon plays in the Pacific undersea cable business.  Apparently it is easier to get permits to land cables in Oregon than it is in California or Washington and our undersea geography is conducive to such projects.

As I have dug deeper into this topic, I put together a spreadsheet of all the different cables that land in Oregon (that I am aware of).  As usual, if I am missing anything, please send me an email.

I am working to add the cable landing stations and cable termination stations to my Portland/Oregon telecom map.

I find it disappointing how little “peering” we have going on in Oregon considering the amount of bandwidth flowing through the state up to Washington, down to California, East to Boise, off the coast to Alaska, and further West to Asia.  With the addition of the new immense capacity of the TPE cable Oregon has even more data flowing through it than ever before.

And just in case you want to know more about these cables, here is a list of all the references I have come across:

• http://bandoncable.org/
• http://www.ofcc.com/
• http://www.alaskaunited.com/
• http://www.southerncrosscables.com/
• http://www.acsalaska.com/business/enterprise/cable-system.asp
• http://tpecable.org/
• http://www.tatacommunications.com/map/chooser/fullscreen.html
• http://www.iscpc.org/cabledb/Eastern_Pacific_Cable_db.htm
• http://en.wikipedia.org/wiki/List_of_international_submarine_communications_cables
• http://en.wikipedia.org/wiki/List_of_domestic_submarine_communications_cables
• http://en.wikipedia.org/wiki/NPC_(cable_system)
• http://en.wikipedia.org/wiki/TPC-5CN_(cable_system)
• http://en.wikipedia.org/wiki/CUCN_(cable_system)
• http://en.wikipedia.org/wiki/Southern_Cross_Cable
• http://en.wikipedia.org/wiki/VSNL_Transpacific
• http://en.wikipedia.org/wiki/TPE_(cable_system)

-Eric

eprosenx Network, Telecom