Back From Silence

I haven't posted much as of late. It's not because I didn't have a lot to say, that's for sure. A couple of months ago I got the itch for a career change. Fortunately, having a CCIE makes this relatively easy to do--recession or no recession.

If any of you out there are concerned about the occasional stories about how the CCIE is becoming worthless, let me assure you it is definitely not. Posting my resume on Monster yielded a barrage of opportunities.

After everything was said and done, I'm now working for Cisco Systems in RTP. Everything is going great and it's truly an excellent company to work for. Now I have the unusual experience of going from being one of the most knowledgable in my company to realizing I have a long way to go.

Long story short, my previous story of becoming board will not be coming to fruition any time soon.

What to do next?

Bottom line, I'm torn. I know I have a lot to learn in a number of areas, but I'm not sure where to go next.

Do I go for a CCSP or CCVP? Or straight to the CCIE written in voice or security?
I'm a little reluctant to work on a second CCIE because I'd like to reap some of the benefits of the first one before working on the next.

My weaknesses in Security have shown over the past few weeks when I've missed little things here and there.

And then there's other non-Cisco options such as CISSP or Ethical Hacker. This summer I'm finishing a Master's in Computer Science, so I just don't think those two certs would matter much with a Master's and a CCIE.

Then there's always the thought of going for an MBA or a PhD in Computer Science. Neither seem too appealing to me at this time.

April Fools RFC

I look forward to these every year.

RFC 5241 - Naming Rights in IETF Protocols
RFC 5242 - A Generalized Unified Character Code: Western European and CJK Sections

I've also added a couple of my all-time favories to the list of links to the right.

More Issues

Naturally, the turning off of DNS doctoring has created more issues. This was, to a certain extent, to be expected. But with our public website being down, it was the surefire way to fix it, so we went with that solution.

The issue now lies on the inside network. Anyone who accesses a site via its public FQDN now must use the outside public IP rather than the inside private IP. As long as sites resided in the DMZ, this was taken care of by reconfiguring the static translation to use the outside public instead of the inside private.

There are two issues where this could not be resolved, however. First, if the server resides on the private network, instead of the DMZ. Unfortunately I inherited a bunch of these. I'm trying to get them removed, but still have a ways to go. Fortunately, I haven't found one that was a show stopper yet. If I do, I'll probably have to NAT it on our inside core switch. But I'd like to avoid that if possible.

The second issue is for people who accesses our DMZ services from remote sites across the VPN. Since they now reference a public instead of private address, they traverse the public Internet. This has created issues for people trying to do things like edit the public website remotely. Naturally the firewall doesn't allow this from the public internet.

We've been able to survive the little issues thus far. We also discovered the reason for the problem. Sever static NAT translations that I inherited were applied backwards. This was causing DNS doctoring to occur to the public internet! I'm not sure why this wasn't broken before.

So we know the fix, now I just have to convince my VP that we won't break the public website again. It's looking like it will be at least 2 weeks before we can put the change through. I just hope we don't put in too many band-aids between now and then that we can't go back to the correct way.

More Issues

Some previous firewall configuration was carried over as part of the migration as well. Unfortunately, this created some more problems. For some reason, DNS doctoring was turned on pretty much everywhere. It was being overused to the point that DNS replies of internet addresses were getting overwritten with DMZ addresses. Naturally, this caused a number of internet services to fail.

Now I really haven't messed wtih DNS doctoring much since the alias days. It seems to be a lot easier now, since all one needs to do is to add the dns option to the static translation.

But the real question is, why wasn't this failing before the upgrade. I don't know. I turned off DNS inspection to kill this. That fixed the issue, but it took time to propogate.

The Default DNS TTL is 3 hours, so anyone who grabbed a bad record would have to wait up to 3 hours for their upstream DNS servers to get corrected.

To check the TTL remaining, nslookup can be used

c:\>nslookup
Default Server: cns.manassaspr.va.dc02.comcast.net
Address: 68.87.73.242
> set debug
> www.somecompany.com
Server: cns.manassaspr.va.dc02.comcast.net
Address: 68.87.73.242
------------------------
Got answer: HEADER: opcode = QUERY, id = 5, rcode = NOERROR
header flags: response, want recursion, recursion avail.
questions = 1, answers = 1, authority records = 0, additional = 0
QUESTIONS: www.somecompany.com, type = A, class = IN
ANSWERS: -> www.somecomapny.com internet address = 10.1.1.100
ttl = 205 (3 mins 25 secs)
------------
Non-authoritative answer:
Name: www.somecompany.com
Address: 10.1.1.100
>

Late Night Troubleshooting

Unfortunately, issues continued late into the night. The joy of working on intermittent issues. On several occasions we thought we had it resolved, only to have the problems return later.

As it turns out, a transparent firewall that permits everything still runs packet inspection. We placed one in between two mail servers. It was inadvertently inspecting SMTP and occasionally killing communication between the servers. This only seemed to happen with a high message load.

Over the years, Cisco has been driving me and many others crazy with smtp. This was actually the first place I looked on our external firewall, and it was not present. I never even thought to look at the transparent firewall until later.

A primer on mailguard, etc.
Since PixOS 4ish, Cisco dabbled in application inspection with the mailguard feature. Since SMTP only has a handful of commands (HELO or EHLO, DATA, etc.) mailguard attempted to only allow these commands and to play with some banners and things as well to help protect the mail server.
In Pix 6ish this was now called fixup smtp 25
And now in Pix 7 it's called inspect esmtp

And over all of these years, the general rule is always to disable it immediately.

There are certain things that Cisco does that amazes me. Why, when after 10 years of configurations being changed out of the box, does Cisco not just make them default? Off the top of my head, some examples are

• Turn off SMTP proxying/fixup/inspect/mailguard
• Disable auto-summarization
• Turn on service password-encryption
• Turn on service timestamps
• Automatic "terminal monitor"

I'm sure Cisco would say it's because of compatibility issues with those upgrading from older code. But come on, somebody upgrading to 12.4 should at least have some inkling about auto-summarization.

Data Center Rebuild

Today we redesigned the core of our datacenter. Some of the configuration tasks included:

• HSRP with object tracking
• dot1q trunking to a router
• transparent firewall BPDU passthrough
• BGP advertise-map configuration

One of the primary benefits of this configuration was it allowed us to add some functionality to the network, while replacing a number of layer 3 switches with layer 2 ones. This allows us to use less expensive resources and redeploy the freed up devices to more suitable locations. It also simplifies the network, which is generally a good thing.

Just from my experience with the CCIE lab, I changed my normal plan of attack here. In the past I was more one who prefered to gut everything and replace it all at once. Now, I tend to follow lab strategy and do one piece at a time, verifying as I go.

As with the lab experience, things go much slower this way, but it gets done right. When something doesn't come up I know exactly where to look for the issue instead of having to troubleshoot everything.

As expected, there were a few bugs to work out, but just about everything ended up as designed. The great work up front and during the implementation by the whole team really paid off.

Some of the learning experiences were

• A firewall in transparent mode can pass layer 2 information, but not CDP. A ethertype access list is required to allow BPDUs to pass. This was necessary so that a backup link can be used while letting rstp block the link. If this didn't work, another option would have been the "switchport backup" option for the 2560/3560 series switches, which functions a bit like a layer 2 version of the backup interface router configuration command.
• It always helps when default gateways are set correctly
• When using Hyperterminal to paste commands, make sure to set the character delay to prevent buffer overflows
• Duplicate IP addresses are very bad things

Most of these were minor and corrected quickly.