Sunday, March 16, 2008

Late Night Troubleshooting

Unfortunately, issues continued late into the night. The joy of working on intermittent issues. On several occasions we thought we had it resolved, only to have the problems return later.

As it turns out, a transparent firewall that permits everything still runs packet inspection. We placed one in between two mail servers. It was inadvertently inspecting SMTP and occasionally killing communication between the servers. This only seemed to happen with a high message load.

Over the years, Cisco has been driving me and many others crazy with smtp. This was actually the first place I looked on our external firewall, and it was not present. I never even thought to look at the transparent firewall until later.

A primer on mailguard, etc.
Since PixOS 4ish, Cisco dabbled in application inspection with the mailguard feature. Since SMTP only has a handful of commands (HELO or EHLO, DATA, etc.) mailguard attempted to only allow these commands and to play with some banners and things as well to help protect the mail server.
In Pix 6ish this was now called fixup smtp 25
And now in Pix 7 it's called inspect esmtp

And over all of these years, the general rule is always to disable it immediately.

There are certain things that Cisco does that amazes me. Why, when after 10 years of configurations being changed out of the box, does Cisco not just make them default? Off the top of my head, some examples are
  • Turn off SMTP proxying/fixup/inspect/mailguard
  • Disable auto-summarization
  • Turn on service password-encryption
  • Turn on service timestamps
  • Automatic "terminal monitor"
I'm sure Cisco would say it's because of compatibility issues with those upgrading from older code. But come on, somebody upgrading to 12.4 should at least have some inkling about auto-summarization.

1 comment:

Anonymous said...


Congrats on your CCIE cert. Great idea in starting a blog dedicated to post-ccie. not many of them around.

i'm also thinking of getting a ccie.

but this post is what raises an eyebrow.

i hate working late nights. i've worked the third shift at a NOC a long time ago and i hated it.

but is IT, right? it comes with the terretory.

i was just wondering, on average - how much weekend/late night work are you involved in.

i'm under the impression that after a CCIE you can kinda dodge IT late nights.

if you have time, you should let us know on avg. how many late night and weekends do you work. or has obtaining the ccie increased more late nights and weekend work?


p.s. i keep hearing good things about the cisco voice certifications.