Tuesday, March 31, 2009

Next Steps

First off, I wanted to say thanks to everyone's comments over the past couple of days. I'm now trying to figure out what to do next.

I scheduled the lab this week because I have a family wedding to attend at the beach next week, so I'll be taking some time to relax and unwind. The big question is what to do next.

My score on the lab ended up right around 50%. I'm not surprised because not only did I spend all my time troubleshooting and point hunting, but I didn't have any time to verify (since I had no shot at even an 80, what was the point?). Typically I like to spend the last hour just going over the entire lab step by step verifying configs.

I could repeat the IE labs and lab up some of the SP mini scenarios . I really do believe 90% of the lab was covered by the IE labs. I would just rank the real thing as more of a 9 on the difficulty scale. I was studying for it like it'd be a 7, as was the case for the R&S lab. It turned out the actual lab was a little harder than the IE labs.

Another option would be to use IPexpert. I used them as well for R&S. I really found the annoying lack of clarity in the IPexpert tasks to be great practice for the lab.

I've also seen some folks recommend iementor.com's products for SP.

It looks like I'll have to wait until July to retake the lab, unless I want to schedule it for the week of Memorial Day, which I really don't.

I also think I'm going to buy a hard copy of at least one MPLS book. I've done ALL of my studying via Books Online and the DocCd, and I think I'd like to have something physical to thumb through.

Monday, March 30, 2009

No Chance

I'm still waiting for the score report, but I know I don't have a chance to pass. Even if I got everything right that I thought I did, I'd still be left with less than 80.

So what happened? I can blame a little bit on poorly written tasks, but for the most part I simply wasn't ready.

There are still too many topics that I can figure out if I spend 15 or 20 minutes troubleshooting. I need to be automatic with all of these.

I lost an hour troubleshooting a stupid mistake that I would have caught right away if I had verified properly. What's worse is I did I verify it, but I missed something. Not only did I lose valuable time, but I kind of spooked myself when I got stuck on something so vital. By time I found it I was a little burned out as well.

I also didn't stick to my game plan. For some silly reason, I opted to skip the VPN diagram right away. I was having a hard time understanding some of the requirements, so I chose to jump right in instead of doing a diagram. What the heck was I thinking? Regardless, this may have made the VPN portion take longer than normal, but it wasn't a killer.

In a nutshell, there are 4 or 5 topics that I need to be COMPLETELY automatic on. I need to have the ability to look at the topology and already know the issues and how to work around them. This is the level I was at when I passed R&S.

When you can look at the topology and think "I bet they're gonna throw this, that, and this other thing at me, and here's how I'll work around it", then you're ready.

All in all, it was a fair exam. If I was a little more solid in 2 or 3 more areas, and had another hour, I believe I would have passed.

Internetwork Expert did a great job for preparing me for the exam. There material is great, I just plain wasn't ready yet. I need to put more time in.

I'm waiting for the score report now to see how I fared on the tasks I think I completed. If my total score is in the 60% range, then I'm right on track. If it's much lower than that, then I really misunderstood some of the requirements or my attention to detail faltered.

I plan on shooting for it again in May.

Sunday, March 29, 2009

Game Plan for tomorrow

Here is how I plan to attack the lab.

1. Contrary to popular strategy, I am NOT going to do a read-through the first time. In the past I've found doing a read-through to be a waste of time because my short-term memory just ain't that good.

2. If there are troubleshooting tasks, review the config and spend 30 minutes or so trying to spot them.

3. Attack L2, IGP, and EGP immediately after. I'll go ahead and configure authentication/encryption type tasks that aren't required for core connectivity, but I won't hesitate to skip them if I hit a snag.

4. Diagram entire BGP/VPN topology. Note that I wait until core is complete before doing this. I like having a good feel for the core before starting the diagram.

5. Complete the VPN configuration. While I'm comfortable with TE now, I'll consider skipping it if it's just a few points because it just seems to have a habit of breaking other things. If I can have VPN completed before lunch, I'll be extremely happy because that should give me plenty of time to polish off the other tasks and do verifications.

6. Diagram and complete multicast. Nothing too fancy of the diagram, I just find it greatly helps me spot rpf and inter-AS issues when I have a separate mcast diagram drawn out. Unfortunately, I know how nasty multicast can be from the R&S lab, so I need prepared to give up these points altogether if need be.

7. Complete the rest of the lab. Consider skipping any traffic filtering related security tasks altogether. If I'm feeling confident in my points, I'm not even going to bother with them because I'm just too afraid of breaking my core. Seems like there's always a routing protocol or tunnel involved that get broken. If I must filter, I'm going to throw an explicit deny log at the end of the access-list so I can see everything that's getting dropped.

At this point, hopefully I have at least an hour left to go back through the lab, step by step, to verify everything. When I passed R&S, I found at least 6 points doing this that were due to stupid omissions (e.g. only applying QoS policy to R5 instead of R5 and R6).

Then, with 30 minutes left, I won't make any more configuration changes unless ABSOLUTELY necessary.

My speed has been fine though, so I'll be a little surprised if I don't have the whole lab wrapped up with 2 hours to spare. If I do, I'll probably keep doing verifications and annoy the proctor with the pickiest little questions I can think of until the labs over or the proctor kicks me out.

And here's what worries me:
Layer 2:
PPPoE. I've done a few basic configs, but it seems like it could get pretty complicated.
l2tp/AToM. There really isn't much to it, so it shouldn't give me any problems.
ATM: I have the basics down, but if some knobs get thrown my way I could have issues.

IGP: IS-IS knobs make me a little nervous.

EGP: I could have spent more time on advanced BGP features like orf and fast external failover. But these seem to be covered well in the doccd.

MPLS: ldp knobs, such as funky neighbor attributes, label filtering, and label retention

VPN: Making sure to spot if/where I need to use domain-id, sham links, and soo. Broken LSPs can be nasty to figure out too.

I'm happy with MDTs and bgp ipv4 multicast now. I'm still a little uncomfortable with MSDP. I think I just have to keep in mind that if I don't have a pim path between two hosts, I'll need two RPs and will need to join them with MSDP.

QoS: They always seem to find something nasty I haven't thought of. I'll just need to spend a little extra time verifying that what I put on is working properly

Security: Bottom line, don't break the core

System Management and IP Services: Gotta find those knobs.

MPLS VPN Inter-AS Feature Guide

This document could be a life saver tomorrow. It's located under the 12.0S(29) feature guides.


It has a wealth of information on the send label command, route reflectors, and peering non-adjacent vpnv4 routers.

Saturday, March 28, 2009

IE SP Vol 2 Lab 10

Ok, I've changed my mind and decided to give lab 10 another try.

4:27pm lab started
4:35pm 1.1 complete, 3
4:43pm 1.2 complete, 3
4:45pm 1.3 complete, 3
4:49pm 1.4 complete, 3
Layer 2 complete, 12/12, 0:22

5:07pm 1.1 complete, 3
5:13pm 1.2 complete, 3
5:15pm 1.3 complete, 3
IGP complete, 9/9, 0:26

5:24pm 3.1 complete, 4
5:32pm 3.2 complete, 3 -- probably gonna need some next-hop-self's later but I'll add them when req'd.
EGP complete, 7/7, 0:17

5:42pm 4.1 complete, 3
6:01pm 4.2 complete, 3
6:09pm 4.3 complete, 3
6:24pm 4.4 complete, 3
MPLS complete, 12/12, 0:52

8:20am back from break
8:33am vpn diagram complete
8:49am 5.1 complete, 3
8:57am 5.2 complete, 3
9:58am 5.3 0/4 -- this one completely blew my mind. So it IS possible to have an interface participate in multiple vrf's using the vrf selection source command. I'm glad I decided to do lab 10 or I would have had no idea about this if I run into it tomorrow.

10:21am back from break
10:54am 4/4. Got this one, and it works beautifully! In summary, vrf leaking occurs on the way out so traffic can leave the vrf and go out to the global routing table. But on the way back in, the PE router needs to know which traffic goes to which vrf. The vrf selection commands enable the router to put the flows into the proper vrf on the way back to the P network. This essentially requires three steps.

1. Create the vrf and assicated rd and rt's on the PE router
2. Leak the vrf routes out to the global routing table for the downstream C networks
3. Inject the upstream C routes into the global routing table so the downstream C networks have reachability back
4. Specify which source networks are associated with which vrfs via the vrf selection source command
5. Enable ip vrf select source and ip vrf receive on the PE upstream interface
6. Resolve any next-hop rechability and MPLS LSP issues.

Sounds like a lot, but after doing a couple it's really not too bad.

11:22am 5.5 complete, 4, trick here was default-information-originate on R4 ipv4 address family.
11:48am 5.6 not complete, 0/4. I got hung up because the debug ip packet didn't show my packets flowing. All I needed to do was to advertise my link in bgp to BB1, but I never bothered to try because I thought something else was broken. Most likely these packets were being cef switched so I didn't even know they were flowing. Painful 4 points to give up...

VPN complete, 14/22, 2:05, 4:02 total

6.1 not complete, 0/3. They got me on this one. I tried to put this on the TE tunnel and the pim adjacencies wouldn't come up. I considered creating a gre tunnel, but figured there was some knob that would allow pim to talk across the TE tunnel. Nope, a gre tunnel was the solution.

6.2 not complete, 0/3. This makes sense at least, now. The new loopback can't pass the rpf check since R4 doesn't have a route to it. We don't want to add full ip reachability, we just want to pass the rpf check. So we add a bgp multicast peer and add the loopback there, so R4 can do the rpf check on the new loopback. Got it.

12:27pm 6.3 complete, 3/3. Have to carry the ipv4 multicast route out to R1.

Multicast complete, 3/9, 0:39

12:42pm 7.1 complete, 3/3
12:51pm 7.2 complte, 3/3
12:54pm 7.3 complete, 3/3
1:00pm 7.4 not complete, 0/3, they got me again, I forgot about the nat. Sad, because I even had NAT written on the diagram.
QoS complete, 9/12, 0:33

1:17pm 8.1 complete, 2/2. More trickery, do to this, besides mpls, ttl expiration messages must also be filtered. The solutions guide didn't catch onto this even, as it only prevents traffic that passes through the TE tunnels from showing the hops. Not all traffic is going through the tunnels!

1:21pm 8.2 not complete, 0/3. I misunderstood and didn't filter the tunnels too. I'm sure a customer would just love an ISP who filtered all tunneling from their network lol.

1:26pm 9.1 not complete, 0/3. I was looking for something totally different.

1:30pm 9.2 complete, 3/3. For once I didn't forget to enable traps.

1:31pm 10.1 complete, 2/2. Scan time

Lab 10 complete, 71/100, 5:45

Wow, not bad for a difficulty 9 lab. There were a few new concepts in this lab that was was able to catch on to and am sure I would be able to configure them next time. I'm actually feeling really good right about now.

Friday, March 27, 2009

IE SP Vol 2 Lab 7 Repeat

Here goes my 2nd attempt at lab 7.

2:15pm lab started
2:25pm 1.1 complete, 3 points
2:29pm 1.2 complete, 2 points
2:34pm 1.3 complete, 3 points
2:36pm 1.4 complete, 3 points
2:42pm 1.5 complete, 3 points
L2 complete, 14/14, 0:27

Taking CoD break

12:40am back from break
12:50am 2.1 complete, 3 points
12:57am 2.2 complete, 2 points
1:00am 2.3 complete, 3 points
1:01am 2.4 complete, 2 points
IGP complete, 10/10, 0:21

1:21am 3.1 complete, 3 points
1:33am 3.2 complete, 2 points
1:46am 3.3 complete, 2 points
1:52am 3.4 complete, 3 points
1:54am 3.5 complete, 3 points
2:03am 3.6 complete, 3 points
EGP complete, 16/16, 1:02

2:13am 4.1 complete, 3 points
MPLS complete, 3/3, 0:10

2:30am vpn diagram complete
2:39am 5.1 complete 3
2:54am 5.2 complete 3
3:22am 5.3 complete 3
3:42am 5.4 complete 3
3:46am 5.5 complete 3
4:22am 5.6 incomplete, 0/4 looked up the answer. This one still baffles me.
4:59am 5.7 complete, 3
5:14am 5.8 complete, 3
VPN mostly complete, 21/25, 3:01 (too long!!!)

5:24am 6.1 complete, 3 watch rfp checks on TE tunnel
5:44am 6.2 not complete, 0, not sure what vlan 7 is. Needed mpls ip on TE tunnel (AGAIN!!!)
6:15am 6.3 not complete, 0, but I did eventually get it working!
Multicast barely complete, 3/9, 1:01 (still making progress...)

6:25am 7.1 complete, 3
6:52am 7.2 not complete. 0/3 I have it configured right but the marking doesn't seem to be occurring.
7:11am 7.3 complete, 3
QoS complete, 6/9, 0:54

10:34am back from break
11:25am 8.1 complete, 3 points--too much time!
11:30am 8.2 wrong, 0/2--no ip unreachable on null0 interfaces, I never would have thought of that!
11:34am 8.3 complete, 3.
Security complete, 6/8, 1:00

11:43am 9.1 not complete, missed snmp ifindex persist and didn't use absolute
11:49am 10.1 not complete, needed to use nat instead of policy routing

Lab 7 complete, 8:11, 79%
The single biggest thing that worries me about this lab is task 5.6. I understand that a route is needed for the PE router so the mpls bindings occur. What is beyond me is why on earth redistribute connected works. Just as I'm typing that, I actually bother to do a sh ip route on R4. Sure enough, shows up as connected, due to it being a peer neighbor route. Ok, I feel better now.

Thursday, March 26, 2009

IE SP Vol 2 Lab 4 Repeat

I'm going to try to do a repeat of labs 4, 7, and 8 before my big day. Here it goes...

4:44am lab started

Layer 2
4:55am 1.1 complete. 3 Vtp was acting up and I had to bounce both sides of the trunk int
4:58am 1.2 complete 3
5:04am 1.3 complete 3
5:07am 1.4 complete 3
Layer 2 complete, 12/12, 0:23

5:10am 2.1 complete 2
5:21am 2.2 complete 3, frame-rel map clns broad
5:25am 2.3 complete 2
6:06am 2.4 complete, 2 --got distracted
6:16am 2.5 complete, 3
IGP complete, 12/12, 1:09

6:27am 3.1 complete, 3
6:40am 3.2 complete, 4
EGP complete, 7/7, 0:34

6:43am 4.1 complete, 3
6:49am 4.2 complete, 2
6:54am 4.3 complete, 2
7:01am 4.4 complete, 3
7:26am 4.5 complete, 3
MPLS complete, 13/13, 0:46

taking lunch
10:33am back from lunch
10:43am vpn diagram complete
10:47am 5.1 complete, 3 points
10:56am 5.2 complete, 3 points
11:00am 5.3 complete, 3 points
11:18am 5.4 complete, 3 points
11:34am 5.5 complete, 3 points
11:53am 5.6 complete, 3 points
12:41pm 5.7 skipped, the export map won't take
BGP almost complete, 18/21, 2:08. Too much time spent troubleshooting 5.7!

taking another lunch.
3:56pm back from lunch
4:10pm 6.1 complete, 3 points
4:13pm 6.2 complete, 2 points
4:17pm 6.3 complete, 3 points
5:06pm 6.4 not complete, 0 points, needed to make msdp router pim dr.
for the second time, task 6.5 killed r1.
Multicast complete, 8/14, 1:10

Total time so far: 6:10

7.1 no issues, 3 points, 0:15
7.2 no issues, 3 points, 0:30
QoS complete, 6/6, 0:45

8.1 no issues, 3 points, 0:05
8.2 no issues, 3 points, 0:05
Security complete, 6/6, 0:10

System Management
9.1 0/0, 0:10, banner motd, and no motd on other ports
9.2 2/2, 0:05
9.3 3/3, 0:10
System Management complete, 5/8, 0:25

IP Services
10.1 2/2, 0:10

Lab complete, 88%. Of the 12 points I missed, at least 6 seem to be due to dynamips issues. The solutions guide shows I was doing mdt's and the nat exercise correctly, but they just didn't want to come up.

By far my biggest weakness is multicast. I'll take a stab at lab 7 tomorrow and will again spend a lot of time on the mulitcast portion. I think I remembered most of the key concepts today, I just ran into some small implementation issues.

Sunday, March 22, 2009

IE SP Vol 2 Lab 10

Here we go, the last of the labs I haven't looked at yet.

12:38am lab started

Layer 2
12:44am 1.1 complete, 3
12:48am 1.2 complete, 3
12:52am 1.3 complete, 3
1:02am 1.4 complete, 3, assuming pvc command satisfies requirement
Layer 2 complete, 12/12, 0:24

1:12am 2.1 complete, 3
1:28am 2.2 complete, 3. Unable to authen RIP. Verified no spaces, etc, assuming BB issue.
1:33am 2.3 complete, 3 points
IGP complete, 3/3, 0:31

1:48am 3.1 complete, 3. Tried confederations but ran into issues so went with local-as.

3:30am back from work break
3:46am 3.2 complete, 3 points
EGP complete, 6/6, 0:31

4:03am 4.1 complete, maybe--te w/o mpls on each interface

Well, after 2.5 hours of working on 4.2, I give up. As far as I can tell, I had everything right 2 hours ago. I ran into an issue where I was somehow missing my loopback 0 ip address, which wouldn't allow me to bring the tunnels up. I added this, and my tunnel through R9 worked fine. but the dynamic tunnel won't come up. I've checked and rechecked my interfaces, but it just won't work.

Saturday, March 21, 2009

IE CoD Service Provider Multicast

Since Multicast is obviously my weakest point now, I've decided to review the IE CoD for Multicast.

1. Don't touch mVPN until each PIM domain has been verified
2. Use debug ip pim to find "RPF lookup failed" messages
3. Enable pim between msdp peers so rp information can be exchanged
4. For multicast testing, getting a ping reply is not a great test. It is only really necessary to verify the downstream traffic is getting to the host. To get icmp replies, it may be necessary to enable multicast routing on the destination and enabling pim there, ensuring it is not the pim dr.
5. mBGP is solely for the purpose of advertising source ips for rpf lookup. Verify using show ip rpf.

Friday, March 20, 2009

IE SP Vol 2 Lab 9

Time once again to change things up for a timed lab. This is the second to last of the Vol 2 labs. It also has a difficulty of 7, which generally means its difficulty closely mirrors that of the real thing. So I need to pay close attention to my performance on this one.

12:15am lab started

Layer 2
12:26am 1.1 complete
12:28am 1.2 complete
12:45am 1.3 complete, fun with inverse-arp. Will probably need point-to-multipoint for ospf.
12:48am 1.4 complete
12:53am 1.5 complete
Layer 2 complete, 10/10, 0:38

12:58am 2.1 complete
1:00am 2.2 complete
1:02am 2.3 complete
1:08am 2.4 complete
1:10am 2.5 complete
1:19am 2.6 complete
1:22am 2.7 complete, had to manually redis loopback on r1
IGP complete, 15/15, 0:29

1:24am 3.1 complete
1:28am 3.2 complete
1:30am 3.3 complete
EGP complete, 7/7, 0:08

1:33am 4.1 complete
1:36am 4.2 complete
2:05am 4.3 complete
MPLS complete, 8/8, 0:35

2:33am forced break
3:59am back from break
4:05am 5.1 complete
4:11am 5.2 complete
4:30am 5.3 complete
4:55am 5.4 complete
5:05am 5.5 complete, as-override
5:26am 5.6 complete, domain-id
5:32am 5.7 complete, export map
5:38am 5.8 complete, one label still shows, but it's in vrf 100, so should be ok
5:47am 5.9 complete
VPN complete, 25/25, 2:16

Time so far: 4:06

6:36am skipping multicast, having issues with p2mp network
taking break
10:25am back from break

10:30am 7.1 complete, assuming it's ok to do this from the router side instead of the switch side
10:35am 7.2 complete
10:40am 7.3 complete
Qos complete, 8/8, 0:15

10:46am 8.1 complete, permit means don't do uRPF check
11:13am 8.2 complete, got pppoe going!!!
11:30am time's up, almost had it
Security incomplete 6/11, 0:44

System Management, no time, 0/3
IP Services, no time, 0/5

Total possible, 79/100.
Time spent: 5:03 :(. I had too many distractions during the night and by time 6:30am got here I was just too tired to go on. I KNOW I could have completed 8.3, 9.1, 10.1, and 10.2. So all that really leaves is multicast. If this were my actual lab, I'm about 99% sure I would have passed it.

Reviewing the solutions to see if I missed anything:
Task 5.8 added the keyword forwarded
For 6.1, it looks like I did what I needed to . I remembered autorp listener and nbma mode, so I'm not sure why I couldn't get it to work.

Realistically, if I had 8 hours, I would have definitely got 8.3, 8.4, and 9.1. Most likely 10.1 and 10.2 as well, but even if I don't include those I'm still looking at 87%, which is passing.

Tuesday, March 17, 2009

IE SP Vol 2 Lab 8

Layer 2
1.1 no issues
1.2 no issues
1.3 no issues
1.4 unfortunately I had to look this one up. All I needed to do was to create an mpls subinterface and configure mpls ip on it. I'm not sure why I couldn't figure that out. Been dealing with pvc's so much lately, and didn't even need one this time.

2.1 no issues
2.2 no issues
2.3 no issues
2.4 no issues

3.1 well I mapped the peer group, but I didn't actually use it. The task could have been a little more restrictive to actually for me to use it.

4.1 no issues
4.2 no issues

5.1 no issues
5.2 no issues, allowas-in
5.3 I used an export map, but the solutions guide wanted soo. Either way works fine.
5.4 I had to get creative on this one. A couple of TE tunnels with MED thrown in seemed to do the trick.
5.5 ok, the TE tunnel killed my redundancy. Now I could have probably done something really fancy with the tunnels to work around this, but it made more sense to get rid of them altogether and follow the solution guide. So I went with local preference, which for this task worked much better. Aside from that, some rip redistribution did the trick.
5.6 finally, an easy one

Monday, March 16, 2009

2 weeks out

I sit for the lab on March 30th, 2 weeks from today.

Thus far, I have completed the following:
IE Vol 1, all labs
IE Vol 2, labs 1-7 (4 and 7 were only partially completed)

At this point I feel pretty comfortable with most of the topics. I'm still a little concerned with the following:

MSDP - I just don't have the big picture on this yet
TE - I can get the tunnels up just fine now, but routing vpn traffic across the tunnel gives me problems
PPPoE - I just plain need more practice
AToM - Haven't touched it

My game plan is as follows:
Squeeze in labs 8, 9, and 10 during the week
Study SP scenarios during the week
3/21 - Repeat Lab 4
3/23 - Repeat Lab 7
3/28 - Whatever I need the most

The big thing I'm doing differently in studying for the SP lab vs R&S is I'm going slowly and banging away until I figure out the issues. I definitely like this approach better. But on these last 3 timed attempts, I'll need to do it like it's for real to help me decide which tasks to skip, etc.

Lab 6 Task 6 - 9

Ok, back to finish off lab 6.

6.1 no issues
6.2 accept-rp to only use certain rp's. Also, need to configure sparse-mode on links between domains.
6.3 advertise the remote router loopback into bgp ipv4 multicast.

7.1 I wasn't paying attention to scale. And yet another lab anomaly.

8.1 Almost, missed the discovery transport address. I don't understand why this is required at all. If I get a question like this on the lab I may add it just to be on the safe side, but it seems irrelevant, unless it is somehow locking down the address.

9.1 no issues

10.1 took me a bit of hunting but I finally found it. ip access-list logging


Even though on the last couple of labs I've been able to get the TE tunnels up, I haven't been successful in actually routing traffic across them

During troubleshooting, I'd see the TE tunnel in the forwarding table, and verified mpls packets flowing through the tunnel. But the packets always seemed to get dropped at the endpoint. I thought this could have been something to do with the implicit-null issue, but forcing explicit nulls never helped.

I also noticed the mpls forwarding table showed no label for the TE tunnel. I thought maybe this was just a quirk of TE tunnels.

As it turns out, I need to enable mpls ip on the tunnel interface.

IE SP Vol 2 Lab 7

I'll have to take a break from lab 6 so I can do my scheduled timed lab 7.

3:00am began
3:08am 1.1 complete
3:11am 1.2 complete
3:18am 1.3 complete, verified using debug ppp authen and shut/no shut int
3:21am 1.4 complete, ppp peer neighbor (default) route allows connectivity
...55 minute break...
4:24am 1.5 complete
Layer 2 complete, 14/14

..50 minute break...
5:05am 2.1 complete
5:07am 2.2 complete
5:32am 2.3 complete, redistribute ip level-2 into level-1 route-map
5:35am 2.4 complete
IGP complete, 10/10

5:47am 3.1 complete
5:58am 3.2 complete
6:02am 3.3 complete
6:07am 3.4 complete
6:15am 3.5 complete, route-maps can set next hops too
6:31am 3.6 complete
BGP complete, 16/16

6:37am 4.1 complete, looks like ldp uses
MPLS complete, 3/3

6:46am 5.1 complete
7:04am 5.2 complete, as override
8:24am 5.3 complete, export maps and playing with the next-hop route-maps
9:54am 5.4 skipped. Ugh, I had all sorts of unrealized issues here. I didn't have an isis adjacency on the atm link. For some reason, I needed to change the atm subinterface type to p2m so it matched with r2's interface only config. After this, I had to ensure all broadcasts were permitted on the pvc's so that mpls would come up. And after all that, once the TE tunnels were up, yet again traffic going across them would drop, so I had to shut the tunnel and move on.
10:10am 5.5 complete

Unfortunately a large number of distractions only let me get about 5 hours toward this lab before I ran out of time.

The final tally is:
projected: 55/100

Task notes:
1.5: Ha, I was right to change a3/0.12 to multipoint. I'd give myself extra credit for that if I could!
After reviewing most of the tasks, I caught on to most of the tricks. Even though I didn't get very far into the lab, I'm very excited with my progress!

Saturday, March 14, 2009

IE SP Vol 2 Lab 6 Continued

5.1 no issues
5.2 something in Dynamips crashed and I had to reload a couple of routers to get the 5-8 adjacency up.
5.3 no issues. Sham links are covered nicely on the doccd.
5.4 no issues.
5.5 no issues
5.6 no issues. I'm getting MUCH better with TE, this one only took me a few minutes.
5.7 this task was a major pain. I went ahead and made an assumption that only vlan78 was required. The first step was obvious, adjust bgp weights so the correct link was preferred. But it wasn't quite so simple. One of the PE routers refused to advertise the required routes. After way too much time troubleshooting, it turned out that this was due to sham-link costs. Adjusting these ensured the path through R5 was used.

The solutions guide just ignored the sham link altogether saying there was a conflict. It then went ahead and used an export map, which basically took the place of my weight adjustment. 6 one way a half dozen the other...

5.8 no issues

5.9 Yet another task that took a heck of a long time. The pings between ASs failed at first, naturally. It didn't take much time in debugging to determine that send-labels was required on the P-P bgp peers, since they were not running mpls with each other and did not split the vrf's out via subinterfaces. Next, I made sure the route-reflector left the next-hop-unchanged to allow the vpn traffic to flow PE-PE to ensure the vpn tag remained entact.

But no matter what I did, the upstream PE router was still seeing the bgp network as sending no label. I spent multiple hours troubleshooting this. Debug mpls packet is becoming my best friend.

So as it turns out, with a little help from the solutions guide, that the route-map is to blame. If a route map is used, set mpls-label must be configured in the route-map, otherwise this will override the send-label portion of the bgp neighbor config.

But even after I configured this, the CE router still couldn't ping AS54. After another hour or two troubleshooting, the problem seemed to be that the upstream P router was trying to send the traffic back via mpls instead of ip. I have no idea why this was happening, but a mpls encapulation failed message was occurring. Guessing that something may have just gotten hosed on the router in all my previous messing around, I gave up and just reloaded the upstream PE routers. Sure enough, everything came up upon the reload. Annoying, but it's working.

Too much time spent, but I think I got a lot of good troubleshooting experience.

5.10 Yet again more nastiness. The basic task was simple--summarize using an aggrigate address to ensure the longer-match prefix is preferred. An interesting trick. But for some reason, the path to R8 breaks once I set this. Tearing down the TE tunnel fixes it. So yet again i put a ton of time into troubleshooting this. It turns out that for some reason R1 is ignoring the sham-link when TE is operational.

A reboot of R1 seemed to take care of the sham-link issue, but my traffic is still not making it back to R8.

After another hour spent of troubleshooting, I'm giving up on this one. I'm going to write it off as some kind of weird incompatibility with sham-links and TE or something like that. Either that or something is still not clicking with TE that's not mentioned in the solutions guide either.

5.11 wow, no issues

5.12 of course, once again issues. This again deals with the sham link and R7 cannot be made to take the backdoor route to R8 to come into nat on R5 in the appropriate interface.

IE SP Vol 2 Lab 6 Continued

4.1 no issues
4.2 no issues

Some good information is contained in the solutions guide for verifying the ldp session is encrypted. I'll all for verifying everything, and fully believe it was the reason I was able to pass R&S the second time.

First, find your ldp/tdp session:

Rack1R4#sh tcp brie
TCB Local Address Foreign Address (state)
6520C0C4 ESTAB
65185A8C ESTAB

Then check out the appropriate tcb (port 711 or 646). Note the flags section.

Rack1R4#sh tcp tcb 6520c0c4
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Connection is ECN Disabled, Mininum incoming TTL 0, Outgoing TTL 255
Local host:, Local port: 646
Foreign host:, Foreign port: 28294

Flags: passive open, retransmission timeout, gen tcbs, md5,

Friday, March 13, 2009

IE SP Vol 2 Lab 6 (Untimed)

I'm gonna see if I can squeeze this lab in on an unscheduled day so I can make it through all of them before the big day. 16 days to go...

Layer 2
1.1 No issues
1.2 No issues
1.3 No issues
1.4 No issues
Layer 2 complete, 12/12.

2.1 No issues
2.2 No issues
2.3 No issues
IGP complete, 9/9

3.1 No issues
3.2 No issues
3.3 No issues
3.4 No issues
3.5 I ALMOST had this. Of course almost equals nothing in the lab. I figured out what needed to be redistributed to get the ebgp adjacency up. I also figured out some admin distance changes needed to be made in as 100. But what I didn't get right is getting traffic from R3 to R6 flowing across the R1-R4 link. I knew it needed to be done, I just couldn't figure out how, within the scope of current BGP adjacencies. I ended up looking this one up. The trick was to advertise BOTH loopbacks on each AS100 router. Then I could easily adjust the metric on the AS 200 ABRs during redistribution.
EGP complete, 12/15

Sunday, March 8, 2009

IE SP Vol 2 Lab 5

12:20am Lab started
12:26am Read through complete

Layer 2
12:33am 1.1 complete, 3 points
12:37am 1.2 complete, 3 points
12:46am 1.3 complete, 3 points
Layer 2 complete, 9/9 points

12:53am 2.1 complete, 3 points "Most appropriate" is awfully subjective
12:58am 2.2 complete, 3 points
1:10am 2.3 complete, 2 points
1:15am 2.4 complete, 3 points
IGP Complete, 11/11 points

1:29am 3.1 complete, 3 points, assuming confederations
1:33am 3.2 complete, 3 points
1:37am 3.3 complete, 3 points
1:43am 3.4 complete, 3 points
skipping 3.5 until neighbors are up
EGP incomplete, 12/15

1:55am 4.1 complete, 3 points
2:04am 4.2 complete, 3 points, changing ethernet link b/w to 100m.
2:13am 4.3 complete, 3 points
2:41am 4.4 incomplete, 0 points, tricky requirements and researching static routes for TE cost some time. Static routes are failing. I think it's a label popping issue.
MPLS complete, 9/13

2:48am 5.1 complete, 3 points
3:13am 5.2 complete, 3 points
3:20am 5.3 complete, 3 points
4:25am spent over an hour troubleshooting 4 measly TE points. Subtracting points.
5:00am spent another 30 minutes working on the earlier skipped bgp task. moving on...
5:18am 5.4 incomplete, 0 points. Finally getting somewhere...
5:29am 5.5 complete, 3 points. A little unsure about the last bullet point.
6:04am 5.6 complete, 3 points. Determined 5.4 was broken, removed.
6:31am 5.7 complete, 3 points
VPN complete, 18/21

6:36 am 6.1 complete, 2/2 points
6:41am 6.2 incomplete, 0/3, unsure how to get rp across mdt
6:56am 6.3 complete, 3/3, points. Unsure about data access-list
7:01am 6.4 incomplete, 0/2, testing failed, I believe it's an rp issue
Multicast complete, 5/10

7.1 incomplete, unsure how to match on traffic going to r6
7.2 incomplete, unsure how to match on traffic gong to r5
7.3 incomplete, same
7.4 skipping
7.5 skipping
Qos incomplete, 0/13

8.1 complete, 3 points

System Management
skipped. I can't remember the trick to this. Something to do with rotary I think?

Ip Services
10.1 complete, 3 points

Lab 5 complete. Highest possible score, 70/100. I ran into some TE issues, some VPN issues, and was pretty much clueless on QoS and Multicast.

Schedule Status

Well, I finished Labs 1-3 and got through the guts of Lab4, but didn't quite get to the fluff. I reviewed it briefly and there didn't seem to be anything too extraordinary in it. But when it comes to the CCIE, the devil is definitely in the details.

Tonight is my first timed lab. I'll be tackling Vol 2 Lab 5. Since I've been taking my sweet time so far, I have no idea how fast or slow I'm actually doing these things. It'll definitely change my plan of attack now that I'll be skipping some tasks.

Here is an updated schedule:
3/9: Vol 2 Lab 5
3/16: Vol 2 Lab 6
3/21: Vol 2 Lab 7
3/23: Vol 2 Lab 8
3/28: Vol 2 Lab 9

Now that leaves me without doing Lab 10. I think I'll try to squeeze lab 10 in during the weeks, as, at least in R&S, the last lab was some ugly impossible thing that was WAY above and beyond the lab requirements.

IE SP Vol 2 Lab 4 Task 6

6.1 no issues
6.2 no issues
6.3 Multicast boundary filters access-list denies--needs permit any at the end.

6.4 Ugh--I forgot all about msdp. I'm gonna have to do some more reading on it. I know I understood it at one point for R&S, but those brain cells have long since died off.

6.5 Crashed my R1. After multiple attempts I can't get it back up :(. There's pretty much no hope of me rebuilding it. I'll have to try some different IOS versions. But no time for that now.

IE SP Vol 2 Lab 4 Task 5

This one started out ok but got rough towards the end.

5.1 Complete, no issues
5.2 Complete, no issues
5.3 Complete, no issues
5.4 Ok, this task is just plan annoying. I hope the real lab doesn't require quite so many assumptions. I'm referring to the "Redistribute where necessary to advertise these prefixes". On top of redistribution, vrf's need to be assigned to interfaces not specified and bgp adjacencies need to be created that aren't specified. I hate when the lab does this. How am I supposed to know that we're allowed to create a bgp adjacency yet not a whole new vrf? Is this really a case of doing whatever I want that doesn't violate the lab requirements? I guess I'd just have to harrass the proctor on lab day...

5.5 Complete, no issues

5.6 I did this one all wrong. I knew it violated requirements but I went ahead an used a static route, simply because that's the way I've always done internet access before. I didn't even think about creating a new vrf and establishing the bgp session inside of it. I'll definitely have to commit that to memory.

5.7 I couldn't get nat working at all. It turns out that I needed to create a new loopback which reflects the nat pool. Something else I'll have to commit to memory, I guess. Another tidbit I should have realized--debug ip packet doesn't show MPLS traffic!!! Duh--MPLS is NOT IP. To track mpls, use debug mpls packets. But it doesn't show a whole lot, other than the labels. I wish there was a way to slow the router down and do deep inspection of mpls packets for debug purposes...

Thursday, March 5, 2009

IE SP Vol2 Lab 4 Task 4

4.1 Complete, no issues
4.2 Complete, no issues
4.3 Complete, no issues. neighbor send-label tells bgp to send the lables.
4.4 and 4.5 Complete--lots of issues!!!

Tasks 4.4 and 4.5 were a major pain. I'm still having a tough time with TE, but I'm getting there slowly but surely. I refused to look up the answers on these tasks and kept pounding away at it until I got it figured out.

Here's a little checklist for getting it setup.

1. Keep in mind that TE tunnles are unidirectional. A tunnel interface does NOT need to exist on both sides, as is common with gre/ipsec tunnels. It is for one-way transfer of traffic. As a matter of fact, by setting up tunnels on both directions, it can exhaust rsvp bandwidth and make bringing up all of the tunnels impossible.

2. mpls traffic-engineering tunnels must be configured globally and at the interface level on all routers in the path, including the head.

3. ip rsvp bandwidth must be reserved on all links in the path, including the head

4. The tunnel must have an ip address (unnumbered is fine), priority, and path configured.

5. If an explicit path is used, all hops must be valid and the correct name or idenitifer must be used.

And here is a list of verification steps:

1. Check sh ip rsvp int on all routers to ensure rsvp is enabled
2. Check sh mpls traf tun on the head to look for error messages and tunnel state
3. Check sh mpls traf top path to see the path the tunnel is taking

For establishing the explicit path, I like to do the following:
1. Configure the tunnel for dynamic
2. Shut down some links to force the tunnel to dynamically take the path you want
3. Configure the explicit-path list to mirror the output of sh mpls traf top path
4. Reconfigure the tunnel with the explicit-path

IE SP Vol2 Lab 4

Here we go, 3 days ahead of schedule, woohoo!

1.1 Complete, no issues
1.2 Complete, no issues
1.3 Complete, no issues
1.4 Complete, surprisingly. Even without inverse-arp or mappings, we can configure a p2p subinterface with a broadcast pvc. Per the solutions guide, the broadcast keyword isn't even required. I guess if you have a p2p subinterface with a single pvc, it assumes to send all traffic out that single pvc.

2.1 Complete, no issues
2.2 Complete, no issues. Minimum amount of commands means specify is-type under router isis instead of on each interface.
2.3 Complete, pretty poorly worded task, but adjusting the hello timers took care of it.
2.4 Complete, no issues, for md5 isis authen we need key-chains.
2.5 Complete, no issues

3.1 Complete, no issues
3.2 Complete, wow, the instructions sure could have been more specific here. Nowhere is it mentioned that redistribution should be done. Nowhere is it mentioned that the peering should come up here, just that it should be configured. I guess in theory I should have read the rest of the lab and figured out the design to know what must have to happen here.

Also, the neighbor next-hop-unchanged command stops an eBGP router from updating next-hop advertised from an iBGP peer.

Wednesday, March 4, 2009

IE SP Vol 2 Lab 3 Complete

I actually feel pretty good about this lab.

Sections 1 - 4 were pretty easy for the most part.
Section 5 was a tough vpn with two P ASs joined by vrf-lite. It turned out to be a great exercise which really boosted my confidence.
Section 6 was a bit over my head when I started, but I feel much better about mdt now. I think I have a pretty good idea what's going on and how to troubleshoot issues.

I'm feeling really good about QoS right now. Funny, I always seem to feel good about QoS yet I tend to get my butt kicked on the lab on this topic.

Security was something completely different. I think I understand how remotely triggered black holes work now.

System management was silly, and IP Services involved ORF, which while a new topic for me, was pretty easy to configure.

So, keeping in mind that I didn't enforce a time limit, I'd score myself about a 79 on this one, since I blew half the multicast and the whole QoS section. Still, not bad for a difficulty 8 lab.

IE SP Vol 2 Lab 3 Task 10

10.1 Complete. While I knew the theory from the CCIP, this is the first time I actually configured ORF. The guide on the doccd doesn't really explain much, the key is to keep in mind that it's backwards from what it appears. The CE is the sender and the PE is the receiver. The CE router defines a prefix-list and sends it to the PE router, which then filters the prefixes it sends accordingly.

IE SP Vol 2 Lab 3 Task 9

9.1 Complete, I hate menus...just seems like stupid router tricks
9,.2 Complete, username secret

IE SP Vol 2 Lab 3 Task 8

Source Based Remotely Triggered Black Hole Filtering. Wow, that's a mouthful. I've never heard of this before.

In a nutshell--part of the problem with a DOS attack is that it consumes bandwidth. Even if a customer drops packets on an inbound access list, the circuit would still be congested.

This is a way to allow a customer to have the provider drop traffic inside of the P network, so the PE-CE link does not get saturated.

To do this, the customer simply advertises a route with a prearranged tag to the PE router. The PE router will then ensure that traffic from this source gets dropped instead of clogging the link.

Sounds great in theory minus one important concept. Single host DOS attacks are really not that big of an issue. Heck, you could just as easily get your ISP on the phone and have them shun a single host. The manual effort of adding tagged null0 routes would do little to stop a distributed DOS attack, or one that randomly spoofs source addresses.

Anyway, as far as the task:
8.1 Mostly Complete. I was close--for being unsure of what they were looking for, I was happy I made it as far as I did.
8.2 Again mostly complete.
8.3 I did it a different way, but complete.

IE Vol 2 Lab 3 Task 7

7.1 Complete. No issues, same as R&S. If you can't use MQC, there's always CAR.

7.2 Complete. No issues, if you can't use an access list, use a custom nbar port-map.

7.3 Complete. Surprisingly, no issues. Like the solutions guide, I opted to not use qos-groups since the underlying packet was marked ef as well.

One note on QoS. I usually don't bother to use ip sla to test it. Big mistake!!! I have to get into the habit of setting this up every time. It's just way to easy to watch the counters to ensure everything is setup correctly.

Monday, March 2, 2009

IE SP Vol 2 Lab 3 Task 6

Ok, this multicast task was a bit rediculous too. Now on the bright side I was pretty much able to nail the config. But when I got down to the test, I was not able to get it to work at all. I went through the solutions guide and primarily found one mistake--that I didn't have the mdt router's loopback interfaces running pim. Even upon fixing this, however, the multicast test still did not work.

-----12 hours later-----

I was really starting to get desperate here, but I FINALLY got everything figured out. It basically boiled down to two things:

1. sparse-mode requires autorp listener for the autorp groups to be able to pass through. Without it, autorp will not function across sparse-mode interfaces.

2. On mVPN, rp's are required for both the main routing instance and the vrf. This was my main mistake. I was missing a static rp on a vrf interface.

So, after a long time of being worried I had no idea what I was doing and that this stuff was way over my head, it turned out to be silly mistakes that did me in.

As with other topics I spent so much time on troubleshooting an issue, I really hope all this time put in helps me the next time I come across mVPN. I'm feeling pretty good about it now, but we'll see how it goes next week.

If multicast contines to be this complex, I may need to draw a mVPN diagram in order to keep all the tunnels and vrf's straight.

IE Vol 2 SP Lab 3 Task 5

5.1 Complete, no issues
5.2 Complete, no issues
5.3 Complete, no issues
5.4 Wow, what a task. I now see why this lab gets an 8. For 2 bullet points, there is a heck of a lot of assumptions that have to be made, tons of redistribution, and a bunch of vrfs to keep track of. I actually made my way through this pretty well, but it certainly took some time. An excellent exercise to validate understanding of the big picture.
5.5, piece of cake after making it through the last task.

Sunday, March 1, 2009

IE Vol 2 SP Lab 3

Here we go with lab 3, right on schedule. Again, I'll be spending a full week on this one, taking my time to research and experiment with anything I'm not confident on. This lab moves up to a difficulty rating of 8, so I'm assuming there will be some curve balls thrown in this time.

1.1 Complete, no issues
1.2 Complete, even in transparent mode, dynamips doesn't seem to allow extended vlans on the ethernet modules.
1.3 Complete, no issues
1.4 Complete, no issues
1.5 Complete, had to change the label protocol to tdp.

2.1 Complete, no issues--no DR? use p2p or p2mp
2.2 Complete, no issues--no area authen? put it on the interface
2.3 Complete, stupid mistake. I couldn't get the isis adjacency up across the frame-relay interface. I thought there may have been some trick here since the frame-relay map had to support clns. Turns out I missed configuring isis on the interface to begin with. A dumb mistake that would have cost me valuable time in the lab.
2.4 Wrong. I used the isis password command. However, this exchanges the passwords in plain text. To do md5, an authentication key-chain must be used ala rip and eigrp.

3.1 complete, no issues
3.2 complete, no issues
3.3 complete, no issues. I'm probably splitting hairs a bit here, but the maximum prefix drops the connection when the limit is exceeded. Since the requirements specify that if the table "continues to grow to 12,000", to me this means at 12,000 the connection should be reset, meaning maximum-prefixes is 11,999. I'd hope the lab wouldn't be quite this particular, but who knows.

4.1 Complete, no issues. Here's where the tdp requirement from earlier gets specified.
4.2 Complete, no issues.

Well so far this lab is laughable. I'm assuming it's gonna it rough here shortly, because the difficulty is nowhere near an 8 so far.

IE Vol2 Lab 2 Complete

Well, I'm on schedule, barely, and ready to do lab 3 next week. There really wasn't anything too substantially scary in this lab. The VPN config was pretty basic and there wasn't any TE. PPPoE was new, but I felt pretty comfortable with it and know how to get to the docs. I'm also getting more comfortable with MPLS QoS. So, aside from a few knobs I think I did ok.

IE Vol 2 Section 10 Complete

10.1 Gotta love those knobs. mpls ldp tcp pak-priority. Now I never would have found this one. Guess I gotta get better at looking over the command-line options.

10.2 Pretty basic netflow setup. I'm not sure if netflow was covered in R&S, but I've worked with it enough where it wasn't an issue.

IE Vol 2 Section 9 Complete

9.1 Complete, no issues--just like R&S
9.2 Complete, no issues--just like R&S
9.3 Complete, no issues--just like R&S

IE Vol 2 Section 8

8.1 Complete. No issues, standard R&S stuff here. The nice part about logging denies, is I find out quickly that ospf has to be permitted in the access list.

8.2 Complete. Maximum routes in the vrf controls the size of the routing table.

IE Vol 2 Section 7

7.1 Complete. No issues, straight R&S stuff.
7.2 Complete. I need to remember not to use the nbar ftp, but to use an access-list of ftp, ftp-data, and gt 1023 to match ftp traffic.
7.3 Complete. I disagree with the solutions guide here. The requirements specify traffic should not be dropped "unless there is congestion in the transit path". To me this means setting DE, but the answers say to use queueing. In the lab a question to the proctor would be in order.
7.4 Complete, Qos-groups again. The key to keep in mind is that when traffic comes in an mpls interface, the only thing to match on is the mpls exp bits. Then on its way out of the router, if the tag is stripped, the exp bits are gone. So it's only possible to match on the packet itself. The other alternative is to use the Qos-group, with which the router associates an incoming packet internally, to remember to apply policies on egress.