CCIE Training

I enjoyed Petr’s article regarding explicit next hop.  It reminded me of a scenario where a redistributed route, going into OSPF conditionally worked, depending on which reachable next hop was used.

Here is the topology for the scenario:

Here is the relevant (and working ) information for R1.

When we replace the static route, with a new reachable next hop, we loose the ability to ping 100.100.100.3

When we change the next hop for the static route, (which is being redistributed into OSPF), the route to 100.100.100.0/24 no longer works, even though we have verified ability to ping the new next hop.

Can you solve this puzzle?  Please post your ideas!

For more troubleshooting scenarios, please see our CCIE Route-Switch workbooks, volume 2, for more than 100 challenging troubleshooting scenarios.

We will post the results right here, in a few days, after you have had a chance to post your comments and ideas.

Best wishes,

Keith

Abstract

This publication briefly covers the use of 3rd party next-hops in OSPF, RIP, EIGRP and BGP routing protocols. Common concepts are introduced and protocol-specific implementations are discussed. Basic understanding of the routing protocol function is required before reading this blog post.

Overview

Third-party next-hop concept appears only to distance vector protocol, or in the parts of the link-state protocols that exhibit distance-vector behavior. The idea is that a distance-vector update carries explicit next-hop value, which is used by receiving side, as opposed to the “implicit” next-hop calculated as the sending router’s address – the source address in the IP header carrying the routing update. Such “explicit” next-hop is called “third-party” next-hop IP address, allowing for pointing to a different next-hop, other than advertising router. Intitively, this is only possible if the advertising and receiving router are on a shared segment, but the “shared segment” concept could be generalized and abstracted. Every popular distance-vector protocols support third party next-hop – RIPv2, EIGRP, OSPF and BGP all carry explicit next-hop value. Look at the figure below – it illustrates the situation where two different distance-vector protocols are running on the shared segment, but none of them runs on all routers attached to the segment. The protocols “overlap” at a “pivotal” router and redistribution is used to provide inter-protocol route exchange.

Per the default distance-vector protocol behavior, traffic from one routing domain going into another has cross the “pivotal” router, the router where the two domains overlap (R3 in our case) – as opposed to going directly to the closes next-hop on the shared segment. The reason for this is that there is no direct “native” update exchange between the hops running different routing protocols. In situations like this, it is beneficial to rewrite the next-hop IP address to point toward the “optimum” exit point, using the “pivotal” router’s knowledge of both routing protocols.

OSPF is somewhat special with respect to the 3rd party next-hop implementation. It supports third-party next-hop in Type-5/7 LSAs (External Routing Information LSA and NSSA External LSA). These LSAs are processed in “distance-vector manner” by every receiving router. By default, the LSA is assumed to advertise the external prefix “connected” to the advertising router. However, if the FA is non-zero, the address in this field is used to calculate the forwarding information, as opposed to default forwarding toward the advertising router. Forwarding Address is always present in Type-7 LSAs, for the reason illustrated on the figure below:

Since there could be multiple ABRs in NSSA area, only one is elected to perform 7-to-5 LSA translation – otherwise the routing information will loop back in the area, unless manual filtering implemented in the ABRs (which is prone to errors). Translating ABR is elected based on the highest Router-ID, and may not be on the optimum path toward the advertising ASBR. Therefore, the forwarding address should prompt the more optimum path, based on the inter-area routing information.

EIGRP

We start with the scenario where we redistribute RIP into EIGRP.

Notice that EIGRP will not insert the third-party next-hop until you apply the command no ip next-hop-self eigrp on R3’s connection to the shared segment. Look at the routing table output prior to applying the no ip next-hop-self eigrp command.

R1#show  ip route eigrp 
     140.1.0.0/16 is variably subnetted, 2 subnets, 2 masks
D EX    140.1.2.2/32
           [170/2560002816] via 140.1.123.3, 00:00:27, FastEthernet0/0

After the command has been applied to R3’s interface:

R1#show  ip route eigrp
     140.1.0.0/16 is variably subnetted, 2 subnets, 2 masks
D EX    140.1.2.2/32
           [170/2560002816] via 140.1.123.2, 00:00:04, FastEthernet0/0

The same behavior is observed when redistributing OSPF into EIGRP, but not when redistributing BGP. For some reason, BGP’s next-hop is not copied into EIGRP, e.g. in the example below, EIGRP will NOT insert the BGP’s next-hop into updates. Notice that you may enable or disable the third-party next-hop behavior in EIGRP using the interface-level command ip next-hop-self eigrp.

RIP

RIP passes the third-party next-hop from OSPF, BGP or EIGRP. For instance, assume EIGRP redistribution into RIP. You have to turn on the no ip split-horizon on R3’s Ethernet connection to get this to work:

R2#show ip route rip 
     140.1.0.0/16 is variably subnetted, 3 subnets, 2 masks
R       140.1.1.1/32 [120/1] via 140.1.123.1, 00:00:17, FastEthernet0/0

Notice the following RIP debugging output, which lists the third-party next-hop:

RIP: received v2 update from 140.1.123.3 on FastEthernet0/0
     140.1.1.1/32 via 140.1.123.1 in 1 hops
     140.1.123.0/24 via 0.0.0.0 in 1 hops

Surprisingly, there is NO need to enable the command no ip split-horizon on the interface when redistributing BGP or OSPF routes into RIP. Seem like only EIGRP to RIP redistribution requires that. Keep in mind, however, that split-horizon is OFF by default on physical frame-relay interfaces. Here is a sample output of redistributing BGP into RIP using the third-party next-hop:

R3#show ip route bgp 
     140.1.0.0/16 is variably subnetted, 3 subnets, 2 masks
B       140.1.2.2/32 [20/0] via 140.1.123.2, 00:22:13
R3#

R1#show ip route rip 
     140.1.0.0/16 is variably subnetted, 3 subnets, 2 masks
R       140.1.2.2/32 [120/1] via 140.1.123.2, 00:00:09, FastEthernet0/0

RIP’s third-party next-hop behavior is fully automatic. You cannot disable or enable it, like you do in EIGRP.

OSPF

Similarly to RIP, OSPF has no problems picking up the third-party next-hop from BGP, EIGRP or RIP. Here is how it would look like (guess which protocol is redistributed into OSPF, based solely on the commands output):

R1#sh ip route ospf 
     140.1.0.0/16 is variably subnetted, 3 subnets, 2 masks
O E2    140.1.2.2/32 [110/1] via 140.1.123.2, 00:34:59, FastEthernet0/0

R1#show ip ospf database external 

            OSPF Router with ID (140.1.1.1) (Process ID 1)

                Type-5 AS External Link States

  Routing Bit Set on this LSA
  LS age: 131
  Options: (No TOS-capability, DC)
  LS Type: AS External Link
  Link State ID: 140.1.2.2 (External Network Number )
  Advertising Router: 140.1.123.3
  LS Seq Number: 80000002
  Checksum: 0xF749
  Length: 36
  Network Mask: /32
        Metric Type: 2 (Larger than any link state path)
        TOS: 0
        Metric: 1
        Forward Address: 140.1.123.2
        External Route Tag: 200

If you’re still guessing, the external protocol is BGP, as could have been seen observing the automatic External Route Tag – OSPF set’s it to the last AS# found in the AS_PATH.

There are special conditions to be met by OSPF for the FA address to be used. First, the interface where the third party next-hop resides should be advertised into OSPF using the network command. Secondly, this interface should not be passive in OSPF and should not have network type point-to-point or point-to-multipoint. Violating any of these conditions will stop OSPF from using the FA in type-5 LSA created for external routes. Violating any of these conditions prevents third-party next-hop installation in the external LSAs.

OSPF is special in one other respect. Distance vector-protocols such as RIP or EIGRP modify the next-hop as soon as they pass the routing information to other devices. That is, the third party next-hop is not maintained through the RIP or EIGRP domain. Contrary to these, OSPF LSAs are flooded within their scope with the FA unmodified. This creates interesting problem: if the FA address is not reachable in the receiving router’s routing table, the external information found in type 7/5 LSA is not used. This situation is discussed in the blog post “OSPF Filtering using FA Address”.

BGP

When you redistribute any protocol into BGP, the system correctly sets the third-party next-hop in the local BGP table. Look at the diagram below, where EIGRP prefixes are being redistributed into BGP AS 300:

R3’s BGP process installs R1 Loopback0 prefix into the BGP table with the next-hop value of R1’s address, not “0.0.0.0” like it would be for locally advertised routes. You will observe the same behavior if you inject EIGRP prefixes into BGP using the network command.

R3#sh ip bgp
BGP table version is 9, local router ID is 140.1.123.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 140.1.1.1/32     140.1.123.1         156160         32768 ?

Furthermore, BGP is supposed to change the next-hop to self when advertising prefixes over eBGP peering sessions. However, when all peers share the same segment, the prefixes re-advertised over the shared segment do not have their next-hop changed. See the diagram below:

Here R1 advertises prefix 140.1.1.1/24 to R3 and R3 re-advertises it back to R2 over the same segment. Unless non-physical interfaces are used to form the BGP sessions (e.g. Loopbacks), the next-hop received from R1 is not changed when passing it down to R2. This implements the default third-party next-hop preservation over eBGP sessions. Look at the sample output for the configuration illustrated above: R1 receives R2’s prefix with unmodified next-hop.

R1#show ip bgp 
BGP table version is 3, local router ID is 140.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
              r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

   Network          Next Hop            Metric LocPrf Weight Path
*> 140.1.1.1/32     0.0.0.0                  0         32768 i
*> 140.1.2.2/32     140.1.123.2                            0 300 200 i

There is a way to disable this default behavior in BGP. A logical assumption would be that using the command neighbor X.X.X.X next-hop-self would work, and it does indeed, in the recent IOS versions. The older IOS, such as 12.2T did not have this command working for eBGP sessions, and your option would have been using a route-map with set ip next-hop command. The route-map method may still be handy, if you want insert totally “bogus” IP next-hop from the shared segment – receiving BGP speaker will accept any IP address that is on the same segment. That is not something you would do in the production environment too often, but definitely an interesting idea for lab practicing. One good use in production is changing the BGP next-hop to an HSRP virtual IP address, to provide physical BGP speaker redundancy. Here is a sample code for setting an explicit next-hop in BGP update:

router bgp 300
 neighbor 140.1.123.1 remote-as 100
 neighbor 140.1.123.1 route-map BGP_NEXT_HOP out
!
route-map BGP_NEXT_HOP permit 10
 set ip next-hop 140.1.123.100

Summary

All popular distance-vector protocols support third-party next-hop insertion. This mechanism is useful on multi-access segments, in situations when you want pass optimum path information between routers belonging to different routing protocols. We illustrated that RIP implements this function automatically, and does not allow any tuning. On the other hand, EIGRP supports third-party next-hop passing from any protocol, other than BGP, and you may turn this function on/off on per-interface basis. Furthermore, OSPF’s special feature is propagation of the third-party next-hop within an area/autonomous system, unlike the distance-vector protocols that reset the next-hop at every hop (considering AS a being a “single-hop” for BGP). Thanks to that feature, OSPF offers interesting possibility to filter external routing information by blocking FA prefix from the routing tables. Finally, BGP gives most flexibility when it comes to the IP next-hop manipulation, allowing for changing it to any value.

Further Reading

Common Routing Problem with OSPF Forwarding Address
OSPF Prefix Filtering Using Forwarding Address
BGP Redundancy using HSRP

Are you a CCNP or CCIE student looking to challenge your perfect knowledge of Catalyst switchport commands?

Take the latest SWITCH Command Recall exam by clicking the link below. Good luck – and let us know how you scored in the comments area of this post.

Remember to read, AND TYPE, very carefully! I failed my first attempt due to just plain sloppiness.

SWITCH Command Recall Exam – L2/L3 Ports

One of the frequent questions I hear regarding L3VPNs, is regarding the bottom VPN label.  In this article, we will focus on the control plane that provides both the VPN and transit labels, and then look at the data plane that results because of those labels.

In the topology, there are 2 customer sites (bottom right, and bottom left).  The BGP, VRFs, Redistribution, etc are all configured to allow us to focus on the control and data plane.   Lets begin by verifying that R1 is sourcing the network, 1.1.1.1/32.

A debug verifies that R1 is sending the updates for 1.1.1.1 to R2.

R2 has learned the route from R1, has assigned a VPN label for it, and has exported it from the VRF into BGP.  This lucky route was assigned the local label of 16 by R2.

We can also look at the MPLS forwarding table on R2 to see the same tag information.

This prefix, as a VPNV4 route, is sent as an update to the iBGP peer R4.   We can force an update with refresh.

The update can be seen on the wire between R2 and R3, (on its way to R4) using a protocol analyzer.  You may also notice that R2 uses outgoing label 19 for forwarding this update to 4.4.4.4   The label can be seen in the MPLS forwarding output above.

The VPN label being advertised in the update is Label 16, which is R2’s local label for the 1.1.1.1 network.

On R4, which will be the ingress PE for transit traffic destined for 1.1.1.1, we can see that the VPN label of 16 is associated with destination network of 1.1.1.1  The next hop of 2.2.2.2 to reach the 1.1.1.1 network, is due to R2 assigning next-hop-self for updates it sends to R4.

We can also see the outgoing MPLS label that R4 will use to reach the next hop of 2.2.2.2.  The label of 18 below, was advertised by R3, as the label to use to reach 2.2.2.2

We can also verify that the route (1.1.1.1) has been imported by R4 into the customer vrf.

So when a transit packet is sent from R5 to 1.1.1.1, R4 should impose 2 labels.   The bottom label will be 16 (the VPN label) for the 1.1.1.1 network (R2 told us about that via the iBGP update), and the top label should be 18 (advertised via LDP from R3), to reach the next hop of 2.2.2.2

On R4 a quick check of the CEF table for the vrf can verify both labels.

A simple trace from R5, to the destination network of 1.1.1.1 should prove all the labels in the path.

The top label of 18 is used to reach the next hop of 2.2.2.2, and the bottom label of 16, which is meaningful to R2, because he sourced it, will be used by R2 in forwarding the transit traffic destined to 1.1.1.1 to the next hop, which is R1.

R3 will pop the transit label off, due to R2 advertising implicit-null for the network 2.2.2.2 (itself).

For more information and step by step training on MPLS, take a look at our newest MPLS self paced course!

If you like, an 8 minute video, that reviews the same steps, may be viewed here.

Thanks for reading!

Keith Barker

In this blog post we’re going to discuss the fundamental logic of how MPLS tunnels allow applications such as L2VPN & L3VPN to work, and how MPLS tunnels enable Service Providers to run what is known as the “BGP Free Core”. In a nutshell, MPLS tunnels allow traffic to transit over devices that have no knowledge of the traffic’s final destination, similar to how GRE tunnels and site-to-site IPsec VPN tunnels work. To accomplish this, MPLS tunnels use a combination of IGP learned information, BGP learned information, and MPLS labels.

In normal IP transit networks, each device in the topology makes a routing decision on a hop-by-hop basis by comparing the destination IP address of the packet to the routing or forwarding table. In MPLS networks, devices make their decision based on the MPLS label contained in the packet that is received. In this manner, MPLS enabled Label Switch Routers (LSRs for short) do not necessarily need IP routing information about all destinations, as long as they know how to forward traffic based on an MPLS label. To demonstrate how this process works, we’ll examine the above topology in two sample cases, first with normal IP packet forwarding, and secondly with IP packet forwarding plus MPLS.

In this topology R4, R5, and R6 represent the Service Provider network, while SW1 and SW2 represent customers that are using the Service Provider for transit. In each example case, the goal of our scenario will be to provide IP packet transport between the 10.1.7.0/24 network attached to SW1, and the 10.1.8.0/24 network attached to SW2.

Case 1: Normal IP Forwarding

In case 1, the devices in the topology are configured as follows. AS 100, which consists of R4, R5, and R6, runs OSPF as an IGP on all internal interfaces, along with a full mesh of iBGP. AS 7, which consists of SW1, and AS 8, which consists of SW2, peer EBGP with AS 100 via R4 and R6 respectively, and advertise the prefixes 10.1.7.0/24 & 10.1.8.0/24 respectively into BGP. The relevant device configurations are as follows. Note that these configs assume that layer 2 connectivity has already been established between the devices.

R4#
interface Loopback0
 ip address 10.1.4.4 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.47.4 255.255.255.0
!
interface FastEthernet0/1
 ip address 10.1.45.4 255.255.255.0
 ip ospf 1 area 0
!
router bgp 100
 neighbor 10.1.6.6 remote-as 100
 neighbor 10.1.6.6 update-source Loopback0
 neighbor 10.1.6.6 next-hop-self
 neighbor 10.1.45.5 remote-as 100
 neighbor 10.1.45.5 next-hop-self
 neighbor 10.1.47.7 remote-as 7

R5#
interface FastEthernet0/0
 ip address 10.1.45.5 255.255.255.0
 ip ospf 1 area 0
!
interface FastEthernet0/1
 ip address 10.1.56.5 255.255.255.0
 ip ospf 1 area 0
!
router bgp 100
 neighbor 10.1.45.4 remote-as 100
 neighbor 10.1.56.6 remote-as 100

R6#
interface Loopback0
 ip address 10.1.6.6 255.255.255.255
 ip ospf 1 area 0
!
interface FastEthernet0/0
 ip address 10.1.56.6 255.255.255.0
 ip ospf 1 area 0
!
interface FastEthernet0/1
 ip address 10.1.68.6 255.255.255.0
!
router bgp 100
 neighbor 10.1.4.4 remote-as 100
 neighbor 10.1.4.4 update-source Loopback0
 neighbor 10.1.4.4 next-hop-self
 neighbor 10.1.56.5 remote-as 100
 neighbor 10.1.56.5 next-hop-self
 neighbor 10.1.68.8 remote-as 8

SW1#
interface Loopback0
 ip address 10.1.7.7 255.255.255.0
!
interface Vlan47
 ip address 10.1.47.7 255.255.255.0
!
router bgp 7
 network 10.1.7.0 mask 255.255.255.0
 neighbor 10.1.47.4 remote-as 100

SW2#
interface Loopback0
 ip address 10.1.8.8 255.255.255.0
!
interface Vlan68
 ip address 10.1.68.8 255.255.255.0
!
router bgp 8
 network 10.1.8.0 mask 255.255.255.0
 neighbor 10.1.68.6 remote-as 100

Next, let’s examine the hop-by-hop packet flow as traffic moves between the 10.1.7.0/24 and 10.1.8.0/24 networks, starting at SW1 towards SW2, and then back in the reverse direction. Note that verification should be done in both directions, as packet flow from the source to destination is independent of packet flow from the destination back to the source.

SW1#show ip route 10.1.8.0
Routing entry for 10.1.8.0/24
  Known via "bgp 7", distance 20, metric 0
  Tag 100, type external
  Last update from 10.1.47.4 00:21:13 ago
  Routing Descriptor Blocks:
  * 10.1.47.4, from 10.1.47.4, 00:21:13 ago
      Route metric is 0, traffic share count is 1
      AS Hops 2
      Route tag 100

On SW1, the prefix 10.1.8.0 is learned via BGP from R4, with a next-hop of 10.1.47.4. Next, SW1 performs a second recursive lookup on the next-hop to see which interface must be used for packet forwarding.

SW1#show ip route 10.1.47.4
Routing entry for 10.1.47.0/24
  Known via "connected", distance 0, metric 0 (connected, via interface)
  Routing Descriptor Blocks:
  * directly connected, via Vlan47
      Route metric is 0, traffic share count is 1

The result of this lookup is that SW1 should use interface Vlan47, which connects towards R4. Assuming that underlying IP address to MAC address resolution is successful, packets going to 10.1.8.0 should be properly routed towards R4. Next, the lookup process continues on R4.

R4#show ip route 10.1.8.0
Routing entry for 10.1.8.0/24
  Known via "bgp 100", distance 200, metric 0
  Tag 8, type internal
  Last update from 10.1.6.6 00:25:19 ago
  Routing Descriptor Blocks:
  * 10.1.6.6, from 10.1.6.6, 00:25:19 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 8

R4 is learning the prefix 10.1.8.0 via iBGP from R6, with a next-hop value of 10.1.6.6, R6’s Loopback0 interface. R4 must now perform an additional recursive lookup to figure out what interface 10.1.6.6 is reachable out.

R4#show ip route 10.1.6.6
Routing entry for 10.1.6.6/32
  Known via "ospf 1", distance 110, metric 3, type intra area
  Last update from 10.1.45.5 on FastEthernet0/1, 00:25:26 ago
  Routing Descriptor Blocks:
  * 10.1.45.5, from 10.1.6.6, 00:25:26 ago, via FastEthernet0/1
      Route metric is 3, traffic share count is 1

R4 knows 10.1.6.6 via OSPF from R5, which uses interface FastEthernet0/1. Assuming layer 2 connectivity is working properly, packets towards 10.1.8.0 are now routed to R5, and the lookup process continues.

R5#show ip route 10.1.8.0
Routing entry for 10.1.8.0/24
  Known via "bgp 100", distance 200, metric 0
  Tag 8, type internal
  Last update from 10.1.56.6 00:24:36 ago
  Routing Descriptor Blocks:
  * 10.1.56.6, from 10.1.56.6, 00:24:36 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 8

R5 is learning the prefix 10.1.8.0 via iBGP from R6, with a next-hop of 10.1.56.6. A recursive lookup on the next-hop, as seen below, indicates that R5 should use interface FastEthernet0/1 to forward packets towards 10.1.8.0.

R5#show ip route 10.1.56.6
Routing entry for 10.1.56.0/24
  Known via "connected", distance 0, metric 0 (connected, via interface)
  Routing Descriptor Blocks:
  * directly connected, via FastEthernet0/1
      Route metric is 0, traffic share count is 1

The lookup process now continues on R6, as seen below.

R6#show ip route 10.1.8.0
Routing entry for 10.1.8.0/24
  Known via "bgp 100", distance 20, metric 0
  Tag 8, type external
  Last update from 10.1.68.8 00:28:58 ago
  Routing Descriptor Blocks:
  * 10.1.68.8, from 10.1.68.8, 00:28:58 ago
      Route metric is 0, traffic share count is 1
      AS Hops 1
      Route tag 8

R6 is learning the prefix 10.1.8.0 via EBGP from SW2, with a next-hop of 10.1.68.8.

R6#show ip route 10.1.68.8
Routing entry for 10.1.68.0/24
  Known via "connected", distance 0, metric 0 (connected, via interface)
  Routing Descriptor Blocks:
  * directly connected, via FastEthernet0/1
      Route metric is 0, traffic share count is 1

A recursive lookup on 10.1.68.8 from R6 dictates that interface FastEthernet0/1 should be used to forward traffic on to SW2.

SW2#show ip route 10.1.8.0
Routing entry for 10.1.8.0/24
  Known via "connected", distance 0, metric 0 (connected, via interface)
  Advertised by bgp 8
  Routing Descriptor Blocks:
  * directly connected, via Loopback0
      Route metric is 0, traffic share count is 1

SW2’s lookup for 10.1.8.0 indicates that the destination is directly connected, and packets are routed to the final destination. For return traffic back to 10.1.7.0, a lookup occurs in the reverse direction similar to what we saw above, starting as SW2, and moving to R6, R5, R4, and then finally SW1.

This example shows how the hop-by-hop routing paradigm works in IPv4 networks. While this type of design works, one of the limitations of IPv4 forwarding is that all devices in the transit path must have routing information for all destinations they are forwarding packets towards. If AS 100 was used for Internet transit in this example, each router in the transit path would need 300,000+ routes in their routing tables in order to provide transit to all Internet destinations. This is just one of the many applications for which MPLS can be helpful. By introducing MPLS into this design, the need for large routing tables can be avoided in the core of the Service Provider network.

Case 2: MPLS Forwarding

By enabling MPLS in the Service Provider network of AS 100, BGP can be disabled in the core, lightening the load on devices that are possibly already taxed for resources. The configuration for MPLS in this scenario is very simple, but the understanding of what happens behind the scenes can be intimidating when learning the technology for the first time. To help with this learning curve, we’ll look at the step by step process that occurs when an MPLS tunnel is functional in AS 100.

The configuration changes necessary to implement MPLS in AS 100 are as follows:

R4#
mpls label protocol ldp
!
interface FastEthernet0/1
 mpls ip
!
router bgp 100
 no neighbor 10.1.45.5 remote-as 100

R5#
mpls label protocol ldp
!
interface FastEthernet0/0
 mpls ip
!
interface FastEthernet0/1
 mpls ip
!
no router bgp 100

R6#
mpls label protocol ldp
!
interface FastEthernet0/0
 mpls ip
!
router bgp 100
 no neighbor 10.1.56.5 remote-as 100

Once MPLS is enabled inside AS 100, BGP can be disabled on R5, and the additional BGP peering statements removed on R4 and R6. The end result of this change is surprising for some, as seen below.

R5#show ip route 10.1.7.0
% Subnet not in table
R5#show ip route 10.1.8.0
% Subnet not in table

SW1#ping 10.1.8.8 source 10.1.7.7

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.8.8, timeout is 2 seconds:
Packet sent with a source address of 10.1.7.7
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms

Although R5 no longer has a route to the prefixes 10.1.7.0/24 or 10.1.8.0/24, it can still provide transit for traffic between them. How is this possible you may ask? The key is that an MPLS tunnel has now been formed between the ingress and egress routers of the Service Provider network, which are R4 and R6 in this case.  To see the operation of the MPLS tunnel, we’ll follow the same lookup process as before, but now R4, R5, and R6 will add an additional MPLS label lookup.

Below SW1 looks up the route for 10.1.8.0/24, and finds that it recurses to R4’s next-hop value reachable via the Vlan47 interface.

SW1#show ip route 10.1.8.0
Routing entry for 10.1.8.0/24
 Known via "bgp 7", distance 20, metric 0
 Tag 100, type external
 Last update from 10.1.47.4 01:02:56 ago
 Routing Descriptor Blocks:
 * 10.1.47.4, from 10.1.47.4, 01:02:56 ago
 Route metric is 0, traffic share count is 1
 AS Hops 2
 Route tag 100

SW1#show ip route 10.1.47.4
Routing entry for 10.1.47.0/24
 Known via "connected", distance 0, metric 0 (connected, via interface)
 Routing Descriptor Blocks:
 * directly connected, via Vlan47
 Route metric is 0, traffic share count is 1

Next, R4 receives packets for this destination and performs its own lookup.

R4#show ip route 10.1.8.0
Routing entry for 10.1.8.0/24
 Known via "bgp 100", distance 200, metric 0
 Tag 8, type internal
 Last update from 10.1.6.6 01:05:15 ago
 Routing Descriptor Blocks:
 * 10.1.6.6, from 10.1.6.6, 01:05:15 ago
 Route metric is 0, traffic share count is 1
 AS Hops 1
 Route tag 8

Like before, R4 finds the route via BGP from R6, with a next-hop of 10.1.6.6.  R4 must now perform a recursive lookup on 10.1.6.6 to find the outgoing interface to reach R6.

R4#show ip route 10.1.6.6
Routing entry for 10.1.6.6/32
 Known via "ospf 1", distance 110, metric 3, type intra area
 Last update from 10.1.45.5 on FastEthernet0/1, 01:06:22 ago
 Routing Descriptor Blocks:
 * 10.1.45.5, from 10.1.6.6, 01:06:22 ago, via FastEthernet0/1
 Route metric is 3, traffic share count is 1

R4’s recursive lookup finds the outgoing interface FastEthernet0/1 with a next-hop of 10.1.45.5.  In normal IP forwarding, the packet would now be sent to the interface driver for layer 2 encapsulation.  However in this case, R4 first checks to see if the interface FastEthernet0/1 is MPLS enabled, as seen below.

R4#show mpls interfaces
Interface              IP            Tunnel   BGP Static Operational
FastEthernet0/1        Yes (ldp)     No       No  No     Yes

Since interface FastEthernet0/1 is running MPLS via Label Distribution Protocol (LDP), R4 now consults the MPLS Label Forwarding Information Base (LFIB) to see if there is an MPLS label assigned to the next-hop we’re trying to reach, 10.1.6.6.

R4#show mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     Pop Label     10.1.56.0/24      0             Fa0/1      10.1.45.5
17     17            10.1.6.6/32       0             Fa0/1      10.1.45.5
18     18            10.1.68.0/24      0             Fa0/1      10.1.45.5

R4 finds an entry for 10.1.6.6/32 in the LFIB, and uses the outgoing label value of 17.  This means that for traffic going to 10.1.8.0/24, the label 17 will be added to the packet header.  In reality this lookup process occurs as one step, which is the lookup in the CEF table.  The below output of the CEF table for the final destination on R4 shows that label 17 will be used, because it is inherited from the next-hop of 10.1.6.6.

R4#show ip cef 10.1.8.0 detail
10.1.8.0/24, epoch 0
 recursive via 10.1.6.6
 nexthop 10.1.45.5 FastEthernet0/1 label 17

Now that the MPLS label lookup is successful, the packet is label switched to R5, which leads us to the key step in this example.  When R5 receives the packet, it sees that it has an MPLS label in the header.  This means that R5 performs a lookup in the MPLS LFIB first, and not in the regular IP routing table.  Specifically R5 sees the label number 17 coming in, which has a match in the LFIB as seen below.

R5#show mpls forwarding-table

Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop

Label  Label or VC   or Tunnel Id      Switched      interface

16     Pop Label     10.1.4.4/32       15447         Fa0/0      10.1.45.4

17     Pop Label     10.1.6.6/32       15393         Fa0/1      10.1.56.6

18     Pop Label     10.1.68.0/24      0             Fa0/1      10.1.56.6

The local label 17 is associated with the destination 10.1.6.6/32.  Although our packets are going to the final destination 10.1.8.0/24, knowing how to get towards the next-hop 10.1.6.6/32 is sufficient for R5, because we know that R6 actually does have the route for the final destination.  Specifically R5’s operation at this point is to remove the label 17 from the packet, and continue to forward the packet towards R6 without an additional label.  This operation is known as the “pop” operation, or label disposition.  This occurs because R5 sees the outgoing label as “no label”, which causes it to remove any MPLS labels from the packet, and continue forwarding it.

On the return trip for packets from 10.1.8.0/24 back to 10.1.7.0/24, R6 adds the label 16 and forwards the packet to R5, then R5 removes the label 16 and forwards the packet to R4.  This can be inferred from the LFIB and CEF table verifications below.

R6#show mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     16            10.1.4.4/32       0             Fa0/0      10.1.56.5
17     Pop Label     10.1.45.0/24      0             Fa0/0      10.1.56.5   

R6#show ip cef 10.1.7.0 detail
10.1.7.0/24, epoch 0
  recursive via 10.1.4.4
    nexthop 10.1.56.5 FastEthernet0/0 label 16

R5#show mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     No Label      10.1.4.4/32       17606         Fa0/0      10.1.45.4
17     No Label      10.1.6.6/32       17552         Fa0/1      10.1.56.6
18     Pop Label     10.1.68.0/24      0             Fa0/1      10.1.56.6

To see this operation in action, we can send traffic from 10.1.7.0/24 to 10.1.8.0/24, and look at the debug mpls packet output on R5.

R5#debug mpls packet
Packet debugging is on

SW1#ping 10.1.8.8 source 10.1.7.7

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.8.8, timeout is 2 seconds:
Packet sent with a source address of 10.1.7.7
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/4/9 ms

R5#
MPLS les: Fa0/0: rx: Len 118 Stack {17 0 254} - ipv4 data
MPLS les: Fa0/1: rx: Len 118 Stack {16 0 254} - ipv4 data
MPLS les: Fa0/0: rx: Len 118 Stack {17 0 254} - ipv4 data
MPLS les: Fa0/1: rx: Len 118 Stack {16 0 254} - ipv4 data
MPLS les: Fa0/0: rx: Len 118 Stack {17 0 254} - ipv4 data
MPLS les: Fa0/1: rx: Len 118 Stack {16 0 254} - ipv4 data
MPLS les: Fa0/0: rx: Len 118 Stack {17 0 254} - ipv4 data
MPLS les: Fa0/1: rx: Len 118 Stack {16 0 254} - ipv4 data
MPLS les: Fa0/0: rx: Len 118 Stack {17 0 254} - ipv4 data
MPLS les: Fa0/1: rx: Len 118 Stack {16 0 254} - ipv4 data

The beauty of this MPLS design is that for any new routes AS 7 or AS 8 advertise into the network, AS 100 does not need to allocate new MPLS labels in the core.  As long as MPLS transport is established between the BGP peering address of the Provider Edge routers (R4 and R6 in this case), traffic for any destinations can transit over the Service Provider’s network without the core needing any further forwarding information.

R4#show ip route 10.1.6.6
Routing entry for 10.1.6.6/32
Known via "ospf 1", distance 110, metric 3, type intra area
Last update from 10.1.45.5 on FastEthernet0/1, 01:06:22 ago
Routing Descriptor Blocks:
* 10.1.45.5, from 10.1.6.6, 01:06:22 ago, via FastEthernet0/1
Route metric is 3, traffic share count is 1

R4#show mpls interfaces
Interface              IP            Tunnel   BGP Static Operational
FastEthernet0/1        Yes (ldp)     No       No  No     Yes
R4#show mpls forw
R4#show mpls forwarding-table
Local  Outgoing      Prefix            Bytes Label   Outgoing   Next Hop
Label  Label or VC   or Tunnel Id      Switched      interface
16     Pop Label     10.1.56.0/24      0             Fa0/1      10.1.45.5
17     17            10.1.6.6/32       0             Fa0/1      10.1.45.5
18     18            10.1.68.0/24      0             Fa0/1      10.1.45.5
R4#show ip cef 10.1.8.0 detail
10.1.8.0/24, epoch 0
recursive via 10.1.6.6
nexthop 10.1.45.5 FastEthernet0/1 label 17

We recently created a new self-paced MPLS course, which walks the learner step by step from concept to implementation for MPLS and L3 VPNs.  Click here for more information.

We wanted to provide our students with advance notification of some upcoming online classes here at INE. While we hope to see many students in the actual live events, on-demand versions will indeed be made available the week following the live, online version.

Join the INE Experts Online in September and October

September 13 – 17th, 2010     CCNA Wireless 5-Day Bootcamp

September 15 – 17th, 2010     Security for CCIE R&S Candidates 3-Day Bootcamp

September 29 – Oct 1, 2010    IPv4/IPv6 Multicast 3-Day Bootcamp

October 4 – 9th, 2010              Online 6-Day CCIE R&S Bootcamp with K. Barker and A. Sequeira

For each new CCIE Testimonial we are extending the seven years of success sale! Share your INE success story and congratulations to the following new CCIE Testimonials who have extended the sale thus far!

Thomas Fischer, CCIE #26636 – Routing & Switching

I am proud to let you know, that I passed my CCIE R&S Lab in Brussels on Aug. 5th. This was my second attempt. I want to express my deepest appreciation for your Products. I am a self-paced student, using Vol1 (*****), Vol2 (****) and Vol4 (***). Thanks INE, it feels so good to have a social life again )

Matthew Ayre, CCIE #26654 – Service Provider

Big shout out for INE and their OEQ / lab preparation resources! I just cleared service provider on second attempt finishing about an hour and a half early. Was ~7% of passing the first time using INE 1 & 2 as my primary material then just drilled down on the finer details reading theory. The workbooks really developed the speed and confidence required to beat the exam!

Prateek Madaan, CCIE #26772 – Security

Had been a long and tough journey. Would really like to thank INE from the Core of my heart for facilitating in imparting the skills required not to just pass the exam but to DESERVE it as well…

There are many workbooks available which I prepared along with INE , do not want to name or list any one of them…or make any comparisons…But in comparisons INEs Security Workbooks may sound tough as compared to others BUT once you go through these workbooks is when you actually feel DESERVED the tag rather than just passing it.. Each of these workbooks and the tasks test each and every technology in detail and till the dead end….

In my last attempt on Version 2 I was deprived of the number by 1%, still followed and trusted INE workbooks and finally it helped….Today I am more happy not to procure the number but to actually have the feeling of confidence that ‘YES this time I deserve to be a CCIE’ and all due to the exhaustive INE workbooks….

Update!
Olusegun Olurotimi Medeyinlo, CCIE #26683 – Routing & Switching

I the Passed the CCIE R&S lab in Brussels on my Second attempt. I’d like to thank the instructors at INE for their excellent workbooks and blogs. Special thanks to Keith Barker for his encouragement and advice.


Now, I have my own CCIE number #22683.

Congratulations to everyone who passed the CCIE Lab Exam. Our instructors, authors, and staff have been committed to helping you pass your exam for the past seven years and we will continue to make your exam our number one priority. Only at INE.

Abstract

In this blog post we are going to review a number of MPLS scaling techniques. Theoretically, the main factors that limit MPLS network growth are:

1. IGP Scaling. Route Summarization, which is the core procedure for scaling of all commonly used IGPs does not work well with MPLS LSPs. We’ll discuss the reasons for this and see what solutions are available to deploy MPLS in presence of IGP route summarization.
2. Forwarding State growth. Deploying MPLE TE may be challenging in large network as number of tunnels grow like O(N^2) where N is the number of TE endpoints (typically the number of PE routers). While most of the networks are not even near the breaking point, we are still going to review techniques that allow MPLS-TE to scale to very large networks (10th of thousands routers).
3. Management Overhead. MPLS requires additional control plane components and therefore is more difficult to manage compared to classic IP networks. This becomes more complicated with the network growth.

The blog post summarizes some recently developed approaches that address the first two of the above mentioned issues. Before we begin, I would like to thank Daniel Ginsburg for introducing me to this topic back in 2007.

IGP Scalability and Route Summarization

IP networks were built around the concept of hierarchical addressing and routing, first introduced by Kleinrock [KLEIN]. Scaling for hierarchically addressed networks is achieved by using topologically-aware addresses that allow for network clustering and routing information hiding by summarizing contiguous address blocks. While other approaches for route scaling has been developed later, the hierarchical approach is the prevalent one in modern networks. Modern IGPs used in SP networks are link-state (ISIS and OSPF) and hence maintain network topology information in addition to the network reachability information. Route summarization creates topological domain boundaries and condenses network reachability information thus having the following important effects on link-state routing protocols:

1. Topological database size is decreased. This reduces the impact on router memory and CPU as smaller database requires less maintenance efforts and consumes less memory.
2. Convergence time within every area is improved as a result of faster SPF, smaller flooding scope and decreased FIB size.
3. Flooding is reduced, since events in one routing domain do not propagate into another as a result of information hiding. This improves routing process stability.

The above positive effects were very important during the earlier days of the Internet, when hardware did not have enough power to easily run link-state routing protocols. Modern advances in router hardware allow for scaling of link state protocols to single-area networks consisting or thousand of routers. Certain optimization procedures mentioned, for instance, in [OSPF-FAST] allow for such large single-area networks to remain stable and converge on sub-second time-scale. Many ISP networks indeed use single-area design for their IGPs, enjoying simplified operations procedures. However, scaling IGPs to tens of thousands nodes will most likely require use of routing areas yet allow for end-to-end MPLS LSPs. Another trend that may call for support of route summarization in MPLS networks is the fact that many enterprises, which typically employ area-based design, are starting their own MPLS deployments.

Before we go into details discussing the effect of route summarization MPLS LSPs, it is worth recalling what negative effects route summarization has in pure IP networks. Firstly, summarization hides network topology and detailed routing information and thus results in suboptimal routing, and even routing loops, in presence of route redistribution between different protocols. This problem is especially visible at larger scale, e.g. for the Internet as a whole. The other serious problem is that summarization hides reachability information. For example, if a single host goes done, the summarized prefix encompassing this hosts’s IP address will remain the same and the failure notification will not leak the event beyond the summarization domain. Being a positive effect of summarization, this behavior has negative impact on inter-area network convergence. For example, this affects BGP next-hop tracking process (see [BGP-NEXTHOP]).

MPLS LSPs and Route Summarization

MPLS LSP is unidirectional tunnel built on switching of locally significant labels. There are multiple ways for constructing MPLS LSPs, but we are concerned with classic LDP signaling at the moment. In order for LDP-signaled LSP to destination (FEC, forwarding equivalency class) X to be contiguous, every hop on path to X should have forwarding state for this FEC. Essentially, forwarding component for MPLS treats X, the FEC, as an endpoint identifier. If two nodes cannot match information about X, they cannot “stitch” the LSP in contiguous manner. In the case of MPLS deployment in IP networks, the FEC is typically the PE’s /32 IP address. IP address has overloaded functions of both location pointer and endpoint identifier. Summarizing the /32 addresses hides the endpoint identity and prohibits LDP nodes to consistently map labels for a given FEC. Look at the diagram below: ABR2 summarizes PE1-PE3 Loopback prefixes, and thus PE1 cannot construct an end-to-end LSP for 10.0.0.1/32, 10.0.0.2/32 or 10.0.0.3/32.

One obvious solution to this problem would be changing the underlying MPLS transport to IP based tunnels, such as mGRE or L2TPv3. Such solutions are available for deploying with L3 and L2 VPNs (see [L3VPN-MGRE] for example) and perfectly work with route summarization. However, by using convenient routing, these solutions lose the powerful feature of MPLS Traffic Engineering. While some Traffic Engineering solutions are available for IP networks, they are not as flexible and feature-rich as MPLS-TE. Therefore, we are not going to look into IP-based tunneling in this publication.

Route Leaking

Route leaking is the most widely deployed technique to allow for end-to-end LSP construction in presence of route summarization. This technique is based on the fact that typically LSPs need only be constructed for the PE addresses. Provided that a separate subnet is selected for the PE Loopback interfaces it is easy to apply fine-tuned control to the PE prefixes in the network. Referring to the figure below, all transit link prefixes (10.0.0.0/16) could be summarized (and even suppressed) and only the necessary PE prefixes (subnet 20.0.0.0/24 on the diagram below) will be leaked. It is also possible to fine-tune LDP to propagate only the prefixes from selected subnet, reducing LDP signaling overhead.

Leaking /32’s also allows for perfect next-hop reachability tracking, as a route for every PE router is individually present all routing tables. Of course, such granularity significantly reduces the positive effect of multi-area design and summarization. If the number of PE routers grows to tens of thousands, network stability would be at risk. However, this approach is by far the most commonly used with multi-area designs nowadays.

Inter-Area RSVP-TE LSPs

MPLS transport is often deployed for the purpose of traffic-engineering. Typically, a full-mesh of MPLS TE LSPs is provisioned between the PE routers in the network to accomplish this goal (strategic Traffic Engineering). The MPLS TE LSPs are signaled by means of TE extensions to RSVP protocol (RSVP-TE), which allows for source-routed LSP construction. A typical RSVP-TE LSP is explicitly routed by the source PE, where explicit route is either manually specified or computed dynamically by constrained SPF (cSPF) algorithm. Notice that it is impossible for a router within one area to run cSPF for another area, as the topology information is hidden (there are certain solutions to overcome this limitation e.g. [PCE-ARCH], but they are out of the scope of this publication).

Based on the single-area limitation, it may seem that MPLS-TE is not applicable to multi-area design. However, a number of extensions to RSVP signaling have been proposed to allow for inter-area LSP construction (see [RSVP-TE-INTERAREA]). In short, such extensions allow for explicitly programming loose inter-area hops (ABRs or ASBRs) and let every ABR expand the loose next-hop (ERO, explicit route object, expansion). The headend router and every ABR run the cSPF algorithm for their areas, such that every segment of Inter-AS LSP is locally optimal.

This approach does not necessarily result in globally optimal path, i.e. the resulting LSP may appear to differ from the shortest IGP path between the two destinations. This is a serious problem, and requires considerable modification to RSVP-TE signaling to be resolved. See the document [PCE-ARCH] for an approach to make MPLS LSPs globally optimal. Next, inter-area TE involves additional management overhead, as the ABR loose next-hops need to be programmed manually, possibly covering backup paths via alternate ABRs. The final issue is less obvious, but may have significant impact in large networks.

Every MPLS-TE constructed LSP is point-to-point (P2P) by its nature. This means that for N endpoints there is a total of N^2 LSPs connecting them, which results in rapidly growing number of forwarding states (e.g. CEF table entries) in the core routers, not to mention the control-plane overhead. This issue becomes serious in very large networks, as it has been thoroughly analyzed in [FARREL-MPLS-TE-SCALE-1 ] and [FARREL-MPLS-TE-SCALE-2]. It is interesting to compare the MPLS-TE scaling properties to those of MPLS LDP. The former constructs P2P LSPs, the latter builds Multipoint-to-Point (MP2P) LSPs that merge toward the tail-end router (see [MINEI-MPLS-SCALING]). This is the direct result of signaling behavior: RSVP is end-to-end and LDP signaling is local between every pair of routers. In result, the number of forwarding states with LDP grows as O(N) compared to O(N^2) with MPLS-TE. Look at the diagram below, that illustrates LSPs constructed using RSVP-TE and LDP.

One solution to the state growth problem would be shrinking the full-mesh of PE-to-PE LSPs, e.g. pushing the MPLS-TE endpoints from PE deeper in the network core, say up to the aggregation layer and then deploying LDP on the PEs and over the MPLS-TE tunnels. This would reduce the size of the tunnel full-mesh significantly, but prevent the use of peripheral connections for MPLS-TE. In other words, this reduces the effect of using MPLS-TE for network resource optimization.

Besides losing full MPLS-TE functionality, the use of unmodified LDP means that route summarization could not deployed in such scenario. Based on these limitations, we are not going to discuss this approach for MPLS-TE scaling in this publication. Notice that replacing LDP with MPLS-TE in the non-core sectors of the network does not solve global traffic engineer problem, though allows for construction locally optimal LSPs in every level. However, similar to the “LDP-in-the-edge approach”, this solution reduces overall effectiveness of MPLS-TE in the network. There are other ways to improve MPLS-TE scaling properties which we are going to discuss further.

Hierarchical RSVP-TE LSPs

As mentioned previously, hierarchies are the key to IP routing scalability. Based on this, it could be reasonable to assume that hierarchical LSPs could be used to solve MPLS-TE scaling problems. Hierarchical LSPs have been in the MPLS standards almost since the inception and their definition has been finalized in RFC 4206. The idea behind the hierarchical LSPs is the use of layered MPLS-TE tunnels. The first layer of MPLS-TE tunnels is used to establish forwarding adjacencies for the link-state IGP. These forwarding adjacencies are then flooded in link-state advertisements and added into Traffic Engineering Database (TED), though not the LSDB. In fact, the first mesh looks exactly like set of IGP logical connections, but get advertised only into TED. It is important that link-state information is not flooded across this mesh, hence the name “forwarding adjacency”. This first layer of MPLS LSPs typically overlays the network core.

The second level of MPLS-TE tunnels is constructed with the first-level mesh added to the TED in mind, using cSPF or manual programming. From the standpoint of the second-level mesh, the physical core topology is hidden and replaced with the full-mesh of RSVP-TE signaled “links”. Effectively, the second-level tunnels are nested within the first level mesh and are never visible to the core network. The second-level mesh normally spans edge-to-edge and connects the PEs. At the data plane, hierarchical LSPs are realized by means of label stacking.

Not every instance of IOS code supports the classic RFC4206 FAs. However, it is possible to add an mGRE tunnel spanning between the P-routers forming the first level mesh and use it to route the PE-PE MPLS TE LSPs (second level). This solution does not require running additional set of IGP adjacencies over the mGRE tunnel and therefore has acceptable scalability properties. The mGRE tunnel could be overlaid over a classic, non-FA RSVP-TE tunnel mesh used for traffic engineering between the P-routers. This solution creates additional overhead for the hierarchical tunnel mesh, but allows for implementing hierarchical LSPs in the IOS code that does not support the RFC 4206 FAs.

At last, it is worth pointing out that Cisco puts different meaning into the term Forwarding Adjacencies, as it is implemented in IOS software. Instead of advertising the MPLS-TE tunnels into TED, Cisco advertises them into LSDB and uses for shortest path construction. These virtual connections are not used for LSA/LSP flooding though, just like the “classic” FAs. Such technique does not allow for hierarchical LSPs, as second-level mesh cannot be signaled over FA defined in this manner, but allows for global visibility of MPLS TE tunnels within a single IGP area, compared to Auto-Route, which only supports local visibility.

Do Hierarchical RSVP-TE LSPs Solve the Scaling Problem?

It may look like that using hierarchical LSPs along with route summarization solves the MPLS scaling problems at once. The network core has to deal with the first-level of MPLS-TE LSPS only, and it is perfectly allows for route summarization, as illustrated on the diagram below:

However, a more detailed analysis performed in [FARREL-MPLS-TE-SCALE-2] shows that in popular ISP network topologies deploying multi-level LSP hierarchies does not result in significant benefits. Firstly, there are still “congestion points” remaining at the level where 1st and 2nd layer-meshes are joined. This layer has to struggle with fast-growing number of LSP states as it has to support both levels of MPLS TE meshes. Furthermore, the management burden associated with deploying multiple MPLS-TE meshes along with Inter-Area MPLS TE tunnels is significant, and seriously impacts network growth. The same arguments apply to the “hybrid” solution that uses mGRE tunnel in the core, as every P router needs to have a CEF entry for every other P router connected to the mGRE tunnel. Not to mention the underlying P-P RSVP-TE mesh adding even more forwarding states in the P-routers.

Multipoint-to-Point RSVP-TE LSPs

What is the key issue that makes MPLS-TE hard to scale? The root cause is the point-to-point nature of MPLS-TE LSPs. This results in rapid forwarding state proliferation in transit nodes. An alternative solution to using LSP hierarchies is the use if Multipoint-to-Point RSVP-TE LSPs. Similar to LDP, it is possible to merge the LSPs that traverse the same egress link and terminate on the same endpoint. With the existing protocol design, RSVP-TE cannot do that, but certain protocol extensions proposed in [YASUKAWA-MP2P-LSP] allow for automatic RSVP-TE merging. Intuitively it is clear that the number of forwarding states with MP2P LSPs grows like O(N) where N is the number of PE routers, compared to O(N^2) in classic MPLS-TE. Effectively, MP2P RSVP-TE signaled LSPs have the same scaling properties as LDP signaled LSPs, with the added benefits of MPLS-TE functionality and Inter-Area signaling. Based on the scaling properties, th use of MP2P Inter-Area LSPs seems to be a promising direction toward scaling MPLS networks.

BGP-Based Hierarchical LSPs

As we have seen, the default operations mode for MPLS-TE does not offer enough scaling even with LSP hierarchies. It is worth asking, whether it is possible to create hierarchical LSPs using signaling other than RSVP-TE. As you remember, BGP extensions could be used to transport MPLS labels. This gives an idea of creating nested LSPs by overlaying BGP mesh over IGP areas. Here is an illustration of this concept:

In this sample scenario, there are three IGP areas, and ABRs optimally summarizing their area address ranges, therefore hiding the PE /32 prefixes. Inside every area, LDP or RSVP-TE could be used for constructing intra-area LSPs, for example LSPs from PEs to ABRs. At the same time, all PEs establish BGP sessions with their nearest ABRs and ABRs connect in full iBGP mesh, treating the PEs as route-reflector clients. This allows for the PEs to propagate their Loopback /32 prefixes via BGP. The iBGP peering should be done using another set of Loopback interfaces (call them IGP-routed) that are used to build transport LSPs inside every area..

The only routers that will see the PE loopback prefixes (BGP routed) are the other PEs and the ABRs. The next step is configuring the ABRs that act as route-reflectors for the PE routers to change the BGP IP next-hop to self and activate MPLS label propagation over all iBGP peering sessions. The net result is overlay label distribution process. Every PE would use two labels in the stack to get to another PE’s Loopback (BGP propagated) by means of BGP next-hop recursive resolution. The topmost label (LDP or RSVP-TE) is used to steer packet to the nearest ABR, using the transport LSP built toward the IGP-routed Loopback interface. The bottom label (BGP propagated) identifies the PE prefix within the context of a given ABR. Every ABR will pop incoming LDP/RSVP-TE label, then swap the PE label to the label understood by another ABR/PE (as signaled via BGP) and then push new LDP label that starts a new LSP to reach the next ABR/PE. Effectively, this implements two-level hierarchical LSP end-to-end between any pair of PEs. This behavior is a result of BGP’s ability to propagate label information and the recursive next-hop resolution process.

How well does this approach scale? Firstly, using BGP for prefix distribution ensures that we may advertise truly large amount of PE prefixes without any serious problems (though DFZ systems operators may disagree). At first sight, routing convergence may seem to be a problem, as loss of any PE router should result in iBGP session timing out based on BGP keepalives. However, if BGP next-hop tracking (see [BGP-NEXTHOP]) is used within every area, then ABRs will be able to detect loss of PE at the pace of IGP converges. Link failures within an area will also be handled by IGP or, possibly, using intra-area protection mechanisms, such as MPLS/IP Fast Re-Route. Now for the drawback of the BGP-signaled hierarchical LSPs:

1. Configuration and Management overhead. In popular BGP-free core design, P routers (which typically include ABRs) do not run BGP. Adding extra mesh of BGP peering sessions requires configuring all PEs and, specifically, ABRs, which involves non-trivial efforts. This slows initial deployment process and complicates further operations.
2. ABR failure protection. Hierarchical LSPs constructed using BGP, consist of multiple disconnected LDP/RSVP-TE signaled segments, e.g. PE-ABR1, ABR1-ABR2. Within the current set of RSVP-TE FRR features, it is not possible to protect the LSP endpoint nodes, due to local significance of the exposed label. There is work in progress to implement endpoint node FRR protection, but this feature is not yet available. This might be a problem, as it makes the network core vulnerable to ABR failures.
3. The amount of forwarding states increases in the PE and (additionally) ABR nodes. However, unlike MPLS TE LSPs this growth is proportional to the number of PE routers, which is approximately the same scaling limitation we would have with MP2P MPLS-TE LSPs. Therefore, growth of forwarding state in the ABRs should not be a big problem, especially since no other nodes than PEs/ABRs are affected.

To summarize, it is possible to deploy hierarchical LSPs and get LDP/single-area TE working with multi-area IGPs without any updates to existing protocols (LDP, BGP, RSVP-TE). The main drawback is excessive management overhead and lack of MPLS FRR protection features for the ABRs. However, this is the only approach that does not require any changes to the running software, as all functionality is implemented using existing protocol features.

LDP Extensions to work with Route Summarization

In order to make native LDP (RFC 3036) work with IGP route summarization it is possible to extend the protocol in some way. Up to date, there are two main approaches: one of them utilizes prefix leaking via LDP and the other implements hierarchical LSPs using LDP.

LDP Prefix Leaking (Interarea LDP)

A very simply extension to LDP allows for performing route summarization. Per the LDP’s RFC, prior to installing a label mapping for prefix X, local router needs to ensure there is an exact match for X in the RIB. RFC 5283 suggest changing this verification procedure to the longest match: if there is a prefix Z in the RIB such that X is a subnet to Z then keep this label mapping. It is important to notice that LFIB is not aggregated in any manner – all label mappings received via LDP are maintained and propagated further. Things are different at the RIB level, however. Prefixes are allowed to be summarized and routing protocol operations are simplified. No end-to-end LSPs are broken, because label mappings for specific prefixes are maintained along the path.

What are the drawbacks of this approach? Obviously, LFIB size growth (forwarding state) is the first one. It is possible to argue that maintaining LFIB is less burdensome compared to maintaining IGP databases, so this could be acceptable. However, it is well known that IGP convergence is seriously affected by FIB sizes, not just IGP protocol data structures, as updating large FIB takes considerable time. Based on this, the LDP “leaking” approach does not solve all scalability issues. On the other hand, keeping detailed information in LFIB allows for end-to-end connectivity tracking, thanks to LDP ordered label propagation. If on area loses a prefix, LDP will signal the loss of label mapping, even though no specific information will ever leak to IGP. This is the flip-side of having detailed information at the forwarding plane level. The other problem that could be pointed out is LDP signaling overhead. However, since LDP is “on-demand” and “distance-vector” protocol it does not impose as much problems as say link-state IGP flooding.

Aggregated FEC Approach

This approach has been suggested by G. Swallow of Cisco Systems – see [SWALLOW-AGG-FEC]. It requires modifications to LDP and slight changes to the forwarding plane behavior. Here is how it works. When an ABR aggregates prefixes {X1…Xn} into a new summary prefix X it generates a label for X (aggregate FEC) and propagates it to other areas, creating an LSP for the aggregate FEC that terminates at the ABR. The LDP mappings are propagated using “aggregate FEC” type to signal special processing for packets matching this prefix. The LSP constructed for such FEC has PHP (penultimate hop popping) disabled for the reason we’ll explain later. All that other routers see in their RIB/FIB is the summary prefix X and the corresponding LFIB element (aggregate FEC/label). In addition to propagating the route/label for X, the same ABR also applies special hash function to the IP addresses {X1…Xn} (the specific prefixes) and generates local labels based on the result of this function. These new algorithmic labels are stored under the context of the “aggregated” label generated for prefix X. That is, these labels should only be interpreted in the association with the “aggregate” label. The algorithmic labels are further stitched with the labels the ABR learns via LDP from the source area for the prefixes {X1…Xn}.

The last piece of the puzzle is how PE creates the label stack for a specific prefix Xi. When a PE attempts to encapsulate a packet destined to Xi on ip2mpls edge it looks up the LFIB for exact match. If no exact match is found, but there is a matching aggregate FEC X, the PE will use the same hash function that ABR used previously on Xi to create an algorithmic label for Xi. The PE then stacks this “algorithmic” label under the label for the aggregate FEC X and sends the packet with two labels – the topmost for the aggregate X and the bottom for Xi. The packet will arrive at the ABR that originated the summary prefix X, with the topmost label NOT being removed by PHP mechanism (as mentioned previously). This allows the ABR to correctly determine the context for the bottom label. The topmost label is removed, and the next de-aggregation label for Xi is used to lookup the real label (stitched with the de-aggregation label for Xi) to be used for further switching.

This method is backward compatible with classic LDP implementation and will interoperate with other LDP deployments. Notice that there is no control-plane correlation between ABRs and PE, like there is in the case of BGP-signaled hierarchical LSPs. Instead, synchronization is achieved by using the same globally known hash function that produces de-aggregation labels. This method reduces control plane overhead associated with hierarchical LSP construction, but has one drawback – there is no end-to-end reachability signaling, like there was in RFC 5283 approach. That is, if an area loses prefix for PE, there is no way to signal this via LDP, as only aggregated FEC is being propagated. The presentation [SWALLOW-SCALING-MPLS] suggests a generic solution to this problem, by means of IGP protocol extension. In addition to flooding a summary prefix, the ABR is responsible for flooding a bit-vector that corresponds to every possible /32 under the summary. For example, for a /16 prefix there should be a 2^16 bit vector, where bit flag being equal to one means the corresponding /32 prefix is reachable and zero means it being unreachable. This scheme allows for certain optimizations, such as using Bloom filters (see [BLOOM-FILTER]) for information compression. This approach is known as Summarized Route Detailed Reachability (SRDR). The SRDR approach solves the problem of hiding the reachability information at the cost of modification to IGP signaling. An alternative is using tuned BGP keepalives. This, however, puts high stress on router’s control plane. A better alternative is using data-plane reachability discovery, such as using multi-hop BFD ([BFD-MULTIHOP]). The last two approaches do not require any modifications to IGP protocols and therefore better interoperate with existing networks.

Hierarchical LDP

This approach, an extension to RFC 5283, has been proposed by Kireeti Kompella of Juniper, but never has been officially documented and presented for peer review. There were only a few presentations made at various conferences, such as [KOMPELLA-HLDP]. No IETF draft is available, so we can only guess about the protocol details. In a nutshell, it seems that the idea is running a an overlay mesh of LDP sessions between PEs/ABRs similar to the BGP approach and using stacked FEC advertisements. The topmost FEC in such advertisement corresponds to the summarized prefix advertised by the ABR. This FEC is flooded across all areas and local mappings are used to construct an LSP terminating at the ABR. So far, this looks similar to the aggregated FEC approach. However, instead of using algorithmic label generation, the PE and ABR directly exchange their bindings on the specific prefixes, using new form of FEC announcement – hierarchical FEC stacking. ABR advertises aggregated FEC along with aggregated label and nested specific labels. The PE knows what labels the ABR is expecting for the specific prefixes, and may construct a two-label stack consisting of the “aggregate” label and “specific” label learned via the directed LDP session. The specific prefixes are accepted by the virtue of RFC 5283 extension, which allows detailed FEC information if there is a summary prefix in the RIB matching the specific prefix.

The hierarchical LDP approach maintains control-plane connection between the PE and the ABRs. Most likely, this means manual configuration of directed LDP sessions, very much similar to the BGP approach. The benefit is the control-plane reachability signaling and better extensibility compared to the Aggregated FEC approach. Another benefit is that BGP mesh is left intact and only LDP configuration has to be modified. However, it seems like that further work on Hierarchical LDP extensions has been abandoned, as there are no publications or discussions on this subject.

Hierarchical LSP Commonalities

So far we reviewed three different approaches to constructing hierarchical LSPs. The first one uses RSVP-TE forwarding adjacencies, the second one uses BGP label propagation and the last two use LDP extensions. All the approaches result in constructing transport LSP segments, terminating at the ABRs. For example, in RSVP-TE approach there are LSPs connecting the ABRs. In BGP approach, there are LSP segments connecting PEs to ABRs. As we mentioned previously, the current set of MPLS FRR features does not protect LSP endpoints. As a direct result, using hierarchical LSPs decreases the effectiveness of MPLS FRR protection. There is a work in progress on extending FRR protection to LSP endpoints, but there are no complete working solutions at the moment.

Summary

We have reviewed various aspects of MPLS technology scaling. The two main ones are scaling IGP protocols by using route summarization/areas and getting MPLS to work with summarization. A number of approaches are available to solve the latter problem, and practically all of them (with except to inter-area MPLS TE) are based on hierarchical LSP construction. Some approaches, such as BGP-signaled hierarchical LSPs are ready to be deployed using existing protocol functionality, at the expense of added management overhead. Others require modifications to control/forwarding plane behavior.

It looked like there was high interest in MPLS scaling problems about 3-4 years ago (2006-2007), but this topic looks to be abandoned nowadays. There is no active work in progress on LDP extensions mentioned above, however Multipoint-to-Point RSVO-TE LSP draft document [YASUKAWA-MP2P-LSP] seems to be making progress through IETF. Based on this, it looks like using Inter-Area RSVP-TE with MP2P extensions is going to be the main solution to scaling the MPLS networks of the future.

Further Reading

RFC 3036: LDP Specification
[FARREL-MPLS-TE-SCALE-1] MPLS-TE Does Not Scale
[RSVP-TE-INTERAREA] Requirements for Inter-Area MPLS TE Traffic Engineering
Inter-Area Traffic Engineering
[PCE-ARCH] A Path-Computation Element Based Architecture
[YASUKAWA-MP2P-LSP] Multipoint-to-Point RSVP-TE Signaled LSPs
[MINEI-MPLS-SCALING] Scaling Considerations in MPLS Networks
[SWALLOW-SCALING-MPLS] George Swallow: Scaling MPLS, MPLS2007 (no public link available)
[KOMPELLSA-HLDP] Techniques for Scaling LDP, Kireeti Kompella MPLS2007 (no public link available)
[SWALLOW-AGG-FEC] Network Scaling with Aggregated IP LSPs
LDP Extensions for Inter-Area LSPs
[KLEIN] Hierarchical Routing for Large Networks
[OSPF-FAST] OSPF Fast Convergence
[L3VPN-MGRE] Layer 3 VPNs over Multipoint GRE
[FARREL-MPLS-TE-SCALE-2] MPLS TE Scaling Analysis
RFC 4206: LSP Hierarchy
[BGP-NEXTHOP] BGP Next-Hop Tracking
[BLOOM-FILTER] Bloom Filter
[MINEI-MPLS-SCALING] Scaling Considerations in MPLS Networks
[L3VPN-MGRE] Layer 3 VPNs over Multipoint GRE
[FARREL-MPLS-TE-SCALE-2] MPLS TE Scaling Analysis
RFC 4206: LSP Hierarchy
[BGP-NEXTHOP] BGP Next-Hop Tracking
[BFD-MULTIHOP] Multihop Extension to BFD

We are so excited here at INE for the live, online 5-Day CCNP bootcamp that starts Monday, August 16, 2010 . I look forward to seeing many of our aspiring CCIE candidates in this course. These students realize that they really need to improve their foundation Tier 1 knowledge as they seek to conquer the Lab Exam beast.

In this blog post, we are going to provide a sneak peek into some of the awesome information shared in the TSHOOT section of the bootcamp regarding the Troubleshooting of EIGRP. This can prove critical in the Troubleshooting and Configuration sections of the CCIE R&S Lab Exam, as well as the TSHOOT CCNP exam (duh!).

Where Is My Neighbor!?!?!?

The first thing that you want to master when it comes to troubleshooting EIGRP is the ‘workflow” that EIGRP follows in its operation. We can subdivide the processes of this exciting protocol into four discrete steps:

• Discovery of neighbors
• Exchange of topology information
• Best path selection
• Neighbor and topology table maintaince

Discovery of Neighbors

Remember that EIGRP discovers neighbors through bi-directional multicast by default. IP protocol 88 and 224.0.0.10 are the key parameters we need to watch out for here. Could there be issues with NBMA pseudo-broadcast support or filtering causing neighbor discovery to fail? Certainly things to examine in the topology. Also, watch out for the neighbor command under the EIGRP routing process. This feature causes the use of unicast packets for neighbor creation exclusively and must be agreed upon by BOTH neighbors.

Another area we need to be aware of is the attributes that must match in order for neighbors to form. Sure this list is nowhere near as lengthy as the parameters that we have to watch out for in OSPF networks, but the list is just as critical:

• Common Subnet (Must be the primary address – not a secondary)
• Autonomous System Number
• Authentication
• K values (metric weights)

Exchange of Topology Information

When it comes to the exchange of topology information, this is done with unicast. Notice that connectivity for neighborship still requires mutlicast communications (unless you are using the neighbor command).

Remember that EIGRP will only advertise those prefixes that it installs in the routing table. This is an excellent time to review the difference from the Topology Table to the Routing Table.

Important troubleshooting considerations for the exchange of topology information include:

• Automatic summarization in use?
• Split horizon settings
• The use of duplicate router IDs preventing external route introduction
• No seed metric set for external prefixes
• Filtering through the use of distribute lists

Please consider these troubleshooting aspects for these two phases of EIGRP’s operation. We will cover more in the next installment coming soon…

Time is a valuable resource in the lab.   In a lab task, if asked to configure a policy-map named “BOB”, it doesn’t get the same point value if we happen to accidentally name it “bob”, especially  if they are looking to see if you configured what they asked for.

The challenge is, that when reviewing a lab task, and we discover that we need to change a name, it could be a hassle, as we need to remove the policy-map, recreate the policy map, and then put it in place again.

So if you are down to the last minute, here is a time saving solution, that can assist with that process.

IOS allows us to rename a policy-map, and the IOS will swap out the name in other areas of the configuration that reference that policy map.

Here is an example, of a policy map from Volume 2, lab 5.

Rack1R5#show run policy-map
Building configuration...

Current configuration : 352 bytes
!
policy-map TRANSIT_RATE_LIMIT
class FRAGMENTS
   police rate 1000000 pps burst 200000 packets
policy-map type port-filter HOST_PORT_FILTER
class CLOSED_PORTS
   drop
policy-map CEF_EXCEPTION_RATE_LIMIT
class class-default
   police rate 100 pps burst 20 packets
policy-map HOST_RATE_LIMIT
class ICMP
   police rate 10 pps burst 5 packets
!
end

Rack1R5#show run | begin control
control-plane host
service-policy input HOST_RATE_LIMIT
service-policy type port-filter input HOST_PORT_FILTER
!
control-plane transit
service-policy input TRANSIT_RATE_LIMIT
!
control-plane cef-exception
service-policy input CEF_EXCEPTION_RATE_LIMIT

Let’s say that after reviewing our configuration, we discovered that the policy-map for the cef-exception sub interface of the control plane should have been named “NEW-NAME-CEF”.

To change it everywhere in the configuration, instead of creating it new, and replacing it, we could simply do this:

Rack1R5(config)#policy-map CEF_EXCEPTION_RATE_LIMIT
Rack1R5(config-pmap)#rename NEW-NAME-CEF

Now, when we look at the configuration, we can see that not only the name has changed for the policy-map, but it also updated our control-plane configuration to reflect the new name there as well:

Rack1R5#show run policy-map
Building configuration...

Current configuration : 340 bytes
!
policy-map TRANSIT_RATE_LIMIT
class FRAGMENTS
   police rate 1000000 pps burst 200000 packets
policy-map type port-filter HOST_PORT_FILTER
class CLOSED_PORTS
   drop
policy-map NEW-NAME-CEF
class class-default
   police rate 100 pps burst 20 packets
policy-map HOST_RATE_LIMIT
class ICMP
   police rate 10 pps burst 5 packets
!
end

Rack1R5#show run | begin control
control-plane host
service-policy input HOST_RATE_LIMIT
service-policy type port-filter input HOST_PORT_FILTER
!
control-plane transit
service-policy input TRANSIT_RATE_LIMIT
!
control-plane cef-exception
service-policy input NEW-NAME-CEF
!
!

Best wishes on your studies, and may your policy-maps be named correctly the first time around.

Keith