CCIE Training

I enjoyed Petr’s article regarding explicit next hop.  It reminded me of a scenario where a redistributed route, going into OSPF conditionally worked, depending on which reachable next hop was used.

Here is the topology for the scenario:

Here is the relevant (and working ) information for R1.

When we replace the static route, with a new reachable next hop, we loose the ability to ping 100.100.100.3

When we change the next hop for the static route, (which is being redistributed into OSPF), the route to 100.100.100.0/24 no longer works, even though we have verified ability to ping the new next hop.

Can you solve this puzzle?  Please post your ideas!

For more troubleshooting scenarios, please see our CCIE Route-Switch workbooks, volume 2, for more than 100 challenging troubleshooting scenarios.

We will post the results right here, in a few days, after you have had a chance to post your comments and ideas.

Best wishes,

Keith

For each new CCIE Testimonial we are extending the seven years of success sale! Share your INE success story and congratulations to the following new CCIE Testimonials who have extended the sale thus far!

Thomas Fischer, CCIE #26636 – Routing & Switching

I am proud to let you know, that I passed my CCIE R&S Lab in Brussels on Aug. 5th. This was my second attempt. I want to express my deepest appreciation for your Products. I am a self-paced student, using Vol1 (*****), Vol2 (****) and Vol4 (***). Thanks INE, it feels so good to have a social life again )

Matthew Ayre, CCIE #26654 – Service Provider

Big shout out for INE and their OEQ / lab preparation resources! I just cleared service provider on second attempt finishing about an hour and a half early. Was ~7% of passing the first time using INE 1 & 2 as my primary material then just drilled down on the finer details reading theory. The workbooks really developed the speed and confidence required to beat the exam!

Prateek Madaan, CCIE #26772 – Security

Had been a long and tough journey. Would really like to thank INE from the Core of my heart for facilitating in imparting the skills required not to just pass the exam but to DESERVE it as well…

There are many workbooks available which I prepared along with INE , do not want to name or list any one of them…or make any comparisons…But in comparisons INEs Security Workbooks may sound tough as compared to others BUT once you go through these workbooks is when you actually feel DESERVED the tag rather than just passing it.. Each of these workbooks and the tasks test each and every technology in detail and till the dead end….

In my last attempt on Version 2 I was deprived of the number by 1%, still followed and trusted INE workbooks and finally it helped….Today I am more happy not to procure the number but to actually have the feeling of confidence that ‘YES this time I deserve to be a CCIE’ and all due to the exhaustive INE workbooks….

Update!
Olusegun Olurotimi Medeyinlo, CCIE #26683 – Routing & Switching

I the Passed the CCIE R&S lab in Brussels on my Second attempt. I’d like to thank the instructors at INE for their excellent workbooks and blogs. Special thanks to Keith Barker for his encouragement and advice.


Now, I have my own CCIE number #22683.

Congratulations to everyone who passed the CCIE Lab Exam. Our instructors, authors, and staff have been committed to helping you pass your exam for the past seven years and we will continue to make your exam our number one priority. Only at INE.

To celebrate INE’s 7th year anniversary, we will be extending a 30% discount on all self-paced products and 15% discount on instructor-led training for 77 hours for each success story we receive during the month of August. Over the years we have helped hundreds of candidates pass the CCIE Lab Exam, and we would like to thank all of you for making it possible.

Use discount code: INE7SP to save 30% on self-paced material or INE7ILT to save 15% on instructor-led training.

For the latest Video Blog – simply click the link below:

The GradedLabs Control Panel

This Video Blog demonstrates and discusses uses for this powerful rack rental tool.

During this video I mentioned that it is now possible to view and download your saved configs, but I forgot to show from where. The location is shown below. Enjoy!

“Why doesn’t this PING work!?!”

Here is a simple 3 router configuration, well at least it is simple on 2 of the 3 routers. R1 and R3 are configured quite traditionally, but R2 is a bit more involved.
Here is the diagram.

Here are the details.

R2 is using a VRF which includes both LAN interfaces. R2 is also acting as a Zone Based Firewall in transparent mode, allowing all ICMP traffic in both directions, as well as SSH from the inside to the outside networks. R2 has a bridged virtual interface in the 10.123.0.0/24 network. All are running OSPF, but pings issued from R2 to the loopbacks of R1 and R3 are failing.

Can you identify why?
Here is the relevant output:

R1#show ip ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
3.3.3.3           1   FULL/DR         00:00:39    10.123.0.3      FastEthernet0/0
10.123.0.2        1   FULL/BDR        00:00:32    10.123.0.2      FastEthernet0/0
R1#show ip route ospf
     3.0.0.0/32 is subnetted, 1 subnets
O       3.3.3.3 [110/2] via 10.123.0.3, 00:01:33, FastEthernet0/0

R1#ping 3.3.3.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 60/88/172 ms
R1#ssh -l admin 3.3.3.3
Password: <password>

R3#show ssh
Connection Version Mode Encryption  Hmac         State                 Username
0          1.99     IN   aes128-cbc  hmac-sha1    Session started       admin
0          1.99     OUT  aes128-cbc  hmac-sha1    Session started       admin
%No SSHv1 server connections running.
R3#exit

[Connection to 3.3.3.3 closed by foreign host]
R1#

Now for R2:

R2#show ip ospf neighbor
Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1           1   FULL/DROTHER    00:00:37    10.123.0.1      BVI1
3.3.3.3           1   FULL/DR         00:00:35    10.123.0.3      BVI1

R2#show ip route ospf

R2#show policy-map type inspect zone-pair
 Zone-pair: zp-in-to-out

  Service-policy inspect : p-in-to-out

    Class-map: c-in-to-out (match-any)
      Match: protocol icmp
        4 packets, 320 bytes
        30 second rate 0 bps
      Match: protocol ssh
        3 packets, 72 bytes
        30 second rate 0 bps
      Inspect
        Packet inspection statistics [process switch:fast switch]
        tcp packets: [4:390]
        icmp packets: [0:50]

        Session creations since subsystem startup or last reset 8
        Current session counts (estab/half-open/terminating) [0:0:0]
        Maxever session counts (estab/half-open/terminating) [2:1:1]
        Last session created 00:02:23
        Last statistic reset never
        Last session creation rate 0
        Maxever session creation rate 3
        Last half-open session total 0

    Class-map: class-default (match-any)
      Match: any
      Drop (default action)
        0 packets, 0 bytes
 Zone-pair: zp-out-to-in

  Service-policy inspect : p-out-to-in

    Class-map: c-out-to-in (match-all)
      Match: protocol icmp
      Inspect
        Packet inspection statistics [process switch:fast switch]
        icmp packets: [0:20]

        Session creations since subsystem startup or last reset 2
        Current session counts (estab/half-open/terminating) [0:0:0]
        Maxever session counts (estab/half-open/terminating) [1:1:0]
        Last session created 00:25:24
        Last statistic reset never
        Last session creation rate 0
        Maxever session creation rate 1
        Last half-open session total 0

    Class-map: class-default (match-any)
      Match: any
      Drop (default action)
        4 packets, 96 bytes

R2#ping 3.3.3.3
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 3.3.3.3, timeout is 2 seconds:
.....
Success rate is 0 percent (0/5)

R2# show run
version 12.4
hostname R2
!
ip vrf myvrf
!
class-map type inspect match-any c-in-to-out
 match protocol icmp
 match protocol ssh
class-map type inspect match-all c-out-to-in
 match protocol icmp
!
policy-map type inspect p-in-to-out
 class type inspect c-in-to-out
  inspect
 class class-default
policy-map type inspect p-out-to-in
 class type inspect c-out-to-in
  inspect
 class class-default
!
zone security inside
zone security outside
zone-pair security zp-in-to-out source inside destination outside
 service-policy type inspect p-in-to-out
zone-pair security zp-out-to-in source outside destination inside
 service-policy type inspect p-out-to-in
bridge irb
!
interface FastEthernet0/0
 ip vrf forwarding myvrf
 no ip address
 zone-member security inside
 bridge-group 1
!
interface FastEthernet0/1
 ip vrf forwarding myvrf
 no ip address
 zone-member security outside
 bridge-group 1
!
interface BVI1
 ip vrf forwarding myvrf
 ip address 10.123.0.2 255.255.255.0
!
router ospf 1 vrf myvrf
 router-id 10.123.0.2
 network 0.0.0.0 255.255.255.255 area 0
!
bridge 1 protocol ieee
bridge 1 route ip
end

Here is R3:

R3#show ip ospf neighbor

Neighbor ID     Pri   State           Dead Time   Address         Interface
1.1.1.1           1   FULL/DROTHER    00:00:32    10.123.0.1      FastEthernet0/1
10.123.0.2        1   FULL/BDR        00:00:31    10.123.0.2      FastEthernet0/1

R3#show ip route ospf
     1.0.0.0/32 is subnetted, 1 subnets
O       1.1.1.1 [110/2] via 10.123.0.1, 00:29:36, FastEthernet0/1

R3#ping 1.1.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 1.1.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 76/117/176 ms
R3#

Similar configuration scenarios are included in both our RS and SC workbooks at INE.

Take a moment, and post your ideas on why the PING from R2 is failing, and thanks for taking the time to assist!

Best wishes, Keith

RFC, or Request for Comments, are documents published that describe various items surrounding computer networking. Generally, these are memorandums published by the Internet Engineering Task Force.

RFCs can be a great resource. For some unknown reason, most candidates preparing for the CCIE don’t take the time to review these documents, which can be very helpful in assisting with understanding the how and why of various networking components. Perhaps the language is a bit dry, or they prefer books with shiny covers.

There are a variety of status classifications. These include, but are not limited to: standards, informational, best current practices. Some are very serious discussions of the deep inner workings, where others are just there for entertainment, such as RFC 1149 and 2549.

If you aren’t sure whether a RFC is intended to be serious or entertainment, check the date. If it was one from 1 April of any year, most likely it falls into the category of entertainment.

http://www.rfc-archive.org/1+april+rfc.php

Language is included to define how an item is intended to behave. RFC 2119 lists some of these requirements. Requirements are shown capitalized, and include the following: MUST, MUST NOT, SHOULD, SHOULD NOT, MAY, RECOMMENDED, NOT RECOMMENDED, OPTIONAL.

RFCs are not a “magic bullet” for lab preparation. Most students that are familiar with RFCs tend to be more comfortable with the technologies discussed.

RFCs can be viewed online at a number of sites, including the following:

http://www.ietf.org/rfc.html

http://www.rfc-editor.org/

Most search engines will also give you results for RFCs by number or topic.

Quick quiz.

Four questions on RFCs that most people are generally familiar with. Questions are True or False, and the answers can be found fairly quickly, if you know where to look.

T or F
RFC 3330, which describes Special Use IPv4 Addresses, is obsolete.

T or F
RFC 1812, which discusses requirements for IPv4 Routers, states that subnet bits MUST be contiguous.

T or F
RFC 2827 discusses ingress filtering mechanisms, including the effects of multihoming.

T or F
RFC 1918 does not address security issues.

How did you do? Two of these are true and two are false. If you got all four correct, congratulations. If you’ve never heard of these RFCs, perhaps it is time to do some additional reading.

Bonus Question:

True or False:
Neither Cisco nor Juniper devices are compliant with RFC 5841.

Recently, there have been a lot of talks around LISP – location and Identity Separation Protocol. This is a “new” technology aiming to resolve some of the Internet scalability issues and which has been implemented in IOS 15.x. In this blog publication we are going to give a general overview of LISP, pointing out benefits as well as drawbacks of the technology.

Hierarchical Routing and its Problems

Ever since ARPANet has been launched, routing in packet switched networks (PSNs) has been based on hierarchical network addressing to achieve scalability. The groundbreaking work by Kleinrock and Kamoun named “Hierarchical Routing for Large Networks, Performance Evaluation and Optimization” ([KLEIN]) clearly outlined all ideas of hierarchical routing. The main result of this work is that for a network of arbitrary connected N nodes it is possible to devise a hierarchical clustering scheme where nodes inside a single cluster only have to know routes to the nodes in the same cluster and other clusters, provided that addresses are assigned to the nodes following the hierarchical structure. The routing is assumed to be classic shortest-path selection process. Under optimal partitioning scheme, the size of routing table on every router would be of order O(log(N)).

Even though this approach promises compact routing tables if implemented properly, there are three main drawbacks here. Firstly, this approach required strict, topology-based addressing, which was perfect in static environment but didn’t work well with mobile nodes, changing their point of attachment. Secondly, it resulted in suboptimal routes, as every cluster had limited knowledge of the outside topology. The resulting routing paths were longer than shortest paths, or as they say have “stretch” factor above 1 – e.g. a stretch of 2 means that the path is twice as longer as optimal shortest path. In addition to these two main issues, hierarchical routing adds management burden of keeping address space hierarchical, following the network clustering structure.

The modern Internet was built on the same shortest path routing concept using hierarchical addressing for routing scalability (though BGP makes it more complicated that it could have been). This model started showing some signs of instability at the end of 90s. Dramatic growth of Internet routing tables put serious burden on the routing systems as a direct result of explosive routing-tables growth. This growth was a result of two main processes: topology-independent addresses that were obtained for Internet multihoming and routing optimization which has been achieved by injecting “longer” or more specific prefixes into Internet routing tables. These processes attempt to overcome limited mobility and suboptimal routing inherent to hierarchical routing. Even though BGP was designed with scalability in mind, carrying modern Internet routing tables in the core networks becomes a huge burden, not only because of the size of RIB and FIB but also because of the network instabilities exposed by poor summarization. There are some approaches to alleviate the problem, such as Virtual Aggregation ([VA-DFZ]) for in-core FIB size reduction or BGP-aware DNS (see [BGP-DNS]) for efficient multihoming based on PA (Provider Aggregatable) addresses, but they do not offer ultimate solution and have not been widely deployed.

What is LISP

The main problem that causes routing table explosion is inefficient multihoming in topology-aware addressing. With topology-based numbering, node address represents its relative location in the topology. At the same time, mobility or ability to change attachment to the topology (e.g. multihoming) requires the node address to be more of node identifier, which does not change with topology re-attachment. The dual function of IPv4 address (or any topology-based address, e.g. IPv6) prevents it from being effectively used in either role, and this fact has been acknowledged for many years. However, business needs do not allow easy transition from a topology-based addressing to any other, more efficient solution, thought there have been offered a lot of those.

What LISP offers is loosening this location/ID duality by creating two parallel IPv4/IPv6 address spaces: one serving to identify locations (RLOCs or routing locators) and another serving to identify endpoint (EID or endpoint identifiers). To separate the two spaced, tunneling is implemented with outer headers using RLOCs and inner header using EIDs. Effectively, the Internet is partitioned into the “core” based on hierarchical topology-based RLOC addressing and “edge” which uses “mobile” EIDs, with both addresses using the same format. Of course, the use of tunneling implies that EIDs have to be somehow mapped to the closes RLOCs. LISP achieves that by using ITRs (ingress tunnel routers) and ETRs (egress tunnel routers). Both types of routers reside on the “edge” of RLOC Internet and perform encapsulation, decapsulation and mapping functions: ITR receives native packets with EID addressing, finds the corresponding RLOC address of the “optimum” ETR and encapsulates the original packets in RLOC header. The packet is then routed across the network core toward the ETR where it is decapsulated and forwarded based on EID address and potentially using some other routing scheme than used for RLOCs.

In short, LISP creates two parallel namespaces (RLOCs and EIDs) mapped to different topologies (routing and endpoints). The first, underlying topology uses classic IPv4 addressing and the overlay topology may use any other (not necessarily hierarchical) identifiers. So far the EID space is normally based on IPv4 32 bit addresses – this allows for maintaing all applications unchanged. It is important to stress out that in theory EID and RLOC address spaces could be completely different: e.g. EIDs could be MAC addresses while RLOCs could be IPv4 addresses, implementing sort of Ethernet tunneling. The main LISP workhorse is the mapping layer, which represents a “directory lookup” finding the RLOC “closest” to the destination EID. There could be multiple schemes used for mapping, such as ALT, CORD, NERD, DHT etc. This is probably the most important component of LISP architecture, and hence is desires special attention.

LISP Mapping Layer

Internet has been dealing with distributed databases for quite some time, starting with DNS and BGP and evolving to Distributed Hash Tables (DHT) used in peer-to-peer overlay networks, such as Chord. You may be surprised to see BGP here, but you should not – with the growth and evolution of Internet, BGP became more of universal information distribution protocol than a simple routing solution.

In general, every scheme could be characterized as Push, Pull or Hybrid model. The Push model works by broadcasting (in some efficient manner) all mapping information to all interested nodes. A good example could be BGP – information such as VPNv4 prefixes, communities and additional attributes is “flooded” to PE routers in SP network. This model present minimum latency on information retrieval, but requires all interested nodes to store large amounts of state information. Pull models, such as DNS or DHT, distribute information more or less evenly across all participating nodes, and interested node need to issue information search request to locate the mapping. Pull models typically build overlay networks to search for information. Various techniques could be used to improve lookup latency and reliability: e.g. local caching and information duplication in multiple nodes. Pull models are in general more scalable than push, but introduce increased operations delays, sometimes poorly predictable. Finally, Hybrid models attempt to combine the best of Push and Pull by pushing (broadcasting) some condensed information (e.g. discovery hints) to the interested nodes but requiring pull operations to retrieve the actual mappings. Hybrid models are more complex but theoretically offer better performance than pure Pull models.

In addition to the amount of state stored in every participating node, another important scalability factor for the mapping layer is the amount (rate) of changes it may handle. It is obvious that Push models offer rapid propagation of changes (normally in incremental manner) and worse scalability compared to Pull models, due to the fact that changes need to propagate to every interested node. Pull models are not very rapid at propagating changes, as this requires finding the authoritative node and then informing it of the updated information. The use of some performance optimization with the Pull models may further degrade update latency, e.g. it may take some time to expire stale cached information (recall DNS caching). Hybrid models may balance between the two endpoints trading off optimal behavior for extra complexity.

We mentioned multiple mapping models, but mainly going to discuss LISP-ALT (LISP alternative topology), which is being actively tested and supported in the latest IOS releases. The core idea of ATL is to maintain IPv4-based EID address spaces strictly hierarchical and aggregatable and overlay additional MP-BGP mesh (ALT-topology) over existing IPv4 network (RLOC topology). Tunneling techniques such as GRE could be used to implement the overlay topology used to route based on EID, with the tunnel endpoints routed in RLOC space. Since EID space follows clean hierarchy, EID BGP tables should small and manageable. Mapping information is not conveyed in overlay BGP mesh, but rather stored in ETRs (or other special devices). The overlay ALT-topology is used to route LISP mapping requests and replies, based on the MP-BGP next-hop values. Effectively, LISP-ALT is a hybrid push and pull model, where overlay network is used to route LISP requests but the actual LISP mapping information is not propagated in BGP, but rather stored in mapping servers and returned in response to explicit requests. This allows for optimum summarization in EID address space.

In addition to ITR/ETR LIST-ALT also adds the “ALT” routers, which serve the purpose of additional logical endpoints for the ALT space. LISP-ALT topology aims at introducing minimal changes to existing software and, just like LISP technology in general, does not affect network endpoints (hosts) at all. Decoupling RLOC and EID topologies allows for making both of them hierarchical and aggregatable, thus reducing BGP table sizes both in network core and edge. This comes at expense of additional management burden required to keep EID/RLOC topologies strictly hierarchical. On positive note, once allocation has been properly done, any changes in physical attachment will not result in EID address space fragmentation – the overlay topology could be simply changed to accommodate for the new connection, without the need of renumbering EID spaces. Additionally, the EID-to-RLOC mapping layer allows for flexible traffic engineering – different RLOCs could be associated with different EID prefixes, even though ALT only propagates aggregated information in BGP.

The idea of separating address spaces and using overlay tunneling is not novel and has been around for years, implemented in MPLS BGP VPNs. In fact, the EID space looks exactly like a VPN implemented by means of VRF and LISP tunnels. The only significant change to this model is the addition of the mapping requests/replies which allow for the use of new unique traffic-engineering capabilities.

Have all problems been solved?

Not completely. Even though in theory the use of LISP-ALT model allows for transitioning to a highly aggegatable EID address space and reducing routing state in both the Internet core and edge domains it requires significant management efforts. This problem boils down to the fact that the same hierarchical routing model is retained with IPv4 LISP-ALT, resulting in the need to carefully manage EID address block allocation and possibly renumbering the Internet core to allow for better aggregation. Other mapping layer implementations, such as LISP-DHT (distributed hash tables) allow for effectively managing non-hierarchical, even flat address space, but have all typical drawbacks of Pull models. Furthermore, the requirement of keeping EID space strictly hierarchical in LISP-ALT seriously impacts “true mobility” implementation, where single EID could move between its points of attachment. However, the problem of “static” mobility or ISP multihoming/changing is effectively addressed, as the potential points of attachment are known in advance and accounted in the ALT topology.

Perspective

Theoretically, LISP model solves the routing table growth problem, at the expense of added management complexity. LISP offers non-disruptive transition approach, which does not affect end systems and allows for incremental deployment. Special techniques (proxy ITRs/ETRs not covered in this post) allow for LISP-enabled internet to interoperate with classic Internet during the transition period. But there is a significant problem: for any given organization, unless it is a Tier-1 ISP, there is no direct benefit of transition to LISP. Indeed, the main goal of LISP is to reduce the complexity of ever-growing DFZ tables, which is mainly a problem for core ISPs. For any company that seats at the “edge” of the Internet, the only visible effect of implementing LISP is increased complexity due to the need of renumbering and implementing an overlay topology. At the same time, the Internet edge is where LISP should be mainly deployed. Just like with IPv6, there is no motivation to transition to the new technology, which makes the perspectives of rapid LISP adoption tough.

In addition to the deployment issues, there is a theoretical problem. LISP-ALT architecture implements the same hierarchical routing model for EIDs that has been used in the Internet for years. We already mentioned some of the problems of hierarchical routing earlier in this publication. The main problem is high management burden, required to coordinate and maintain hierarchical address space. Another problem is highly suboptimal routing, which manifests in excessive stretch on the topology exhibiting scale-free (power-law) behavior. Unfortunately, modern Internet seems to be a scale-free graph, which means hierarchical routing is not the best choice (see [COMPACT]). However, dismantling the hierarchical routing model requires clean-slate redesign of modern Internet, which is simply inacceptable. And thus, just like with IPv6, the Internet society is stuck in a difficult situation, where each solution is either not very effective or requires major Internet redesign.

Further Reading

The below is the minimum recommended reading for the topics of location and ID separation. The [LISP-IPJ] contains excellent additional bibliography, which is highly recommended for further reading. Notice that there are proposal similar to LISTP, such as ILNP and IRON, which are not covered in this publication but reference below.

[KLEIN] “Hierarchical Routing for Large Networks, Performance Evaluation and Optimization”, Computer Networks, Vol. 1, No. 3, pp. 155–174, January 1977
[VA-DFZ] “Virtual Aggregation” http://www-europe.cisco.com/web/about/ac123/ac147/archived_issues/ipj_13-1/131_aggregation.html
[BGP-DNS] “BGP DNS” href=http://www.ripe.net/ripe/meetings/ripe-41/presentations/routing-opperman/index.html
[LISP-QA] “LISP Questions and Answers” http://cisco.biz/en/US/prod/collateral/iosswrel/ps6537/ps6554/ps6599/ps10800/qa_c67-582925.html
[LISP-IPJ] “LISP” http://www.ciscosystems.com/web/about/ac123/ac147/archived_issues/ipj_11-1/111_lisp.html
[LIS-CONF] “LISP Configuration Guide” http://www.cisco.com/en/US/docs/ios/lisp/configuration/guide/LISP_configuration_guide.pdf
[COMPACT] “On compact routing for the Internet” http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.102.5763&rep=rep1&type=pdf
[ILNP] “ILNP Concept of Operations” http://ilnp.cs.st-andrews.ac.uk/docs/id/draft-rja-ilnp-intro-03.txt
[IRON] “The Internet Routing Overlay Network” http://tools.ietf.org/html/draft-templin-iron-05

Summer was in full swing, and it was over 105 degrees Fahrenheit outside.   Bob was told it was a “dry heat”, but he thought “so is my oven”.  Needless to say, Bob was glad to be in the data center, where the temperature and humidity controls kept it very cold.   He had been asked to setup up a basic route-map with BGP, and here is the diagram he worked from.

The goal, was to modify BGP,  so that all traffic going towards the 1.1.1.0 network (which is sourced from AS1), traveling either from or through AS23, would only use the 13.0.0.0/24 segment (between R3 and R1), and not use the 10.0.0.0/24 segment (between R2 and R1) as a transit path.
Bob reviewed some of the BGP topics he had recently learned.   Here is the list he made of possibilities:
R1 could pre-pend to the AS path for advertisements of the 1.1.1.0/24 prefix when it is sent to R2 from R1.   This way, AS23 would see a better path through R3 rather than R2.  He tried this using the following on R1:

ip prefix-list JUST-1.1.1.0 seq 5 permit 1.1.1.0/24

route-map PRE-PEND permit 10
 match ip address prefix-list JUST-1.1.1.0
 set as-path prepend 1
route-map PRE-PEND permit 20

router bgp 1
 neighbor 10.0.0.2 route-map PRE-PEND out

Bob cleared the BGP session, just to be sure.    Unfortunately, some traffic destined to 1.1.1.0 was still flowing over the 10.0.0.0 network between R2 and R1.

Bob decided to try another approach, and instead of R1 trying to make AS23 think the path on 10.0.0.0 was worse, perhaps he would tell R3 to make the path on 13.0.0.0 look better.    He considered weight, but then realized that would only work for R3, and not every other device in AS23.    So Bob decided to use local-preference.  On R3, he tried using local-preference, to specify that when a BGP update came in from R1 for 1.1.1.0, R3 would set the local-preference to 250 for that prefix, in hopes that this would allow traffic destined for 1.1.1.0 go exclusively through the 13.0.0.0 segment between R3 and R1.   Unfortunately, even with this change, Bob noticed that traffic destined to 1.1.1.0 from our through AS23 still crossed on the link between R2 and R1.

Below are the configurations for R1, R2 and R3 along with the relevant show commands.

Can you assist Bob?   What can he do?  What did he do wrong, if anything?

Post your ideas and comments below!

R1:

version 12.4
hostname R1
interface Loopback0
 ip address 1.1.1.1 255.255.255.0
 ip ospf network point-to-point

interface FastEthernet0/0
 ip address 10.0.0.1 255.255.255.0
 ip ospf 1 area 1

interface FastEthernet1/0
 ip address 13.0.0.1 255.255.255.0
 ip ospf 1 area 1

router bgp 1
 no synchronization
 bgp log-neighbor-changes
 network 1.1.1.0 mask 255.255.255.0
 neighbor 10.0.0.2 remote-as 23
 neighbor 10.0.0.2 route-map PRE-PEND out
 neighbor 13.0.0.3 remote-as 23
 no auto-summary

ip prefix-list JUST-1.1.1.0 seq 5 permit 1.1.1.0/24

route-map PRE-PEND permit 10
 match ip address prefix-list JUST-1.1.1.0
 set as-path prepend 1

route-map PRE-PEND permit 20

R2:

version 12.4
hostname R2
interface FastEthernet0/0
 ip address 10.0.0.2 255.255.255.0
 ip ospf 1 area 1

interface FastEthernet0/1
 ip address 23.0.0.2 255.255.255.0
 ip ospf 1 area 1

router bgp 23
 no synchronization
 bgp log-neighbor-changes
 neighbor 10.0.0.1 remote-as 1
 neighbor 23.0.0.3 remote-as 23
 no auto-summary
!

R3:

version 12.4
hostname R3
interface FastEthernet0/0
 ip address 13.0.0.3 255.255.255.0
 ip ospf 1 area 1

interface FastEthernet0/1
 ip address 23.0.0.3 255.255.255.0
 ip ospf 1 area 1

router bgp 23
 no synchronization
 bgp log-neighbor-changes
 neighbor 13.0.0.1 remote-as 1
 neighbor 13.0.0.1 route-map SET-LOCAL-PREF in
 neighbor 23.0.0.2 remote-as 23
 no auto-summary

ip prefix-list LOCAL-PREF-250 seq 5 permit 1.1.1.0/24

route-map SET-LOCAL-PREF permit 10
 match ip address prefix-list LOCAL-PREF-250
 set local-preference 250

route-map SET-LOCAL-PREF permit 20

Show commands R1:

R1#show ip bgp summary
BGP router identifier 1.1.1.1, local AS number 1
BGP table version is 2, main routing table version 2
1 network entries using 120 bytes of memory
1 path entries using 52 bytes of memory
2/1 BGP path/bestpath attribute entries using 248 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
Bitfield cache entries: current 1 (at peak 2) using 32 bytes of memory
BGP using 452 total bytes of memory
BGP activity 2/1 prefixes, 2/1 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.0.2        4    23      77      73        2    0    0 00:29:01        0
13.0.0.3        4    23      74      74        2    0    0 00:29:01        0

R1#show ip bgp
BGP table version is 2, local router ID is 1.1.1.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network          Next Hop            Metric LocPrf Weight Path
*> 1.1.1.0/24       0.0.0.0                  0         32768 i

R1#show ip route | begin resort
Gateway of last resort is not set

1.0.0.0/24 is subnetted, 1 subnets
C       1.1.1.0 is directly connected, Loopback0
23.0.0.0/24 is subnetted, 1 subnets
O       23.0.0.0 [110/2] via 13.0.0.3, 00:48:43, FastEthernet1/0
                 [110/2] via 10.0.0.2, 00:48:09, FastEthernet0/0
10.0.0.0/24 is subnetted, 1 subnets
C       10.0.0.0 is directly connected, FastEthernet0/0
13.0.0.0/24 is subnetted, 1 subnets
C       13.0.0.0 is directly connected, FastEthernet1/0

Show commands R2:

R2#show ip bgp summary
BGP router identifier 2.2.2.2, local AS number 23
BGP table version is 14, main routing table version 14
1 network entries using 120 bytes of memory
2 path entries using 104 bytes of memory
3/1 BGP path/bestpath attribute entries using 372 bytes of memory
2 BGP AS-PATH entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
Bitfield cache entries: current 1 (at peak 2) using 32 bytes of memory
BGP using 676 total bytes of memory
BGP activity 1/0 prefixes, 4/2 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
10.0.0.1        4     1      73      77       14    0    0 00:29:07        1
23.0.0.3        4    23      71      73       14    0    0 01:04:54        1

R2#show ip bgp
BGP table version is 14, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network          Next Hop            Metric LocPrf Weight Path
*>i1.1.1.0/24       13.0.0.1                 0    250      0 1 i
*                   10.0.0.1                 0             0 1 1 i

R2#show ip route | begin resort
Gateway of last resort is not set

1.0.0.0/24 is subnetted, 1 subnets
B       1.1.1.0 [200/0] via 13.0.0.1, 00:28:37
2.0.0.0/24 is subnetted, 1 subnets
C       2.2.2.0 is directly connected, Loopback0
23.0.0.0/24 is subnetted, 1 subnets
C       23.0.0.0 is directly connected, FastEthernet0/1
10.0.0.0/24 is subnetted, 1 subnets
C       10.0.0.0 is directly connected, FastEthernet0/0
13.0.0.0/24 is subnetted, 1 subnets
O       13.0.0.0 [110/2] via 23.0.0.3, 00:48:16, FastEthernet0/1
                 [110/2] via 10.0.0.1, 00:49:19, FastEthernet0/0

Show commands R3:

R3#show ip bgp summary
BGP router identifier 3.3.3.3, local AS number 23
BGP table version is 6, main routing table version 6
1 network entries using 120 bytes of memory
1 path entries using 52 bytes of memory
3/1 BGP path/bestpath attribute entries using 372 bytes of memory
1 BGP AS-PATH entries using 24 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
Bitfield cache entries: current 1 (at peak 2) using 32 bytes of memory
BGP using 600 total bytes of memory
BGP activity 1/0 prefixes, 5/4 paths, scan interval 60 secs

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd
13.0.0.1        4     1      74      74        6    0    0 00:29:09        1
23.0.0.2        4    23      73      71        6    0    0 01:04:56        0

R3#show ip bgp
BGP table version is 6, local router ID is 3.3.3.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network          Next Hop            Metric LocPrf Weight Path
*> 1.1.1.0/24       13.0.0.1                 0    250      0 1 i

R3#show ip route | begin resort
Gateway of last resort is not set

1.0.0.0/24 is subnetted, 1 subnets
B       1.1.1.0 [20/0] via 13.0.0.1, 00:28:39
3.0.0.0/24 is subnetted, 1 subnets
C       3.3.3.0 is directly connected, Loopback0
23.0.0.0/24 is subnetted, 1 subnets
C       23.0.0.0 is directly connected, FastEthernet0/1
10.0.0.0/24 is subnetted, 1 subnets
O       10.0.0.0 [110/2] via 23.0.0.2, 00:48:18, FastEthernet0/1
                 [110/2] via 13.0.0.1, 00:48:48, FastEthernet0/0
13.0.0.0/24 is subnetted, 1 subnets
C       13.0.0.0 is directly connected, FastEthernet0/0

Best wishes,

Keith

And the answer is:

Thanks to you, and your 50+ posts, bob got his answer.   By reading your responses, Bob learned the following:

For R2, the BGP next hop for the best route, is still 13.0.0.1, even though it is learned from R3.     R3 doesn’t bother to change the next-hop attribute when learning routes via a eBGP neighbor (R1).    With R2 having 2 equal cost paths (OSPF) for the next hop of 13.0.0.1, R2 would load balance the traffic over the 10.0.0.0 and 23.0.0.0 networks for traffic going to 1.1.1.0/24

One solution would be to have R3 use next-hop-self for updates sent to R2.  Then R2 would see the next hop as 23.0.0.3, and all the traffic would be forwarded to R3 as desired.

The update on R3 would look like this:

router bgp 23
 neighbor 23.0.0.2 next-hop-self

This would cause R2, to have the BGP table of this:

R2#show ip bgp
BGP table version is 4, local router ID is 2.2.2.2
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete

Network          Next Hop            Metric LocPrf Weight Path
*>i1.1.1.0/24       23.0.0.3                 0    250      0 1 i
*                   10.0.0.1                 0             0 1 1 i

Another option would be increasing the OSPF cost on R2’s 10.0.0.0/24 interface, so that it wouldn’t be considered an equal cost to get to 13.0.0.1 (the previous next hop before the change we just made).

Thanks everyone for all your assistance!    You rock.

Summer is here and it’s time to get certified!  Join us during the Summer of Success by attending one of our bootcamps and save up-to $1000.  Get $500 off any one week on-site bootcamp or $1000 off any two week on-site bootcamp when you purchase during this limited offer.  This special promotion applies to CCIE Routing & Switching, CCIE Voice, CCIE Service Provider or CCIE Security.

To take advantage of this special promotion, use discount code 1WEEKBOOTCAMP or 2WEEKBOOTCAMP when you check out at http://www.ine.com.

It isn’t my fault, they configured it that way before I got here! That was the entry level technician’s story Monday morning, and he was sticking to it. 

Here is the rest of the story.   Over the weekend, some testing had been done regarding a proposed BGP configuration.   The objective was simple, R1 and R3 needed to ping each others loobacks at 1.1.1.1 and 3.3.3.3 respectively, with those 2 networks, being carried by BGP.  R2 is performing NAT.    The topology diagram looks like this:

The ping between loopbacks didn’t work, but R1 and R3 had these console messages:

R1#
%TCP-6-BADAUTH: No MD5 digest from 10.0.0.3(179) to 10.0.0.1(28556) (RST)
R1#
%TCP-6-BADAUTH: No MD5 digest from 10.0.0.3(179) to 10.0.0.1(28556) (RST)
R1#

R3#
%TCP-6-BADAUTH: No MD5 digest from 23.0.0.1(179) to 23.0.0.3(59922) (RST)
R3#
%TCP-6-BADAUTH: No MD5 digest from 23.0.0.1(179) to 23.0.0.3(59922) (RST)
R3#

The senior engineer looked at the configurations for R1, R2 and R3 and found 5 specific items, each of which was independently causing a failure.

Here is the challenge:  Can you find 1 or more of them?

Let us know what your troubleshooting skills can find, and post your comments here on the blog.

Here are the configurations for the 3 routers:

R1#show run
version 12.4
hostname R1
!
interface Loopback0
 ip address 1.1.1.1 255.255.255.0
!
interface FastEthernet0/0
 ip address 10.0.0.1 255.255.255.0
!
router ospf 1
 network 10.0.0.0 0.0.0.255 area 0
!
router bgp 1
 no synchronization
 bgp log-neighbor-changes
 network 1.1.1.1 mask 255.255.255.255
 neighbor 10.0.0.3 remote-as 3
 neighbor 10.0.0.3 password cisco
 no auto-summary
!
end
R1#

R2#show run
version 12.4
hostname R2
!
interface Loopback0
 ip address 2.2.2.2 255.255.255.0
!
interface FastEthernet0/0
 ip address 10.0.0.2 255.255.255.0
 ip nat inside
 ip virtual-reassembly
!
interface FastEthernet0/1
 ip address 23.0.0.2 255.255.255.0
 ip nat outside
 ip virtual-reassembly
!
router ospf 1
 network 2.2.2.2 0.0.0.0 area 0
 network 10.0.0.2 0.0.0.0 area 0
 network 23.0.0.2 0.0.0.0 area 0
!
ip nat inside source static 10.0.0.1 23.0.0.1
ip nat outside source static 23.0.0.3 10.0.0.3
!
end

R3#show run
version 12.4
hostname R3
!
interface Loopback0
 ip address 3.3.3.3 255.255.255.0
!
interface FastEthernet0/1
 ip address 23.0.0.3 255.255.255.0
!
router ospf 1
 log-adjacency-changes
 network 23.0.0.0 0.0.0.255 area 0
!
router bgp 3
 no synchronization
 bgp log-neighbor-changes
 network 3.3.3.3 mask 255.255.255.255
 neighbor 23.0.0.1 remote-as 1
 neighbor 23.0.0.1 password cisco123
 no auto-summary
!
end
R3#

Let us know what you find!

Best wishes,

Keith

UPDATE:   ANSWERS

Your contributions and input is great.  You ROCK!

I have summarized the 5 specific errors/issues with the configuration, and here they are:

• R2: NAT isn’t fully baked. Can fix with  “ip nat outside source static 23.0.0.3 10.0.0.3 add-route” (or we could manually add the route as well).
• R1 & R3: The BGP passwords don’t match, but it doesn’t matter. BGP authentication doesn’t work between NAT’d BGP neighbors, so it would have to be removed.
• R1 & R3: Incorrect network statements for loopback addresses on both BGP routers (incorrect mask)
• R1 & R3: Ebgp-multihop statements are needed on both neighbors (not directly connected EBGP)
• R2: R2 doesn’t know how to reach 1.1.1.1 or 3.3.3.3 (non-BGP routing issue)

Again, thanks for the time and effort invested in this solution, and in learning in general.   I appreciate you!

Best wishes,

Keith