Solution Lab 2 for MPLS Lab 010

Why is there no connectivity between the CE1-A router's loopback0 interface and IP address 10.160.0.1?

To identify the cause of the problem we will follow the methodical step-by-step process of troubleshooting.

There are two domains where issues could cause connection loss:
1. Customer networks.
2. ISP Infrastructure.

Topology:



Let's start with the customer site first, since the problem originally was spotted at the client's device CE1-A, we will go there and confirm that the issue is still present:
CE1-A#ping 10.160.0.1 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.160.0.1, timeout is 2 seconds:
Packet sent with a source address of 10.155.0.1
.....
Success rate is 0 percent (0/5)
CE1-A#

The output confirms that the problem still exists. 


While being at the CLI of CE1-A verify the routing table:
CE1-A#show ip route
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
C 10.150.0.0/30 is directly connected, GigabitEthernet0/1
L 10.150.0.1/32 is directly connected, GigabitEthernet0/1
C 10.155.0.0/24 is directly connected, Loopback0
L 10.155.0.1/32 is directly connected, Loopback0
i L2 10.165.0.0/24 [115/10] via 10.150.0.2, 00:13:45, GigabitEthernet0/1
CE1-A#
CE1-A#show ip route 10.160.0.1 255.255.255.255
% Subnet not in table
CE1-A#

Clearly that the destination prefix is missing from the RIB at CE1-A. We know that network 10.160.0.0/24 is off the router CE1-B, before involving the ISP in the troubleshooting effort, let's go over to the CE1-B CLI and check if the subnet is up and running:
CE1-B#show ip route 10.160.0.0 255.255.255.0
Routing entry for 10.160.0.0/24
Known via "connected", distance 0, metric 0 (connected, via interface)
Routing Descriptor Blocks:
* directly connected, via Loopback0
Route metric is 0, traffic share count is 1
CE1-B#

CE1-B#show ip route 10.160.0.1 255.255.255.255
Routing entry for 10.160.0.1/32
Known via "connected", distance 0, metric 0 (connected)
Routing Descriptor Blocks:
* directly connected, via Loopback0
Route metric is 0, traffic share count is 1
CE1-B#


CE1-B#show ip route connected
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 6 subnets, 3 masks
C 10.150.0.4/30 is directly connected, GigabitEthernet0/1
L 10.150.0.5/32 is directly connected, GigabitEthernet0/1
C 10.160.0.0/24 is directly connected, Loopback0
L 10.160.0.1/32 is directly connected, Loopback0
CE1-B#


Investigating the routing entries shows us that at the origin location subnet is working properly, but one more thing is necessary to check if you want to completely make sure that the problem is not due to the customer's infrastructure misconfiguration. At the client's location for CE-PE routing, IS-IS protocol is used, now let's verify if the IS-IS neighborship is up and CE1-B router is advertising the 10.160.0.0/24 network to the ISP's PE4 router: 

CE1-B#show isis neighbors
System Id Type Interface IP Address State Holdtime Circuit Id
PE4 L2 Gi0/1 10.150.0.6 UP 24 00
CE1-B#


CE1-B#show isis database
IS-IS Level-2 Link State Database:
LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL
PE4.00-00 0x00000004 0x65C4 513 0/0/0
CE1-B.00-00 * 0x00000004 0xFE85 1176 0/0/0

CE1-B#show isis database detail CE1-B.00-00
IS-IS Level-2 LSP CE1-B.00-00
LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL
CE1-B.00-00 * 0x00000004 0xFE85 1147 0/0/0
Area Address: 49.dddd
NLPID: 0xCC
Hostname: CE1-B
Metric: 10 IS-Extended PE4.00
IP Address: 10.160.0.1
Metric: 10 IP 10.150.0.4/30
Metric: 10 IP 10.160.0.0/24


The output confirms that the network 10.160.0.0/24 is being advertised to the ISP's router PE4. Additionally, it is good to check if router CE1-B has CE1-A router's subnet in its database:
CE1-B#show isis database detail PE4.00-00
 IS-IS Level-2 LSP PE4.00-00
LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL
PE4.00-00 0x00000005 0x63C5 1072 0/0/0
Area Address: 49.dddd
NLPID: 0xCC
Hostname: PE4
Metric: 10 IS-Extended CE1-B.00
IP Address: 10.150.0.6
Metric: 10 IP 10.150.0.4/30
Metric: 0 IP 10.155.0.0/24
Metric: 0 IP 10.165.0.0/24
CE1-B#


Now that 10.155.0.0/24 has been learned from ISP's device PE4, we can conclude that the problem is with network 10.160.0.0/24 is not being advertised to the router CE1-A and we can get ISP involved into the troubleshooting process. Okay, let's switch roles from being the network engineer at the customer site to the ISP. Troubleshooting ticket has been open with ISP, all findings were passed to the network support team at ISP.
The network engineer at the ISP decides to start the investigation of the problem at the router PE2 since this is the place troublesome device CE1-A is connected to.

PE2#show isis neighbors
Tag null:
System Id Type Interface IP Address State Holdtime Circuit Id
CE1-A L2 Gi0/3 10.150.0.1 UP 22 01

IS-IS neighborship with CE1-A is UP. Next, let's see the routing table for VRF CE1:

PE2#show ip route vrf CE1
Routing Table: CE1
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
C 10.150.0.0/30 is directly connected, GigabitEthernet0/3
L 10.150.0.2/32 is directly connected, GigabitEthernet0/3
i L2 10.155.0.0/24 [115/20] via 10.150.0.1, 00:47:31, GigabitEthernet0/3
B 10.165.0.0/24 [200/20] via 10.100.0.13, 00:46:08


Indeed, the 10.160.0.0/24 is missing. Check on the BGP VPNv4 table:
PE2#show bgp vpnv4 unicast vrf CE1
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1900:1900 (default for vrf CE1)
*> 10.155.0.0/24 10.150.0.1 20 32768 ?
*>i 10.165.0.0/24 10.100.0.13 20 100 0 ?


The prefix is not in the BGP VPNv4 table, but we can derive that the problem might not be related to the PE2 router because other BGP prefixes are being learned. That is why it been decided to go the router PE4 and start troubleshooting there. We will start with the BGP VPNv4 table:

PE4#show bgp vpnv4 unicast vrf CE1
BGP table version is 8, local router ID is 10.100.0.14
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 1900:1900 (default for vrf CE1)
*>i 10.155.0.0/24 10.100.0.12 20 100 0 ?
*>i 10.165.0.0/24 10.100.0.13 20 100 0 ?


Looking at the output, right away you can see a lot of information pointing out that problem is on this device, first, there are two entries in the BGP table, both are denoted with "i" meaning these prefixes learned via internal bgp and they coming from other nodes if the local device would inject prefix into the BGP table the "i" will be missing, and that is how we know that PE4 does not redistribute network of CE1-B 10.160.0.0/24 into the MPLS core, also the next-hop addresses are IP addresses of the loopback0 interfaces of PE2 and PE3 routers. We will not look at the IS-IS router configuration because from the information in previous steps while interrogating the CE1-B router, we saw that there some other routes learned from BGP, that is why we can conclude that redistribution is working properly from BGP into IS-IS. But we do need to check what is going on with BGP configuration of redistribution:

PE4#show running-config | section bgp
redistribute bgp 64500
router bgp 64500
bgp router-id 10.100.0.14
bgp log-neighbor-changes
neighbor 10.100.0.11 remote-as 64500
neighbor 10.100.0.11 update-source Loopback0
neighbor 10.100.0.12 remote-as 64500
neighbor 10.100.0.12 update-source Loopback0
neighbor 10.100.0.13 remote-as 64500
neighbor 10.100.0.13 update-source Loopback0
!
address-family vpnv4
neighbor 10.100.0.11 activate
neighbor 10.100.0.11 send-community extended
neighbor 10.100.0.12 activate
neighbor 10.100.0.12 send-community extended
neighbor 10.100.0.13 activate
neighbor 10.100.0.13 send-community extended
exit-address-family
!
address-family ipv4 vrf CE1
redistribute isis level-1
exit-address-family

Under the address-family portion of the output, you can see that BGP performs redistribution of the IS-IS into the MPLS backbone network but there is the problem with the line! If you spot it then you are good to go, go ahead and fix it, otherwise, let me show you what is the problem.
IS-IS protocol for CE-PE routing on all the PE routers have been configured to run as a level-2 type and if you look at the IS-IS database you will not find any type-1 prefixes only type-2 but BGP is configured to redistribute the Type-1 IS-IS prefixes, that is why router PE4 does not redistribute subnet 10.160.0.0/24. 



Correct the problem:
PE4(config)#router bgp 64500
PE4(config-router)#address-family ipv4 unicast vrf CE1
PE4(config-router-af)#no redistribute isis
PE4(config-router-af)#redistribute isis level-2
PE4(config-router-af)#
Now back on the router CE1-A verify if prefix 10.160.0.0/24 is installed into the routing table, then if it is positive, ping the 10.160.0.1 using loopback0 interface as a source.



Summary:
Understanding the way these complex networks work is very beneficial to your troubleshooting skills, knowing where to start identifying the reasons for the cause of the problems in the topology sometimes is difficult because you do not have all necessary pieces of information to conduct the investigation and it also depends on many other factors which are obstacles in the way to solve the complications preventing the production to operate correctly, that is why you need to learn not only the technical aspects of the job as troubleshooter but as well human factors that will drastically improve your ability to resolve the urgent matter, know your peers, know what other people are able to do that will help you to fix the issues, collect the information about other employees' abilities, technical skills and duties, this is managerial expertise that will allow you to become an excellent network engineer with the great troubleshooting dexterity.  

Comments

Popular Posts