Solution Lab 4 for MPLS Lab 024
Description:
This is a solution lab, in the step-by-step instruction, an explanation will be given on how to with systematic approach troubleshoot the connectivity problem. Additionally troubleshooting dependency model will be used to facilitate the problem-solving effort.
Topology:
Troubleshooting dependency model for topology used in Lab 024:
Lab procedure:
Step1. Confirm the problem:
Access the CLI of CE1-A and CE1-B routers and ping the 8.8.8.8 public address sourcing from the loopback1 interface.
CE1-A#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1
.....
Success rate is 0 percent (0/5)
CE1-A#
CE1-B#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.20.1
.....
Success rate is 0 percent (0/5)
CE1-B#
Both outputs verified that users in both networks 192.168.10.0/24 and 192.168.20.0/24 unable to access the Internet.
Step2. Using the model to troubleshoot the problem:
Back to the model, looking at the top of the model we have two top-level end-to-end logical data paths that have been affected by some events in topology. Now it is your job to find the culprit by using the model. Where to begin? Since both paths are affected you can start with PATH1 for now, looking at the PATH1 section in the table you can see that the higher level is 5 and this is IP connectivity with LAN network 192.168.10.0/24 let's issue commands that confirms that LAN network is operational, the show ip route command and ping 192.168.10.1 have to be executed at the point where LAN network is attached, which is the CE1-A router:
CE1-A#show ip route connected
Gateway of last resort is 10.0.0.26 to network 0.0.0.0
10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C 10.0.0.24/30 is directly connected, GigabitEthernet0/1
L 10.0.0.25/32 is directly connected, GigabitEthernet0/1
192.168.0.0/32 is subnetted, 2 subnets
C 192.168.0.2 is directly connected, Loopback0
192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.10.0/24 is directly connected, Loopback1
L 192.168.10.1/32 is directly connected, Loopback1
CE1-A#ping 192.168.10.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.10.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
CE1-A#
The proper operation of Level5 has been confirmed, moving down the PATH1 section to the Level4, which is eBGP with PE3, verify the BGP neighborship and received prefixes with PE3 router:
CE1-A#show ip bgp summary
BGP router identifier 192.168.10.1, local AS number 65001
BGP table version is 8, main routing table version 8
7 network entries using 1008 bytes of memory
7 path entries using 588 bytes of memory
5/5 BGP path/bestpath attribute entries using 800 bytes of memory
2 BGP AS-PATH entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2444 total bytes of memory
BGP activity 7/0 prefixes, 7/0 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.0.26 4 65000 247 244 8 0 0 03:38:17 5
CE1-A#show ip bgp
BGP table version is 8, local router ID is 192.168.10.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0 10.0.0.26 0 65000 i
*> 10.0.0.8/30 10.0.0.26 0 65000 ?
r> 10.0.0.24/30 10.0.0.26 0 0 65000 ?
*> 192.168.0.1/32 10.0.0.26 0 65000 65002 i
*> 192.168.0.2/32 0.0.0.0 0 32768 i
*> 192.168.10.0 0.0.0.0 0 32768 i
*> 192.168.20.0 10.0.0.26 0 65000 65002 i
CE1-A#
The proper operation of Level4 has been confirmed. Next, Level3 which is Internet Access, and from Step1 you already know that here where problem potentially present, since you have reached the problematic level it is safe to assume that all levels in the section PATH1 below could be broken too.
Now it's time to move to the "Internet Access" section of the model, you can conclude from the descriptions of the levels in this section that all configurations pertained to the PE routers and since you are investigating the PATH1 communication, you can start with PE3 router:
Is there a problem at level5, "Propagation of 0.0.0.0/0 to CE nodes":
PE3#show bgp vrf CE1 all summary
For address family: IPv4 Unicast
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.0.25 4 65001 259 262 12 0 0 03:52:10 2
This command identifies the neighbor ip address which you need to determine which prefixes are being sent.
PE3#show bgp vrf CE1 all neighbors 10.0.0.25 advertised-routes
For address family: IPv4 Unicast
BGP table version is 12, local router ID is 192.168.0.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10:10 (default for vrf CE1)
*>i 10.0.0.8/30 192.168.0.6 0 100 0 ?
*> 10.0.0.24/30 0.0.0.0 0 32768 ?
*>i 192.168.0.1/32 192.168.0.6 0 100 0 65002 i
*>i 192.168.20.0 192.168.0.6 0 100 0 65002 i
Total number of prefixes 4
The line "Originating default network 0.0.0.0" show that propagation is occurs.
Is there a problem at level4: "Default Static Route inside VRF CE1"
PE3#show ip route vrf CE1 static
Routing Table: CE1
Gateway of last resort is 192.168.0.3 to network 0.0.0.0
S* 0.0.0.0/0 [1/0] via 192.168.0.3
PE3#show running-config | section ip route
ip route 192.168.10.0 255.255.255.0 GigabitEthernet0/2 10.0.0.25
ip route vrf CE1 0.0.0.0 0.0.0.0 192.168.0.3 global
Everything seems alright with level4, default static route comrise proparly with global argument.
Is there a problem at level3: "Static Routes for CE's Networks"
PE3#show ip route static
Gateway of last resort is 192.168.0.3 to network 0.0.0.0
S 192.168.10.0/24 [1/0] via 10.0.0.25, GigabitEthernet0/2
The output of the global routing table for static route shows that the static route has no configurational mistakes in it, the network is right, next-hop address and exit interface all configured properly.
Is there a problem at level2: Redistribution by BGP
PE3#show ip protocols | section bgp
Routing Protocol is "bgp 65000"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
IGP synchronization is disabled
Automatic route summarization is disabled
Redistributing: static
Neighbor(s):
Address FiltIn FiltOut DistIn DistOut Weight RouteMap
192.168.0.3
192.168.0.6
Maximum path: 1
Routing Information Sources:
Gateway Distance Last Update
192.168.0.3 200 04:09:26
192.168.0.6 200 04:09:26
Distance: external 20 internal 200 local 200
Redistribution of static routes enabled for the BGP routing protocol. Let's see if the PE1 router is receiving CEs' networks.
PE1#show bgp ipv4 unicast
BGP table version is 6, local router ID is 192.168.0.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0 50.0.0.1 0 500 i
*> 50.0.0.0/16 50.0.0.1 0 0 500 i
*> 75.100.0.0/20 0.0.0.0 0 32768 i
*>i 192.168.10.0 192.168.0.7 0 100 0 ?
*>i 192.168.20.0 192.168.0.6 0 100 0 ?
PE1#show ip route bgp
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
Gateway of last resort is 50.0.0.1 to network 0.0.0.0
B* 0.0.0.0/0 [20/0] via 50.0.0.1, 04:11:46
50.0.0.0/8 is variably subnetted, 3 subnets, 3 masks
B 50.0.0.0/16 [20/0] via 50.0.0.1, 04:11:46
B 192.168.10.0/24 [200/0] via 192.168.0.7, 04:11:46
B 192.168.20.0/24 [200/0] via 192.168.0.6, 04:11:46
Now that you have determined that pretty much all levels in "Internet Access" section working properly, there is only one level left.
Is there a problem at level1: NAT configured at the remote PE.
Simply ping 8.8.8.8 address sourcing it from lo0 interface on the NAT performing router PE1 will show you the operation of the NAT:
PE1#ping 8.8.8.8 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.3
.....
Success rate is 0 percent (0/5)
PE1#ping 8.8.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/3 ms
PE1#
From the outputs, you can derive the conclusion that internet destination is reachable from PE1's outgoing interface facing the ISP, meaning that at least you do not have to troubleshoot the connectivity to the ISP but on the other hand it is NAT overload still is not working.
PE1#show ip nat statistics
Total active translations: 0 (0 static, 0 dynamic; 0 extended)
Peak translations: 0
Outside interfaces:
GigabitEthernet0/2
Inside interfaces:
GigabitEthernet0/1, Loopback0
Hits: 0 Misses: 0
CEF Translated packets: 0, CEF Punted packets: 0
Expired translations: 0
Dynamic mappings:
-- Inside Source
[Id: 1] access-list NAT pool PUBLIC_POOL refcount 0
Total doors: 0
Appl doors: 0
Normal doors: 0
Queued Packets: 0
PE1#
The output now indicates that NAT interfaces configured properly, it uses the access-list named NAT and nat pool named PUBLIC_POOL.
The next step will be verification of the ACL to see if it contains proper networks to be translated:
PE1#show ip access-lists NAT
Standard IP access list NAT
10 permit 10.0.0.0, wildcard bits 0.0.0.255
20 permit 192.168.0.0, wildcard bits 0.0.0.255 (5 matches)
30 permit 192.168.10.0, wildcard bits 0.0.0.255
40 permit 192.168.20.0, wildcard bits 0.0.0.255
It seems that all networks identified in the ACL are what is being expected.
There are a couple more components to check for the proper NAT operation those are NAT statement and NAT pool, let's see the running-config:
PE1#show running-config | section ip nat
ip nat inside
ip nat inside
ip nat outside
ip nat pool PUBLIC_RANGE 75.100.15.0 75.100.15.254 netmask 255.255.255.0
ip nat inside source list NAT pool PUBLIC_POOL overload
PE1#
Here it comes the wrong NAT statement using the pool name PUBLIC_POOL instead of PUBLIC_RANGE. Let's fix the problem:
PE1(config)#no ip nat inside source list NAT pool PUBLIC_POOL overload
PE1(config)#
PE1(config)#ip nat inside source list NAT pool PUBLIC_RANGE overload
PE1(config)#
Lastly, let's ping again the 8.8.8.8 public ip address from the loopback0 interface and then verify NAT translation:
PE1#ping 8.8.8.8 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.3
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/5 ms
PE1#show ip nat translations
Pro Inside global Inside local Outside local Outside global
icmp 75.100.15.1:2 192.168.0.3:2 8.8.8.8:2 8.8.8.8:2
PE1#
Since you restore the NAT operation on the PE1 router, let's assume that the problem is fixed at the CE sites too and you can go to verify the NAT operation back on CEs:
CE1-A#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1
.....
Success rate is 0 percent (0/5)
CE1-A#
CE1-B#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.20.1
.....
Success rate is 0 percent (0/5)
CE1-B#
Unfortunately, the problem is still present, it is time to continue the hypothesis using the troubleshooting dependency model since you have reached the last level1 within "Internet Access" section of the model, you might need to start investigating the "MPLS ISP" section.
MPLS ISP section of the model:
But before you go through the levels you need to look at the model again and think if you can skip some of the levels because of the troubleshooting effort you have already been through, which can expedite the process, the "MPLS ISP" section of the model contains top-level and 8 underlying levels, starting with level8 you can assume that IP connectivity to the CE routers is working because you already have verified previously, regarding the level7, eBGP is working too, you know that because you checked it previously as well, level6, VRF also can be considered working properly because PE3 router has installed static route to CE1's LAN network in its global routing table and it was redistributed by BGP to PE1 router and the LAN network is available from PE3 itself.
Now, level5 comes and it is MP-BGP, two things to verify at this level, first is the BGPv4 neighborship of PE routers but you do not really need to do so because the PE1 router already has CEs' LAN networks in its BGPv4 table meaning that BGPv4 neighborships are working. The second thing is the BGP neighborship as well but only for VPNV4:
PE1#show bgp vpnv4 unicast all summary
BGP router identifier 192.168.0.3, local AS number 65000
BGP table version is 1, main routing table version 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
192.168.0.6 4 65000 355 355 1 0 0 05:18:43 0
192.168.0.7 4 65000 358 361 1 0 0 05:18:43 0
PE1#
It seems that BGP VPNV4 neighborships are working too, so what is the problem then?
Is there a problem at level4: "MPLS, LDP"?
Overall, you can conclude that there is working connectivity between PE3 and CE1-A' LAN network, and probably between PE2 and CE1-B' LAN network as well. Also, router PE1 able to communicate with outside networks via ISP directly and with the use of NAT overload. But proper communication over MPLS cloud between PE routers is unconfirmed yet, from PE1 let's ping loopback0 interfaces on PE2 and PE3 sourcing from loopback0 interface:
PE1#ping 192.168.0.6 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.6, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.3
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/5/7 ms
PE1#ping 192.168.0.7 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.7, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.3
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/5/7 ms
PE1#
Though, the IPv4 connectivity exists between PE routers but does uninterrupted LSP from PE1 to PE2 and from PE1 to PE2 exist too? Let's verify this too:
PE3#ping mpls ipv4 192.168.0.3/32 source 192.168.0.7
Sending 5, 100-byte MPLS Echos to 192.168.0.3/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
BBBBB
Success rate is 0 percent (0/5)
PE3#
PE2#ping mpls ipv4 192.168.0.3/32 source 192.168.0.6
Sending 5, 100-byte MPLS Echos to 192.168.0.3/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
BBBBB
Success rate is 0 percent (0/5)
PE2#
The output of the ping mpls command shows that both LSP paths to the PE1 from PE2 and PE3 somewhere in the MPLS cloud get unlabeled, hence the problem exists, let's use traceroute mpls command to identify the source where packets on the way to PE1 get unlabeled:
PE3#traceroute mpls ipv4 192.168.0.3/32 source 192.168.0.7
Tracing MPLS Label Switched Path to 192.168.0.3/32, timeout is 2 seconds
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
0 10.0.0.30 MRU 1500 [Labels: 101 Exp: 0]
B 1 10.0.0.29 MRU 1504 [No Label] 7 ms
. 2 *
B 3 10.0.0.29 MRU 1504 [No Label] 4 ms
B 4 10.0.0.29 MRU 1504 [No Label] 4 ms
B 5 10.0.0.29 MRU 1504 [No Label] 4 ms
B 6 10.0.0.29 MRU 1504 [No Label] 5 ms
B 7 10.0.0.29 MRU 1504 [No Label] 4 ms
B 8 10.0.0.29 MRU 1504 [No Label] 4 ms
B 9 10.0.0.29 MRU 1504 [No Label] 4 ms
B 10 10.0.0.29 MRU 1504 [No Label] 4 ms
B 11 10.0.0.29 MRU 1504 [No Label] 3 ms
B 12 10.0.0.29 MRU 1504 [No Label] 4 ms
B 13 10.0.0.29 MRU 1504 [No Label] 4 ms
B 14 10.0.0.29 MRU 1504 [No Label] 4 ms
B 15 10.0.0.29 MRU 1504 [No Label] 3 ms
B 16 10.0.0.29 MRU 1504 [No Label] 4 ms
B 17 10.0.0.29 MRU 1504 [No Label] 3 ms
B 18 10.0.0.29 MRU 1504 [No Label] 4 ms
B 19 10.0.0.29 MRU 1504 [No Label] 6 ms
B 20 10.0.0.29 MRU 1504 [No Label] 3 ms
B 21 10.0.0.29 MRU 1504 [No Label] 4 ms
B 22 10.0.0.29 MRU 1504 [No Label] 4 ms
B 23 10.0.0.29 MRU 1504 [No Label] 3 ms
B 24 10.0.0.29 MRU 1504 [No Label] 3 ms
B 25 10.0.0.29 MRU 1504 [No Label] 3 ms
B 26 10.0.0.29 MRU 1504 [No Label] 4 ms
B 27 10.0.0.29 MRU 1504 [No Label] 3 ms
B 28 10.0.0.29 MRU 1504 [No Label] 3 ms
B 29 10.0.0.29 MRU 1504 [No Label] 3 ms
B 30 10.0.0.29 MRU 1504 [No Label] 4 ms
PE3#
Very intersting output, it shows that PE3 uses label 101 of P1 to reach 192.168.0.3 but then P1 has no outgoing label assinged to the IP address of the PE1 loopback0 interface.
Next step is to check the mpls forwarding-table for the prefix 192.168.0.3/32 on the provider router P1:
P1#show mpls forwarding-table 192.168.0.3 255.255.255.255
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
101 No Label 192.168.0.3/32 108177 Gi0/1 10.0.0.18
The output shows that the outgoing label is missing, meaning that there could be a label filter in place or MPLS is not enabled on the adjacent router's interface, from the topology it is obvious that router P1 is connected to the router P2.
Next, at the router P2, verify MPLS forwarding table for the prefix 192.168.0.3/32:
P2#show mpls forwarding-table 192.168.0.3 255.255.255.255
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
None No Label 192.168.0.3/32 0 Gi0/2 10.0.0.14
Neither local or outgoing labels are assigned for the prefix.
Let's check the LDP neighborships on the P2:
P2#show mpls ldp neighbor
P2#
There are no neighbors, this is an indication that something is not configured properly.
Verify which interfaces are enabled for mpls:
P2#show mpls interfaces
Interface IP Tunnel BGP Static Operational
P2#
None of the interfaces participate in the LDP process.
Let's see if interfaces are configured for mpls:
P2#show running-config interface g0/1
Building configuration...
Current configuration : 161 bytes
!
interface GigabitEthernet0/1
description to P1
ip address 10.0.0.18 255.255.255.252
ip router isis
duplex full
speed auto
media-type rj45
mpls ip
end
P2#show running-config interface g0/2
Building configuration...
Current configuration : 162 bytes
!
interface GigabitEthernet0/2
description to PE1
ip address 10.0.0.13 255.255.255.252
ip router isis
duplex full
speed auto
media-type rj45
mpls ip
end
Now that you can see that mpls is enabled under the interfaces, the question rises is it enabled globally?
P2#show running-config | section mpls ip
no mpls ip
mpls ip
mpls ip
P2#
Here it is the problem, that caused the connectivity issue related to the mpls itself. The label switching path was interrupted at the P2 provider router. Now it time to fix the problem and verify the solution:
P2(config)#mpls ip
P2(config)#end
P2#
Verify LSP path from PE3:
PE3#ping mpls ipv4 192.168.0.3/32 source 192.168.0.7
Sending 5, 100-byte MPLS Echos to 192.168.0.3/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/6/8 ms
PE3#
You can also confirm the LSP from PE2 as well.
Verify the Internet access from CE1-A and CE1-B:
CE1-A#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/6/8 ms
CE1-A#
CE1-B#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.20.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/6/8 ms
CE1-B#
The lab is complete, both end-to-end logical data paths are restored, now users in both LAN networks are able to return to work.
Summary:
Using the troubleshooting dependency model can improve communication between network engineers with different level of experience when it comes to the problem-solving situation, the model can be used as the checklist to prevent engineers from going in circles while trying to understand what is not working on the network right now, in addition, the model is about data flow, you put virtual paths on the top and then underneath all technologies used in the topology which allow that path to operate. The model can be organized into the tiers in a hierarchical fashion with multiple levels as shown in this lab when you exhaust possible troubleshooting actions within one tier you can jump to one below and continue your troubleshooting effort.
This is a solution lab, in the step-by-step instruction, an explanation will be given on how to with systematic approach troubleshoot the connectivity problem. Additionally troubleshooting dependency model will be used to facilitate the problem-solving effort.
Topology:
Troubleshooting dependency model for topology used in Lab 024:
| Layers | Technologies or Entity | Description |
| Top Level | CE1-A 192.168.10.0/24 to 8.8.8.8 | End-to-End Data Path One |
| Top Level | CE1-B 192.168.20.0/24 to 8.8.8.8 | End-to-End Data Path Two |
| PATH 1: | ||
| Level5 | IPv4 Connectivity for 192.168.10.0/24 | The network is installed in the RIB |
| Level4 | eBGP with PE3 | Neighborship, Prefixes exchange |
| Level3 | Internet Access for CE subnets | Set of technologies used to ensure Internet |
| Level2 | MPLS ISP | Depend on entire MPLS infrastructure |
| Level1 | ISP for MPLS network | Connectivity to ISP from PE1 |
| PATH 2: | ||
| Level5 | IPv4 Connectivity for 192.168.20.0/24 | The network is installed in the RIB |
| Level4 | eBGP with PE2 | Neighborship, Prefixes exchange |
| Level3 | Internet Access for CE subnets | Set of technologies used to ensure Internet |
| Level2 | MPLS ISP | Depend on entire MPLS infrastructure |
| Level1 | ISP for MPLS network | Connectivity to ISP from PE1 |
| Internet Access: | ||
| Level5 | Propagation of 0.0.0.0/0 to CE nodes | PE routers advertise to the CEs via eBGP |
| Level4 | Default Static Route inside VRF CE1 | Configured on PEs attached to CEs |
| Level3 | Static Routes for CE's Networks | inside Global Routing Table within MPLS core |
| Level2 | Redistribution by BGP | Make sure that NAT PE has routes to CEs networks |
| Level1 | NAT configured at the remote PE | NAT overload performed at the PE1 router |
| MPLS ISP: | ||
| Top Level | PE to ISP Logical Data Path | From Interfaces facing CE node to ISP |
| Level8 | IPv4 connectivity to CE sites | Proper interface IP configuration in place |
| Level7 | eBGP | Connection to CE routers |
| Level6 | VRF | Virtual separation of customers' networks |
| Level5 | MP-BGP | BGP VPNV4 communications between PEs |
| Level4 | MPLS, LDP | Label Switching Path ensured between PEs |
| Level3 | IGP: Integrated IS-IS | Facilitates the MPLS forwarding table |
| Level2 | IPv4 connectivity between nodes (links) | Within MPLS infrastructure |
| Level1 | ISP (BGPv4, IPv4) | Provides connection to the outside networks |
Lab procedure:
Step1. Confirm the problem:
Access the CLI of CE1-A and CE1-B routers and ping the 8.8.8.8 public address sourcing from the loopback1 interface.
CE1-A#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1
.....
Success rate is 0 percent (0/5)
CE1-A#
CE1-B#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.20.1
.....
Success rate is 0 percent (0/5)
CE1-B#
Both outputs verified that users in both networks 192.168.10.0/24 and 192.168.20.0/24 unable to access the Internet.
Step2. Using the model to troubleshoot the problem:
Back to the model, looking at the top of the model we have two top-level end-to-end logical data paths that have been affected by some events in topology. Now it is your job to find the culprit by using the model. Where to begin? Since both paths are affected you can start with PATH1 for now, looking at the PATH1 section in the table you can see that the higher level is 5 and this is IP connectivity with LAN network 192.168.10.0/24 let's issue commands that confirms that LAN network is operational, the show ip route command and ping 192.168.10.1 have to be executed at the point where LAN network is attached, which is the CE1-A router:
CE1-A#show ip route connected
Gateway of last resort is 10.0.0.26 to network 0.0.0.0
10.0.0.0/8 is variably subnetted, 3 subnets, 2 masks
C 10.0.0.24/30 is directly connected, GigabitEthernet0/1
L 10.0.0.25/32 is directly connected, GigabitEthernet0/1
192.168.0.0/32 is subnetted, 2 subnets
C 192.168.0.2 is directly connected, Loopback0
192.168.10.0/24 is variably subnetted, 2 subnets, 2 masks
C 192.168.10.0/24 is directly connected, Loopback1
L 192.168.10.1/32 is directly connected, Loopback1
CE1-A#ping 192.168.10.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.10.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
CE1-A#
The proper operation of Level5 has been confirmed, moving down the PATH1 section to the Level4, which is eBGP with PE3, verify the BGP neighborship and received prefixes with PE3 router:
CE1-A#show ip bgp summary
BGP router identifier 192.168.10.1, local AS number 65001
BGP table version is 8, main routing table version 8
7 network entries using 1008 bytes of memory
7 path entries using 588 bytes of memory
5/5 BGP path/bestpath attribute entries using 800 bytes of memory
2 BGP AS-PATH entries using 48 bytes of memory
0 BGP route-map cache entries using 0 bytes of memory
0 BGP filter-list cache entries using 0 bytes of memory
BGP using 2444 total bytes of memory
BGP activity 7/0 prefixes, 7/0 paths, scan interval 60 secs
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.0.26 4 65000 247 244 8 0 0 03:38:17 5
CE1-A#show ip bgp
BGP table version is 8, local router ID is 192.168.10.1
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0 10.0.0.26 0 65000 i
*> 10.0.0.8/30 10.0.0.26 0 65000 ?
r> 10.0.0.24/30 10.0.0.26 0 0 65000 ?
*> 192.168.0.1/32 10.0.0.26 0 65000 65002 i
*> 192.168.0.2/32 0.0.0.0 0 32768 i
*> 192.168.10.0 0.0.0.0 0 32768 i
*> 192.168.20.0 10.0.0.26 0 65000 65002 i
CE1-A#
The proper operation of Level4 has been confirmed. Next, Level3 which is Internet Access, and from Step1 you already know that here where problem potentially present, since you have reached the problematic level it is safe to assume that all levels in the section PATH1 below could be broken too.
Now it's time to move to the "Internet Access" section of the model, you can conclude from the descriptions of the levels in this section that all configurations pertained to the PE routers and since you are investigating the PATH1 communication, you can start with PE3 router:
Is there a problem at level5, "Propagation of 0.0.0.0/0 to CE nodes":
PE3#show bgp vrf CE1 all summary
For address family: IPv4 Unicast
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.0.0.25 4 65001 259 262 12 0 0 03:52:10 2
This command identifies the neighbor ip address which you need to determine which prefixes are being sent.
PE3#show bgp vrf CE1 all neighbors 10.0.0.25 advertised-routes
For address family: IPv4 Unicast
BGP table version is 12, local router ID is 192.168.0.7
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
Route Distinguisher: 10:10 (default for vrf CE1)
*>i 10.0.0.8/30 192.168.0.6 0 100 0 ?
*> 10.0.0.24/30 0.0.0.0 0 32768 ?
*>i 192.168.0.1/32 192.168.0.6 0 100 0 65002 i
*>i 192.168.20.0 192.168.0.6 0 100 0 65002 i
Total number of prefixes 4
The line "Originating default network 0.0.0.0" show that propagation is occurs.
Is there a problem at level4: "Default Static Route inside VRF CE1"
PE3#show ip route vrf CE1 static
Routing Table: CE1
Gateway of last resort is 192.168.0.3 to network 0.0.0.0
S* 0.0.0.0/0 [1/0] via 192.168.0.3
PE3#show running-config | section ip route
ip route 192.168.10.0 255.255.255.0 GigabitEthernet0/2 10.0.0.25
ip route vrf CE1 0.0.0.0 0.0.0.0 192.168.0.3 global
Everything seems alright with level4, default static route comrise proparly with global argument.
Is there a problem at level3: "Static Routes for CE's Networks"
PE3#show ip route static
Gateway of last resort is 192.168.0.3 to network 0.0.0.0
S 192.168.10.0/24 [1/0] via 10.0.0.25, GigabitEthernet0/2
The output of the global routing table for static route shows that the static route has no configurational mistakes in it, the network is right, next-hop address and exit interface all configured properly.
Is there a problem at level2: Redistribution by BGP
PE3#show ip protocols | section bgp
Routing Protocol is "bgp 65000"
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
IGP synchronization is disabled
Automatic route summarization is disabled
Redistributing: static
Neighbor(s):
Address FiltIn FiltOut DistIn DistOut Weight RouteMap
192.168.0.3
192.168.0.6
Maximum path: 1
Routing Information Sources:
Gateway Distance Last Update
192.168.0.3 200 04:09:26
192.168.0.6 200 04:09:26
Distance: external 20 internal 200 local 200
Redistribution of static routes enabled for the BGP routing protocol. Let's see if the PE1 router is receiving CEs' networks.
PE1#show bgp ipv4 unicast
BGP table version is 6, local router ID is 192.168.0.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale, m multipath, b backup-path, f RT-Filter,
x best-external, a additional-path, c RIB-compressed,
t secondary path,
Origin codes: i - IGP, e - EGP, ? - incomplete
RPKI validation codes: V valid, I invalid, N Not found
Network Next Hop Metric LocPrf Weight Path
*> 0.0.0.0 50.0.0.1 0 500 i
*> 50.0.0.0/16 50.0.0.1 0 0 500 i
*> 75.100.0.0/20 0.0.0.0 0 32768 i
*>i 192.168.10.0 192.168.0.7 0 100 0 ?
*>i 192.168.20.0 192.168.0.6 0 100 0 ?
PE1#show ip route bgp
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, H - NHRP, l - LISP
a - application route
+ - replicated route, % - next hop override, p - overrides from PfR
Gateway of last resort is 50.0.0.1 to network 0.0.0.0
B* 0.0.0.0/0 [20/0] via 50.0.0.1, 04:11:46
50.0.0.0/8 is variably subnetted, 3 subnets, 3 masks
B 50.0.0.0/16 [20/0] via 50.0.0.1, 04:11:46
B 192.168.10.0/24 [200/0] via 192.168.0.7, 04:11:46
B 192.168.20.0/24 [200/0] via 192.168.0.6, 04:11:46
Now that you have determined that pretty much all levels in "Internet Access" section working properly, there is only one level left.
Is there a problem at level1: NAT configured at the remote PE.
Simply ping 8.8.8.8 address sourcing it from lo0 interface on the NAT performing router PE1 will show you the operation of the NAT:
PE1#ping 8.8.8.8 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.3
.....
Success rate is 0 percent (0/5)
PE1#ping 8.8.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/3 ms
PE1#
From the outputs, you can derive the conclusion that internet destination is reachable from PE1's outgoing interface facing the ISP, meaning that at least you do not have to troubleshoot the connectivity to the ISP but on the other hand it is NAT overload still is not working.
PE1#show ip nat statistics
Total active translations: 0 (0 static, 0 dynamic; 0 extended)
Peak translations: 0
Outside interfaces:
GigabitEthernet0/2
Inside interfaces:
GigabitEthernet0/1, Loopback0
Hits: 0 Misses: 0
CEF Translated packets: 0, CEF Punted packets: 0
Expired translations: 0
Dynamic mappings:
-- Inside Source
[Id: 1] access-list NAT pool PUBLIC_POOL refcount 0
Total doors: 0
Appl doors: 0
Normal doors: 0
Queued Packets: 0
PE1#
The output now indicates that NAT interfaces configured properly, it uses the access-list named NAT and nat pool named PUBLIC_POOL.
The next step will be verification of the ACL to see if it contains proper networks to be translated:
PE1#show ip access-lists NAT
Standard IP access list NAT
10 permit 10.0.0.0, wildcard bits 0.0.0.255
20 permit 192.168.0.0, wildcard bits 0.0.0.255 (5 matches)
30 permit 192.168.10.0, wildcard bits 0.0.0.255
40 permit 192.168.20.0, wildcard bits 0.0.0.255
It seems that all networks identified in the ACL are what is being expected.
There are a couple more components to check for the proper NAT operation those are NAT statement and NAT pool, let's see the running-config:
PE1#show running-config | section ip nat
ip nat inside
ip nat inside
ip nat outside
ip nat pool PUBLIC_RANGE 75.100.15.0 75.100.15.254 netmask 255.255.255.0
ip nat inside source list NAT pool PUBLIC_POOL overload
PE1#
Here it comes the wrong NAT statement using the pool name PUBLIC_POOL instead of PUBLIC_RANGE. Let's fix the problem:
PE1(config)#no ip nat inside source list NAT pool PUBLIC_POOL overload
PE1(config)#
PE1(config)#ip nat inside source list NAT pool PUBLIC_RANGE overload
PE1(config)#
Lastly, let's ping again the 8.8.8.8 public ip address from the loopback0 interface and then verify NAT translation:
PE1#ping 8.8.8.8 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.3
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/5 ms
PE1#show ip nat translations
Pro Inside global Inside local Outside local Outside global
icmp 75.100.15.1:2 192.168.0.3:2 8.8.8.8:2 8.8.8.8:2
PE1#
Since you restore the NAT operation on the PE1 router, let's assume that the problem is fixed at the CE sites too and you can go to verify the NAT operation back on CEs:
CE1-A#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1
.....
Success rate is 0 percent (0/5)
CE1-A#
CE1-B#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.20.1
.....
Success rate is 0 percent (0/5)
CE1-B#
Unfortunately, the problem is still present, it is time to continue the hypothesis using the troubleshooting dependency model since you have reached the last level1 within "Internet Access" section of the model, you might need to start investigating the "MPLS ISP" section.
MPLS ISP section of the model:
But before you go through the levels you need to look at the model again and think if you can skip some of the levels because of the troubleshooting effort you have already been through, which can expedite the process, the "MPLS ISP" section of the model contains top-level and 8 underlying levels, starting with level8 you can assume that IP connectivity to the CE routers is working because you already have verified previously, regarding the level7, eBGP is working too, you know that because you checked it previously as well, level6, VRF also can be considered working properly because PE3 router has installed static route to CE1's LAN network in its global routing table and it was redistributed by BGP to PE1 router and the LAN network is available from PE3 itself.
Now, level5 comes and it is MP-BGP, two things to verify at this level, first is the BGPv4 neighborship of PE routers but you do not really need to do so because the PE1 router already has CEs' LAN networks in its BGPv4 table meaning that BGPv4 neighborships are working. The second thing is the BGP neighborship as well but only for VPNV4:
PE1#show bgp vpnv4 unicast all summary
BGP router identifier 192.168.0.3, local AS number 65000
BGP table version is 1, main routing table version 1
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
192.168.0.6 4 65000 355 355 1 0 0 05:18:43 0
192.168.0.7 4 65000 358 361 1 0 0 05:18:43 0
PE1#
It seems that BGP VPNV4 neighborships are working too, so what is the problem then?
Is there a problem at level4: "MPLS, LDP"?
Overall, you can conclude that there is working connectivity between PE3 and CE1-A' LAN network, and probably between PE2 and CE1-B' LAN network as well. Also, router PE1 able to communicate with outside networks via ISP directly and with the use of NAT overload. But proper communication over MPLS cloud between PE routers is unconfirmed yet, from PE1 let's ping loopback0 interfaces on PE2 and PE3 sourcing from loopback0 interface:
PE1#ping 192.168.0.6 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.6, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.3
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/5/7 ms
PE1#ping 192.168.0.7 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.0.7, timeout is 2 seconds:
Packet sent with a source address of 192.168.0.3
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/5/7 ms
PE1#
Though, the IPv4 connectivity exists between PE routers but does uninterrupted LSP from PE1 to PE2 and from PE1 to PE2 exist too? Let's verify this too:
PE3#ping mpls ipv4 192.168.0.3/32 source 192.168.0.7
Sending 5, 100-byte MPLS Echos to 192.168.0.3/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
BBBBB
Success rate is 0 percent (0/5)
PE3#
PE2#ping mpls ipv4 192.168.0.3/32 source 192.168.0.6
Sending 5, 100-byte MPLS Echos to 192.168.0.3/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
BBBBB
Success rate is 0 percent (0/5)
PE2#
The output of the ping mpls command shows that both LSP paths to the PE1 from PE2 and PE3 somewhere in the MPLS cloud get unlabeled, hence the problem exists, let's use traceroute mpls command to identify the source where packets on the way to PE1 get unlabeled:
PE3#traceroute mpls ipv4 192.168.0.3/32 source 192.168.0.7
Tracing MPLS Label Switched Path to 192.168.0.3/32, timeout is 2 seconds
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
0 10.0.0.30 MRU 1500 [Labels: 101 Exp: 0]
B 1 10.0.0.29 MRU 1504 [No Label] 7 ms
. 2 *
B 3 10.0.0.29 MRU 1504 [No Label] 4 ms
B 4 10.0.0.29 MRU 1504 [No Label] 4 ms
B 5 10.0.0.29 MRU 1504 [No Label] 4 ms
B 6 10.0.0.29 MRU 1504 [No Label] 5 ms
B 7 10.0.0.29 MRU 1504 [No Label] 4 ms
B 8 10.0.0.29 MRU 1504 [No Label] 4 ms
B 9 10.0.0.29 MRU 1504 [No Label] 4 ms
B 10 10.0.0.29 MRU 1504 [No Label] 4 ms
B 11 10.0.0.29 MRU 1504 [No Label] 3 ms
B 12 10.0.0.29 MRU 1504 [No Label] 4 ms
B 13 10.0.0.29 MRU 1504 [No Label] 4 ms
B 14 10.0.0.29 MRU 1504 [No Label] 4 ms
B 15 10.0.0.29 MRU 1504 [No Label] 3 ms
B 16 10.0.0.29 MRU 1504 [No Label] 4 ms
B 17 10.0.0.29 MRU 1504 [No Label] 3 ms
B 18 10.0.0.29 MRU 1504 [No Label] 4 ms
B 19 10.0.0.29 MRU 1504 [No Label] 6 ms
B 20 10.0.0.29 MRU 1504 [No Label] 3 ms
B 21 10.0.0.29 MRU 1504 [No Label] 4 ms
B 22 10.0.0.29 MRU 1504 [No Label] 4 ms
B 23 10.0.0.29 MRU 1504 [No Label] 3 ms
B 24 10.0.0.29 MRU 1504 [No Label] 3 ms
B 25 10.0.0.29 MRU 1504 [No Label] 3 ms
B 26 10.0.0.29 MRU 1504 [No Label] 4 ms
B 27 10.0.0.29 MRU 1504 [No Label] 3 ms
B 28 10.0.0.29 MRU 1504 [No Label] 3 ms
B 29 10.0.0.29 MRU 1504 [No Label] 3 ms
B 30 10.0.0.29 MRU 1504 [No Label] 4 ms
PE3#
Very intersting output, it shows that PE3 uses label 101 of P1 to reach 192.168.0.3 but then P1 has no outgoing label assinged to the IP address of the PE1 loopback0 interface.
Next step is to check the mpls forwarding-table for the prefix 192.168.0.3/32 on the provider router P1:
P1#show mpls forwarding-table 192.168.0.3 255.255.255.255
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
101 No Label 192.168.0.3/32 108177 Gi0/1 10.0.0.18
The output shows that the outgoing label is missing, meaning that there could be a label filter in place or MPLS is not enabled on the adjacent router's interface, from the topology it is obvious that router P1 is connected to the router P2.
Next, at the router P2, verify MPLS forwarding table for the prefix 192.168.0.3/32:
P2#show mpls forwarding-table 192.168.0.3 255.255.255.255
Local Outgoing Prefix Bytes Label Outgoing Next Hop
Label Label or Tunnel Id Switched interface
None No Label 192.168.0.3/32 0 Gi0/2 10.0.0.14
Neither local or outgoing labels are assigned for the prefix.
Let's check the LDP neighborships on the P2:
P2#show mpls ldp neighbor
P2#
There are no neighbors, this is an indication that something is not configured properly.
Verify which interfaces are enabled for mpls:
P2#show mpls interfaces
Interface IP Tunnel BGP Static Operational
P2#
None of the interfaces participate in the LDP process.
Let's see if interfaces are configured for mpls:
P2#show running-config interface g0/1
Building configuration...
Current configuration : 161 bytes
!
interface GigabitEthernet0/1
description to P1
ip address 10.0.0.18 255.255.255.252
ip router isis
duplex full
speed auto
media-type rj45
mpls ip
end
P2#show running-config interface g0/2
Building configuration...
Current configuration : 162 bytes
!
interface GigabitEthernet0/2
description to PE1
ip address 10.0.0.13 255.255.255.252
ip router isis
duplex full
speed auto
media-type rj45
mpls ip
end
Now that you can see that mpls is enabled under the interfaces, the question rises is it enabled globally?
P2#show running-config | section mpls ip
no mpls ip
mpls ip
mpls ip
P2#
Here it is the problem, that caused the connectivity issue related to the mpls itself. The label switching path was interrupted at the P2 provider router. Now it time to fix the problem and verify the solution:
P2(config)#mpls ip
P2(config)#end
P2#
Verify LSP path from PE3:
PE3#ping mpls ipv4 192.168.0.3/32 source 192.168.0.7
Sending 5, 100-byte MPLS Echos to 192.168.0.3/32,
timeout is 2 seconds, send interval is 0 msec:
Codes: '!' - success, 'Q' - request not sent, '.' - timeout,
'L' - labeled output interface, 'B' - unlabeled output interface,
'D' - DS Map mismatch, 'F' - no FEC mapping, 'f' - FEC mismatch,
'M' - malformed request, 'm' - unsupported tlvs, 'N' - no label entry,
'P' - no rx intf label prot, 'p' - premature termination of LSP,
'R' - transit router, 'I' - unknown upstream index,
'X' - unknown return code, 'x' - return code 0
Type escape sequence to abort.
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/6/8 ms
PE3#
You can also confirm the LSP from PE2 as well.
Verify the Internet access from CE1-A and CE1-B:
CE1-A#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.10.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/6/8 ms
CE1-A#
CE1-B#ping 8.8.8.8 source lo1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
Packet sent with a source address of 192.168.20.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 5/6/8 ms
CE1-B#
The lab is complete, both end-to-end logical data paths are restored, now users in both LAN networks are able to return to work.
Summary:
Using the troubleshooting dependency model can improve communication between network engineers with different level of experience when it comes to the problem-solving situation, the model can be used as the checklist to prevent engineers from going in circles while trying to understand what is not working on the network right now, in addition, the model is about data flow, you put virtual paths on the top and then underneath all technologies used in the topology which allow that path to operate. The model can be organized into the tiers in a hierarchical fashion with multiple levels as shown in this lab when you exhaust possible troubleshooting actions within one tier you can jump to one below and continue your troubleshooting effort.
Comments
Post a Comment