Solution Lab 3 for MPLS Lab 015
How to identify the problem with missing routes?
Topology:
Step1: Access both CE1-A and CE1-B verify their routing tables:
CE1-A#show ip route ospf
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 35 subnets, 3 masks
O IA 10.150.0.4/30 [110/2] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.150.0.8/30 [110/2] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.160.0.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.1.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.2.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.3.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.4.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.5.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.6.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.7.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.165.0.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.1.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.3.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.4.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.5.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.6.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.7.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
CE1-B#show ip route ospf
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 35 subnets, 3 masks
O IA 10.150.0.0/30 [110/2] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.150.0.8/30 [110/2] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.0.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.1.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.2.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.3.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.4.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.5.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.6.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.7.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.0.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.1.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.3.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.4.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.5.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.6.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.7.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
Both outputs confirm that the network 10.165.2.0/24 is not present. Next, you need to think within the domain of the customer system before going to investigate the provider infrastructure. You see, that most of the 10.165.x.x subnets are in the routing tables meaning that there could be a problem with router CE1-C that causing the missing prefix, and additionally there are two locations that have been affected by this, so we can assume and go straight to the console of the CE1-C and check what is going on there.
Step2. Verify if the router CE1-C is the location of the problem, access this router's CLI and verify interfaces' configuration:
CE1-C#show ip interface brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 10.255.2.236 YES NVRAM administratively down down
GigabitEthernet0/1 10.150.0.9 YES NVRAM up up
GigabitEthernet0/2 unassigned YES unset administratively down down
GigabitEthernet0/3 unassigned YES unset administratively down down
Loopback0 10.165.0.1 YES NVRAM up up
Loopback1 10.165.1.1 YES NVRAM up up
Loopback2 10.165.3.1 YES NVRAM up up
Loopback3 unassigned YES unset up up
Loopback4 10.165.4.1 YES NVRAM up up
Loopback5 10.165.5.1 YES NVRAM up up
Loopback6 10.165.6.1 YES NVRAM up up
Loopback7 10.165.7.1 YES NVRAM up up
CE1-C#
Right from the first shot, we see that there is the problem here, it appears that interface loopback3 is missing IP parameters and the interface loopback2 has the IP address that supposes to be assigned to the loopback3 interface. Furthermore, let's verify the running-config for these two loopback interfaces, just to make sure that there are no secondary IP addresses configured:
CE1-C#show running-config interface lo2
Building configuration...
Current configuration : 96 bytes
!
interface Loopback2
ip address 10.165.3.1 255.255.255.0
ip ospf network point-to-point
end
CE1-C#show running-config interface lo3
Building configuration...
Current configuration : 74 bytes
!
interface Loopback3
no ip address
ip ospf network point-to-point
end
Step3. Apply the changes to the CE1-C router's interfaces loopback 2 and 3:
CE1-C(config)#interface lo2
CE1-C(config-if)#ip address 10.165.2.1 255.255.255.0
CE1-C(config-if)#exit
CE1-C(config)#interface lo3
CE1-C(config-if)#ip address 10.165.3.1 255.255.255.0
CE1-C(config-if)#end
Step4. Verify local routing table for connected networks:
CE1-C#show ip route connected
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 36 subnets, 3 masks
C 10.150.0.8/30 is directly connected, GigabitEthernet0/1
L 10.150.0.9/32 is directly connected, GigabitEthernet0/1
C 10.165.0.0/24 is directly connected, Loopback0
L 10.165.0.1/32 is directly connected, Loopback0
C 10.165.1.0/24 is directly connected, Loopback1
L 10.165.1.1/32 is directly connected, Loopback1
C 10.165.2.0/24 is directly connected, Loopback2
L 10.165.2.1/32 is directly connected, Loopback2
C 10.165.3.0/24 is directly connected, Loopback3
L 10.165.3.1/32 is directly connected, Loopback3
C 10.165.4.0/24 is directly connected, Loopback4
L 10.165.4.1/32 is directly connected, Loopback4
C 10.165.5.0/24 is directly connected, Loopback5
L 10.165.5.1/32 is directly connected, Loopback5
C 10.165.6.0/24 is directly connected, Loopback6
L 10.165.6.1/32 is directly connected, Loopback6
C 10.165.7.0/24 is directly connected, Loopback7
L 10.165.7.1/32 is directly connected, Loopback7
Subnet 10.165.2.0/24 is in the routing table, and while we are at the CLI of CE1-C it will be a good step for documentation purposes to check if CE1-C advertises this route to its peer.
Determine this router's OSPF router-ID:
CE1-C#show ip ospf | section ID
Routing Process "ospf 10" with ID 10.165.7.1
Use the router-ID to check Type-1 LSA in the OSPF database:
CE1-C#show ip ospf database router adv-router 10.165.7.1
OSPF Router with ID (10.165.7.1) (Process ID 10)
Router Link States (Area 0)
LS age: 367
Options: (No TOS-capability, DC)
LS Type: Router Links
Link State ID: 10.165.7.1
Advertising Router: 10.165.7.1
LS Seq Number: 80000006
Checksum: 0xC3D
Length: 132
Number of Links: 9
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.0.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.1.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.4.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.5.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.6.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.7.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.2.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.3.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Transit Network
(Link ID) Designated Router address: 10.150.0.9
(Link Data) Router Interface address: 10.150.0.9
Number of MTID metrics: 0
TOS 0 Metrics: 1
The bold portion of the output shows that router CE1-C is now sending LSA about 10.165.2.0/24 network.
Step5. Back on the routers CE1-A and CE1-B check again the routing table for OSPF prefixes:
CE1-A#show ip route ospf | section 10.165.2.0/24
O IA 10.165.2.0/24 [110/3] via 10.150.0.2, 00:13:11, GigabitEthernet0/1
Router CE1-A has the subnet, let's see if it able to ping loopback2 interface of CE1-C router:
CE1-A#ping 10.165.2.1 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.165.2.1, timeout is 2 seconds:
Packet sent with a source address of 10.155.0.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/7 ms
Alright, the problem with CE1-A unable to access resources on the 10.165.2.0/24 is resolved. Next is CE1-B router:
CE1-B#show ip route ospf | section 10.165.2.0/24
empty
Unfortunately, the 10.165.2.0/24 is not in the routing table of CE1-B, further investigation is required. From step one we know that this router is receiving other subnets of CE1-C node but not 10.165.2.0/24, we found that the problem originally was because of improper configuration on the CE1-C router, after the applied changes to this node the problem has been resolved for CE1-A but not for CE1-B! This leads to the conclusion that there might be some incorrect configuration at the ISP site since we experience the problem at the CE1-B router lets go the closest node PE4 and verify if that node has 10.165.2.0/24 network in its possession.
Step6: Interrogate the router PE4, let's start with BGP VPNv4 table:
PE4#show bgp vpnv4 unicast vrf CE1 10.165.2.0/24
BGP routing table entry for 2020:2020:10.165.2.0/24, version 50
Paths: (1 available, best #1, table CE1)
Not advertised to any peer
Refresh Epoch 1
Local
10.100.0.13 (metric 30) (via default) from 10.100.0.13 (10.100.0.13)
Origin incomplete, metric 2, localpref 100, valid, internal, best
Extended Community: RT:2020:2020 OSPF DOMAIN ID:0x0005:0x0000000A0200
OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.150.0.10:0
mpls labels in/out nolabel/1328
rx pathid: 0, tx pathid: 0x0
This output shows that the route is on the PE4 and it coming from PE3. Now, let's verify the routing table for VRF CE1:
PE4#show ip route vrf CE1 | section 10.165.2.0/24
B 10.165.2.0/24 [200/2] via 10.100.0.13, 00:35:29
The prefix is in the routing table for CE1 VRF, that is great, but why is this router do not send this route to the CE1-B? it does redistribute other routes from 10.165.x.x networks but not 10.165.2.0/24. OSPF routes over MPLS coming to the PE routers as inter-area type meaning that they are type 3 summary LSA, checking the OSPF database next will prove that there might be some kind of filtration in place:
PE4#show ip ospf database
OSPF Router with ID (10.150.0.6) (Process ID 10)
Router Link States (Area 0)
Link ID ADV Router Age Seq# Checksum Link count
10.150.0.6 10.150.0.6 519 0x80000005 0x00EAA7 1
10.160.7.1 10.160.7.1 753 0x80000005 0x00582C 9
Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
10.150.0.5 10.160.7.1 753 0x80000004 0x0093F4
Summary Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
10.150.0.0 10.150.0.6 519 0x80000004 0x002F45
10.150.0.8 10.150.0.6 519 0x80000004 0x00DE8D
10.155.0.0 10.150.0.6 519 0x80000004 0x000F5C
10.155.1.0 10.150.0.6 519 0x80000004 0x000466
10.155.2.0 10.150.0.6 519 0x80000004 0x00F870
10.155.3.0 10.150.0.6 519 0x80000004 0x00ED7A
10.155.4.0 10.150.0.6 519 0x80000004 0x00E284
10.155.5.0 10.150.0.6 519 0x80000004 0x00D78E
10.155.6.0 10.150.0.6 519 0x80000004 0x00CC98
10.155.7.0 10.150.0.6 519 0x80000004 0x00C1A2
10.165.0.0 10.150.0.6 519 0x80000004 0x0096CA
10.165.1.0 10.150.0.6 519 0x80000004 0x008BD4
10.165.3.0 10.150.0.6 519 0x80000002 0x0079E6
10.165.4.0 10.150.0.6 519 0x80000004 0x006AF2
10.165.5.0 10.150.0.6 519 0x80000004 0x005FFC
10.165.6.0 10.150.0.6 519 0x80000004 0x005407
10.165.7.0 10.150.0.6 519 0x80000004 0x004911
The output reveals that the network 10.165.2.0/24 is not in the list of the summary LSAs.
Step7. Looking if there is the filtering in place that prevent PE4 from sending the subnet to the CE1-B router:
PE4#show ip access-lists
empty
PE4#show ip prefix-list
ip prefix-list CE1-C: 1 entries
seq 5 permit 10.165.2.0/24
Here it goes, the prefix-list named CE1-C is identifying the 10.165.2.0/24, let's see if this prefix-list active:
PE4#show ip prefix-list detail
Prefix-list with the last deletion/insertion: CE1-C
ip prefix-list CE1-C:
count: 1, range entries: 0, sequences: 5 - 5, refcount: 3
seq 5 permit 10.165.2.0/24 (hit count: 1, refcount: 1)
The hit count shows that the prefix-list has been used. Let's check if there is any route-map matching this prefix-list:
PE4#show route-map
route-map FILTER_INTO_OSPF, deny, sequence 10
Match clauses:
ip address prefix-lists: CE1-C
Set clauses:
Policy routing matches: 0 packets, 0 bytes
route-map FILTER_INTO_OSPF, permit, sequence 20
Match clauses:
Set clauses:
Policy routing matches: 0 packets, 0 bytes
As outlined in the route-map the sequence 10 denies the prefix-list CE1-C and sequence 20 permits anything else. Now it is a time to see where the route-map is being used, we can assume that it applied to the router ospf configuration:
PE4#show running-config | section router ospf
router ospf 10 vrf CE1
redistribute bgp 64500 subnets route-map FILTER_INTO_OSPF
network 10.150.0.4 0.0.0.3 area 0
PE4#
Finally, we found the cause of the issue, now we need to apply the proper changed to the PE4 router.
Step8. Modify the PE4's configuration to allow the prefix 10.165.2.0/24 to be redistributed, we actually do not know the purpose of the route-map and who put in place this filtration, that is why we are not going to remove this but will adjust the route-map sequence 10 to permit instead of denying when later investigate why this has been done:
PE4(config)#route-map FILTER_INTO_OSPF permit 10
PE4(config-route-map)#end
Confirm that route-map sequence 10 now is permitting the ip prefix-list CE1-C:
PE4#show route-map
route-map FILTER_INTO_OSPF, permit, sequence 10
Match clauses:
ip address prefix-lists: CE1-C
Set clauses:
Policy routing matches: 0 packets, 0 bytes
route-map FILTER_INTO_OSPF, permit, sequence 20
Match clauses:
Set clauses:
Policy routing matches: 0 packets, 0 bytes
Step9. Verify again if network 10.165.2.0/24 has been propagated to the router CE1-B:
CE1-B#show ip route ospf | section 10.165.2.0/24
O IA 10.165.2.0/24 [110/3] via 10.150.0.6, 00:03:21, GigabitEthernet0/1
CE1-B#
CE1-B#ping 10.165.2.1 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.165.2.1, timeout is 2 seconds:
Packet sent with a source address of 10.160.0.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/5 ms
CE1-B#
Summary:
This lab is a great tool to develop the skill needed for the creation of network documentation when troubleshooting the networking systems, you can learn how to analyze your finding of the faulty devices and configurations then put it into the record for future use.
Topology:
Step1: Access both CE1-A and CE1-B verify their routing tables:
CE1-A#show ip route ospf
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 35 subnets, 3 masks
O IA 10.150.0.4/30 [110/2] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.150.0.8/30 [110/2] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.160.0.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.1.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.2.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.3.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.4.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.5.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.6.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.160.7.0/24 [110/3] via 10.150.0.2, 00:41:06, GigabitEthernet0/1
O IA 10.165.0.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.1.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.3.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.4.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.5.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.6.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
O IA 10.165.7.0/24 [110/3] via 10.150.0.2, 00:41:07, GigabitEthernet0/1
CE1-B#show ip route ospf
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 35 subnets, 3 masks
O IA 10.150.0.0/30 [110/2] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.150.0.8/30 [110/2] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.0.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.1.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.2.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.3.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.4.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.5.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.6.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.155.7.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.0.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.1.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.3.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.4.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.5.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.6.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
O IA 10.165.7.0/24 [110/3] via 10.150.0.6, 00:41:43, GigabitEthernet0/1
Both outputs confirm that the network 10.165.2.0/24 is not present. Next, you need to think within the domain of the customer system before going to investigate the provider infrastructure. You see, that most of the 10.165.x.x subnets are in the routing tables meaning that there could be a problem with router CE1-C that causing the missing prefix, and additionally there are two locations that have been affected by this, so we can assume and go straight to the console of the CE1-C and check what is going on there.
Step2. Verify if the router CE1-C is the location of the problem, access this router's CLI and verify interfaces' configuration:
CE1-C#show ip interface brief
Interface IP-Address OK? Method Status Protocol
GigabitEthernet0/0 10.255.2.236 YES NVRAM administratively down down
GigabitEthernet0/1 10.150.0.9 YES NVRAM up up
GigabitEthernet0/2 unassigned YES unset administratively down down
GigabitEthernet0/3 unassigned YES unset administratively down down
Loopback0 10.165.0.1 YES NVRAM up up
Loopback1 10.165.1.1 YES NVRAM up up
Loopback2 10.165.3.1 YES NVRAM up up
Loopback3 unassigned YES unset up up
Loopback4 10.165.4.1 YES NVRAM up up
Loopback5 10.165.5.1 YES NVRAM up up
Loopback6 10.165.6.1 YES NVRAM up up
Loopback7 10.165.7.1 YES NVRAM up up
CE1-C#
Right from the first shot, we see that there is the problem here, it appears that interface loopback3 is missing IP parameters and the interface loopback2 has the IP address that supposes to be assigned to the loopback3 interface. Furthermore, let's verify the running-config for these two loopback interfaces, just to make sure that there are no secondary IP addresses configured:
CE1-C#show running-config interface lo2
Building configuration...
Current configuration : 96 bytes
!
interface Loopback2
ip address 10.165.3.1 255.255.255.0
ip ospf network point-to-point
end
CE1-C#show running-config interface lo3
Building configuration...
Current configuration : 74 bytes
!
interface Loopback3
no ip address
ip ospf network point-to-point
end
Step3. Apply the changes to the CE1-C router's interfaces loopback 2 and 3:
CE1-C(config)#interface lo2
CE1-C(config-if)#ip address 10.165.2.1 255.255.255.0
CE1-C(config-if)#exit
CE1-C(config)#interface lo3
CE1-C(config-if)#ip address 10.165.3.1 255.255.255.0
CE1-C(config-if)#end
Step4. Verify local routing table for connected networks:
CE1-C#show ip route connected
Gateway of last resort is not set
10.0.0.0/8 is variably subnetted, 36 subnets, 3 masks
C 10.150.0.8/30 is directly connected, GigabitEthernet0/1
L 10.150.0.9/32 is directly connected, GigabitEthernet0/1
C 10.165.0.0/24 is directly connected, Loopback0
L 10.165.0.1/32 is directly connected, Loopback0
C 10.165.1.0/24 is directly connected, Loopback1
L 10.165.1.1/32 is directly connected, Loopback1
C 10.165.2.0/24 is directly connected, Loopback2
L 10.165.2.1/32 is directly connected, Loopback2
C 10.165.3.0/24 is directly connected, Loopback3
L 10.165.3.1/32 is directly connected, Loopback3
C 10.165.4.0/24 is directly connected, Loopback4
L 10.165.4.1/32 is directly connected, Loopback4
C 10.165.5.0/24 is directly connected, Loopback5
L 10.165.5.1/32 is directly connected, Loopback5
C 10.165.6.0/24 is directly connected, Loopback6
L 10.165.6.1/32 is directly connected, Loopback6
C 10.165.7.0/24 is directly connected, Loopback7
L 10.165.7.1/32 is directly connected, Loopback7
Subnet 10.165.2.0/24 is in the routing table, and while we are at the CLI of CE1-C it will be a good step for documentation purposes to check if CE1-C advertises this route to its peer.
Determine this router's OSPF router-ID:
CE1-C#show ip ospf | section ID
Routing Process "ospf 10" with ID 10.165.7.1
Use the router-ID to check Type-1 LSA in the OSPF database:
CE1-C#show ip ospf database router adv-router 10.165.7.1
OSPF Router with ID (10.165.7.1) (Process ID 10)
Router Link States (Area 0)
LS age: 367
Options: (No TOS-capability, DC)
LS Type: Router Links
Link State ID: 10.165.7.1
Advertising Router: 10.165.7.1
LS Seq Number: 80000006
Checksum: 0xC3D
Length: 132
Number of Links: 9
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.0.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.1.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.4.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.5.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.6.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.7.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.2.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Stub Network
(Link ID) Network/subnet number: 10.165.3.0
(Link Data) Network Mask: 255.255.255.0
Number of MTID metrics: 0
TOS 0 Metrics: 1
Link connected to: a Transit Network
(Link ID) Designated Router address: 10.150.0.9
(Link Data) Router Interface address: 10.150.0.9
Number of MTID metrics: 0
TOS 0 Metrics: 1
The bold portion of the output shows that router CE1-C is now sending LSA about 10.165.2.0/24 network.
Step5. Back on the routers CE1-A and CE1-B check again the routing table for OSPF prefixes:
CE1-A#show ip route ospf | section 10.165.2.0/24
O IA 10.165.2.0/24 [110/3] via 10.150.0.2, 00:13:11, GigabitEthernet0/1
Router CE1-A has the subnet, let's see if it able to ping loopback2 interface of CE1-C router:
CE1-A#ping 10.165.2.1 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.165.2.1, timeout is 2 seconds:
Packet sent with a source address of 10.155.0.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/7 ms
Alright, the problem with CE1-A unable to access resources on the 10.165.2.0/24 is resolved. Next is CE1-B router:
CE1-B#show ip route ospf | section 10.165.2.0/24
empty
Unfortunately, the 10.165.2.0/24 is not in the routing table of CE1-B, further investigation is required. From step one we know that this router is receiving other subnets of CE1-C node but not 10.165.2.0/24, we found that the problem originally was because of improper configuration on the CE1-C router, after the applied changes to this node the problem has been resolved for CE1-A but not for CE1-B! This leads to the conclusion that there might be some incorrect configuration at the ISP site since we experience the problem at the CE1-B router lets go the closest node PE4 and verify if that node has 10.165.2.0/24 network in its possession.
Step6: Interrogate the router PE4, let's start with BGP VPNv4 table:
PE4#show bgp vpnv4 unicast vrf CE1 10.165.2.0/24
BGP routing table entry for 2020:2020:10.165.2.0/24, version 50
Paths: (1 available, best #1, table CE1)
Not advertised to any peer
Refresh Epoch 1
Local
10.100.0.13 (metric 30) (via default) from 10.100.0.13 (10.100.0.13)
Origin incomplete, metric 2, localpref 100, valid, internal, best
Extended Community: RT:2020:2020 OSPF DOMAIN ID:0x0005:0x0000000A0200
OSPF RT:0.0.0.0:2:0 OSPF ROUTER ID:10.150.0.10:0
mpls labels in/out nolabel/1328
rx pathid: 0, tx pathid: 0x0
This output shows that the route is on the PE4 and it coming from PE3. Now, let's verify the routing table for VRF CE1:
PE4#show ip route vrf CE1 | section 10.165.2.0/24
B 10.165.2.0/24 [200/2] via 10.100.0.13, 00:35:29
The prefix is in the routing table for CE1 VRF, that is great, but why is this router do not send this route to the CE1-B? it does redistribute other routes from 10.165.x.x networks but not 10.165.2.0/24. OSPF routes over MPLS coming to the PE routers as inter-area type meaning that they are type 3 summary LSA, checking the OSPF database next will prove that there might be some kind of filtration in place:
PE4#show ip ospf database
OSPF Router with ID (10.150.0.6) (Process ID 10)
Router Link States (Area 0)
Link ID ADV Router Age Seq# Checksum Link count
10.150.0.6 10.150.0.6 519 0x80000005 0x00EAA7 1
10.160.7.1 10.160.7.1 753 0x80000005 0x00582C 9
Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
10.150.0.5 10.160.7.1 753 0x80000004 0x0093F4
Summary Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
10.150.0.0 10.150.0.6 519 0x80000004 0x002F45
10.150.0.8 10.150.0.6 519 0x80000004 0x00DE8D
10.155.0.0 10.150.0.6 519 0x80000004 0x000F5C
10.155.1.0 10.150.0.6 519 0x80000004 0x000466
10.155.2.0 10.150.0.6 519 0x80000004 0x00F870
10.155.3.0 10.150.0.6 519 0x80000004 0x00ED7A
10.155.4.0 10.150.0.6 519 0x80000004 0x00E284
10.155.5.0 10.150.0.6 519 0x80000004 0x00D78E
10.155.6.0 10.150.0.6 519 0x80000004 0x00CC98
10.155.7.0 10.150.0.6 519 0x80000004 0x00C1A2
10.165.0.0 10.150.0.6 519 0x80000004 0x0096CA
10.165.1.0 10.150.0.6 519 0x80000004 0x008BD4
10.165.3.0 10.150.0.6 519 0x80000002 0x0079E6
10.165.4.0 10.150.0.6 519 0x80000004 0x006AF2
10.165.5.0 10.150.0.6 519 0x80000004 0x005FFC
10.165.6.0 10.150.0.6 519 0x80000004 0x005407
10.165.7.0 10.150.0.6 519 0x80000004 0x004911
The output reveals that the network 10.165.2.0/24 is not in the list of the summary LSAs.
Step7. Looking if there is the filtering in place that prevent PE4 from sending the subnet to the CE1-B router:
PE4#show ip access-lists
empty
PE4#show ip prefix-list
ip prefix-list CE1-C: 1 entries
seq 5 permit 10.165.2.0/24
Here it goes, the prefix-list named CE1-C is identifying the 10.165.2.0/24, let's see if this prefix-list active:
PE4#show ip prefix-list detail
Prefix-list with the last deletion/insertion: CE1-C
ip prefix-list CE1-C:
count: 1, range entries: 0, sequences: 5 - 5, refcount: 3
seq 5 permit 10.165.2.0/24 (hit count: 1, refcount: 1)
The hit count shows that the prefix-list has been used. Let's check if there is any route-map matching this prefix-list:
PE4#show route-map
route-map FILTER_INTO_OSPF, deny, sequence 10
Match clauses:
ip address prefix-lists: CE1-C
Set clauses:
Policy routing matches: 0 packets, 0 bytes
route-map FILTER_INTO_OSPF, permit, sequence 20
Match clauses:
Set clauses:
Policy routing matches: 0 packets, 0 bytes
As outlined in the route-map the sequence 10 denies the prefix-list CE1-C and sequence 20 permits anything else. Now it is a time to see where the route-map is being used, we can assume that it applied to the router ospf configuration:
PE4#show running-config | section router ospf
router ospf 10 vrf CE1
redistribute bgp 64500 subnets route-map FILTER_INTO_OSPF
network 10.150.0.4 0.0.0.3 area 0
PE4#
Finally, we found the cause of the issue, now we need to apply the proper changed to the PE4 router.
Step8. Modify the PE4's configuration to allow the prefix 10.165.2.0/24 to be redistributed, we actually do not know the purpose of the route-map and who put in place this filtration, that is why we are not going to remove this but will adjust the route-map sequence 10 to permit instead of denying when later investigate why this has been done:
PE4(config)#route-map FILTER_INTO_OSPF permit 10
PE4(config-route-map)#end
Confirm that route-map sequence 10 now is permitting the ip prefix-list CE1-C:
PE4#show route-map
route-map FILTER_INTO_OSPF, permit, sequence 10
Match clauses:
ip address prefix-lists: CE1-C
Set clauses:
Policy routing matches: 0 packets, 0 bytes
route-map FILTER_INTO_OSPF, permit, sequence 20
Match clauses:
Set clauses:
Policy routing matches: 0 packets, 0 bytes
Step9. Verify again if network 10.165.2.0/24 has been propagated to the router CE1-B:
CE1-B#show ip route ospf | section 10.165.2.0/24
O IA 10.165.2.0/24 [110/3] via 10.150.0.6, 00:03:21, GigabitEthernet0/1
CE1-B#
CE1-B#ping 10.165.2.1 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.165.2.1, timeout is 2 seconds:
Packet sent with a source address of 10.160.0.1
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/5 ms
CE1-B#
Summary:
This lab is a great tool to develop the skill needed for the creation of network documentation when troubleshooting the networking systems, you can learn how to analyze your finding of the faulty devices and configurations then put it into the record for future use.
Comments
Post a Comment