CCIE RS Workbook | CCIE Security Workbook | CCIE SP Workbook| CCIE Collaboration Workbook | CCIE Wireless Workbook | CCIE Data Center Workbook

Troubleshooting IS-IS Routing Update Problems

< Free Open Study >

Configuration in IS-IS is fairly simple and straightforward. In the two-stage process discussed in Chapter 10, the routing process is enabled globally, and IS-IS adjacency formation and LSP flooding is enabled on an interface by applying the command ip router isis. This command also puts the IP subnet information of the interface into the router’s LSP that is generated and flooded to adjacent neighbors. This section covers IS-IS routing update problems on the premise that there are no adjacency problems. This essentially implies that troubleshooting any routing update problems should start with verifying the appropriate adjacencies. Adjacency problems and related troubleshooting methodology are discussed extensively in the previous sections.

The following routing update problems are covered in this section:

  • Route advertisement problems

  • Route flaps

  • Route redistribution problems

One important method for troubleshooting routing problems in IS-IS is by direct inspection of the contents of link-state packets (LSPs). Depending on its configuration, an IS-IS router generates an LSP for each of the levels of routing that it participates in—Level 1 LSP, Level 2 LSP, or both. Inspection of the LSP contents of the expected source of a route can help diagnose routing advertisement problems in cases when no obvious adjacency problems exist. The show isis database detail command displays the contents of a specific LSP. Example 11-25 shows sample output from this command.

Example 11-25 Displaying the Routing Information in an LSP
RT1#show isis database level-1 RT2.00-00 detail



IS-IS Level-2 LSP RT2.00-00

LSPID             LSP Seq Num  LSP Checksum  LSP Holdtime      ATT/P/OL

RT2.00-00         0x00001C9C   0x5F3E        1015              0/0/0

  Area Address: 49.0002

  NLPID:        0xCC

  Hostname: RT2

  IP Address:   11.1.1.2

  Metric: 10         IS-Extended RT1.00

  Metric: 10         IP 10.1.2.0/24

  Metric: 0          IP 11.1.1.2/32

  Metric: 10         IP 11.1.1.6/32

  Metric: 10         IP 192.168.1.0/30

RT2.00-00 is the LSP ID of RT2. Detail output of the LSP, with ID RT2.00-00, shows the IP subnets for directly connected links, together with their metric information.

Another interesting command is show isis topology, which displays a list of all known routers. For example, Example 11-26 shows the IS-IS topology for Figure 11-6 as captured on RT1.

Figure 11-6. A Simple IS-IS Network Topology

image

The backbone (Level 2) is the shaded area. Example 11-26 shows the Level 1 and Level 2 data-bases of RT1. The Level 1 database features RT3 and RT5, confirming that these routers are indeed in the same area as RT1. The Level 2 database describes the backbone, which features RT2 and RT3. RT2 is not in the same area as RT1, but it is connected to the backbone, whereas RT3 is a Level 1–2 router in the same area as RT1.

All this information gleaned from Example 11-26 agrees with the layout in Figure 11-6; this makes the show isis topology command an invaluable command for troubleshooting routing-related problems.

Example 11-26 Obtaining IS-IS Topology Information
RT1#show isis topology



IS-IS paths to level-1 routers

System Id       Metric  Next-Hop        Interface       SNPA

RT1             --

RT3             10      RT3             Et0/0           00d0.58eb.f841

RT5             10      RT5             Et0/0           00d0.58eb.ff01



IS-IS paths to level-2 routers

System Id       Metric  Next-Hop        Interface       SNPA

RT1             --

RT2             10      RT2             Se0/0           *HDLC*

RT3             10      RT3             Et0/0           00d0.58eb.f841

Route Advertisement Problems

Most route advertisement problems can be narrowed down to configuration problems at the source or LSP propagation issues.

Suppose that there is concern that a specific route is missing on RT5 (see Figure 11-6). You can start troubleshooting the problem by obtaining more information on the topology of the network. This should help you determine which router is the original source of the route. Suppose that the route 11.1.1.2/32 turns out to be the loopback address of RT2; the flowchart in Figure 11-7 can be followed to narrow the cause of the problem.

Figure 11-7. Flowchart for Troubleshooting Missing Routes

image

Using knowledge about the topology, you know that RT2 and RT5 are not in the same area. In this case, if route leaking is not enabled in the network, RT5, which is a Level 1–only router, should not see any route from RT2. Hence, tracking the problem takes you along the flowchart through boxes 0-1-7-10-5, where you determine that this is normal behavior. As a Level 1–only router, RT5 should follow a default router to the nearest Level 1 router to get to remote areas.

If you end up in Box 9 or Box 11 along the flowchart, the next case studies and flowcharts might help narrow the problem.

The procedure covered by the flowchart in Figure 11-7 provides a generic method for trouble-shooting missing route problems. The following sections discuss procedures for specific scenarios.

Local Routes Not Being Advertised to Remote

Because IS-IS is a link-state protocol, IS-IS routers depend on LSP flooding to obtain topology and routing information. For example, during stable conditions, each IS-IS router in an area will have the same Level 1 link-state database, which contains the LSPs generated by every router in the area. Dijkstra’s algorithm is run over the LS database to obtain the best path to every advertised route. Therefore, if a route is missing at a section of the area, it’s because the routers in that section did not receive the original LSP, or the LSP was received corrupted and, there-fore, was purged. An even simpler reason could be that the route was not even put into the LSP at the source. The flowchart in Figure 11-8 provides the troubleshooting methodology for such problems. The output of debug isis update-packets and debug isis snp-packets, as demon-strated in Example 11-27, could help decipher this sort of problem if it is related to LSP flooding or issues with link-state database synchronization.

Figure 11-8. Flowchart for Troubleshooting Route Advertisement Problems

image

Step 8 of the flowchart for troubleshooting route advertisement problems calls for enabling the debugging of LSP updates if it is determined that an LSP is not making it to certain remote locations of the IS-IS domain. Example 11-27 shows an output of the debug isis update-packets command. The highlighted lines show RT1 flooding its LSP and also receiving an LSP from RT2. Because the adjacency was just brought up, the output also shows the one-time exchange of CSNPs on point-to-point links between RT1 and RT2.

Example 11-27 Debugging IS-IS Routing Update Problems
RT1#debug isis update-packets



Mar  2 23:25:02: %LINEPROTO-5-UPDOWN: Line protocol on Interface Serial0/0, chp

Mar  2 23:25:02: ISIS-Update: Building L2 LSP

Mar  2 23:25:02: ISIS-Update: No change, suppress L2 LSP 0000.0000.0001.00-00,0

Mar  2 23:25:03: %CLNS-5-ADJCHANGE: ISIS: Adjacency to RT2 (Serial0/0) Up, newy

Mar  2 23:25:07: ISIS-Update: Building L2 LSP

Mar  2 23:25:07: ISIS-Update: TLV contents different, code 16

Mar  2 23:25:07: ISIS-Update: Full SPF required

Mar  2 23:25:07: ISIS-Update: Sending L2 LSP 0000.0000.0001.00-00, seq 160, ht0

Mar  2 23:25:07: ISIS-Update: Rec L2 LSP 0000.0000.0002.00-00, seq 1D16, ht 11,

Mar  2 23:25:07: ISIS-Update: from SNPA *HDLC* (Serial0/0)

Mar  2 23:25:07: ISIS-Update: LSP newer than database copy

Mar  2 23:25:07: ISIS-Update: No change

Mar  2 23:25:08: ISIS-SNP: Rec L2 CSNP from 0000.0000.0002 (Serial0/0)

Mar  2 23:25:08: ISIS-SNP: Rec L2 PSNP from 0000.0000.0002 (Serial0/0)

Mar  2 23:25:08: ISIS-SNP: PSNP entry 0000.0000.0001.00-00, seq 160, ht 1197

Mar  2 23:25:08: ISIS-Update: Sending L2 CSNP on Serial0/0

Mar  2 23:25:08: ISIS-Update: Build L2 PSNP entry for 0000.0000.0002.00-00, se6

Mar  2 23:25:08: ISIS-Update: Build L2 PSNP entry for 0000.0000.0006.00-00, se2

Mar  2 23:25:08: ISIS-Update: Sending L2 PSNP on Serial0/0

Mar  2 23:25:09: ISIS-Update: Building L1 LSP

Mar  2 23:25:09: ISIS-Update: Important fields changed

Mar  2 23:25:09: ISIS-Update: Important fields changed

Mar  2 23:25:09: ISIS-Update: Full SPF required

Mar  2 23:25:09: ISIS-Update: Sending L1 LSP 0000.0000.0001.00-00, seq 15A, ht0

Mar  2 23:25:09: ISIS-Update: Sending L1 CSNP on Ethernet0/0

_____________________________________________________________________________________

RT5#debug isis snp-packets

IS-IS CSNP/PSNP packets debugging is on

RT5#

Mar  6 20:02:28: ISIS-SNP: Rec L1 CSNP from 0000.0000.0001 (Ethernet0/0)

Mar  6 20:02:28: ISIS-SNP: CSNP range 0000.0000.0000.00-00 to FFFF.FFFF.FFFF.FFF

Mar  6 20:02:28: ISIS-SNP: Same entry 0000.0000.0001.00-00, seq 15D

Mar  6 20:02:28: ISIS-SNP: Same entry 0000.0000.0001.01-00, seq 104

Mar  6 20:02:28: ISIS-SNP: Same entry 0000.0000.0005.00-00, seq FEA

The debug isis snp-packets command enables debugging of database synchronization on broadcast interfaces and can troubleshoot route update problems over media such as LANs.

The highlighted line indicates receipt of a CSNP by RT5 from the DIS (RT1). By comparing the contents of the CSNP with the local Level 1 database, RT5 determines that no changes occurred in all known LSPs.

Solution Summary

As depicted by the flowchart in Figure 11-8, there could be many potential causes for problems in which routes are not reaching remote locations in the network. In the extreme case, the route might not be making it into the routing table because of a software glitch relating to IS-IS data structures (Steps 4 and 5). Such situations might need to be reported to the Cisco Technical Assistance Center (TAC) for further assistance.

In other cases, the LSPs might not be reaching some parts of the network because they are getting corrupted in flight (Step 9). Such problems can be determined by a combination of activities, such as debugging the update process and monitoring router logs for LSP errors. If the culprit device is located, it can be isolated or replaced to address the problem.

In most cases, however, the problem could be caused by a trivial configuration error, such as IS-IS not being enabled on an interface (Step 11). Applying the appropriate interface command (ip router isis) to the interface should be sufficient to address the problem.

Situations involving route redistribution or route-leaking problems are addressed in the next section.

Route Redistribution and Level 2-to-Level 1 Route-Leaking Problems

Cisco IOS Software allows routes from other routing sources, such as static, connected interfaces, and other routing protocols, to be redistributed into IS-IS Level 1 routing, Level 2 routing, or both as external routes. Technically, external routes should exist only in the Level 2 routing environment, but Cisco provides a configuration attribute that allows redistribution into Level 1 for practical operational purposes. For example, service providers using IS-IS as IGP with flat Level 1-only domains might need this capability. The existence of the IP External Reachability TLV in a Level 1 LSP should not pose any interoperability issues because IS-IS routers are required to skip TLVs that they don’t understand when they parse an IS-IS packet.

Redistribution in IS-IS is straightforward with hardly any potential pitfalls except occasional, unexpected software bugs. Sometimes, effective metric assignment becomes a challenge in redistribution scenarios. When configuring redistribution, the operator is asked to specify internal or external metrics. The default is the external metric type, which bumps up the metric by 128 on Cisco routers. The increase by 128 instead of 64 is the result of an incorrect bit setting in the default metric field when using narrow metrics. This issue is discussed in the configur-ation section as a Cisco IOS Software implementation issue that has been retained for backward compatibility.

The rather simple flowchart in Figure 11-9 should provide guidance for troubleshooting redistribution problems.

Figure 11-9. Flowchart to Resolve Redistribution Problems

image

Route-Flapping Problem

Route flaps normally are caused by unstable links between the source of the route and the location where the flap is being observed. Typically, multiple routes are impacted at the same time because of an adjacency change affecting many LSPs, all of which might carry numerous IP prefixes. Also, route flaps could induce consistent running of the SPF process, causing potentially dangerous high CPU utilization on route processors that might crash affected routers. If the advertised LSP changes affect only IP prefixes, only partial route calculation (PRC) is run instead of full SPF calculations. PRC is less costly in terms of CPU cycles than full SPF runs. The flowchart in Figure 11-10 provides guidance for troubleshooting route-flapping problems.

Figure 11-10. Flowchart for Troubleshooting Route-Flapping Problems

image

Apart from certain destinations not being reachable—obviously implying routing problems—high CPU utilization by the SPF process certainly should flag instabilities in the network. High CPU utilization can be observed through the IOS command line interface (CLI), network-management applications, or special network-performance analysis tools, such as CiscoWorks 2000, HP Openview, and so on.

From the Cisco IOS Software CLI, the show process cpu command can obtain CPU utilization information. If any indication of high CPU utilization caused by the SPF process is determined, the show isis spf-log command can determine SPF-related events that might cause the high CPU utilization. Example 11-28 provides sample output of this command.

Example 11-28 Tracking SPF Process-Related Events
RT1#show isis spf-log



   Level 1 SPF log

  When   Duration  Nodes  Count     Last trigger LSP   Triggers

03:40:08       0      3      1                        PERIODIC

03:25:08       0      3      1                        PERIODIC

03:10:07       0      3      1                        PERIODIC

02:55:07       0      3      1                        PERIODIC

02:40:07       0      3      1                        PERIODIC

02:25:06       0      3      1                        PERIODIC

02:10:06       0      3      1                        PERIODIC

01:56:08       0      2      1             RT1.01-00  TLVCONTENT

01:55:06       0      2      1                        PERIODIC

01:40:06       0      2      1                        PERIODIC

01:36:31       0      2      1             RT5.00-00  LSPEXPIRED

01:28:31       0      2      2             RT1.01-00  NEWADJ TLVCONTENT

01:28:25       0      3      1             RT5.00-00  NEWLSP

01:25:06       0      3      1                        PERIODIC

01:10:06       0      3      1                        PERIODIC

The output in Example 11-28 lists SPF process runs by time, trigger, and duration. You might recall that LSPs are refreshed every 15 minutes, triggering periodic SPF calculations. Such events are labeled periodic in the show isis spf-log output. It also shows that at 01:25:25, the process was run over an insignificant period of time because of receipt of a new LSP from RT5. Table 11-2 lists and describes common SPF triggers.

Table 11-2. SPF Process Triggers

Trigger Code

Description

AREASET

Area change

ATTACHFLAG

Attached bit setting change

CLEAR

Manual clear

CONFIG

Configuration change

DELADJ

Adjacency deletion

DIS

DIS election

ES

End system information change

HIPPITY LSPDB

Overload bit state change

IP_DEF_ORIG

Default information change

IPDOWN

Connected IP prefix down

IP_EXTERNAL

Route redistribution change

IPIA

Interarea route change

IPUP

Connected IP prefix up

NEWADJ

New neighbor adjacency up

NEWLEVEL

IS-IS process level changed

NEWMETRIC

New metric assigned to an interface

NEWSYSID

New system ID assigned

PERIODIC

Periodic SPF process (LSPDB refresh interval)

TLVCODE

LSP with a new TLV code field received

TLVCONTENT

LSP with changed TLV contents received

The following IS-IS debugging commands also can narrow SPF-related problems:

  • debug isis spf-events

  • debug isis spf-triggers

  • debug isis spf-statistics

  • debug isis update-packets

  • debug isis adj-packets

Use caution when enabling debugging in situations in which the route processor’s CPU already is overtaxed. Example 11-29 provides sample output of the debug isis spf-events command, which displays the events following an interface shut to simulate a link flap. Lines indicating LSPs flagged for recalculation as a result of this event are highlighted. Also, events flagging computation of the Level 1 and Level 2 SPF trees are highlighted.

Example 11-29 debug isis spf-events Command Output Displays Events Following an Interface Shut
RT1(config-if)#debug isis spf-events



Mar  6 20:17:26: ISIS-SPF: L1 LSP 1 (0000.0000.0001.00-00) flagged for recalculC

Mar  6 20:17:26: ISIS-SPF: L1 LSP 5 (0000.0000.0005.00-00) flagged for recalculC

Mar  6 20:17:28: %LINK-5-CHANGED: Interface Ethernet0/0, changed state to admin down

Mar  6 20:17:28: ISIS-SPF: Compute L1 SPT

Mar  6 20:17:28: ISIS-SPF: 3 nodes for level-1

Mar  6 20:17:28: ISIS-SPF: Move 0000.0000.0001.00-00 to PATHS, metric 0

Mar  6 20:17:28: ISIS-SPF: Add 0000.0000.0001.01-00 to TENT, metric 10

Mar  6 20:17:28: ISIS-SPF: Add 0000.0000.0001 to L1 route table, metric 0

Mar  6 20:17:28: ISIS-SPF: Move 0000.0000.0001.01-00 to PATHS, metric 10

Mar  6 20:17:28: ISIS-SPF: Aging L1 LSP 1 (0000.0000.0001.00-00), version 214

Mar  6 20:17:28: ISIS-SPF: Aging L2 LSP 2 (0000.0000.0001.00-00), version 208

Mar  6 20:17:28: ISIS-SPF: Aging L1 LSP 3 (0000.0000.0001.01-00), version 207

Mar  6 20:17:28: ISIS-SPF: Aging L2 LSP 4 (0000.0000.0002.00-00), version 209

Mar  6 20:17:28: ISIS-SPF: Aging L1 LSP 5 (0000.0000.0005.00-00), version 207

Mar  6 20:17:28: ISIS-SPF: Aging L2 LSP 6 (0000.0000.0006.01-00), version 112

Mar  6 20:17:28: ISIS-SPF: Aging L2 LSP 7 (0000.0000.0006.00-00), version 114

Mar  6 20:17:28: ISIS-SPF: Aging L2 LSP 8 (0000.0000.0001.01-00), version 1

Mar  6 20:17:29: %LINEPROTO-5-UPDOWN: Line protocol on Interface Ethernet0/0

Mar  6 20:17:33: ISIS-SPF: Compute L2 SPT

Mar  6 20:17:33: ISIS-SPF: 5 nodes for level-2

Mar  6 20:17:33: ISIS-SPF: Move 0000.0000.0001.00-00 to PATHS, metric 0

Mar  6 20:17:33: ISIS-SPF: Add 49.0001 to L2 route table, metric 0

Mar  6 20:17:33: ISIS-SPF: Add 0000.0000.0001.01-00 to TENT, metric 10

Mar  6 20:17:33: ISIS-SPF: considering adj to 0000.0000.0002 (Serial0/0) metric

Mar  6 20:17:33: ISIS-SPF:   (accepted)

Mar  6 20:17:33: ISIS-SPF: Add 0000.0000.0002.00-00 to TENT, metric 10

Mar  6 20:17:33: ISIS-SPF:   Next hop 0000.0000.0002 (Serial0/0)

Mar  6 20:17:33: ISIS-SPF: Move 0000.0000.0001.01-00 to PATHS, metric 10

Mar  6 20:17:33: ISIS-SPF: Move 0000.0000.0002.00-00 to PATHS, metric 10

Mar  6 20:17:33: ISIS-SPF: Add 49.0002 to L2 route table, metric 10

Mar  6 20:17:33: ISIS-SPF: Redundant IP route 10.1.2.0/255.255.255.0, metric 20d

Mar  6 20:17:33: ISIS-SPF: Redundant IP route 11.1.1.2/255.255.255.255, metric d

Mar  6 20:17:33: ISIS-SPF: Redundant IP route 11.1.1.6/255.255.255.255, metric d

Mar  6 20:17:33: ISIS-SPF: Add 192.168.1.0/255.255.255.252 to IP route table, m0

Mar  6 20:17:33: ISIS-SPF: Next hop 0000.0000.0002/192.168.1.2 (Serial0/0) (rej)

Solution Summary

As shown in the flowchart for troubleshooting route flapping problems (see Figure 11-10), most route-flapping problems can be traced to unstable links in the network (see Step 2). Physical connectivity problems are addressed by troubleshooting the physical and data link layers.

In more complicated scenarios, however, the flaps could be caused by LSP corruption storms or even a routing loop. This might be the case when there are no unstable links in the network but CPU utilization is high, indicating continuous running of the SPF process (see Step 4). The show isis spf-log command can provide indication of which LSPs are changing the most frequently and are triggering the SPF calculations. Similar clues can be gleaned by enabling debugging of the update process with debug isis update-packets. This should be done with care so that the CPU is not overloaded. The logs also can be observed for LSP error loggings, which could give an indication of packet corruption by a culprit device (see Step 7). Zeroing in on any continuously changing LSPs is critical for determining and addressing the problem.


    < Free Open Study >

    Subscribe to Get Free 6 DAYS CCIE Course

     As CCIE instructors we provide this free offer to help any one who is interested in being a CCIE certificate engineer .
    All the below tips are FREE!!!.

    • Latest CCIE certification information.
    • Free advice for any CCIE exam.
    • Free tips on how to become a CCIE network engineer.



    Powered by WPSubscribers
    Your privacy will never be compromised

    CCIE RS Workbook | CCIE Security Workbook | CCIE SP Workbook| CCIE Collaboration Workbook | CCIE Wireless Workbook | CCIE Data Center Workbook

    Free CCIE 6 DAYS Courses Download.