Table of Contents

Problem Solving with Subsystems
Identifying Startup Problems
Troubleshooting the Power Subsystem
Troubleshooting the Cooling Subsystem
Troubleshooting the Processor Subsystem

Troubleshooting the RP and SP (or SSP)
Troubleshooting the Interface Processors

Troubleshooting the Installation

Your router went through extensive testing and burn-in before leaving the factory. However, if you encounter problems starting up, use the information in this chapter to help isolate the cause. Problems with the initial startup will most likely be caused by the source power or an interface processor that has become dislodged from the backplane. Although an overtemperature condition is unlikely at initial startup, the environmental monitoring functions are included because they also monitor internal voltages.

At the initial system boot, you should verify the following:

When each of these conditions is met, the hardware installation is complete, and you should proceed to the appropriate software configuration documentation to configure the system and the individual interfaces. If the startup sequence fails before these conditions are met, use the procedures in this chapter to isolate and, if possible, resolve the problem.

• Date you received the router

• Chassis serial number (located on a label on the right rear deck of the chassis)

• Type of software and release number

• Brief description of the problem you are having

• Brief explanation of the steps you have already taken to isolate and resolve the problem

• Maintenance agreement or warranty information

Figure 4-1 shows the general troubleshooting strategy described in this chapter. Refer to this chart as necessary to follow the steps to isolate problems to a specific subsystem, and resolve the problem if possible.

The rest of this chapter describes troubleshooting methods and defines how the router is divided into subsystems for more efficient problem solving. A description of a normal startup sequence contains pointers to sections in this chapter that contain troubleshooting procedures for specific components so that you can determine where your system is having trouble and then troubleshoot that specific component or subsystem.

Problem Solving with Subsystems

The key to problem solving the system is to try to isolate the problem to a specific subsystem. The first step in solving startup problems is to compare what the system is doing to what it should be doing. Since a startup problem is usually attributable to a single component, it is more efficient to first isolate the problem to a subsystem rather than troubleshoot each component in the system. For these troubleshooting procedures, consider the following subsystems:

• Power subsystem—This subsystem comprises the AC-input or DC-input power supply, the external power cord, and the internal power harness that connects to the backplane.

• Cooling subsystem—This subsystem comprises the fan tray assembly, which includes the six individual fans in the array, the fan control board, and the tray itself. All six fans should be operating whenever the system power is on. The system automatically shuts down if any one or more of the fans is operating outside the specified range, or shuts down completely. The variable speed feature allows the fans to operate at a slower speed and provide quieter operation when the internal chassis temperature is within the normal operating range. If the internal temperature exceeds a specified temperature, the fan speed increases to move more cooling air through the chassis. As a result, it may be difficult to determine whether or not the fans are operating in noisy, air-conditioned rooms. When viewing the chassis from the noninterface processor end, the fan array spans the right wall of the chassis interior. If you determine that the fans are not operating, contact a customer service representative immediately. There are no installation adjustments that you should make if it does not function properly at initial startup.

• Processor subsystem—This subsystem includes the RP, SP (or SSP), and all interface processors. The RP contains the system operating software, and the enabled LED on the SP (or SSP) and each interface processor indicates whether or not the RP was able to initialize it. Remember that an interface processor that is partially installed in the backplane will cause the system to hang and crash.

Identifying Startup Problems

When you start up the router for the first time, you should observe the startup sequence described in the section "Starting the Router" in the chapter "Installing the Router." This section contains a more detailed description of the normal startup sequence and describes the steps to take if the system does not perform that sequence as expected.

By checking the state of the LEDs on the power supply and processor modules (the RP, SP [or SSP], and interface processors), you can determine when and where the system failed in the startup sequence. Use the following descriptions to isolate the problem to a subsystem, then proceed to the appropriate sections (indicated in each description) to try to resolve the problem.

When you start up the system by turning on the main system power switch on the interface processor end of the power supply, the following should occur:

• The DC OK LED should go on immediately, and it should remain on as long as the system is receiving source power and the power switch is in the on (|) position. If this LED does not go on, or if it goes off while the power switch is still on, there could be a problem with either the source power, the internal DC power that the power supply distributes to the internal components, or the cooling subsystem.

  The green DC OK LED indicates the status of the power supply and internal DC voltages. The DC OK LED stays on when all of the following conditions are met:

• AC-input power supply is on and receiving 100 to 240 VAC, 50 to 60 Hz source power or the DC-input power supply is on and receiving -40 to -52 VDC (-48 VDC nominal) in North America, or -56 and -72 VDC (-60 VDC nominal) in the European Community.

• Power supply is providing the +5, +12, -12, and +24 VDC to internal components.

• All internal DC voltages are within tolerance.

  If the source power or any of the internal DC voltages exceeds allowable tolerances, the DC OK LED will not go on, or will go off shortly after you turn on the power. Because both the RP (which uses +5, +12, -12 VDC), and the fan tray (which uses +24 VDC) are required for operation, a problem with any of the internal DC lines can prevent the system from starting up or continuing operation.

  For example, if there is a problem with the +24 VDC line that supplies the fan tray, the system will start up but also recognize that the fans are not operating. The system will initiate a fan failure shutdown sequence, display the appropriate warning messages, then shut down after two minutes. If there is a problem with any of the other internal DC lines, the RP will not be able to initialize the system software, so the system might attempt to start up and fail during the boot sequence.

  Depending upon when the DC OK LED goes off, proceed as follows:

• If the DC OK LED stays off (if it never goes on) when you turn on the power switch, there is a problem with either the source power or the DC power that is distributed to the internal components. Proceed to the section "Troubleshooting the Power Subsystem."

• If the DC OK LED goes on temporarily, then goes off within 30 seconds, the system is most likely shutting itself down because it detected an out-of-tolerance power or temperature condition within the power supply. Proceed to the section "Troubleshooting the Power Subsystem."

• If the DC OK LED goes on, and the system starts up as expected but then displays the following message and shuts down after two minutes, there is a problem with the fan tray. Proceed to the section "Troubleshooting the Cooling Subsystem."

%ENVM-2-FAN: Fan array has failed, shutdown in 2 minutes
 

• If the DC OK LED stays off yet the system starts up correctly, displays the preceding message, and shuts down after about two minutes, there is a problem with the +24VDC line to the fan tray. Proceed to the section "Troubleshooting the Power Subsystem."

• As soon as you turn on the power switch, you should immediately hear the fans operating. If you are in an unusually noisy environment, such as an air-conditioned wiring closet with other equipment noise, place your hand near the left side of the chassis (when facing the interface processor end); you should feel the exhaust air that is being forced out the side of the chassis.

  • If the DC OK LED is on but the fans are not operating, there is a problem with the fan tray or with one of the fans. (The system will shut itself down if it detects that any of the fans are not functioning properly.) Proceed to the section "Troubleshooting the Cooling Subsystem."

  • If the DC OK LED is on at startup and fans are operating, but the system shuts down after two minutes, there is a problem with the fan tray. Proceed to the section "Troubleshooting the Cooling Subsystem."

• When you turn on the system power, the LEDs on the RP, which is located in the top processor slot (see Figure 1-3) should go on and off as follows:

  • The RP boot error LED goes on and stays on for 1 to 2 seconds when the system boot sequence is initialized, but it should otherwise remain off. If the system software is unable to start up, this LED will go on and stay on.

  • The RP normal LED goes on after the system has completed a successful boot to indicate normal system operation. This LED should remain on during system operation.

  • The halt CPU LED should always remain off. This LED goes on only if the system detects a processor hardware failure.

  If the normal LED on the RP does not go on, or if either the boot error or halt CPU LED goes on and stays on, proceed to the section "Troubleshooting the Cooling Subsystem."

• The enabled LED on the SP (or SSP) and on each interface processor goes on when the RP has completed initialization of the interface processor or SP (or SSP) for operation. This LED indicates that the SP (or SSP) or interface processor is receiving power and has been recognized by the RP; it does not indicate the state of the individual interfaces. It does, however, indicate that an interface processor or SP (or SSP) contains a valid microcode version. If an enabled LED fails to go on, proceed to the section "Troubleshooting the Interface Processors."

When the LEDs indicate that the system has initialized successfully, the system banner (similar to the following example) should be displayed on the console screen.

System Bootstrap, Version 4.6(5), SOFTWARE
Copyright (c) 1986-1995 by cisco Systems
RP1 processor with 16384 Kbytes of memory
### (text omitted) ###
F3: 2012356+47852+194864 at 0x1000
 
 
               Restricted Rights Legend
 
Use, duplication, or disclosure by the Government is
subject to restrictions as set forth in subparagraph
(c) of the Commercial Computer Software - Restricted
Rights clause at FAR sec. 52.227-19 and subparagraph
(c) (1) (ii) of the Rights in Technical Data and Computer
Software clause at DFARS sec. 252.227-7013.
 
              cisco Systems, Inc.
              170 West Tasman Drive
              San Jose, California 95134-1706
 
GS Software (GS7), Version 10.3(1) [fc3], RELEASE SOFTWARE
Copyright (c) 1986-1995 by cisco Systems, Inc.
 
RP1 (68040) processor with 16384K bytes of memory.
(text omitted)
 
Press RETURN to get started!
 

If it is not displayed, refer to the section "Connecting the Console Terminal" in the chapter "Installing the Router" to verify that the terminal is set correctly and that it is properly connected to the RP console port.

Troubleshooting the Power Subsystem

Check the following to help isolate the problem:

• On the interface processor end of the power supply, is the DC OK LED on?

  • If yes, the power source is good, and the power supply is functional.

  • If no but the fans are operating and LEDs on the processor modules are on, suspect a faulty power supply LED. The RP uses +5, +12, -12 VDC, and the fan tray uses +24 VDC; therefore, if both the RP and the fans are operating, all internal DC lines are within tolerance.

  • If no and there is not other obvious activity, first suspect that the power switch is not fully in the on (|) position. Examine the switch and ensure that it is completely pushed into the on (|) position. (Refer to the section "Starting the Router" in the chapter "Installing the Router")

  • If the system power switch is set correctly and the DC OK LED remains off, suspect the power source or the power cable. Turn off the power switch, connect the power cable to another power source if one is available, and turn the switch back on. If the LED then goes on, the problem is the first power source.

  • If the LED fails to go on after you connect the power supply to a new power source, swap the power cable with a replacement if one is available and turn the switch back on. If the DC OK LED then goes on, the power cable is faulty. (The AC power cable for the Cisco 7000 is compatible with the Cisco 7010. You supply the DC power cable.)

  • Ensure that the fan tray is seated properly. Refer to the section "Replacing the Fan Tray" in the chapter "Maintenance" to remove and reseat the fan tray. Ensure that the fan control board edge connector is inserted fully in the backplane socket. After you replace the chassis cover panel, try starting the system again.

  • If the LED still fails to go on when connected to a different power source with a new power cable, the power supply is probably faulty. If a spare power supply is available, replace the existing power supply with the spare and restart the system. (Refer to the section "Removing and Replacing the Power Harness Cover" in the chapter "Maintenance.") If the DC OK LED then goes on, return the faulty power supply for replacement.

If you are unable to resolve the problem, or if you determine that either the power supply or power cable is faulty, contact a service representative for instructions.

Troubleshooting the Cooling Subsystem

Check the following to help isolate the problem:

• When you start up the system, do the fans go on?

  To determine whether the fans are operating, listen for the fan motors. In noisy environments, place your hand next to the left side of the chassis (when facing the interface processor end of the router) to feel for air being forced out the side vents.

• If yes, the +24 VDC line to the fan tray is good, but there might be a problem with the software or an individual fan.

• If no, there is a problem with the fan tray or the +24 VDC power. Ensure that the DC OK LED is on. If the LED is off, there could be a problem with the +24 VDC supply to the fan tray. Refer to the section "Troubleshooting the Power Subsystem."

• If no and the DC OK LED is on, ensure that the fan tray is seated properly. Refer to the section "Replacing the Fan Tray" in the chapter "Maintenance" to remove and reseat the fan tray. Ensure that the fan control board edge connector is inserted fully in the backplane socket. After you replace the chassis cover panel, try starting the system again.

• Do the system and the fans start up but shut down after about two minutes?

  • If you have changed the configuration register boot field jumper settings or altered the configuration file boot instructions, the system could be booting a software image (earlier than Software Release 9.17[6]) that does not recognize the signals from the fan control board (and therefore assumes that the cooling subsystem is not operating).

  Use the show version command to display the current software version and ensure that the currently running software image is Software Release 9.17(6) or later. You will have to enter this command quickly after the system boot is complete and before the system reaches the two-minute shutdown timeout.
  

  The following message, if displayed, indicates that one or more fans has failed or is operating out of tolerance.
  

%ENVM-2-FAN: Fan array has failed, shutdown in 2 minutes
 

If one or more fans or the fan control board fails, you must replace the fan tray.

• The following message, if displayed, indicates that the system has detected an overtemperature condition or out-of-tolerance power inside the chassis.

Queued messages:
%ENVM-1-SHUTDOWN: Environmental Monitor initiated shutdown
 

If an environmental shutdown results from an out-of-tolerance power condition, the DC OK LED will go off before the system shuts down. (Refer to the section "Troubleshooting the Power Subsystem.") Although an overtemperature condition is unlikely at initial startup, ensure that heated exhaust air from other equipment is not entering the inlet vents, and that there is sufficient clearance around the sides of the chassis to allow cooling air to flow. Refer to the guidelines in the chapter "Preparing for Installation" for acceptable site configurations.

This message could also indicate a faulty component or temperature sensor. Before the system shuts down, use the show environment or show environment table command to display the internal chassis environment. (Refer to the chapter "Product Overview" for detailed descriptions.)

If you are still unable to resolve the problem, contact a service representative for further instructions.

Troubleshooting the Processor Subsystem

The processor subsystem comprises the RP, SP (or SSP), and interface processors. The RP and SP (or SSP) are required system components; the system cannot operate unless both the RP and SP (or SSP) are installed properly. However, because the CxBus interface processors support OIR, the system can operate without any interface processors installed as long as none are in partial contact with the backplane pins; an interface processor that is partially connected to the backplane will send incomplete signals to the processor, which will fault the bus and cause the system to hang. Therefore, first ensure that the RP and SP (or SSP) are installed properly and the system software has initialized successfully. Then, if necessary, you can troubleshoot individual interface processors.

Troubleshooting the RP and SP (or SSP)

These procedures assume that the RP and router are in the original factory configuration, and that you have not changed any configuration register settings or made changes to your configuration file. If you have made such changes, refer to the section "RP and RSP7000 Configurations" in the chapter "Maintenance" to ensure that all jumpers are set to the factory default. For example, the configuration register boot field jumper should be set to boot from ROM (0x0001).

If the RP LEDs do not go on as expected, check the following items to help isolate the problem.

• Do all of the RP LEDs remain off when system power switch is turned on?

  • If all RP LEDs stay off, first refer to the sections "Troubleshooting the Power Subsystem" and "Troubleshooting the Cooling Subsystem" to ensure that both the fan and power supply are functioning properly.

  • If the SP (or SSP) enabled LED is off and any of the RP LEDs are on, the SP (or SSP) may not be seated in its slot properly. Turn off the system power, loosen the captive installation screws on the SP (or SSP), and use the ejector levers to eject then reseat the SP (or SSP) in its slot. Retighten the captive screws and restart the system.

  • If the power supply and fans appear operational, but none of the RP LEDs are on, suspect that an improperly connected RP, SP (or SSP), or interface processor has hung the bus. Turn the system power switch off and, on each processor module, loosen the captive installation screws and use the ejector levers to eject and reseat each board. (For a description and illustration of the ejector levers, refer to the section "Ejector Levers" in the chapter "Maintenance.") Tighten all captive installation screws, then restart the system.

• Is the RP normal LED on?

  • If yes, the system software has initialized successfully, and the system is operational.

• Is the boot error LED on?

  • If yes, the system software is unable to start up. Turn the system power off and then on again.

  • If you have a spare RP with the system software ROMs installed, turn off the system power and replace the installed RP with the spare.

  • If after several attempts the boot error LED continues to go on, try to estimate the amount of time that elapses between power on and when the LED goes on, and contact a service representative for instructions.

• Is the RP CPU halt LED on?

  • If yes, the system has detected a processor hardware failure. Contact a service representative for instructions.

Troubleshooting the Interface Processors

Check the following to help isolate the problem:

• Are all SP (or SSP) and interface processor enabled LEDs on?

  • If yes, the system is operational. Proceed to the instructions for configuring the interfaces in the appropriate software configuration documentation.

• Are any SP (or SSP) and interface processor enabled LEDs on?

  • If none of the enabled LEDs are on, first check the RP normal LED, which will be on if the system booted successfully. The normal LED should be on if the system.

  • If the enabled LED on the SP (or SSP) is off, suspect that the SP (or SSP) has pulled away from the backplane. Turn all system power off, then use the ejector levers to eject and then reseat the SP (or SSP). (For a description and illustration of the ejector levers, refer to the section "Online Insertion and Removal Information" in the chapter "Maintenance.") Tighten the captive installation screws and restart the system.

  • If the enabled LED on an individual interface processor is off, suspect that the interface processor has pulled away from the backplane. You do not have to turn off the system power to remove and replace an interface processor. Use the ejector levers to eject and then reseat the interface processor, then tighten both of the captive installation screws. (For a description and illustration of the ejector levers, refer to the section "Ejector Levers" in the chapter "Maintenance.") After the system reinitializes the interfaces, the enabled LED on the interface processor should go on.

If you experience trouble with the startup that is not resolved with these procedures, contact a service representative for assistance and further instructions.

Posted: Wed Oct 31 18:02:42 PST 2001
All contents are Copyright © 1992--2001 Cisco Systems, Inc. All rights reserved.
Important Notices and Privacy Statement.