Editing IBM RS/6000 Troubleshooting Guide

This guide documents fault diagnosis for the '''[[IBM RS/6000]]''' family (machine types '''7011 / 7012 / 7013 / 7015 / 7020 / 7025 / 7026 / 7043 / 7044 / 7248'''). RS/6000 troubleshooting differs significantly from PC troubleshooting in that '''every RS/6000 has a 3- or 4-digit LED operator panel''' that displays both BIST (Built-In Self-Test) and POST codes, and that operating-system errors are reported with structured '''SRN''' (Service Request Number) codes that map directly to IBM-supplied FRU lists.

== The Operator Panel ==

Every RS/6000 carries an operator panel with a 3-digit or 4-digit LED display (some later 7026 / 7044 systems use a 16-character LCD instead). The display shows:

* '''Codes during power-on''' — BIST and POST progress; halt codes if a fault is detected.
* '''Codes after AIX boots''' — operating-system service codes; "888" flashing for kernel halt.
* '''Codes during shutdown''' — graceful shutdown progress.

The complete reference is the per-machine '''Operator Guide''' / '''Service Guide''' (SA38-05xx family) plus the kev009 mirror of IBM's "RS/6000 3-Digit Display Codes" document.<ref>http://ps-2.kev009.com/jasper/aix/rs-leds.errcodes.html</ref><ref>http://ps-2.kev009.com/pimpworks/ibm/aixled.html</ref>

== BIST Codes (Hardware POST) ==

BIST runs first at power-on, before any firmware initialisation. BIST codes are the lowest-level fault indication.

{| class="wikitable styled-table" style="width:100%; text-align:center;"
|+'''Common BIST / hardware POST codes'''
! Code !! Meaning
|-
| 100 || BIST is running. Cleared on success.
|-
| 101 || BIST is running. Cleared on success.
|-
| 102 || BIST checksum on the boot ROM failed. ROM corrupted or failed.
|-
| 103 || BIST timed out. CPU or service-processor fault.
|-
| 104 || Equipment Check (general hardware fault).
|-
| 105 || ROS (Read-Only Storage) test failure.
|-
| 106 || L2 cache failure (where fitted).
|-
| 110 || Power Good not received from PSU in time.
|-
| 111 || Bus interface fault on the planar.
|-
| 112 || Watchdog timer expired during BIST.
|-
| 113 || Reset issued by service processor — usually transient.
|-
| 120–129 || Memory subsystem BIST failures.
|-
| 130–139 || I/O subsystem BIST failures.
|-
| 140–149 || SCSI subsystem BIST failures.
|-
| 150–159 || Graphics adapter BIST failures.
|}

The exact per-code FRU list is in each machine's service guide. The above is the cross-family pattern.

== IPL POST Codes (Firmware Boot) ==

After BIST, the firmware (proprietary ROS on POWER1/POWER2; PReP on 7248; Open Firmware on CHRP) runs IPL. IPL codes are in the 200–400 range.

{| class="wikitable styled-table" style="width:100%; text-align:center;"
|+'''Common IPL / firmware POST codes'''
! Code !! Meaning
|-
| 200 || '''Mode switch in SECURE position''' — boot attempted but blocked.<ref>http://ps-2.kev009.com/jasper/aix/older.rs-leds.errcodes.html</ref>
|-
| 201 || '''Checkstop during IPL''' — major hardware fault, CPU halted.<ref>http://ps-2.kev009.com/jasper/aix/older.rs-leds.errcodes.html</ref>
|-
| 202 || NVRAM read failure. Replace NVRAM module (see [[IBM RS/6000 Maintenance Guide]]).
|-
| 203 || NVRAM CRC failure. Re-enter SMS configuration.
|-
| 204 || NVRAM write failure.
|-
| 205 || Service processor not responding.
|-
| 210 || Memory configuration error. Re-seat memory cards.
|-
| 211 || Memory ECC test failure.
|-
| 220 || I/O slot configuration error.
|-
| 221 || PCI bus configuration error (CHRP machines).
|-
| 222 || PCI device enumeration failure.
|-
| 230 || SCSI controller not responding.
|-
| 231 || SCSI bus configuration error.
|-
| 232 || No boot device found on SCSI.
|-
| 240 || Boot disk not in boot list.
|-
| 241 || Boot image corrupt.
|-
| 250 || Network boot started.
|-
| 260 || Diagnostic mode requested (key in SERVICE position).
|-
| 299 || IPL completed; AIX kernel handed control.
|}

The specific code-to-FRU mapping is in the per-machine service guide. The above is the cross-family pattern.

== AIX-Generated Codes (After Kernel Boot) ==

Once AIX is running, the LED panel is driven by the kernel. Codes in the 500–900 range indicate runtime / operational events.

{| class="wikitable styled-table" style="width:100%; text-align:center;"
|+'''Common AIX runtime codes'''
! Code !! Meaning
|-
| 500 || Init started.
|-
| 511 || Filesystem checks running.
|-
| 517 || /etc/inittab being processed.
|-
| 520 || Init complete, daemons starting.
|-
| 540 || Network initialising.
|-
| 551 || Login prompt presented.
|-
| 553 || /etc/rc.tcpip running.
|-
| 581 || ODM (Object Data Manager) reconfiguring.
|-
| 700 || Program Interrupt — kernel panic or invalid instruction.<ref>https://sysadminera.com/2017/02/04/the-led-codes-of-aix/</ref>
|-
| 888 || '''Unexpected system halt''' (kernel panic / hardware fault). Code flashes. See below.<ref>https://sysadminera.com/2017/02/04/the-led-codes-of-aix/</ref>
|}

=== The 888 Halt Sequence ===

When the kernel panics, the operator panel cycles through a sequence beginning with a flashing '''888'''. The format is:

  888 102 xxx yyy

Where:

* '''102''' — software-induced crash (the kernel called the panic routine).
* '''xxx''' — crash code:
:: '''300''' = Data Storage Interrupt (DSI; bad memory access).
:: '''700''' = Program Interrupt (invalid instruction, trap, panic).
:: '''0c0''' = successful kernel dump completed.
:: '''0c5''' = kernel dump attempted but failed.
* '''yyy''' — dump status code.

Press the '''Reset''' button on the operator panel to advance through the sequence. Record every code before resetting.

== SRN — Service Request Numbers ==

When AIX detects a hardware error or when a diagnostic is run, the error is reported as an '''SRN''' (Service Request Number) in the format:

  sss-rrr

Where '''sss''' is the source code (CPU, memory, SCSI, etc.) and '''rrr''' is the reason. SRNs map directly to the FRU list in the per-machine service guide. Run '''diag''' from AIX to see active and historical SRNs.

Examples (from kev009 LED codes reference):<ref>http://ps-2.kev009.com/jasper/aix/older.rs-leds.errcodes.html</ref>

* '''101-xxx''' — Memory.
* '''165-xxx''' — Planar logic.
* '''201-xxx''' — Memory test.
* '''221-xxx''' — Memory adapter.
* '''260-xxx''' — Display adapter.
* '''651-xxx''' — Internal SCSI device.
* '''767-xxx''' — Graphics adapter.

== SCSI-Specific Codes ==

The 0540-class SCSI error codes appear during IPL or AIX boot when the SCSI subsystem cannot enumerate devices.

{| class="wikitable styled-table" style="width:100%; text-align:center;"
|+'''Common SCSI LED / SRN patterns'''
! Code area !! Meaning !! First action
|-
| 0540 || SCSI controller initialisation timeout || Reseat SCSI controller card; check SCSI cable
|-
| 0552 || SCSI bus parity error || Replace SCSI cable, then terminator, then drive
|-
| 0556 || SCSI hard reset failure || Check for missing terminator at end of bus
|-
| 0581 || No bootable device on SCSI bus || Run SMS, verify boot list
|}

== Stiction Diagnosis ==

If the operator panel halts at a 230 / 232 / 0540 / 0556 SCSI code immediately after power-on and the SCSI drive is silent (not spinning), suspect spindle stiction. See [[IBM RS/6000 Maintenance Guide]] for the field stiction fix.

== Mode Switch ==

The operator panel includes a key switch with positions:

* '''NORMAL''' — boot the default OS from the boot list.
* '''SERVICE''' — boot the standalone diagnostic image (from disk or CD-ROM).
* '''SECURE''' — block all boots (used for transport; produces a '''200''' code on attempted boot).

Always check the mode switch position before troubleshooting an apparent boot failure — a key turned to SECURE is a very common "first day in the lab" symptom.

== SMS — System Management Services (CHRP Machines) ==

On 7025 / 7026 / 7043 / 7044, the SMS menu is the user-facing configuration interface. Entry keys at POST:

* '''F1''' — graphical SMS (at the keyboard icon).
* '''F4''' — text-mode SMS (English).
* '''F5''' — boot from default boot list ('''5''' on an ASCII terminal).
* '''F6''' — boot from the multiboot menu.
* '''F8''' — Open Firmware "ok" prompt.

If the system halts at a POST code before reaching the keyboard icon, SMS cannot be entered — diagnose the halt first.

== Memory Faults ==

* '''201''' / '''211''' / '''221''' — memory test or memory adapter failure. Reseat memory cards; on POWER2 39H/397 the memory must be in matched pairs.
* On 7012-39H / 7012-397 / 7013-590 / 7013-595 — pulling one card of a pair will halt POST with a 210 / 211 code.
* ECC scrubbing on long-uptime systems can produce gradual "memory degraded" entries in '''errpt''' — replace the indicated DIMM/SIMM before the second bit error in a word fails outright.

== Graphics Faults ==

* '''No video, system POSTs to 299 / 551 (login prompt) on serial console''' — graphics adapter problem. Try a different GXT card.
* '''Video on initial POST but blank by login prompt''' — AIX driver loaded for a card that isn't present, or wrong DSP file. Boot to single-user mode and run '''lsdev -C''' to verify the graphics device.
* '''GXT800P / GXT3000P pulling down +5 V''' — tantalum decoupling cap failed short. Recap.

== NVRAM Faults ==

NVRAM-class faults appear as:

* '''202 / 203 / 204''' at IPL — NVRAM read / CRC / write failure.
* '''Boot list reset to defaults every power cycle''' — Dallas TimeKeeper battery dead.
* '''Clock at epoch (1 Jan 1970 or 1 Jan 1980) after every cold boot''' — same.

Fix: replace the Dallas TimeKeeper module (see [[IBM RS/6000 Maintenance Guide]]).

== Service Processor Faults (Later Machines) ==

7025 / 7026 / 7044 carry a separate service processor that runs independently of the main CPU. If the service processor fails:

* '''No LED panel display at power-on''' — service processor not booting.
* '''LED display stuck at "STBY" or "0000"''' — service processor running but not transitioning to IPL.
* '''Service processor log full''' — accumulated thermal / fan / ECC events; clear via '''cfgmgr''' or the SP menu.

Service processor reset on 7025-F50 / F80 is via the small recessed switch on the rear of the chassis.

== PSU Faults ==

* '''Dead — no fans, no power''': PSU input rectifier or bulk capacitor. Discharge before any work.
* '''Fans spin briefly, then click-retry''': Power Good not asserted. Could be PSU fold-back, shorted planar tantalum, or '''on 7013 500-series''', a fan sense signal missing.<ref>https://www.ardent-tool.com/RS6000/docs/pdf/38053100.pdf</ref>
* '''Boots cold, fails when warm''': aged secondary electrolytics.
* '''Audible whine, smell of fish''': RIFA X2 cap venting.
* '''Rails low/high''': PSU feedback path issue. Recap.

== Diagnostic Workflow ==

# Confirm the mode switch is in NORMAL.
# Power on. Watch the LED panel.
# If LED stops at a code, look up the code in the per-machine service guide first; if not found, in the cross-family table above; finally in kev009.<ref>http://ps-2.kev009.com/jasper/aix/rs-leds.errcodes.html</ref>
# If LED reaches 299 but AIX does not start, switch to a serial console (DB-9 on the system, 9600/8/N/1) and watch for kernel boot messages.
# If AIX boots but '''errpt''' shows ongoing hardware events, run '''diag''' to get the SRN.
# Cross-reference the SRN to the FRU list in the service guide; replace the indicated FRU(s).

== Common Faults and Resolutions ==

* '''No POST, no LED activity''' — PSU dead. Check rails. Check internal fuse if fitted.
* '''Halts at 200''' — mode switch in SECURE. Turn key.
* '''Halts at 201''' — checkstop. Reseat CPU MCM card; on POWER1/POWER2 7012/7013, suspect SMD electrolyte leakage on planar.
* '''Halts at 202 / 203''' — NVRAM dead. Replace Dallas module.
* '''Halts at 211''' — memory pair mismatch on POWER2. Reseat memory cards.
* '''Halts at 230 / 232 / 0540 / 0556''' — SCSI fault. Cable, terminator, drive stiction.
* '''Halts at 260''' — diagnostic mode (NORMAL key was in SERVICE).
* '''POST reaches 299 but AIX dies during boot''' — corrupted AIX rootvg. Boot from AIX install CD and use the maintenance shell.
* '''Boot loops to SMS''' — boot list points at a missing device; correct via SMS → Multiboot.
* '''Random reboots under load''' — PSU rails sagging, or thermal event due to failed CPU fan.
* '''Graphics card not detected''' — Open Firmware did not enumerate it; reseat and check for tantalum failure.

== Service Processor Logs (CHRP Machines) ==

On 7025 / 7026 / 7044, the service processor maintains an event log accessible via:

* The SP menu (boot to SMS, then "Service Processor Setup" → "Read Service Processor Log").
* AIX command '''snap -a''' (collects all logs to /tmp/ibmsupt for IBM service).
* AIX command '''errpt -a''' (formatted error report).

== Reading the 3-digit LED codes ==

The RS/6000 shows its boot and diagnostic progress, and any fault, as a '''3-digit code on the operator-panel LED display'''. Read the code first — it points straight at the failing area:

<templatestyles src="Template:StyledTable/styles.css" />
[[File:IBM RS-6000 (photo).jpg|thumb|right|300px|IBM RS/6000. Source: Wikimedia Commons.]]
{| class="wikitable styled-table" style="width:80%; text-align:left;"
|+'''RS/6000 operator-panel codes (selection)'''
! Code !! Meaning
|-
| 215 || Low-voltage condition (irrecoverable) — power-supply fault
|-
| 221 || NVRAM CRC error — reset by re-running IPL in Service mode
|-
| 223 || Attempting to IPL from the SCSI devices in the NVRAM boot list (stuck here = boot-device/SCSI problem)
|-
| 21c || L2 cache not detected
|-
| F57 || Bad or low backup battery
|-
| F6A / F7A / F7B || SCSI init / NVRAM init / NVRAM CRC check
|}

The full published code list is in the references.<ref name="rs6k">[http://ps-2.kev009.com/jasper/aix/rs-leds.errcodes.html RS/6000 3-Digit Display Codes], Kev009; and the IBM RS/6000 service manuals. Source for the operator-panel LED codes and the NVRAM-battery behaviour.</ref>

== NVRAM battery ==

The RS/6000 keeps its configuration and IPL device list in battery-backed NVRAM. A '''dead or removed battery''' loses the settings and can leave the machine unbootable or "confused" — some 43P systems are hard to recover after the backup battery is pulled. Replace a low battery (F57), and after any NVRAM loss re-enter the boot configuration in Service mode.<ref name="rs6k">[http://ps-2.kev009.com/jasper/aix/rs-leds.errcodes.html RS/6000 3-Digit Display Codes], Kev009; and the IBM RS/6000 service manuals. Source for the operator-panel LED codes and the NVRAM-battery behaviour.</ref>

== SCSI and memory ==

Most RS/6000 boot hangs are the '''SCSI subsystem''' (the codes step through SCSI init and the IPL-device search) or bad '''memory SIMMs'''. Reseat the SIMMs and the SCSI cabling and terminators, and confirm the boot device against the NVRAM IPL list.<ref name="rs6k">[http://ps-2.kev009.com/jasper/aix/rs-leds.errcodes.html RS/6000 3-Digit Display Codes], Kev009; and the IBM RS/6000 service manuals. Source for the operator-panel LED codes and the NVRAM-battery behaviour.</ref>

== References ==

<references />

== Related Pages ==

* [[IBM RS/6000]]
* [[IBM RS/6000 Maintenance Guide]]
* [[IBM RS/6000 Capacitor Replacement Guide]]
* [[IBM RS/6000 Firmware Update Guide]]

== References ==

* [http://ps-2.kev009.com/jasper/aix/rs-leds.errcodes.html RS/6000 3-Digit Display Codes — kev009 mirror]. Authoritative LED code reference.
* [http://ps-2.kev009.com/jasper/aix/older.rs-leds.errcodes.html RS/6000 older 3-Digit Display Codes]. Specific BIST/IPL code listings.
* [http://ps-2.kev009.com/pimpworks/ibm/aixled.html IBM AIX LED diagnostic codes — kev009]. Cross-reference list.
* [https://sysadminera.com/2017/02/04/the-led-codes-of-aix/ The LED Codes of AIX — sysadminera]. Modern restated reference for 888 halt sequence.
* [https://www.ardent-tool.com/RS6000/docs/pdf/38053100.pdf IBM SA38-0531-00 — RS/6000 7013 500-series Installation and Service Guide]. PSU + planar service.
* [https://sharktastica.co.uk/resources/docs/IBM_SA38-0512-03_RS6000_98_4.pdf IBM SA38-0512-03 — RS/6000 7043 / 7248 Service Guide]. CHRP / PReP service.
* [https://www.manualslib.com/manual/901093/Ibm-Rs-6000-Enterprise-Server-M80.html?page=356 IBM RS/6000 M80 Service Manual]. Common firmware error codes chapter (applies broadly across CHRP RS/6000s).
* [https://www.infania.net/misc/rs6000-firmware/7043140I.html 7043-140 firmware and SMS keys]. F1/F4/F5/F6/F8 entry-key reference.
* [http://ps-2.kev009.com/tl/techlib/qna/faxes/html/krn/krn4.htm IBM TechLib krn4 — RS/6000 boot keys].

{{Navbox-IBMComputers|state=collapsed}}

[[Category:IBM]]
[[Category:Troubleshooting Guides]]