Jump to content

IBM RS/6000 Troubleshooting Guide

From RetroTechCollection

This guide documents fault diagnosis for the IBM RS/6000 family (machine types 7011 / 7012 / 7013 / 7015 / 7020 / 7025 / 7026 / 7043 / 7044 / 7248). RS/6000 troubleshooting differs significantly from PC troubleshooting in that every RS/6000 has a 3- or 4-digit LED operator panel that displays both BIST (Built-In Self-Test) and POST codes, and that operating-system errors are reported with structured SRN (Service Request Number) codes that map directly to IBM-supplied FRU lists.

The Operator Panel

[edit | edit source]

Every RS/6000 carries an operator panel with a 3-digit or 4-digit LED display (some later 7026 / 7044 systems use a 16-character LCD instead). The display shows:

  • Codes during power-on — BIST and POST progress; halt codes if a fault is detected.
  • Codes after AIX boots — operating-system service codes; "888" flashing for kernel halt.
  • Codes during shutdown — graceful shutdown progress.

The complete reference is the per-machine Operator Guide / Service Guide (SA38-05xx family) plus the kev009 mirror of IBM's "RS/6000 3-Digit Display Codes" document.[1][2]

BIST Codes (Hardware POST)

[edit | edit source]

BIST runs first at power-on, before any firmware initialisation. BIST codes are the lowest-level fault indication.

Common BIST / hardware POST codes
Code Meaning
100 BIST is running. Cleared on success.
101 BIST is running. Cleared on success.
102 BIST checksum on the boot ROM failed. ROM corrupted or failed.
103 BIST timed out. CPU or service-processor fault.
104 Equipment Check (general hardware fault).
105 ROS (Read-Only Storage) test failure.
106 L2 cache failure (where fitted).
110 Power Good not received from PSU in time.
111 Bus interface fault on the planar.
112 Watchdog timer expired during BIST.
113 Reset issued by service processor — usually transient.
120–129 Memory subsystem BIST failures.
130–139 I/O subsystem BIST failures.
140–149 SCSI subsystem BIST failures.
150–159 Graphics adapter BIST failures.

The exact per-code FRU list is in each machine's service guide. The above is the cross-family pattern.

IPL POST Codes (Firmware Boot)

[edit | edit source]

After BIST, the firmware (proprietary ROS on POWER1/POWER2; PReP on 7248; Open Firmware on CHRP) runs IPL. IPL codes are in the 200–400 range.

Common IPL / firmware POST codes
Code Meaning
200 Mode switch in SECURE position — boot attempted but blocked.[3]
201 Checkstop during IPL — major hardware fault, CPU halted.[4]
202 NVRAM read failure. Replace NVRAM module (see IBM RS/6000 Maintenance Guide).
203 NVRAM CRC failure. Re-enter SMS configuration.
204 NVRAM write failure.
205 Service processor not responding.
210 Memory configuration error. Re-seat memory cards.
211 Memory ECC test failure.
220 I/O slot configuration error.
221 PCI bus configuration error (CHRP machines).
222 PCI device enumeration failure.
230 SCSI controller not responding.
231 SCSI bus configuration error.
232 No boot device found on SCSI.
240 Boot disk not in boot list.
241 Boot image corrupt.
250 Network boot started.
260 Diagnostic mode requested (key in SERVICE position).
299 IPL completed; AIX kernel handed control.

The specific code-to-FRU mapping is in the per-machine service guide. The above is the cross-family pattern.

AIX-Generated Codes (After Kernel Boot)

[edit | edit source]

Once AIX is running, the LED panel is driven by the kernel. Codes in the 500–900 range indicate runtime / operational events.

Common AIX runtime codes
Code Meaning
500 Init started.
511 Filesystem checks running.
517 /etc/inittab being processed.
520 Init complete, daemons starting.
540 Network initialising.
551 Login prompt presented.
553 /etc/rc.tcpip running.
581 ODM (Object Data Manager) reconfiguring.
700 Program Interrupt — kernel panic or invalid instruction.[5]
888 Unexpected system halt (kernel panic / hardware fault). Code flashes. See below.[6]

The 888 Halt Sequence

[edit | edit source]

When the kernel panics, the operator panel cycles through a sequence beginning with a flashing 888. The format is: 

 888 102 xxx yyy

Where:

  • 102 — software-induced crash (the kernel called the panic routine).
  • xxx — crash code:
300 = Data Storage Interrupt (DSI; bad memory access).
700 = Program Interrupt (invalid instruction, trap, panic).
0c0 = successful kernel dump completed.
0c5 = kernel dump attempted but failed.
  • yyy — dump status code.

Press the Reset button on the operator panel to advance through the sequence. Record every code before resetting.

SRN — Service Request Numbers

[edit | edit source]

When AIX detects a hardware error or when a diagnostic is run, the error is reported as an SRN (Service Request Number) in the format: 

 sss-rrr

Where sss is the source code (CPU, memory, SCSI, etc.) and rrr is the reason. SRNs map directly to the FRU list in the per-machine service guide. Run diag from AIX to see active and historical SRNs.

Examples (from kev009 LED codes reference):[7]

  • 101-xxx — Memory.
  • 165-xxx — Planar logic.
  • 201-xxx — Memory test.
  • 221-xxx — Memory adapter.
  • 260-xxx — Display adapter.
  • 651-xxx — Internal SCSI device.
  • 767-xxx — Graphics adapter.

SCSI-Specific Codes

[edit | edit source]

The 0540-class SCSI error codes appear during IPL or AIX boot when the SCSI subsystem cannot enumerate devices.

Common SCSI LED / SRN patterns
Code area Meaning First action
0540 SCSI controller initialisation timeout Reseat SCSI controller card; check SCSI cable
0552 SCSI bus parity error Replace SCSI cable, then terminator, then drive
0556 SCSI hard reset failure Check for missing terminator at end of bus
0581 No bootable device on SCSI bus Run SMS, verify boot list

Stiction Diagnosis

[edit | edit source]

If the operator panel halts at a 230 / 232 / 0540 / 0556 SCSI code immediately after power-on and the SCSI drive is silent (not spinning), suspect spindle stiction. See IBM RS/6000 Maintenance Guide for the field stiction fix.

Mode Switch

[edit | edit source]

The operator panel includes a key switch with positions:

  • NORMAL — boot the default OS from the boot list.
  • SERVICE — boot the standalone diagnostic image (from disk or CD-ROM).
  • SECURE — block all boots (used for transport; produces a 200 code on attempted boot).

Always check the mode switch position before troubleshooting an apparent boot failure — a key turned to SECURE is a very common "first day in the lab" symptom.

SMS — System Management Services (CHRP Machines)

[edit | edit source]

On 7025 / 7026 / 7043 / 7044, the SMS menu is the user-facing configuration interface. Entry keys at POST:

  • F1 — graphical SMS (at the keyboard icon).
  • F4 — text-mode SMS (English).
  • F5 — boot from default boot list (5 on an ASCII terminal).
  • F6 — boot from the multiboot menu.
  • F8 — Open Firmware "ok" prompt.

If the system halts at a POST code before reaching the keyboard icon, SMS cannot be entered — diagnose the halt first.

Memory Faults

[edit | edit source]
  • 201 / 211 / 221 — memory test or memory adapter failure. Reseat memory cards; on POWER2 39H/397 the memory must be in matched pairs.
  • On 7012-39H / 7012-397 / 7013-590 / 7013-595 — pulling one card of a pair will halt POST with a 210 / 211 code.
  • ECC scrubbing on long-uptime systems can produce gradual "memory degraded" entries in errpt — replace the indicated DIMM/SIMM before the second bit error in a word fails outright.

Graphics Faults

[edit | edit source]
  • No video, system POSTs to 299 / 551 (login prompt) on serial console — graphics adapter problem. Try a different GXT card.
  • Video on initial POST but blank by login prompt — AIX driver loaded for a card that isn't present, or wrong DSP file. Boot to single-user mode and run lsdev -C to verify the graphics device.
  • GXT800P / GXT3000P pulling down +5 V — tantalum decoupling cap failed short. Recap.

NVRAM Faults

[edit | edit source]

NVRAM-class faults appear as:

  • 202 / 203 / 204 at IPL — NVRAM read / CRC / write failure.
  • Boot list reset to defaults every power cycle — Dallas TimeKeeper battery dead.
  • Clock at epoch (1 Jan 1970 or 1 Jan 1980) after every cold boot — same.

Fix: replace the Dallas TimeKeeper module (see IBM RS/6000 Maintenance Guide).

Service Processor Faults (Later Machines)

[edit | edit source]

7025 / 7026 / 7044 carry a separate service processor that runs independently of the main CPU. If the service processor fails:

  • No LED panel display at power-on — service processor not booting.
  • LED display stuck at "STBY" or "0000" — service processor running but not transitioning to IPL.
  • Service processor log full — accumulated thermal / fan / ECC events; clear via cfgmgr or the SP menu.

Service processor reset on 7025-F50 / F80 is via the small recessed switch on the rear of the chassis.

PSU Faults

[edit | edit source]
  • Dead — no fans, no power: PSU input rectifier or bulk capacitor. Discharge before any work.
  • Fans spin briefly, then click-retry: Power Good not asserted. Could be PSU fold-back, shorted planar tantalum, or on 7013 500-series, a fan sense signal missing.[8]
  • Boots cold, fails when warm: aged secondary electrolytics.
  • Audible whine, smell of fish: RIFA X2 cap venting.
  • Rails low/high: PSU feedback path issue. Recap.

Diagnostic Workflow

[edit | edit source]
  1. Confirm the mode switch is in NORMAL.
  2. Power on. Watch the LED panel.
  3. If LED stops at a code, look up the code in the per-machine service guide first; if not found, in the cross-family table above; finally in kev009.[9]
  4. If LED reaches 299 but AIX does not start, switch to a serial console (DB-9 on the system, 9600/8/N/1) and watch for kernel boot messages.
  5. If AIX boots but errpt shows ongoing hardware events, run diag to get the SRN.
  6. Cross-reference the SRN to the FRU list in the service guide; replace the indicated FRU(s).

Common Faults and Resolutions

[edit | edit source]
  • No POST, no LED activity — PSU dead. Check rails. Check internal fuse if fitted.
  • Halts at 200 — mode switch in SECURE. Turn key.
  • Halts at 201 — checkstop. Reseat CPU MCM card; on POWER1/POWER2 7012/7013, suspect SMD electrolyte leakage on planar.
  • Halts at 202 / 203 — NVRAM dead. Replace Dallas module.
  • Halts at 211 — memory pair mismatch on POWER2. Reseat memory cards.
  • Halts at 230 / 232 / 0540 / 0556 — SCSI fault. Cable, terminator, drive stiction.
  • Halts at 260 — diagnostic mode (NORMAL key was in SERVICE).
  • POST reaches 299 but AIX dies during boot — corrupted AIX rootvg. Boot from AIX install CD and use the maintenance shell.
  • Boot loops to SMS — boot list points at a missing device; correct via SMS → Multiboot.
  • Random reboots under load — PSU rails sagging, or thermal event due to failed CPU fan.
  • Graphics card not detected — Open Firmware did not enumerate it; reseat and check for tantalum failure.

Service Processor Logs (CHRP Machines)

[edit | edit source]

On 7025 / 7026 / 7044, the service processor maintains an event log accessible via:

  • The SP menu (boot to SMS, then "Service Processor Setup" → "Read Service Processor Log").
  • AIX command snap -a (collects all logs to /tmp/ibmsupt for IBM service).
  • AIX command errpt -a (formatted error report).
[edit | edit source]

References

[edit | edit source]


  1. ↑ http://ps-2.kev009.com/jasper/aix/rs-leds.errcodes.html
  2. ↑ http://ps-2.kev009.com/pimpworks/ibm/aixled.html
  3. ↑ http://ps-2.kev009.com/jasper/aix/older.rs-leds.errcodes.html
  4. ↑ http://ps-2.kev009.com/jasper/aix/older.rs-leds.errcodes.html
  5. ↑ https://sysadminera.com/2017/02/04/the-led-codes-of-aix/
  6. ↑ https://sysadminera.com/2017/02/04/the-led-codes-of-aix/
  7. ↑ http://ps-2.kev009.com/jasper/aix/older.rs-leds.errcodes.html
  8. ↑ https://www.ardent-tool.com/RS6000/docs/pdf/38053100.pdf
  9. ↑ http://ps-2.kev009.com/jasper/aix/rs-leds.errcodes.html
Retrieved from "https://wiki.retrotechcollection.com/index.php?title=IBM_RS/6000_Troubleshooting_Guide&oldid=10918"