Replacing a Bad Disk

Because disks are physical devices, their hardware can fail, necessitating their replacement. After a failing disk is replaced with a new one (retaining the hardware address of the original disk to avoid confusion), the data must be restored to that disk from a backup.

Since the disk was under LVM control, it can have physical extents for several logical volumes on it. The layout of those logical volumes must first be restored and the data for each of those logical volumes restored from backup.

This section provides a step-by-step guide to replacing a faulty LVM disk.

Review “Preparing for LVM System Recovery” for steps to perform before a disk fails. Read this section carefully, and implement the required procedures as soon as possible. Your system recovery might rely on these steps.

If you have any questions about the recovery process, contact your local HP Customer Response Center for assistance.




	TIP: For an in-depth discussion of disk failures, see the white paper When Good Disks Go Bad: Dealing with Disk Failures under LVM, available at http://docs.hp.com. It covers additional topics such as recognizing a disk failure, identifying the failing disk, and choosing the appropriate resolution, such as removing the disk instead of replacing it. The paper also covers releases prior to HP-UX 11i Version 3.

Disk Replacement Prerequisites

Once you have isolated a failed disk, the replacement process depends on answers to the following questions:

Is the disk hot-swappable?
You can remove or add an inactive hot-swappable hard disk drive module to a system while power is still on and the SCSI bus is still active. That is, you can replace or remove a hot-swappable disk from a system without turning off the power to the entire system.
Consult your system hardware manuals for information about which disks in your system are hot-swappable. Specifications for other hard disks are available in their installation manuals at http://docs.hp.com.
If your disk is not hot-swappable, you must schedule system down time to replace the disk.

Is the disk the root disk or part of the root volume group?

If the root disk is failing, the replacement process includes steps to set up the boot area; in addition, you might have to boot from its mirror if the primary root disk has failed. If a failing root disk is not mirrored, you must reinstall to the replacement disk or recover it from an Ignite-UX backup.

To determine whether the disk is in the root volume group, use the lvlnboot command with the –v option. It lists the disks in the root volume group, and any special volumes configured on them. For example:

# lvlnboot –v
Boot Definitions for Volume Group /dev/vg00:
Physical Volumes belonging in Root Volume Group:
        /dev/disk/disk47_p2 -- Boot Disk
Boot: lvol1     on:     /dev/disk/disk47_p2
Root: lvol3     on:     /dev/disk/disk47_p2
Swap: lvol2     on:     /dev/disk/disk47_p2
Dump: lvol2     on:     /dev/disk/disk47_p2, 0

What logical volumes are on the disk, and are they mirrored?

After you replace the disk, you might need to restore data from backups. However, you must only recover data for a subset of the logical volumes in the volume group. Only the logical volumes that actually have physical extents on the disk are affected. In addition, if a logical volume is mirrored, there is probably a current copy of the data on the mirror, so it does not need to be recovered from backup.

You can find the list of logical volumes using the disk with the pvdisplay command. With the –v option, pvdisplay shows a listing of all the physical extents on a physical volume and to what logical volume they belong. This list is long; pipe it to more or send it to a file. For example:

# pvdisplay -v /dev/disk/disk3 | more
...
   --- Distribution of physical volume ---
   LV Name            LE of LV  PE for LV 
   /dev/vg00/lvol5    50        50
   /dev/vg00/lvol6    245       245
 ...

In this example, logical volumes /dev/vg00/lvol5 and /dev/vg00/lvol6 have physical extents on this disk, so you must restore lvol5 and lvol6 only.

If pvdisplay fails, you can see any configuration documentation you created in advance, or use the vgcfgdisplay command, available from your HP support representative.

For each of the logical volumes affected, use lvdisplay to determine if the number of mirror copies is greater than zero. This verifies that the logical volume is mirrored. For example:

# lvdisplay /dev/vg00/lvol1
--- Logical volumes ---
LV Name                 /dev/vg00/lvol1
VG Name                 /dev/vg00
LV Permission           read/write
LV Status               available/syncd
Mirror copies           1
Consistency Recovery    MWC
Schedule                parallel
LV Size (Mbytes)        300
Current LE              75
Allocated PE            150
Stripes                 0
Stripe Size (Kbytes)    0
Bad block               off
Allocation              strict/contiguous
IO Timeout (Seconds)    default

The number of mirror copies is not zero; therefore, the logical volume is mirrored.

Use lvdisplay again to determine which logical extents are mapped onto the suspect disk, and whether there is a current copy of that data on another disk. With the –v option, lvdisplay shows every logical extent, its mapping to any physical extents, and the status of those physical extents (stale or current).

This listing can be quite long, so use grep to confine the listing to the disk that is being replaced. For example:

# lvdisplay -v /dev/vg00/lvol1 | grep –e /dev/disk/disk3 –e ’???’
   00000 /dev/disk/disk3  00000 current  /dev/disk/disk6  00000 current
   00001 /dev/disk/disk3  00001 current  /dev/disk/disk6  00001 current
   00002 /dev/disk/disk3  00002 current  /dev/disk/disk6  00002 current
   00003 /dev/disk/disk3  00003 current  /dev/disk/disk6  00003 current
   00004 /dev/disk/disk3  00004 current  /dev/disk/disk6  00004 current
   00005 /dev/disk/disk3  00005 current  /dev/disk/disk6  00005 current
...

In this example, all lvol1 physical extents on /dev/disk/disk3 have a current copy elsewhere on the system, specifically on /dev/disk/disk6. If /dev/disk/disk3 was unavailable when the volume group was activated, its column contains a ‘???’ instead of the disk name.

Based on the gathered information, choose the appropriate procedure.

Replacing a Mirrored Nonboot Disk

Use this procedure if all the physical extents on the disk have copies on another disk, and the disk is not a boot disk. If the disk contains any unmirrored logical volumes or any mirrored logical volumes without an available and current mirror copy, see “Replacing an Unmirrored Nonboot Disk”.

For this example, the disk to be replaced is at lunpath hardware path 0/1/1/1.0x3.0x0, with device special files named /dev/disk/disk14 and /dev/rdisk/disk14. Follow these steps:

Save the hardware paths to the disk.

Run the ioscan command and note the hardware paths of the failed disk.

# ioscan –m lun /dev/disk/disk14
Class I Lun H/W Path      Driver  S/W State H/W Type Health  Description
========================================================================
disk 14 64000/0xfa00/0x0  esdisk  CLAIMED   DEVICE   offline HP MSA Vol
        0/1/1/1.0x3.0x0
                  /dev/disk/disk14      /dev/rdisk/disk14

In this example, the LUN instance number is 14, the LUN hardware path is 64000/0xfa00/0x0, and the lunpath hardware path is 0/1/1/1.0x3.0x0.

When the failed disk is replaced, a new LUN instance and LUN hardware path are created. To identify the disk after it is replaced, you must use the lunpath hardware path (0/1/1/1.0x3.0x0).

Halt LVM access to the disk.
If the disk is not hot-swappable, power off the system to replace it. By shutting down the system, you halt LVM access to the disk, so you can skip this step.
If the disk is hot-swappable, detach it using the –a option of the pvchange command:
# pvchange -a N /dev/disk/disk14
Replace the disk.
For the hardware details on how to replace the disk, see the hardware administrator's guide for the system or disk array.
If the disk is hot-swappable, replace it.
If the disk is not hot-swappable, shut down the system, turn off the power, and replace the disk. Reboot the system.
Notify the mass storage subsystem that the disk has been replaced.
If the system was not rebooted to replace the failed disk, then run scsimgr before using the new disk as a replacement for the old disk. For example:
# scsimgr replace_wwid –D /dev/rdisk/disk14
This command allows the storage subsystem to replace the old disk’s LUN World-Wide-Identifier (WWID) with the new disk’s LUN WWID. The storage subsystem creates a new LUN instance and new device special files for the replacement disk.

Determine the new LUN instance number for the replacement disk.

For example:

# ioscan –m lun
Class I Lun H/W Path      Driver  S/W State H/W Type Health  Description
========================================================================
disk 14 64000/0xfa00/0x0  esdisk  NO_HW     DEVICE   offline HP MSA Vol
                  /dev/disk/disk14      /dev/rdisk/disk14
...
disk 28 64000/0xfa00/0x1c esdisk  CLAIMED   DEVICE   online  HP MSA Vol
        0/1/1/1.0x3.0x0
                  /dev/disk/disk28      /dev/rdisk/disk28

In this example, LUN instance 28 was created for the new disk, with LUN hardware path 64000/0xfa00/0x1c, device special files /dev/disk/disk28 and /dev/rdisk/disk28, at the same lunpath hardware path as the old disk, 0/1/1/1.0x3.0x0. The old LUN instance 14 for the old disk now has no lunpath associated with it.




	NOTE: If the system was rebooted to replace the failed disk, then ioscan –m lun does not display the old disk.

Assign the old instance number to the replacement disk.

For example:

# io_redirect_dsf -d /dev/disk/disk14 -n /dev/disk/disk28

This assigns the old LUN instance number (14) to the replacement disk. In addition, the device special files for the new disk are renamed to be consistent with the old LUN instance number. The following ioscan –m lun output shows the result:

# ioscan –m lun /dev/disk/disk14
Class I Lun H/W Path      Driver  S/W State H/W Type Health  Description
========================================================================
disk 14 64000/0xfa00/0x1c esdisk  CLAIMED   DEVICE   online  HP MSA Vol
        0/1/1/1.0x3.0x0
                  /dev/disk/disk14      /dev/rdisk/disk14

The LUN representation of the old disk with LUN hardware path 64000/0xfa00/0x0 was removed. The LUN representation of the new disk with LUN hardware path 64000/0xfa00/0x1c was reassigned from LUN instance 28 to LUN instance 14 and its device special files were renamed as /dev/disk/disk14 and /dev/rdisk/disk14.

Restore LVM configuration information to the new disk.
For example:
# vgcfgrestore -n /dev/vgnn /dev/rdisk/disk14

Restore LVM access to the disk.

If you did not reboot the system in Step 2, reattach the disk as follows:

# pvchange –a y /dev/disk/disk14

If you did reboot the system, reattach the disk by reactivating the volume group as follows:

# vgchange -a y /dev/vgnn




	NOTE: The vgchange command with the `-a y` option can be run on a volume group that is deactivated or already activated. It attaches all paths for all disks in the volume group and resumes automatically recovering any disks in the volume group that had been offline or any disks in the volume group that were replaced. Therefore, run vgchange only after all work has been completed on all disks and paths in the volume group, and it is necessary to attach them all.

Because all the data on the replaced disk was mirrored, you do not need to do anything else; LVM automatically synchronizes the data on the disk with the other mirror copies of the data.

Replacing an Unmirrored Nonboot Disk

Use this procedure if any of the physical extents on the disk do not have mirror copies elsewhere, and your disk is not a boot disk.

In this example, the disk to be replaced is at lunpath hardware path 0/1/1/1.0x3.0x0, with device special files named /dev/disk/disk14 and /dev/rdisk/disk14. Follow these steps:

Save the hardware paths to the disk.

Run the ioscan command and note the hardware paths of the failed disk.

# ioscan –m lun /dev/disk/disk14
Class I Lun H/W Path      Driver  S/W State H/W Type Health  Description
========================================================================
disk 14 64000/0xfa00/0x0  esdisk  CLAIMED   DEVICE   offline HP MSA Vol
        0/1/1/1.0x3.0x0
                  /dev/disk/disk14      /dev/rdisk/disk14

In this example, the LUN instance number is 14, the LUN hardware path is 64000/0xfa00/0x0, and the lunpath hardware path is 0/1/1/1.0x3.0x0.

When the failed disk is replaced, a new LUN instance and LUN hardware path are created. To identify the disk after it is replaced, you must use the lunpath hardware path (0/1/1/1.0x3.0x0).

Halt LVM access to the disk.
If the disk is not hot-swappable, power off the system to replace it. By shutting down the system, you halt LVM access to the disk, so you can skip this step.
If the disk is hot-swappable, disable user and LVM access to all unmirrored logical volumes.
First, disable user access to all unmirrored logical volumes. Halt any applications and unmount any file systems using these logical volumes. This prevents the applications or file systems from writing inconsistent data over the newly restored replacement disk.
For each unmirrored logical volume using the disk:
1. Use the fuser command to make sure no one is accessing the logical volume, either as a raw device or as a file system. If users have files open in the file system or it is their current working directory, fuser reports their process IDs.
  For example, if the logical volume was /dev/vg01/lvol1, enter the following command:
  # fuser -cu dev/vg01/lvol1 /dev/vg01/lvol1: 27815c(root) 27184c(root)
2. If fuser reports process IDs using the logical volume, use the ps command to map the list of process IDs to processes, and then determine whether you can halt those processes.
  For example, look up processes 27815 and 27184 as follows:
  # ps -fp27815 -p27184 UID PID PPID C STIME TTY TIME COMMAND root 27815 27184 0 09:04:05 pts/0 0:00 vi test.c root 27184 27182 0 08:26:24 pts/0 0:00 -sh
3. If so, use fuser with the –k option to kill all processes accessing the logical volume.
  The example processes are noncritical, so kill them as follows:
  # fuser -ku dev/vg01/lvol1 /dev/vg01/lvol1: 27815c(root) 27184c(root)
4. If the logical volume is being used as a file system, unmount it as follows:
  # umount /dev/vg01/lvol1
NOTE: If you cannot stop the applications using the logical volume, or you cannot unmount the file system, you must shut down the system.
After disabling user access to the unmirrored logical volumes, disable LVM access to the disk:
# pvchange -a N /dev/disk/disk14
Replace the disk.
For the hardware details on how to replace the disk, see the hardware administrator’s guide for the system or disk array.
If the disk is hot-swappable, replace it.
If the disk is not hot-swappable, shut down the system, turn off the power, and replace the disk. Reboot the system.
Notify the mass storage subsystem that the disk has been replaced.
If the system was not rebooted to replace the failed disk, then run scsimgr before using the new disk as a replacement for the old disk. For example:
# scsimgr replace_wwid –D /dev/rdisk/disk14
This command allows the storage subsystem to replace the old disk’s LUN World-Wide-Identifier (WWID) with the new disk’s LUN WWID. The storage subsystem creates a new LUN instance and new device special files for the replacement disk.

Determine the new LUN instance number for the replacement disk.

For example:

# ioscan –m lun
Class I Lun H/W Path      Driver  S/W State H/W Type Health  Description
========================================================================
disk 14 64000/0xfa00/0x0  esdisk  NO_HW     DEVICE   offline HP MSA Vol
                  /dev/disk/disk14      /dev/rdisk/disk14
...
disk 28 64000/0xfa00/0x1c esdisk  CLAIMED   DEVICE   online  HP MSA Vol
        0/1/1/1.0x3.0x0
                  /dev/disk/disk28      /dev/rdisk/disk28




	NOTE: If the system was rebooted to replace the failed disk, then ioscan –m lun does not display the old disk.

Assign the old instance number to the replacement disk.

For example:

# io_redirect_dsf -d /dev/disk/disk14 -n /dev/disk/disk28

# ioscan –m lun /dev/disk/disk14
Class I Lun H/W Path      Driver  S/W State H/W Type Health  Description
========================================================================
disk 14 64000/0xfa00/0x1c esdisk  CLAIMED   DEVICE   online  HP MSA Vol
        0/1/1/1.0x3.0x0
                  /dev/disk/disk14      /dev/rdisk/disk14

Restore LVM configuration information to the new disk.
For example:
# vgcfgrestore -n /dev/vgnn /dev/rdisk/disk14

Restore LVM access to the disk.

If you did not reboot the system in Step 2, reattach the disk as follows:

# pvchange –a y /dev/disk/disk14

If you did reboot the system, reattach the disk by reactivating the volume group as follows:

# vgchange -a y /dev/vgnn




	NOTE: The vgchange command with the `-a y` option can be run on a volume group that is deactivated or already activated. It attaches all paths for all disks in the volume group and resumes automatically recovering any disks in the volume group that had been offline or any disks in the volume group that were replaced. Therefore, run vgchange only after all work has been completed on all disks and paths in the volume group, and it is necessary to attach them all.

Recover any lost data.

LVM recovers all the mirrored logical volumes on the disk, and starts that recovery when the volume group is activated.

For all the unmirrored logical volumes that you identified in Step 2, restore the data from backup and reenable user access as follows:

For raw volumes, restore the full raw volume using the utility that was used to create your backup. Then restart the application.

For file systems, you must re-create the file systems first. For example:

# newfs -F fstype /dev/vgnn/rlvolnn

Use the logical volume's character device file for the newfs command. For file systems that had nondefault configurations, see newfs(1M) for the correct options.

After creating the file system, mount it under the mount point that it previously occupied. Then restore the data for that file system from your full backups.




	TIP: To make the file system re-creation step easier, record how they were originally created. You can change other file system parameters, such as those used to tune file system performance. The file system must be at least as large as before the disk failure.

Replacing a Mirrored Boot Disk

There are two additional operations you must perform when replacing a mirrored boot disk:

You must initialize boot information on the replacement disk.
If the replacement requires rebooting the system, and the primary boot disk is being replaced, you must boot from the alternate boot disk.

In this example, the disk to be replaced is at lunpath hardware path 0/1/1/1.0x3.0x0, with device special files named /dev/disk/disk14 and /dev/rdisk/disk14. The system is an HP Integrity server, so the physical volume names must specify the HP-UX partition on the boot disk (/dev/disk/disk14_p2 and /dev/disk/disk14_p2).

Save the hardware paths to the disk.

Run the ioscan command and note the hardware paths of the failed disk as follows:

# ioscan –m lun /dev/disk/disk14
Class I Lun H/W Path      Driver  S/W State H/W Type Health  Description
========================================================================
disk 14 64000/0xfa00/0x0  esdisk  CLAIMED   DEVICE   offline HP MSA Vol
        0/1/1/1.0x3.0x0
                  /dev/disk/disk14      /dev/rdisk/disk14
                  /dev/disk/disk14_p1   /dev/rdisk/disk14_p1
                  /dev/disk/disk14_p2   /dev/rdisk/disk14_p2
                  /dev/disk/disk14_p3   /dev/rdisk/disk14_p3

In this example, the LUN instance number is 14, the LUN hardware path is 64000/0xfa00/0x0, and the lunpath hardware path is 0/1/1/1.0x3.0x0.

When the failed disk is replaced, a new LUN instance and LUN hardware path are created. To identify the disk after it is replaced, you must use the lunpath hardware path (0/1/1/1.0x3.0x0).

Halt LVM access to the disk.
If the disk is not hot-swappable, power off the system to replace it. By shutting down the system, you halt LVM access to the disk, so you can skip this step.
If the disk is hot-swappable, detach the device using the –a option of the pvchange command:
# pvchange -a N /dev/disk/disk14_p2
NOTE: On an HP 9000 server, the boot disk is not partitioned so the physical volume refers to the entire disk, not the HP-UX partition. Use the following command:
# pvchange -a N /dev/disk/disk14
Replace the disk.
For the hardware details on how to replace the disk, see the hardware administrator’s guide for the system or disk array.
If the disk is hot-swappable, replace it.
If the disk is not hot-swappable, shut down the system, turn off the power, and replace the disk. Reboot the system. Two problems can occur:
- If you replaced the disk that you normally boot from, the replacement disk does not contain the information needed by the boot loader. In this case, interrupt the boot process and boot from the mirror boot disk, which is configured as the alternate boot path.
- If there are only two disks in the root volume group, the system probably fails its quorum check as described in “Volume Group Activation Failures”. It can panic early in the boot process with the message:
  panic: LVM: Configuration failure
  In this situation, you must override quorum to boot successfully. Do this by interrupting the boot process and adding the –lq option to the boot command.
For information on the boot process and how to select boot options, see HP-UX System Administrator's Guide: Configuration Management.
Notify the mass storage subsystem that the disk has been replaced.
If the system was not rebooted to replace the failed disk, then run scsimgr before using the new disk as a replacement for the old disk. For example:
# scsimgr replace_wwid –D /dev/rdisk/disk14
This command allows the storage subsystem to replace the old disk’s LUN World-Wide-Identifier (WWID) with the new disk’s LUN WWID. The storage subsystem creates a new LUN instance and new device special files for the replacement disk.

Determine the new LUN instance number for the replacement disk.

For example:

# ioscan –m lun
Class I Lun H/W Path      Driver  S/W State H/W Type Health  Description
========================================================================
disk 14 64000/0xfa00/0x0  esdisk  NO_HW     DEVICE   offline HP MSA Vol
                  /dev/disk/disk14      /dev/rdisk/disk14
                  /dev/disk/disk14_p1   /dev/rdisk/disk14_p1
                  /dev/disk/disk14_p2   /dev/rdisk/disk14_p2
                  /dev/disk/disk14_p3   /dev/rdisk/disk14_p3
...
disk 28 64000/0xfa00/0x1c esdisk  CLAIMED   DEVICE   online  HP MSA Vol
        0/1/1/1.0x3.0x0
                  /dev/disk/disk28      /dev/rdisk/disk28




	NOTE: If the system was rebooted to replace the failed disk, then ioscan –m lun does not display the old disk.

(HP Integrity servers only) Partition the replacement disk.
Partition the disk using the idisk command and a partition description file, and create the partition device files using insf, as described in “Mirroring the Boot Disk on HP Integrity Servers”.

Assign the old instance number to the replacement disk.

For example:

# io_redirect_dsf -d /dev/disk/disk14 -n /dev/disk/disk28

# ioscan –m lun /dev/disk/disk14
Class I Lun H/W Path      Driver  S/W State H/W Type Health  Description
========================================================================
disk 14 64000/0xfa00/0x1c esdisk  CLAIMED   DEVICE   online  HP MSA Vol
        0/1/1/1.0x3.0x0
                  /dev/disk/disk14      /dev/rdisk/disk14
                  /dev/disk/disk14_p1   /dev/rdisk/disk14_p1
                  /dev/disk/disk14_p2   /dev/rdisk/disk14_p2
                  /dev/disk/disk14_p3   /dev/rdisk/disk14_p3

Restore LVM configuration information to the new disk.

For example:

# vgcfgrestore -n /dev/vg00 /dev/rdisk/disk14_p2

NOTE: On an HP 9000 server, the boot disk is not partitioned, so the physical volume refers to the entire disk, not the HP-UX partition. Use the following command:

# vgcfgrestore -n /dev/vg00 /dev/rdisk/disk14

Restore LVM access to the disk.

If you did not reboot the system in Step 2, reattach the disk as follows:

# pvchange –a y /dev/disk/disk14_p2

On an HP 9000 server, use this command:

# pvchange –a y /dev/disk/disk14

If you did reboot the system, reattach the disk by reactivating the volume group as follows:

# vgchange -a y /dev/vg00




	NOTE: The vgchange command with the `-a y` option can be run on a volume group that is deactivated or already activated. It attaches all paths for all disks in the volume group and resumes automatically recovering any disks in the volume group that had been offline or any disks in the volume group that were replaced. Therefore, run vgchange only after all work has been completed on all disks and paths in the volume group, and it is necessary to attach them all.

Initialize boot information on the disk.
For an HP Integrity server, set up the boot area and update the autoboot file in the disk's EFI partition as described in step 5 and step 6 of “Mirroring the Boot Disk on HP Integrity Servers”.
For an HP 9000 server, set up the boot area and update the autoboot file as described in step 4 and step 5 of “Mirroring the Boot Disk on HP 9000 Servers”.

Replacing an Unmirrored Boot Disk

With the failure of an unmirrored boot disk, you have lost the only copy of information that is required to boot the system. You must reinstall to the replacement disk, or recover it from an Ignite-UX backup.

Replacing a Bad Disk

Technical documentation

» Table of Contents

» Glossary

» Index

Disk Replacement Prerequisites

Replacing a Mirrored Nonboot Disk

Replacing an Unmirrored Nonboot Disk

Replacing a Mirrored Boot Disk

Replacing an Unmirrored Boot Disk