Preparing Your Systems

To enable a system to be included in a cluster, you must enable HP-UX root access to the system by the root user of every other potential cluster node. The Serviceguard mechanism for doing this is the file $SGCONF/cmclnodelist. This is sometimes referred to as a “bootstrap” file because Serviceguard consults it only when configuring a node into a cluster for the first time; it is ignored after that. It does not exist by default, but you will need to create it.

You may want to add a comment such as the following at the top of the file:

###########################################################

# Do not edit this file!

# Serviceguard uses this file only to authorize access to an

# unconfigured node. Once the node is configured,

# Serviceguard will not consult this file.

###########################################################

The format for entries in cmclnodelist is as follows:

[hostname] [user] [#Comment]

For example:

This example grants root access to the node on which this cmclnodelist file resides to root users on the nodes gryf, sly, and bit.

Serviceguard also accepts the use of a “+” in the cmclnodelist file; this indicates that the root user on any Serviceguard node can configure Serviceguard on this node.

	IMPORTANT: If $SGCONF/cmclnodelist does not exist, Serviceguard will look at ~/.rhosts. HP strongly recommends that you use cmclnodelist.

	NOTE: When you upgrade a cluster from Version A.11.15 or earlier, entries in $SGCONF/cmclnodelist are automatically updated to Access Control Policies in the cluster configuration file. All non-root user-hostname pairs are assigned the role of Monitor.

The HP-UX root user on any cluster node can configure the cluster. This requires that Serviceguard on one node be able to recognize the root user on another.

Serviceguard uses the identd daemon to verify user names, and, in the case of a root user, verification succeeds only if identd returns the username root. Because identd may return the username for the first match on UID 0, you must check /etc/passwd on each node you intend to configure into the cluster, and ensure that the entry for the root user comes before any other entry with a UID of 0.

About identd

HP strongly recommends that you use identd for user verification, so you should make sure that each prospective cluster node is configured to run it. identd is usually started by inetd from /etc/inetd.conf.

	NOTE: If the -T option to identd is available on your system, you should set it to 120 (-T120); this ensures that a connection inadvertently left open will be closed after two minutes. The identd entry in /etc/inetd.conf should look like this: auth stream tcp6 wait bin /usr/lbin/identd identd -T120 Check the man page for identd to determine whether the -T option is supported for your version of identd

(It is possible to disable identd, though HP recommends against doing so. If for some reason you have to disable identd, see “Disabling identd”.)

For more information about identd, see the white paper Securing Serviceguard at http://docs.hp.com -> High Availability -> Serviceguard -> White Papers, and the identd (1M) manpage.

When you employ any user-level Serviceguard command (including cmviewcl), the command uses the name service you have configured (such as DNS) to obtain the addresses of all the cluster nodes. If the name service is not available, the command could hang or return an unexpected networking error message.

	NOTE: If such a hang or error occurs, Serviceguard and all protected applications will continue working even though the command you issued does not. That is, only the Serviceguard configuration commands (and corresponding Serviceguard Manager functions) are affected, not the cluster daemon or package services.

The procedure that follows shows how to create a robust name-resolution configuration that will allow cluster nodes to continue communicating with one another if a name service fails. If a standby LAN is configured, this approach also allows the cluster to continue to function fully (including commands such as cmrunnode and cmruncl) after the primary LAN has failed.

	NOTE: If a NIC fails, the affected node will be able to fail over to a standby LAN so long as the node is running in the cluster. But if a NIC that is used by Serviceguard fails when the affected node is not running in the cluster, Serviceguard will not be able to restart the node. (For instructions on replacing a failed NIC, see “Replacing LAN or Fibre Channel Cards”.)

	NOTE: HP recommends that you also make the name service itself highly available, either by using multiple name servers or by configuring the name service into a Serviceguard package.

After you configure the cluster and create the cluster lock volume group and physical volume, you should create a backup of the volume group configuration data on each lock volume group. Use the vgcfgbackup command for each lock volume group you have configured, and save the backup file in case the lock configuration must be restored to a new disk with the vgcfgrestore command following a disk failure.

	NOTE: You must use the vgcfgbackup and vgcfgrestore commands to back up and restore the lock volume group configuration data regardless of how you create the lock volume group.

You can use the idisk utility to create a partition for a lock LUN in a cluster that will contain only HP Integrity servers. Use the procedure that follows; see the idisk (1m) manpage for more information. Do this on one of the nodes in the cluster that will use this lock LUN.

	CAUTION: Before you start, make sure the disk or LUN that is to be partitioned has no data on it that you need. idisk will destroy any existing data.

Use cmquerycl -L to create a cluster configuration file that defines the lock LUN.

These commands create a configuration file which you can apply to the cluster configuration when you are ready to do so; see “Distributing the Binary Configuration File ”. See also “Specifying a Lock LUN”.

	CAUTION: Once you have specified the lock LUN in the cluster configuration file, running cmapplyconf will destroy any data on the LUN.

The procedure described in this section uses physical volume groups for mirroring of individual disks to ensure that each logical volume is mirrored to a disk on a different I/O bus. This kind of arrangement is known as PVG-strict mirroring. It is assumed that your disk hardware is already configured in such a way that a disk to be used as a mirror copy is connected to each node on a different bus from the bus that is used for the other (primary) copy.

For more information on using LVM, refer to the Logical Volume Management volume of the HP-UX System Administrator’s Guide.

You can use the System Management Homepage to create or extend volume groups and create logical volumes. From the System Management Homepage, choose Disks and File Systems. Make sure you create mirrored logical volumes with PVG-strict allocation.

When you have created the logical volumes and created or extended the volume groups, specify the filesystem that is to be mounted on the volume group, then skip ahead to the section “Deactivating the Volume Group”.

To configure the volume groups from the command line, proceed as follows.

If your volume groups have not been set up, use the procedures that follow. If you have already done LVM configuration, skip ahead to the section “Configuring the Cluster.”

Obtain a list of the disks on both nodes and identify which device files are used for the same disk on both. Use the following command on each node to list available disks as they are known to each system:

In the following examples, we use /dev/rdsk/c1t2d0 and /dev/rdsk/c0t2d0, which happen to be the device names for the same disks on both ftsys9 and ftsys10. In the event that the device file names are different on the different nodes, make a careful note of the correspondences.

	NOTE: Under agile addressing, the physical devices in these examples would have names such as /dev/rdisk/disk1 and /dev/rdisk/disk2. See “About Device File Names (Device Special Files)”.

On the configuration node (ftsys9), use the pvcreate command to define disks as physical volumes. This only needs to be done on the configuration node. Use the following commands to create two physical volumes for the sample configuration:

Using PV Strict Mirroring Use the following steps to build a volume group on the configuration node (ftsys9). Later, the same volume group will be created on other nodes.

Creating Logical Volumes Use the following command to create logical volumes (the example is for /dev/vgdatabase):

This command creates a 120 MB mirrored volume named lvol1. The name is supplied by default, since no name is specified in the command. The -s g option means that mirroring is PVG-strict, that is, the mirror copies of data will be in different physical volume groups.

	NOTE: If you are using disk arrays in RAID 1 or RAID 5 mode, omit the -m 1 and -s g options.

Creating File Systems If your installation uses filesystems, create them next. Use the following commands to create a filesystem for mounting on the logical volume just created:

Distributing Volume Groups to Other Nodes After creating volume groups for cluster data, you must make them available to any cluster node that will need to activate the volume group. The cluster lock volume group must be made available to all nodes.

Deactivating the Volume Group At the time you create the volume group, it is active on the configuration node (ftsys9, for example).The next step is to umount the file system and deactivate the volume group; for example, on ftsys9:

	NOTE: Do this during this set-up process only, so that activation and mounting can be done by the package control script at run time. You do not need to deactivate and unmount a volume simply in order to create a map file (as in step 1 of the procedure that follows).

Distributing the Volume Group Use the following commands to set up the same volume group on another cluster node. In this example, the commands set up a new volume group on ftsys10 which will hold the same physical volume that was available on ftsys9. You must carry out the same procedure separately for each node on which the volume group's package can run.

To set up the volume group on ftsys10, use the following steps:

Making Physical Volume Group Files Consistent Skip ahead to the next section if you do not use physical volume groups for mirrored individual disks in your disk configuration.

Different volume groups may be activated by different subsets of nodes within a Serviceguard cluster. In addition, the physical volume name for any given disk may be different on one node from what it is on another. For these reasons, you must carefully merge the /etc/lvmpvg files on all nodes so that each node has a complete and consistent view of all cluster-aware disks as well as of its own private (non-cluster-aware) disks. To make merging the files easier, be sure to keep a careful record of the physical volume group names on the volume group planning worksheet (described in Chapter 4 “Planning and Documenting an HA Cluster ”).

Use the following procedure to merge files between the configuration node (ftsys9) and a new node (ftsys10) to which you are importing volume groups:

Creating Additional Volume Groups The foregoing sections show in general how to create volume groups and logical volumes for use with Serviceguard. Repeat the procedure for as many volume groups as you need to create, substituting other volume group names, logical volume names, and physical volume names. Pay close attention to the disk device names, which can vary from one node to another.

	NOTE: Check the Serviceguard, SGeRAC, and SMS Compatibility and Feature Matrix and the latest Release Notes for your version of Serviceguard for up-to-date information about support for CVM (and CFS - Cluster File System): http://www.docs.hp.com -> High Availability -> Serviceguard).

If you are using CVM 3.5 and you are about to create disk groups for the first time, you need to initialize the Volume Manager. This is done by creating a disk group known as rootdg that contains at least one disk. Use the following command once only, immediately after installing VxVM on each node:

vxinstall

This displays a menu-driven program that steps you through the VxVM initialization sequence. From the main menu, choose the “Custom” option, and specify the disk you wish to include in rootdg.

	IMPORTANT: The rootdg for the Veritas Cluster Volume Manager 3.5 is not the same as the HP-UX root disk if an LVM volume group is used for the HP-UX root disk filesystem. Note also that rootdg cannot be used for shared storage. However, rootdg can be used for other local filesystems (e.g., /export/home), so it need not be wasted. (CVM 4.1 and later do not require you to create rootdg.) Note that you should create a rootdg disk group only once on each node.

You can use the vxvmconvert(1m) utility to convert LVM volume groups into VxVM disk groups. Before you can do this, the volume group must be deactivated, which means that any package that uses the volume group must be halted. Follow the conversion procedures outlined in the Veritas Volume Manager Migration Guide for your version of VxVM. Before you start, be sure to create a backup of each volume group’s configuration with the vgcfgbackup command, and make a backup of the data in the volume group. Appendix G “Migrating from LVM to VxVM Data Storage ” for more information about conversion.

You need to initialize the physical disks that will be employed in VxVM disk groups. To initialize a disk, log on to one node in the cluster, then use the vxdiskadm program to initialize multiple disks, or use the vxdisksetup command to initialize one disk at a time, as in the following example:

/usr/lib/vxvm/bin/vxdisksetup -i c0t3d2

If a physical disk has been previously used with LVM, you should use the pvremove command to delete the LVM header data from all the disks in the volume group. In addition, if the LVM disk was previously used in a cluster, you have to re-initialize the disk with the pvcreate -f command to remove the cluster ID from the disk.

	NOTE: These commands make the disk and its data unusable by LVM, and allow it to be initialized by VxVM. (The commands should only be used if you have previously used the disk with LVM and do not want to save the data on it.)

You can remove LVM header data from the disk as in the following example (note that all data on the disk will be erased): pvremove /dev/rdsk/c0t3d2

Then, use the vxdiskadm program to initialize multiple disks for VxVM, or use the vxdisksetup command to initialize one disk at a time, as in the following example:

/usr/lib/vxvm/bin/vxdisksetup -i c0t3d2

Use vxdiskadm, or use the vxdg command, to create disk groups, as in the following example:

vxdg init logdata c0t3d2

Verify the configuration with the following command:

vxdg list

Use the vxassist command to create logical volumes. The following is an example:

vxassist -g logdata make log_files 1024m

This command creates a 1024 MB volume named log_files in a disk group named logdata. The volume can be referenced with the block device file /dev/vx/dsk/logdata/log_files or the raw (character) device file /dev/vx/rdsk/logdata/log_files. Verify the configuration with the following command:

vxprint -g logdata

The output of this command is shown in the following example:

	NOTE: The specific commands for creating mirrored and multi-path storage using VxVM are described in the Veritas Volume Manager Reference Guide.

If your installation uses filesystems, create them next. Use the following commands to create a filesystem for mounting on the logical volume just created:

After creating the disk groups that are to be used by Serviceguard packages, use the following command with each disk group to allow the disk group to be deported by the package control script on other cluster nodes:

vxdg deport <DiskGroupName>

where <DiskGroupName> is the name of the disk group that will be activated by the control script.

When all disk groups have been deported, you must issue the following command on all cluster nodes to allow them to access the disk groups:

vxdctl enable

After deporting disk groups, they are not available for use on the node until they are imported again either by a package control script or with a vxdg import command. If you need to import a disk group manually for maintenance or other purposes, you import it, start up all its logical volumes, and mount filesystems as in the following example:

vxdg import dg_01

vxvol -g dg_01 startall

mount /dev/vx/dsk/dg_01/myvol /mountpoint

	NOTE: Unlike LVM volume groups, VxVM disk groups are not entered in the cluster configuration file, nor in the package configuration file.

At system reboot time, the cmcluster RC script does a vxdisk clearimport on all disks formerly imported by the system, provided they have the noautoimport flag set, and provided they are not currently imported by another running node. The clearimport clears the host ID on the disk group, to allow any node that is connected to the disk group to import it when the package moves from one node to another.

Using the clearimport at reboot time allows Serviceguard to clean up following a node failure, for example, a system crash during a power failure. Disks that were imported at the time of the failure still have the node’s ID written on them, and this ID must be cleared before the rebooting node or any other node can import them with a package control script.

Note that the clearimport is done for disks previously imported with noautoimport set on any system that has Serviceguard installed, whether it is configured in a cluster or not.