Log Consolidation Configuration

To start the log consolidation wizard, issue the following command:

/opt/dsau/sbin/clog_wizard

For a standalone system, the wizard first displays introductory paragraphs explaining log consolidation and then asks:

Answer yes (y) or press Enter. The next question is:

You can configure this system hostname as either a:
        - Consolidation server
        - Client that forwards logs to a remote consolidation server
Do you want to configure hostname as a Consolidation
Server? (y/n) [y]:

Answer yes (y). The wizard then prompts:

It is typically best to select a dedicated filesystem for the consolidated logs. Since consolidated logs like syslog can grow rapidly, HP also recommends that the filesystem be configured for “largefiles.” For this example, a filesystem named “/clog” is used.

Next, the wizard prompts for the client’s transport:

Selecting TCP does not necessarily preclude the use of UDP forwarded log messages by clients. Whether the log consolidator allows TCP log messages exclusively, depends on whether the system is consolidating its own local syslog file. See below for details.

There is no reserved port for the TCP transport of syslog-ng. Any port that is not in use can be chosen. HP recommends that the administrator choose a port from the reserved range, that is, ports below 1024. Only privileged processes on a remote system can connect to privileged ports. This provides only a weak security guarantee because it implies that the administrator trusts the remote system. See the ssh section in the log forwarding client section for establishing stronger security guarantees “Manually Configuring a Standalone Log Forwarding Client”.

The /etc/services file documents the well-known reserved ports. When choosing a reserved port, the wizard will check both /etc/services and use “netstat -an” to check that the port is not in use.

Note that syslogd uses UDP port 514. TCP port 514 is reserved for use by remsh. remsh is not a secure protocol and is disabled at many sites. If remsh has been disabled on the consolidator, you could use TCP port 514. This has the advantage that it is a privileged port and it is the same as the UDP port number so it is easy to remember and manage. However, if the administrator changes the system to re-enable the use of remsh, syslog-ng would have to be reconfigured to use a new port and all the client systems that forward to this system would have to be updated.

Unlike UDP, TCP is a connection-oriented protocol. Each log forwarding client using TCP will have a connection to the log consolidation server. In order to avoid denial of service attacks, the default number of TCP connections accepted by syslog-ng is limited to 10 connections. For larger numbers of clients, edit the consolidation server’s /etc/syslog-ng.conf.server file. Find the TCP source line in the file:

source s_syslog_tcp { tcp(port(tcp_port) keep-alive(yes));};

and add a max-connections() attribute as follows:

source s_syslog_tcp { tcp(port(tcp_port) keep-alive(yes) max-connections(N)); };

where N is the expected number of clients.

Next, the wizard prompts for which local logs should be consolidated:

Log files that reside on this system can be consolidated.

Would you like to consolidate this system's syslogs? (y/n) [y]:

Answering yes places this log consolidation system’s own local syslog data in the consolidated log along with the client system's syslog data. To preserve the priority and facility of syslog entries, UDP local loopback is used, and syslog is configured to also forward entries to its local UDP port 514. syslog-ng is configured to read from this port. Thus, consolidating this system’s syslogs allows clients to also connect to this log consolidation server via UDP port 514, even if TCP transport is specified earlier. If you choose not to consolidate this system’s syslogs, then choosing a TCP transport earlier will require that all log forwarding clients be configured to use the TCP transport. The wizard displays a summary of all the configuration choices made by the administrator:

If these choices are correct, continue:

The wizard displays its progress by describing which files are being modified and warns that Ctrl/C is disabled until configuration is done. For a complete description of the modified files, refer to “Manually Configuring Log Consolidation”.

When running the clog_wizard (see clog_wizard(1M)) in a cluster, first make sure that all the cluster members are up and available. The wizard needs to perform configuration operations on each member. It only needs to be run once, from any member of the cluster. If you run the wizard more than once, additional prompts may appear.

The wizard will set up and create a Serviceguard package for the consolidated logging service. Make sure that this cluster’s MAX_CONFIGURED_PACKAGES value can accommodate the additional package. For more information on this setting, please refer to the Managing Serviceguard manual which is part of the Serviceguard documentation set.

The wizard first displays introductory paragraphs explaining log consolidation (wizard output may wrap differently from what is shown here):

The next prompt is:

Answer yes (y).

In a cluster, the wizard configures syslog-ng to be highly available using a Serviceguard package. For consolidated logging, the package name is clog. The LVM storage configuration and network configuration for the package must be set up before continuing or before running the wizard. For additional details, refer to the section “Creating a Storage Infrastructure with LVM” in the chapter “Building an HA Cluster Configuration,” in the Managing Serviceguard manual.

	NOTE: The wizard only supports creating packages based on LVM volume groups. When using CFS or VxVM, manual configuration is required. See the section “Manually Configuring Log Consolidation” for details.

The wizard prompts for the following, all of which should have already been configured:

Next, the wizard prompts for the clients' transport.

Note that selecting TCP does not necessarily preclude the use of UDP forwarded log messages by clients. Whether the log consolidator allows TCP log messages exclusively depends on whether the system is consolidating its own local syslog file. See below for details.

When answering Yes to TCP, you must select a free TCP port. This port must be free on all cluster members. See the section “Configuring a Log Forwarding Client Using clog_wizard” using the clog_wizard for details on choosing a TCP port.

Next the wizard prompts for which local logs should be consolidated:

In a Serviceguard cluster, you can consolidate all the member-specific syslog files into a single consolidated syslog containing syslog.log, mail.log and syslog-ng.log. Each member-specific package log can also be consolidated.

Note that if you choose to consolidate the cluster’s syslogs, the remote clients can also forward UDP syslog messages to the cluster, regardless of the answer to the “Do you want to use the TCP protocol” question. If you choose not to consolidate this cluster’s syslogs, then choosing a TCP transport earlier requires that all log forwarding clients be configured to use the TCP transport.

The consolidated logs are placed in the filesystem associated with the package. Using the example above, the consolidated syslog.log would be located here:

/clog/syslog/syslog.log,mail.log,syslog-ng.log

The consolidated package logs would be located here:

/clog/package/package1.log,package2.log, ...,packageN.log

The wizard now has all the data it needs to configure the consolidated logging package. It displays a summary confirmation screen and then performs the configuration:

In a Serviceguard cluster, the adoptive node for the clog package performs the log consolidation functions. All the other cluster members participate as log forwarding clients and send log messages to the relocatable IP address of the clog package.

DSAU maintains two configuration files that control whether the instance of syslog-ng on a particular cluster member operates as a consolidation server or a log forwarding client: /etc/syslog-ng.conf.server and /etc/syslog-ng.conf.client. The symbolic link /etc/syslog-ng.conf points to one of the configuration files. When the cluster is booted, all the members start as log forwarding clients with syslog-ng running on each member. The /sbin/init.d/syslog-ng startup script sets the symbolic link /etc/syslog-ng.conf to point to /etc/syslog-ng.conf.client.

When the clog package starts, the adoptive node restarts that instance of syslog-ng as a log consolidation server instance. The package script resets the /etc/syslog-ng.conf symbolic link to point to /etc/syslog-ng.conf.server. If the clog package is halted, the symlink is changed to point to /etc/syslog-ng.conf.client and the syslog-ng instance on that member restarted. Note that when a cluster is a log consolidation server, and the package is down, no log consolidation occurs and forwarded log messages are lost.

Cluster members can forward log messages to the consolidator using either UDP or TCP. Within a Serviceguard cluster, ssh port forwarding is not used. ssh port forwarding can be used to secure the log traffic forwarded by remote clients outside the cluster. For additional information, refer to “ssh Port Forwarding”.

The Distributed Systems Administration Utilities require Serviceguard 11.17 or later. With Serviceguard 11.17 or later, when members are added to or deleted from cluster or packages are added and deleted, the DSAU consolidated logging tools will automatically take the appropriate configuration actions. Specifically:

When there is a failure on the adoptive node, it takes a finite amount of time for the clog package to fail over to another cluster member. The longer this failover time, the more likely that messages could be lost from the consolidated log. Use the following guidelines to minimize message loss during failover.

There are two ways to configure a log forwarding client: as a standalone machine or as a Serviceguard cluster. When configuring a cluster as a log forwarding client, all the members of the cluster will be configured identically as clients. The wizard asks the same questions and performs the same configuration actions for single systems and for clusters. The examples below show use of the clog wizard on a Serviceguard cluster. After starting clog_wizard, answer “yes” to the following question:

Do you want to configure log consolidation? (y/n) [y]:

or press Enter. The next question is:

Answer “No” here. At this point you are configuring a log forwarding client. The wizard displays the following:

After entering the hostname or IP address of the log consolidation server, the wizard asks if you want to use the TCP transport when forwarding log messages:

Standard syslogd forwards messages using the UDP protocol. UDP is a high-performance, broadcast-oriented protocol with no flow control or message delivery verification. syslog-ng supports syslogd’s UDP protocol and a TCP protocol. The TCP transport offers both flow control and message delivery checks. However, since TCP is a connection-oriented protocol, it requires additional resources on the log consolidation server. The consolidation server’s max-connections() attribute must be set according to the maximum number of expected clients. Refer to the section “Configuring a Log Consolidation Standalone Server with clog_wizard” for a discussion of the max-connections() setting.

If you answer “yes” to using TCP, the next question asks for the TCP port to forward messages to:

You must use the TCP port selected by the system administrator of the log consolidation server. If the clog_wizard was used to configure the server, the port number is saved in /etc/rc.config.d/syslog-ng as the variable CLOG_TCP_PORT. In this example, TCP port 1776 was used. If you answer “yes” to the TCP question, the following question is displayed:

Choose yes in order to use ssh port forwarding. This will encrypt all the traffic sent from this local log forwarding client to the log consolidator.

	NOTE: A special ssh security configuration is required on the server when a Serviceguard cluster is the log consolidation server. For details, refer to “ssh Port Forwarding”.

ssh port forwarding requires an additional free TCP port on the local client system:

You need to choose a free port on this cluster for ssh port forwarding. The port chosen should be free on all cluster nodes.

Enter the ssh port to be used for port forwarding []: 1775

The same guidelines for choosing a free syslog-ng TCP port apply to this port. For details, refer to “Configuring a Log Consolidation Standalone Server with clog_wizard”. In this example, the local port 1775 was used. For a Serviceguard cluster log forwarding client, the cluster’s syslogs and package logs can be forwarded to the log consolidation server. For a standalone system, the wizard asks only about forwarding syslog messages:

When forwarding a cluster’s package logs, manual configuration is required on the consolidation server in order to add the syslog-ng filtering lines to cause these package logs to be consolidated into their own unique files. See “Manually Configuring a Serviceguard Cluster as a Log Forwarding Client” for details.

After all the questions have been answered, the clog_wizard displays the following summary screen:

Confirm your answers with a “yes” response and the wizard summarizes the configuration steps that it performs:

For additional information on the configuration actions performed by the clog_wizard, refer to “Manually Configuring a Serviceguard Cluster as a Log Consolidation Server”.

Start by configuring the standard syslogd to co-exist with a syslog-ng consolidator. By default, syslogd listens for incoming log messages on UDP port 514. If you want to accept UDP syslog messages from remote clients or consolidate this server’s local syslogs, syslog-ng must listen on UDP port 514. Edit /etc/rc.config.d/syslogd and change SYSLOGD_OPTS to add the -N switch, which prevents syslogd from listening on port 514. For example:

SYSLOGD_OPTS=“-D -N”

If you want the local syslog messages from the log consolidation server itself to be part of the consolidated syslog, edit the consolidator’s /etc/syslog.conf file to forward log messages to port 514 on the local host where they will be read by syslog-ng. Using the HP-UX default /etc/syslog.conf as an example, add the following lines:

where log-consolidation-server is the fully qualified domain name of the consolidation server. The name must be fully qualified or syslogd will not forward the messages properly.

	NOTE: There must be a <tab> before each @ sign.

If you have customized syslog.conf, make sure to add the forwarding lines for your customizations as well.

syslogd must be stopped and restarted for these changes to take effect, using the following commands:

# /sbin/init.d/syslogd stop

# /sbin/init.d/syslogd start

With syslogd appropriately configured, now configure syslog-ng.

Start with the same syslog-ng.conf templates used by the clog_wizard. Copy /opt/dsau/share/clog/templates/syslog-ng.conf.server.template to /etc/syslog-ng.conf.server. This file has tokens named <%token-name%> that are replaced by the wizard based on the administrator’s answers to the wizard’s questions.

Replace the tokens as follows:

Configuring a Serviceguard cluster as a log consolidation server is similar to the steps for a single system. All cluster members must be up and accessible before proceeding.

Create the configuration files described below on every cluster member. The simplest approach is to configure one member completely and then copy each configuration file cluster-wide. The cexec and ccp tools can simplify replicating changes cluster-wide.

For a cluster configuration, syslog-ng is configured as a package so the log consolidation service is highly available. The package must be named clog and the package configuration files require the following information:

Complete IP address registration and storage/filesystem configuration before continuing. For additional information on creating the Serviceguard storage/filesystem configuration for a package, refer to the Managing Serviceguard manual.

For an overview of how to configure consolidated logging in a cluster, see the section “Cluster Configuration Notes for clog”.

Creating the clog Package

To create the consolidated logging or clog package, start by copying the package templates:

# mkdir /etc/cmcluster/clog # cd /etc/cmcluster/clog # cp /opt/dsau/share/serviceguard/templates/clog.conf.template clog.conf # cp /opt/dsau/share/serviceguard/templates/clog.script.template clog # chmod +x clog

Both the clog.conf package configuration file and the clog package control script need to be edited to replace marker tokens with the correct values.

For clog.conf, there is only one token to replace, <%SG_PKG_SUBNET%>. This identifies the package’s subnet. It is the same as the subnet value in the package control script. Use netstat -i to help identify the proper subnet for the package’s IP address.

For the package control script, clog, make the changes described below.

The default script template assumes you are using an LVM-based storage configuration. Replace the volume group/filesystem related tokens as appropriate for the package’s storage configuration as follows:

1. Find the line “VG[0]=“<%SG_PKG_VOL_GRP%>”” and replace the token with the name of the VM volume group for the package. For example:

   VG[0]=“vgclog”

   If using VxVM, comment out the LVM Volume Group line VG[0]=”<%SG_PKG_VOL_GRP%>”. Uncomment the line “VXVM_DG[0]=” and put in the VxVM Disk Group.

2. Find the line “LV[0]=“<%SG_PKG_LOG_VOL%>”” and replace the token with the full name of the logical volume. For example:

   LV[0]=“/dev/vgclog/lvol1”

3. Find the line “FS[0]=“<%SG_PKG_FS%>”” and replace the token with the name of the filesystem created for this package. For example:

   FS[0]=“/clog”

   All the consolidated logs will reside on this filesystem. The specific location for the consolidated package logs and the consolidated syslogs is specified in the /etc/syslog-ng.conf.server file. Using /clog as the example, the default locations based on the template /etc/syslog-ng.conf.server file are:

   /clog/syslog/syslog.log /clog/packages/package name.log

4. Find the line “FS_MOUNT_OPT[0]=“<%SG_PKG_MNT_OPT%>”:” and replace the token with the filesystem’s mount options. For example:

   FS_MOUNT_OPT[0]=-o rw,largefiles

5. Find the line “FS_TYPE[0]=“<%SG_PKG_FS_TYPE%>”” and replace the token with the filesystem type. For example:

   FS_TYPE[0]=vxfs

6. Find the line “FS_UMOUNT_OPT[0]=“<%SG_PKG_FS_UMOUNT_OPT%>”” and replace the token with any filesystem umount options. The token can be removed and this option left blank if there are no special umount options. For example:

   FS_UMOUNT_OPT[0]=“”

7. Find the line “FS_FSCK_OPT[0]=“<%SG_PKG_FS_FSCK_OPT%>”” and replace the token with any filesystem specific fsck options. The token can be deleted and this option left blank. For example:

   FS_FSCK_OPT[0]=

8. Find the line “IP[0]=“<%SG_PKG_IP%>”” and replace the token with the IP address of the clog package. For example:

   IP[0]= 192.119.152.3

9. Find the line “SUBNET[0]=“<%SG_PKG_SUBNET%>”” and replace the token with the subnet for the packages IP address. Use netstat -i to help determine the subnet. For example:

   SUBNET[0]= 192.119.152.0

You next need to distribute the package files cluster-wide. To do this, perform the following steps:

1. First, create the package directory on all the other members:

   # cexec mkdir /etc/cmcluster/clog

2. Copy the package control script and package ASCII configuration file:

   # ccp clog clog.conf /usr/local/cmcluster/conf/clog/

3. Update the /etc/rc.config.d/syslog-ng file, by adding the following lines:

   CLOG_PKG_VOL_GRP=LVM-volume-group CLOG_PKG_LOG_VOL=logical-volume(full path) CLOG_PKG_FS=filesystem mount point where the consolidated logs are stored CLOG_PKG_MNT_OPT=file systems mount options such as -o rw,largefiles CLOG_PKG_FS_TYPE=file system type CLOG_PKG_IP=IP address of the clog package CLOG_PKG_SUBNET=subnet of the clog package’s IP address CLOG_SYSTEM_LOG_CONSOLIDATION_DIR=file system mount point/syslog CLOG_SERVICEGUARD_PACKAGE_LOG_CONSOLIDATION_DIR=file system mount point/packages CLOG_PKG_RETRY_TIMES=1 CLOG_PKG_MONITOR_INTERVAL=5

4. Distribute it cluster-wide:

   # ccp /etc/rc.config.d/syslog-ng /etc/rc.config.d/

You can configure either a standalone system or a Serviceguard cluster as log forwarding clients. You can also manually configure Serviceguard package logs as if they were syslog data. For each case, you set up both syslogd and syslog-ng.

When remote Serviceguard clusters forward package log data to a log consolidation server, the default is to place all forwarded log messages in the consolidated syslog.log file on the consolidation server. It can be much more convenient to place these messages in cluster-specific consolidated package log files instead of in the consolidated syslog.log file. This can be achieved using syslog-ng’s filtering rules as follows: