Securing Consolidated Logs

On a standard HP-UX system, all users can view the system’s local /var/adm/syslog/syslog.log. Access to consolidated logs is typically restricted. The log consolidation server system itself is usually a restricted access system with strict security policies in place.

Log File Protections

One level of protection is the permissions on the consolidated log files themselves. This is controlled using the syslog-ng.conf.server file. Each syslog-ng “file” destination can have specific permissions specified. If the log directory for a consolidated file does not exist, syslog-ng can be instructed to create it (create_dirs(yes)) and set the directory’s ownership and permissions on the directory as well. For example,

destination d_file { file(“/clog/test/example.log” );
 dir_owner(root);
 dir_group(sys);
 dir_perm(0600);
 owner(root);
 group(sys);
 perm(0600);
};

ssh Port Forwarding

ssh port forwarding sets up a tunnel for the log traffic between the syslog-ng log forwarding client and the syslog-ng log consolidation server. This ssh-based tunnel is only available when using the TCP transport, not UDP. Also, ssh port forwarding is not used when forwarding log traffic within a Serviceguard cluster (member to member). Standard TCP or UDP is used for intra-cluster log traffic.

ssh port forwarding is transparent to syslog-ng. The /etc/syslog-ng.conf.client file is configured so that syslog-ng forwards log traffic to a local port managed by ssh. The local ssh connects to the remote sshd on the log consolidation server and sshd opens the standard syslog-ng TCP port. The remote log consolidation believes it has a standard log forwarding client and is unaware of the tunneling taking place.

One problem with ssh tunneling is failure of the log consolidation server. If the syslog-ng server stops or crashes, the remote ssh tunnels disconnect. The client ssh tunnels will try to reconnect at one minute intervals. The reconnect time is configurable.

Each failed reconnect attempt is logged to the client’s local syslog.log. During this time, syslog-ng’s client log (/var/adm/syslog/syslog-ng.log) will show it trying to reconnect to the tunnel. The default reconnect time is 10 seconds. This is configurable using syslog-ng’s global setting "time_reopen(seconds)" parameter. See the syslog-ng open source reference manual (/opt/dsau/doc/syslog-ng) for details.

ssh Port Forwarding to a Serviceguard Cluster Log Consolidator

When using ssh port forwarding with a Serviceguard cluster as the log consolidation server, a special ssh configuration is required.

In general, using ssh port forwarding requires that the log consolidation server perform a key exchange with the log forwarding client. Specifically, the ssh public key for the remote log forwarding client must be added to the consolidation server’s authorized keys file. Also, the fingerprint for the log consolidation server is added to the log forwarding client’s /.ssh/known_hosts file. The client log forwarder is a trusted system after this key exchange, and the consolidation server does not need to prompt for any ssh passwords at this point.

Since the consolidation server is a package, it can potentially run on every member of the cluster. This key exchange between the remote log forwarding client and a cluster member must be replicated for each cluster member. Each cluster member has to establish the same trust relationship to the log forwarding clients.

A problem can arise with the log forwarding client’s known_host fingerprints . When using a package’s relocatable IP address for the initial ssh key exchange, the client will have the adoptive node’s fingerprint added to its local /.ssh/known_hosts file. When the package fails over and the ssh connection is reestablished, the new adoptive node will have a different fingerprint and ssh will detect this as a man-in-the-middle attack and refuse to reestablish the ssh tunnel.

In order to prevent this, each cluster member must look like the same system from the perspective of the log forwarding clients. This can be achieved by having each cluster member use an identical host key. The ssh host keys are located in /opt/ssh/etc and contained in the following files:

ssh_host_key
ssh_host_key.pub
ssh_host_dsa_key
ssh_host_dsa_key.pub
ssh_host_rsa_key
ssh_host_rsa_key.pub

Pick one of the cluster members and copy these files to the same directory on the other cluster members. Using the “cluster copy” or ccp tool is a quick way to do this, using the following commands:

# cd /opt/ssh/etc/

# ccp ssh_host_* /opt/ssh/etc/

Then from each log consolidation client, perform a standard ssh key exchange with the relocatable IP address of the clog package. One way to do this is using the csshsetup tool (see csshsetup(1)), as follows:

# csshsetup DNS name of the clog package

csshsetup will prompt for the password of the cluster in order to do the initial key exchange.

clog Network Port Usage

syslog and syslog-ng require specific network ports to be available for correct operation. These ports are the following:

UDP 514 – this port is used by syslogd clients for forwarding log messages
TCP port selected port - the administrator chooses which TCP port a syslog-ng log consolidator uses to receive messages.
TCP port 22 – When using ssh port forwarding to create encrypted tunnels, the remote clients communicate with the log consolidation server’s sshd daemon. In a default configuration, this daemon listens on TCP port 22.

Using Bastille to Harden the System

Bastille is a security-hardening lockdown tool that can be used to enhance the security of the HP-UX operating system. It provides customized lockdown on a system-by-system basis by allowing the administrator to choose which security features to enable or disable from hardening/lockdown checklists.

Bastille can be used to harden a log consolidation server by enabling security tools such as IP filtering. If IP filtering is enabled, the ports described in “clog Network Port Usage” must not be blocked.

Additionally, Bastille asks the following questions that impact a log consolidation system:

Do you want to BLOCK incoming Secure Shell connections with IPFilter?

When configuring a log consolidation server, answer No to the question if you plan to support clients using the tcp transport and ssh tunneled connections to the server.

Would you like to restrict the system logging daemon to local connections?

Answering yes to this question adds the -N option to /etc/syslog.conf. When configuring a log consolidation server, this option is required. The clog_wizard adds it automatically; the manual configuration instructions also explain the appropriate edits to /etc/syslog.conf.

Securing Consolidated Logs

Technical documentation

» Table of Contents

» Index

Log File Protections

ssh Port Forwarding

ssh Port Forwarding to a Serviceguard Cluster Log Consolidator

clog Network Port Usage

Using Bastille to Harden the System