cPanel : SSHD Rootkit

We have been seeing this SSHD rootkit from time to time with much improved versions 🙂

One could see number of SSH processes on the server with nothing under process details. There is a huge discussion thread at WHT www.webhostingtalk.com/showthread.php?t=1235797

The current one which I have seen today was with keyutils-libs legit version name. However the “Signature” was missing.

root@linuxbabu [/var/log]# ls -la /lib64 | grep libkeyutils
lrwxrwxrwx 1 root root 18 Jun 22 2012 libkeyutils.so.1 -> libkeyutils.so.1.3*
-rwxr-xr-x 1 root root 35320 Jun 22 2012 libkeyutils.so.1.3*

root@linuxbabu [/var/log]# rpm -qi keyutils-libs
Name : keyutils-libs Relocations: (not relocatable)
Version : 1.4 Vendor: CentOS
Release : 4.el6 Build Date: Fri 22 Jun 2012 01:20:38 AM CDT
Install Date: Tue 14 Jan 2014 04:56:45 AM CST Build Host: c6b10.bsys.dev.centos.org
Group : System Environment/Base Source RPM: keyutils-1.4-4.el6.src.rpm
Size : 59336 License: GPLv2+ and LGPLv2+
Signature : (none)
URL : http://people.redhat.com/~dhowells/keyutils/
Summary : Key utilities library
Description :
This package provides a wrapper library for the key management facility system
calls.

I matched the size from a clean server and found different in the file size of libkeyutils.so.1.3.

So I reinstalled keyutils-libs replacing the file using yum.

yum reinstall keyutils-libs -y

After reinstall I could see the file size changes and also the Signature shown was correct :

root@linuxbabu [~]# ls -la /lib64/libkeyutils*
lrwxrwxrwx 1 root root 18 Feb 25 01:03 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3*
-rwxr-xr-x 1 root root 10192 Jun 22 2012 /lib64/libkeyutils.so.1.3*
root@linuxbabu [~]# rpm -qi keyutils-libs
Name : keyutils-libs Relocations: (not relocatable)
Version : 1.4 Vendor: CentOS
Release : 4.el6 Build Date: Fri 22 Jun 2012 01:20:38 AM CDT
Install Date: Tue 25 Feb 2014 01:03:47 AM CST Build Host: c6b10.bsys.dev.centos.org
Group : System Environment/Base Source RPM: keyutils-1.4-4.el6.src.rpm
Size : 36624 License: GPLv2+ and LGPLv2+
Signature : RSA/SHA1, Sun 24 Jun 2012 05:18:51 PM CDT, Key ID 0946fca2c105b9de
Packager : CentOS BuildSystem
URL : http://people.redhat.com/~dhowells/keyutils/
Summary : Key utilities library
Description :
This package provides a wrapper library for the key management facility system
calls.
root@linuxbabu [~]#

Hope this helps to disable the rootkit and avoid further damage to the server.

Another MySQL daemon already running with the same unix socket.

After upgrading MySQL binaries mysqld will not start at all any more and shows below error :

CT-1977-bash-4.1# service mysqld start
Another MySQL daemon already running with the same unix socket.
Starting mysqld: [FAILED]
CT-1977-bash-4.1#

MySQL service does not shut down gracefully during the OS reboot, leaving the old /var/lib/mysql/mysql.sock such that mysqld will not start up. Some people were able to reproduce this error on a CentOS 6.5 KVM guest virtual system by rebooting the host CentOS 6.5 system. CentOS is supposed to gracefully shut down the guest systems, but this seems to be failing in the case of mysqld.

Confirmed Red Hat Linux 6.5 bug – https://bugzilla.redhat.com/show_bug.cgi?id=1037650

Issue discussion on MySQL bug tracker – http://bugs.mysql.com/bug.php?id=71086

Simple steps to reproduce this issue:

service mysqld start
killall -9 mysqld_safe mysqld
service mysqld start

A quick way to restart MySQL is as below :
Remove socket file and restart mysql:
CT-1977-bash-4.1# ls -la /var/lib/mysql/mysql.sock
srwxrwxrwx 1 mysql mysql 0 Jan 8 20:13 /var/lib/mysql/mysql.sock
CT-1977-bash-4.1# rm /var/lib/mysql/mysql.sock
CT-1977-bash-4.1# /etc/init.d/mysqld restart
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]
CT-1977-bash-4.1#

A work around for the issue is to modify ‘/etc/init.d/mysqld’ script:

Make a backup copy of the startup script.
cp -p /etc/init.d/mysqld /etc/init.d/mysqld.orig

Edit the file /etc/init.d/mysqld to look as below :
=========================
# if fuser "$socketfile" &>/dev/null ; then
# echo "Socket file $socketfile exists. Is another MySQL daemon already running with the same unix socket?"
# action $"Starting $prog: " /bin/false
# return 1

# We check if there is already a process using the socket file,
# since otherwise this init script could report false positive
# result and mysqld_safe would remove the socket file, which
# actually uses a different daemon.
if fuser "$socketfile" &>/dev/null ; then
echo "Socket file $socketfile exists. Is another MySQL daemon already running with the same unix socket?"
action $"Starting $prog: " /bin/false
return 1
fi
=========================

Now restart MySQL :
CT-1977-bash-4.1# service mysqld start
Starting mysqld: [ OK ]
CT-1977-bash-4.1#