OpenVZ 7 Upgrade Error !

While upgrading OpenVZ server I got an error as below :

====
--> Finished Dependency Resolution
Error: Package: 10:qemu-kvm-vz-2.9.0-16.3.vz7.8.x86_64 (factory)
Requires: libspice-server.so.1(SPICE_SERVER_0.12.5)(64bit)
You could try using --skip-broken to work around the problem
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
grub2-theme-openvz-1.4-1.vl7.noarch has installed conflicts grub2-theme-openvz: grub2-theme-openvz-1.4-1.vl7.noarch

====

This is a known bug and has been patched by OpenVZ Team and below is the fix :

Update vzlinux-release
yum update vzlinux-release
Enable virtuozzolinux-factory repo – edit /etc/yum.repos.d/vzlinux.repo or launch
yum-config-manager --enable virtuozzolinux-factory
Actually launch “yum update”
yum update

This fixes the bug. πŸ™‚

Reference : https://bugs.openvz.org/browse/OVZ-6924

Disable MySQL strict mode!

We recently migrated WHMCS to latest CentOS 7 server with cPanel. The WHMCS ClientArea showed White screen when “Client logins were used”. We found that the server had MySQL strict mode enabled which was causing this problem.

I disabled MySQL strict mode as below :

Edited /usr/my.cnf and changed below value :

sql_mode=NO_ENGINE_SUBSTITUTION,NO_AUTO_CREATE_USER

Restarted MySQL :

/scripts/restartsrv mysql

voila this worked…… earlier I was trying to change sql_mode under /etc/my.cnf however found MySQL is using different location /usr/my.cnf

WordPress :: xmlrpc.php Attack!

WordPress is the most targeted CMS nowadays and needs to be updated regularly. Recently I have seen attacks on wordpress xmlrpc.php using POST requests and the attack is large enough to take down / freeze the server.

The top or ps aufxw shows most of the xmlrpc.php requests as below :

==========
linuxbabu 4414 1.3 0.4 85512 35544 ? R 20:57 0:00 | \_ /usr/bin/php /home/linuxbabu/public_html/wordpress/xmlrpc.php
nobody 3876 0.0 0.1 25936 10852 ? S 20:52 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
linuxbabu 4413 1.3 0.4 85512 35672 ? R 20:57 0:00 | \_ /usr/bin/php /home/linuxbabu/public_html/wordpress/xmlrpc.php
nobody 3877 0.0 0.1 25936 10852 ? S 20:52 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
linuxbabu 4418 1.3 0.3 82936 33472 ? R 20:57 0:00 | \_ /usr/bin/php /home/linuxbabu/public_html/wordpress/xmlrpc.php
nobody 3878 0.0 0.1 25936 10764 ? S 20:52 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL
linuxbabu 4381 1.3 0.5 91580 42368 ? R 20:56 0:01 | \_ /usr/bin/php /home/linuxbabu/public_html/wordpress/xmlrpc.php
nobody 3879 0.0 0.1 25936 10768 ? S 20:52 0:00 \_ /usr/local/apache/bin/httpd -k start -DSSL

89.248.168.164 - - [30/Jul/2014:16:51:27 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.174.101 - - [30/Jul/2014:17:03:11 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.168.164 - - [30/Jul/2014:17:03:09 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.168.164 - - [30/Jul/2014:17:06:38 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.174.101 - - [30/Jul/2014:17:09:30 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.174.101 - - [30/Jul/2014:17:16:27 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.168.164 - - [30/Jul/2014:17:16:26 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.174.101 - - [30/Jul/2014:17:35:03 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)"
89.248.168.164 - - [30/Jul/2014:17:37:30 -0500] "POST /xmlrpc.php HTTP/1.0" 200 370 "-" "Mozilla/4.0 (compatible: MSIE 7.0; Windows NT 6.0)
==========

First thing you would think is why not delete xmlrpc.php, NO that will generate 404 error and all attack will process the 404 page that too wont help reduce the load.

Here is a quick way to get control of the server :

Add below to .htaccess :

#redirect xmlrpc.php attack to attacking servers
Redirect 301 /xmlrpc.php http://127.0.0.1

You would see a drastic load change and load back to normal and get back your smile πŸ™‚

cPanel :: upcp failed, exited with code 25

I noticed below error while updating cPanel to latest one of the server:

Running `/usr/local/cpanel/scripts/updatenow --upcp --log=/var/cpanel/updatelogs/update.9320814564.log` failed, exited with code 25 (signal = 0)

Below was the OS installed on the server :

root@linuxbabu [~]# cat /etc/redhat-release
CentOS release 6.5 (Final)
root@linuxbabu [~]#

Further found that the cPanel config had the OS information missing as below :

root@linuxbabu [~]# cat /var/cpanel/sysinfo.config
# This values in this file are calculated and updated if necessary nightly. If you wish to override
# these values, populate lock= with a comma delimited list of keys you don't want updated.
# This will allow you to change those values to suit your needs.
#
# Example: if you set lock like this, then rpm_dist and rpm_arch will not be updated nightly:
# lock=rpm_dist,rpm_arch
#
ises=2
lock=
release=6.5
rpm_arch=x86_64
rpm_dist=centos
rpm_dist_ver=unknown
root@linuxbabu [~]#

I corrected the OS rpm_dist_ver under the config file and the upcp worked fine :

root@linuxbabu [~]# cat /var/cpanel/sysinfo.config | grep rpm_dist_ver
rpm_dist_ver=6
root@linuxbabu [~]#

cPanel : SSHD Rootkit

We have been seeing this SSHD rootkit from time to time with much improved versions πŸ™‚

One could see number of SSH processes on the server with nothing under process details. There is a huge discussion thread at WHT www.webhostingtalk.com/showthread.php?t=1235797

The current one which I have seen today was with keyutils-libs legit version name. However the “Signature” was missing.

root@linuxbabu [/var/log]# ls -la /lib64 | grep libkeyutils
lrwxrwxrwx 1 root root 18 Jun 22 2012 libkeyutils.so.1 -> libkeyutils.so.1.3*
-rwxr-xr-x 1 root root 35320 Jun 22 2012 libkeyutils.so.1.3*

root@linuxbabu [/var/log]# rpm -qi keyutils-libs
Name : keyutils-libs Relocations: (not relocatable)
Version : 1.4 Vendor: CentOS
Release : 4.el6 Build Date: Fri 22 Jun 2012 01:20:38 AM CDT
Install Date: Tue 14 Jan 2014 04:56:45 AM CST Build Host: c6b10.bsys.dev.centos.org
Group : System Environment/Base Source RPM: keyutils-1.4-4.el6.src.rpm
Size : 59336 License: GPLv2+ and LGPLv2+
Signature : (none)
URL : http://people.redhat.com/~dhowells/keyutils/
Summary : Key utilities library
Description :
This package provides a wrapper library for the key management facility system
calls.

I matched the size from a clean server and found different in the file size of libkeyutils.so.1.3.

So I reinstalled keyutils-libs replacing the file using yum.

yum reinstall keyutils-libs -y

After reinstall I could see the file size changes and also the Signature shown was correct :

root@linuxbabu [~]# ls -la /lib64/libkeyutils*
lrwxrwxrwx 1 root root 18 Feb 25 01:03 /lib64/libkeyutils.so.1 -> libkeyutils.so.1.3*
-rwxr-xr-x 1 root root 10192 Jun 22 2012 /lib64/libkeyutils.so.1.3*
root@linuxbabu [~]# rpm -qi keyutils-libs
Name : keyutils-libs Relocations: (not relocatable)
Version : 1.4 Vendor: CentOS
Release : 4.el6 Build Date: Fri 22 Jun 2012 01:20:38 AM CDT
Install Date: Tue 25 Feb 2014 01:03:47 AM CST Build Host: c6b10.bsys.dev.centos.org
Group : System Environment/Base Source RPM: keyutils-1.4-4.el6.src.rpm
Size : 36624 License: GPLv2+ and LGPLv2+
Signature : RSA/SHA1, Sun 24 Jun 2012 05:18:51 PM CDT, Key ID 0946fca2c105b9de
Packager : CentOS BuildSystem
URL : http://people.redhat.com/~dhowells/keyutils/
Summary : Key utilities library
Description :
This package provides a wrapper library for the key management facility system
calls.
root@linuxbabu [~]#

Hope this helps to disable the rootkit and avoid further damage to the server.

Another MySQL daemon already running with the same unix socket.

After upgrading MySQL binaries mysqld will not start at all any more and shows below error :

CT-1977-bash-4.1# service mysqld start
Another MySQL daemon already running with the same unix socket.
Starting mysqld: [FAILED]
CT-1977-bash-4.1#

MySQL service does not shut down gracefully during the OS reboot, leaving the old /var/lib/mysql/mysql.sock such that mysqld will not start up. Some people were able to reproduce this error on a CentOS 6.5 KVM guest virtual system by rebooting the host CentOS 6.5 system. CentOS is supposed to gracefully shut down the guest systems, but this seems to be failing in the case of mysqld.

Confirmed Red Hat Linux 6.5 bug – https://bugzilla.redhat.com/show_bug.cgi?id=1037650

Issue discussion on MySQL bug tracker – http://bugs.mysql.com/bug.php?id=71086

Simple steps to reproduce this issue:

service mysqld start
killall -9 mysqld_safe mysqld
service mysqld start

A quick way to restart MySQL is as below :
Remove socket file and restart mysql:
CT-1977-bash-4.1# ls -la /var/lib/mysql/mysql.sock
srwxrwxrwx 1 mysql mysql 0 Jan 8 20:13 /var/lib/mysql/mysql.sock
CT-1977-bash-4.1# rm /var/lib/mysql/mysql.sock
CT-1977-bash-4.1# /etc/init.d/mysqld restart
Stopping mysqld: [ OK ]
Starting mysqld: [ OK ]
CT-1977-bash-4.1#

A work around for the issue is to modify ‘/etc/init.d/mysqld’ script:

Make a backup copy of the startup script.
cp -p /etc/init.d/mysqld /etc/init.d/mysqld.orig

Edit the file /etc/init.d/mysqld to look as below :
=========================
# if fuser "$socketfile" &>/dev/null ; then
# echo "Socket file $socketfile exists. Is another MySQL daemon already running with the same unix socket?"
# action $"Starting $prog: " /bin/false
# return 1

# We check if there is already a process using the socket file,
# since otherwise this init script could report false positive
# result and mysqld_safe would remove the socket file, which
# actually uses a different daemon.
if fuser "$socketfile" &>/dev/null ; then
echo "Socket file $socketfile exists. Is another MySQL daemon already running with the same unix socket?"
action $"Starting $prog: " /bin/false
return 1
fi
=========================

Now restart MySQL :
CT-1977-bash-4.1# service mysqld start
Starting mysqld: [ OK ]
CT-1977-bash-4.1#

Virtuozzo : RTNETLINK answers: Operation not supported

Here is a new bug which I faced on virtuozzo 4 and 4.6 for Centos 6 VPS.

VPS fails to add IP during start or networking fails when network is restarted :

-bash-4.1# /etc/init.d/network restart
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: RTNETLINK answers: Operation not supported
Failed to bring up lo.
[FAILED]
Bringing up interface venet0: RTNETLINK answers: Operation not supported
Failed to bring up venet0.
[FAILED]
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
-bash-4.1#

ifconfig shows blank as networking fails to start :

-bash-4.1# ifconfig
-bash-4.1# rpm -q iproute
iproute-2.6.32-31.el6.x86_64
-bash-4.1# cat /etc/redhat-release
CentOS release 6.5 (Final)
-bash-4.1#

Here is the fix, download the rpm’s as per your VPS arch.

for 64-bit systems: http://mirror.centos.org/centos/6.4/os/x86_64/Packages/iproute-2.6.32-23.el6.x86_64.rpm
for 32-bit systems: http://mirror.centos.org/centos/6.4/os/i386/Packages/iproute-2.6.32-23.el6.i686.rpm

Networking is not available in VPS so download it on the Hardware Node :

cd /root/dino
wget http://mirror.centos.org/centos/6.4/os/x86_64/Packages/iproute-2.6.32-23.el6.x86_64.rpm

Copy it to the affected VPS :
cp iproute-2.6.32-23.el6.x86_64.rpm /vz/private/VEID/fs/root/root/
replace VEID with the affected VPS ID.

vzctl enter VEID

-bash-4.1# cd /root/
-bash-4.1# rpm -q iproute
iproute-2.6.32-31.el6.x86_64
-bash-4.1# rpm -e iproute --nodeps
-bash-4.1# rpm -Uvh iproute-2.6.32-23.el6.x86_64.rpm
Preparing... ########################################### [100%]
1:iproute ########################################### [100%]
-bash-4.1# /etc/init.d/network restart
Shutting down loopback interface: [ OK ]
Bringing up loopback interface: [ OK ]
Bringing up interface venet0: Determining if ip address 127.0.0.1 is already in use for device venet0...
SIOCADDRT: Network is unreachable
SIOCADDRT: Network is unreachable
[ OK ]
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
RTNETLINK answers: File exists
-bash-4.1#

Now you can see the IP’s responding. Please make sure you check the RPM and OS versions:

Parallels KB : http://kb.parallels.com/en/118992

UPDATE :

to avoid updates due to yum append iproute* to exclude line in file /etc/yum.conf

cPanel : /root/ Inode issue!

Today I had an issue on / with shortage of disk inodes on a cPanel dedicated server.

In computing, an inode is a data structure on a traditional Unix-style file system such as UFS. An inode stores basic information about a regular file, directory, or other file system object.

In layman’s term – Every time a file is created or uploaded on a server, an inode is created. Simply say, inode is the count of the number of files on your server may it be a VPS or dedicated server.

The number of inodes used can be checked using below command (example is for /root) :

for i in /root ; do echo $i; find $i -type f | wc -l ;done

For my current issue it was the comet directory which was using up the space / inodes. The comet directory is usually filled on a high mail traffic server.

/root/.cpanel/comet

You can safely clean up the comet directory by running the following command that will remove all files in this directory that have not been accessed in more than three days.

/usr/local/cpanel/bin/purge_dead_comet_files

Example :

root@linuxbabu [~]# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda6 960992 960992 0 100% /
root@linuxbabu [~]#

root@linuxbabu [~]# /usr/local/cpanel/bin/purge_dead_comet_files
******Cleaning up comet for root...Done
root@linuxbabu [~]#

root@linuxbabu [~]# df -i /
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/sda6 960992 31278 929714 4% /
root@linuxbabu [~]#

This should free your Inodes / Disk space.
πŸ™‚

wp-cron.php – High CPU usage

What is wp-cron.php ?

This file is a PHP script which runs all the automated tasks that let WordPress do all it’s wonderful tricks. Some examples include:

 

  • Posting content when it is scheduled to be posted at specific times
  • Check all pending comments for spam (if you have plugins like Akismet running)
  • Send emails (i.e. if you have the option enabled where you get emailed whenever a comment is posted, this script handles the email)

Basically wp-cron.php is the automatic part of WordPress.

WP-Cron.php is a very common cause of high CPU loads. Disabling WP-Cron can drastically reduce CPU-load and prevent the chances your account is suspended due to exceeding your resources.

You can disable WP-Cron by editing your wp-config.php and adding the following line;

define('DISABLE_WP_CRON', 'true');

Create a cron job and run wp-cron.php every hour or two using the following command:

wget -O /dev/null http://www.example.com/wp-cron.php?doing_wp_cron

OR (if wget is disabled) :

cd /home/cpanel_user/public_html; php -q wp-cron.php

πŸ™‚

cPanel :: cPHulk error – Error while connecting to MySQL

WHM shows below error for cPHulk Brute Force Protection :

=========
cPHulk Brute Force Protection
Mysql is currently disabled. To enable mysql go to: Service Manager
Once there check Enable and Monitor for mysql. Then at the bottom of the page click save.

=========

Running below command should fix the issue :

/usr/local/cpanel/bin/hulkdsetup

Sample Output :
===========
root@1 [/]# /usr/local/cpanel/bin/hulkdsetup
hulkdsetup: synchronizing database schema

## mysqldiff 0.43
##
## Run on Sat Apr 6 04:05:04 2013
## Options: debug=0, host=localhost
##
## — db: cphulkd (host=localhost)
## +++ file: /usr/local/cpanel/etc/cphulkd_db.sql

CREATE TABLE auths (
SERVER char(128) NOT NULL,
USER char(128) NOT NULL,
PASS char(128) NOT NULL,
PRIMARY KEY (SERVER,USER)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE blacklist (
IP char(128) NOT NULL,
ISPREFIX int(1) DEFAULT ‘0’,
UNIQUE KEY IP (IP),
KEY ISPREFIX_index (ISPREFIX),
KEY IP_index (IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE brutes (
IP char(255) NOT NULL DEFAULT ”,
NOTES text,
BRUTETIME datetime DEFAULT NULL,
EXPTIME datetime DEFAULT NULL,
PRIMARY KEY (IP),
KEY EXPTIME_index (EXPTIME)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE good_logins (
USER char(128) NOT NULL,
IP char(255) DEFAULT NULL,
LOGINSERVICE char(64) DEFAULT NULL,
LOGINTIME datetime DEFAULT NULL,
KEY LOGINTIME_LOGINSERVICE_USER_index (LOGINTIME,LOGINSERVICE,USER)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE logins (
USER char(128) NOT NULL,
IP char(255) DEFAULT NULL,
SERVICE char(64) DEFAULT NULL,
STATUS int(1) DEFAULT NULL,
LOGINTIME datetime DEFAULT NULL,
KEY LOGINTIME_SERVICE_STATUS_USER_index (LOGINTIME,SERVICE,STATUS,USER),
KEY LOGINTIME_IP_index (LOGINTIME,IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE report (
type char(16) NOT NULL,
service char(16) NOT NULL,
login_service char(16) DEFAULT NULL,
ip char(200) DEFAULT NULL,
user char(100) DEFAULT NULL,
failcount int(11) DEFAULT NULL,
logintime datetime DEFAULT NULL
) ENGINE=MyISAM DEFAULT CHARSET=latin1

CREATE TABLE whitelist (
IP char(128) NOT NULL,
ISPREFIX int(1) DEFAULT ‘0’,
UNIQUE KEY IP (IP),
KEY ISPREFIX_index (ISPREFIX),
KEY IP_index (IP)
) ENGINE=MyISAM DEFAULT CHARSET=latin1
root@1 [/]#

===========