Investigating and cleaning up a WordPress installation after a hacking incident

If the appearance or behavior of your site has changed, this may mean that it has been compromised. In this article, we will cover how our Incident Response Team handles such cases. We have selected a WordPress installation for this example, and while the details will be different, the logical framework of the investigation is nearly identical for all common web applications.

The Incident

Any WordPress installation will contain a number of standard core files in its main directory, and the same goes for all major CMS applications, so you should look for unexpected and unusual files and subdirectories in your web root directory. In this case, the user has the following two files in their WordPress installation directory:

  133ja3lore.php
  a1cw42ipim.php

We will be using some of the standard Unix command line tools in order to gain more information about these files, and about how they originally appeared on the account. We provide SSH access out of the box, as well as full Apache access logs, so you could reproduce these steps on your own.

Obtaining Timestamps

The stat utility gives us a lot of information about files, and this is how its output looks like:

[17:09:32] server~$ stat ~/www/www/133ja3lore.php
  File: ~/www/www/133ja3lore.php
  Size: 4909          Blocks: 16         IO Block: 4096   regular file
Device: fc00h/64512d    Inode: 202768472   Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 4731/user)   Gid: ( 4674/user)
Access: 2018-02-02 15:16:02.000000000 +0800
Modify: 2018-02-02 15:16:02.000000000 +0800
Change: 2018-08-07 20:49:47.771134758 +0800
 Birth:
[17:09:43] server~$ stat ~/www/www/a1cw42ipim.php
  File: ~/www/www/a1cw42ipim.php
  Size: 4909          Blocks: 16         IO Block: 4096   regular file
Device: fc00h/64512d    Inode: 202768462   Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 4731/user)   Gid: ( 4674/user)
Access: 2018-02-02 15:16:02.000000000 +0800
Modify: 2018-02-02 15:16:02.000000000 +0800
Change: 2018-08-07 19:41:46.676971426 +0800
 Birth:

The "~" symbol stands for "/home/$USER" where $USER is your hosting username.

The main difference between the mtime (modification time) and ctime (change time) is that mtime shows the last change of the contents of the file, while ctime shows the last change of the contents of the file or of one of its attributes (such as permissions or owner). Depending on the situation, we may search our logs for the ctime attribute, the mtime attribute, or even both.

In this case, we will take the ctime timestamp for further analysis, because for both files, it is the more recent attribute. We will also start by searching for the ctime timestamp we obtained from a1cw42ipim.php, because it was changed about an hour earlier than the same attribute of the other file. We will assume that it was uploaded first and should take us closer to the original Point of Entry.

Log Analysis

Current, real-time access logs are available from the Logs section of the Control Panel – this is helpful if the attack was very recent. In this case, the original event occurred a long time ago, so we will be using archived copies of the logs. These archives are stored in the Logs directory of the account, and can be viewed from the File Manager, via FTP, or through SSH. The archives are in the gzip format, so you can use the zgrep command to search in them. Furthermore, log archives are split into individual files, with names based on subdomain and day, so this is how a search for the timestamp obtained previously would look like:

zgrep "19:41:" ~/logs/2018-08/www-07.log.gz

When checking the access logs, we pay special attention to any POST requests that we discover. Unlike GET requests, where all data is encoded in the URL, POST requests include data in the body of the request. Most file uploads and setting updates use this method, and this is how malware is usually uploaded, too. The search yields the following lines of interest to our investigation:

www.domain.com 192.0.2.1 - - [07/Aug/2018:19:41:44 +0800] "POST /um-api/route/um!core!Files/ajax_image_upload/a2c75736fe HTTP...
www.domain.com 192.0.2.2 - - [07/Aug/2018:19:41:45 +0800] "POST /wp-content/uploads/ultimatemember/temp/Ao3uKpx8B9klgpJ6Ra7A8...

You should not expect to discover the Point of Entry on the first try. The malware that we investigate may have been uploaded by another malicious file, and more often than not, the same process will have to be repeated several times before the Point of Entry can be identified. It is also entirely possible that the account was compromised on another server and was then migrated. Additionally, this process is tuned and specific for our environment. However, any respectable hosting provider should be able to provide you with the access and tools required to perform these steps.

Once we have identified the original Point of Entry, we can begin working on cleaning up the account and hardening it against similar issues in the future. Unless a safe backup exists, this task can appear daunting, but it is actually pretty straightforward with a modern CMS.

Finding all executable files on your account and manually reviewing them for malware (since automated scanners do not work well enough for obfuscated PHP code) can range from difficult to outright impossible. Luckily, with a modern web application such as WordPress, you can simply re-install it while preserving your uploaded files. This is the fastest and the most effective method to bringing your site back up in a clean and secure state. We use it every day on all but the most specific and niche cases.

Save the wp-config.php file, your images, and your personal files

The wp-config.php file contains the basic configuration for your site, including the information necessary to connect to the database server. When saving it, open it and inspect its contents for anything suspicious, like a very long first line containing unusual characters. As a rule of thumb, if your text editor has an unusually long horizontal scroll bar, the file that you are editing is likely to be compromised. If that’s the case with the wp-config.php file, copy the database connection strings to a clean wp-config.php file from a fresh WordPress archive.

Your images and personal files should be stored in the wp-content/uploads directory. Ideally, there should be no PHP files at this location. Unfortunately, some plugins and themes may store some of their files in subdirectories there, making the task of correctly identifying and removing malware even harder. An easy way to list all PHP files in a given directory is to run the following command via SSH:

find ~/www/www/wp-content/uploads/ -iname "*.php"

If you receive no output, you can safely copy the uploads directory. Otherwise, you should review the identified PHP files manually in order to see if they were placed there by a harmless plugin, or by an attacker.

Before you delete anything, you should first get a list of your active plugins. You can either do this from the WordPress Dashboard, or by typing the following command in the SSH terminal inside your WordPress directory (e.g. ~/www/www):

wp plugin list --status=active

This is how the command’s output may look like:

 +----------------------------------+----------+-----------+---------+
 | name                             | status   | update    | version |
 +----------------------------------+----------+-----------+---------+
 | classic-editor                   |  active  | none      | 1.5     |
 | really-simple-ssl                |  active  | none      | 3.2.3   |
 +----------------------------------+----------+-----------+---------+

You can do the same for your active theme:

wp theme list --status=active

Delete the entire folder where WordPress is installed

The wp-config.php file and the uploads directory are all the files that you need to preserve from your current WordPress installation, since most of your site resides in the database. In some cases, the database needs cleaning, too – but these are relatively rare, and not as trivial to secure. By now, you should also have a list of your active plugins and themes, so that you can download their latest versions from the official vendors.

Upload a new clean full package of the latest WordPress version

You can obtain WordPress from their official website and install it manually, or you could use our automated installer. In either case, you can now copy the previously saved wp-config.php file and the uploads directory. You can also install all plugins and themes that you had been using previously. If they are publicly available in the WordPress repository, you can do so from the terminal:

wp plugin install really-simple-ssl classic-editor --activate

At this point, you can test your site – ideally, it should work just like it did before the incident. In the odd chance that it does not, you should contact our Support Team.

Additional precautions

When a WordPress site has been compromised, we assume that the attackers know its administrative credentials. This is why you should review and update the passwords for all administrative users. You should also double-check for any unusual users created around or after the attack.

We also encourage you to update your Control Panel password, as well as the passwords for all MySQL users. You should note that following a password change for your MySQL user, you should also update your wp-config.php file accordingly.