Maybe you saw a warning when accessing your site this morning. On occasion, you may have been redirected to another site altogether – a site that was trying to sell something to your visitors. Perhaps just above your header image, you caught a glimpse of an almost invisible message from a “hacker,” claiming that your security was “w34k.” Or maybe you were simply greeted by a very sudden and unexpected White Screen of Death.

The root cause behind all these events could be the same: a single hacking incident. With the rising availability of automated software that can either brute-force their way into your site’s Dashboard, exploit a known vulnerability in one of its many plugins, or efficiently do both, it should come as no surprise that we see more and more compromised sites every day. While we do our best to proactively secure our infrastructure against common attacks, our clients install their own applications on their own hosting space, which always involves a security risk.

In this article, we will cover how the experts in our Incident Response Team approach such cases. We have selected a WordPress installation for this example, but while the details will be different, the logical framework of the investigation is nearly identical for all common web applications.

The Incident

We perform regular antivirus checks for all hosting accounts. However, due to the nature of PHP obfuscation, malware can remain undiscovered for a while. In this case, the customer notified us that they received a malicious content warning when visiting their website, so our colleagues went in to manually review their files. Usually, a WordPress installation will contain a number of standard core files in its main directory, and the same goes for all major CMS applications – so naturally, a couple of files grabbed their attention with their unusual filenames:

133ja3lore.php
a1cw42ipim.php

From this point on, we will be using some of the standard Unix command line tools in order to gain more information about these files, and about how they originally appeared on the account. All of our hosting plans come with SSH access out of the box, as well as full Apache access logs, so you could reproduce these steps on your own.

Obtaining Timestamps

The stat utility gives us a lot of information about files, and this is how its output looks like:

[17:09:32] server~$ stat /home/user/www/www/133ja3lore.php
  File: /home/user/www/www/133ja3lore.php
  Size: 4909          Blocks: 16         IO Block: 4096   regular file
Device: fc00h/64512d    Inode: 202768472   Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 4731/user)   Gid: ( 4674/user)
Access: 2018-02-02 15:16:02.000000000 +0800
Modify: 2018-02-02 15:16:02.000000000 +0800
Change: 2018-08-07 20:49:47.771134758 +0800
 Birth:
[17:09:43] server~$ stat /home/user/www/www/a1cw42ipim.php
  File: /home/user/www/www/a1cw42ipim.php
  Size: 4909          Blocks: 16         IO Block: 4096   regular file
Device: fc00h/64512d    Inode: 202768462   Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 4731/user)   Gid: ( 4674/user)
Access: 2018-02-02 15:16:02.000000000 +0800
Modify: 2018-02-02 15:16:02.000000000 +0800
Change: 2018-08-07 19:41:46.676971426 +0800
 Birth:

The main difference between the mtime (modification time) and ctime (change time) is that mtime shows the last change of the contents of the file, while ctime shows the last change of the contents of the file or of one of its attributes (such as permissions or owner). Depending on the situation, we may search our logs for the ctime attribute, the mtime attribute, or even both.

In this case, we take the ctime timestamp for further analysis, because for both files, it is the more recent attribute. We will also start by searching for the timestamp we obtained from a1cw42ipim.php, because it was changed about an hour earlier than the same attribute of the other file – hence, we assume that it was uploaded first, and takes us closer to the original Point of Entry.

Log Analysis

Current, real-time access logs are available from the Logs section of the Control Panel – this is helpful if the attack was very recent. In this case, the original event occurred a long time ago, so we will be using archived copies of the logs. These archives are stored in the logs directory of the account, and can be viewed from the File Manager, via FTP, or through SSH. The archives are in the gzip format, so you can use the zgrep command to search within them. Furthermore, log archives are split into individual files, with names based on subdomain and day, so this is how a search for the timestamp obtained previously would look like:

zgrep "19:41:" /home/user/logs/2018-08/www-07.log.gz

When checking the access logs, we pay special attention to any POST requests that we discover. Unlike GET requests, where all data is encoded in the URL, POST requests include data in the body of the request. Most file uploads and setting updates use this method, and this is how malware is usually uploaded, too. The search yields the following lines of interest to our investigation:

www.domain.com 192.0.2.1 - - [07/Aug/2018:19:41:44 +0800] "POST /um-api/route/um!core!Files/ajax_image_upload/a2c75736fe HTTP/1.1" 200 407 "domain.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36" 0 0 "off:-:-" 5045 299593 192.0.2.2 domain.com

www.domain.com 192.0.2.2 - - [07/Aug/2018:19:41:45 +0800] "POST /wp-content/uploads/ultimatemember/temp/Ao3uKpx8B9klgpJ6Ra7A8z0jycfjvtiZZDrPtZao/stream_photo_9c8d90bc587c22ae9aef83fcdb2a02d0_5b69857918ac2.php HTTP/1.1" 200 494 "domain.com" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.75 Safari/537.36" 0 0 "off:-:-" 686 1047741 192.0.2.2 domain.com

Sections of the URLs in these POST requests – um-api and ultimatemember – suggest that the attacker gained access through the popular Ultimate Member plugin. Around the time of this incident, there was a well-documented vulnerability that was massively exploited online, which tells us how the malware was originally uploaded on the account.

You should note that we rarely get as “lucky” on the first try. The malware that we investigate may have been uploaded by another malicious file, and more often than not, the same process will have to be repeated several times before the Point of Entry has been identified. It is also entirely possible that the account was compromised on another server before being migrated to us. Additionally, this process is tuned and specific for our environment. However, any respectable hosting provider should be able to provide you with the access and tools required to perform these steps.

Once we have identified the original Point of Entry, we can begin working on cleaning up the account, and hardening it against similar issues in the future. We will discuss this in more detail in our next post on the subject.

Avatar
Author

Web hosting provider since 2001. At the moment we host over 59,000 web sites for customers in over 140 countries around the globe.