My WordPress got hacked and here is a brief story on How Content Change Monitoring helped me detect the Malware in the first place and steps I took to remove malware from WordPress
First suscipions : WordPress Hacked!
I had just completed an internal round of testing for my new SEO Change Monitor Plugin. The next phase was to broaden the testing and monitor larger set of domains. So I installed this on various WordPress websites to begin extended testing.
I installed on Sunday and got a few on-page alerts. On Page alerts do not need any history to generate any alerts. Only when somethings were to change it would be captured during the next scan.
Oh boy! the plugin is not ready yet…
Next day morning, I got an email alert from one of the installs and the email had unusually large number of alerts. My immediate thoughts were “Oh boy! the plugin is not ready yet… looks like it is generating duplicate or erroneous alerts”
In the first iteration of the Email Alert design, I decided not to include additional details as this might produce a huge amount of text.
Here is a snapshot of that email
Anxious and already thinking ahead “Did the scanning algorithm fail… what do I need to fix in the diff algorithm…”.
I said “Relax.. just login and check the details”
The detailed alert and all the changes are captured within the plugin dashboard.
Let us zoom in
Looking at the details, adjusting my eyes.. squinting and moving my head back and forth.. trying to make sense of what I am seeing.
In my mind…” I am pretty sure that all the demo data was disabled and removed and I certainly never listed or sold any “LOUIS VUITTON”.. Oh No! I am hacked..
Getting Hacked generates all kinds of negative emotions but having had this experience with my past WordPress sites I didn’t feel the usual panic.
Oh Lord! Why did you let my website get Hacked!
Instead I reached out and complained to my almighty.. “Oh Lord! Why did you let my website get hacked .. the very website where I am about to launch my new software.. What is this .. Is this a test?”
I was calm after this, rolled up my sleeves… not again.. “Lets work to clean this wordpress malware”, I said!
Backup and Export all WordPress content
I did a complete export of all my post and pages as XML file dump.
I was prepared to start from scratch, reinstall WordPress and then import all posts and pages.
I checked the plugins sections. I had several disabled ones, I went and deleted all disabled plugins. I deactivated several and kept bare minimum needed to run the website.
Next I went to the themes and deleted all but the active theme.
Installed WordFence and GOTMLS
I installed WordFence, ran the scan and it found several infected files. WordFence gave me the option to “Clean all infected files” and “Delete all infected files”. I tried the first option and ran the scan again. This time the number of issues reduced so I opted to “Delete all infected files”. Ran the scan again.. There was one particular file “index.php” in the root folder that WordFence wasn’t able to delete.
I logged into the terminal and see that the file ownership was different than the rest of the files. Next I installed GOTMLS , donated to get the core updates, ran the scan.
My site is hosted with Cloudways and it provides “Master” credentials and “App” credentials. You can set the file ownership either belonging to Master or app level. Since all ownership was mixed up, I tried a reset from the terminal.
chown -R master *This did not work (not enough permissions). The nice support person at Cloudways instructed me to use the Admin UI to reset the file ownership. I did and almost all files were changed except this “index.php”.
This was really strange. I got a copy of clean index.php. I opened it in notepad++ and verified, it had just few lines of understandable code. The clean index.php was only 400 bytes whereas the infected one was around 8k bytes.
See how clean and innocent this file is?
I opened the infected file and boy.. it was like staring into the dark canvas of an evil mind.. nope sorry I did not understand any bit of it nor did I try hard enough.
All I knew that it was all encoded and a perfect signature of a malware. I was just not aware of how notorious this evil malware was.
Malware Detection and Evil Behavior
I simply FTP the clean file and changed the file ownership. This seemed like an easy fix. I refresh the ftp client and started rubbing my eyes. The file is still at 8k. Strangely the file permissions were reset to ‘-r–r–r-‘ and ownership changed back to “App’ level.
I went to the terminal and did a delete on the file. No matter what I tried to get rid of the file, the evil file re-appeared in the time it took me to type ‘ls -l ‘. I could almost see that file laughing back at me!
I was convinced this was certainly the evil type and summoned all the Divine force I needed to fight this.
I noticed two files .htaccess and index.php playing this “Whack-a-mole” game with me. I kept changing file permissions and deleting the files and they would resurrect right back.
Cloudways support person was still on the chat with me and I asked if she can use her magical powers and delete it from their side. She tried and reached out to senior team members and nothing could remove this damn file.
I googled and found “Malcare in the advert section”. They claimed instant removal of malware. I signed up and went through the process of installing the plugin and synching the website.
It seemed happy to alert me “Yes your website is hacked!” Click upgrade to remove the virus instantly. They have a claim on their offer “if not removed, will return 3x the money”.
I signed up and provided all FTP credentials. I assumed this would be something like an instant help. But I had to raise a support ticket and send them an email. After 30 minutes I check my support ticket and got a message we have now removed the infected file, your website is clean. I went to the Malcare dashboard and they indeed were “Claiming victory over this virus!”
Hacked Malware Index.php and .htaccess regenerating
I went to my FTP client, refresh the folder and cannot believe what I see.
The evil index.php is right there staring back at me. I post an update to the ticket and tell them that this is not fixed yet. The file is still “Present”.
In the meanwhile, I am still trying to work with Cloudways support channel
I need help with the malware
I hired malcare and they said it cleaned the files but again I see the infected files
I go to command line and delete the file, in few seconds I see a new infected file
who is writing this file
it has to be some process on cloudways server
so please help finding the culpritAfter few minutes, I got a message from Aman at Malcare and instructed to restart the server. As per his comments, there is a server level process that is creating these files. That made sense but I am like “How is this possible? This is my VPS”
Anyways, I hop into Cloudways dashboard and restart the server from the UI. I go back to the terminal, delete the file and the index.php file is still sitting there. It simply won’t go.
I go back to the Cloudways dashboard, do a “Server Stop”.. Pray for few seconds and then do a “Server Start”.
I check and the malware index.php file is still not going away.. Happily protecting its turf.
How to find PHP process running in Linux
I collect the command to find php process and discover that there is one “php-fpm” process running. I try to issue a kill on that process but do not have enough privileges. So I ask Cloudways to run the command. The nice support team at Cloudways reassure me that they are working on it. After several minutes I get confirmation that the process is killed.
I notice that the evil index.php is no longer resurrecting.
Meanwhile I get a confirmation from MalCare team as well that the infection file is removed and apparently not coming back.
Every day I login to see if that Jeepers Creepers has come back… so far it is all good.
There are many things that are still not clear
- How did the Malware easily get in and hack my website whereas I have to lookup passwords for login into my wordpress dashboard?
- How did it manage to setup a server level process getting unlimited CPU time whereas PHP gives me only 30 to 60 seconds of continuous CPU time?
Summary of steps : How to Remove Malware from WordPress
- Remove all deactivated plugins
- Remove all unused themes
- Install WordFence
- Install GotMLS (donated and got all core updates)
- I went with Malcare annual plan as it seems to be cheapest of all services
- Engaged Cloudways support who are always willing to work with you
- Changed all master login credentails
- Changed all app level credentials
- I wish there was a way within Cloudways to turn off the application but keep the server running. This way it would help restrict any outside interference during the malware removal.
- If you are using WordPress, you will be hacked. Just make sure not to panic and be prepared to invest in securing your wordpress.
