Here’s a post where I learn a little humility, and everyone else gets to see that “it really can happen to anyone” isn’t just something we say to be nice.
I used to be so proud of myself. I’ve never had a serious hacking incident on my website (just a few incidents where hackers tried and failed to get in), and I was certain that I was smarter than the all the other techs who admitted in private FB groups that their servers had been hacked.
I was so sure of myself that I even released a short ebook on the topic.
That sense of superiority ended on Saturday when I discovered that my server had been hacked. I had several demo sites on there, and a few client sites, and most of the eleven sites were redirecting visitors to what looked like random blog posts (I still don’t understand it).
Everyone, including credit card companies, huge tech firms, the US govt, and retailers like B&N get hacked at some point, and on Saturday it was my turn to learn that my security wasn’t as good as I thought it was.
Fortunately for me, this ended up being more an embarrassment than a crisis. My hosting company has just finished restoring my server from a clean backup, and while they were doing that I was sitting here thinking about what I did wrong and what I need to do differently to keep this from happening again. And since I like to write out the lessons I learned (this helps them stick in my mind) I figured it would be useful to also turn it into a blog post.
Here are a few things about hacked websites that I didn’t know Saturday morning.
Note: I am leaving out the more technical and process-oriented lessons because they’re not very useful to the average user.
Edit: I can’t believe I forgot this, but the first thing I learned is:
Great tech support is a lifesaver
I cannot recommend my hosting company enough. PeoplesHost tech support is highly competent, and having them as a backstop reduced my sense of panic from “gut-wrenching” to “this is effing annoying”.
I am leasing a server from them, and I cannot tell you how happy I am. If I had had to deal with this at my previous hosting company, MediaTemple, I would adopted Plan N (fake my death and flee to Argentina under an assumed name) as being the easier option.
Instead, I can sit here writing this post, letting you know that I learned:
If one site gets hacked, they all get hacked
When other web techs told me that hackers can use an infected site on a server to target other sites on the same server, I always assumed that this could only happen on disreputable hosting companies like Bluehost (which is notorious for being insecure and unsafe). I thought this was a sign of improperly configured servers that lacked basic security.
I could not have been more wrong.
I will never know which site was hacked first, but I did learn on Saturday that hackers got into all the folders on my server, including the ones that were not related to any site.
What I learned from this is that even if you do everything right, you can still get hacked. It also taught me that:
Firewall plugins are only good for so much (even the paid plugins)
One of the security measures I take whenever I build a site (or take on the role of website admin) is install a firewall and security plugin. I used to think that so long as the plugin was properly configured, the site was safe from getting hacked.
A security plugin is useful, and you should definitely have one on your WP site, but it is only good for so much. If the hackers find a flaw in a plugin’s code, or if they hack another site on your server, that security plugin will not stop them from attacking your site from behind its defenses.
In fact, one of the hacked sites on my server belonged to a client who had purchased a paid support plan from Wordfence, and it still got hacked. That extra cost offered no extra protection.
That’s not to say that paying for anti-malware support is a waste of money; to be clear, I think the paid plans can still be worth the expense. When you buy one, what you’re really paying for is insurance against getting hacked and needing a tech to clean up your site.
And that is because:
Once a site is hacked, it has to be fixed by a person
One of the tools that used to be in my toolbox was a service called Malcare. This is a malware removal service with automated tools that are supposed to de-hack a site with one click.
Alas, that didn’t happen. The Malcare plugin couldn’t even identify all of the hacked files, much less fix them. I had to access my server over SSH and check each file and folder one by one for evidence that it had been hacked. (While very informative, that was as tedious as it sounds.)
If your site gets hacked, someone is going to have to fix it by hand. And depending on their skill level, they might not be able to do so. I spent four or five hours on Saturday trying to de-hack my server before I realized I was out of my depth, and decided that I had to restore the server from a backup.
Sometimes restoring from a clean backup is your best option
After Malcare failed me, I had only two options: Fix the hack myself, or hire someone. I didn’t have to look at my bank balance to know I could not afford to hire anyone, and after I realized I lacked the skill to resolve this myself, I was left with no choice but to restore a backup copy of my server.
Now that it’s done, I think that should have been my first choice because it would have saved me a lot of time and aggravation. I could have skipped over trying to fix the problem and gone directly to making sure it didn’t happen again.
Edit: And here’s a sixth thing I learned.
Online malware scanners are mostly useless
I used to swear by Sucuri’s Sitecheck scanner. I thought that if it could not find malware then there was no malware to be found. But then I had to clean up my server, and I learned that external scanners can really only do so much.
Sucuri could not find any sign that my sites were hacked, and yet I could see the code the hackers had inserted in the sites. This showed me that a negative result from Sucuri was meaningless, and that the only way to confirm a hack was by looking at the site’s code itself.
As astute reader may have noticed that I didn’t mention how I got hacked.
I do not know exactly why I got hacked, but the techs at my hosting company and I both think that the most likely cause was the Zero-Day security flaw discovered in the Social Warfare plugin a couple weeks ago. One of the client sites on my server has that plugin but did not get updated, and that site compromised the whole server.
At the time I was hesitant to touch their site because I had neither explicit nor implicit permission to manage that client’s site. (Yes, I do offer free updates with hosting, but that client predated my hosting plan, leaving me in what I felt was an awkward position.)
This was a mistake I will not repeat. In the future I will respond to all security issues on my server with the same process I use for taking care of clients’ sites; I will proactively resolve the issue, and then tell the affected parties.
That is not SOP for my industry, but I have learned first-hand that it should be.
Thank you for a very informative post that must have been a little embarrassing to write given some of your previous public statements. (Hee hee.) Glad you got everything sorted out with minimal damage.
You say you now think a clean restore should have been your first choice because it would have saved you a lot of time and aggravation, but it sounds as if a lot of the knowledge you gained and a lot of the change in your general strategy came from that time and aggravation.
The Malcare failure illustrates a fundamental problem with automation–including Amazon’s algorithms. You have to know a process in great detail before you can automate it. Automation cannot deal with something outside of it’s preset parameters. That’s what GPMs (General Purpose Monkeys) are for.
Yes. I’ve been there. Once, so many “invisible” pages were added to my old wordpress site with malicious code that my site got black listed by google. That took THREE MONTHS to fix. OMG.
Now I use WP Engine for my wordpress hosting. It’s very expensive compared to my hosting before (right now I’m paying $75/month for three sites). However, they proactively monitor my site for hacking and fix things quickly when there are any issues. Their support is AMAZING! I’ve live chatted with them at all hours of the day/night and they always respond quickly and help me fix my problem.
I agree, good tech support is crucial.
Flee. Not flea.
thanks – I blame autocorrect