The Wonderful World of Warfare or (WWW)

Dealing With Various Cyber Attacks

Over the last few days it’s come to my attention that my VPS had been compromised for the first time in a little under a decade and so I thought I’d write a little on the subject. After all, what better way to breakout into blogging than to complain. It would seem that many people are unaware of the constant brute force attacks and vulnerability scans going on in the background of their servers. Particularly, users with several online and home servers (e.g. OVH and home/office – like me) seem to either not care to deal with individual server attacks, believe these attacks are singular occurrences, or make the mistake of thinking that nobody would attack their small business server. In any case, the result is a broken server, a spam-flagged IP, and possibly a hurt ego.

The fact is, most servers all around the world are under all sorts of different attacks.

Kaspersky Threat Map

Of course, the above map is a honeypot index, so it’s not an absolutely accurate representation of on-going attacks. However, it serves to show the internet is a dangerous place filled with bots and not protecting your servers is a mistake you’ll wish you never made – and sadly one that I did. See, I believed that with enough strength behind my passwords even billions of bruteforce attacks on my server would take millennia before they made progress. Technically, I wasn’t wrong, but I didn’t account for site, application, and kernel vulnerabilities. Sometimes hackers don’t need to find your password because they can access your server directly.

Yesterday and the day before then I received emails from OVH that look like this:

Naturally, my first instinct was to check server mail logs to see what was happening. As luck would have it though, I was blissfully unaware of the magnitude of the attacks. A quick look into logs shows this:

less /usr/local/psa/var/log/maillog

On a minute basis, my server was getting bruteforced for potential account logins and emails. The bots had already found a working login and were actively spamming mail, and yet there were still more looking for access! Regardless, these logs were entirely useless because sifting through this articled diarrhea would take longer than to rebuild my server from scratch after a format. What happened next is not an activity I visually cataloged (who would think to?) and took an obscene amount of time. First I ran ClamAV on webspaces and found hundreds of malicious scripts riddled across one particular domain. Obviously, the next step was to disable this domain. Then, thank god we live in the age of visual interfaces, I installed ImmunityAV on my plesk panel and ran their built-in cleaner. You might be thinking ‘oh no, was this all a sales pitch?’ No, because it didn’t work. I updated my WordPress instances, updated my plugins, and updated MyPHP. Momentarily after ImmunityAV cleaned my webspaces, not an hour had passed, and ClamAV was reporting the same malicious scripts on my server.

Clearly, there was a massive problem with either a WordPress theme or plugin. I could figure out the problem in time but this was an impending issue – my clients need mail access. I couldn’t just delete my client’s website or permanently block off port 25. The next logical step was to set up outgoing filter rules on my server.

This workaround halted the problem but did not propose a solution. So next I needed to scold my clients for using bootleg versions of plugins and themes (or otherwise ‘nulled’), repair their site, and contact the greater looming elephant in the room: my servers are getting attacked every second of the day and I have absolutely no security precautions besides a strong password.

It was time to setup fail2ban and modsec and introduce some strict rulesets. OWASP is a great starter ruleset for modsec and immediately cut down my access logs by half. Fail2ban was a little more difficult to configure in a reasonable way. I don’t want clients to be locked out for years at a time for screwing up a password, but I also recognize that most panel login attempts are not by my clients. The compromise was 2 login attempts (30 minute cooldown) with a year long IP ban if the user attempts to login through an alternate protocol/port after their ban. Additionally, any intruders attempting to enter credentials for any non-existent users receive much more severe and permanent punishment. The results?

We have a lot of banned IPs and a slightly reduced amount of login attempts compared to when I first started my endeavour to fix the problem. Despite my efforts, my servers are under constant attack and further mitigation is virtually impossible on a hosted server with multiple clients (I can’t ban regions, can’t set ranges, and can’t make a whitelist). However, fear not, if you own a home server or you’re the only user of your panel, then you’re in luck. The best solution is often the simplest, and the rule of Occam’s razor applies here just as well. Rather than filtering for who cannot connect to your server you should filter for who can. Set up a whitelist for your own IP range and block all ports you don’t use. If you have a dynamic IP consider investing in either a VPN with reserved IPs or into hardware such as PFSense which can act as a VPN when connected to your server. I personally will be updating my server prices in the future and implementing an enterprise-level VPN solution for my customers to connect through.