collapse

* Notice

Important notice (31 July): We have recently recovered from a nearly two day downtime due to an attack. No data was lost or stolen but the server has been reinstalled as a precaution. Please let us know if you encounter any issues. We apologise for the unacceptable inconvenience. Please read here for more information.

Author [EN] [PL] [ES] [PT] [IT] [DE] [FR] [NL] [TR] [SR] [AR] [RU] Topic: 29-31 July 2020 downtime  (Read 113 times)

Offline samspin

  • Administratrix
  • Ancillus
  • *****
  • Posts: 118
  • Reputation: +8/-1
29-31 July 2020 downtime
« on: July 31, 2020, 06:13:49 pm »

Following a nearly 48 hour downtime, I would like to make a statement to reassure that no data was stolen. For some background: we host our server at OVH. One of OVH's great features is anti-DDOS protection, something that has helped us out before. Which is all well and good for outside sources, except... if it comes from within their own network. In which case OVH would have no choice but to take direct action on such sources. This happened in our case, as will be explained below.

In our case, a vulnerability in Apache, the software we use to run our site, was discovered. This vulnerability was caused by a misconfiguration on my part, I have to hold my hands up on that. More specifically, an attacker took advantage of a proxy module that had been left unintentionally open. This meant anyone could send 'requests' for Apache to connect to any other site and act as a relay. It appears someone discovered this by attempting this on hundreds of websites, including ours. Once successful, they noted this and when the time was right, launched a DDOS attack from a botnet to send multiple proxy requests to Apache on our server. This was evident because when I glanced over the log files to discover how this all happened, there were multiple source IP addresses involved. The final target was a fellow OVH customer's server running TeamSpeak. This could have overwhelmed their server had it not been for OVH's stringent monitoring, and they acted to immediately switch off our server to stop the attack, and place it in "rescue mode" to allow me to attempt to download all data safely. As neither me nor OVH were sure if our server was completely compromised, the next step was for me to grab all the data and then reinstall from scratch as a precaution. It was only later on while I was rebuilding that I glanced over the log files and discovered what happened.

Unfortunately the downtime was a lot longer than it could have been. The SQL databases that power the forum (as well as other WOD sites we host) were not properly shut down since the server had been hurriedly shut off. As it was not possible for me to start the server normally during recovery to export the databases to a portable format this made life very difficult. I could have used a month old backup but this would have meant losing forum threads/private messages within the last month. Without getting too technical, I had to find the exact version of MariaDB we had been using. Installing a server from scratch usually causes the very latest version to be installed, and it turned out an update had been released within the last couple of weeks and we had not applied it by the time of the attack. This broke compatibility with the database files, so finding the earlier version of MariaDB was necessary. Once I found this, I was finally able to get the site back online and then upgrade to the newest release. I've learned a lot from this situation and more regular backups of the databases is something I'll definitely be doing (we usually do these once a month).

You may ask why we had a proxy module enabled in Apache in the first place? Well, it's because of the need to maintain the old Gamespy era version of Planet Vampire ( usually located at archive.planetvampire.com ) as there are a number of articles and sites that still link to it. It is written in an old format that will only run on a Windows instance, and I needed to run a virtual machine to host this archive properly. The proxy module would accept requests for archive.planetvampire.com to be forwarded to the virtual machine which serves the archive site. This virtual machine is still in the process of being re-uploaded (as of 18:00 GMT) and this time will be using a dedicated IP for it, with Cloudflare in the middle for additional protection like the rest of the site. This negates the need for having any proxy module at all which should avoid nasty attacks being possible again for the foreseeable future.

I found no evidence of any hidden scripts or malware, and in any case a full reinstall has enabled me to ensure we only have what we need on this server. I can therefore assure you that any personal data is safe. I would like to thank OVH for continuing to have us as a customer and for acting so quickly to contain this incident. I have exchanged a few messages with them and explained what happened, and they have been very accommodating when I needed help to access the server. Hopefully that clears up any questions anyone may have. If not, please PM me with any concerns. I may add more here at a later date to address any further concerns anyone has.

EDIT 01 AUG 2020: The archive site is now running again.
« Last Edit: August 01, 2020, 11:06:41 am by samspin »

 

SimplePortal 2.3.7 © 2008-2020, SimplePortal