Compromised Server Security Measures

Recently, my employer's web server hosted on Rackspace Cloud was compromised (yes I still have a day job), and I helped resolve the issue, consequently formulating some security measures which I hope to impart today wishing you won't have to go through the upheavels we had. You may also learn more on this from Linux Server Security and Secure Architectures with OpenBSD.

From the experience we had at work, here are some of the measures I would advocate (some advise you may have encountered somewhere else):

• Isolate access to the compromised server
• Secure the workstation that solely accesses the server
• For web application scripts, have trusted copy in version control systems
• Your .htaccess may be the culprit
• Setup routine file change monitor

As our modus operandi goes, below are the rationale and discussion of the measures above.

Isolate access to the compromised server
As the saying goes: too many cooks spoil the stew. This has got to be the cardinal rule number zero (0); and by isolation, we mean only one (1 and no more) authorized and trusted user should have access to the server. To limit access to the server, we change the access credentials (i.e. passwords) to the compromised server and disseminate this information on a need-to-know basis, expecting to have these results: (1) we minimize the possibility of recurring server compromise, (2) this will make it easier to pinpoint where the security breach is coming from, and (3) this measure alone may avoid the incident from happening again.

Secure the workstation that solely accesses the server
No secured server is secure enough if it's accessed from a vulnerable system/workstation. Sadly, there are server configurations available now that are only accessed via Windows, which in my opinion, is a very dangerous value proposition. I have nothing personal against Microsoft, but if it's possible, never use Windows to access your secured servers in the first place. Microsoft Windows is the Mary Mallon of computing, by themselves harmless but they make you sick. As of this writing, my gripe against Rackspace Cloud is that instead of SSH access, they only have FTP remote connection, making the access to the server unsecured not just from the workstation end, but by the remote connection protocol itself. Should the workstation, for lack of better option, be a Windows computer, it should have very minimal or no other software installed other than the needed application to access the server to have less probability of problem to ensue.

For web application scripts, have trusted copy in version control systems
This may be more of a preventive measure that should have been done before a server is compromised. However, should the incident occur and hopefully with very little blame and fingerpointing, it's still great to know that we can have this to ensure server file integrity, easily determine if any text/script file is modified, and quickly assess the changes and impact on the features and performance of the server. In contrast to my previous article on software development systems and tools, I would advise you to use centralized version control systems like Subversion since you'll need to have a single trusted source of the web application script in this scenario.
Update: the process I propose is illustrated below

Thanks to http://www.gliffy.com for making this illustration pretty

Your .htaccess may be the target and culprit
Based on our recent experience, the compromised file on the compromised server is the .htaccess file, and for some good reason: (1) it can't be seen from FTP if accessed in Unix-like systems (i.e. Ubuntu), but visible on Windows machines, and (2) being a hidden file that's technically the gateway to the whole dynamic web application, it can mess things up before handing over the reins to the supposed to be legitimate application script(s). This further reinforces the requirement that your workstation should be secure.

Setup routine file change monitor
You may have an extensive file integrity check utility in place like Tripwire on GNU/Linux systems (free and open source version here) or mtree on OpenBSD to regularly log and help you monitor changes to server text files. For those who may not be able to avail of these option on their server, you may even use MD5 digest hash to verify your files are intact.

Should the server in question not be viable for its intended purpose again, we have to find a function that will serve our other objectives without causing critical interruption or economic sabotage.

What most guys had in mind when the word "server" was created.