« Two-factor authentication is a necessary step in the arms race | Main | Built-in not bolt-on »

April 05, 2005

Red Hat Linux vs Microsoft Windows: what really matters

Many people have been talking about the recent paper that compares Microsoft Windows with Red Hat Linux. The paper does a role-based comparison of Windows Server 2003 with Red Hat Enterprise Linux ES 3, meaning that they are compared when functioning just as web-servers. The conclusion of the report is that Windows in its default installation was more secure than Red Hat during 2004, with 52 vulnerabitlities for Windows vs 132 for Red Hat during the year, and 31 days of risk vs 70.

Of course, this paper created quite a firestorm, not least because the ongoing argument of whether Linux is more secure than Windows is like a religious war, and the paper was funded by Microsoft, which only increases the controversy.

After reading the paper, I have to say that I think it is a very valuable contribution to the debate, because it tries to quantify the difference between the two systems. Perhaps the data is not accurate, as some would argue, but the paper gives a full description of the methodology used, and so should be easily replicable. I'd love to see some of the naysayers repeat the analysis with their own data and see what comes up.

I do have a few quibbles though. Firstly, this paper compares two vendors, Microsoft and Red Hat, which doesn't necessarily give a good idea of how secure Linux itself is. As the authors point out:

One interesting aspect of the challenge faced by Red Hat that is not obvious from a simple examination of the raw numbers is the delay between a fix becoming available within a product, and the inclusion of that product as an "approved" Red Hat package.

They go on to give an example of a MySQL bug that was fixed in June but only incorporated in the Red Hat package in November. And since they only look at fixes that come officially from Red Hat, this introduces a vendor specific bias. One of the reasons that this bias has an impact is that there are many vendors of Linux, but only one of Windows. With many - if not most - Windows vulnerabilities, Microsoft is given advance warning by the people who discovered the vulnerability, and given time to fix it before the vulnerability is made public. By contrast, Red Hat would not be given such advance warning, since it doesn't control the source code of Linux. Hence the task faced by Red Hat is much more daunting.

This has interesting ramifications for the security of open source. Firstly, it is potentially less secure because vulnerabilities are publically known before the community has time to fix them, unlike closed source. On the flip side, though, there is more knowledge about the existence of vulnerabilities. The thing that I find unnerving is that if a third party finds a vulnerability in Windows, and reports it to Microsoft to give them time to fix it before making it public, someone else could also have found that same vulnerability, and could be actively exploiting it without anyone knowing. How much this happens we don't know, and will probably never know, but it's certainly possible. What this could mean is that closed source will have less exposure to known vulnerabilities, but potentially more exposure to zero-day or unknown vulnerabilities, although of course this is wild speculation.

Another major potential issue with regard to the vulnerabilities count is that they only consider vulnerabilities that were fixed by the vendor during 2004. If the vulnerability existed in 2003 and was not fixed then, it is counted, and any vulnerabilities discovered in 2004 that were not fixed are not counted. To quote the paper:

we will not consider vulnerabilities announced in 2004 but fixed in 2005.

Of course, what this means is that if a vendor gets a 1000 vulnerabilities in 2004 and doesn't fix any of them, then they will not show up in the analysis. The authors try to iron out this effect by including vulnerabilities from 2003 that were not fixed in 2003, but I think what would make more sense is to consider all vulnerabilities from 2004, and make a note of which were fixed, and which weren't, including those carried over from earlier years. After all, if we want to know the days of risk during 2004, we have to include all vulnerabilities discovered during 2004 and not fixed.

And now a comment on methodology: they compared the systems in their default "out-of-the-box" configurations, without any security hardening. That is a fair way to go about it, but one objection that I have is that they totally disregarded the effect of the firewalls that exist on both these systems. I can't imagine anyone would actually deploy these servers without the firewalls turned on, especially on the Red Hat box since the firewall is on by default.

So what I'd like to see is a comparison in more meaningful terms, i.e. if I deploy these servers in default mode, connected to the internet, what are the chances that someone out there can own my box? Let's forget password stealing, and rather focus on application level vulnerabilities. Just reporting the total number of vulnerabilities, and using the ICAT severity ratings, doesn't give me much idea of the risk. In particular, ICAT calls a vulnerability severe if an attacker can remotely get root or user access, or a local user can get root access. Now these are all quite different things. If there are no remotely exploitable vulnerabilities, I don't care about local vulnerabilities. On the other hand, if there are no local rootable vulnerabilites, then the remote non-rootable vulnerabilities are so much less severe.

What I would suggest is the following metrics:

1. The total number of remote rootable vulnerabilities, assuming the firewall is on.

2. The total number of remote user access vulnerabilities, assuming the firewall is on.

3. The total number of local rootable vulnerabilities.

4. The days of risk for each category above.

These metrics would give a much more understandable idea of the risk than those in the paper. And who knows, Microsoft Windows may turn out to be more secure than Red Hat Linux...

Comments

Post a comment