I reported on this in the Guardian. Interesting piece to research. First, the history. You can find the exchange between Karl Roeckx and Ulf Möller here. An unfortunate mistake; I make mistakes too (it was my fault that a name was misspelt in the Guardian piece, for example), so rather than heap blame on individuals I suggest this is more about a problem with the process; the only people making significant changes to the source code of such an critical library should be the committers responsible for that library. No doubt the incident is prompting a review of the process for updating Debian, Ubuntu and other distros; perhaps we will end up with a slower but less vulnerable flow of updates.

Second, a remark from Tim Callan at Verisign which there was not room for in this piece. I asked him whether Verisign knows which of the certificates it has issued are bad. “Unfortunately we don’t have those key pairs to look at them and scan them and tell which ones are good and which ones are not,” he told me. All Verisign can do is to ask its customers to check, which Callan says it is doing “very very aggressively.” In mitigation, Verisign does have a record of what operating system was used to purchase the certificate, but this is not the same thing; it is an imperfect process. The only fix is to revoke and replace the bad ones, which the company is offering to do for free.

Third, there are two distinct risks here. First, weak SSL certificates. Versign is embarrassed because it has been issuing weak certificates; its core product has been undermined. However, according to Netcraft, of the 870,000 secure web servers on the Internet, only 20,000 report themselves as Debian and 4,000 as Ubuntu. The true figure will be somewhat more than that, but that is a relatively small proportion; and exploiting the weakness takes a bit of effort.

The second problem is the possibility of intercepting or cracking SSH tunnels used to administer affected servers. We saw this demonstrated at a hacking briefing run by NCC Group yesterday. Let’s assume that administrators use SSH authenticated with a private key – a common scenario – and that the key was generated by the faulty Open SSL library. I suspect this will have been true for many more than 20,000 servers, though a lot will now have been fixed. All you need to do is to run a script against that server armed with a list of the possible keys – under a thousand, according to the demo we saw*. When you get a hit, you can connect to that server, most likely with full root permissions.

The most hardened servers will not be so easy to crack. They will authenticate as a user with limited rights, and use su to elevate. They will limit access to specific IP addresses. They will use additional passphrases. And they will have changed the keys within hours of the problem being discovered.

Still, there are plenty of less secure servers out there, so what that means is that an unknown number of servers will have been compromised, and more will follow. If you are lucky, the intruders will hack your website and do obvious damage so the server will get cleaned up. If you are unlucky, the intruder will be discreet and quietly start stealing credit card numbers, or taking advantage of any information or privileges obtained to get access to additional servers or data, or make occasional use of the server in botnet attacks. Who knows?

Servers getting rooted is not a new problem; and it’s not yet clear whether this incident is more than a ripple. Colin Phipps at Netcraft doesn’t think it is. “We’ll see a lot of panicked system administrators,” he told me, “and we’ll see a lot of scepticism about open source.” That last point is probably the most significant.

*I’m told this was artificially reduced for the demo – but there are only 32,676 keys possible private keys to brute force access. However, even using the full set of 2048-bit RSA keys NCC Group successfully broke into a system which used Debian to generate SSH keys in 20 minutes, and think it could often be done in half that time.