An unknown certificate is a failure in SSL/TLS, and that's how it should be. Ever since Firefox 3 came out, the way it presents SSL-enabled Web sites with self-signed certificates has been called scary and hurtful. Untrusted self-signed certificates should be scary because untrusted self-signed certificates are a failure in SSL/TLS, and a failure in your authentication and encryption mechanism should be treated as serious. Encryption with unknown parties is useless.Self-signed certificates can be perfectly trustworthy, but you have to know you have the certificate that corresponds to the server you're communicating with. The question is one of identity. SSL does provide encryption, but equally important, SSL lets the Web browser authenticate the Web server as a trusted entity. It's the authenticated identity that's important. Let me start from the beginning.

A digital certificate is a blob of data typical in the X.509V3 format, which contains information about the certificate holder. In the case of a Web server, it contains the Web server's host name, validity date, and maybe some other data. When a company wants to have a signed Web server certificate from a public certificate authority like VeriSign or Comodo, it has to prove to the CA that it is the rightful owner of the domain and is authorized to make the request. The Web server admin creates a certificate request, which creates a public/private key pair (the private key is always kept private), then creates a request with any required information like the corresponding public key. The request is signed with the private key and sent to the CA. The CA does its checks to make sure the request is from a valid source and then checks to see that the request was signed by the public keys corresponding private key. If that all works, then the CA adds its information to the certificate and signs the request using its private key. Through the magic of public key cryptography, your browser can validate the CA's signature using the signing CA's public key. You trust the certificate because the signing CA has validated that the request was from an authorized source.

Here's an analog example: When you travel abroad, you present your passport to enter a country. That passport is created by your government and is trusted by other governments as valid. It's quite simple in principle.

Broadly speaking, there are two types of certificates. Trusted certificates are certificates that are either signed by a trusted certificate authority or the self-signed certificate is known to the browser. The other type of certificate is untrusted, which are certificates where the signing CA or self-signed certificate is unknown to the browser.

The browser considers a certificate trusted if the signing CA certificate or self-signed certificate is stored in the browser's certificate store. You can see the certificate store in IE7 by going to Tool->Internet Options->Content->Certificates or in Firefox 2 via Tools->Options->Advanced->Encryption->View Certificates. If the signing CA certificate or the self-signed certificate is not present in the browser store, then it's untrusted. The browser assumes that if the signing CA certificate or self-signed certificate is present, an authorized party such as the browser vendor or you, the user, put it there.

When a browser receives a certificate, it checks to make sure the certificate is within the validity period (not expired), that the common name matches the requested host name, and some other stuff. If those all check out, the browser then looks to see if the certificate is trusted. If it's signed by a CA, the browser then looks for the signing CA certificate in its certificate store. The signing CA certificate contains the public key, which corresponds to the signing CA private key. The private key is used to sign the certificate, and the public key is used to validate the Web server certificate's signature. If the signature matches, then the browser assumes that the certificate was signed by the trusted CA. If the certificate is self-signed, then the browser uses the public key in the trusted self-signed certificate to validate the signature of the certificate presented by the Web server. If the signature fails, then the certificate is not from a trustworthy source. Back to the passport example, the passport has been forged or has a problem.

Whom Do You Trust? Why do we trust the certificates that are shipped with browsers? Because hopefully the browser vendors have validated in some way that the signing CA certificates are from the legitimate source. Perhaps they visit the CA's offices in person to get a copy of the CA certificates, or they download them from CA's Web site and then call the CA to validate the certificates.

Now what about self-signed certificates? Self-signed certificates are perfectly valid and trustworthy provided that you can validate the self-signed certificate before trusting it. You do that by getting a copy of the self-signed certificate through some trusted method and then when presented with the self-signed certificate through SSL/TLS, you manually validate it with the copy you have. Check the signature thumbprints, for example. If they match, then you simply add the self-signed certificate to your trusted browser store and you're done.

Firefox 3's explanation of why the certificate is untrusted is perfectly clear if you bother to read the message. "Unknown Authority Certificate is not trusted, because it hasn't been verified by a recognized authority." That means the certificate has not been signed by a trusted CA nor has the certificate been trusted by you. So be an authority! Get a copy of the certificate, validate the Web server certificate, and create a permanent exception. No problem. It's a one-time deal per self-signed certificate.

You can simply trust self-signed certificates without validating them first, but then you really don't know who you're creating a secure connection with, do you? And for those commercial sites that have invalid certificates -- ones that have expired or are self-signed -- those are failures, too, and you shouldn't trust them either.

There is apparently a lot of confusion and misinformation about SSL/TLS and digital certificates, and I plan on giving this topic more thorough treatment in the future.