2018-03-05 Verifying X.509 Peer Certificate Strength

The X.509 certificates of the SSL/TLS protocol contain cryptographically-secure data, but only if the cryptographic algorithms are of sufficient strength to secure that data. Most Certificate Authorities (CAs) that issue certificates choose relatively "weak" algorithms in order to bean-count compute cycles rather than provide extra security (for example, SHA-256 instead of SHA-512). In addition, trying to clairvoyantly guess algorithm deprecation times by setting expiration dates on the certificate is either dangerous in the case of a lengthy expiration date, or irritating in the case of a short expiration date. Alternatively, revoking individual, insecure certificates is tedious at best. Rather than trust CAs, rely on expiration dates, or revoke certificates, I decided instead to implement server-side checks of client certificate strength on my InspIRCd server; specifically, I implemented checks for the public key size and signature algorithms used in the certificates. This blog entry details the hacks methods I used in implementing the checks.

OpenSSL Data Structure Primer

First, a few notes on some of the data structures used in OpenSSL. An SSL_CTX object is basically a factory for individual SSL connections which are then represented by SSL objects. X.509 certificates are represented by, intuitively enough, X509 structures. Ciphers, hashes, and keys are represented by a set of abstract interfaces prefixed with EVP (EnVeloPe), thus the EVP_PKEY interface can be used to access the public key of a certificate; the other interfaces were not relevant to me.

ASN.1 Primer

X.509 makes use of an, in my opinion, labyrinthian, standard known as Abstract Syntax Notation One (ASN.1). A full description is outside the scope of both this article and my mortal understanding; however, it has some very useful properties that I was able to make use of. ASN.1 objects, at least in OpenSSL, have a Numeric ID (NID), Short Name (SN), and Long Name (LN), and OpenSSL provides an API that easily translates between these. The function OBJ_txt2nid() translates the specified string, which can be either a Short Name or a Long Name, to a NID while the OBJ_nid2ln() and OBJ_nid2sn() functions can turn the specified NID into either a Long Name or a Short Name, respectively. The relevant object definitions can be found within your system include directory (assuming development headers are installed) under openssl/obj_macs.h.

These translation functions made it possible for me to translate from arbitrary user-defined strings in a configuration file to relevant OpenSSL objects, without having to write some unmaintainable kludge that maps configuration options to objects. Note, however, that not all the names were exactly what I expected. For example, rsa and rsaEncryption refer to two different objects. A thorough Internet search reveals that the two objects are used in different contexts; sadly, the question's answer doesn't seem to be quite correct as I was getting an rsaEncryption object from an X.509 certificate despite the answer stating it was for PKCS1 objects. The naming thus seems rather idiosyncratic, but rolling with it seems to work well enough.

Implementation

The first step was to get the SSL object for the connection; this was provided to me by the program. Likewise, the program then called SSL_get_peer_certificate() on the SSL object in order to get the client's certificate (note that, in this context, the client is the peer of the server, as the client is on the other side of the connection) as an X509* object. From here the methods to verify the key size and to verify the signature algorithm on the certificate diverged.

Verifying the Key Size

In order to verify the key size, the first step was to get the public key from the X509* object by using the X509_get_pubkey() function. The next step was to get the type of the public key before comparing it to the respective minimum size for that key type (or failing if the key type was not found); this was be done by calling EVP_PKEY_id() in order to return the type as a NID (note that the EVP_PKEY_RSA family of macros, which are used to identify key types, are actually defined based on their respective NIDs). Versions of OpenSSL prior to the 1.1 series allow accessing the type member of the EVP_PKEY object directly, and confusingly enough, the EVP_PKEY_type() function serves an entirely different purpose than returning the key type as a NID, thus EVP_PKEY_id() had to be used. After the key type had been retrieved the size of the key was retrieved via the EVP_PKEY_bits() function, which returned the key size in bits. Confusingly enough, there is also an EVP_PKEY_size() function which serves a different purpose; it returns, in bytes, the maximum size of a signature that can be created by the specified key and is meant to be used when allocating memory, but that number, depending on key type, may have nothing to do with key size, so I had to use EVP_PKEY_bits() instead. Thus, using EVP_PKEY_id() and EVP_PKEY_bits(), I was able to verify key type and size.

Verifying the Signature Algorithm

Verifying the certificate signature algorithm was much more straightforward than verifying the key type and size. Getting the signature algorithm NID can be done by simply calling X509_get_signature_nid(). That's it. No weird gotchas like in the previous subsection. Phew.

Verifying a Chain of Peer Certificates

Astute readers may have noticed the singular form of SSL_get_peer_certificate(), which is a problem, as the client may in fact present multiple certificates in the form of a certificate chain to the server, and it would be silly to check only one of the certificates in the chain for proper strength. Thankfully, the function SSL_get_peer_cert_chain() can be used to retrieve the other certificates (but not the last certificate, the one returned by SSL_get_peer_certificate()) in the chain as a STACK_OF(X509)* objects that can be iterated through and checked one-by-one for meeting proper strength requirements (use of OpenSSL's STACK_OF API is outside the scope of this document).

Figure: Certificate chain where A is a trusted CA of the server and the client must send its certificate C followed by the intermediate certificate B.

After implementing a check for the entire peer certificate chain, my next problem was testing verification of the chain. Consider the chain A -> B -> C, where A is trusted by the server and B and C are sent by the client. I had been using OpenSSL's s_client command for testing, so the obvious approach was to place B and C into the file passed via the -cert argument; the first problem with this approach was that the certificates must be in a certain order, specifically, the sender's certificate must come first, followed by the certificate certifying the sender's certificate, and so on (see the RFC). In this case that meant sending C followed by B; specifying the certificates in the wrong order (B followed by C) was obvious, because s_client tried to use C's private key to verify B, which, of course, failed. Putting them in the correct order, however, sent C while silently ignoring B. After much gnashing of teeth, I decided to look into the source code, and, while combing through the parsing of the -cert option noticed an undocumented (yes, not even in the man pages nor -help output) -cert_chain option. After looking into it I ran the s_client command again with C sent via the -cert option and B sent via the -cert_chain option and was able to actually send a certificate chain to the server, and sanity was once again restored to my life.

Testing

Manually verifying changes is both a pain and error-prone, so I created a bash script in order to automatically test strong cert verification for InspIRCd on my system. The tests check that the program runs without strong verification enabled, then turns verification on and checks that users can connect when their certificates use proper key type+sizes and algorithms, then checks that errors occur when invalid configuration options (such as non-existent algorithms) are specified, then finally checks that verification works correctly when the client passes a chain of certificates. It's not particularly exciting, but it gives peace of mind when rebasing on top of the latest stable release and when porting to newer versions of both OpenSSL and InspIRCd.

Future Work

After adding this feature I managed to find a series of functions that appeared as if they might do my heavy lifting for me. The functions, found under the include directory as ssl/ssl.h were SSL_set1_sigalgs_list(), SSL_set1_client_sigalgs_list(), and a few similar variations. Sadly, testing these show that they do not actually have the same functionality, although the signature algorithm list displayed after connecting via s_client was indeed different; presumably the signature algorithm is for session negation rather than peer certificate verification, but I am not yet sure as I have not yet Read The "Friendly" Manual RFC.

The online man pages for the function also mentioned an SSL_CONF API, which might make option parsing easier; regardless, I didn't use it this time.