While a university student, I had a part-time job evolving a Mac point-of-sale application to use a client-server architecture. During the course of that project, I struggled to implement an early version of certificate pinning due to unexpected behavior with certificate verification in OpenSSL.

rgov

Nov 23, 2009

Weird OpenSSL behavior: VERIFY_PEER callback is returning 0, but the connection isn't terminating per the documentation.

After some investigation, I discovered that Apple had shipped a patch to OpenSSL intended to allow users to override certificate trust settings, but in the process had accidentally and entirely broken certificate verification.

I reported the vulnerability to Apple Product Security along with a hopeful inquiry about summer internships. They swiftly scheduled an interview and ultimately extended an internship offer, which led to a full-time position after graduation.

rgov

Dec 6, 2009

@AppleEngineer Does Apple ever have internship opportunities in the product security team?

Not wanting to jeopardize my internship, I never published the details of the vulnerability. Nearly 17 years later, my original write-up is reproduced below with minimal edits.

Timeline

• 2009-11-24: Initial discovery.
• 2010-01-14: Reported to Apple Product Security.
• 2010-01-15: Apple acknowledges issue; invites reporter to phone interview.
• 2010-01-20: Interview with Apple Product Security; Apple extends internship offer.
• 2010-11-10: Apple releases fix in Mac OS X 10.6.5, 300 days from initial report.

---

It appears that Apple has added undocumented new code to its distribution of the OpenSSL cryptographic library, libcrypto, to integrate trust settings from the user and system keychains into the process of verifying the peer certificate during a connection handshake.

The normal verification procedure looks something like this:

1. The certificate goes through a preverification process, which checks that it was issued by an application-supplied certificate authority, and also that it has not expired and has not been modified.

2. Optionally, an application-defined callback is fired which allows additional verification. Its return value overrides the preverification result.

3. If verification failed, the handshake ends and the connection is terminated immediately. Otherwise, the handshake continues.

The patch, dating back to April 2009 at least, modifies the procedure to insert a new step after the callback:

3. If verification has so far not succeeded, see if the certificate was signed by an authority trusted by the user's or the system's keychain.

4. If verification ultimately failed, the handshake ends and the connection is terminated. Otherwise, the handshake continues.

The insertion of the keychain step raises problems for applications which significantly rely on the callback to verify peer certificates. Take for instance a web browser, which must verify that the common name of a server certificate matches the domain name for the site. Due to Apple's changes, even if the callback rejected the certificate due to a mismatched domain name, verification would succeed so long as the certificate was signed by a certificate trusted by the system.

This design makes it trivial to get an application to blindly accept a certificate, as any certificate signed by an authority trusted by Mac OS X will pass verification.

Demonstration

You can download a sample program I've written to demonstrate the override, a very basic HTTPS server written in C whose verification callback intentionally rejects all peer certificates: Makefile · ssldemo.c

To start listening with the server,

$ make listen
[*] Initialized SSL library
[+] Loaded certificate and private key
[+] Installed peer certificate verification callback
[+] Set up socket to listen on https://localhost:8080

The Makefile will generate two client certificates, client.pem, which is self-signed, and signed.pem, which is signed by the former. We can check that the server accepts neither certificate as valid.

$ make connect
[-] Secure connection refused
$ make connect-signed
[-] Secure connection refused

Next, we can add the client.pem certificate to the user's keychain and mark the certificate as trusted. (This will prompt the user for his or her keychain password.) Afterwards, we can try to connect once again:

$ make trust
$ make connect
[+] Secure connection established
$ make connect-signed
[+] Secure connection established

This time, both connections bypass the rejection decreed by the verification callback, and successfully connect.

(Simply run make clean to remove the trusted certificate from your keychain.)

Faulty Error Check

Apple's patch is intended to only perform the keychain search if OpenSSL's error code indicates that the certificate's issuer was unknown:

/* Try OpenSSL, if we get a local certificate issue verify against trusted roots */
ret = X509_verify_cert_orig(ctx);
if (ret != 1 && (ctx->error & X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY)) {
    /* ... */
}

However, Apple uses the bitwise and (&) operator to test for the error condition, despite the error code not being a bitfield. Inadvertently, this means that the majority of OpenSSL error codes trigger the behavior; only a few do not:

• X509_V_OK
• X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT
• X509_V_ERR_UNABLE_TO_GET_CRL
• X509_V_ERR_CRL_SIGNATURE_FAILURE
• X509_V_ERR_CERT_NOT_YET_VALID
• X509_V_ERR_CERT_HAS_EXPIRED
• X509_V_ERR_CRL_NOT_YET_VALID
• X509_V_ERR_KEYUSAGE_NO_CERTSIGN

If your callback is affected by this behavior, you have the option of clearing the error value with X509_STORE_CTX_set_error() and thereby bypassing the unwelcome verification. Unfortunately, this may have unintended consequences with error handling, and possibly lead to other security flaws.

int verify_callback(int preverify_ok, X509_STORE_CTX *ctx)
{
    X509_STORE_CTX_set_error(ctx, X509_V_OK);
    /* ... */
}

Conclusion

It is understandable that Apple would like to make keychain trust settings "just work" in applications which do not use OS X's own security APIs. However, the haphazard way this patch was developed and the consequences of the changes are too great for it to go undocumented.

Until a less abrasive method of integrating the keychain with OpenSSL is developed, the changes should be reverted to the documented, dependable behavior.