An Analysis of the OpenSSL SSL Handshake Error State Security Bypass (CVE-2017-3737)

OpenSSL is a widely used library for SSL and TLS protocol implementation that secures data using encryption and decryption based on cryptographic functions. However, a Security Bypass vulnerability – recently addressed in a patch by the OpenSSL Project –can be exploited to make vulnerable SSL clients or remote SSL servers send clean application data without encryption.

This Security Bypass vulnerability (CVE-2017-3737) is caused by an error when the SSL_read or SSL_write function handles an "error state" during an SSL handshake. In this paper the FortiGuard Labs team examines the root cause of this vulnerability.

The "error state" mechanism was introduced in OpenSSL beginning with version 1.0.2b, It is used to make OpenSSL move into an error state whenever a fatal error occurs during the SSL handshake that would fail if the SSL handshake continued. If SSL_read or SSL_write function is called directly, it checks the SSL handshake state and performs a new SSL handshake automatically if no handshake has been initiated. If a fatal error occurs during the SSL handshake, OpenSSL moves into the error state and returns an error message to the caller. However, the problem occurs if the caller doesn't check the error state and simply calls the SSL_read or SSL_write function again, because it then sends application data without encryption.   

The following code snippet was taken from OpenSSL 1.0.2m. (Comments added by me have been highlighted.)

ssl/s3_pkt.c:

638     int ssl3_write_bytes(SSL *s, int type, const void *buf_, int len)

639     {

640         const unsigned char *buf = buf_;

641         int tot;

642         unsigned int n, nw;

643     #if !defined(OPENSSL_NO_MULTIBLOCK) && EVP_CIPH_FLAG_TLS1_1_MULTIBLOCK

644         unsigned int max_send_fragment;

645     #endif

646         SSL3_BUFFER *wb = &(s->s3->wbuf);

647         int i;

648    

649         s->rwstate = SSL_NOTHING;

650         OPENSSL_assert(s->s3->wnum <= INT_MAX);

651         tot = s->s3->wnum;

652         s->s3->wnum = 0;

653    

654         if (SSL_in_init(s) && !s->in_handshake) { //checks to see if the SSL handshake state is initiated. The state will be SSL_ST_INIT the first time.

655             i = s->handshake_func(s);  //performs a new SSL handshake if no handshake has been initiated.

656             if (i < 0)

657                 return (i);

658             if (i == 0) {

659                 SSLerr(SSL_F_SSL3_WRITE_BYTES, SSL_R_SSL_HANDSHAKE_FAILURE);

660                 return -1;

661             }

662         }

ssl3_write_bytes() is called by the SSL_write function to send the application data. It checks the SSL handshake state and performs the SSL handshake if needed.

ssl/s3_clnt.c:

898     int ssl3_get_server_hello(SSL *s)

899     {

900         STACK_OF(SSL_CIPHER) *sk;

901         const SSL_CIPHER *c;

902         CERT *ct = s->cert;

903         unsigned char *p, *d;

904         int i, al = SSL_AD_INTERNAL_ERROR, ok;

….

1077        if (i < 0) {

1078            /* we did not say we would use this cipher */

1079            al = SSL_AD_ILLEGAL_PARAMETER;

1080            SSLerr(SSL_F_SSL3_GET_SERVER_HELLO, SSL_R_WRONG_CIPHER_RETURNED);

1081            goto f_err;  //a fatal error occurs

1082        }

….

1170     f_err:

1171        ssl3_send_alert(s, SSL3_AL_FATAL, al); //sends an SSL alert packet

1172     err:

1173        s->state = SSL_ST_ERR; //moves into an error state

1174        return (-1);

1175    }

We used a vulnerable SSL client as a target during the test. During the SSL handshake it received a malformed server hello message from the SSL server controlled by the attacker. ssl3_get_server_hello() is called to handle this server hello message and a fatal error occurs, causing OpenSSL to move into an error state by setting s->state from SSL_ST_INIT to SSL_ST_ERR.

ssl/s3_pkt.c:

638     int ssl3_write_bytes(SSL *s, int type, const void *buf_, int len)

639     {

640         const unsigned char *buf = buf_;

641         int tot;

642         unsigned int n, nw;

643     #if !defined(OPENSSL_NO_MULTIBLOCK) && EVP_CIPH_FLAG_TLS1_1_MULTIBLOCK

644         unsigned int max_send_fragment;

645     #endif

646         SSL3_BUFFER *wb = &(s->s3->wbuf);

647         int i;

648    

649         s->rwstate = SSL_NOTHING;

650         OPENSSL_assert(s->s3->wnum <= INT_MAX);

651         tot = s->s3->wnum;

652         s->s3->wnum = 0;

653    

654         if (SSL_in_init(s) && !s->in_handshake) { // SSL_in_init is called again to check the state

If the vulnerable SSL client doesn't check the error state and call SSL_write function to send application data again, ssl3_write_bytes() is called and uses SSL_in_init() to check the handshake state again.

include/openssl/ssl.h:

1749    # define SSL_in_init(a)                  (SSL_state(a)&SSL_ST_INIT) //s->state is now SSL_ST_ERR, and check returns are false.

This time the check fails, the SSL handshake is bypassed  and the application data will be sent without encryption.

The following traffic dump shows how the clean application data is sent:

During this attack, the attacker entices the vulnerable SSL client to connect to a malicious SSL server. The SSL client may bypass the handshake process and send the application data without encryption. The SSL server may also have the same vulnerability if SSL_read or SSL_write function is called directly.

NOTE: authentication is NOT required to exploit this vulnerability.

IPS Signature

FortiGuard released IPS signature OpenSSL.Handshake.Error.State.Security.Bypass to address this vulnerability.

Sign up for our weekly FortiGuard Labs intel briefs or to be a part of our open beta of Fortinet’s FortiGuard Threat Intelligence Service.