Issue with IBM JDK 7 or 8 resulting SSL handshake failure

[bxu1999]

Trying to upgrade to java-cloudant library 2.3.0 with the okhttp dependency, and encountered a weird SSL handshake failure. For testing purpose, I created a simple test Java program that just connects to my Cloudant account and list all of my databases. Please see the attached simple code below. It works perfectly fine with Oracle Java 7 or 8, but when running it in IBM JDK 7 or 8, it fails and produces an SSL handshake failures. Attached the simple java class file here:
ListDBTest.zip
You need to enter your Cloudant account, username, and password in the file to run it, and also use -Djavax.net.debug=all to see all the SSL handshake trace if needed.

Some excerpt of error log:
......
main, handling exception: javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
main, called close()
main, called closeInternal(true)
Feb 25, 2016 9:38:01 AM com.cloudant.http.interceptors.CookieInterceptor interceptResponse
SEVERE: Failed to get response code from request
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at com.ibm.jsse2.j.a(j.java:33)
at com.ibm.jsse2.j.a(j.java:31)
at com.ibm.jsse2.qc.b(qc.java:190)
at com.ibm.jsse2.qc.a(qc.java:456)
at com.ibm.jsse2.qc.h(qc.java:352)
at com.ibm.jsse2.qc.a(qc.java:523)
at com.ibm.jsse2.qc.startHandshake(qc.java:730)
at com.squareup.okhttp.Connection.connectTls(Connection.java:235)
at com.squareup.okhttp.Connection.connectSocket(Connection.java:199)
at com.squareup.okhttp.Connection.connect(Connection.java:172)
......

Now thinking of SSL handshake failures, one might think of invalid certificate and trust CA issues or cipher incompatibility, so I spent quite some time to trace the SSL hand shaking, did not find any such issue. Actually the first round SSL handshake was successful (from ClientHello, ServerHello, ServerHelloDone, to Finished), and the client and Cloudant was communicating with app data on user authentication and setting cookie. But in the IBM JDK 7 or 8 case, the SSL connection then broken down after that at the CookieInterceptor class (the client would re-initiate an SSL hand-shake, but the server will no longer respond as it thinks there is already an established communication session), while running under Oracle Java 7 or 8 it continues and works fine.

So my guess is that somehow the interceptor mechanism (possibly the CookieInterceptor) somehow disrupts the normal SSL flow in the IBM JDK 7 or 8 case, resulting the problem, on the other hand it does not cause any issue in Oracle Java 7 or 8.

Please let me know if there are some configuration changes or other changes I can apply to prevent this issue. Your help is greatly appreciated.

Thanks much.
Bing.

[ricellis]

@bxu1999 thanks for reporting this - I'm looking into it. Do you see the issue without the okhttp dependency?

[ricellis]

I was able to reproduce the issue, but only when using the okhttp dependency. It seems that this problem is caused by some interaction between okhttp and the IBM JDKs. You should be able workaround by dropping the optional okhttp dependency.

[bxu1999]

@ricellis I've not tried without the okhttp dependency. (1) The questions I have are: what benefits do I lose if I don't include okhttp? and if I don't use okhttp, will I still have the connection pool capability available? The connection pool is what I want to utilize and very important as our eventual code is a web application as a server/service in WAS Liberty on Bluemix/Cloud Foundry. (2) Also how do I drop the okhttp dependency? From the build file, it says that compile requires okhttp, then how do I disable its use in my client application? (3) can you please look into fixing the issue as lost of WAS Liberty based applications are probably run with IBM JDK 7 or 8 as recommended.
Thanks much.
Bing.

[ricellis]

@bxu1999
1)
Under the covers the http requests go through the Java default HttpURLConnection. This does have connection pooling enabled by default and it uses a JVM wide connection pool, this is all documented in the javadoc.

Previous versions of java-cloudant allowed a per client instance connection pool and the primary reason for enabling the optional okhttp dependency was to maintain compatibility with that option after we switched away from Apache HTTP client to HttpURLConnection.

The build file you are referring to is for the compilation of our java-cloudant cloudant-client archives, which does require the okhttp dependency for compilation. Okhttp is not required for compiling your application. The dependency snippets for applications using our library both with and without the okhttp dependency are documented in our README. Essentially if you have only this line you will have the dependency on our cloudant client without okhttp:

dependencies {
    compile group: 'com.cloudant', name: 'cloudant-client', version: '2.3.0'
}

3. Running on the IBM JDKs appears to work normally without okhttp and that would be my recommendation. I will nonetheless continue to try and identify the cause of the issue.

[bxu1999]

@ricellis yes, that's perfect. I was using an Eclipse web project so I added all those okhttp jar files into my application project as I thought they are also needed for my application as I'm building the java-cloudant project itself. After I removed them from my application, it works now. That's great.

Also you answered my question in the connection pool, i.e. not using okhttp does not impact our requirements at all.

So thanks so much for your quick response, and we are unblocked. Best luck in your effort fixing the okhttp and IBM JDK compatibility issue :-)

Thanks so much.

[ricellis]

I have confirmed that this javax.net.ssl.SSLHandshakeException happens with okhttp and IBM Java without java-cloudant involved - so we shouldn't need to make a fix here. There is, however, a potential workaround we could add to java-cloudant to configure the okhttp client with all the enabled protocols/ciphers instead of using its default ConnectionSpec.MODERN_TLS spec, but since IBM Java does support the necessary protocols and ciphers that change shouldn't be necessary and I await some feedback from IBM Java on this issue.

[bxu1999]

OK, sounds good. Thanks much for the update.

[ricellis]

The underlying cause of this issue is that okhttp filters the list of ciphers (based on the name returned by the JVM). The IBM Java returns the cipher suite names prefixed SSL_ whilst Oracle Java returns them prefixed TLS_. They are named TLS_ in RFC5246 and in okhttp. It seems there is no way to get the names returned with a TLS_ prefix in IBM Java. So our best option here is the workaround I described in #215 (comment) which will use a ConnectionSpec identical to the JVM default (i.e. the same as what would be used for HttpURLConnection in the case where okhttp is not being used) and any constraints on what cipher suites the client should or shouldn't accept can be configured consistently at the JVM level.

[iosnewbie2016]

Hi @ricellis - Am running into a similar issue when using okhttp with IBM. We use IBM WAS 8.5.5. I am not clear with your comment "So our best option here is the workaround I described in #215 (comment) which will use a ConnectionSpec identical to the JVM default (i.e. the same as what would be used for HttpURLConnection in the case where okhttp is not being used) and any constraints on what cipher suites the client should or shouldn't accept can be configured consistently at the JVM level.".

I need to use okhttp3, because I use another github library, which in turn uses this. Can you pls help me understand more on what can be done to fix this?

[ricellis]

@iosnewbie2016 this issue was resolved by implementing the workaround described above in java-cloudant's configuration of OkHttp to use the JVM's enabled cipher suite names instead of the restricted set of okhttp defaults. It all definitely worked after that change. If you are seeing a problem now please open a new issue completing the template to help us investigate. Thanks.

[iosnewbie2016]

Hi @ricellis - Am using a different library and not java-cloudant. Am receiving a similar error. So am trying to understand the nature of the changes done, so that I can try the similar ones in the library am using and see if it works. Thanks for your help.

[ricellis]

@iosnewbie2016 oh I see - well you can see the changes I made here:
#220

[csckcac]

I just comment this issue for adding notes about IBM Java 8.0 (in WAS 9) is using "SSL_" cipher suite name prefix instead of "TLS_"

You may be able to fix this problem by adding the following Generic JVM arguments
-Dhttps.protocols=TLSv1.2 -Djdk.tls.client.protocols=TLSv1.2 -Dcom.ibm.jsse2.overrideDefaultTLS=true