Kafka output and SSL/TLS

[elvarb]

I have set up SSL with Kafka 0.9, created the CA, certs and keys for 1/4 Kafka nodes. Created the Client cert/key and tested it from Logstash 5.0 beta and that works.

When using Telegraf I'm getting this error

2016/04/29 12:47:26 Attempting connection to output: kafka
2016/04/29 12:47:27 Failed to connect to output kafka, retrying in 15s, error was 'kafka: client has run out of available brokers to talk to (Is your cluster reachable?)'
2016/04/29 12:47:43 kafka: client has run out of available brokers to talk to (Is your cluster reachable?)

And this is my Telegraf config

[outputs.kafka]
    # URLs of kafka brokers
    brokers = ["confluent-1:9093"]

    ssl_ca = "ca.pem"
    ssl_cert = "elvarx1.pem"
    ssl_key = "elvarx1.key"
    insecure_skip_verify = true

    topic = "telegraf2"
    data_format = "influx"
    required_acks = 1
    max_retry = 3

The main difference here from Logstash is that I'm using the raw pem and key files here but with Logstash I'm using JKS stores created from the raw pem and key files.

[elvarb]

Seems that this is the issue I'm having, IBM/sarama#643

[elvarb]

Damn politics!

http://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#importlimits

So OpenJDK cant support higher ciphers

[elvarb]

It goes deeper
https://bugzilla.redhat.com/show_bug.cgi?id=1019554

[sparrc]

interesting, thanks for digging @elvarb, what's the conclusion? Don't run Kafka with Oracle Java 8?

[elvarb]

By default using Confluent 2.0 with Kafka 0.9 and the latest OpenJDK these ciphers are available. Found by using sslscan

 Supported Server Cipher(s):
   Accepted  TLSv1  256 bits  DHE-DSS-AES256-SHA
   Accepted  TLSv1  128 bits  DHE-DSS-AES128-SHA
   Accepted  TLSv1  128 bits  EDH-DSS-DES-CBC3-SHA
   Accepted  TLS11  256 bits  DHE-DSS-AES256-SHA
   Accepted  TLS11  128 bits  DHE-DSS-AES128-SHA
   Accepted  TLS11  128 bits  EDH-DSS-DES-CBC3-SHA
   Accepted  TLS12  256 bits  DHE-DSS-AES256-GCM-SHA384
   Accepted  TLS12  256 bits  DHE-DSS-AES256-SHA256
   Accepted  TLS12  256 bits  DHE-DSS-AES256-SHA
   Accepted  TLS12  128 bits  DHE-DSS-AES128-GCM-SHA256
   Accepted  TLS12  128 bits  DHE-DSS-AES128-SHA256
   Accepted  TLS12  128 bits  DHE-DSS-AES128-SHA
   Accepted  TLS12  128 bits  EDH-DSS-DES-CBC3-SHA

 Preferred Server Cipher(s):
   SSLv2  0 bits    (NONE)
   TLSv1  256 bits  DHE-DSS-AES256-SHA
   TLS11  256 bits  DHE-DSS-AES256-SHA
   TLS12  256 bits  DHE-DSS-AES256-GCM-SHA384

From the Golang docs these are available. https://golang.org/pkg/crypto/tls/

        TLS_RSA_WITH_RC4_128_SHA                uint16 = 0x0005
        TLS_RSA_WITH_3DES_EDE_CBC_SHA           uint16 = 0x000a
        TLS_RSA_WITH_AES_128_CBC_SHA            uint16 = 0x002f
        TLS_RSA_WITH_AES_256_CBC_SHA            uint16 = 0x0035
        TLS_RSA_WITH_AES_128_GCM_SHA256         uint16 = 0x009c
        TLS_RSA_WITH_AES_256_GCM_SHA384         uint16 = 0x009d
        TLS_ECDHE_ECDSA_WITH_RC4_128_SHA        uint16 = 0xc007
        TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA    uint16 = 0xc009
        TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA    uint16 = 0xc00a
        TLS_ECDHE_RSA_WITH_RC4_128_SHA          uint16 = 0xc011
        TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA     uint16 = 0xc012
        TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA      uint16 = 0xc013
        TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA      uint16 = 0xc014
        TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256   uint16 = 0xc02f
        TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 uint16 = 0xc02b
        TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384   uint16 = 0xc030
        TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 uint16 = 0xc02c

Still trying to get the extra ciphers from Oracle in.

[elvarb]

The extra ciphers probably only work with the Oracle JDK http://armoredbarista.blogspot.is/2013/10/how-to-use-ecc-with-openjdk.html

[elvarb]

I have been beating my head against the wall for the last days, no matter what I do I can not get Kafka to allow for other ciphers. Can't spend more time on this, but my results are that at least on Centos you can not use SSL when using Kafka from Go programs.

[elvarb]

Using any other provider than some default one Kafka uses is impossible, always says provider not found.

For example this

/bin/kafka-server-start /etc/kafka/server.properties --override ssl.provider=sun.security.ec.SunEC

[2016-05-02 13:48:48,252] FATAL [Kafka Server 11], Fatal error during KafkaServer startup. Prepare to shutdown (kafka.server.KafkaServer)
org.apache.kafka.common.KafkaException: org.apache.kafka.common.KafkaException: java.security.NoSuchProviderException: no such provider: sun.security.ec.SunEC
        at org.apache.kafka.common.network.SslChannelBuilder.configure(SslChannelBuilder.java:44)

[elvarb]

Created a ticket with the Kafka project https://issues.apache.org/jira/browse/KAFKA-3647

[elvarb]

Solution found,

When you create the keystore use -keyalg RSA, then you will get the right ciphers.