x509: certificate signed by unknown authority

[kcyea]

Hi All,

I am trying to run the exporter by using Docker command to extract metrics from our confluent cloud setup.

docker run \
  -e CCLOUD_API_KEY=$CCLOUD_API_KEY \
  -e CCLOUD_API_SECRET=$CCLOUD_API_SECRET \
  -e CCLOUD_CLUSTER=lkc-abc123 \
  -p 2112:2112 \
  dabz/ccloudexporter:latest

But I am getting the following error:

{
  "error": "Get \"https://api.telemetry.confluent.cloud/v2/metrics/cloud/descriptors/resources\": x509: certificate signed by unknown authority",
  "level": "fatal",
  "msg": "HTTP query for the descriptor endpoint failed",
  "time": "2021-12-08T08:42:53Z"
}

Is it due to we enable the X.509 certificate at confluent cloud? Anyone know how to solve it? Appreciate your help. Thanks.

[Dabz]

Hi @kcyea , somehow I am not able to reproduce locally this issue.
If I recall properly, Confluent Cloud changed the root CA recently, this change might have impacted some users.

Let me double check and check if the new root CA could have impacted the exporter.

[Dabz]

By quickly looking at the TLS certificates bundled with the exporter, it looks valid. It includes Subject: C = US, O = Amazon, CN = Amazon Root CA 1 that is used as a root CA.

Are you using a HTTPS proxy? If not, could you provide the output of openssl s_client -connect api.telemetry.confluent.cloud:443 ?

[kcyea]

Hi @Dabz, thanks for your prompt response. By the way, how do I check if I am using the HTTPS proxy? Our confluent cloud has a self signed certificate, will that caused an issue?

[kcyea]

By running the docker build using the Dockerfile, it got the following error:

Sending build context to Docker daemon  342.5kB
Step 1/9 : FROM golang:1.14 AS builder
 ---> 21a5635903d6
Step 2/9 : COPY . /src
 ---> a2c57fa4ecdb
Step 3/9 : WORKDIR /src
 ---> Running in 83f65b2d3af7
Removing intermediate container 83f65b2d3af7
 ---> ee7a58f75911
Step 4/9 : RUN CGO_ENABLED=0 GOOS=linux go install -ldflags "-X github.com/Dabz/ccloudexporter/cmd/internal/collector.Version=$(git rev-parse --short HEAD)" ./...
 ---> Running in ba1ee90eb393
go: github.com/prometheus/[email protected]: Get "https://proxy.golang.org/github.com/prometheus/client_golang/@v/v1.5.1.mod": x509: certificate signed by unknown authority
The command '/bin/sh -c CGO_ENABLED=0 GOOS=linux go install -ldflags "-X github.com/Dabz/ccloudexporter/cmd/internal/collector.Version=$(git rev-parse --short HEAD)" ./...' returned a non-zero code: 1

By running openssl s_client -connect api.telemetry.confluent.cloud:443, it gave me "Connected" status with some information on our certificate.

[Dabz]

I am not sure why the docker build failed (I assume because $(git rev-parse --short HEAD) failed). Could you share the output of the openssl s_client -connect api.telemetry.confluent.cloud:443?

[kcyea]

Hi @Dabz, I think I couldn't share it because it contains the certificate information, sorry about that. But i had mock the information and the output is similar to something like the following, not sure if you able to track something?

depth=2 C = US, O = MyComp, CN = MyComp Root CA
verify error:num=19:self signed certificate in certificate chain
verify return:1
depth=2 C = US, O = MyComp, CN = MyComp Root CA
verify return:1
depth=1 C = US, CN = mycomp-trusted.prod.com
verify return:1
depth=0 CN = *.telemetry.confluent.cloud
verify return:1
---
Certificate chain
 0 s:CN = *.telemetry.confluent.cloud
   i:C = US, CN = mycomp-trusted.prod.com
 1 s:C = US, CN = mycomp-trusted.prod.com
   i:C = US, O = MyComp, CN = MyComp Root CA
 2 s:C = US, O = MyComp, CN = MyComp Root CA
   i:C = US, O = MyComp, CN = MyComp Root CA
---
Server certificate
-----BEGIN CERTIFICATE-----
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
-----END CERTIFICATE-----
subject=CN = *.telemetry.confluent.cloud

issuer=C = US, CN = mycomp-trusted.prod.com

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: XXXX, P-256, 256 bits
---
SSL handshake has read 4340 bytes and written 457 bytes
Verification error: self signed certificate in certificate chain
---
New, TLSv1.2, Cipher is xxxxx-RSA-AES128-xxx-SHA256
Server public key is 2048 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : xxxxx-RSA-AES128-xxx-SHA256
    Session-ID: xxx
    Session-ID-ctx:
    Master-Key: xxx
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1638968682
    Timeout   : 7200 (sec)
    Verify return code: 19 (self signed certificate in certificate chain)
    Extended master secret: no

[Dabz]

Hi, the problem is that there is a proxy between the exporter and the Metrics API endpoint.
As the certificate used by the proxy is not trusted by the exporter, you get a X509 exception.

The only way to fix it is to trust your custom CA in the exporter, to do that, you will need to add your certificate in https://github.com/Dabz/ccloudexporter/blob/master/Dockerfile#L9 and rebuild the image. You could also see if there is a way to bypass the proxy.

I hope that's clear

[kcyea]

Thanks for your explanation @Dabz . I will try to add the certificate but rebuild the image by using docker build?

[Dabz]

Yeah, you would need to do a docker build -t dabz/ccloudexporter:latest . (or equivalent)

[kcyea]

Thanks @Dabz. I tried to rebuild the docker and it showed me the error I showed to you earlier.

go: github.com/prometheus/[email protected]: Get "https://proxy.golang.org/github.com/prometheus/client_golang/@v/v1.5.1.mod": x509: certificate signed by unknown authority
The command '/bin/sh -c CGO_ENABLED=0 GOOS=linux go install -ldflags "-X github.com/Dabz/ccloudexporter/cmd/internal/collector.Version=$(git rev-parse --short HEAD)" ./...' returned a non-zero code: 1

It seems that it related to client_golang as well :(

[Dabz]

oh yeah, goland is failing to download dependencies due to the same x509 issue. I am not sure how to bypass this issue .
Maybe try adding the certificate BEFORE compiling? ADD docker/telemetry-confluent-cloud-chain.pem /etc/ssl/certs/

[kcyea]

Yeah, it is not helping even I switch the sequence. Seem not something that easy to fix :)

[Dabz]

It looks like golang is fetching TLS trusted certs from: https://stackoverflow.com/questions/40051213/where-is-golang-picking-up-root-cas-from . You could also build your image in a non-proxy environment then export/import it to your env