Rolling CAs not working in 2.9

[AndreasBergmeier6176]

Report

We have a working configuration for 2.8.
When we try to use the same configuration with 2.9 accessing the bootstrap-servers with our CAs fail:

We use:

spec:
  triggers:
  - authenticationRef:
      name: creds

with:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: creds
  namespace: ns
spec:
  secretTargetRef:
  - key: tls
    name: secret
    parameter: tls
  # list of all cas that are currently active
  - key: root-ca-new.pub.crt
    name: tracking-cas
    parameter: ca
  - key: foo.crt
    name: signing
    parameter: cert
  - key: private-key
    name: private-key
    parameter: key
  - key: password
    name: private-key
    parameter: keyPassword

Expected Behavior

Either it should work or at least the error should indicate what I need to change in the config.

Actual Behavior

Error is:

2023-03-07T12:07:47Z	ERROR	scalehandler	error resolving auth params	{"scalerIndex": 0, "object": {"apiVersion": "keda.sh/v1alpha1", "kind": "ScaledObject", "namespace": "analyzing-joreetz", "name": "controller"}, "error": "error creating kafka client: kafka: client has run out of available brokers to talk to: 3 errors occurred:\n\t* x509: certificate signed by unknown authority\n\t* x509: certificate signed by unknown authority\n\t* x509: certificate signed by unknown authority\n"}

Steps to Reproduce the Problem

Use a config with multiple CAs in a file.

Logs from KEDA operator

Stacktrace to above error message is:

github.com/kedacore/keda/v2/pkg/scaling.(*scaleHandler).buildScalers
/workspace/pkg/scaling/scale_handler.go:543

KEDA Version

2.9.3

Kubernetes Version

1.25

Platform

Google Cloud

Scaler Details

Kafka

Anything else?

Seems like multiple CAs were supported before but now just one is picked.

[JorTurFer]

Hi,
I have found the cause of it. Before this commit, we didn't validate the server certificate 4a90de8#diff-297745b4ce02635e0ca7d26e79c91e8a1418fabfb6c0f21cab249d6a259640c2 ignoring it always.
This is really insecure and doesn't make sense considering that we are also setting the trusted CAs in the line above.

I'm going to test what happens if the same file contains multiple certificates to be sure if the problem is related with that or not (because it's important to know it). v2.10 will allow setting multiple CA at KEDA level, so I guess that it will solve the issue just providing you already existing secret if that problem is related with multi-certificates (at least to document this limitation for previous versions). I guess that the server is responding with a TLS cert signed by one registered CA, right?

[abergmeier]

Thanks for the effort.

v2.10 will allow setting multiple CA at KEDA level

Just to double-check. These CAs will then be used to communicate with the Kafka Bootstrap Servers?
Better yet - is there an integration test that validates that this scenario works and/or will continue to work for Kafka?

I guess that the server is responding with a TLS cert signed by one registered CA, right?

It does.

[JorTurFer]

Just to double-check. These CAs will then be used to communicate with the Kafka Bootstrap Servers?

Those CAs will be used to trust on server certificates. If the server certificates are signed by any registered CA, KEDA will accept them. I'd have to test if the server has to be signed by any register CA (being a child cert signed from a registered CA) or if it's possible to register as CA the exposed certificate itself (just registering the server certificate). I have already tested it and both are correct, at least with the new approach
In your case, what are you registering? The CA used for signing the server certs or just the server certs?

Better yet - is there an integration test that validates that this scenario works and/or will continue to work for Kafka?

Not for kafka explicitly because Kafka is more complicated to spin up, but there is an e2e test that tests it with a rest API exposing a self-signed certificate. Basically we register KEDA adding the custom CA and then we use the metrics-api scaler to validate that KEDA trust in that self-signed CA

BTW, and this is important, using KEDA v2.8 you are not validating the CAs at all. I mean, the code that we fixed was ignoring the certificate validation (that's why it works). I clarify it because maybe it's a security problem as you are skipping the certificate validation (or maybe not, but I prefer to clarify this point)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed due to inactivity.