SNI is getting dropped when routing Cross Cluster TCP with tls passthrough between Edge and service clusters

[rsvenkatesh]

We are trying out a cross cluster Contour deployment with one deployment at the Edge and one Contour deployment per client cluster.
This deployment works as expected for most use cases for http/https as long as we have Host header rewritten in the httpproxy of the Edge eg:.
requestHeadersPolicy:
set:
- name: Host
value: echocluster.vdp-dev.vmware.com
However, routing on TCP with tls passthrough gets a SSL_ERROR_SYSCALL for curl request on the TCP fqdn.
Eg: curl -v https://tcpecho.vdp-dev.vmware.com/health.
I think the issue can be solved if there is a way to inject SNI equivalent to the requestHeadersPolicy in http . Wondering if anyone has faced and solved this use case before?

Service and HTTPproxy config details:

Edge cluster GW to Tenanat cluster GW

 apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"projectcontour.io/upstream-protocol.tls":"443,https"},"name":"contour-tenant-service-echo-two","namespace":"internal-edge"},"spec":{"externalName":"ac21a09e1281a4e26b619e7e349ffb55-fecd45979a3a85a6.elb.us-east-2.amazonaws.com","ports":[{"name":"https","port":443,"protocol":"TCP","targetPort":443}],"sessionAffinity":"None","type":"ExternalName"}}
    projectcontour.io/upstream-protocol.tls: 443,https
  creationTimestamp: "2020-05-05T21:47:20Z"
  name: contour-tenant-service-echo-two
  namespace: internal-edge
  resourceVersion: "169319659"
  selfLink: /api/v1/namespaces/internal-edge/services/contour-tenant-service-echo-two
  uid: 88ada13a-992a-4e14-9267-31b08b01742e
spec:
  externalName: ac21a09e1281a4e26b619e7e349ffb55-fecd45979a3a85a6.elb.us-east-2.amazonaws.com
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  sessionAffinity: None
  type: ExternalName
status:
  loadBalancer: {}

HTTPProxy :

  apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"projectcontour.io/v1","kind":"HTTPProxy","metadata":{"annotations":{},"labels":{"app":"echocluster"},"name":"echocluster-tcp-mgmt01","namespace":"internal-edge"},"spec":{"tcpproxy":{"requestHeadersPolicy":{"set":[{"name":"Host","value":"tcpecho.vdp-dev.vmware.com"}]},"services":[{"name":"contour-tenant-service-echo-two","port":443,"protocol":"tls"}]},"virtualhost":{"fqdn":"tcpecho.vdp-dev.vmware.com","tls":{"passthrough":true}}}}
  creationTimestamp: "2020-05-09T17:25:56Z"
  generation: 2
  labels:
    app: echocluster
  name: echocluster-tcp-mgmt01
  namespace: internal-edge
  resourceVersion: "171130972"
  selfLink: /apis/projectcontour.io/v1/namespaces/internal-edge/httpproxies/echocluster-tcp-mgmt01
  uid: 5552b643-384a-451b-bbda-a637d08a30dd
spec:
  tcpproxy:
    requestHeadersPolicy:
      set:
      - name: Host
        value: tcpecho.vdp-dev.vmware.com
    services:
    - name: contour-tenant-service-echo-two
      port: 443
      protocol: tls
  virtualhost:
    fqdn: tcpecho.vdp-dev.vmware.com
    tls:
      passthrough: true
status:
  currentStatus: valid
  description: valid HTTPProxy

---

Tenant Cluster GW to upstream service

apiVersion: v1
kind: Service
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"labels":{"app":"echocluster"},"name":"echocluster-tcp-tls","namespace":"projectcontour"},"spec":{"ports":[{"port":8443,"protocol":"TCP","targetPort":8083}],"selector":{"app":"echocluster"},"sessionAffinity":"None","type":"ClusterIP"}}
  creationTimestamp: "2020-05-08T23:22:05Z"
  labels:
    app: echocluster
  name: echocluster-tcp-tls
  namespace: projectcontour
  resourceVersion: "85546672"
  selfLink: /api/v1/namespaces/projectcontour/services/echocluster-tcp-tls
  uid: 8eeb585f-d8a9-4a8f-8b7f-d3c0d96d6ff4
spec:
  clusterIP: 100.70.143.12
  ports:
  - port: 8443
    protocol: TCP
    targetPort: 8083
  selector:
    app: echocluster
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

HTTPProxy

apiVersion: projectcontour.io/v1
kind: HTTPProxy
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"projectcontour.io/v1","kind":"HTTPProxy","metadata":{"annotations":{},"labels":{"app":"echocluster"},"name":"echocluster-tcp","namespace":"projectcontour"},"spec":{"tcpproxy":{"services":[{"name":"echocluster-tcp-tls","port":8443}]},"virtualhost":{"fqdn":"tcpecho.vdp-dev.vmware.com","tls":{"passthrough":true}}}}
  creationTimestamp: "2020-05-08T23:22:05Z"
  generation: 2
  labels:
    app: echocluster
  name: echocluster-tcp
  namespace: projectcontour
  resourceVersion: "85546676"
  selfLink: /apis/projectcontour.io/v1/namespaces/projectcontour/httpproxies/echocluster-tcp
  uid: adaa571b-4868-4070-a35d-8eda62adb127
spec:
  tcpproxy:
    services:
    - name: echocluster-tcp-tls
      port: 8443
  virtualhost:
    fqdn: tcpecho.vdp-dev.vmware.com
    tls:
      passthrough: true
status:
  currentStatus: valid
  description: valid HTTPProxy

[stevesloka]

The issue here is that requests are getting routed to example.com then proxied to another instance of Contour in a different cluster at foo.com. In the upstream cluster, the route is set to example.com and Envoy 404's since it doesn't have a match since it receives foo.com.

On the HTTP-side, we have a Host rewrite to foo.com such that the SNI on the upstream request matches, but this doesn't exist for TCP.

I think the SNI dynamic forward proxy might work (https://www.envoyproxy.io/docs/envoy/latest/configuration/listeners/network_filters/sni_dynamic_forward_proxy_filter) (envoyproxy/envoy#10448), but it's in Envoy v1.15 and also needs the v3 xDS API.

[jpeach]

The issue here is that requests are getting routed to example.com then proxied to another instance of Contour in a different cluster at foo.com. In the upstream cluster, the route is set to example.com and Envoy 404's since it doesn't have a match since it receives foo.com.

Can you show me where this is happening in he config above?

[xaleeks]

Closing due to lack of response. Feel free to reopen if this is still an issue