Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ingress controller TLS with wildcard certificate #242

Closed
janhommes opened this issue Feb 7, 2017 · 18 comments
Closed

Ingress controller TLS with wildcard certificate #242

janhommes opened this issue Feb 7, 2017 · 18 comments

Comments

@janhommes
Copy link

janhommes commented Feb 7, 2017

I try to add a wildcard certificate as the default tls certificate but I getting the following error message:

2017-02-07T09:12:39.341667667Z W0207 09:12:39.341613       5 queue.go:87] 
requeuing default/ingress-ssl, err asn1: structure error: tags don't match
(16 vs {class:0 tag:2 length:1 isCompound:false}) 
{optional:false explicit:false application:false defaultValue:<nil> tag:<nil> 
stringType:0 timeType:0 set:false omitEmpty:false} tbsCertificate @2

The same message I get if I use it only on a ingress rule but not if I use a self-signed cert. This certificate works in another nginx installation. Is this a bug or is my cert misconfigured?

@cmluciano
Copy link
Contributor

May be dup of #195

@janhommes
Copy link
Author

Mh, I don't think it's a duplicate because I am using it as a default certificate with:

        args:
        - /nginx-ingress-controller
        - --default-backend-service=$(POD_NAMESPACE)/default-http-backend
        - --default-ssl-certificate=default/ingress-ssl

What exactly is asn1 used for? Is it used in the controller to validate the certificate or is it used for something else?

@rikatz
Copy link
Contributor

rikatz commented Feb 8, 2017

@janhommes What's the structure of your cert? Can you post the X509 PEM (only the public key) somewhere, so I can take a deeper look?

Thanks!

@janhommes
Copy link
Author

the structure is the following:

-----BEGIN RSA PRIVATE KEY-----
[...]
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
MIIFZDCCBEygAwIBAgIQXrqpJFtqo2CeoIkz5viEejANBgkqhkiG9w0BAQsFADBC
MQswCQYDVQQGEwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEbMBkGA1UEAxMS
UmFwaWRTU0wgU0hBMjU2IENBMB4XDTE2MTExODAwMDAwMFoXDTE3MTExODIzNTk1
OVowFjEUMBIGA1UEAwwLKi5ibG9nbm8uZGUwggEiMA0GCSqGSIb3DQEBAQUAA4IB
DwAwggEKAoIBAQCVfo0nqPu8OprDOW6ENaIJDYT9wpbeVvO1dTXKW3ccuOltZ7tW
Hlef8wYWkgv86oZpXsi+1luP8PdxFE2GG5wXW53a5GX2QolUIRPRdaOt+DcD+7UY
AaVyI46T/pE2JWOyut98oFLcqkokbbHn3/nc3lbszAi6C0H5Hu1cweCzAz35DGiK
28ijx31Xbum3v1/aedhCcJuAH1ktPUe0jmWw596VSXD0A/eMF+j+UFtj7X6iYnBt
g+jZhZrW4pifpnfoLzCDLh10npq02pthEClBf99jdKhKrShFqlzuLDQ/uHOK2NZ8
AZ2MrMfBfIAy3WxV0ZZ5ctQ0RqAom7K22qmtAgMBAAGjggKAMIICfDAhBgNVHREE
GjAYggsqLmJsb2duby5kZYIJYmxvZ25vLmRlMAkGA1UdEwQCMAAwKwYDVR0fBCQw
IjAgoB6gHIYaaHR0cDovL2dwLnN5bWNiLmNvbS9ncC5jcmwwbwYDVR0gBGgwZjBk
BgZngQwBAgEwWjAqBggrBgEFBQcCARYeaHR0cHM6Ly93d3cucmFwaWRzc2wuY29t
L2xlZ2FsMCwGCCsGAQUFBwICMCAMHmh0dHBzOi8vd3d3LnJhcGlkc3NsLmNvbS9s
ZWdhbDAfBgNVHSMEGDAWgBSXwidQnsLJ7AyIMsh8reKmAU/abzAOBgNVHQ8BAf8E
BAMCBaAwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMFcGCCsGAQUFBwEB
BEswSTAfBggrBgEFBQcwAYYTaHR0cDovL2dwLnN5bWNkLmNvbTAmBggrBgEFBQcw
AoYaaHR0cDovL2dwLnN5bWNiLmNvbS9ncC5jcnQwggEDBgorBgEEAdZ5AgQCBIH0
BIHxAO8AdgDd6x0reg1PpiCLga2BaHB+Lo6dAdVciI09EcTNtuy+zAAAAVh2ygRC
AAAEAwBHMEUCIQCdREfJxYoo/Xx2LKi7KulEFczPWAdEVLBt5NDg2D/53gIgA1ge
I5oIVNn8ihcGSPEy1LvQmwdvg4VN+38WKzoBgbEAdQDuS723dc5guuFCaR+r4Z5m
ow9+X7By2IMAxHuJeqj9ywAAAVh2ygSKAAAEAwBGMEQCIGR2frNmJB5d3XTggIPN
eiegmwAl/n7NT3MFIXzZtZJSAiAMaN6tvLj8W/ffrPc6W5jAQEXD5/DEwhKz0SzI
a3mUxjANBgkqhkiG9w0BAQsFAAOCAQEAaXnMUb5u+/3/yTy8hbjkzZLIIT4YDNy1
xuZh7eJtGQOOS5XTnXhapiiIiLG+RdadsawN1w1eWSQuPt5zaYICDhdAWJrtQDC9
WKuvitdRLyYCI3gIKqwp0oyw5C9IZXzGRgt9ajmm+QqWQx/v8DB6RogN459epI0z
dAfvQoTT8QnBfPDlZj2MMzye6TTXd3I+G3agpfaiSrThuJGLduMTJ4gzZiNw4Ebl
dznVlF90koC++vOYBukxzkKaj0u9VrK4DSNwzcqcwpp3en+Xdmzhi1MOTZ8OY+wB
Yej38+pxuz83PRW3qWhDwfpCE+dxq2uF7CzeQ1Z5xIvNj72llDnbvQ==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDVDCCAjygAwIBAgIDAjRWMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMDIwNTIxMDQwMDAwWhcNMjIwNTIxMDQwMDAwWjBCMQswCQYDVQQG
EwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEbMBkGA1UEAxMSR2VvVHJ1c3Qg
R2xvYmFsIENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEA2swYYzD9
9BcjGlZ+W988bDjkcbd4kdS8odhM+KhDtgPpTSEHCIjaWC9mOSm9BXiLnTjoBbdq
fnGk5sRgprDvgOSJKA+eJdbtg/OtppHHmMlCGDUUna2YRpIuT8rxh0PBFpVXLVDv
iS2Aelet8u5fa9IAjbkU+BQVNdnARqN7csiRv8lVK83Qlz6cJmTM386DGXHKTubU
1XupGc1V3sjs0l44U+VcT4wt/lAjNvxm5suOpDkZALeVAjmRCw7+OC7RHQWa9k0+
bw8HHa8sHo9gOeL6NlMTOdReJivbPagUvTLrGAMoUgRx5aszPeE4uwc2hGKceeoW
MPRfwCvocWvk+QIDAQABo1MwUTAPBgNVHRMBAf8EBTADAQH/MB0GA1UdDgQWBBTA
ephojYn7qwVkDBF9qn1luMrMTjAfBgNVHSMEGDAWgBTAephojYn7qwVkDBF9qn1l
uMrMTjANBgkqhkiG9w0BAQUFAAOCAQEANeMpauUvXVSOKVCUn5kaFOSPeCpilKIn
Z57QzxpeR+nBsqTP3UEaBU6bS+5Kb1VSsyShNwrrZHYqLizz/Tt1kL/6cdjHPTfS
tQWVYrmm3ok9Nns4d0iXrKYgjy6myQzCsplFAMfOEVEiIuCl6rYVSAlk6l5PdPcF
PseKUgzbFbS9bZvlxrFUaKnjaZC2mqUPuLk/IH2uSrW4nOQdtqvmlKXBx4Ot2/Un
hw4EbNX/3aBd7YdStysVAq45pmp06drE57xNNB6pXE0zX5IJL4hmXXeXxx12E6nV
5fEWCRE11azbJHFwLJhWC9kXtNHjUStedejV0NxPNO3CBWaAocvmMw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIETTCCAzWgAwIBAgIDAjpxMA0GCSqGSIb3DQEBCwUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMTMxMjExMjM0NTUxWhcNMjIwNTIwMjM0NTUxWjBCMQswCQYDVQQG
EwJVUzEWMBQGA1UEChMNR2VvVHJ1c3QgSW5jLjEbMBkGA1UEAxMSUmFwaWRTU0wg
U0hBMjU2IENBMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAu1jBEgEu
l9h9GKrIwuWF4hdsYC7JjTEFORoGmFbdVNcRjFlbPbFUrkshhTIWX1SG5tmx2GCJ
a1i+ctqgAEJ2sSdZTM3jutRc2aZ/uyt11UZEvexAXFm33Vmf8Wr3BvzWLxmKlRK6
msrVMNI4/Bk7WxU7NtBDTdFlodSLwWBBs9ZwF8w5wJwMoD23ESJOztmpetIqYpyg
C04q18NhWoXdXBC5VD0tA/hJ8LySt7ecMcfpuKqCCwW5Mc0IW7siC/acjopVHHZD
dvDibvDfqCl158ikh4tq8bsIyTYYZe5QQ7hdctUoOeFTPiUs2itP3YqeUFDgb5rE
1RkmiQF1cwmbOwIDAQABo4IBSjCCAUYwHwYDVR0jBBgwFoAUwHqYaI2J+6sFZAwR
fap9ZbjKzE4wHQYDVR0OBBYEFJfCJ1CewsnsDIgyyHyt4qYBT9pvMBIGA1UdEwEB
/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgEGMDYGA1UdHwQvMC0wK6ApoCeGJWh0
dHA6Ly9nMS5zeW1jYi5jb20vY3Jscy9ndGdsb2JhbC5jcmwwLwYIKwYBBQUHAQEE
IzAhMB8GCCsGAQUFBzABhhNodHRwOi8vZzIuc3ltY2IuY29tMEwGA1UdIARFMEMw
QQYKYIZIAYb4RQEHNjAzMDEGCCsGAQUFBwIBFiVodHRwOi8vd3d3Lmdlb3RydXN0
LmNvbS9yZXNvdXJjZXMvY3BzMCkGA1UdEQQiMCCkHjAcMRowGAYDVQQDExFTeW1h
bnRlY1BLSS0xLTU2OTANBgkqhkiG9w0BAQsFAAOCAQEANevhiyBWlLp6vXmp9uP+
bji0MsGj21hWID59xzqxZ2nVeRQb9vrsYPJ5zQoMYIp0TKOTKqDwUX/N6fmS/Zar
RfViPT9gRlATPSATGC6URq7VIf5Dockj/lPEvxrYrDrK3maXI67T30pNcx9vMaJR
BBZqAOv5jUOB8FChH6bKOvMoPF9RrNcKRXdLDlJiG9g4UaCSLT+Qbsh+QJ8gRhVd
4FB84XavXu0R0y8TubglpK9YCa81tGJUheNI3rzSkHp6pIQNo0LyUcDUrVNlXWz4
Px8G8k/Ll6BKWcZ40egDuYVtLLrhX7atKz4lecWLVtXjCYDqwSfC2Q7sRwrp0Mr8
2A==
-----END CERTIFICATE-----

I tested it with traefik-ingress-controller and there it is working.

@rikatz
Copy link
Contributor

rikatz commented Feb 9, 2017

@janhommes it seems to be something with EC (elliptic curves) certificates (i guess, not sure).

What's the version of ingress controller you've used? Maybe this was compiled with go 1.6. Need to check if compiling with go 1.7 solves the problem.

@janhommes
Copy link
Author

@rikatz I used gcr.io/google_containers/nginx-ingress-controller:0.9.0-beta.1 but tested also version 0.8.3, both the same problem.

@rikatz
Copy link
Contributor

rikatz commented Feb 9, 2017

@janhommes I've created a nginx ingress controller based on 0.9.0-beta.1, but compiled with go1.7.4

Can you test this, and check if the problem persists?

The image is rpkatz/nginx-ingress-controller-go17:0.9.0-beta1

@janhommes
Copy link
Author

@rikatz sorry, problem still exist. Interesting is that it shows the above error for every certificate in my cluster, even when I don't use it in an ingress.

@rikatz
Copy link
Contributor

rikatz commented Feb 28, 2017

@janhommes reading here some other issues in other software, a question: Is your private key encrypted (requesting password)?

@rikatz
Copy link
Contributor

rikatz commented Feb 28, 2017

@janhommes I could reproduce this here with a simple program trying to Parse X509 certificate, but with the key BEFORE the cert. Don't know if this still happens on a valid kubernetes program, so I'll keep looking at this trying to reproduce and see if this is a problem with encrypted private key or with the certificate concatenation order.

@janhommes
Copy link
Author

@rikatz My private key is not encrypted. It's just a plain textfile in the format above. So you mean I can get around this problem by changing the order of the key and certificate in the file?

I switched to taerefik and there the cert is working. But I prefer to use the nginx controller.

@rikatz
Copy link
Contributor

rikatz commented Mar 1, 2017 via email

@rikatz
Copy link
Contributor

rikatz commented Mar 1, 2017

@janhommes I've verified in your post earlier and yes, it's probably because RSA Key is appearing before the certs.

Just to know, how are you creating the TLS secret for the ingress controller?

According to ingress code, the cert is always written BEFORE the key, as the following:

https://github.com/kubernetes/ingress/blob/master/core/pkg/net/ssl/ssl.go#L46 and then https://github.com/kubernetes/ingress/blob/master/core/pkg/net/ssl/ssl.go#L54

It's strange that the certificate inside the ingress contains the key before the cert :)

I'll wait for your answer before commiting a new code like the following, as it seems the secret is not being created correctly :)

pemBlock, restBlock := pem.Decode(pemCerts)
if pemBlock == nil {
	return nil, fmt.Errorf("No valid PEM formatted block found")
}
for pemBlock.Type != "CERTIFICATE" {
	pemBlock, restBlock = pem.Decode(restBlock)
}

Thanks!

@janhommes
Copy link
Author

@rikatz Now its working! Thanks.

I was using a .crt with a private key included (as shown here).

My tls.crt was looking like this:
-----BEGIN RSA PRIVATE KEY-----
(Your Private Key: your_domain_name.key)
-----END RSA PRIVATE KEY-----
-----BEGIN CERTIFICATE-----
(Your Primary SSL certificate: your_domain_name.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Intermediate certificate: DigiCertCA.crt)
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
(Your Root certificate: TrustedRoot.crt)
-----END CERTIFICATE-----

And my tls.key:
-----BEGIN RSA PRIVATE KEY-----
(Your Private Key: your_domain_name.key)
-----END RSA PRIVATE KEY-----

I removed the RSA PRIVATE KEY part in the tls.crt and now the ingress controller is working. I used this command to create it: kubectl create secret tls ingress-ssl --key tls.key --cert tls.crt

Maybe it is just because I missunderstand the whole .crt, .pem, .key stuff. But for example nginx support this format, maybe the ingress controller should be flexible enougth to support this format.

Thanks for you help!

@rikatz
Copy link
Contributor

rikatz commented Mar 1, 2017

@janhommes Glad to hear this :)

here is an example of the creation of this certs also.

As NGINX supports this format (actually, the final file created in the ingress-controller container is a concatenation of cert and key file), it's necessary to make sure ingress is universal.

Anyway, I think it's a good idea to verify if the generated file starts with the 'BEGIN CERTIFICATE' and give user a warning if it doesn't find what is expected, instead of issuing a generic Go error.

Will open a PR to make at least this verification.

@rikatz
Copy link
Contributor

rikatz commented Mar 21, 2017

@janhommes @aledbf I think this one can be closed :)

@aledbf
Copy link
Member

aledbf commented Mar 26, 2017

@janhommes Please reopen if you still have issues

@aledbf aledbf closed this as completed Mar 26, 2017
@sht5
Copy link

sht5 commented Nov 22, 2018

@rikatz your a rock star!
i was working with Traefik and moved to Nginx and couldn't understand what broke.
found this through google
thanks for helping!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants