Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] KerberosAuthenticationIT fails in CI #32498

Closed
danielmitterdorfer opened this issue Jul 31, 2018 · 5 comments · Fixed by #32514 or #32588
Closed

[CI] KerberosAuthenticationIT fails in CI #32498

danielmitterdorfer opened this issue Jul 31, 2018 · 5 comments · Fixed by #32514 or #32588
Assignees
@bizybot
Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) >test-failure Triaged test failures from CI

Comments

@danielmitterdorfer
Copy link
Member

Build failure link: https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.4+multijob-unix-compatibility/os=centos/7/console

./gradlew :x-pack:qa:kerberos-tests:integTestRunner \
  -Dtests.seed=96A4AD9F42092F61 \
  -Dtests.class=org.elasticsearch.xpack.security.authc.kerberos.KerberosAuthenticationIT \
  -Dtests.security.manager=true \
  -Dtests.locale=und \
  -Dtests.timezone=Etc/GMT-11

testLoginByKeytab and testLoginByUsernamePassword fail with:

KrbException: Identifier doesn't match expected value (906)

and also

KrbException: Server not found in Kerberos database (7) - LOOKING_UP_SERVER

This issue does not reproduce locally.
@danielmitterdorfer danielmitterdorfer added >test-failure Triaged test failures from CI :Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) labels Jul 31, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-security

danielmitterdorfer added a commit that referenced this issue Jul 31, 2018
@danielmitterdorfer
Mute KerberosAuthenticationIT
02e9388 
Relates #32498
danielmitterdorfer added a commit that referenced this issue Jul 31, 2018
@danielmitterdorfer
Mute KerberosAuthenticationIT
adb93da 
Relates #32498
danielmitterdorfer added a commit that referenced this issue Jul 31, 2018
@danielmitterdorfer
Mute KerberosAuthenticationIT
635ee1c 
Relates #32498
@danielmitterdorfer
Copy link
Member Author

The tests are muted now on 6.4, 6.x and master (see above for commit ids).

@danielmitterdorfer
Copy link
Member Author

Another failure happened in https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=centos/1214/console.

jaymode added a commit to jaymode/elasticsearch that referenced this issue Jul 31, 2018
@jaymode
Use hostname instead of IP with SPNEGO test
892c496 
This change updates KerberosAuthenticationIT to resolve the host used
to connect to the test cluster. This is needed because the host could
be an IP address but SPNEGO requires a hostname to work properly. This
is done by adding a hook in ESRestTestCase for building the HttpHost
from the host and port.

Additionally, the project now specifies the IPv4 loopback address as
the http host. This is done because we need to be able to resolve the
address used for the HTTP transport before the node starts up, but the
http.ports file is not written until the node is started.

Closes elastic#32498
@jaymode jaymode mentioned this issue Jul 31, 2018
Merged
@jaymode
Copy link
Member

jaymode commented Jul 31, 2018

This is failing due to the use of [::1] as the address in the rest client and the fact that the test always expects the http address to be localhost. I opened #32514 to address this

bizybot pushed a commit that referenced this issue Aug 1, 2018
@jaymode @bizybot
Use hostname instead of IP with SPNEGO test (#32514)
f2f33f3 
This change updates KerberosAuthenticationIT to resolve the host used
to connect to the test cluster. This is needed because the host could
be an IP address but SPNEGO requires a hostname to work properly. This
is done by adding a hook in ESRestTestCase for building the HttpHost
from the host and port.

Additionally, the project now specifies the IPv4 loopback address as
the http host. This is done because we need to be able to resolve the
address used for the HTTP transport before the node starts up, but the
http.ports file is not written until the node is started.

Closes #32498
bizybot pushed a commit that referenced this issue Aug 1, 2018
@jaymode
Use hostname instead of IP with SPNEGO test (#32514)
f206783 
This change updates KerberosAuthenticationIT to resolve the host used
to connect to the test cluster. This is needed because the host could
be an IP address but SPNEGO requires a hostname to work properly. This
is done by adding a hook in ESRestTestCase for building the HttpHost
from the host and port.

Additionally, the project now specifies the IPv4 loopback address as
the http host. This is done because we need to be able to resolve the
address used for the HTTP transport before the node starts up, but the
http.ports file is not written until the node is started.

Closes #32498
bizybot pushed a commit that referenced this issue Aug 1, 2018
@jaymode
Use hostname instead of IP with SPNEGO test (#32514)
c2fff29 
This change updates KerberosAuthenticationIT to resolve the host used
to connect to the test cluster. This is needed because the host could
be an IP address but SPNEGO requires a hostname to work properly. This
is done by adding a hook in ESRestTestCase for building the HttpHost
from the host and port.

Additionally, the project now specifies the IPv4 loopback address as
the http host. This is done because we need to be able to resolve the
address used for the HTTP transport before the node starts up, but the
http.ports file is not written until the node is started.

Closes #32498
@colings86
Copy link
Contributor

colings86 commented Aug 2, 2018 •
edited
Loading

Looks like this is still failing on the same tests though with a different (my related) exception. The build in question is:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+6.x+multijob-unix-compatibility/os=centos/1217/console

It now seems to fail with the following exception suggesting that the host name is not resolvable on the CI server:

08:08:14   1> >>> KrbKdcReq send: kdc=kerberos.build.elastic.co UDP:88, timeout=3000, number of retries =3, #bytes=142
08:08:14   1> >>> KrbKdcReq send: error trying kerberos.build.elastic.co:88
08:08:14   1> java.net.UnknownHostException: kerberos.build.elastic.co: Name or service not known
08:08:14   1> 	at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
08:08:14   1> 	at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:928)
08:08:14   1> 	at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1323)
08:08:14   1> 	at java.net.InetAddress.getAllByName0(InetAddress.java:1276)
08:08:14   1> 	at java.net.InetAddress.getAllByName(InetAddress.java:1192)
08:08:14   1> 	at java.net.InetAddress.getAllByName(InetAddress.java:1126)
08:08:14   1> 	at java.net.InetAddress.getByName(InetAddress.java:1076)
08:08:14   1> 	at sun.security.krb5.internal.UDPClient.<init>(NetClient.java:187)
08:08:14   1> 	at sun.security.krb5.internal.NetClient.getInstance(NetClient.java:45)
08:08:14   1> 	at sun.security.krb5.KdcComm$KdcCommunication.run(KdcComm.java:393)

Reopening this issue first as it seems to be very related to this fix but if its an infra issue instead I can raise an infra issue for it.

As before this does not reproduce locally.

@colings86 colings86 reopened this Aug 2, 2018
bizybot referenced this issue in bizybot/elasticsearch Aug 2, 2018
[Kerberos] Use canonical host name
0272de7 
The Apache Http components support for Spnego scheme
uses canonical name by default.
Also when resolving host name, on centos by default
there are other aliases so adding them to the
DelegationPermission.

Closes#32498
@bizybot bizybot mentioned this issue Aug 2, 2018
Merged
bizybot referenced this issue Aug 6, 2018
@bizybot
[Kerberos] Use canonical host name (#32588)
615aa85 
The Apache Http components support for Spnego scheme
uses canonical name by default.
Also when resolving host name, on centos by default
there are other aliases so adding them to the
DelegationPermission.

Closes#32498
bizybot referenced this issue Aug 6, 2018
@bizybot
[Kerberos] Use canonical host name (#32588)
f102836 
The Apache Http components support for Spnego scheme
uses canonical name by default.
Also when resolving host name, on centos by default
there are other aliases so adding them to the
DelegationPermission.

Closes#32498
bizybot referenced this issue Aug 6, 2018
@bizybot
[Kerberos] Use canonical host name (#32588)
d5fafbf 
The Apache Http components support for Spnego scheme
uses canonical name by default.
Also when resolving host name, on centos by default
there are other aliases so adding them to the
DelegationPermission.

Closes#32498
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue Sep 19, 2018
@DaveCTurner
Remove no-op test
3ebbefd 
This test was introduced when muting some other tests in elastic#32498, but not
removed when the tests were unmuted in elastic#32514.
@DaveCTurner DaveCTurner mentioned this issue Sep 19, 2018
Merged
DaveCTurner added a commit that referenced this issue Sep 20, 2018
@DaveCTurner
Remove no-op test (#33861)
6d3a791 
This test was introduced when muting some other tests in #32498, but not
removed when the tests were unmuted in #32514.
DaveCTurner added a commit that referenced this issue Sep 20, 2018
@DaveCTurner
Remove no-op test (#33861)
96b4e21 
This test was introduced when muting some other tests in #32498, but not
removed when the tests were unmuted in #32514.
kcm pushed a commit that referenced this issue Oct 30, 2018
@DaveCTurner @kcm
Remove no-op test (#33861)
758cf24 
This test was introduced when muting some other tests in #32498, but not
removed when the tests were unmuted in #32514.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@bizybot bizybot

Labels
:Security/Authentication Logging in, Usernames/passwords, Realms (Native/LDAP/AD/SAML/PKI/etc) >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
5 participants
@colings86 @bizybot @danielmitterdorfer @jaymode @elasticmachine