Minikube time sync issues

[bkeelan]

minikube version v1.19.0
vm-driver hyperkit

Was hoping to reopen an older issue #1378 as timesyncd was removed from the config with this commit

[minikube] ~ $ minikube ssh
                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ timedatectl
               Local time: Tue 2021-04-27 06:39:17 UTC
           Universal time: Tue 2021-04-27 06:39:17 UTC
                 RTC time: Tue 2021-04-27 06:39:17
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

I realize the date is back in sync here but I just fixed it. I'm running into the same issues pulling images and connecting to ECR as the original post.

[afbjorklund]

I'm not sure if there is any harm in running another NTP server, in case the VM is drifting for some reason ?

I don't think removing systemd-timesyncd was intended, so if it is gone now I think that could have happen by mistake

EDIT: Still available. But since it is default already, buildroot removes it from the defconfig.

config BR2_PACKAGE_SYSTEMD_TIMESYNCD
	bool "enable timesync daemon"
	default y
	help
	  systemd-timesyncd is a service that may be used to
	  synchronize the local system clock with a Network Time
	  Protocol server.

	  This simple NTP solution is an alternative to sntp/ntpd from
	  the ntp package.

	  http://www.freedesktop.org/software/systemd/man/systemd-timesyncd.html

BR2_INIT_SYSTEMD=y
BR2_PACKAGE_SKELETON_INIT_SYSTEMD=y
BR2_PACKAGE_SYSTEMD_ARCH_SUPPORTS=y
BR2_PACKAGE_SYSTEMD=y
BR2_PACKAGE_SYSTEMD_PSTORE=y
BR2_PACKAGE_SYSTEMD_HOSTNAMED=y
BR2_PACKAGE_SYSTEMD_HWDB=y
BR2_PACKAGE_SYSTEMD_LOGIND=y
BR2_PACKAGE_SYSTEMD_MACHINED=y
BR2_PACKAGE_SYSTEMD_MYHOSTNAME=y
BR2_PACKAGE_SYSTEMD_NETWORKD=y
BR2_PACKAGE_SYSTEMD_RESOLVED=y
BR2_PACKAGE_SYSTEMD_TIMEDATED=y
BR2_PACKAGE_SYSTEMD_TIMESYNCD=y
BR2_PACKAGE_SYSTEMD_TMPFILES=y
BR2_PACKAGE_SYSTEMD_VCONSOLE=y
BR2_PACKAGE_SYSTEMD_BOOTCHART_ARCH_SUPPORTS=y
BR2_PACKAGE_HOST_SYSTEMD=y

The daemon is available here, but disabled (on VirtualBox)

Can you post the output of:
sudo systemctl status systemd-timesyncd
sudo timedatectl timesync-status
sudo timedatectl show-timesync

[bkeelan]

Hmm, looks like it's enabled but unable to hit the NTPServer

● systemd-timesyncd.service - Network Time Synchronization
     Loaded: loaded (/usr/lib/systemd/system/systemd-timesyncd.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/systemd-timesyncd.service.d
             └─disable-virtualbox.conf
     Active: active (running) since Tue 2021-04-27 06:33:31 UTC; 2h 56min ago
       Docs: man:systemd-timesyncd.service(8)
   Main PID: 1770 (systemd-timesyn)
     Status: "Idle."
      Tasks: 3 (limit: 9282)
     Memory: 1.4M
     CGroup: /system.slice/systemd-timesyncd.service
             └─1770 /usr/lib/systemd/systemd-timesyncd

Apr 27 07:23:29 minikube systemd-timesyncd[1770]: Network configuration changed, trying to establish connection.
Apr 27 07:23:39 minikube systemd-timesyncd[1770]: Timed out waiting for reply from 192.168.65.1:123 (192.168.65.1).
Apr 27 07:23:50 minikube systemd-timesyncd[1770]: Timed out waiting for reply from 192.168.65.1:123 (192.168.65.1).
Apr 27 07:58:08 minikube systemd-timesyncd[1770]: Timed out waiting for reply from 192.168.65.1:123 (192.168.65.1).
Apr 27 08:32:27 minikube systemd-timesyncd[1770]: Timed out waiting for reply from 192.168.65.1:123 (192.168.65.1).
Apr 27 08:33:29 minikube systemd-timesyncd[1770]: Network configuration changed, trying to establish connection.
Apr 27 08:33:29 minikube systemd-timesyncd[1770]: Network configuration changed, trying to establish connection.
Apr 27 08:33:29 minikube systemd-timesyncd[1770]: Network configuration changed, trying to establish connection.
Apr 27 08:33:39 minikube systemd-timesyncd[1770]: Timed out waiting for reply from 192.168.65.1:123 (192.168.65.1).
Apr 27 09:07:57 minikube systemd-timesyncd[1770]: Timed out waiting for reply from 192.168.65.1:123 (192.168.65.1).

$ sudo timedatectl timesync-status
       Server: 192.168.65.1 (192.168.65.1)
Poll interval: 34min 8s (min: 32s; max 34min 8s)
 Packet count: 0

$ sudo timedatectl show-timesync
LinkNTPServers=192.168.65.1
FallbackNTPServers=time1.google.com time2.google.com time3.google.com time4.google.com
ServerName=192.168.65.1
ServerAddress=192.168.65.1
RootDistanceMaxUSec=5s
PollIntervalMinUSec=32s
PollIntervalMaxUSec=34min 8s
PollIntervalUSec=34min 8s
Frequency=0

                         _             _
            _         _ ( )           ( )
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ timedatectl
               Local time: Tue 2021-04-27 09:32:20 UTC
           Universal time: Tue 2021-04-27 09:32:20 UTC
                 RTC time: Tue 2021-04-27 15:44:06
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: no
              NTP service: active
          RTC in local TZ: no

[afbjorklund]

Looks like this is specific to hyperkit, no issue with kvm2:

                         _             _            
            _         _ ( )           ( )           
  ___ ___  (_)  ___  (_)| |/')  _   _ | |_      __  
/' _ ` _ `\| |/' _ `\| || , <  ( ) ( )| '_`\  /'__`\
| ( ) ( ) || || ( ) || || |\`\ | (_) || |_) )(  ___/
(_) (_) (_)(_)(_) (_)(_)(_) (_)`\___/'(_,__/'`\____)

$ timedatectl
               Local time: Wed 2021-04-28 05:27:52 UTC
           Universal time: Wed 2021-04-28 05:27:52 UTC
                 RTC time: Wed 2021-04-28 05:27:53
                Time zone: Etc/UTC (UTC, +0000)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no
$ systemd-detect-virt 
kvm

Wonder where that LinkNTPServers comes from ?

FallbackNTPServers=time1.google.com time2.google.com time3.google.com time4.google.com
ServerName=time1.google.com
ServerAddress=216.239.35.0

$ more /etc/systemd/timesyncd.conf 
#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it
#  under the terms of the GNU Lesser General Public License as published by
#  the Free Software Foundation; either version 2.1 of the License, or
#  (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See timesyncd.conf(5) for details.

[Time]
#NTP=
#FallbackNTP=time1.google.com time2.google.com time3.google.com time4.google.com
#RootDistanceMaxSec=5
#PollIntervalMinSec=32
#PollIntervalMaxSec=2048

Possibly from DHCP.

$ networkctl --no-pager list
IDX LINK        TYPE     OPERATIONAL SETUP     
  1 lo          loopback carrier     unmanaged 
  2 eth0        ether    routable    configured
  3 eth1        ether    routable    configured
  4 sit0        sit      off         unmanaged 
  5 docker0     bridge   routable    unmanaged 
  7 veth472bf63 ether    carrier     unmanaged 

6 links listed.
$ networkctl --no-pager status eth0 eth1
● 2: eth0                                                              
             Link File: n/a                                            
          Network File: /etc/systemd/network/20-dhcp.network           
                  Type: ether                                          
                 State: routable (configured)
            HW Address: 52:54:00:56:82:7b                              
                   MTU: 1500 (min: 68, max: 65535)                     
  Queue Length (Tx/Rx): 1/1                                            
      Auto negotiation: no                                             
                 Speed: n/a                                            
               Address: 192.168.39.51 (DHCP4)                          

● 3: eth1                                                              
             Link File: n/a                                            
          Network File: /etc/systemd/network/20-dhcp.network           
                  Type: ether                                          
                 State: routable (configured)
            HW Address: 52:54:00:d6:2b:63                              
                   MTU: 1500 (min: 68, max: 65535)                     
  Queue Length (Tx/Rx): 1/1                                            
      Auto negotiation: no                                             
                 Speed: n/a                                            
               Address: 192.168.122.26 (DHCP4)                         
               Gateway: 192.168.122.1                                  
                   DNS: 192.168.122.1                                  

[afbjorklund]

Also moby/hyperkit#110 (comment)

Apparently HyperKit assumes that a NTP server is running on the host ?

Timed out waiting for reply from 192.168.65.1:123 (192.168.65.1).

I think that address is the same as gateway.docker.internal, i.e. host.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[nickcaballero]

Seeing the same issue on my environment using hyperkit without vpnkit. DNS resolution for the NTP server time4.google.com fails inside the VM, I'm guessing because the only way to reach internet is via HTTP proxy.

[nickcaballero]

Adding IPs of the Google time servers to /etc/hosts helps with the DNS resolution issue.

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.