Enhance condition to take full snapshot during startup.

[ishan16696]

How to categorize this issue?
/area monitoring
/area backup
/kind enhancement

What would you like to be added:
During startup, etcd-backup-restore should also consider configured FullSnapshotSchedule along with timeStamp of last full snapshot, so that it won't missed any full snapshot.

Why is this needed:
Currently, alert KubeEtcdFullBackupFailed is calculated based on Last FullSnapshot Timestamp, so it checks whether backup-restore has taken a full snapshot within last 24h or not. But we have observed some false KubeEtcdFullBackupFailed alerts for some shoots if shoot is hibernated and wake up within 24h and determineBackupSchedule is calculating FullSnapshotSchedule on the basis of maintenance window.

Take this Scenario:

1. Suppose, shoot have maintenance window m1.
2. etcd-backup-restore took a full snapshot at timestamp t1 .
3. Then the cluster was hibernated at t2(before maintenance window i.e t2<m1) and woken up again at t3. (t3>t2>t1 && t3>m1)
4. When Cluster was woken up, t3-t1< 24h (24h didn’t pass yet), so no new full snapshot had taken by backup-restore as it calculates 23.5h from timeStamp of last full snapshot.
5. Past 24h, at t4 alert KubeEtcdFullBackupFailed checks the last timestamp of full snapshot and it found t4-t1>24h, now an alert has been raised and backup-restore is waiting to take full snapshot according to --schedule passed to backup-restore and it was calculated on basis of maintenance window of shoot (m1 of next day) which is not yet reached.

/cc @timuthy @shreyas-s-rao

[shreyas-s-rao]

@ishan16696 thanks for resolving this issue. Can you please also raise another issue to calculate the previous full snapshot scheduled time based purely on the cron schedule? As discussed in #574 (review) thread

[ishan16696]

Hi @shreyas-s-rao ,
I have created an issue for follow up: #587

[shreyas-s-rao]

Thanks! 🚀