Sasha/ha

[klizhentas]

Description

NOTE Do not merge this yet, let's merge this after we release 2.0

This Pull Request refines failure modes for Teleport.

Without this change set:

• there was in-memory cache that was only invalidated on server restart)
• in case if all auth servers are down, no new SSH sessions could be created

With this change set:

• teleport provides explicit control over cache TTL for Proxy and Node
• cache is persistent and on-disk
• in case if all auth servers are down, all systems including Proxy (reverse tunnels, etc) and Nodes are functioning
• introduces throttling for failed auth servers to avoid locks on connection timeouts
• introduces client-side caching for discovery data
• introduces new expires metadata property uniformly supported by all resource objects

Documentation

New section cache is now supported in config. This setting is node-local, as it affects local caches for Node and Proxy.

Default value in case if config is not supplied is enabled cache with ttl up to 20 hours

teleport:
  cache:
     # turn off local cache
     enabled: no

teleport:
  cache:
     # turn off local cache
     enabled: yes
    # invalidate cache after 20 hours
     ttl: 20h

teleport:
  cache:
     # turn off local cache
     enabled: yes
    # never expires means that cache for persistent non-expiring values will 
    # never expire effectively keeping SSH or Proxy nodes in "standalone" mode.
    never_expires: yes

[klizhentas]

retest this please

[kontsevoy]

hehe, this is basically done but not merged, right?

[kontsevoy]

Comment regarding the docs: ttl and never_expires are mutually exclusive options without natural expression for preference. It's better to have "never" or "no" (or 0) as acceptable values for ttl to express "never expires" semantics.

[klizhentas]

@kontsevoy comments make sense I will update the config.

[kontsevoy]

@klizhentas do you want to add something to documentation regarding this? Folks are often asking how to login into a cluster if the CA is not available. I can document it if you sent me the rough draft.