add MADR about new features/experiments lifecycle

[lahabana]

Description

We usually add new features/change of behaviours through "experiments".

We don't have a good practice around how experiments land as default.
Let's codify this in a MADR.

IMO here's what we should do:

• land a new conf in experimental with a value of false
• once the feature is mature switch it to true
• once we're all ready with this remove the config altogether

This MADR should also make a difference between rewrites which should have experimental booleans and features that should just be added in the config.

For example we have ExperimentalKDSEventBasedWatchdog. It's unfortunate because if we were to make it not an experiment we'd need to ask people using it to change their config to adopt it.

This MADR should ensure we make it as easy as possible for people to adopt experiments and never need to change configs if the experiment is a success.

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.

[github-actions]

This issue was inactive for 90 days. It will be reviewed in the next triage meeting and might be closed.
If you think this issue is still relevant, please comment on it or attend the next triage meeting.