chore: [release-3.0.x] feat: Updated best practises for labels (#12783)

Co-authored-by: Jay Clifford <[email protected]>

Author: grafanabot

docs/sources/get-started/labels/bp-labels.md

@@ -12,7 +12,7 @@ Grafana Loki is under active development, and we are constantly working to impro
 
 ## Static labels are good
 
-Things like, host, application, and environment are great labels. They will be fixed for a given system/app and have bounded values. Use static labels to make it easier to query your logs in a logical sense (e.g. show me all the logs for a given application and specific environment, or show me all the logs for all the apps on a specific host).
+Use labels for things like regions, clusters, servers, applications, namespaces, and environments. They will be fixed for a given system/app and have bounded values. Use static labels to make it easier to query your logs in a logical sense (for example, show me all the logs for a given application and specific environment, or show me all the logs for all the apps on a specific host).
 
 ## Use dynamic labels sparingly
 
@@ -33,15 +33,24 @@ What you want to avoid is splitting a log file into streams, which result in chu
 
 It’s not critical that every chunk be full when flushed, but it will improve many aspects of operation. As such, our current guidance here is to avoid dynamic labels as much as possible and instead favor filter expressions. For example, don’t add a `level` dynamic label, just `|= "level=debug"` instead.
 
+Here are some best practices for using dynamic labels with Loki:
+- Ensure the labels have low cardinality, ideally limited to tens of values.
+- Use labels with long-lived values, such as the initial segment of an HTTP path: `/load`, `/save`, `/update`.
+  - Do not extract ephemeral values like a trace ID or an order ID into a label; the values should be static, not dynamic.
+- Only add labels that users will frequently use in their queries.
+  - Don’t increase the size of the index and fragment your log streams if nobody is actually using these labels. This will degrade performance. 
+
 ## Label values must always be bounded
 
 If you are dynamically setting labels, never use a label which can have unbounded or infinite values. This will always result in big problems for Loki.
 
 Try to keep values bounded to as small a set as possible. We don't have perfect guidance as to what Loki can handle, but think single digits, or maybe 10’s of values for a dynamic label. This is less critical for static labels. For example, if you have 1,000 hosts in your environment it's going to be just fine to have a host label with 1,000 values.
 
+As a general rule, you should try to keep any single tenant in Loki to less than **100,000 active streams**, and less than a million streams in a 24-hour period.  These values are for HUGE tenants, sending more than **10 TB** a day. If your tenant is 10x smaller, you should have at least 10x fewer labels.
+
 ## Be aware of dynamic labels applied by clients
 
-Loki has several client options: [Promtail]({{< relref "../../send-data/promtail" >}}) (which also supports systemd journal ingestion and TCP-based syslog ingestion), [Fluentd]({{< relref "../../send-data/fluentd" >}}), [Fluent Bit]({{< relref "../../send-data/fluentbit" >}}), a [Docker plugin](/blog/2019/07/15/lokis-path-to-ga-docker-logging-driver-plugin-support-for-systemd/), and more!
+Loki has several client options: [Grafana Alloy](https://grafana.com/docs/alloy/latest/), [Promtail](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/promtail/) (which also supports systemd journal ingestion and TCP-based syslog ingestion), [Fluentd](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/fluentd/), [Fluent Bit](https://grafana.com/docs/loki/<LOKI_VERSION>/send-data/fluentbit/), a [Docker plugin](https://grafana.com/docs/loki/MLOKI_VERSION>/send-data/docker-driver/), and more.
 
 Each of these come with ways to configure what labels are applied to create log streams. But be aware of what dynamic labels might be applied.
 Use the Loki series API to get an idea of what your log streams look like and see if there might be ways to reduce streams and cardinality.