[o11y] Support Simple Schema for Observability

[YANG-DB]

Is your feature request related to a problem?

Observability refers to the ability to monitor and diagnose systems and applications in real-time, in order to understand how they are behaving and identify potential issues or research the root cause of an error in the system.

Metrics, Traces & Logs (AKA Observability signals) present a critical component providing quantifiable data about the performance and behavior of systems and applications. The importance of supporting well structured Observability schema lies in the fact that it enables better analysis and understanding of the system from the outside.

A structured schema provides a clear, consistent format, making it easier for observability tools to process and aggregate the data.
This in turn makes it easier for engineers to understand the performance and behavior of their systems, and quickly identify potential issues.

When data is unstructured, it can be difficult for observability tools to extract meaningful information from them.

For example, if the data for a particular metric is not consistently recorded in the same format, it can be difficult to compare and analyze performance data over time. Similarly, if traces are not consistently categorized, it can be difficult to understand their context and significance.

With Simple Schema for Observability structured schema in place, observability tools can automatically extract and aggregate data, create custom common dashboards - making it easier to understand the system at a higher level.

This can help teams quickly identify performance bottlenecks, track changes in the system over time, and make informed decisions about system performance optimization.

What solution would you like?
Simple Schema for Observability will support a unified structured definition for the major Observability signals including

• Traces
• Metrics
• Logs
• Alerts

Simple Schema for Observability will conform to OTEL standards and will allow transparent integration with OTEL exporters.

Logs will be supported in a wider extent so that detailed information arriving from the collecting agents would be organized into a well structured format - also following the logs ECS convention .

Logs will support both the general OTEL standard and when possible allow extended information to be recorded in a standard format.

All the supported signals will have a companion json schema file defining the expected schema that must be followed .

What alternatives have you considered?
Only supporting OTEL signals without the enriched structured logs information and without the capability for supporting ECS based logs.

Do you have any additional context?

• Support OTEL based Trace Index Mapping in place #1388
• Support OTEL based Metrics Index Mapping in place #1389
• [FEATURE] Add Observability Simple Schema Support #1404
• OpenSearch exporter support open-telemetry/opentelemetry-collector-contrib#7905
• OpenSearch exporter enhancements open-telemetry/opentelemetry-collector-contrib#12323