Document the new 9.1 OpenTelemetry meters

Snippets/Core/Core.sln

@@ -12,6 +12,8 @@ Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Core_8.1", "Core_8.1\Core_8
 EndProject
 Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Core_9", "Core_9\Core_9.csproj", "{5CE08C72-7A6B-44B8-8CC2-4D787D30A155}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Core_9.1", "Core_9.1\Core_9.1.csproj", "{9D189D02-8AF6-4B92-ABFC-224246946917}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
@@ -38,6 +40,10 @@ Global
 		{5CE08C72-7A6B-44B8-8CC2-4D787D30A155}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{5CE08C72-7A6B-44B8-8CC2-4D787D30A155}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{5CE08C72-7A6B-44B8-8CC2-4D787D30A155}.Release|Any CPU.Build.0 = Release|Any CPU
+		{9D189D02-8AF6-4B92-ABFC-224246946917}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{9D189D02-8AF6-4B92-ABFC-224246946917}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{9D189D02-8AF6-4B92-ABFC-224246946917}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{9D189D02-8AF6-4B92-ABFC-224246946917}.Release|Any CPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(SolutionProperties) = preSolution
 		HideSolutionNode = FALSE

Snippets/Core/Core_9.1/Core_9.1.csproj

@@ -0,0 +1,16 @@
+<Project Sdk="Microsoft.NET.Sdk">
+
+  <PropertyGroup>
+    <TargetFramework>net8.0</TargetFramework>
+    <RootNamespace>Core9</RootNamespace>
+  </PropertyGroup>
+
+  <ItemGroup>
+    <FrameworkReference Include="Microsoft.AspNetCore.App" />
+  </ItemGroup>
+
+  <ItemGroup>
+    <PackageReference Include="NServiceBus" Version="9.1.*" />
+  </ItemGroup>
+
+</Project>

monitoring/metrics/definitions.md

@@ -15,42 +15,42 @@ Gathering metrics is important to know how a system works and if it works proper
 
 NServiceBus and ServiceControl capture a number of different metrics about a running endpoint including the processing time, the number of messages in each queue (differentiating between those pulled from the queue, those processed successfully, and those which failed processing), as well as "critical time".
 
-
 ### Processing time
 
-Processing time is the time it takes for an endpoint to **successfully** invoke all handlers and sagas for a single incoming message. Processing failures are not included in the processing time metric.
+Processing time is the time it takes for an endpoint to **successfully** process an incoming message. It includes:
+
+- The execution of all handlers and sagas for the incoming message
+- The execution of the incoming message processing pipeline, which includes deserialization and where applicable, user-defined pipeline behaviors or saga-loading time.
+
+Processing failures are not included in the processing time metric.
 
 > [!NOTE]
 > Processing time does not include the time to store the outbox operation, transmit outgoing messages to the transport, fetch the incoming message, and complete the incoming message (i.e. commit the transport transaction or acknowledge the message).
 
-
 ### Number of messages pulled from queue
 
 This metric measures the total number of messages that the endpoint reads from its input queue regardless of whether message processing succeeds or fails.
 
-
 ### Number of messages successfully processed
 
 This metric measures the total number of messages that the endpoint successfully processes. In order for a message to be counted by this metric, all handlers must have executed without throwing an exception.
 
-
 ### Number of message processing failures
 
 This metric measures the total number of messages that the endpoint has failed to process. In order for a message to be counted by this metric, at least one handler must have thrown an exception.
 
-
 ### Critical time
 
 Critical time is the time between when a message is sent and when it is fully processed. It is a combination of:
 
- * Network send time: The time a message spends on the network before arriving in the destination queue
- * Queue wait time: The time a message spends in the destination queue before being picked up and processed
- * Processing time: The time it takes for the destination endpoint to process the message
+- Network send time: The time a message spends on the network before arriving in the destination queue
+- Queue wait time: The time a message spends in the destination queue before being picked up and processed
+- Processing time: The time it takes for the destination endpoint to process the message
 
 Critical time does _not_ include:
 
-* The time to store the outbox operation, transmit messages to the transport, and complete the incoming message (i.e. commit the transport transaction or acknowledge) because the `TimeSent` header is added with the current time during the dispatch phase, after the outbox operation has completed.
-* The time a delayed message is held by a timeout mechanism. (NServiceBus version 7.7 and above.)
+- The time to store the outbox operation, transmit messages to the transport, and complete the incoming message (i.e. commit the transport transaction or acknowledge) because the `TimeSent` header is added with the current time during the dispatch phase, after the outbox operation has completed.
+- The time a delayed message is held by a timeout mechanism. (NServiceBus version 7.7 and above.)
 
 > [!NOTE]
 > Due to the fact that the critical time is calculated based on timestamps taken on two different machines (the sender and the receiver of a message), it is affected by the [clock drift problem](https://en.wikipedia.org/wiki/Clock_drift). In cases where the clocks of the machines differ significantly, the critical time may be reported as a negative value. Use well-known clock synchronization solutions such as NTP to mitigate the issue.

nservicebus/operations/opentelemetry.md

@@ -3,10 +3,75 @@ title: OpenTelemetry
 summary: Observability of NServiceBus endpoints with OpenTelemetry
 component: core
 reviewed: 2022-06-29
+versions: '[8,)'
 related:
  - samples/open-telemetry
 ---
 
 NServiceBus version 8 and above supports [OpenTelemetry](https://opentelemetry.io/docs/instrumentation/net/) through traces, metrics, and logging.
 
-partial: content
+Enable OpenTelemetry instrumentation in NServiceBus:
+
+snippet: opentelemetry-enableinstrumentation
+
+With OpenTelemetry instrumentation enabled, tracing, metrics, and logging can be individually configured via the OpenTelemetry API itself.
+
+## Traces
+
+NServiceBus endpoints generate OpenTelemetry traces for incoming and outgoing messages. To capture trace information, add the `NServiceBus.Core` activity source to the OpenTelemetry configuration:
+
+snippet: opentelemetry-enabletracing
+
+See the [OpenTelemetry samples](/samples/open-telemetry/) for instructions on how to send trace information to different tools.
+
+## Meters
+
+NServiceBus endpoints can be configured to expose metrics related to message processing. To capture meter information, add the `NServiceBus.Core` meter source to the OpenTelemetry configuration:
+
+snippet: opentelemetry-enablemeters
+
+partial: meters
+
+See the [OpenTelemetry samples](/samples/open-telemetry/) for instructions on how to send metric information to different tools.
+
+## Logging
+
+NServiceBus supports logging out of the box. To collect OpenTelemetry-compatible logging in NServiceBus endpoints, it's possible to configure the endpoint to connect traces and logging when using `Microsoft.Extensions.Logging` package. See the [_Connecting OpenTelemetry traces and logs_ sample](/samples/open-telemetry/logging) for more details.
+
+## Alignment of host identifier
+
+It is recommended to align the instance identifier between NServiceBus and OpenTelemetry so all logs, metrics, traces and audit messages can be correlated by a host (instance) if needed.
+
+> [!NOTE]
+> The OpenTelemetry specification recommends this to be a random uuid. However, it may also be a [deterministic uuid v5](https://opentelemetry.io/docs/specs/semconv/attributes-registry/service/#service-attributes) (i.e. hash of machine name and  endpointname).
+
+NServiceBus adds a [host identifier to all audit messages](/nservicebus/hosting/override-hostid.md) and this instance identifier is also used to show [performance metrics for each running instance in ServicePulse](/monitoring/metrics/in-servicepulse.md). The [instance identifier used for ServicePulse value can be overriden](/monitoring/metrics/install-plugin.md#configuration-instance-id).
+
+OpenTelemetry also allows customizing the instance id used for `service.instance.id` in various ways.
+
+Consider aligning the instance ID used by OpenTelemetry and ServiceControl metrics API.
+
+#### Example
+
+```c#
+// Generate (deterministic) instance ID shared by all components
+
+// Generate deterministic uuid v4 viahttps://github.com/Faithlife/FaithlifeUtility/blob/master/src/Faithlife.Utility/GuidUtility.cs
+var deterministicValue = "MyEndpoint@" + Dns.GetHostName();
+Guid serviceInstanceId = GuidUtility.Create(new Guid("4d63009a-8d0f-11ee-aad7-4c796ed8e320", deterministicValue)) // or Guid.NewGuid()
+
+// OpenTelemetry
+services.AddOpenTelemetry()
+    .ConfigureResource(builder => builder.AddService("MyService", serviceInstanceId: serviceInstanceId.ToString()))
+
+// NServiceBus
+endpointConfiguration.UniquelyIdentifyRunningInstance()
+    .UsingCustomDisplayName("original-instance");
+    .UsingCustomIdentifier(serviceInstanceId)
+
+// ServiceControl Metrics
+endpointConfiguration
+    .EnableMetrics()
+     // Not required when already set via UsingCustomIdentifier
+    .SendMetricDataToServiceControl("particular.monitoring", TimeSpan.FromMinutes(1), serviceInstanceId.ToString());
+```

nservicebus/operations/opentelemetry_meters_core_[8,9.1).partial.md

@@ -0,0 +1,5 @@
+### Emitted meters
+
+- `nservicebus.messaging.successes` - Total number of messages processed successfully by the endpoint
+- `nservicebus.messaging.fetches` - Total number of messages fetched from the queue by the endpoint
+- `nservicebus.messaging.failures` - Total number of messages processed unsuccessfully by the endpoint

nservicebus/operations/opentelemetry_meters_core_[9.1,).partial.md

@@ -0,0 +1,13 @@
+### Emitted meters
+
+- `nservicebus.messaging.successes` - Total number of messages processed successfully by the endpoint
+- `nservicebus.messaging.fetches` - Total number of messages fetched from the queue by the endpoint
+- `nservicebus.messaging.failures` - Total number of messages processed unsuccessfully by the endpoint
+- `nservicebus.messaging.handler_time` - The time the user handling code takes to handle a message
+- `nservicebus.messaging.processing_time` - The time the endpoint takes to process a message from when it's fetched from the input queue to when processing completes. It includes:
+  - Invoking all handlers and sagas for a single incoming message
+  - Invoking the incoming message processing pipeline, which includes steps like deserialization or user defined pipeline behaviors.
+- `nservicebus.messaging.critical_time` - The time between when a message is sent and when it is fully processed. It is a combination of:
+  - Network send time: The time a message spends on the network before arriving in the destination queue
+  - Queue wait time: The time a message spends in the destination queue before being picked up and processed
+  - Processing time: The time it takes for the destination endpoint to process the message