Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OPC Publisher 2.9.4's Prometheus metrics partially missing probably due to size limit #2197

Closed
jacqueskang opened this issue Mar 4, 2024 · 6 comments
Closed

OPC Publisher 2.9.4's Prometheus metrics partially missing probably due to size limit #2197

jacqueskang opened this issue Mar 4, 2024 · 6 comments
Labels
bug Something isn't working feature request New feature or request
Milestone
2.9.9

Comments

@jacqueskang
Copy link

jacqueskang commented Mar 4, 2024 •
edited
Loading

Describe the bug
It appears that OPC Publisher v2.9.4's prometheus endpoint (http://<opc_publisher>/metrics) does not always expose all metrics documented in this article.

It seems that the metrics are somehow truncated when the total data size exceeds certain threshold.

To Reproduce

Please find below a sample of raw metrics content.

Many iiot metrics (e.g., iiot_edge_publisher_messages_total) are missing. Instead the content is filled with a lot of non-documented metrics such as dns_lookup_*, process_runtime_dotnet_*, kestrel_*, http_server_request_*, etc.

When comes to iiot metrics, there are a lot of metadata filled with long but low-value default values such as siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>".

# TYPE dns_lookup_duration_seconds histogram
# UNIT dns_lookup_duration_seconds seconds
# HELP dns_lookup_duration_seconds Measures the time taken to perform a DNS lookup.
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.005"} 159 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.01"} 159 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.025"} 160 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.05"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.075"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.1"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.25"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.5"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.75"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="1"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="2.5"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="5"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="7.5"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="10"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="+Inf"} 161 1709313757021
dns_lookup_duration_seconds_sum{dns_question_name="czmbompx0001.mbo.cz.corp"} 0.17188980000000012 1709313757021
dns_lookup_duration_seconds_count{dns_question_name="czmbompx0001.mbo.cz.corp"} 161 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.005"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.01"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.025"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.05"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.075"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.1"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.25"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.5"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.75"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="1"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="2.5"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="5"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="7.5"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="10"} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="",le="+Inf"} 1 1709313757021
dns_lookup_duration_seconds_sum{dns_question_name=""} 1.34E-05 1709313757021
dns_lookup_duration_seconds_count{dns_question_name=""} 1 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.005"} 3 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.01"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.025"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.05"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.075"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.1"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.25"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.5"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.75"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="1"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="2.5"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="5"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="7.5"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="10"} 4 1709313757021
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="+Inf"} 4 1709313757021
dns_lookup_duration_seconds_sum{dns_question_name="czmbomii2103.ls.ege.ds"} 0.018285299999999997 1709313757021
dns_lookup_duration_seconds_count{dns_question_name="czmbomii2103.ls.ege.ds"} 4 1709313757021

# TYPE process_runtime_dotnet_gc_collections_count_total counter
# HELP process_runtime_dotnet_gc_collections_count_total Number of garbage collections that have occurred since process start.
process_runtime_dotnet_gc_collections_count_total{generation="gen2"} 39 1709313757021
process_runtime_dotnet_gc_collections_count_total{generation="gen1"} 2175 1709313757021
process_runtime_dotnet_gc_collections_count_total{generation="gen0"} 12367 1709313757021

# TYPE process_runtime_dotnet_gc_objects_size_bytes gauge
# UNIT process_runtime_dotnet_gc_objects_size_bytes bytes
# HELP process_runtime_dotnet_gc_objects_size_bytes Count of bytes currently in use by objects in the GC heap that haven't been collected yet. Fragmentation and other GC committed memory pools are excluded.
process_runtime_dotnet_gc_objects_size_bytes 56505552 1709313757021

# TYPE process_runtime_dotnet_gc_allocations_size_bytes_total counter
# UNIT process_runtime_dotnet_gc_allocations_size_bytes_total bytes
# HELP process_runtime_dotnet_gc_allocations_size_bytes_total Count of bytes allocated on the managed GC heap since the process start. .NET objects are allocated from this heap. Object allocations from unmanaged languages such as C/C++ do not use this heap.
process_runtime_dotnet_gc_allocations_size_bytes_total 209160601232 1709313757021

# TYPE process_runtime_dotnet_gc_committed_memory_size_bytes gauge
# UNIT process_runtime_dotnet_gc_committed_memory_size_bytes bytes
# HELP process_runtime_dotnet_gc_committed_memory_size_bytes The amount of committed virtual memory for the managed GC heap, as observed during the latest garbage collection. Committed virtual memory may be larger than the heap size because it includes both memory for storing existing objects (the heap size) and some extra memory that is ready to handle newly allocated objects in the future. The value will be unavailable until at least one garbage collection has occurred.
process_runtime_dotnet_gc_committed_memory_size_bytes 78585856 1709313757021

# TYPE process_runtime_dotnet_gc_heap_size_bytes gauge
# UNIT process_runtime_dotnet_gc_heap_size_bytes bytes
# HELP process_runtime_dotnet_gc_heap_size_bytes The heap size (including fragmentation), as observed during the latest garbage collection. The value will be unavailable until at least one garbage collection has occurred.
process_runtime_dotnet_gc_heap_size_bytes{generation="gen0"} 0 1709313757021
process_runtime_dotnet_gc_heap_size_bytes{generation="gen1"} 776824 1709313757021
process_runtime_dotnet_gc_heap_size_bytes{generation="gen2"} 24646224 1709313757021
process_runtime_dotnet_gc_heap_size_bytes{generation="loh"} 35314040 1709313757021
process_runtime_dotnet_gc_heap_size_bytes{generation="poh"} 454704 1709313757021

# TYPE process_runtime_dotnet_gc_heap_fragmentation_size_bytes gauge
# UNIT process_runtime_dotnet_gc_heap_fragmentation_size_bytes bytes
# HELP process_runtime_dotnet_gc_heap_fragmentation_size_bytes The heap fragmentation, as observed during the latest garbage collection. The value will be unavailable until at least one garbage collection has occurred.
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="gen0"} 0 1709313757021
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="gen1"} 11904 1709313757021
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="gen2"} 2164792 1709313757021
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="loh"} 9288000 1709313757021
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="poh"} 1608 1709313757021

# TYPE process_runtime_dotnet_gc_duration_nanoseconds_total counter
# UNIT process_runtime_dotnet_gc_duration_nanoseconds_total nanoseconds
# HELP process_runtime_dotnet_gc_duration_nanoseconds_total The total amount of time paused in GC since the process start.
process_runtime_dotnet_gc_duration_nanoseconds_total 44959829000 1709313757021

# TYPE process_runtime_dotnet_jit_il_compiled_size_bytes_total counter
# UNIT process_runtime_dotnet_jit_il_compiled_size_bytes_total bytes
# HELP process_runtime_dotnet_jit_il_compiled_size_bytes_total Count of bytes of intermediate language that have been compiled since the process start.
process_runtime_dotnet_jit_il_compiled_size_bytes_total 2696862 1709313757021

# TYPE process_runtime_dotnet_jit_methods_compiled_count_total counter
# HELP process_runtime_dotnet_jit_methods_compiled_count_total The number of times the JIT compiler compiled a method since the process start. The JIT compiler may be invoked multiple times for the same method to compile with different generic parameters, or because tiered compilation requested different optimization settings.
process_runtime_dotnet_jit_methods_compiled_count_total 36197 1709313757021

# TYPE process_runtime_dotnet_jit_compilation_time_nanoseconds_total counter
# UNIT process_runtime_dotnet_jit_compilation_time_nanoseconds_total nanoseconds
# HELP process_runtime_dotnet_jit_compilation_time_nanoseconds_total The amount of time the JIT compiler has spent compiling methods since the process start.
process_runtime_dotnet_jit_compilation_time_nanoseconds_total 23153214300 1709313757021

# TYPE process_runtime_dotnet_monitor_lock_contention_count_total counter
# HELP process_runtime_dotnet_monitor_lock_contention_count_total The number of times there was contention when trying to acquire a monitor lock since the process start. Monitor locks are commonly acquired by using the lock keyword in C#, or by calling Monitor.Enter() and Monitor.TryEnter().
process_runtime_dotnet_monitor_lock_contention_count_total 613109 1709313757021

# TYPE process_runtime_dotnet_thread_pool_threads_count gauge
# HELP process_runtime_dotnet_thread_pool_threads_count The number of thread pool threads that currently exist.
process_runtime_dotnet_thread_pool_threads_count 3 1709313757021

# TYPE process_runtime_dotnet_thread_pool_completed_items_count_total counter
# HELP process_runtime_dotnet_thread_pool_completed_items_count_total The number of work items that have been processed by the thread pool since the process start.
process_runtime_dotnet_thread_pool_completed_items_count_total 7699737 1709313757021

# TYPE process_runtime_dotnet_thread_pool_queue_length gauge
# HELP process_runtime_dotnet_thread_pool_queue_length The number of work items that are currently queued to be processed by the thread pool.
process_runtime_dotnet_thread_pool_queue_length 0 1709313757021

# TYPE process_runtime_dotnet_timer_count gauge
# HELP process_runtime_dotnet_timer_count The number of timer instances that are currently active. Timers can be created by many sources such as System.Threading.Timer, Task.Delay, or the timeout in a CancellationSource. An active timer is registered to tick at some point in the future and has not yet been canceled.
process_runtime_dotnet_timer_count 16 1709313757021

# TYPE process_runtime_dotnet_assemblies_count gauge
# HELP process_runtime_dotnet_assemblies_count The number of .NET assemblies that are currently loaded.
process_runtime_dotnet_assemblies_count 209 1709313757021

# TYPE process_runtime_dotnet_exceptions_count_total counter
# HELP process_runtime_dotnet_exceptions_count_total Count of exceptions that have been thrown in managed code, since the observation started. The value will be unavailable until an exception has been thrown after OpenTelemetry.Instrumentation.Runtime initialization.
process_runtime_dotnet_exceptions_count_total 595 1709313757021

# TYPE iiot_edge_publisher_module_start gauge
# HELP iiot_edge_publisher_module_start Publisher module started.
iiot_edge_publisher_module_start{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 1 1709313757021

# TYPE iiot_edge_publisher_module_state gauge
# HELP iiot_edge_publisher_module_state Publisher module runtime state.
iiot_edge_publisher_module_state{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 1 1709313757021

# TYPE iiot_edge_publisher_certificate_renewal_count_total counter
# HELP iiot_edge_publisher_certificate_renewal_count_total Publisher certificate renewals.
iiot_edge_publisher_certificate_renewal_count_total{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 0 1709313757021

# TYPE kestrel_active_connections gauge
# HELP kestrel_active_connections Number of connections that are currently active on the server.
kestrel_active_connections{network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 1 1709313757021

# TYPE kestrel_connection_duration_seconds histogram
# UNIT kestrel_connection_duration_seconds seconds
# HELP kestrel_connection_duration_seconds The duration of connections on the server.
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.01"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.02"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.05"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.1"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.2"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.5"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="1"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="2"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="5"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="10"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="30"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="60"} 0 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="120"} 2085 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="300"} 2085 1709313757021
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="+Inf"} 2085 1709313757021
kestrel_connection_duration_seconds_sum{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 154889.00442710007 1709313757021
kestrel_connection_duration_seconds_count{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 2085 1709313757021

# TYPE kestrel_queued_connections gauge
# HELP kestrel_queued_connections Number of connections that are currently queued and are waiting to start.
kestrel_queued_connections{network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 0 1709313757021

# TYPE http_server_active_requests gauge
# HELP http_server_active_requests Number of active HTTP server requests.
http_server_active_requests{http_request_method="GET",url_scheme="http"} 1 1709313757021

# TYPE http_server_request_duration_seconds histogram
# UNIT http_server_request_duration_seconds seconds
# HELP http_server_request_duration_seconds Duration of HTTP server requests.
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.005"} 1799 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.01"} 2028 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.025"} 2083 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.05"} 2084 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.075"} 2084 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.1"} 2084 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.25"} 2084 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.5"} 2085 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.75"} 2085 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="1"} 2085 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="2.5"} 2085 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="5"} 2085 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="7.5"} 2085 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="10"} 2085 1709313757021
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="+Inf"} 2085 1709313757021
http_server_request_duration_seconds_sum{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http"} 6.767059300000002 1709313757021
http_server_request_duration_seconds_count{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http"} 2085 1709313757021

# TYPE aspnetcore_routing_match_attempts_total counter
# HELP aspnetcore_routing_match_attempts_total Number of requests that were attempted to be matched to an endpoint.
aspnetcore_routing_match_attempts_total{aspnetcore_routing_match_status="failure"} 2086 1709313757021

# TYPE iiot_edge_publisher_client_count gauge
# HELP iiot_edge_publisher_client_count Number of clients.
iiot_edge_publisher_client_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 2 1709313757021

# TYPE iiot_edge_publisher_missing_keep_alives_total counter
# HELP iiot_edge_publisher_missing_keep_alives_total Number of missing keep alives in subscription.
iiot_edge_publisher_missing_keep_alives_total{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_missing_keep_alives_total{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_missing_keep_alives_total{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_missing_keep_alives_total{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021
iiot_edge_publisher_missing_keep_alives_total{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021
iiot_edge_publisher_missing_keep_alives_total{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021

# TYPE iiot_edge_publisher_unassigned_notification_count_total counter
# HELP iiot_edge_publisher_unassigned_notification_count_total Number of notifications that could not be assigned.
iiot_edge_publisher_unassigned_notification_count_total{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_unassigned_notification_count_total{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_unassigned_notification_count_total{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_unassigned_notification_count_total{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021
iiot_edge_publisher_unassigned_notification_count_total{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021
iiot_edge_publisher_unassigned_notification_count_total{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021

# TYPE iiot_edge_publisher_good_nodes gauge
# HELP iiot_edge_publisher_good_nodes Monitored items successfully created.
iiot_edge_publisher_good_nodes{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_good_nodes{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_good_nodes{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_good_nodes{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1000 1709313757021
iiot_edge_publisher_good_nodes{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1000 1709313757021
iiot_edge_publisher_good_nodes{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 801 1709313757021

# TYPE iiot_edge_publisher_bad_nodes gauge
# HELP iiot_edge_publisher_bad_nodes Monitored items with errors.
iiot_edge_publisher_bad_nodes{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_bad_nodes{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_bad_nodes{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_bad_nodes{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021
iiot_edge_publisher_bad_nodes{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021
iiot_edge_publisher_bad_nodes{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021

# TYPE iiot_edge_publisher_monitored_items gauge
# HELP iiot_edge_publisher_monitored_items Total monitored item count.
iiot_edge_publisher_monitored_items{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 1000 1709313757021
iiot_edge_publisher_monitored_items{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 1000 1709313757021
iiot_edge_publisher_monitored_items{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 801 1709313757021
iiot_edge_publisher_monitored_items{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1000 1709313757021
iiot_edge_publisher_monitored_items{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1000 1709313757021
iiot_edge_publisher_monitored_items{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 801 1709313757021

# TYPE iiot_edge_publisher_publish_requests_per_subscription gauge
# HELP iiot_edge_publisher_publish_requests_per_subscription Good publish requests per subsciption.
iiot_edge_publisher_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1 1709313757021
iiot_edge_publisher_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1 1709313757021
iiot_edge_publisher_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1 1709313757021

# TYPE iiot_edge_publisher_good_publish_requests_per_subscription gauge
# HELP iiot_edge_publisher_good_publish_requests_per_subscription Good publish requests per subsciption.
iiot_edge_publisher_good_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_good_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_good_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_good_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1 1709313757021
iiot_edge_publisher_good_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1 1709313757021
iiot_edge_publisher_good_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1 1709313757021

# TYPE iiot_edge_publisher_bad_publish_requests_per_subscription gauge
# HELP iiot_edge_publisher_bad_publish_requests_per_subscription Bad publish requests per subsciption.
iiot_edge_publisher_bad_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_bad_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_bad_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_bad_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021
iiot_edge_publisher_bad_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021
iiot_edge_publisher_bad_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 0 1709313757021

# TYPE iiot_edge_publisher_min_publish_requests_per_subscription gauge
# HELP iiot_edge_publisher_min_publish_requests_per_subscription Min publish requests queued per subsciption.
iiot_edge_publisher_min_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_min_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_min_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 0 1709313757021
iiot_edge_publisher_min_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1 1709313757021
iiot_edge_publisher_min_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1 1709313757021
iiot_edge_publisher_min_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",securityMode="SignAndEncrypt",siteId="<<UnknownSite>>",writerGroupId="x"} 1 1709313757021

# TYPE iiot_edge_publisher_client_connectivity_state gauge
# HELP iiot_edge_publisher_client_connectivity_state Client connectivity state.
iiot_edge_publisher_client_connectivity_state{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 5 1709313757021

# TYPE iiot_edge_publisher_client_subscription_count gauge
# HELP iiot_edge_publisher_client_subscription_count Number of client managed subscriptions.
iiot_edge_publisher_client_subscription_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 3 1709313757021

# TYPE iiot_edge_publisher_client_connectivity_retry_count gauge
# HELP iiot_edge_publisher_client_connectivity_retry_count Number of connectivity retries on this connection.
iiot_edge_publisher_client_connectivity_retry_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 0 1709313757021

# TYPE iiot_edge_publisher_client_ref_count gauge
# HELP iiot_edge_publisher_client_ref_count Number of references to this client.
iiot_edge_publisher_client_ref_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 3 1709313757021

# TYPE iiot_edge_publisher_client_good_publish_requests_count gauge
# HELP iiot_edge_publisher_client_good_publish_requests_count Number of good publish requests.
iiot_edge_publisher_client_good_publish_requests_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 3 1709313757021

# TYPE iiot_edge_publisher_client_bad_publish_requests_count gauge
# HELP iiot_edge_publisher_client_bad_publish_requests_count Number of bad publish requests.
iiot_edge_publisher_client_bad_publish_requests_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 0 1709313757021

# TYPE iiot_edge_publisher_client_min_publish_requests_count gauge
# HELP iiot_edge_publisher_client_min_publish_requests_count Number of min publish requests that should be queued.
iiot_edge_publisher_client_min_publish_requests_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 3 1709313757021

# TYPE iiot_edge_publisher_client_outstanding_requests_count gauge
# HELP iiot_edge_publisher_client_outstanding_requests_count Number of outstanding requests.
iiot_edge_publisher_client_outstanding_requests_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 3 1709313757021

# TYPE iiot_edge_publisher_client_publish_timeout_count gauge
# HELP iiot_edge_publisher_client_publish_timeout_count Number of timed out requests.
iiot_edge_publisher_client_publish_timeout_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 0 1709313757021

# TYPE iiot_edge_publisher_session_count gauge
# HELP iiot_edge_publisher_session_count Number of active sessions.
iiot_edge_publisher_session_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher"} 1 1709313757021

# TYPE iiot_edge_publisher_messages_duration histogram
# HELP iiot_edge_publisher_messages_duration Histogram of message sending durations.
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="0"} 0 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="5"} 2550 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="10"} 5964 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="25"} 6213 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="50"} 6220 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="75"} 6220 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="100"} 6220 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="250"} 6222 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="500"} 6225 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="750"} 6225 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="1000"} 6225 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="2500"} 6226 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="5000"} 6226 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="7500"} 6226 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="10000"} 6226 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>",le="+Inf"} 6226 1709313757021
iiot_edge_publisher_messages_duration_sum{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 42920 1709313757021
iiot_edge_publisher_messages_duration_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="<<UnknownWriterGroup>>"} 6226 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="0"} 0 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="5"} 28952 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="10"} 55891 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="25"} 57313 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="50"} 57366 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="75"} 57366 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="100"} 57366 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="250"} 57369 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="500"} 57372 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="750"} 57372 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="1000"} 57372 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="2500"} 57372 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="5000"} 57372 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="7500"} 57372 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="10000"} 57372 1709313757021
iiot_edge_publisher_messages_duration_bucket{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x",le="+Inf"} 57373 1709313757021
iiot_edge_publisher_messages_duration_sum{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x"} 391736 1709313757021
iiot_edge_publisher_messages_duration_count{publisherId="iot-gptmpxp01-euw-pro-core-01_device_czmbompx0001_module_opc-publisher",siteId="<<UnknownSite>>",writerGroupId="x"} 57373 1709313757021

# EOF

In order to reduce the content size, we have configured the "publisherId" and "siteId" with shorter string, and the metrics are back afterwards.

Expected behavior

Prometheus endpoint always return all documented metrics

Screenshots
N/A

Desktop (please complete the following information):

  • OS: Ubuntu
  • Browser N/A
  • Version 20.04

Additional context
OPC Publisher version: 2.9.4
@jacqueskang
Copy link
Author

jacqueskang commented Mar 4, 2024 •
edited
Loading

After have specified "publisherId" and "siteId" with a short string, more metrics are available:

# TYPE process_runtime_dotnet_gc_collections_count_total counter
# HELP process_runtime_dotnet_gc_collections_count_total Number of garbage collections that have occurred since process start.
process_runtime_dotnet_gc_collections_count_total{generation="gen2"} 16 1709536344519
process_runtime_dotnet_gc_collections_count_total{generation="gen1"} 193 1709536344519
process_runtime_dotnet_gc_collections_count_total{generation="gen0"} 4299 1709536344519

# TYPE process_runtime_dotnet_gc_objects_size_bytes gauge
# UNIT process_runtime_dotnet_gc_objects_size_bytes bytes
# HELP process_runtime_dotnet_gc_objects_size_bytes Count of bytes currently in use by objects in the GC heap that haven't been collected yet. Fragmentation and other GC committed memory pools are excluded.
process_runtime_dotnet_gc_objects_size_bytes 50195032 1709536344519

# TYPE process_runtime_dotnet_gc_allocations_size_bytes_total counter
# UNIT process_runtime_dotnet_gc_allocations_size_bytes_total bytes
# HELP process_runtime_dotnet_gc_allocations_size_bytes_total Count of bytes allocated on the managed GC heap since the process start. .NET objects are allocated from this heap. Object allocations from unmanaged languages such as C/C++ do not use this heap.
process_runtime_dotnet_gc_allocations_size_bytes_total 64268684480 1709536344519

# TYPE process_runtime_dotnet_gc_committed_memory_size_bytes gauge
# UNIT process_runtime_dotnet_gc_committed_memory_size_bytes bytes
# HELP process_runtime_dotnet_gc_committed_memory_size_bytes The amount of committed virtual memory for the managed GC heap, as observed during the latest garbage collection. Committed virtual memory may be larger than the heap size because it includes both memory for storing existing objects (the heap size) and some extra memory that is ready to handle newly allocated objects in the future. The value will be unavailable until at least one garbage collection has occurred.
process_runtime_dotnet_gc_committed_memory_size_bytes 67608576 1709536344519

# TYPE process_runtime_dotnet_gc_heap_size_bytes gauge
# UNIT process_runtime_dotnet_gc_heap_size_bytes bytes
# HELP process_runtime_dotnet_gc_heap_size_bytes The heap size (including fragmentation), as observed during the latest garbage collection. The value will be unavailable until at least one garbage collection has occurred.
process_runtime_dotnet_gc_heap_size_bytes{generation="gen0"} 2303008 1709536344519
process_runtime_dotnet_gc_heap_size_bytes{generation="gen1"} 1577664 1709536344519
process_runtime_dotnet_gc_heap_size_bytes{generation="gen2"} 17829680 1709536344519
process_runtime_dotnet_gc_heap_size_bytes{generation="loh"} 30729712 1709536344519
process_runtime_dotnet_gc_heap_size_bytes{generation="poh"} 428376 1709536344519

# TYPE process_runtime_dotnet_gc_heap_fragmentation_size_bytes gauge
# UNIT process_runtime_dotnet_gc_heap_fragmentation_size_bytes bytes
# HELP process_runtime_dotnet_gc_heap_fragmentation_size_bytes The heap fragmentation, as observed during the latest garbage collection. The value will be unavailable until at least one garbage collection has occurred.
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="gen0"} 1991680 1709536344519
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="gen1"} 25872 1709536344519
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="gen2"} 360984 1709536344519
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="loh"} 7523144 1709536344519
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="poh"} 0 1709536344519

# TYPE process_runtime_dotnet_gc_duration_nanoseconds_total counter
# UNIT process_runtime_dotnet_gc_duration_nanoseconds_total nanoseconds
# HELP process_runtime_dotnet_gc_duration_nanoseconds_total The total amount of time paused in GC since the process start.
process_runtime_dotnet_gc_duration_nanoseconds_total 14780224000 1709536344519

# TYPE process_runtime_dotnet_jit_il_compiled_size_bytes_total counter
# UNIT process_runtime_dotnet_jit_il_compiled_size_bytes_total bytes
# HELP process_runtime_dotnet_jit_il_compiled_size_bytes_total Count of bytes of intermediate language that have been compiled since the process start.
process_runtime_dotnet_jit_il_compiled_size_bytes_total 2520844 1709536344519

# TYPE process_runtime_dotnet_jit_methods_compiled_count_total counter
# HELP process_runtime_dotnet_jit_methods_compiled_count_total The number of times the JIT compiler compiled a method since the process start. The JIT compiler may be invoked multiple times for the same method to compile with different generic parameters, or because tiered compilation requested different optimization settings.
process_runtime_dotnet_jit_methods_compiled_count_total 33952 1709536344519

# TYPE process_runtime_dotnet_jit_compilation_time_nanoseconds_total counter
# UNIT process_runtime_dotnet_jit_compilation_time_nanoseconds_total nanoseconds
# HELP process_runtime_dotnet_jit_compilation_time_nanoseconds_total The amount of time the JIT compiler has spent compiling methods since the process start.
process_runtime_dotnet_jit_compilation_time_nanoseconds_total 13506887400 1709536344519

# TYPE process_runtime_dotnet_monitor_lock_contention_count_total counter
# HELP process_runtime_dotnet_monitor_lock_contention_count_total The number of times there was contention when trying to acquire a monitor lock since the process start. Monitor locks are commonly acquired by using the lock keyword in C#, or by calling Monitor.Enter() and Monitor.TryEnter().
process_runtime_dotnet_monitor_lock_contention_count_total 223139 1709536344519

# TYPE process_runtime_dotnet_thread_pool_threads_count gauge
# HELP process_runtime_dotnet_thread_pool_threads_count The number of thread pool threads that currently exist.
process_runtime_dotnet_thread_pool_threads_count 5 1709536344519

# TYPE process_runtime_dotnet_thread_pool_completed_items_count_total counter
# HELP process_runtime_dotnet_thread_pool_completed_items_count_total The number of work items that have been processed by the thread pool since the process start.
process_runtime_dotnet_thread_pool_completed_items_count_total 2423197 1709536344519

# TYPE process_runtime_dotnet_thread_pool_queue_length gauge
# HELP process_runtime_dotnet_thread_pool_queue_length The number of work items that are currently queued to be processed by the thread pool.
process_runtime_dotnet_thread_pool_queue_length 0 1709536344519

# TYPE process_runtime_dotnet_timer_count gauge
# HELP process_runtime_dotnet_timer_count The number of timer instances that are currently active. Timers can be created by many sources such as System.Threading.Timer, Task.Delay, or the timeout in a CancellationSource. An active timer is registered to tick at some point in the future and has not yet been canceled.
process_runtime_dotnet_timer_count 16 1709536344519

# TYPE process_runtime_dotnet_assemblies_count gauge
# HELP process_runtime_dotnet_assemblies_count The number of .NET assemblies that are currently loaded.
process_runtime_dotnet_assemblies_count 205 1709536344519

# TYPE process_runtime_dotnet_exceptions_count_total counter
# HELP process_runtime_dotnet_exceptions_count_total Count of exceptions that have been thrown in managed code, since the observation started. The value will be unavailable until an exception has been thrown after OpenTelemetry.Instrumentation.Runtime initialization.
process_runtime_dotnet_exceptions_count_total 156 1709536344519

# TYPE iiot_edge_publisher_module_start gauge
# HELP iiot_edge_publisher_module_start Publisher module started.
iiot_edge_publisher_module_start{publisherId="mpx"} 1 1709536344519

# TYPE iiot_edge_publisher_module_state gauge
# HELP iiot_edge_publisher_module_state Publisher module runtime state.
iiot_edge_publisher_module_state{publisherId="mpx"} 1 1709536344519

# TYPE iiot_edge_publisher_certificate_renewal_count_total counter
# HELP iiot_edge_publisher_certificate_renewal_count_total Publisher certificate renewals.
iiot_edge_publisher_certificate_renewal_count_total{publisherId="mpx"} 1 1709536344519

# TYPE kestrel_active_connections gauge
# HELP kestrel_active_connections Number of connections that are currently active on the server.
kestrel_active_connections{network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 2 1709536344519

# TYPE kestrel_connection_duration_seconds histogram
# UNIT kestrel_connection_duration_seconds seconds
# HELP kestrel_connection_duration_seconds The duration of connections on the server.
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.01"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.02"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.05"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.1"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.2"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.5"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="1"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="2"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="5"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="10"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="30"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="60"} 0 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="120"} 713 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="300"} 713 1709536344519
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="+Inf"} 713 1709536344519
kestrel_connection_duration_seconds_sum{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 52986.10865109997 1709536344519
kestrel_connection_duration_seconds_count{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 713 1709536344519

# TYPE kestrel_queued_connections gauge
# HELP kestrel_queued_connections Number of connections that are currently queued and are waiting to start.
kestrel_queued_connections{network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 0 1709536344519

# TYPE http_server_active_requests gauge
# HELP http_server_active_requests Number of active HTTP server requests.
http_server_active_requests{http_request_method="GET",url_scheme="http"} 1 1709536344519

# TYPE http_server_request_duration_seconds histogram
# UNIT http_server_request_duration_seconds seconds
# HELP http_server_request_duration_seconds Duration of HTTP server requests.
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.005"} 578 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.01"} 673 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.025"} 712 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.05"} 713 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.075"} 713 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.1"} 713 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.25"} 714 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.5"} 714 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.75"} 714 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="1"} 714 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="2.5"} 714 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="5"} 714 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="7.5"} 714 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="10"} 714 1709536344519
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="+Inf"} 714 1709536344519
http_server_request_duration_seconds_sum{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http"} 2.703283400000002 1709536344519
http_server_request_duration_seconds_count{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http"} 714 1709536344519

# TYPE dns_lookup_duration_seconds histogram
# UNIT dns_lookup_duration_seconds seconds
# HELP dns_lookup_duration_seconds Measures the time taken to perform a DNS lookup.
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.005"} 49 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.01"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.025"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.05"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.075"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.1"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.25"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.5"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="0.75"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="1"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="2.5"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="5"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="7.5"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="10"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbompx0001.mbo.cz.corp",le="+Inf"} 52 1709536344519
dns_lookup_duration_seconds_sum{dns_question_name="czmbompx0001.mbo.cz.corp"} 0.05540119999999998 1709536344519
dns_lookup_duration_seconds_count{dns_question_name="czmbompx0001.mbo.cz.corp"} 52 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.005"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.01"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.025"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.05"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.075"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.1"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.25"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.5"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.75"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="1"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="2.5"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="5"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="7.5"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="10"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="",le="+Inf"} 1 1709536344519
dns_lookup_duration_seconds_sum{dns_question_name=""} 7.31E-05 1709536344519
dns_lookup_duration_seconds_count{dns_question_name=""} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.005"} 1 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.01"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.025"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.05"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.075"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.1"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.25"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.5"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="0.75"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="1"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="2.5"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="5"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="7.5"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="10"} 2 1709536344519
dns_lookup_duration_seconds_bucket{dns_question_name="czmbomii2103.ls.ege.ds",le="+Inf"} 2 1709536344519
dns_lookup_duration_seconds_sum{dns_question_name="czmbomii2103.ls.ege.ds"} 0.011269300000000001 1709536344519
dns_lookup_duration_seconds_count{dns_question_name="czmbomii2103.ls.ege.ds"} 2 1709536344519

# TYPE aspnetcore_routing_match_attempts_total counter
# HELP aspnetcore_routing_match_attempts_total Number of requests that were attempted to be matched to an endpoint.
aspnetcore_routing_match_attempts_total{aspnetcore_routing_match_status="failure"} 715 1709536344519

# TYPE iiot_edge_publisher_client_count gauge
# HELP iiot_edge_publisher_client_count Number of clients.
iiot_edge_publisher_client_count{publisherId="mpx"} 1 1709536344519

# TYPE iiot_edge_publisher_heartbeats_total counter
# HELP iiot_edge_publisher_heartbeats_total Total Heartbeats delivered for processing.
iiot_edge_publisher_heartbeats_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_heartbeats_per_second gauge
# HELP iiot_edge_publisher_heartbeats_per_second Opc Cyclic reads/second delivered for processing.
iiot_edge_publisher_heartbeats_per_second{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_heartbeats_per_second_last_min gauge
# HELP iiot_edge_publisher_heartbeats_per_second_last_min Opc Cyclic reads/second delivered for processing in last 60s.
iiot_edge_publisher_heartbeats_per_second_last_min{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_cyclicreads_total counter
# HELP iiot_edge_publisher_cyclicreads_total Total Cyclic reads delivered for processing.
iiot_edge_publisher_cyclicreads_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_cyclicreads_per_second gauge
# HELP iiot_edge_publisher_cyclicreads_per_second Opc Cyclic reads/second delivered for processing.
iiot_edge_publisher_cyclicreads_per_second{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_cyclicreads_per_second_last_min gauge
# HELP iiot_edge_publisher_cyclicreads_per_second_last_min Opc Cyclic reads/second delivered for processing in last 60s.
iiot_edge_publisher_cyclicreads_per_second_last_min{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_modelchanges_total counter
# HELP iiot_edge_publisher_modelchanges_total Total Number of changes found in the address spaces of the connected servers.
iiot_edge_publisher_modelchanges_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_modelchanges_per_second gauge
# HELP iiot_edge_publisher_modelchanges_per_second Address space Model changes/second delivered for processing.
iiot_edge_publisher_modelchanges_per_second{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_modelchanges_per_second_last_min gauge
# HELP iiot_edge_publisher_modelchanges_per_second_last_min Address space Model changes/second delivered for processing in last 60s.
iiot_edge_publisher_modelchanges_per_second_last_min{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_value_changes_total counter
# HELP iiot_edge_publisher_value_changes_total Total Opc Value changes delivered for processing.
iiot_edge_publisher_value_changes_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 3823081 1709536344519

# TYPE iiot_edge_publisher_value_changes_per_second gauge
# HELP iiot_edge_publisher_value_changes_per_second Opc Value changes/second delivered for processing.
iiot_edge_publisher_value_changes_per_second{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 17.5605147029454 1709536344519

# TYPE iiot_edge_publisher_value_changes_per_second_last_min gauge
# HELP iiot_edge_publisher_value_changes_per_second_last_min Opc Value changes/second delivered for processing in last 60s.
iiot_edge_publisher_value_changes_per_second_last_min{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 1209 1709536344519

# TYPE iiot_edge_publisher_events_total counter
# HELP iiot_edge_publisher_events_total Total Opc Events delivered for processing.
iiot_edge_publisher_events_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_events_per_second gauge
# HELP iiot_edge_publisher_events_per_second Opc Events/second delivered for processing.
iiot_edge_publisher_events_per_second{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_events_per_second_last_min gauge
# HELP iiot_edge_publisher_events_per_second_last_min Opc Events/second delivered for processing in last 60s.
iiot_edge_publisher_events_per_second_last_min{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_event_notifications_total counter
# HELP iiot_edge_publisher_event_notifications_total Total Opc Event notifications delivered for processing.
iiot_edge_publisher_event_notifications_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_event_notifications_per_second gauge
# HELP iiot_edge_publisher_event_notifications_per_second Opc Event notifications/second delivered for processing.
iiot_edge_publisher_event_notifications_per_second{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_event_notifications_per_second_last_min gauge
# HELP iiot_edge_publisher_event_notifications_per_second_last_min Opc Event notifications/second delivered for processing in last 60s.
iiot_edge_publisher_event_notifications_per_second_last_min{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_data_changes_total counter
# HELP iiot_edge_publisher_data_changes_total Total Opc Data change notifications delivered for processing.
iiot_edge_publisher_data_changes_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 512780 1709536344519

# TYPE iiot_edge_publisher_data_changes_per_second gauge
# HELP iiot_edge_publisher_data_changes_per_second Opc Data change notifications/second delivered for processing.
iiot_edge_publisher_data_changes_per_second{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 2.355346572065202 1709536344519

# TYPE iiot_edge_publisher_data_changes_per_second_last_min gauge
# HELP iiot_edge_publisher_data_changes_per_second_last_min Opc Data change notifications/second delivered for processing in last 60s.
iiot_edge_publisher_data_changes_per_second_last_min{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 171 1709536344519

# TYPE iiot_edge_publisher_queue_overflows_total counter
# HELP iiot_edge_publisher_queue_overflows_total Total values received with a queue overflow indicator.
iiot_edge_publisher_queue_overflows_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_queue_overflows_per_second gauge
# HELP iiot_edge_publisher_queue_overflows_per_second Values with overflow indicator/second received.
iiot_edge_publisher_queue_overflows_per_second{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_queue_overflows_per_second_last_min gauge
# HELP iiot_edge_publisher_queue_overflows_per_second_last_min Values with overflow indicator/second received in last 60s.
iiot_edge_publisher_queue_overflows_per_second_last_min{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_keep_alive_notifications_total counter
# HELP iiot_edge_publisher_keep_alive_notifications_total Total Opc keep alive notifications delivered for processing.
iiot_edge_publisher_keep_alive_notifications_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 8957 1709536344519

# TYPE iiot_edge_publisher_subscriptions gauge
# HELP iiot_edge_publisher_subscriptions Number of Writers/Subscriptions in the writer group.
iiot_edge_publisher_subscriptions{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 3 1709536344519

# TYPE iiot_edge_publisher_connection_retries gauge
# HELP iiot_edge_publisher_connection_retries OPC UA connect retries.
iiot_edge_publisher_connection_retries{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_is_connection_ok gauge
# HELP iiot_edge_publisher_is_connection_ok OPC UA endpoints that are successfully connected.
iiot_edge_publisher_is_connection_ok{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 1 1709536344519

# TYPE iiot_edge_publisher_is_disconnected gauge
# HELP iiot_edge_publisher_is_disconnected OPC UA endpoints that are disconnected.
iiot_edge_publisher_is_disconnected{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_encoded_notifications_total counter
# HELP iiot_edge_publisher_encoded_notifications_total Number of successfully processed subscription notifications received from OPC client.
iiot_edge_publisher_encoded_notifications_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 512771 1709536344519

# TYPE iiot_edge_publisher_dropped_notifications_total counter
# HELP iiot_edge_publisher_dropped_notifications_total Number of incoming subscription notifications that are too big to be processed based on the message size limits or other issues with the notification.
iiot_edge_publisher_dropped_notifications_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_processed_messages_total counter
# HELP iiot_edge_publisher_processed_messages_total Number of successfully generated messages that are to be sent using the message sender
iiot_edge_publisher_processed_messages_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 21772 1709536344519

# TYPE iiot_edge_publisher_notifications_per_message_average gauge
# HELP iiot_edge_publisher_notifications_per_message_average Average subscription notifications packed into a message
iiot_edge_publisher_notifications_per_message_average{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 23.551855594341262 1709536344519

# TYPE iiot_edge_publisher_encoded_message_size_average gauge
# HELP iiot_edge_publisher_encoded_message_size_average Average size of a message through the lifetime of the encoder.
iiot_edge_publisher_encoded_message_size_average{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 37349.0943413558 1709536344519

# TYPE iiot_edge_publisher_chunk_size_average gauge
# HELP iiot_edge_publisher_chunk_size_average IoT Hub chunk size average
iiot_edge_publisher_chunk_size_average{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 9.118431235682568 1709536344519

# TYPE iiot_edge_publisher_message_split_ratio_max gauge
# HELP iiot_edge_publisher_message_split_ratio_max The message split ration specifies into how many messages a subscription notification had to be split. Less is better for performance. If the number is large user should attempt to limit the number of notifications in a message using configuration.
iiot_edge_publisher_message_split_ratio_max{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_send_queue_dropped_count_total counter
# HELP iiot_edge_publisher_send_queue_dropped_count_total Telemetry messages dropped due to overflow.
iiot_edge_publisher_send_queue_dropped_count_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_send_queue_size gauge
# HELP iiot_edge_publisher_send_queue_size Telemetry messages queued for sending upstream.
iiot_edge_publisher_send_queue_size{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_batch_input_queue_size gauge
# HELP iiot_edge_publisher_batch_input_queue_size Telemetry messages queued for sending upstream.
iiot_edge_publisher_batch_input_queue_size{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 21 1709536344519

# TYPE iiot_edge_publisher_encoding_input_queue_size gauge
# HELP iiot_edge_publisher_encoding_input_queue_size Telemetry messages queued for sending upstream.
iiot_edge_publisher_encoding_input_queue_size{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_encoding_output_queue_size gauge
# HELP iiot_edge_publisher_encoding_output_queue_size Telemetry messages queued for sending upstream.
iiot_edge_publisher_encoding_output_queue_size{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_messages_total counter
# HELP iiot_edge_publisher_messages_total Number of IoT messages successfully sent via transport.
iiot_edge_publisher_messages_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 21769 1709536344519

# TYPE iiot_edge_publisher_messages_per_second gauge
# HELP iiot_edge_publisher_messages_per_second Messages/second sent via transport.
iiot_edge_publisher_messages_per_second{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0.0999913076998132 1709536344519

# TYPE iiot_edge_publisher_message_send_failures_total counter
# HELP iiot_edge_publisher_message_send_failures_total Number of failures sending a network message.
iiot_edge_publisher_message_send_failures_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_sent_iot_messages_total counter
# HELP iiot_edge_publisher_sent_iot_messages_total Number of IoT messages successfully sent via transport.
iiot_edge_publisher_sent_iot_messages_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_sent_iot_messages_per_second gauge
# HELP iiot_edge_publisher_sent_iot_messages_per_second Messages/second sent via transport.
iiot_edge_publisher_sent_iot_messages_per_second{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_iothub_queue_dropped_count_total counter
# HELP iiot_edge_publisher_iothub_queue_dropped_count_total Telemetry messages dropped due to overflow.
iiot_edge_publisher_iothub_queue_dropped_count_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_iothub_queue_size gauge
# HELP iiot_edge_publisher_iothub_queue_size Telemetry messages queued for sending upstream.
iiot_edge_publisher_iothub_queue_size{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_failed_iot_messages_total counter
# HELP iiot_edge_publisher_failed_iot_messages_total Number of failures sending a network message.
iiot_edge_publisher_failed_iot_messages_total{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_missing_keep_alives_total counter
# HELP iiot_edge_publisher_missing_keep_alives_total Number of missing keep alives in subscription.
iiot_edge_publisher_missing_keep_alives_total{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519
iiot_edge_publisher_missing_keep_alives_total{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1 1709536344519
iiot_edge_publisher_missing_keep_alives_total{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_unassigned_notification_count_total counter
# HELP iiot_edge_publisher_unassigned_notification_count_total Number of notifications that could not be assigned.
iiot_edge_publisher_unassigned_notification_count_total{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519
iiot_edge_publisher_unassigned_notification_count_total{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519
iiot_edge_publisher_unassigned_notification_count_total{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_good_nodes gauge
# HELP iiot_edge_publisher_good_nodes Monitored items successfully created.
iiot_edge_publisher_good_nodes{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1000 1709536344519
iiot_edge_publisher_good_nodes{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1000 1709536344519
iiot_edge_publisher_good_nodes{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 801 1709536344519

# TYPE iiot_edge_publisher_bad_nodes gauge
# HELP iiot_edge_publisher_bad_nodes Monitored items with errors.
iiot_edge_publisher_bad_nodes{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519
iiot_edge_publisher_bad_nodes{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519
iiot_edge_publisher_bad_nodes{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_monitored_items gauge
# HELP iiot_edge_publisher_monitored_items Total monitored item count.
iiot_edge_publisher_monitored_items{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1000 1709536344519
iiot_edge_publisher_monitored_items{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1000 1709536344519
iiot_edge_publisher_monitored_items{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 801 1709536344519

# TYPE iiot_edge_publisher_publish_requests_per_subscription gauge
# HELP iiot_edge_publisher_publish_requests_per_subscription Good publish requests per subsciption.
iiot_edge_publisher_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1 1709536344519
iiot_edge_publisher_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1 1709536344519
iiot_edge_publisher_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1 1709536344519

# TYPE iiot_edge_publisher_good_publish_requests_per_subscription gauge
# HELP iiot_edge_publisher_good_publish_requests_per_subscription Good publish requests per subsciption.
iiot_edge_publisher_good_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1 1709536344519
iiot_edge_publisher_good_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1 1709536344519
iiot_edge_publisher_good_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1 1709536344519

# TYPE iiot_edge_publisher_bad_publish_requests_per_subscription gauge
# HELP iiot_edge_publisher_bad_publish_requests_per_subscription Bad publish requests per subsciption.
iiot_edge_publisher_bad_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519
iiot_edge_publisher_bad_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519
iiot_edge_publisher_bad_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 0 1709536344519

# TYPE iiot_edge_publisher_min_publish_requests_per_subscription gauge
# HELP iiot_edge_publisher_min_publish_requests_per_subscription Min publish requests queued per subsciption.
iiot_edge_publisher_min_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($c9cd47b65f358320b6316b09c8ea7fc0cf1f3fde)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1 1709536344519
iiot_edge_publisher_min_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($effa52693153959ccadeaf3bb6ad951452666611)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1 1709536344519
iiot_edge_publisher_min_publish_requests_per_subscription{dataSetWriterId="<<UnknownDataSet>>_($a6be81dbb77b4bcdb45eecf042e4d61ffd9a8ce7)",endpointUrl="opc.tcp://czmbomii2103.ls.ege.ds:49320",publisherId="mpx",securityMode="SignAndEncrypt",siteId="czmbompx0001",writerGroupId="x"} 1 1709536344519

# TYPE iiot_edge_publisher_client_connectivity_state gauge
# HELP iiot_edge_publisher_client_connectivity_state Client connectivity state.
iiot_edge_publisher_client_connectivity_state{publisherId="mpx"} 5 1709536344519

# TYPE iiot_edge_publisher_client_subscription_count gauge
# HELP iiot_edge_publisher_client_subscription_count Number of client managed subscriptions.
iiot_edge_publisher_client_subscription_count{publisherId="mpx"} 3 1709536344519

# TYPE iiot_edge_publisher_client_connectivity_retry_count gauge
# HELP iiot_edge_publisher_client_connectivity_retry_count Number of connectivity retries on this connection.
iiot_edge_publisher_client_connectivity_retry_count{publisherId="mpx"} 0 1709536344519

# TYPE iiot_edge_publisher_client_ref_count gauge
# HELP iiot_edge_publisher_client_ref_count Number of references to this client.
iiot_edge_publisher_client_ref_count{publisherId="mpx"} 3 1709536344519

# TYPE iiot_edge_publisher_client_good_publish_requests_count gauge
# HELP iiot_edge_publisher_client_good_publish_requests_count Number of good publish requests.
iiot_edge_publisher_client_good_publish_requests_count{publisherId="mpx"} 3 1709536344519

# TYPE iiot_edge_publisher_client_bad_publish_requests_count gauge
# HELP iiot_edge_publisher_client_bad_publish_requests_count Number of bad publish requests.
iiot_edge_publisher_client_bad_publish_requests_count{publisherId="mpx"} 0 1709536344519

# TYPE iiot_edge_publisher_client_min_publish_requests_count gauge
# HELP iiot_edge_publisher_client_min_publish_requests_count Number of min publish requests that should be queued.
iiot_edge_publisher_client_min_publish_requests_count{publisherId="mpx"} 3 1709536344519

# TYPE iiot_edge_publisher_client_outstanding_requests_count gauge
# HELP iiot_edge_publisher_client_outstanding_requests_count Number of outstanding requests.
iiot_edge_publisher_client_outstanding_requests_count{publisherId="mpx"} 3 1709536344519

# TYPE iiot_edge_publisher_client_publish_timeout_count gauge
# HELP iiot_edge_publisher_client_publish_timeout_count Number of timed out requests.
iiot_edge_publisher_client_publish_timeout_count{publisherId="mpx"} 0 1709536344519

# TYPE http_client_active_requests gauge
# HELP http_client_active_requests Number of outbound HTTP requests that are currently active on the client.
http_client_active_requests{http_request_method="POST",server_address="workload.sock",url_scheme="http"} 0 1709536344519

# TYPE http_client_request_duration_seconds histogram
# UNIT http_client_request_duration_seconds seconds
# HELP http_client_request_duration_seconds The duration of outbound HTTP requests.
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.005"} 0 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.01"} 0 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.025"} 0 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.05"} 0 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.075"} 0 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.1"} 0 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.25"} 1 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.5"} 1 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.75"} 1 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="1"} 1 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="2.5"} 1 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="5"} 1 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="7.5"} 1 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="10"} 1 1709536344519
http_client_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="+Inf"} 1 1709536344519
http_client_request_duration_seconds_sum{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http"} 0.1859284 1709536344519
http_client_request_duration_seconds_count{http_request_method="POST",http_response_status_code="201",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http"} 1 1709536344519

# TYPE http_client_open_connections gauge
# HELP http_client_open_connections Number of outbound HTTP connections that are currently active or idle on the client.
http_client_open_connections{http_connection_state="idle",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http"} 0 1709536344519
http_client_open_connections{http_connection_state="active",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http"} 0 1709536344519

# TYPE http_client_connection_duration_seconds histogram
# UNIT http_client_connection_duration_seconds seconds
# HELP http_client_connection_duration_seconds The duration of successfully established outbound HTTP connections.
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.01"} 0 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.02"} 0 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.05"} 0 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.1"} 0 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.2"} 1 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.5"} 1 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="1"} 1 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="2"} 1 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="5"} 1 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="10"} 1 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="30"} 1 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="60"} 1 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="120"} 1 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="300"} 1 1709536344519
http_client_connection_duration_seconds_bucket{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="+Inf"} 1 1709536344519
http_client_connection_duration_seconds_sum{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http"} 0.172 1709536344519
http_client_connection_duration_seconds_count{network_protocol_version="1.1",server_address="workload.sock",url_scheme="http"} 1 1709536344519

# TYPE http_client_request_time_in_queue_seconds histogram
# UNIT http_client_request_time_in_queue_seconds seconds
# HELP http_client_request_time_in_queue_seconds The amount of time requests spent on a queue waiting for an available connection.
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.005"} 0 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.01"} 0 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.025"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.05"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.075"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.1"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.25"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.5"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="0.75"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="1"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="2.5"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="5"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="7.5"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="10"} 1 1709536344519
http_client_request_time_in_queue_seconds_bucket{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http",le="+Inf"} 1 1709536344519
http_client_request_time_in_queue_seconds_sum{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http"} 0.0164352 1709536344519
http_client_request_time_in_queue_seconds_count{http_request_method="POST",network_protocol_version="1.1",server_address="workload.sock",url_scheme="http"} 1 1709536344519

# TYPE iiot_edge_publisher_session_count gauge
# HELP iiot_edge_publisher_session_count Number of active sessions.
iiot_edge_publisher_session_count{publisherId="mpx"} 1 1709536344519

# TYPE iiot_edge_publisher_messages_duration histogram
# HELP iiot_edge_publisher_messages_duration Histogram of message sending durations.
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="0"} 0 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="5"} 12057 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="10"} 21285 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="25"} 21740 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="50"} 21765 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="75"} 21766 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="100"} 21767 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="250"} 21768 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="500"} 21769 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="750"} 21769 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="1000"} 21769 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="2500"} 21769 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="5000"} 21769 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="7500"} 21769 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="10000"} 21769 1709536344519
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x",le="+Inf"} 21769 1709536344519
iiot_edge_publisher_messages_duration_sum{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 123716 1709536344519
iiot_edge_publisher_messages_duration_count{publisherId="mpx",siteId="czmbompx0001",writerGroupId="x"} 21769 1709536344519

# EOF

@jacqueskang jacqueskang changed the title OPC Publisher 2.9.4's Prometheus metrics partially missing probably due to unknown total size limit OPC Publisher 2.9.4's Prometheus metrics partially missing probably due to size limit Mar 4, 2024
@marcschier marcschier added the bug Something isn't working label Mar 6, 2024
@marcschier marcschier added this to the 2.9.5 milestone Mar 6, 2024
@marcschier
Copy link
Collaborator

marcschier commented Mar 7, 2024 •
edited
Loading

Likely due to the default of MaxMetricStreams in OTEL library being 1000. You could try setting

if (exporterOptions.Endpoint.Host != Otlp.OtlpEndpointDisabled)
{
    builder.SetMaxMetricStreams(3000);
   ...

in the Configuration.cs file of the module. If that works, please let me know. Otherwise, I have added to 2.9.5 release.

@jacqueskang
Copy link
Author

jacqueskang commented Mar 15, 2024 •
edited
Loading

I think we should fix the issue by removing low-value and non-documented metrics.

For me increasing the size limit by 300% is not reliable: why 3000 is enough? Why not 5000 or 10000?

I attached below another sample we captured in production. Half of the metrics response is filled with http_server_request_duration_seconds_bucket. This metric could be useful for a web site but probably not for opc publisher.

We care more abount "iiot_" metrics which are unfortunately almost missing

We are paying Azure for transfering metrics over IoT Hub and storaging them in Log analytics workspace. We should have an option to disable the metrics which we don't need

# TYPE process_runtime_dotnet_gc_collections_count_total counter
# HELP process_runtime_dotnet_gc_collections_count_total Number of garbage collections that have occurred since process start.
process_runtime_dotnet_gc_collections_count_total{generation="gen2"} 33 1710519082698
process_runtime_dotnet_gc_collections_count_total{generation="gen1"} 131 1710519082698
process_runtime_dotnet_gc_collections_count_total{generation="gen0"} 3366 1710519082698

# TYPE process_runtime_dotnet_gc_objects_size_bytes gauge
# UNIT process_runtime_dotnet_gc_objects_size_bytes bytes
# HELP process_runtime_dotnet_gc_objects_size_bytes Count of bytes currently in use by objects in the GC heap that haven't been collected yet. Fragmentation and other GC committed memory pools are excluded.
process_runtime_dotnet_gc_objects_size_bytes 175815176 1710519082698

# TYPE process_runtime_dotnet_gc_allocations_size_bytes_total counter
# UNIT process_runtime_dotnet_gc_allocations_size_bytes_total bytes
# HELP process_runtime_dotnet_gc_allocations_size_bytes_total Count of bytes allocated on the managed GC heap since the process start. .NET objects are allocated from this heap. Object allocations from unmanaged languages such as C/C++ do not use this heap.
process_runtime_dotnet_gc_allocations_size_bytes_total 192879568168 1710519082698

# TYPE process_runtime_dotnet_gc_committed_memory_size_bytes gauge
# UNIT process_runtime_dotnet_gc_committed_memory_size_bytes bytes
# HELP process_runtime_dotnet_gc_committed_memory_size_bytes The amount of committed virtual memory for the managed GC heap, as observed during the latest garbage collection. Committed virtual memory may be larger than the heap size because it includes both memory for storing existing objects (the heap size) and some extra memory that is ready to handle newly allocated objects in the future. The value will be unavailable until at least one garbage collection has occurred.
process_runtime_dotnet_gc_committed_memory_size_bytes 240283648 1710519082698

# TYPE process_runtime_dotnet_gc_heap_size_bytes gauge
# UNIT process_runtime_dotnet_gc_heap_size_bytes bytes
# HELP process_runtime_dotnet_gc_heap_size_bytes The heap size (including fragmentation), as observed during the latest garbage collection. The value will be unavailable until at least one garbage collection has occurred.
process_runtime_dotnet_gc_heap_size_bytes{generation="gen0"} 319520 1710519082698
process_runtime_dotnet_gc_heap_size_bytes{generation="gen1"} 6565936 1710519082698
process_runtime_dotnet_gc_heap_size_bytes{generation="gen2"} 90860264 1710519082698
process_runtime_dotnet_gc_heap_size_bytes{generation="loh"} 63800504 1710519082698
process_runtime_dotnet_gc_heap_size_bytes{generation="poh"} 683816 1710519082698

# TYPE process_runtime_dotnet_gc_heap_fragmentation_size_bytes gauge
# UNIT process_runtime_dotnet_gc_heap_fragmentation_size_bytes bytes
# HELP process_runtime_dotnet_gc_heap_fragmentation_size_bytes The heap fragmentation, as observed during the latest garbage collection. The value will be unavailable until at least one garbage collection has occurred.
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="gen0"} 6816 1710519082698
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="gen1"} 127008 1710519082698
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="gen2"} 10170888 1710519082698
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="loh"} 24415312 1710519082698
process_runtime_dotnet_gc_heap_fragmentation_size_bytes{generation="poh"} 0 1710519082698

# TYPE process_runtime_dotnet_gc_duration_nanoseconds_total counter
# UNIT process_runtime_dotnet_gc_duration_nanoseconds_total nanoseconds
# HELP process_runtime_dotnet_gc_duration_nanoseconds_total The total amount of time paused in GC since the process start.
process_runtime_dotnet_gc_duration_nanoseconds_total 25368350000 1710519082698

# TYPE process_runtime_dotnet_jit_il_compiled_size_bytes_total counter
# UNIT process_runtime_dotnet_jit_il_compiled_size_bytes_total bytes
# HELP process_runtime_dotnet_jit_il_compiled_size_bytes_total Count of bytes of intermediate language that have been compiled since the process start.
process_runtime_dotnet_jit_il_compiled_size_bytes_total 2993582 1710519082698

# TYPE process_runtime_dotnet_jit_methods_compiled_count_total counter
# HELP process_runtime_dotnet_jit_methods_compiled_count_total The number of times the JIT compiler compiled a method since the process start. The JIT compiler may be invoked multiple times for the same method to compile with different generic parameters, or because tiered compilation requested different optimization settings.
process_runtime_dotnet_jit_methods_compiled_count_total 40299 1710519082698

# TYPE process_runtime_dotnet_jit_compilation_time_nanoseconds_total counter
# UNIT process_runtime_dotnet_jit_compilation_time_nanoseconds_total nanoseconds
# HELP process_runtime_dotnet_jit_compilation_time_nanoseconds_total The amount of time the JIT compiler has spent compiling methods since the process start.
process_runtime_dotnet_jit_compilation_time_nanoseconds_total 15148355600 1710519082698

# TYPE process_runtime_dotnet_monitor_lock_contention_count_total counter
# HELP process_runtime_dotnet_monitor_lock_contention_count_total The number of times there was contention when trying to acquire a monitor lock since the process start. Monitor locks are commonly acquired by using the lock keyword in C#, or by calling Monitor.Enter() and Monitor.TryEnter().
process_runtime_dotnet_monitor_lock_contention_count_total 845435 1710519082698

# TYPE process_runtime_dotnet_thread_pool_threads_count gauge
# HELP process_runtime_dotnet_thread_pool_threads_count The number of thread pool threads that currently exist.
process_runtime_dotnet_thread_pool_threads_count 3 1710519082698

# TYPE process_runtime_dotnet_thread_pool_completed_items_count_total counter
# HELP process_runtime_dotnet_thread_pool_completed_items_count_total The number of work items that have been processed by the thread pool since the process start.
process_runtime_dotnet_thread_pool_completed_items_count_total 6048454 1710519082698

# TYPE process_runtime_dotnet_thread_pool_queue_length gauge
# HELP process_runtime_dotnet_thread_pool_queue_length The number of work items that are currently queued to be processed by the thread pool.
process_runtime_dotnet_thread_pool_queue_length 0 1710519082698

# TYPE process_runtime_dotnet_timer_count gauge
# HELP process_runtime_dotnet_timer_count The number of timer instances that are currently active. Timers can be created by many sources such as System.Threading.Timer, Task.Delay, or the timeout in a CancellationSource. An active timer is registered to tick at some point in the future and has not yet been canceled.
process_runtime_dotnet_timer_count 9 1710519082698

# TYPE process_runtime_dotnet_assemblies_count gauge
# HELP process_runtime_dotnet_assemblies_count The number of .NET assemblies that are currently loaded.
process_runtime_dotnet_assemblies_count 213 1710519082698

# TYPE process_runtime_dotnet_exceptions_count_total counter
# HELP process_runtime_dotnet_exceptions_count_total Count of exceptions that have been thrown in managed code, since the observation started. The value will be unavailable until an exception has been thrown after OpenTelemetry.Instrumentation.Runtime initialization.
process_runtime_dotnet_exceptions_count_total 156 1710519082698

# TYPE iiot_edge_publisher_module_start gauge
# HELP iiot_edge_publisher_module_start Publisher module started.
iiot_edge_publisher_module_start{publisherId="mpx"} 1 1710519082698

# TYPE iiot_edge_publisher_module_state gauge
# HELP iiot_edge_publisher_module_state Publisher module runtime state.
iiot_edge_publisher_module_state{publisherId="mpx"} 1 1710519082698

# TYPE iiot_edge_publisher_certificate_renewal_count_total counter
# HELP iiot_edge_publisher_certificate_renewal_count_total Publisher certificate renewals.
iiot_edge_publisher_certificate_renewal_count_total{publisherId="mpx"} 0 1710519082698

# TYPE kestrel_active_connections gauge
# HELP kestrel_active_connections Number of connections that are currently active on the server.
kestrel_active_connections{network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 2 1710519082698

# TYPE kestrel_connection_duration_seconds histogram
# UNIT kestrel_connection_duration_seconds seconds
# HELP kestrel_connection_duration_seconds The duration of connections on the server.
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.01"} 0 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.02"} 0 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.05"} 3 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.1"} 3 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.2"} 3 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="0.5"} 3 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="1"} 3 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="2"} 3 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="5"} 3 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="10"} 23 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="30"} 23 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="60"} 23 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="120"} 412 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="300"} 412 1710519082698
kestrel_connection_duration_seconds_bucket{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80",le="+Inf"} 412 1710519082698
kestrel_connection_duration_seconds_sum{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 28979.42869240001 1710519082698
kestrel_connection_duration_seconds_count{network_protocol_name="http",network_protocol_version="1.1",network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 412 1710519082698

# TYPE kestrel_queued_connections gauge
# HELP kestrel_queued_connections Number of connections that are currently queued and are waiting to start.
kestrel_queued_connections{network_transport="tcp",network_type="ipv6",server_address="::ffff:172.18.0.5",server_port="80"} 0 1710519082698

# TYPE http_server_active_requests gauge
# HELP http_server_active_requests Number of active HTTP server requests.
http_server_active_requests{http_request_method="GET",url_scheme="http"} 1 1710519082698
http_server_active_requests{http_request_method="POST",url_scheme="http"} 0 1710519082698

# TYPE http_server_request_duration_seconds histogram
# UNIT http_server_request_duration_seconds seconds
# HELP http_server_request_duration_seconds Duration of HTTP server requests.
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.005"} 132 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.01"} 275 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.025"} 389 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.05"} 393 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.075"} 393 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.1"} 393 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.25"} 394 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.5"} 394 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.75"} 394 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="1"} 394 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="2.5"} 395 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="5"} 395 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="7.5"} 395 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="10"} 395 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="+Inf"} 395 1710519082698
http_server_request_duration_seconds_sum{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http"} 4.961818300000001 1710519082698
http_server_request_duration_seconds_count{http_request_method="GET",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http"} 395 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="0.005"} 5 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="0.01"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="0.025"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="0.05"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="0.075"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="0.1"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="0.25"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="0.5"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="0.75"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="1"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="2.5"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="5"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="7.5"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="10"} 6 1710519082698
http_server_request_duration_seconds_bucket{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http",le="+Inf"} 6 1710519082698
http_server_request_duration_seconds_sum{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http"} 0.012533699999999998 1710519082698
http_server_request_duration_seconds_count{aspnetcore_request_is_unhandled="true",http_request_method="GET",http_response_status_code="404",network_protocol_version="1.1",url_scheme="http"} 6 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.005"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.01"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.025"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.05"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.075"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.1"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.25"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.75"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="1"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="2.5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="7.5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="10"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="+Inf"} 1 1710519082698
http_server_request_duration_seconds_sum{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http"} 0.0139666 1710519082698
http_server_request_duration_seconds_count{http_request_method="GET",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.005"} 3 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.01"} 3 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.025"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.05"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.075"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.1"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.25"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.5"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.75"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="1"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="2.5"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="5"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="7.5"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="10"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="+Inf"} 4 1710519082698
http_server_request_duration_seconds_sum{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http"} 0.031230400000000002 1710519082698
http_server_request_duration_seconds_count{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http"} 4 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="0.005"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="0.01"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="0.025"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="0.05"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="0.075"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="0.1"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="0.25"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="0.5"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="0.75"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="1"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="2.5"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="5"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="7.5"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="10"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http",le="+Inf"} 2 1710519082698
http_server_request_duration_seconds_sum{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http"} 0.0078034 1710519082698
http_server_request_duration_seconds_count{http_request_method="GET",http_response_status_code="401",http_route="v{version:apiVersion}/configuration",network_protocol_version="1.1",url_scheme="http"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.005"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.01"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.025"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.05"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.075"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.1"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.25"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="0.75"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="1"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="2.5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="7.5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="10"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http",le="+Inf"} 1 1710519082698
http_server_request_duration_seconds_sum{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http"} 0.0037561 1710519082698
http_server_request_duration_seconds_count{http_request_method="POST",http_response_status_code="200",network_protocol_version="1.1",url_scheme="http"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="0.005"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="0.01"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="0.025"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="0.05"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="0.075"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="0.1"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="0.25"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="0.5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="0.75"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="1"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="2.5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="7.5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="10"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http",le="+Inf"} 1 1710519082698
http_server_request_duration_seconds_sum{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http"} 0.0150159 1710519082698
http_server_request_duration_seconds_count{http_request_method="POST",http_response_status_code="401",http_route="v{version:apiVersion}/configuration/diagnostics",network_protocol_version="1.1",url_scheme="http"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.005"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.01"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.025"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.05"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.075"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.1"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.25"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.5"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="0.75"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="1"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="2.5"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="5"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="7.5"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="10"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http",le="+Inf"} 2 1710519082698
http_server_request_duration_seconds_sum{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http"} 0.0054338 1710519082698
http_server_request_duration_seconds_count{http_request_method="POST",http_response_status_code="405",network_protocol_version="1.1",url_scheme="http"} 2 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.005"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.01"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.025"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.05"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.075"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.1"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.25"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.5"} 0 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="0.75"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="1"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="2.5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="7.5"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="10"} 1 1710519082698
http_server_request_duration_seconds_bucket{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http",le="+Inf"} 1 1710519082698
http_server_request_duration_seconds_sum{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http"} 0.7441505 1710519082698
http_server_request_duration_seconds_count{http_request_method="GET",http_response_status_code="200",http_route="v{version:apiVersion}/reset",network_protocol_version="1.1",url_scheme="http"} 1 1710519082698

# TYPE dns_lookup_duration_seconds histogram
# UNIT dns_lookup_duration_seconds seconds
# HELP dns_lookup_duration_seconds Measures the time taken to perform a DNS lookup.
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="0.005"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="0.01"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="0.025"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="0.05"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="0.075"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="0.1"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="0.25"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="0.5"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="0.75"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="1"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="2.5"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="5"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="7.5"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="10"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmpx0001.gow.pl.corp",le="+Inf"} 30 1710519082698
dns_lookup_duration_seconds_sum{dns_question_name="plgowmpx0001.gow.pl.corp"} 0.0367265 1710519082698
dns_lookup_duration_seconds_count{dns_question_name="plgowmpx0001.gow.pl.corp"} 30 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.005"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.01"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.025"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.05"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.075"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.1"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.25"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.5"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="0.75"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="1"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="2.5"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="5"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="7.5"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="10"} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="",le="+Inf"} 1 1710519082698
dns_lookup_duration_seconds_sum{dns_question_name=""} 6.04E-05 1710519082698
dns_lookup_duration_seconds_count{dns_question_name=""} 1 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="0.005"} 11 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="0.01"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="0.025"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="0.05"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="0.075"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="0.1"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="0.25"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="0.5"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="0.75"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="1"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="2.5"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="5"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="7.5"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="10"} 12 1710519082698
dns_lookup_duration_seconds_bucket{dns_question_name="plgowmii2103.ls.ege.ds",le="+Inf"} 12 1710519082698
dns_lookup_duration_seconds_sum{dns_question_name="plgowmii2103.ls.ege.ds"} 0.0390277 1710519082698
dns_lookup_duration_seconds_count{dns_question_name="plgowmii2103.ls.ege.ds"} 12 1710519082698

# TYPE aspnetcore_routing_match_attempts_total counter
# HELP aspnetcore_routing_match_attempts_total Number of requests that were attempted to be matched to an endpoint.
aspnetcore_routing_match_attempts_total{aspnetcore_routing_match_status="failure"} 403 1710519082698
aspnetcore_routing_match_attempts_total{aspnetcore_routing_is_fallback="false",aspnetcore_routing_match_status="success",http_route="(missing)"} 3 1710519082698
aspnetcore_routing_match_attempts_total{aspnetcore_routing_is_fallback="false",aspnetcore_routing_match_status="success",http_route="v{version:apiVersion}/reset"} 5 1710519082698
aspnetcore_routing_match_attempts_total{aspnetcore_routing_is_fallback="false",aspnetcore_routing_match_status="success",http_route="v{version:apiVersion}/configuration"} 2 1710519082698
aspnetcore_routing_match_attempts_total{aspnetcore_routing_is_fallback="false",aspnetcore_routing_match_status="success",http_route="v{version:apiVersion}/configuration/diagnostics"} 1 1710519082698

# TYPE iiot_edge_publisher_client_count gauge
# HELP iiot_edge_publisher_client_count Number of clients.
iiot_edge_publisher_client_count{publisherId="mpx"} 2 1710519082698

# TYPE iiot_edge_publisher_client_connectivity_state gauge
# HELP iiot_edge_publisher_client_connectivity_state Client connectivity state.
iiot_edge_publisher_client_connectivity_state{publisherId="mpx"} 5 1710519082698

# TYPE iiot_edge_publisher_client_subscription_count gauge
# HELP iiot_edge_publisher_client_subscription_count Number of client managed subscriptions.
iiot_edge_publisher_client_subscription_count{publisherId="mpx"} 0 1710519082698

# TYPE iiot_edge_publisher_client_connectivity_retry_count gauge
# HELP iiot_edge_publisher_client_connectivity_retry_count Number of connectivity retries on this connection.
iiot_edge_publisher_client_connectivity_retry_count{publisherId="mpx"} 0 1710519082698

# TYPE iiot_edge_publisher_client_ref_count gauge
# HELP iiot_edge_publisher_client_ref_count Number of references to this client.
iiot_edge_publisher_client_ref_count{publisherId="mpx"} 0 1710519082698

# TYPE iiot_edge_publisher_client_good_publish_requests_count gauge
# HELP iiot_edge_publisher_client_good_publish_requests_count Number of good publish requests.
iiot_edge_publisher_client_good_publish_requests_count{publisherId="mpx"} 0 1710519082698

# TYPE iiot_edge_publisher_client_bad_publish_requests_count gauge
# HELP iiot_edge_publisher_client_bad_publish_requests_count Number of bad publish requests.
iiot_edge_publisher_client_bad_publish_requests_count{publisherId="mpx"} 0 1710519082698

# TYPE iiot_edge_publisher_client_min_publish_requests_count gauge
# HELP iiot_edge_publisher_client_min_publish_requests_count Number of min publish requests that should be queued.
iiot_edge_publisher_client_min_publish_requests_count{publisherId="mpx"} 1 1710519082698

# TYPE iiot_edge_publisher_client_outstanding_requests_count gauge
# HELP iiot_edge_publisher_client_outstanding_requests_count Number of outstanding requests.
iiot_edge_publisher_client_outstanding_requests_count{publisherId="mpx"} 0 1710519082698

# TYPE iiot_edge_publisher_client_publish_timeout_count gauge
# HELP iiot_edge_publisher_client_publish_timeout_count Number of timed out requests.
iiot_edge_publisher_client_publish_timeout_count{publisherId="mpx"} 0 1710519082698

# TYPE iiot_edge_publisher_session_count gauge
# HELP iiot_edge_publisher_session_count Number of active sessions.
iiot_edge_publisher_session_count{publisherId="mpx"} 2 1710519082698

# TYPE iiot_edge_publisher_messages_duration histogram
# HELP iiot_edge_publisher_messages_duration Histogram of message sending durations.
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="0"} 0 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="5"} 5513 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="10"} 30618 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="25"} 35563 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="50"} 35647 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="75"} 35648 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="100"} 35649 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="250"} 35653 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="500"} 35654 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="750"} 35657 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="1000"} 35657 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="2500"} 35657 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="5000"} 35657 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="7500"} 35657 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="10000"} 35657 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x",le="+Inf"} 35658 1710519082698
iiot_edge_publisher_messages_duration_sum{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x"} 349484 1710519082698
iiot_edge_publisher_messages_duration_count{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="x"} 35658 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="0"} 0 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="5"} 1 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="10"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="25"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="50"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="75"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="100"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="250"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="500"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="750"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="1000"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="2500"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="5000"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="7500"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="10000"} 8 1710519082698
iiot_edge_publisher_messages_duration_bucket{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test",le="+Inf"} 8 1710519082698
iiot_edge_publisher_messages_duration_sum{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test"} 56 1710519082698
iiot_edge_publisher_messages_duration_count{publisherId="mpx",siteId="plgowmpx0001",writerGroupId="test"} 8 1710519082698

# EOF

@marcschier marcschier added the feature request New feature or request label Mar 18, 2024
@alxy
Copy link

alxy commented Apr 30, 2024

@jacqueskang

We are paying Azure for transfering metrics over IoT Hub and storaging them in Log analytics workspace. We should have an option to disable the metrics which we don't need

If you are using metricscollector to collect and send the metrics to Azure, I can highly recommend to setup the environment variable AllowedMetrics=iiot_edge_publisher_* or similar on this module. As we all know, log analytics is really not the most cheap storage in the world...

@jacqueskang
Copy link
Author

jacqueskang commented Jun 11, 2024 •
edited
Loading

I would really appreciate if this regression can be fixed instead of being postponed again and again to later version.

We have deployed in production some Azure Monitor alert rules and dashboard relying on Prometheus metrics which are now missing since v2.9.4. This bug makes our operation team difficult to monitor the system.

@marcschier
Copy link
Collaborator

marcschier commented Jun 17, 2024 •
edited
Loading

In 2.9.9 we are adding a--ari command line option to enable all automatic .net runtime instrumentation. By default it will be off. This will reduce the metrics significantly and align with previous behavior. We will also disable "total" suffix, which can be enabled by --ats command line option. Furthermore response caching will be disabled, cannot be enabled/configured.

@marcschier marcschier mentioned this issue Jun 17, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working feature request New feature or request
Projects
None yet
Milestone
2.9.9
Development

No branches or pull requests
3 participants
@alxy @marcschier @jacqueskang