Failed to query Prometheus and fetched metrics

[ketanadiyal]

I have followed the step as per given and configured the default config.toml file and changed the NSE related paramter after run the manually ./mii-lama.bin --config config.toml but getting following error.
time=2025-02-25T14:50:20.764+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:164 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.142:9100 metric=packet_errors error="response contains no result data"
time=2025-02-25T14:50:20.764+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:211 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.142:9100 metric=failure_count error="response contains no result data"
time=2025-02-25T14:50:20.764+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:131 msg="Failed to query Prometheus for database status" component=mii-lama host=192.168.201.142:9100 metric=status error="response contains no result data"
time=2025-02-25T14:50:20.765+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:85 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.142:9100 metric=disk error="response contains no result data"
time=2025-02-25T14:50:20.765+05:30 level=DEBUG source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:181 msg="fetched metrics" component=mii-lama host=192.168.201.142:9100 data={PacketErrors:0}
time=2025-02-25T14:50:20.765+05:30 level=DEBUG source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:148 msg="fetched metrics" component=mii-lama host=192.168.201.142:9100 data={Status:0}
time=2025-02-25T14:50:20.765+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:199 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.142:9100 metric=throughput error="response contains no result data"
time=2025-02-25T14:50:20.765+05:30 level=DEBUG source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:228 msg="fetched metrics" component=mii-lama host=192.168.201.142:9100 data="{Throughput:0 FailureCount:0}"
time=2025-02-25T14:50:20.765+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:96 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.142:9100 metric=uptime error="response contains no result data"
time=2025-02-25T14:50:20.766+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:131 msg="Failed to query Prometheus for database status" component=mii-lama host=192.168.201.143:9100 metric=status error="response contains no result data"
time=2025-02-25T14:50:20.766+05:30 level=DEBUG source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:148 msg="fetched metrics" component=mii-lama host=192.168.201.143:9100 data={Status:0}
time=2025-02-25T14:50:20.767+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:63 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.142:9100 metric=cpu error="response contains no result data"
time=2025-02-25T14:50:20.767+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:164 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.143:9100 metric=packet_errors error="response contains no result data"
time=2025-02-25T14:50:20.767+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:211 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.143:9100 metric=failure_count error="response contains no result data"
time=2025-02-25T14:50:20.767+05:30 level=DEBUG source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:181 msg="fetched metrics" component=mii-lama host=192.168.201.143:9100 data={PacketErrors:0}
time=2025-02-25T14:50:20.767+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:131 msg="Failed to query Prometheus for database status" component=mii-lama host=192.168.201.78:9100 metric=status error="response contains no result data"
time=2025-02-25T14:50:20.772+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:74 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.143:9100 metric=memory error="response contains no result data"
time=2025-02-25T14:50:20.772+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:85 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.143:9100 metric=disk error="response contains no result data"
time=2025-02-25T14:50:20.773+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:96 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.143:9100 metric=uptime error="response contains no result data"
time=2025-02-25T14:50:20.773+05:30 level=DEBUG source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:113 msg="fetched metrics" component=mii-lama host=192.168.201.143:9100 data="{CPU:0 Mem:0 Disk:0 Uptime:0}"
time=2025-02-25T14:50:20.773+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:63 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.78:9100 metric=cpu error="response contains no result data"
time=2025-02-25T14:50:20.774+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:74 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.78:9100 metric=memory error="response contains no result data"
time=2025-02-25T14:50:20.775+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:85 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.78:9100 metric=disk error="response contains no result data"
time=2025-02-25T14:50:20.775+05:30 level=ERROR source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:96 msg="Failed to query Prometheus" component=mii-lama host=192.168.201.78:9100 metric=uptime error="response contains no result data"
time=2025-02-25T14:50:20.775+05:30 level=DEBUG source=/home/runner/work/mii-lama/mii-lama/cmd/app.go:113 msg="fetched metrics" component=mii-lama host=192.168.201.78:9100 data="{CPU:0 Mem:0 Disk:0 Uptime:0}"
time=2025-02-25T14:50:20.775+05:30 level=INFO source=/home/runner/work/mii-lama/mii-lama/internal/nse/nse.go:253 msg="Preparing to send hardware metrics" component=mii-lama login_id=apim_13 member_id=13 exchange_id=1 host=192.168.201.142:9100 URL=https://lama.uat.nseindia.com/api/V1/metrics/hardware payload="{"memberId":"13605","exchangeId":1,"sequenceId":1,"timestamp":1740475220,"payload":[{"applicationId":1,"metricData":[{"key":"cpu","value":{"min":0,"max":0,"avg":0,"med":0}},{"key":"memory","value":{"min":0,"max":0,"avg":0,"med":0}},{"key":"disk","value":{"min":0,"max":0,"avg":0,"med":0}},{"key":"uptime","value":{"min":0,"max":0,"avg":0,"med":0}}]}]}" headers="map[Accept-Language:[en-US] Content-Type:[application/json] Cookie:[test] Referer:[https://lama.uat.nseindia.com] User-Agent:[LAMAAPI/1.0.0]]"

Please let us know where we have to miss the step or any setting.

[mr-karan]

Looks like your Prometheus instance doesn't have the required node_exporter metrics. Please try running the PromQL queries against the Prometheus API to confirm the same.

[ketanadiyal]

Thanks for your update. I have change the method. Now we have configured mii-lama with docker and it's working fine but only received Avg responce.

How to get the other data -- Min,Max,Avg,Mid
is there any method or any single query to fetch all. Please suggest and help.

===================output of the Avg responce=======
time=2025-02-25T15:30:32.839Z level=INFO source=/home/runner/work/mii-lama/mii-lama/internal/nse/nse.go:253 msg="Preparing to send hardware metrics" component=mii-lama login_id=apim_1 member_id=1 exchange_id=1 host=192.168.201.142:9100 URL=https://lama.uat.nseindia.com/api/V1/metrics/hardware payload="{"memberId":"1","exchangeId":1,"sequenceId":1,"timestamp":1740497432,"payload":[{"applicationId":1,"metricData":[{"key":"cpu","value":{"min":0,"max":0,"avg":34.74,"med":0}},{"key":"memory","value":{"min":0,"max":0,"avg":0,"med":0}},{"key":"disk","value":{"min":0,"max":0,"avg":75.32,"med":0}},{"key":"uptime","value":{"min":0,"max":0,"avg":399049,"med":0}}]}]}" headers="map[Accept-Language:[en-US] Content-Type:[application/json] Cookie:[test] Referer:[https://lama.uat.nseindia.com] User-Agent:[LAMAAPI/1.0.0]]"

================ Configration parameter=============
[metrics.hardware] # Define Prometheus queries for hardware metrics

List of hosts to fetch metrics for. Keep this empty to fetch metrics for all hosts defined in prometheus.config_path file.

cpu = '100 * (1 - avg(rate(node_cpu_seconds_total{mode="idle", instance="%s"}[5m])))'
disk = '100 - ((node_filesystem_avail_bytes{instance="%s",device!"rootfs"} * 100) / node_filesystem_size_bytes{instance="%s",device!"rootfs"})'
hosts = []
memory = '(1 - ((node_memory_MemFree_bytes{instance="%s"} + node_memory_Buffers_bytes{instance="%s"} + node_memory_Cached_bytes{instance="%s"}) / node_memory_MemTotal_bytes{instance="%s"})) * 100'
uptime = '(node_time_seconds{instance="%s"} - node_boot_time_seconds{instance="%s"}) / 60'

[ketanadiyal]

I have manually checked the PromQL queries and received successful responses:

100 * (1 - min(rate(node_cpu_seconds_total{mode="idle", instance="192.168.201.142:9100"}[5m])))
100 * (1 - max(rate(node_cpu_seconds_total{mode="idle", instance="192.168.201.142:9100"}[5m])))
100 * (1 - avg(rate(node_cpu_seconds_total{mode="idle", instance="192.168.201.142:9100"}[5m])))
However, MII-LAMA is not picking up the values correctly. Additionally, I have observed that when configuring min and max instead of avg, MII-LAMA only picks up the avg value.

As per the logs, the min, max, and mid values are showing as zero. Could you please check and advise on this issue?

[mr-karan]

Hi. As of now the queries are defined in config.toml only compute the average values. The min/max values should have separate queries but this version doesn't support it yet. There are some new API params (like locationID) which are proposed to be added to the official spec. Once the official spec includes the new parameters like locationID, we'll be in a good position to implement separate queries for min/max values