diff --git a/.github/workflows/test.yml b/.github/workflows/test.yml index dc58ca283b..203deb8568 100644 --- a/.github/workflows/test.yml +++ b/.github/workflows/test.yml @@ -6,7 +6,7 @@ on: - 'release/*' pull_request: env: - CORE_REPO_SHA: eb98182acc54faeffc7440f9fbce8e71b4575fcc + CORE_REPO_SHA: f81381cf8aca64a707d934f20c6c27d40b949dce jobs: build: diff --git a/CHANGELOG.md b/CHANGELOG.md index 6cd943faab..d7597561a3 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,23 +7,29 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased](https://github.com/open-telemetry/opentelemetry-python/compare/v1.10.0-0.29b0...HEAD) +### Fixed + +- `opentelemetry-instrumentation-flask` Fix non-recording span bug + ([#999])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/999) +- `opentelemetry-instrumentation-tornado` Fix non-recording span bug + ([#999])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/999) + ### Added + - `opentelemetry-instrumentation-django` Capture custom request/response headers in span attributes ([#1024])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1024) - `opentelemetry-instrumentation-asgi` Capture custom request/response headers in span attributes ([#1004])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1004) - `opentelemetry-instrumentation-psycopg2` extended the sql commenter support of dbapi into psycopg2 ([#940](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/940)) -- `opentelemetry-instrumentation-flask` Fix non-recording span bug - ([#999])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/999) -- `opentelemetry-instrumentation-tornado` Fix non-recording span bug - ([#999])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/999) -- `opentelemetry-instrumentation-pyramid` Handle non-HTTPException exceptions - ([#1001](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1001)) - `opentelemetry-instrumentation-falcon` Falcon: Capture custom request/response headers in span attributes ([#1003])(https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1003) - `opentelemetry-instrumentation-elasticsearch` no longer creates unique span names by including search target, replaces them with `` and puts the value in attribute `elasticsearch.target` ([#1018](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1018)) +- `opentelemetry-instrumentation-pyramid` Handle non-HTTPException exceptions + ([#1001](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1001)) +- `opentelemetry-instrumentation-system-metrics` restore `SystemMetrics` instrumentation as `SystemMetricsInstrumentor` + ([#1012](https://github.com/open-telemetry/opentelemetry-python-contrib/pull/1012)) ## [1.10.0-0.29b0](https://github.com/open-telemetry/opentelemetry-python/releases/tag/v1.10.0-0.29b0) - 2022-03-10 diff --git a/docs-requirements.txt b/docs-requirements.txt index c7b3cb7df2..4efc0ca46e 100644 --- a/docs-requirements.txt +++ b/docs-requirements.txt @@ -12,7 +12,6 @@ sphinx-autodoc-typehints # Required by opentelemetry-instrumentation fastapi>=0.65.2 -psutil~=5.7.0 pymemcache~=1.3 # Required by conf @@ -29,6 +28,7 @@ flask~=2.0 falcon~=2.0 grpcio~=1.27 mysql-connector-python~=8.0 +psutil>=5 pymongo~=3.1 PyMySQL~=0.9.3 pyramid>=1.7 diff --git a/instrumentation/README.md b/instrumentation/README.md index caacdafdf6..d609e42326 100644 --- a/instrumentation/README.md +++ b/instrumentation/README.md @@ -33,6 +33,7 @@ | [opentelemetry-instrumentation-sqlalchemy](./opentelemetry-instrumentation-sqlalchemy) | sqlalchemy | | [opentelemetry-instrumentation-sqlite3](./opentelemetry-instrumentation-sqlite3) | sqlite3 | | [opentelemetry-instrumentation-starlette](./opentelemetry-instrumentation-starlette) | starlette ~= 0.13.0 | +| [opentelemetry-instrumentation-system-metrics](./opentelemetry-instrumentation-system-metrics) | psutil >= 5 | | [opentelemetry-instrumentation-tornado](./opentelemetry-instrumentation-tornado) | tornado >= 5.1.1 | | [opentelemetry-instrumentation-urllib](./opentelemetry-instrumentation-urllib) | urllib | | [opentelemetry-instrumentation-urllib3](./opentelemetry-instrumentation-urllib3) | urllib3 >= 1.0.0, < 2.0.0 | diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/LICENSE b/instrumentation/opentelemetry-instrumentation-system-metrics/LICENSE new file mode 100644 index 0000000000..261eeb9e9f --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/LICENSE @@ -0,0 +1,201 @@ + Apache License + Version 2.0, January 2004 + http://www.apache.org/licenses/ + + TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION + + 1. Definitions. + + "License" shall mean the terms and conditions for use, reproduction, + and distribution as defined by Sections 1 through 9 of this document. + + "Licensor" shall mean the copyright owner or entity authorized by + the copyright owner that is granting the License. + + "Legal Entity" shall mean the union of the acting entity and all + other entities that control, are controlled by, or are under common + control with that entity. For the purposes of this definition, + "control" means (i) the power, direct or indirect, to cause the + direction or management of such entity, whether by contract or + otherwise, or (ii) ownership of fifty percent (50%) or more of the + outstanding shares, or (iii) beneficial ownership of such entity. + + "You" (or "Your") shall mean an individual or Legal Entity + exercising permissions granted by this License. + + "Source" form shall mean the preferred form for making modifications, + including but not limited to software source code, documentation + source, and configuration files. + + "Object" form shall mean any form resulting from mechanical + transformation or translation of a Source form, including but + not limited to compiled object code, generated documentation, + and conversions to other media types. + + "Work" shall mean the work of authorship, whether in Source or + Object form, made available under the License, as indicated by a + copyright notice that is included in or attached to the work + (an example is provided in the Appendix below). + + "Derivative Works" shall mean any work, whether in Source or Object + form, that is based on (or derived from) the Work and for which the + editorial revisions, annotations, elaborations, or other modifications + represent, as a whole, an original work of authorship. For the purposes + of this License, Derivative Works shall not include works that remain + separable from, or merely link (or bind by name) to the interfaces of, + the Work and Derivative Works thereof. + + "Contribution" shall mean any work of authorship, including + the original version of the Work and any modifications or additions + to that Work or Derivative Works thereof, that is intentionally + submitted to Licensor for inclusion in the Work by the copyright owner + or by an individual or Legal Entity authorized to submit on behalf of + the copyright owner. For the purposes of this definition, "submitted" + means any form of electronic, verbal, or written communication sent + to the Licensor or its representatives, including but not limited to + communication on electronic mailing lists, source code control systems, + and issue tracking systems that are managed by, or on behalf of, the + Licensor for the purpose of discussing and improving the Work, but + excluding communication that is conspicuously marked or otherwise + designated in writing by the copyright owner as "Not a Contribution." + + "Contributor" shall mean Licensor and any individual or Legal Entity + on behalf of whom a Contribution has been received by Licensor and + subsequently incorporated within the Work. + + 2. Grant of Copyright License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + copyright license to reproduce, prepare Derivative Works of, + publicly display, publicly perform, sublicense, and distribute the + Work and such Derivative Works in Source or Object form. + + 3. Grant of Patent License. Subject to the terms and conditions of + this License, each Contributor hereby grants to You a perpetual, + worldwide, non-exclusive, no-charge, royalty-free, irrevocable + (except as stated in this section) patent license to make, have made, + use, offer to sell, sell, import, and otherwise transfer the Work, + where such license applies only to those patent claims licensable + by such Contributor that are necessarily infringed by their + Contribution(s) alone or by combination of their Contribution(s) + with the Work to which such Contribution(s) was submitted. If You + institute patent litigation against any entity (including a + cross-claim or counterclaim in a lawsuit) alleging that the Work + or a Contribution incorporated within the Work constitutes direct + or contributory patent infringement, then any patent licenses + granted to You under this License for that Work shall terminate + as of the date such litigation is filed. + + 4. Redistribution. You may reproduce and distribute copies of the + Work or Derivative Works thereof in any medium, with or without + modifications, and in Source or Object form, provided that You + meet the following conditions: + + (a) You must give any other recipients of the Work or + Derivative Works a copy of this License; and + + (b) You must cause any modified files to carry prominent notices + stating that You changed the files; and + + (c) You must retain, in the Source form of any Derivative Works + that You distribute, all copyright, patent, trademark, and + attribution notices from the Source form of the Work, + excluding those notices that do not pertain to any part of + the Derivative Works; and + + (d) If the Work includes a "NOTICE" text file as part of its + distribution, then any Derivative Works that You distribute must + include a readable copy of the attribution notices contained + within such NOTICE file, excluding those notices that do not + pertain to any part of the Derivative Works, in at least one + of the following places: within a NOTICE text file distributed + as part of the Derivative Works; within the Source form or + documentation, if provided along with the Derivative Works; or, + within a display generated by the Derivative Works, if and + wherever such third-party notices normally appear. The contents + of the NOTICE file are for informational purposes only and + do not modify the License. You may add Your own attribution + notices within Derivative Works that You distribute, alongside + or as an addendum to the NOTICE text from the Work, provided + that such additional attribution notices cannot be construed + as modifying the License. + + You may add Your own copyright statement to Your modifications and + may provide additional or different license terms and conditions + for use, reproduction, or distribution of Your modifications, or + for any such Derivative Works as a whole, provided Your use, + reproduction, and distribution of the Work otherwise complies with + the conditions stated in this License. + + 5. Submission of Contributions. Unless You explicitly state otherwise, + any Contribution intentionally submitted for inclusion in the Work + by You to the Licensor shall be under the terms and conditions of + this License, without any additional terms or conditions. + Notwithstanding the above, nothing herein shall supersede or modify + the terms of any separate license agreement you may have executed + with Licensor regarding such Contributions. + + 6. Trademarks. This License does not grant permission to use the trade + names, trademarks, service marks, or product names of the Licensor, + except as required for reasonable and customary use in describing the + origin of the Work and reproducing the content of the NOTICE file. + + 7. Disclaimer of Warranty. Unless required by applicable law or + agreed to in writing, Licensor provides the Work (and each + Contributor provides its Contributions) on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or + implied, including, without limitation, any warranties or conditions + of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A + PARTICULAR PURPOSE. You are solely responsible for determining the + appropriateness of using or redistributing the Work and assume any + risks associated with Your exercise of permissions under this License. + + 8. Limitation of Liability. In no event and under no legal theory, + whether in tort (including negligence), contract, or otherwise, + unless required by applicable law (such as deliberate and grossly + negligent acts) or agreed to in writing, shall any Contributor be + liable to You for damages, including any direct, indirect, special, + incidental, or consequential damages of any character arising as a + result of this License or out of the use or inability to use the + Work (including but not limited to damages for loss of goodwill, + work stoppage, computer failure or malfunction, or any and all + other commercial damages or losses), even if such Contributor + has been advised of the possibility of such damages. + + 9. Accepting Warranty or Additional Liability. While redistributing + the Work or Derivative Works thereof, You may choose to offer, + and charge a fee for, acceptance of support, warranty, indemnity, + or other liability obligations and/or rights consistent with this + License. However, in accepting such obligations, You may act only + on Your own behalf and on Your sole responsibility, not on behalf + of any other Contributor, and only if You agree to indemnify, + defend, and hold each Contributor harmless for any liability + incurred by, or claims asserted against, such Contributor by reason + of your accepting any such warranty or additional liability. + + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "[]" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright [yyyy] [name of copyright owner] + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/MANIFEST.in b/instrumentation/opentelemetry-instrumentation-system-metrics/MANIFEST.in new file mode 100644 index 0000000000..aed3e33273 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/MANIFEST.in @@ -0,0 +1,9 @@ +graft src +graft tests +global-exclude *.pyc +global-exclude *.pyo +global-exclude __pycache__/* +include CHANGELOG.md +include MANIFEST.in +include README.rst +include LICENSE diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/README.rst b/instrumentation/opentelemetry-instrumentation-system-metrics/README.rst new file mode 100644 index 0000000000..7a6c65b59b --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/README.rst @@ -0,0 +1,24 @@ +OpenTelemetry System Metrics Instrumentation +============================================ + +|pypi| + +.. |pypi| image:: https://badge.fury.io/py/opentelemetry-instrumentation-system-metrics.svg + :target: https://pypi.org/project/opentelemetry-instrumentation-system-metrics/ + +Instrumentation to collect system performance metrics. + + +Installation +------------ + +:: + + pip install opentelemetry-instrumentation-system-metrics + + +References +---------- +* `OpenTelemetry System Metrics Instrumentation `_ +* `OpenTelemetry Project `_ + diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg new file mode 100644 index 0000000000..84af307853 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.cfg @@ -0,0 +1,55 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +[metadata] +name = opentelemetry-instrumentation-system-metrics +description = OpenTelemetry System Metrics Instrumentation +long_description = file: README.rst +long_description_content_type = text/x-rst +author = OpenTelemetry Authors +author_email = cncf-opentelemetry-contributors@lists.cncf.io +url = https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-system-metrics +platforms = any +license = Apache-2.0 +classifiers = + Development Status :: 4 - Beta + Intended Audience :: Developers + License :: OSI Approved :: Apache Software License + Programming Language :: Python + Programming Language :: Python :: 3 + Programming Language :: Python :: 3.6 + Programming Language :: Python :: 3.7 + Programming Language :: Python :: 3.8 + Programming Language :: Python :: 3.9 + Programming Language :: Python :: 3.10 + +[options] +python_requires = >=3.6 +package_dir= + =src +packages=find_namespace: +install_requires = + opentelemetry-api ~= 1.10.0 + opentelemetry-sdk ~= 1.10.0 + +[options.extras_require] +test = + opentelemetry-test-utils == 0.29b0 + +[options.packages.find] +where = src + +[options.entry_points] +opentelemetry_instrumentor = + system_metrics = opentelemetry.instrumentation.system_metrics:SystemMetricsInstrumentor diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/setup.py b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.py new file mode 100644 index 0000000000..a3baaf78f9 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/setup.py @@ -0,0 +1,99 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +# DO NOT EDIT. THIS FILE WAS AUTOGENERATED FROM templates/instrumentation_setup.py.txt. +# RUN `python scripts/generate_setup.py` TO REGENERATE. + + +import distutils.cmd +import json +import os +from configparser import ConfigParser + +import setuptools + +config = ConfigParser() +config.read("setup.cfg") + +# We provide extras_require parameter to setuptools.setup later which +# overwrites the extras_require section from setup.cfg. To support extras_require +# section in setup.cfg, we load it here and merge it with the extras_require param. +extras_require = {} +if "options.extras_require" in config: + for key, value in config["options.extras_require"].items(): + extras_require[key] = [v for v in value.split("\n") if v.strip()] + +BASE_DIR = os.path.dirname(__file__) +PACKAGE_INFO = {} + +VERSION_FILENAME = os.path.join( + BASE_DIR, + "src", + "opentelemetry", + "instrumentation", + "system_metrics", + "version.py", +) +with open(VERSION_FILENAME, encoding="utf-8") as f: + exec(f.read(), PACKAGE_INFO) + +PACKAGE_FILENAME = os.path.join( + BASE_DIR, + "src", + "opentelemetry", + "instrumentation", + "system_metrics", + "package.py", +) +with open(PACKAGE_FILENAME, encoding="utf-8") as f: + exec(f.read(), PACKAGE_INFO) + +# Mark any instruments/runtime dependencies as test dependencies as well. +extras_require["instruments"] = PACKAGE_INFO["_instruments"] +test_deps = extras_require.get("test", []) +for dep in extras_require["instruments"]: + test_deps.append(dep) + +extras_require["test"] = test_deps + + +class JSONMetadataCommand(distutils.cmd.Command): + + description = ( + "print out package metadata as JSON. This is used by OpenTelemetry dev scripts to ", + "auto-generate code in other places", + ) + user_options = [] + + def initialize_options(self): + pass + + def finalize_options(self): + pass + + def run(self): + metadata = { + "name": config["metadata"]["name"], + "version": PACKAGE_INFO["__version__"], + "instruments": PACKAGE_INFO["_instruments"], + } + print(json.dumps(metadata)) + + +setuptools.setup( + cmdclass={"meta": JSONMetadataCommand}, + version=PACKAGE_INFO["__version__"], + extras_require=extras_require, +) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py new file mode 100644 index 0000000000..3317b3fc65 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/__init__.py @@ -0,0 +1,569 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +Instrument to report system (CPU, memory, network) and +process (CPU, memory, garbage collection) metrics. By default, the +following metrics are configured: + +.. code:: python + + { + "system.cpu.time": ["idle", "user", "system", "irq"], + "system.cpu.utilization": ["idle", "user", "system", "irq"], + "system.memory.usage": ["used", "free", "cached"], + "system.memory.utilization": ["used", "free", "cached"], + "system.swap.usage": ["used", "free"], + "system.swap.utilization": ["used", "free"], + "system.disk.io": ["read", "write"], + "system.disk.operations": ["read", "write"], + "system.disk.time": ["read", "write"], + "system.network.dropped.packets": ["transmit", "receive"], + "system.network.packets": ["transmit", "receive"], + "system.network.errors": ["transmit", "receive"], + "system.network.io": ["trasmit", "receive"], + "system.network.connections": ["family", "type"], + "runtime.memory": ["rss", "vms"], + "runtime.cpu.time": ["user", "system"], + } + +Usage +----- + +.. code:: python + + from opentelemetry._metrics import set_meter_provider + from opentelemetry.instrumentation.system_metrics import SystemMetricsInstrumentor + from opentelemetry.sdk._metrics import MeterProvider + from opentelemetry.sdk._metrics.export import ConsoleMetricExporter, PeriodicExportingMetricReader + + exporter = ConsoleMetricExporter() + + set_meter_provider(MeterProvider(PeriodicExportingMetricReader(exporter))) + SystemMetricsInstrumentor().instrument() + + # metrics are collected asynchronously + input("...") + + # to configure custom metrics + configuration = { + "system.memory.usage": ["used", "free", "cached"], + "system.cpu.time": ["idle", "user", "system", "irq"], + "system.network.io": ["trasmit", "receive"], + "runtime.memory": ["rss", "vms"], + "runtime.cpu.time": ["user", "system"], + } + SystemMetricsInstrumentor(config=configuration).instrument() + +API +--- +""" + +import gc +import os +from platform import python_implementation +from typing import Collection, Dict, Iterable, List, Optional + +import psutil + +from opentelemetry._metrics import get_meter +from opentelemetry._metrics.measurement import Measurement +from opentelemetry.instrumentation.instrumentor import BaseInstrumentor +from opentelemetry.instrumentation.system_metrics.package import _instruments +from opentelemetry.instrumentation.system_metrics.version import __version__ +from opentelemetry.sdk.util import get_dict_as_key + +_DEFAULT_CONFIG = { + "system.cpu.time": ["idle", "user", "system", "irq"], + "system.cpu.utilization": ["idle", "user", "system", "irq"], + "system.memory.usage": ["used", "free", "cached"], + "system.memory.utilization": ["used", "free", "cached"], + "system.swap.usage": ["used", "free"], + "system.swap.utilization": ["used", "free"], + "system.disk.io": ["read", "write"], + "system.disk.operations": ["read", "write"], + "system.disk.time": ["read", "write"], + "system.network.dropped.packets": ["transmit", "receive"], + "system.network.packets": ["transmit", "receive"], + "system.network.errors": ["transmit", "receive"], + "system.network.io": ["trasmit", "receive"], + "system.network.connections": ["family", "type"], + "runtime.memory": ["rss", "vms"], + "runtime.cpu.time": ["user", "system"], +} + + +class SystemMetricsInstrumentor(BaseInstrumentor): + def __init__( + self, + labels: Optional[Dict[str, str]] = None, + config: Optional[Dict[str, List[str]]] = None, + ): + super().__init__() + if config is None: + self._config = _DEFAULT_CONFIG + else: + self._config = config + self._labels = {} if labels is None else labels + self._meter = None + self._python_implementation = python_implementation().lower() + + self._proc = psutil.Process(os.getpid()) + + self._system_cpu_time_labels = self._labels.copy() + self._system_cpu_utilization_labels = self._labels.copy() + + self._system_memory_usage_labels = self._labels.copy() + self._system_memory_utilization_labels = self._labels.copy() + + self._system_swap_usage_labels = self._labels.copy() + self._system_swap_utilization_labels = self._labels.copy() + + self._system_disk_io_labels = self._labels.copy() + self._system_disk_operations_labels = self._labels.copy() + self._system_disk_time_labels = self._labels.copy() + self._system_disk_merged_labels = self._labels.copy() + + self._system_network_dropped_packets_labels = self._labels.copy() + self._system_network_packets_labels = self._labels.copy() + self._system_network_errors_labels = self._labels.copy() + self._system_network_io_labels = self._labels.copy() + self._system_network_connections_labels = self._labels.copy() + + self._runtime_memory_labels = self._labels.copy() + self._runtime_cpu_time_labels = self._labels.copy() + self._runtime_gc_count_labels = self._labels.copy() + + def instrumentation_dependencies(self) -> Collection[str]: + return _instruments + + def _instrument(self, **kwargs): + meter_provider = kwargs.get("meter_provider") + self._meter = get_meter( + __name__, + __version__, + meter_provider, + ) + + self._meter.create_observable_counter( + callback=self._get_system_cpu_time, + name="system.cpu.time", + description="System CPU time", + unit="seconds", + ) + + self._meter.create_observable_gauge( + callback=self._get_system_cpu_utilization, + name="system.cpu.utilization", + description="System CPU utilization", + unit="1", + ) + + self._meter.create_observable_gauge( + callback=self._get_system_memory_usage, + name="system.memory.usage", + description="System memory usage", + unit="bytes", + ) + + self._meter.create_observable_gauge( + callback=self._get_system_memory_utilization, + name="system.memory.utilization", + description="System memory utilization", + unit="1", + ) + + self._meter.create_observable_gauge( + callback=self._get_system_swap_usage, + name="system.swap.usage", + description="System swap usage", + unit="pages", + ) + + self._meter.create_observable_gauge( + callback=self._get_system_swap_utilization, + name="system.swap.utilization", + description="System swap utilization", + unit="1", + ) + + # TODO Add _get_system_swap_page_faults + + # self._meter.create_observable_counter( + # callback=self._get_system_swap_page_faults, + # name="system.swap.page_faults", + # description="System swap page faults", + # unit="faults", + # value_type=int, + # ) + + # TODO Add _get_system_swap_page_operations + # self._meter.create_observable_counter( + # callback=self._get_system_swap_page_operations, + # name="system.swap.page_operations", + # description="System swap page operations", + # unit="operations", + # value_type=int, + # ) + + self._meter.create_observable_counter( + callback=self._get_system_disk_io, + name="system.disk.io", + description="System disk IO", + unit="bytes", + ) + + self._meter.create_observable_counter( + callback=self._get_system_disk_operations, + name="system.disk.operations", + description="System disk operations", + unit="operations", + ) + + self._meter.create_observable_counter( + callback=self._get_system_disk_time, + name="system.disk.time", + description="System disk time", + unit="seconds", + ) + + # TODO Add _get_system_filesystem_usage + + # self.accumulator.register_valueobserver( + # callback=self._get_system_filesystem_usage, + # name="system.filesystem.usage", + # description="System filesystem usage", + # unit="bytes", + # value_type=int, + # ) + + # TODO Add _get_system_filesystem_utilization + # self._meter.create_observable_gauge( + # callback=self._get_system_filesystem_utilization, + # name="system.filesystem.utilization", + # description="System filesystem utilization", + # unit="1", + # value_type=float, + # ) + + # TODO Filesystem information can be obtained with os.statvfs in Unix-like + # OSs, how to do the same in Windows? + + self._meter.create_observable_counter( + callback=self._get_system_network_dropped_packets, + name="system.network.dropped_packets", + description="System network dropped_packets", + unit="packets", + ) + + self._meter.create_observable_counter( + callback=self._get_system_network_packets, + name="system.network.packets", + description="System network packets", + unit="packets", + ) + + self._meter.create_observable_counter( + callback=self._get_system_network_errors, + name="system.network.errors", + description="System network errors", + unit="errors", + ) + + self._meter.create_observable_counter( + callback=self._get_system_network_io, + name="system.network.io", + description="System network io", + unit="bytes", + ) + + self._meter.create_observable_up_down_counter( + callback=self._get_system_network_connections, + name="system.network.connections", + description="System network connections", + unit="connections", + ) + + self._meter.create_observable_counter( + callback=self._get_runtime_memory, + name=f"runtime.{self._python_implementation}.memory", + description=f"Runtime {self._python_implementation} memory", + unit="bytes", + ) + + self._meter.create_observable_counter( + callback=self._get_runtime_cpu_time, + name=f"runtime.{self._python_implementation}.cpu_time", + description=f"Runtime {self._python_implementation} CPU time", + unit="seconds", + ) + + self._meter.create_observable_counter( + callback=self._get_runtime_gc_count, + name=f"runtime.{self._python_implementation}.gc_count", + description=f"Runtime {self._python_implementation} GC count", + unit="bytes", + ) + + def _uninstrument(self, **__): + pass + + def _get_system_cpu_time(self) -> Iterable[Measurement]: + """Observer callback for system CPU time""" + for cpu, times in enumerate(psutil.cpu_times(percpu=True)): + for metric in self._config["system.cpu.time"]: + if hasattr(times, metric): + self._system_cpu_time_labels["state"] = metric + self._system_cpu_time_labels["cpu"] = cpu + 1 + yield Measurement( + getattr(times, metric), self._system_cpu_time_labels + ) + + def _get_system_cpu_utilization(self) -> Iterable[Measurement]: + """Observer callback for system CPU utilization""" + + for cpu, times_percent in enumerate( + psutil.cpu_times_percent(percpu=True) + ): + for metric in self._config["system.cpu.utilization"]: + if hasattr(times_percent, metric): + self._system_cpu_utilization_labels["state"] = metric + self._system_cpu_utilization_labels["cpu"] = cpu + 1 + yield Measurement( + getattr(times_percent, metric) / 100, + self._system_cpu_utilization_labels, + ) + + def _get_system_memory_usage(self) -> Iterable[Measurement]: + """Observer callback for memory usage""" + virtual_memory = psutil.virtual_memory() + for metric in self._config["system.memory.usage"]: + self._system_memory_usage_labels["state"] = metric + if hasattr(virtual_memory, metric): + yield Measurement( + getattr(virtual_memory, metric), + self._system_memory_usage_labels, + ) + + def _get_system_memory_utilization(self) -> Iterable[Measurement]: + """Observer callback for memory utilization""" + system_memory = psutil.virtual_memory() + + for metric in self._config["system.memory.utilization"]: + self._system_memory_utilization_labels["state"] = metric + if hasattr(system_memory, metric): + yield Measurement( + getattr(system_memory, metric) / system_memory.total, + self._system_memory_utilization_labels, + ) + + def _get_system_swap_usage(self) -> Iterable[Measurement]: + """Observer callback for swap usage""" + system_swap = psutil.swap_memory() + + for metric in self._config["system.swap.usage"]: + self._system_swap_usage_labels["state"] = metric + if hasattr(system_swap, metric): + yield Measurement( + getattr(system_swap, metric), + self._system_swap_usage_labels, + ) + + def _get_system_swap_utilization(self) -> Iterable[Measurement]: + """Observer callback for swap utilization""" + system_swap = psutil.swap_memory() + + for metric in self._config["system.swap.utilization"]: + if hasattr(system_swap, metric): + self._system_swap_utilization_labels["state"] = metric + yield Measurement( + getattr(system_swap, metric) / system_swap.total, + self._system_swap_utilization_labels, + ) + + def _get_system_disk_io(self) -> Iterable[Measurement]: + """Observer callback for disk IO""" + for device, counters in psutil.disk_io_counters(perdisk=True).items(): + for metric in self._config["system.disk.io"]: + if hasattr(counters, f"{metric}_bytes"): + self._system_disk_io_labels["device"] = device + self._system_disk_io_labels["direction"] = metric + yield Measurement( + getattr(counters, f"{metric}_bytes"), + self._system_disk_io_labels, + ) + + def _get_system_disk_operations(self) -> Iterable[Measurement]: + """Observer callback for disk operations""" + for device, counters in psutil.disk_io_counters(perdisk=True).items(): + for metric in self._config["system.disk.operations"]: + if hasattr(counters, f"{metric}_count"): + self._system_disk_operations_labels["device"] = device + self._system_disk_operations_labels["direction"] = metric + yield Measurement( + getattr(counters, f"{metric}_count"), + self._system_disk_operations_labels, + ) + + def _get_system_disk_time(self) -> Iterable[Measurement]: + """Observer callback for disk time""" + for device, counters in psutil.disk_io_counters(perdisk=True).items(): + for metric in self._config["system.disk.time"]: + if hasattr(counters, f"{metric}_time"): + self._system_disk_time_labels["device"] = device + self._system_disk_time_labels["direction"] = metric + yield Measurement( + getattr(counters, f"{metric}_time") / 1000, + self._system_disk_time_labels, + ) + + def _get_system_disk_merged(self) -> Iterable[Measurement]: + """Observer callback for disk merged operations""" + + # FIXME The units in the spec is 1, it seems like it should be + # operations or the value type should be Double + + for device, counters in psutil.disk_io_counters(perdisk=True).items(): + for metric in self._config["system.disk.time"]: + if hasattr(counters, f"{metric}_merged_count"): + self._system_disk_merged_labels["device"] = device + self._system_disk_merged_labels["direction"] = metric + yield Measurement( + getattr(counters, f"{metric}_merged_count"), + self._system_disk_merged_labels, + ) + + def _get_system_network_dropped_packets(self) -> Iterable[Measurement]: + """Observer callback for network dropped packets""" + + for device, counters in psutil.net_io_counters(pernic=True).items(): + for metric in self._config["system.network.dropped.packets"]: + in_out = {"receive": "in", "transmit": "out"}[metric] + if hasattr(counters, f"drop{in_out}"): + self._system_network_dropped_packets_labels[ + "device" + ] = device + self._system_network_dropped_packets_labels[ + "direction" + ] = metric + yield Measurement( + getattr(counters, f"drop{in_out}"), + self._system_network_dropped_packets_labels, + ) + + def _get_system_network_packets(self) -> Iterable[Measurement]: + """Observer callback for network packets""" + + for device, counters in psutil.net_io_counters(pernic=True).items(): + for metric in self._config["system.network.dropped.packets"]: + recv_sent = {"receive": "recv", "transmit": "sent"}[metric] + if hasattr(counters, f"packets_{recv_sent}"): + self._system_network_packets_labels["device"] = device + self._system_network_packets_labels["direction"] = metric + yield Measurement( + getattr(counters, f"packets_{recv_sent}"), + self._system_network_packets_labels, + ) + + def _get_system_network_errors(self) -> Iterable[Measurement]: + """Observer callback for network errors""" + for device, counters in psutil.net_io_counters(pernic=True).items(): + for metric in self._config["system.network.errors"]: + in_out = {"receive": "in", "transmit": "out"}[metric] + if hasattr(counters, f"err{in_out}"): + self._system_network_errors_labels["device"] = device + self._system_network_errors_labels["direction"] = metric + yield Measurement( + getattr(counters, f"err{in_out}"), + self._system_network_errors_labels, + ) + + def _get_system_network_io(self) -> Iterable[Measurement]: + """Observer callback for network IO""" + + for device, counters in psutil.net_io_counters(pernic=True).items(): + for metric in self._config["system.network.dropped.packets"]: + recv_sent = {"receive": "recv", "transmit": "sent"}[metric] + if hasattr(counters, f"bytes_{recv_sent}"): + self._system_network_io_labels["device"] = device + self._system_network_io_labels["direction"] = metric + yield Measurement( + getattr(counters, f"bytes_{recv_sent}"), + self._system_network_io_labels, + ) + + def _get_system_network_connections(self) -> Iterable[Measurement]: + """Observer callback for network connections""" + # TODO How to find the device identifier for a particular + # connection? + + connection_counters = {} + + for net_connection in psutil.net_connections(): + for metric in self._config["system.network.connections"]: + self._system_network_connections_labels["protocol"] = { + 1: "tcp", + 2: "udp", + }[net_connection.type.value] + self._system_network_connections_labels[ + "state" + ] = net_connection.status + self._system_network_connections_labels[metric] = getattr( + net_connection, metric + ) + + connection_counters_key = get_dict_as_key( + self._system_network_connections_labels + ) + + if connection_counters_key in connection_counters: + connection_counters[connection_counters_key]["counter"] += 1 + else: + connection_counters[connection_counters_key] = { + "counter": 1, + "labels": self._system_network_connections_labels.copy(), + } + + for connection_counter in connection_counters.values(): + yield Measurement( + connection_counter["counter"], + connection_counter["labels"], + ) + + def _get_runtime_memory(self) -> Iterable[Measurement]: + """Observer callback for runtime memory""" + proc_memory = self._proc.memory_info() + for metric in self._config["runtime.memory"]: + if hasattr(proc_memory, metric): + self._runtime_memory_labels["type"] = metric + yield Measurement( + getattr(proc_memory, metric), + self._runtime_memory_labels, + ) + + def _get_runtime_cpu_time(self) -> Iterable[Measurement]: + """Observer callback for runtime CPU time""" + proc_cpu = self._proc.cpu_times() + for metric in self._config["runtime.cpu.time"]: + if hasattr(proc_cpu, metric): + self._runtime_cpu_time_labels["type"] = metric + yield Measurement( + getattr(proc_cpu, metric), + self._runtime_cpu_time_labels, + ) + + def _get_runtime_gc_count(self) -> Iterable[Measurement]: + """Observer callback for garbage collection""" + for index, count in enumerate(gc.get_count()): + self._runtime_gc_count_labels["count"] = str(index) + yield Measurement(count, self._runtime_gc_count_labels) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/package.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/package.py new file mode 100644 index 0000000000..570690e27f --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/package.py @@ -0,0 +1,16 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +_instruments = ("psutil >= 5",) diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/version.py b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/version.py new file mode 100644 index 0000000000..5743fb8ce1 --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/src/opentelemetry/instrumentation/system_metrics/version.py @@ -0,0 +1,15 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +__version__ = "0.29b0" diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/__init__.py b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py new file mode 100644 index 0000000000..aeee669f9b --- /dev/null +++ b/instrumentation/opentelemetry-instrumentation-system-metrics/tests/test_system_metrics.py @@ -0,0 +1,715 @@ +# Copyright The OpenTelemetry Authors +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# pylint: disable=protected-access + +from collections import namedtuple +from platform import python_implementation +from unittest import mock + +from opentelemetry.instrumentation.system_metrics import ( + SystemMetricsInstrumentor, +) +from opentelemetry.sdk._metrics import MeterProvider +from opentelemetry.sdk._metrics.export import InMemoryMetricReader +from opentelemetry.test.test_base import TestBase + + +def _mock_netconnection(): + NetConnection = namedtuple( + "NetworkConnection", ["family", "type", "status"] + ) + Type = namedtuple("Type", ["value"]) + return [ + NetConnection( + family=1, + status="ESTABLISHED", + type=Type(value=2), + ), + NetConnection( + family=1, + status="ESTABLISHED", + type=Type(value=1), + ), + ] + + +class _SystemMetricsResult: + def __init__(self, attributes, value) -> None: + self.attributes = attributes + self.value = value + + +class TestSystemMetrics(TestBase): + def setUp(self): + super().setUp() + self.implementation = python_implementation().lower() + self._patch_net_connections = mock.patch( + "psutil.net_connections", _mock_netconnection + ) + self._patch_net_connections.start() + + def tearDown(self): + super().tearDown() + self._patch_net_connections.stop() + SystemMetricsInstrumentor().uninstrument() + + def test_system_metrics_instrument(self): + reader = InMemoryMetricReader() + meter_provider = MeterProvider(metric_readers=[reader]) + system_metrics = SystemMetricsInstrumentor() + system_metrics.instrument(meter_provider=meter_provider) + metrics = reader.get_metrics() + metric_names = list({x.name for x in metrics}) + self.assertEqual(len(metric_names), 17) + + observer_names = [ + "system.cpu.time", + "system.cpu.utilization", + "system.memory.usage", + "system.memory.utilization", + "system.swap.usage", + "system.swap.utilization", + "system.disk.io", + "system.disk.operations", + "system.disk.time", + "system.network.dropped_packets", + "system.network.packets", + "system.network.errors", + "system.network.io", + "system.network.connections", + f"runtime.{self.implementation}.memory", + f"runtime.{self.implementation}.cpu_time", + f"runtime.{self.implementation}.gc_count", + ] + + for observer in metric_names: + self.assertIn(observer, observer_names) + observer_names.remove(observer) + + def _assert_metrics(self, observer_name, reader, expected): + assertions = 0 + for metric in reader.get_metrics(): # pylint: disable=protected-access + for expect in expected: + if ( + metric.attributes == expect.attributes + and metric.name == observer_name + ): + self.assertEqual( + metric.point.value, + expect.value, + ) + assertions += 1 + self.assertEqual(len(expected), assertions) + + def _test_metrics(self, observer_name, expected): + reader = InMemoryMetricReader() + meter_provider = MeterProvider(metric_readers=[reader]) + + system_metrics = SystemMetricsInstrumentor() + system_metrics.instrument(meter_provider=meter_provider) + self._assert_metrics(observer_name, reader, expected) + + # This patch is added here to stop psutil from raising an exception + # because we're patching cpu_times + # pylint: disable=unused-argument + @mock.patch("psutil.cpu_times_percent") + @mock.patch("psutil.cpu_times") + def test_system_cpu_time(self, mock_cpu_times, mock_cpu_times_percent): + CPUTimes = namedtuple("CPUTimes", ["idle", "user", "system", "irq"]) + mock_cpu_times.return_value = [ + CPUTimes(idle=1.2, user=3.4, system=5.6, irq=7.8), + CPUTimes(idle=1.2, user=3.4, system=5.6, irq=7.8), + ] + + expected = [ + _SystemMetricsResult( + { + "cpu": 1, + "state": "idle", + }, + 1.2, + ), + _SystemMetricsResult( + { + "cpu": 1, + "state": "user", + }, + 3.4, + ), + _SystemMetricsResult( + { + "cpu": 1, + "state": "system", + }, + 5.6, + ), + _SystemMetricsResult( + { + "cpu": 1, + "state": "irq", + }, + 7.8, + ), + _SystemMetricsResult( + { + "cpu": 2, + "state": "idle", + }, + 1.2, + ), + _SystemMetricsResult( + { + "cpu": 2, + "state": "user", + }, + 3.4, + ), + _SystemMetricsResult( + { + "cpu": 2, + "state": "system", + }, + 5.6, + ), + _SystemMetricsResult( + { + "cpu": 2, + "state": "irq", + }, + 7.8, + ), + ] + self._test_metrics("system.cpu.time", expected) + + @mock.patch("psutil.cpu_times_percent") + def test_system_cpu_utilization(self, mock_cpu_times_percent): + CPUTimesPercent = namedtuple( + "CPUTimesPercent", ["idle", "user", "system", "irq"] + ) + mock_cpu_times_percent.return_value = [ + CPUTimesPercent(idle=1.2, user=3.4, system=5.6, irq=7.8), + CPUTimesPercent(idle=1.2, user=3.4, system=5.6, irq=7.8), + ] + + expected = [ + _SystemMetricsResult({"cpu": 1, "state": "idle"}, 1.2 / 100), + _SystemMetricsResult({"cpu": 1, "state": "user"}, 3.4 / 100), + _SystemMetricsResult({"cpu": 1, "state": "system"}, 5.6 / 100), + _SystemMetricsResult({"cpu": 1, "state": "irq"}, 7.8 / 100), + _SystemMetricsResult({"cpu": 2, "state": "idle"}, 1.2 / 100), + _SystemMetricsResult({"cpu": 2, "state": "user"}, 3.4 / 100), + _SystemMetricsResult({"cpu": 2, "state": "system"}, 5.6 / 100), + _SystemMetricsResult({"cpu": 2, "state": "irq"}, 7.8 / 100), + ] + self._test_metrics("system.cpu.utilization", expected) + + @mock.patch("psutil.virtual_memory") + def test_system_memory_usage(self, mock_virtual_memory): + VirtualMemory = namedtuple( + "VirtualMemory", ["used", "free", "cached", "total"] + ) + mock_virtual_memory.return_value = VirtualMemory( + used=1, free=2, cached=3, total=4 + ) + + expected = [ + _SystemMetricsResult({"state": "used"}, 1), + _SystemMetricsResult({"state": "free"}, 2), + _SystemMetricsResult({"state": "cached"}, 3), + ] + self._test_metrics("system.memory.usage", expected) + + @mock.patch("psutil.virtual_memory") + def test_system_memory_utilization(self, mock_virtual_memory): + VirtualMemory = namedtuple( + "VirtualMemory", ["used", "free", "cached", "total"] + ) + mock_virtual_memory.return_value = VirtualMemory( + used=1, free=2, cached=3, total=4 + ) + + expected = [ + _SystemMetricsResult({"state": "used"}, 1 / 4), + _SystemMetricsResult({"state": "free"}, 2 / 4), + _SystemMetricsResult({"state": "cached"}, 3 / 4), + ] + self._test_metrics("system.memory.utilization", expected) + + @mock.patch("psutil.swap_memory") + def test_system_swap_usage(self, mock_swap_memory): + SwapMemory = namedtuple("SwapMemory", ["used", "free", "total"]) + mock_swap_memory.return_value = SwapMemory(used=1, free=2, total=3) + + expected = [ + _SystemMetricsResult({"state": "used"}, 1), + _SystemMetricsResult({"state": "free"}, 2), + ] + self._test_metrics("system.swap.usage", expected) + + @mock.patch("psutil.swap_memory") + def test_system_swap_utilization(self, mock_swap_memory): + SwapMemory = namedtuple("SwapMemory", ["used", "free", "total"]) + mock_swap_memory.return_value = SwapMemory(used=1, free=2, total=3) + + expected = [ + _SystemMetricsResult({"state": "used"}, 1 / 3), + _SystemMetricsResult({"state": "free"}, 2 / 3), + ] + self._test_metrics("system.swap.utilization", expected) + + @mock.patch("psutil.disk_io_counters") + def test_system_disk_io(self, mock_disk_io_counters): + DiskIO = namedtuple( + "DiskIO", + [ + "read_count", + "write_count", + "read_bytes", + "write_bytes", + "read_time", + "write_time", + "read_merged_count", + "write_merged_count", + ], + ) + mock_disk_io_counters.return_value = { + "sda": DiskIO( + read_count=1, + write_count=2, + read_bytes=3, + write_bytes=4, + read_time=5, + write_time=6, + read_merged_count=7, + write_merged_count=8, + ), + "sdb": DiskIO( + read_count=9, + write_count=10, + read_bytes=11, + write_bytes=12, + read_time=13, + write_time=14, + read_merged_count=15, + write_merged_count=16, + ), + } + + expected = [ + _SystemMetricsResult({"device": "sda", "direction": "read"}, 3), + _SystemMetricsResult({"device": "sda", "direction": "write"}, 4), + _SystemMetricsResult({"device": "sdb", "direction": "read"}, 11), + _SystemMetricsResult({"device": "sdb", "direction": "write"}, 12), + ] + self._test_metrics("system.disk.io", expected) + + @mock.patch("psutil.disk_io_counters") + def test_system_disk_operations(self, mock_disk_io_counters): + DiskIO = namedtuple( + "DiskIO", + [ + "read_count", + "write_count", + "read_bytes", + "write_bytes", + "read_time", + "write_time", + "read_merged_count", + "write_merged_count", + ], + ) + mock_disk_io_counters.return_value = { + "sda": DiskIO( + read_count=1, + write_count=2, + read_bytes=3, + write_bytes=4, + read_time=5, + write_time=6, + read_merged_count=7, + write_merged_count=8, + ), + "sdb": DiskIO( + read_count=9, + write_count=10, + read_bytes=11, + write_bytes=12, + read_time=13, + write_time=14, + read_merged_count=15, + write_merged_count=16, + ), + } + + expected = [ + _SystemMetricsResult({"device": "sda", "direction": "read"}, 1), + _SystemMetricsResult({"device": "sda", "direction": "write"}, 2), + _SystemMetricsResult({"device": "sdb", "direction": "read"}, 9), + _SystemMetricsResult({"device": "sdb", "direction": "write"}, 10), + ] + self._test_metrics("system.disk.operations", expected) + + @mock.patch("psutil.disk_io_counters") + def test_system_disk_time(self, mock_disk_io_counters): + DiskIO = namedtuple( + "DiskIO", + [ + "read_count", + "write_count", + "read_bytes", + "write_bytes", + "read_time", + "write_time", + "read_merged_count", + "write_merged_count", + ], + ) + mock_disk_io_counters.return_value = { + "sda": DiskIO( + read_count=1, + write_count=2, + read_bytes=3, + write_bytes=4, + read_time=5, + write_time=6, + read_merged_count=7, + write_merged_count=8, + ), + "sdb": DiskIO( + read_count=9, + write_count=10, + read_bytes=11, + write_bytes=12, + read_time=13, + write_time=14, + read_merged_count=15, + write_merged_count=16, + ), + } + + expected = [ + _SystemMetricsResult( + {"device": "sda", "direction": "read"}, 5 / 1000 + ), + _SystemMetricsResult( + {"device": "sda", "direction": "write"}, 6 / 1000 + ), + _SystemMetricsResult( + {"device": "sdb", "direction": "read"}, 13 / 1000 + ), + _SystemMetricsResult( + {"device": "sdb", "direction": "write"}, 14 / 1000 + ), + ] + self._test_metrics("system.disk.time", expected) + + @mock.patch("psutil.net_io_counters") + def test_system_network_dropped_packets(self, mock_net_io_counters): + NetIO = namedtuple( + "NetIO", + [ + "dropin", + "dropout", + "packets_sent", + "packets_recv", + "errin", + "errout", + "bytes_sent", + "bytes_recv", + ], + ) + mock_net_io_counters.return_value = { + "eth0": NetIO( + dropin=1, + dropout=2, + packets_sent=3, + packets_recv=4, + errin=5, + errout=6, + bytes_sent=7, + bytes_recv=8, + ), + "eth1": NetIO( + dropin=9, + dropout=10, + packets_sent=11, + packets_recv=12, + errin=13, + errout=14, + bytes_sent=15, + bytes_recv=16, + ), + } + + expected = [ + _SystemMetricsResult( + {"device": "eth0", "direction": "receive"}, 1 + ), + _SystemMetricsResult( + {"device": "eth0", "direction": "transmit"}, 2 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "receive"}, 9 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "transmit"}, 10 + ), + ] + self._test_metrics("system.network.dropped_packets", expected) + + @mock.patch("psutil.net_io_counters") + def test_system_network_packets(self, mock_net_io_counters): + NetIO = namedtuple( + "NetIO", + [ + "dropin", + "dropout", + "packets_sent", + "packets_recv", + "errin", + "errout", + "bytes_sent", + "bytes_recv", + ], + ) + mock_net_io_counters.return_value = { + "eth0": NetIO( + dropin=1, + dropout=2, + packets_sent=3, + packets_recv=4, + errin=5, + errout=6, + bytes_sent=7, + bytes_recv=8, + ), + "eth1": NetIO( + dropin=9, + dropout=10, + packets_sent=11, + packets_recv=12, + errin=13, + errout=14, + bytes_sent=15, + bytes_recv=16, + ), + } + + expected = [ + _SystemMetricsResult( + {"device": "eth0", "direction": "receive"}, 4 + ), + _SystemMetricsResult( + {"device": "eth0", "direction": "transmit"}, 3 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "receive"}, 12 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "transmit"}, 11 + ), + ] + self._test_metrics("system.network.packets", expected) + + @mock.patch("psutil.net_io_counters") + def test_system_network_errors(self, mock_net_io_counters): + NetIO = namedtuple( + "NetIO", + [ + "dropin", + "dropout", + "packets_sent", + "packets_recv", + "errin", + "errout", + "bytes_sent", + "bytes_recv", + ], + ) + mock_net_io_counters.return_value = { + "eth0": NetIO( + dropin=1, + dropout=2, + packets_sent=3, + packets_recv=4, + errin=5, + errout=6, + bytes_sent=7, + bytes_recv=8, + ), + "eth1": NetIO( + dropin=9, + dropout=10, + packets_sent=11, + packets_recv=12, + errin=13, + errout=14, + bytes_sent=15, + bytes_recv=16, + ), + } + + expected = [ + _SystemMetricsResult( + {"device": "eth0", "direction": "receive"}, 5 + ), + _SystemMetricsResult( + {"device": "eth0", "direction": "transmit"}, 6 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "receive"}, 13 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "transmit"}, 14 + ), + ] + self._test_metrics("system.network.errors", expected) + + @mock.patch("psutil.net_io_counters") + def test_system_network_io(self, mock_net_io_counters): + NetIO = namedtuple( + "NetIO", + [ + "dropin", + "dropout", + "packets_sent", + "packets_recv", + "errin", + "errout", + "bytes_sent", + "bytes_recv", + ], + ) + mock_net_io_counters.return_value = { + "eth0": NetIO( + dropin=1, + dropout=2, + packets_sent=3, + packets_recv=4, + errin=5, + errout=6, + bytes_sent=7, + bytes_recv=8, + ), + "eth1": NetIO( + dropin=9, + dropout=10, + packets_sent=11, + packets_recv=12, + errin=13, + errout=14, + bytes_sent=15, + bytes_recv=16, + ), + } + + expected = [ + _SystemMetricsResult( + {"device": "eth0", "direction": "receive"}, 8 + ), + _SystemMetricsResult( + {"device": "eth0", "direction": "transmit"}, 7 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "receive"}, 16 + ), + _SystemMetricsResult( + {"device": "eth1", "direction": "transmit"}, 15 + ), + ] + self._test_metrics("system.network.io", expected) + + @mock.patch("psutil.net_connections") + def test_system_network_connections(self, mock_net_connections): + NetConnection = namedtuple( + "NetworkConnection", ["family", "type", "status"] + ) + Type = namedtuple("Type", ["value"]) + mock_net_connections.return_value = [ + NetConnection( + family=1, + status="ESTABLISHED", + type=Type(value=2), + ), + NetConnection( + family=1, + status="ESTABLISHED", + type=Type(value=1), + ), + ] + + expected = [ + _SystemMetricsResult( + { + "family": 1, + "protocol": "udp", + "state": "ESTABLISHED", + "type": Type(value=2), + }, + 1, + ), + _SystemMetricsResult( + { + "family": 1, + "protocol": "tcp", + "state": "ESTABLISHED", + "type": Type(value=1), + }, + 1, + ), + ] + self._test_metrics("system.network.connections", expected) + + @mock.patch("psutil.Process.memory_info") + def test_runtime_memory(self, mock_process_memory_info): + + PMem = namedtuple("PMem", ["rss", "vms"]) + + mock_process_memory_info.configure_mock( + **{"return_value": PMem(rss=1, vms=2)} + ) + + expected = [ + _SystemMetricsResult({"type": "rss"}, 1), + _SystemMetricsResult({"type": "vms"}, 2), + ] + self._test_metrics(f"runtime.{self.implementation}.memory", expected) + + @mock.patch("psutil.Process.cpu_times") + def test_runtime_cpu_time(self, mock_process_cpu_times): + + PCPUTimes = namedtuple("PCPUTimes", ["user", "system"]) + + mock_process_cpu_times.configure_mock( + **{"return_value": PCPUTimes(user=1.1, system=2.2)} + ) + + expected = [ + _SystemMetricsResult({"type": "user"}, 1.1), + _SystemMetricsResult({"type": "system"}, 2.2), + ] + self._test_metrics(f"runtime.{self.implementation}.cpu_time", expected) + + @mock.patch("gc.get_count") + def test_runtime_get_count(self, mock_gc_get_count): + + mock_gc_get_count.configure_mock(**{"return_value": (1, 2, 3)}) + + expected = [ + _SystemMetricsResult({"count": "0"}, 1), + _SystemMetricsResult({"count": "1"}, 2), + _SystemMetricsResult({"count": "2"}, 3), + ] + self._test_metrics(f"runtime.{self.implementation}.gc_count", expected) diff --git a/opentelemetry-contrib-instrumentations/setup.cfg b/opentelemetry-contrib-instrumentations/setup.cfg index 4fb943ec9a..730f1ca55e 100644 --- a/opentelemetry-contrib-instrumentations/setup.cfg +++ b/opentelemetry-contrib-instrumentations/setup.cfg @@ -60,6 +60,7 @@ install_requires = opentelemetry-instrumentation-sqlalchemy==0.29b0 opentelemetry-instrumentation-sqlite3==0.29b0 opentelemetry-instrumentation-starlette==0.29b0 + opentelemetry-instrumentation-system-metrics==0.29b0 opentelemetry-instrumentation-tornado==0.29b0 opentelemetry-instrumentation-urllib==0.29b0 opentelemetry-instrumentation-urllib3==0.29b0 diff --git a/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py b/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py index dd0955a2cb..66d4980ad6 100644 --- a/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py +++ b/opentelemetry-instrumentation/src/opentelemetry/instrumentation/bootstrap_gen.py @@ -128,6 +128,10 @@ "library": "starlette ~= 0.13.0", "instrumentation": "opentelemetry-instrumentation-starlette==0.29b0", }, + "psutil": { + "library": "psutil >= 5", + "instrumentation": "opentelemetry-instrumentation-system-metrics==0.29b0", + }, "tornado": { "library": "tornado >= 5.1.1", "instrumentation": "opentelemetry-instrumentation-tornado==0.29b0", diff --git a/tox.ini b/tox.ini index 79f63d3f2e..1fd07e51d1 100644 --- a/tox.ini +++ b/tox.ini @@ -161,6 +161,10 @@ envlist = ; opentelemetry-instrumentation-sklearn py3{6,7,8}-test-instrumentation-sklearn + ; opentelemetry-instrumentation-system-metrics + py3{6,7,8,9,10}-test-instrumentation-system-metrics + ; instrumentation-system-metrics intentionally excluded from pypy3 + ; opentelemetry-instrumentation-tornado py3{6,7,8,9,10}-test-instrumentation-tornado pypy3-test-instrumentation-tornado @@ -276,6 +280,7 @@ changedir = test-instrumentation-sqlalchemy{11,14}: instrumentation/opentelemetry-instrumentation-sqlalchemy/tests test-instrumentation-sqlite3: instrumentation/opentelemetry-instrumentation-sqlite3/tests test-instrumentation-starlette: instrumentation/opentelemetry-instrumentation-starlette/tests + test-instrumentation-system-metrics: instrumentation/opentelemetry-instrumentation-system-metrics/tests test-instrumentation-tornado: instrumentation/opentelemetry-instrumentation-tornado/tests test-instrumentation-wsgi: instrumentation/opentelemetry-instrumentation-wsgi/tests test-instrumentation-httpx{18,21}: instrumentation/opentelemetry-instrumentation-httpx/tests @@ -354,6 +359,8 @@ commands_pre = starlette: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-starlette[test] + system-metrics: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-system-metrics[test] + tornado: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-tornado[test] jinja2: pip install {toxinidir}/instrumentation/opentelemetry-instrumentation-jinja2[test] @@ -465,6 +472,7 @@ commands_pre = python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-mysql[test] python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-httpx[test] python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-aws-lambda[test] + python -m pip install -e {toxinidir}/instrumentation/opentelemetry-instrumentation-system-metrics[test] python -m pip install -e {toxinidir}/exporter/opentelemetry-exporter-datadog[test] python -m pip install -e {toxinidir}/exporter/opentelemetry-exporter-richconsole[test] python -m pip install -e {toxinidir}/sdk-extension/opentelemetry-sdk-extension-aws[test]