Adding platform telemetry

README.md

@@ -30,9 +30,12 @@
 * [Technical Design Details](#technical-design-details)
 * [Extensibility Options](#extensibility-options)
 
+**[Data/Telemetry](#Data/Telemetry)**
+
 **[Contributions and Feedback](#contribute)**
 
 **[License](#license)**
+
 ***
 # Key Features
 ## Run any ONNX model
@@ -160,6 +163,10 @@ To tune performance for ONNX models, the [ONNX Go Live tool "OLive"](https://git
 transform](include/onnxruntime/core/optimizer/graph_transformer.h)
 * [Add a new rewrite rule](include/onnxruntime/core/optimizer/rewrite_rule.h)
 
+***
+# Data/Telemetry
+This project may collect usage data and send it to Microsoft to help improve our products and services. See the [privacy statement](docs/Privacy.md) for more details.
+
 ***
 # Contribute
 We welcome contributions! Please see the [contribution guidelines](CONTRIBUTING.md).
@@ -171,6 +178,7 @@ For any feedback or to report a bug, please file a [GitHub Issue](https://github
 This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
 For more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
 or contact [[email protected]](mailto:[email protected]) with any additional questions or comments.
+
 ***
 # License
 [MIT License](LICENSE)

cmake/onnxruntime_common.cmake

@@ -16,6 +16,8 @@ set(onnxruntime_common_src_patterns
     "${ONNXRUNTIME_ROOT}/core/platform/env.cc"
     "${ONNXRUNTIME_ROOT}/core/platform/env_time.h"
     "${ONNXRUNTIME_ROOT}/core/platform/env_time.cc"
+    "${ONNXRUNTIME_ROOT}/core/platform/telemetry.h"
+    "${ONNXRUNTIME_ROOT}/core/platform/telemetry.cc"
 )
 
 if(WIN32)

csharp/src/Microsoft.ML.OnnxRuntime/NativeMethods.cs

@@ -22,6 +22,8 @@ public struct OrtApi
         public IntPtr GetErrorMessage;
         public IntPtr CreateEnv;
         public IntPtr CreateEnvWithCustomLogger;
+        public IntPtr EnableTelemetryEvents;
+        public IntPtr DisableTelemetryEvents;
         public IntPtr CreateSession;
         public IntPtr CreateSessionFromArray;
         public IntPtr Run;

docs/C_API.md

@@ -29,3 +29,6 @@
 The example below shows a sample run using the SqueezeNet model from ONNX model zoo, including dynamically reading model inputs, outputs, shape and type information, as well as running a sample vector and fetching the resulting class probabilities for inspection.
 
 * [../csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/C_Api_Sample.cpp](../csharp/test/Microsoft.ML.OnnxRuntime.EndToEndTests.Capi/C_Api_Sample.cpp)
+
+## Telemetry
+To turn on/off telemetry collection on official Windows builds, please use Enable/DisableTelemetryEvents() in the C API. See the [Privacy](./Privacy.md) page for more information on telemetry collection and Microsoft's privacy policy.

docs/Privacy.md

@@ -0,0 +1,18 @@
+# Privacy
+
+## Data Collection
+The software may collect information about you and your use of the software and send it to Microsoft. Microsoft may use this information to provide services and improve our products and services. You may turn off the telemetry as described in the repository. There are also some features in the software that may enable you and Microsoft to collect data from users of your applications. If you use these features, you must comply with applicable law, including providing appropriate notices to users of your applications together with a copy of Microsoft's privacy statement. Our privacy statement is located at https://go.microsoft.com/fwlink/?LinkID=824704. You can learn more about data collection and use in the help documentation and our privacy statement. Your use of the software operates as your consent to these practices.
+
+***
+
+### Private Builds
+No data collection is performed when using your private builds.
+
+### Official Builds
+Currently telemetry is only implemented for Windows builds, but may be expanded in the future to cover other platforms. Telemetry is turned OFF by default while this feature is in BETA. When the feature moves from BETA to RELEASE, developers should expect telemetry to be ON by default when using the Official Builds. This is implemented via 'Platform Telemetry' per vendor platform providers (see telemetry.h).
+
+#### Technical Details
+The Windows provider uses the [TraceLogging](https://docs.microsoft.com/en-us/windows/win32/tracelogging/trace-logging-about) API for its implementation.
+
+For API usage details to turn this on/off, please check the API pages:
+* [C API](./C_API.md#telemetry)

include/onnxruntime/core/common/common.h

@@ -74,6 +74,9 @@ using common::Status;
   static_cast<void>(fn)
 
 std::vector<std::string> GetStackTrace();
+// these is a helper function that gets defined by platform/Telemetry
+void LogRuntimeError(uint32_t session_id, const common::Status& status, const char* file,
+                     const char* function, uint32_t line);
 
 // __PRETTY_FUNCTION__ isn't a macro on gcc, so use a check for _MSC_VER
 // so we only define it as one for MSVC
@@ -137,16 +140,25 @@ std::vector<std::string> GetStackTrace();
   ORT_DISALLOW_COPY_AND_ASSIGNMENT(TypeName);           \
   ORT_DISALLOW_MOVE(TypeName)
 
-#define ORT_RETURN_IF_ERROR(expr)          \
-  do {                                     \
-    auto _status = (expr);                 \
-    if ((!_status.IsOK())) return _status; \
+#define ORT_RETURN_IF_ERROR_SESSIONID(expr, session_id)  \
+  do {                                       \
+    auto _status = (expr);                   \
+    if ((!_status.IsOK())) {                 \
+      ::onnxruntime::LogRuntimeError(session_id, _status, __FILE__, __FUNCTION__, __LINE__); \
+      return _status;                        \
+    }                                        \
   } while (0)
 
+#define ORT_RETURN_IF_ERROR_SESSIONID_(expr) ORT_RETURN_IF_ERROR_SESSIONID(expr, session_id_)
+#define ORT_RETURN_IF_ERROR(expr) ORT_RETURN_IF_ERROR_SESSIONID(expr, 0)
+
 #define ORT_THROW_IF_ERROR(expr)               \
   do {                                         \
     auto _status = (expr);                     \
-    if ((!_status.IsOK())) ORT_THROW(_status); \
+    if ((!_status.IsOK())) {                   \
+      ::onnxruntime::LogRuntimeError(0, _status, __FILE__, __FUNCTION__, __LINE__); \
+      ORT_THROW(_status);                      \
+    }                                          \
   } while (0)
 
 // use this macro when cannot early return

include/onnxruntime/core/common/version.h

@@ -0,0 +1,6 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License.
+
+#pragma once
+
+#define ONNXRUNTIME_VERSION_STRING "1.0"

include/onnxruntime/core/session/onnxruntime_c_api.h

@@ -249,6 +249,10 @@ struct OrtApi {
                                                       _In_ const char* logid,
                                                       _Outptr_ OrtEnv** out)NO_EXCEPTION;
 
+  // Platform telemetry events are on by default since they are lightweight.  You can manually turn them off.
+  OrtStatus*(ORT_API_CALL* EnableTelemetryEvents)(_In_ const OrtEnv* env)NO_EXCEPTION;
+  OrtStatus*(ORT_API_CALL* DisableTelemetryEvents)(_In_ const OrtEnv* env)NO_EXCEPTION;
+
   // TODO: document the path separator convention? '/' vs '\'
   // TODO: should specify the access characteristics of model_path. Is this read only during the
   // execution of OrtCreateSession, or does the OrtSession retain a handle to the file/directory

include/onnxruntime/core/session/onnxruntime_cxx_api.h

@@ -107,6 +107,9 @@ struct Env : Base<OrtEnv> {
   Env(OrtLoggingLevel default_logging_level, const char* logid, OrtLoggingFunction logging_function, void* logger_param);
   explicit Env(OrtEnv* p) : Base<OrtEnv>{p} {}
 
+  Env& EnableTelemetryEvents();
+  Env& DisableTelemetryEvents();
+
   static const OrtApi* s_api;
 };
 

include/onnxruntime/core/session/onnxruntime_cxx_inline.h

@@ -93,6 +93,16 @@ inline Env::Env(OrtLoggingLevel default_warning_level, const char* logid, OrtLog
   ThrowOnError(g_api->CreateEnvWithCustomLogger(logging_function, logger_param, default_warning_level, logid, &p_));
 }
 
+inline Env& Env::EnableTelemetryEvents() {
+  ThrowOnError(g_api->EnableTelemetryEvents(p_));
+  return *this;
+}
+
+inline Env& Env::DisableTelemetryEvents() {
+  ThrowOnError(g_api->DisableTelemetryEvents(p_));
+  return *this;
+}
+
 inline CustomOpDomain::CustomOpDomain(const char* domain) {
   ThrowOnError(g_api->CreateCustomOpDomain(domain, &p_));
 }

onnxruntime/core/platform/env.h

@@ -26,6 +26,7 @@ limitations under the License.
 #include "core/common/common.h"
 #include "core/framework/callback.h"
 #include "core/platform/env_time.h"
+#include "core/platform/telemetry.h"
 
 #ifndef _WIN32
 #include <sys/types.h>
@@ -133,6 +134,9 @@ class Env {
   // returns the name that LoadDynamicLibrary() can use
   virtual std::string FormatLibraryFileName(const std::string& name, const std::string& version) const = 0;
 
+  // \brief returns a provider that will handle telemetry on the current platform
+  virtual const Telemetry& GetTelemetryProvider() const = 0;
+
  protected:
   Env();
 

onnxruntime/core/platform/posix/env.cc

@@ -266,8 +266,14 @@ class PosixEnv : public Env {
     return filename;
   }
 
+  // \brief returns a provider that will handle telemetry on the current platform
+  const Telemetry& GetTelemetryProvider() const override {
+    return telemetry_provider_;
+  }
+
  private:
   PosixEnv() = default;
+  Telemetry telemetry_provider_;
 };
 
 }  // namespace

onnxruntime/core/platform/telemetry.cc

@@ -0,0 +1,59 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License.
+
+#include "core/platform/telemetry.h"
+#include "core/platform/env.h"
+
+namespace onnxruntime {
+
+void LogRuntimeError(uint32_t sessionId, const common::Status& status, const char* file,
+                     const char* function, uint32_t line)
+{
+  const Env& env = Env::Default();
+  env.GetTelemetryProvider().LogRuntimeError(sessionId, status, file, function, line);
+}
+
+void Telemetry::EnableTelemetryEvents() const {
+}
+
+void Telemetry::DisableTelemetryEvents() const {
+}
+
+void Telemetry::LogProcessInfo() const {
+}
+
+void Telemetry::LogSessionCreation(uint32_t session_id, int64_t ir_version, const std::string& model_producer_name,
+                                   const std::string& model_producer_version, const std::string& model_domain,
+                                   const std::unordered_map<std::string, int>& domain_to_version_map,
+                                   const std::string& model_graph_name,
+                                   const std::unordered_map<std::string, std::string>& model_metadata,
+                                   const std::string& loadedFrom, const std::vector<std::string>& execution_provider_ids) const {
+  ORT_UNUSED_PARAMETER(session_id);
+  ORT_UNUSED_PARAMETER(ir_version);
+  ORT_UNUSED_PARAMETER(model_producer_name);
+  ORT_UNUSED_PARAMETER(model_producer_version);
+  ORT_UNUSED_PARAMETER(model_domain);
+  ORT_UNUSED_PARAMETER(domain_to_version_map);
+  ORT_UNUSED_PARAMETER(model_graph_name);
+  ORT_UNUSED_PARAMETER(model_metadata);
+  ORT_UNUSED_PARAMETER(loadedFrom);
+  ORT_UNUSED_PARAMETER(execution_provider_ids);
+}
+
+void Telemetry::LogRuntimeError(uint32_t session_id, const common::Status& status, const char* file,
+                                       const char* function, uint32_t line) const {
+  ORT_UNUSED_PARAMETER(session_id);
+  ORT_UNUSED_PARAMETER(status);
+  ORT_UNUSED_PARAMETER(file);
+  ORT_UNUSED_PARAMETER(function);
+  ORT_UNUSED_PARAMETER(line);
+}
+
+void Telemetry::LogRuntimePerf(uint32_t session_id, uint32_t total_runs_since_last, int64_t total_run_duration_since_last) const {
+  ORT_UNUSED_PARAMETER(session_id);
+  ORT_UNUSED_PARAMETER(total_runs_since_last);
+  ORT_UNUSED_PARAMETER(total_run_duration_since_last);
+}
+
+}  // namespace onnxruntime
+

onnxruntime/core/platform/telemetry.h

@@ -0,0 +1,55 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License.
+
+#pragma once
+
+#include <string>
+#include <vector>
+#include <unordered_map>
+
+#include "core/common/status.h"
+#include "core/common/common.h"
+
+namespace onnxruntime {
+
+/**
+  * Configuration information for a session.
+  * An interface used by the onnxruntime implementation to
+  * access operating system functionality for telemetry
+  * 
+  * look at env.h and the Env objection which is the activation factory
+  * for telemetry instances
+  * 
+  * All Telemetry implementations are safe for concurrent access from
+  * multiple threads without any external synchronization.
+  */
+class Telemetry {
+ public:
+  // don't create these, use Env::GetTelemetryProvider() instead
+  // this constructor is made public so that other platform Env providers can
+  // use this base class as a "stub" implementation
+  Telemetry() = default;
+  virtual ~Telemetry() = default;
+
+  virtual void EnableTelemetryEvents() const;
+  virtual void DisableTelemetryEvents() const;
+
+  virtual void LogProcessInfo() const;
+
+  virtual void LogSessionCreation(uint32_t session_id, int64_t ir_version, const std::string& model_producer_name,
+                                  const std::string& model_producer_version, const std::string& model_domain,
+                                  const std::unordered_map<std::string, int>& domain_to_version_map,
+                                  const std::string& model_graph_name,
+                                  const std::unordered_map<std::string, std::string>& model_metadata,
+                                  const std::string& loadedFrom, const std::vector<std::string>& execution_provider_ids) const;
+
+  virtual void LogRuntimeError(uint32_t session_id, const common::Status& status, const char* file,
+                               const char* function, uint32_t line) const;
+
+  virtual void LogRuntimePerf(uint32_t session_id, uint32_t total_runs_since_last, int64_t total_run_duration_since_last) const;
+
+ private:
+  ORT_DISALLOW_COPY_ASSIGNMENT_AND_MOVE(Telemetry);
+};
+
+}  // namespace onnxruntime

onnxruntime/core/platform/windows/env.cc

@@ -25,6 +25,7 @@ limitations under the License.
 
 #include "core/common/logging/logging.h"
 #include "core/platform/env.h"
+#include "core/platform/windows/telemetry.h"
 
 namespace onnxruntime {
 
@@ -211,6 +212,11 @@ class WindowsEnv : public Env {
     ORT_NOT_IMPLEMENTED(__FUNCTION__, " is not implemented");
   }
 
+  // \brief returns a provider that will handle telemetry on the current platform
+  const Telemetry& GetTelemetryProvider() const override {
+    return telemetry_provider_;
+  }
+
  private:
   WindowsEnv()
       : GetSystemTimePreciseAsFileTime_(nullptr) {
@@ -228,8 +234,8 @@ class WindowsEnv : public Env {
 
   typedef VOID(WINAPI* FnGetSystemTimePreciseAsFileTime)(LPFILETIME);
   FnGetSystemTimePreciseAsFileTime GetSystemTimePreciseAsFileTime_;
+  WindowsTelemetry telemetry_provider_;
 };
-
 }  // namespace
 
 #if defined(PLATFORM_WINDOWS)