docs: Mark OEP-26 accepted, update

oeps/architectural-decisions/oep-0026-arch-realtime-events.rst

@@ -8,31 +8,45 @@ OEP-26: Real-time Events
 +-----------------+--------------------------------------------------------+
 | Last Modified   | 2019-01-16                                             |
 +-----------------+--------------------------------------------------------+
-| Authors         | Nimisha Asthagiri <[email protected]>,                   |
-|                 | Danial Malik <[email protected]>,                   |
-|                 | Ed Zarecor <[email protected]>                               |
+| Authors         | - Nimisha Asthagiri <[email protected]>,                 |
+|                 | - Danial Malik <[email protected]>,                 |
+|                 | - Ed Zarecor <[email protected]>                             |
 +-----------------+--------------------------------------------------------+
-| Arbiter         | Brian Mesick <bmesick@axim.org>                        |
+| Arbiter         | Ty Hob <ty@axim.org>                                   |
 +-----------------+--------------------------------------------------------+
-| Status          | Provisional                                            |
+| Status          | Accepted                                               |
 +-----------------+--------------------------------------------------------+
 | Type            | Architecture                                           |
 +-----------------+--------------------------------------------------------+
 | Created         | 2018-07-11                                             |
 +-----------------+--------------------------------------------------------+
+| Last Updated    | 2025-02-05                                             |
++-----------------+--------------------------------------------------------+
 | `Review Period` | 2018-11-29 - 2018-12-20                                |
 +-----------------+--------------------------------------------------------+
 
+
 .. contents::
    :local:
    :depth: 3
 
+Status
+******
+
+As of 2025-02-05, this OEP has been accepted. It has been implemented in Open edX as the `event_routing_backends`_ plugin, which is currently in use for delivering real-time analytics to Aspects and to deliver real-time events to external systems via xAPI and Caliper. While that project adheres closely to this OEP, there are also other event-based communication mechanisms within the platform that are covered by other OEPs.
+
+`OEP-41 <oep-0041-arch-async-server-event-messaging>`_ and `OEP-52 <oep-0052-arch-event-bus-architecture>`_ cover the general format and conventions for generic asynchronous event messaging across and within services. While this OEP concerns itself learner-specific event sharing. There is also some overlap, for instance when event-routing-backends is configured to send events to an event bus, however this OEP is not considered superceded at this time.
+
+Where possible this document has been updated to match the implemented reality. However some sections may be out of date. The `event_routing_backends`_ plugin is the authoritative source for the current implementation.
+
+.. _event_routing_backends: https://github.com/openedx/event-routing-backends/
+
 Context
 *******
 
-Currently, the Open edX system captures LMS interactions in persisted event streams (tracking logs) of documented proprietary JSON structures, as specified in `Events in the Tracking Logs`_. The events are emitted by backend servers and frontend apps, including web and mobile apps. The logs are shared and used *asynchronously* for various data analyses and processing.
+The Open edX system captures LMS interactions in persisted event streams (tracking logs) of documented proprietary JSON structures, as specified in `Events in the Tracking Logs`_. The events are emitted by backend servers and frontend apps, including web and mobile apps. The logs are shared and mostly used *asynchronously* for various data analyses and processing, though through different log handlers / forwarders they can be used for more real-time applications.
 
-Of late, there are emerging use cases that require notifying external systems of these LMS events in real-time, preferably in industry standard formats (such as xAPI_ and Caliper_). By **real-time** events, we mean pushing data as fast as possible in an automated and synchronous communication at a speed within a few hundred milliseconds (to satisfy `human perception`_). Use cases, such as Adaptive Learning, do require this sub-second communication time. For other use cases, however, **near-real-time** communication (with a delay of multiple minutes) is sufficient. Unless explicitly called out in the document, near-real-time is included when we refer to real-time. Both are orders of magnitude faster than the current **batch processing** available via the tracking logs.
+There are emerging use cases that require notifying external systems of these LMS events in real-time, preferably in industry standard formats (such as xAPI_ and Caliper_). By **real-time** events, we mean pushing data as fast as possible in an automated and synchronous communication at a speed within a few hundred milliseconds (to satisfy `human perception`_). Use cases, such as Adaptive Learning, do require this sub-second communication time. For other use cases, however, **near-real-time** communication (with a delay of multiple minutes) is sufficient. Unless explicitly called out in the document, near-real-time is included when we refer to real-time. Both are orders of magnitude faster than the **batch processing** of tracking logs.
 
 This OEP proposes an evolutionary change to Open edX's eventing system that will enable near realtime event-based feedback loops.  Events have the potential to connect users, organizations, and learning services in ways that enrich learning on the platform.
 
@@ -42,7 +56,6 @@ This document describes a design proposal for supporting standard real-time even
 .. _xAPI: https://xapi.com/
 .. _Caliper: https://www.imsglobal.org/activity/caliper
 .. _human perception: https://www.pubnub.com/blog/how-fast-is-realtime-human-perception-and-technology/
-.. _Gartner reports: https://hackernoon.com/by-2020-50-of-managed-apis-projected-to-be-event-driven-88f7041ea6d8
 
 
 Use cases
@@ -98,7 +111,7 @@ Use Case: External Service
 
 **Requirement:** Real-time (*sub-second*) or near-real-time (*minutes*) communications depending on integration type
 
-As we move towards creating a more extensible platform for Open edX, we want to provide integration options between the Open edX platform and external services. External services that enhance the learning, authoring, or LMS administrative experiences will benefit greatly from a real-time eventing framework. Some examples include an external social learning service and a commercial learning analytics tool. *Theoretically, the other use cases are specific cases of this* *use case*.
+As we move towards creating a more extensible platform for Open edX, we want to provide integration options between the Open edX platform and external services. External services that enhance the learning, authoring, analytics, or LMS administrative experiences will benefit greatly from a real-time eventing framework. Some examples include an external social learning service and a learning analytics tool. *Theoretically, the other use cases are specific cases of this* *use case*.
 
 .. image:: oep-0026/use_cases.png
 
@@ -139,7 +152,7 @@ For details on integrating with Caliper, please see the :ref:`caliper_realtime_e
 Anonymized User ID
 ==================
 
-Users will be identified to external systems using a UUID that is associated uniquely with a single user and the external system type with which the UUID can be shared.  This decision overrides :ref:`oep-32` and is captured in `ADR 0001-externalid.rst`_.
+By default users will be identified to external systems using a UUID that is associated uniquely with a single user and the external system type with which the UUID can be shared.  This decision overrides :ref:`oep-32` and is captured in `ADR 0001-externalid.rst`_. Some external systems may require a different identifier for the user to support, for example, xAPI specifies a `mbox` field that is an email address. The system will allow for the configuration of a different identifier for each external system type to allow for flexibility of integration with external systems.
 
 .. _ADR 0001-externalid.rst: https://github.com/openedx/edx-platform/blob/e1c8fb51bec2be3aa8da9750cb52b01728d1a740/openedx/core/djangoapps/external_user_ids/docs/decisions/0001-externalid.rst
 
@@ -225,12 +238,6 @@ Decisions & Consequences
 
 * **Event-driven APIs at Scale** - The Context_ section describes the motivation and recent use cases for supporting real-time events. This capability has the potential to dramatically improve how external services can integrate and extend the Open edX platform.  Real-time events will provide a mechanism for loosely coupled integrations with a reliable, standardized contract.
 
-  A big consideration and concern that is sorely missing from this version of the OEP is explicit recommendations on the infrastructure that will be used to support scalability. On one hand, the advantage is that this agnostic approach allows Open edX instances to reuse the core capabilities (and modular subcomponents) without being tied to a specific scalable technology. On the other hand, we run the risk of needing to reimplement initial implementations if a chosen technology's design is fundamentally counter to our choice of boundaries.
-
-* **Emphasis on user privacy** - We are taking a conservative approach by minimizing the PII that is sent to consumers. The trade-off is that consumers may find the received user identifiers limiting. However, at this time, it's unclear whether adaptive engines, which are written generically for all users, need PII to be effective. They need the ability to bind events together and track pathways and progress for users, but they can do so with any unique identifier - hence the introduction of the :ref:`oep-26-user-id`.
-
-  For Enterprise and other use cases, sharing PII may be required. We have chosen to keep those use cases in mind, but not target them initially, with the understanding that future work would be needed to address those needs.
-
 * **Implementation of an LRS is out of scope** - As mentioned in :ref:`oep-26-lrs`, creating an Open edX specific LRS not planned.
 
   As a consequence, adaptive engines may need to maintain their own LRS if they need to refer back to previous events. Given our business research to date, it seems many adaptive engines are already maintaining their own custom-optimized storage of event data.
@@ -253,7 +260,7 @@ Here are a list of current Open edX frameworks that are related to "eventing" bu
 
 * **Event-driven microservices architecture** - Using event-driven asynchronous messaging between microservices is a design pattern we have adopted for the Open edX system. Such a loosely coupled architecture provides optimal scalability, resiliency and responsiveness, per recommendations in the `Reactive Manifesto`_.
 
-  As summarized in Pivotal's `Messaging Patterns for Event-Driven Microservices`_, there are various integration frameworks for supporting eventing frameworks. We currently use the "Asynchronous Command Calls" (2nd) pattern via Django Celery. We may eventually trial and adopt the "Event Firehose" (1st) pattern, using a technology such as Apache Kafka.
+  As summarized in Pivotal's `Messaging Patterns for Event-Driven Microservices`_, there are various integration frameworks for supporting eventing frameworks. By default we currently use the "Asynchronous Command Calls" (2nd) pattern via `Django Celery`_. As of February of 2025 both event-tracking and event-routing-backends can be configured to use the Kafka or Redis event bus implementations for sending real-time events as defined in :ref:`oep-52`.
 
   However, the requirements for that infrastructure are different from those captured in this OEP. This OEP requires support for standardized protocol(s) and configuration and access control (including PII considerations) for external consumers. Although it is theoretically possible for microservices to use this OEP's framework for inter-communication, it is not the intention. Events exchanged between microservices are within a firewall and can use non-standard schemas.
 
@@ -266,3 +273,104 @@ Here are a list of current Open edX frameworks that are related to "eventing" bu
   Although it is possible for this OEP's real-time eventing framework to send events targeted to IoT and personal devices, those events will not be translated nor customized for each individual recipient, nor be adaptive to the individual's policies and time sensitivities. ACE would be a better alternative for those requirements.
 
 .. _Automated Communication Engine (ACE): https://edx-ace.readthedocs.io/en/latest/index.html
+
+Change History
+**************
+
+2025-02-05
+==========
+
+* Mark as accepted
+* Update to reflect current implementation
+* `PR #678 <https://github.com/openedx/open-edx-proposals/pull/678>`_
+
+2023-11-06
+==========
+
+* Fixed various typos
+* `PR #537 <https://github.com/openedx/open-edx-proposals/pull/537>`_
+
+2023-04-14
+==========
+
+* Replace tCRIL references with Axim references
+* `PR #464 <https://github.com/openedx/open-edx-proposals/pull/464>`_
+
+2022-11-04
+==========
+
+* Change Ed Zarecor from arbiter to author, add Brian Mesick as Arbiter
+* Clarify language and update to reflect current thoughts
+* Remove references to Validator Processor
+* Change implementation of an LRS from "Deferring implementation" to "Out of scope"
+* `PR #405 <https://github.com/openedx/open-edx-proposals/pull/405>`_
+
+2022-09-27
+==========
+
+* Change arbiter from Brian Wilson to Ed Zarecor
+* `PR #386 <https://github.com/openedx/open-edx-proposals/pull/386>`_
+
+2022-09-16
+==========
+
+* Change Github organization references from edx to openedx
+* `PR #376 <https://github.com/openedx/open-edx-proposals/pull/376>`_
+
+2021-01-20 - 2022-08-02
+=======================
+
+* RsT style cleanup and minor content updates
+* `PR #365 <https://github.com/openedx/open-edx-proposals/pull/365>`_
+* `PR #248 <https://github.com/openedx/open-edx-proposals/pull/248>`_
+* `PR #245 <https://github.com/openedx/open-edx-proposals/pull/245>`_
+* `PR #206 <https://github.com/openedx/open-edx-proposals/pull/206>`_
+
+2021-04-09
+==========
+
+* Change http URLs to https
+* `PR #200 <https://github.com/openedx/open-edx-proposals/pull/200>`_
+
+2021-01-20
+==========
+
+* Clarify when we should build on Django admin vs settings or a self-service portal
+* `PR #156 <https://github.com/openedx/open-edx-proposals/pull/156>`_
+
+2020-12-07
+==========
+
+* Update example code
+* `PR #175 <https://github.com/openedx/open-edx-proposals/pull/175>`_
+
+2020-09-20 - 2020-09-30
+=======================
+
+* Cleaning up links and references
+* `PR #162 <https://github.com/openedx/open-edx-proposals/pull/162>`_
+* `PR #163 <https://github.com/openedx/open-edx-proposals/pull/163>`_
+
+2020-06-04
+==========
+
+* Significant updates to Caliper documentation
+* `PR #152 <https://github.com/openedx/open-edx-proposals/pull/152>`_
+
+2019-05-25
+==========
+
+* Update title to remove prefixing zeros for consistency
+* `PR #115 <https://github.com/openedx/open-edx-proposals/pull/115>`_
+
+2019-05-15
+==========
+
+* Update anonymized user ID section to unique user id as part of adding OEP-32
+* `PR #103 <https://github.com/openedx/open-edx-proposals/pull/103>`_
+
+2019-01-19
+==========
+
+* Initial publication
+* `PR #73 <https://github.com/openedx/open-edx-proposals/pull/73>`_