Replace context enter and exit events with single "context switched" event

[rhansen]

Feature or enhancement

Proposal:

(split off of #99633 (comment) to keep related pull requests and discussion better organized)

Starting in v3.14, C code can subscribe to "context enter" and "context exit" events:

cpython/Include/cpython/context.h

Lines 30 to 60 in 120729d

	typedef enum {
	Py_CONTEXT_EVENT_ENTER,
	Py_CONTEXT_EVENT_EXIT,
	} PyContextEvent;
	/*
	* Callback to be invoked when a context object is entered or exited.
	*
	* The callback is invoked with the event and a reference to
	* the context after its entered and before its exited.
	*
	* if the callback returns with an exception set, it must return -1. Otherwise
	* it should return 0
	*/
	typedef int (*PyContext_WatchCallback)(PyContextEvent, PyContext *);
	/*
	* Register a per-interpreter callback that will be invoked for context object
	* enter/exit events.
	*
	* Returns a handle that may be passed to PyContext_ClearWatcher on success,
	* or -1 and sets and error if no more handles are available.
	*/
	PyAPI_FUNC(int) PyContext_AddWatcher(PyContext_WatchCallback callback);
	/*
	* Clear the watcher associated with the watcher_id handle.
	*
	* Returns 0 on success or -1 if no watcher exists for the provided id.
	*/
	PyAPI_FUNC(int) PyContext_ClearWatcher(int watcher_id);

See gh-119333 for some background.

I would like to replace the just-added "context enter" and "context exit" events with a single "context switched" event:

diff --git a/Include/cpython/context.h b/Include/cpython/context.h
index ec72966e82c..f9638b0abe5 100644
--- a/Include/cpython/context.h
+++ b/Include/cpython/context.h
@@ -28,8 +28,12 @@ PyAPI_FUNC(int) PyContext_Enter(PyObject *);
 PyAPI_FUNC(int) PyContext_Exit(PyObject *);
 
 typedef enum {
-   Py_CONTEXT_EVENT_ENTER,
-   Py_CONTEXT_EVENT_EXIT,
+    /*
+     * The current context has switched to a different context.  The object
+     * passed to the watch callback is the now-current contextvars.Context
+     * object.
+     */
+    Py_CONTEXT_SWITCHED = 1,
 } PyContextEvent;
 
 /*
diff --git a/Python/context.c b/Python/context.c
index ddb03555f9e..6c9f68583c2 100644
--- a/Python/context.c
+++ b/Python/context.c
@@ -192,7 +192,7 @@ _PyContext_Enter(PyThreadState *ts, PyObject *octx)
     ts->context = Py_NewRef(ctx);
     ts->context_ver++;
 
-    notify_context_watchers(Py_CONTEXT_EVENT_ENTER, ctx, ts);
+    notify_context_watchers(Py_CONTEXT_SWITCHED, ctx, ts);
     return 0;
 }
 
@@ -226,13 +226,13 @@ _PyContext_Exit(PyThreadState *ts, PyObject *octx)
         return -1;
     }
 
-    notify_context_watchers(Py_CONTEXT_EVENT_EXIT, ctx, ts);
     Py_SETREF(ts->context, (PyObject *)ctx->ctx_prev);
     ts->context_ver++;
 
     ctx->ctx_prev = NULL;
     ctx->ctx_entered = 0;
 
+    notify_context_watchers(Py_CONTEXT_SWITCHED, (PyContext *)ts->context, ts);
     return 0;
 }
 

Rationale: Users want to know when the current context switches to a different context object. Right now this happens when and only when a context is entered or exited, so the enter and exit events are synonymous with "switched". However, if the changes proposed for gh-99633 are implemented, the current context will also switch for reasons other than context enter or exit. Since users actually care about context switches and not enter or exit, replacing the enter and exit events with a single switched event would make the new feature compatible with gh-99633.

The current "context exit" event is emitted just before exiting the context. The new "context switched" event would be emitted after the context is exited to match the semantics users expect of an event with a past-tense name. If users need the ability to clean up before the switch takes effect, another "context is about to switch" event can be added in the future. I am not proposing to add such an event for this issue because no users have requested it yet (YAGNI).

I would love to get this resolved before v3.14 is released so that we don't have to worry about breaking backwards compatibility.

Has this already been discussed elsewhere?

I have already discussed this feature proposal on another bug report.

Links to previous discussion of this feature:

#99633 (comment)

Linked PRs

• gh-124872: Move PyThreadState to first argument for consistency #124774
• gh-124872: Refine contextvars documentation #124773
• gh-124872: Back up exception before calling PyContext_WatchCallback #124741
• gh-124872: Change PyContext_WatchCallback to take PyObject #124737
• gh-124872: Replace enter/exit events with "switched" #124776
• [3.13] gh-124872: Refine contextvars documentation (GH-124773) #125233
• gh-124872: Rename blurb file to reference the correct issue #125285
• gh-124872: Replace enter/exit events with "switched" #125532
• gh-124872: Mark the thread's default context as entered #125638

[ZeroIntensity]

Generally, it's a bad idea to create several different PRs for the same issue -- you can put all the needed changes in the same PR.

[rhansen]

Apologies—I was under the impression that this project squashes pull requests when merging, which I don't want (the commits are intended to be separate). I can combine them into a single PR if that would be preferred.

[ZeroIntensity]

We do squash, but why do you want the commits to be separate? It's harder to review and backport across multiple PRs

[rhansen]

I'm following the Python Developer's Guide:

• Do not fix more than one issue in the same commit (except, of course, if one code change fixes all of them).
• Do not do cosmetic changes to unrelated code in the same commit as some feature/bugfix.

Each of these PRs is focused on only one thing, and they are independent from each other (but all support the overall goal of this issue).

In general, it's bad practice to have large monolithic commits that do multiple separable things. Small, focused, and independent commits have several advantages:

• They are better for locating the root cause of a bug by bisecting.
• They are easier to cherry-pick and rebase (less likely to have merge conflicts).
• They are easier to revert if a bug is found.
• Most importantly, they are easier to understand both by current reviewers and by future code archaeologists trying to understand past design decisions.

I can put all of the commits in a single pull request, but if they are squashed then much will be lost.

[ZeroIntensity]

Each of these PRs is focused on only one thing, and they are independent from each other (but all support the overall goal of this issue).

In that case, it's better to make a new issue for each PR (I don't think it's worth doing so at this point, but this is just for future reference). If they really are all "independent", then having all the PRs under one issue makes this more difficult for us. If, for example, one PR gets merged while another gets shot down, what do we do about this issue? Does it stay closed or open?

• Most importantly, they are easier to understand both by current reviewers and by future code archaeologists trying to understand past design decisions.

Eh, looking through a tree of "related yet independent" PRs is arguably more difficult for people trying to understand design decisions. With one PR, all the reviews are in one place. For example, #124774 looks like a cosmetic change -- it would honestly fit better in a larger PR, where there's lots of other refactoring going on (and also easier to review without being brushed off as cosmetic).

I can put all of the commits in a single pull request, but if they are squashed then much will be lost.

GitHub saves the commits for us :)

In terms of bisecting, we can always checkout the branch that got merged and bisect to individual commits on it. But in my experience, we haven't ever needed to do that. Once we've found a bad commit, it's typically easy to figure out what went wrong without doing any extra micro-bisections.

[rhansen]

Each of these PRs is focused on only one thing, and they are independent from each other (but all support the overall goal of this issue).

In that case, it's better to make a new issue for each PR (I don't think it's worth doing so at this point, but this is just for future reference).

Creating a separate issue per PR seems like a heavyweight process to me. Why have two places to discuss the same thing? Though I understand if it makes it easier for the project to track done/pending status.

Also, I think it's easier to understand relationships between changes if there is one issue with N associated pull requests vs. N separate issues with one corresponding pull request each.

If they really are all "independent", then having all the PRs under one issue makes this more difficult for us. If, for example, one PR gets merged while another gets shot down, what do we do about this issue? Does it stay closed or open?

How about: "If there are any open PRs associated with an issue, then the issue is not yet resolved."

• Most importantly, they are easier to understand both by current reviewers and by future code archaeologists trying to understand past design decisions.

Eh, looking through a tree of "related yet independent" PRs is arguably more difficult for people trying to understand design decisions. With one PR, all the reviews are in one place. For example, #124774 looks like a cosmetic change -- it would honestly fit better in a larger PR, where there's lots of other refactoring going on (and also easier to review without being brushed off as cosmetic).

I think the ideal solution would be to require contributors to indicate whether their PR is intended to be squashed vs. rebased-then-merged (aka semi-linear history):

• Squash PRs would be treated as they are now.
• Rebase-then-merge PRs would be rebased onto latest main then merged with a merge commit (git merge --no-ff). Contributors would be expected to frequently rebase and force-push their PR branch to address review comments, resolve conflicts with main, etc.

Rebase-then-merge PRs would be easier to review/manage/understand vs. N separate PRs, but would still have the advantages of N small and focused commits. The N commits would be grouped together in the Git history, linked to main with a merge commit that references the PR. This makes it easy to see that they are related and all part of a particular pull request. Also, Git's --first-parent option can be used to bisect/examine the history at PR granularity.

Unfortunately, the review experience is still not great when force pushes are involved. Also, GitHub has poor/no support for semi-linear history (GitLab is superior in this regard). It can still be done manually, but this is error-prone and tedious. Maybe bots can help. Maybe someone has already written one.

I can put all of the commits in a single pull request, but if they are squashed then much will be lost.

GitHub saves the commits for us :)

git blame doesn't walk GitHub PR commits.

In terms of bisecting, we can always checkout the branch that got merged and bisect to individual commits on it.

This doesn't work for PRs written with the expectation that the commits will be squashed. Such PRs often have intermediate commits that are works-in-progress (they don't compile, don't pass tests, etc.).

But in my experience, we haven't ever needed to do that. Once we've found a bad commit, it's typically easy to figure out what went wrong without doing any extra micro-bisections.

True.

[ZeroIntensity]

All in all, this isn't common practice for CPython. Let's try to stick with tradition :)

[rhansen]

On a related note, it would be nice if this tradition was documented in the devguide. :)

[ZeroIntensity]

It is, sort of. There's supposed to be an implied 1:1 correspondence between issues and PRs (with the exception of backports).

[picnixz]

On the essence of this issue: while I agree that most of the time you only want to know whether the context switched or not, is there actually a way to keep the enter/exit events?

When you enter or exit, you switch. But what if you really want to, say, track down the enter/exit events and not just any kind of switch? (like, you want to count that the number of times you exited is the same as the number of times you entered). I don't know whether this is me who's trying to find issues where there are none, but I usually like events with granularity, even though I don't know if I would use them for now.

[rhansen]

Per the PR description, the related PRs: #124737, #124741, and #124774 need to be merged first.

All of the related PRs have been merged (thank you!) except #124741. I rebased #124776 (the core of this issue) onto latest main, excluding #124741 which appears to be a bit more controversial so I didn't want it to block #124776.

#124776 is now marked as ready for review, and this issue can be closed once that PR is merged even if #124741 is still open.

(Personally, I would have rathered all the needed changes in one PR, but the damage is done at this point.)

Me too. Normally I would put all commits in a single PR, but this project squashes all PRs. I've been burned in the past by single commits doing too many things—for example, bugs introduced by failing to correctly revert or cherry-pick a portion of a monolithic commit. If it was possible to choose a semi-linear merge instead of a squash when opening a PR I would have opened a single PR with multiple commits.