Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Store a reference to the engine in receiver objects #478

Merged
merged 2 commits into from
Jul 4, 2023

Conversation

shsms
Copy link
Contributor

@shsms shsms commented Jul 3, 2023

This PR fixes a bug which occurs when creating and using a higher order formula engine, without holding a reference to it.

@shsms shsms requested a review from a team as a code owner July 3, 2023 12:45
@shsms shsms requested a review from llucax July 3, 2023 12:45
@github-actions github-actions bot added the part:data-pipeline Affects the data pipeline label Jul 3, 2023
Copy link
Contributor

@llucax llucax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe this is not necessary. Let's discuss it first.

Copy link
Contributor

@llucax llucax left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, we agreed this is necessary because tasks as saved as weak references by the loop (https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task). I would just add the link to the docs saying this for extra clarity and for me it is good to go.

If you create a new specific formula engine receiver that has an explicit formula engine attribute, then this wouldn't be a hack, right? You say this is a hack only because we are sticking an attribute to a receiver that doesn't have one, right?

shsms added 2 commits July 3, 2023 15:10
This is a hack to ensure that the lifetime of the engine is tied to the
lifetime of the receiver.  This is necessary because the engine is a task that
runs forever, and in cases where higher order built for example with the below
idiom, the user would hold no references to the engine and it could get
garbage collected before the receiver.

    formula = (grid_power_engine + bat_power_engine).build().new_receiver()

Signed-off-by: Sahas Subramanian <[email protected]>
Signed-off-by: Sahas Subramanian <[email protected]>
@github-actions github-actions bot added the part:docs Affects the documentation label Jul 3, 2023
@shsms
Copy link
Contributor Author

shsms commented Jul 3, 2023

we agreed this is necessary because tasks as saved as weak references by the loop (https://docs.python.org/3/library/asyncio-task.html#asyncio.create_task).

right, so without this, the engine holds a strong reference to the task that runs the formula evaluator. But there might be cases where users don't have a reference to the engine, like in the example above.

Then the user would have a reference to the receiver, and it will have a reference to the channel, but the channel doesn't hold on to anything, so the engine would be dangling, ready for clean up.

I would just add the link to the docs saying this for extra clarity and for me it is good to go.

done.

If you create a new specific formula engine receiver that has an explicit formula engine attribute, then this wouldn't be a hack, right? You say this is a hack only because we are sticking an attribute to a receiver that doesn't have one, right?

I think if we do that, we'd be transferring the hack to the receiver, because it would have to have generics to get the typing right, not that we're short on typing hacks in this package. Also, it won't be enough to just create a receiver, we'd have to create a specialized channel also, because the receivers are actually created by the channels, and the Broadcast channel would only create its default receiver, unless it is patched or overridden. Right now, it is a hack, but it is at least easy and readable.

@matthias-wende-frequenz
Copy link
Contributor

Right now, it is a hack, but it is at least easy and readable.

given that this actually gives a dangling reference a place, I wouldn't call it a hack but rather a bugfix.

@shsms
Copy link
Contributor Author

shsms commented Jul 3, 2023

I wouldn't call it a hack but rather a bugfix.

It is a hack, because we are injecting an attribute with an undocumented name that can't be found anywhere in the channels repo. But as @llucax accurately put it,

the hack is not saving the reference but how are you saving the reference,

But yes, it is also a bug fix. So I've updated the PR description to say just the what, and not the how.

@shsms shsms added this to the v0.22.0 milestone Jul 3, 2023
@llucax
Copy link
Contributor

llucax commented Jul 3, 2023

I think if we do that, we'd be transferring the hack to the receiver, because it would have to have generics to get the typing right, not that we're short on typing hacks in this package. Also, it won't be enough to just create a receiver, we'd have to create a specialized channel also, because the receivers are actually created by the channels, and the Broadcast channel would only create its default receiver, unless it is patched or overridden. Right now, it is a hack, but it is at least easy and readable.

Yeah, true, it is not that easy to make it proper into the receiver. I didn't think it was worth writing a new receiver to make it less hacky, I think the hack is super contained and hidden, but more thinking about it there was a la hacky way to fix the bug.

@shsms shsms added this pull request to the merge queue Jul 4, 2023
Merged via the queue into frequenz-floss:v0.x.x with commit 1b7053e Jul 4, 2023
@shsms shsms deleted the store-engines branch July 4, 2023 08:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
part:data-pipeline Affects the data pipeline part:docs Affects the documentation
Projects
Development

Successfully merging this pull request may close these issues.

3 participants