Implement Receiver resource filtering with CEL #948

bigkevmcd · 2024-10-10T05:37:38Z

This implementation allows filtering of wildcarded resources for receivers using CEL expressions.

Closes: #491

bigkevmcd · 2024-10-10T05:39:57Z

@stefanprodan @matheuscscp I reworked it to filter the resources using CEL.

I'm not quite sure that this is really that valuable, I suspect label filtering is also more efficient?

There's some optimisation of the CEL that could be done within requestReconciliation to reduce the creation of CEL values and I can do that but I thought I'd get this out for comment.

internal/server/receiver_handlers.go

matheuscscp

Thanks again for all the work here, @bigkevmcd! I'm very excited for this feature 😄

This is just a preliminary review. I will try to review this again soon.

internal/server/receiver_handlers.go

stefanprodan

LGTM

Thanks @bigkevmcd 🥇

matheuscscp

LGTM! Thanks @bigkevmcd! Pretty cool contribution 👍

internal/server/cel.go

internal/server/receiver_handlers.go

api/v1/receiver_types.go

stefanprodan · 2024-11-01T20:31:01Z

@bigkevmcd this is good to go, please signoff all your commits and force push.

internal/controller/receiver_controller.go

internal/server/cel.go

bigkevmcd · 2024-11-13T16:43:36Z

@darkowlzz It'll be a couple of weeks before I can devote more time to these things, but I will get to them.

darkowlzz

Almost concluding my CEL research with this. I don't think there's any hurry. This can be addressed whenever you're back to it. 🙂

internal/server/cel.go

pjbgf

@bigkevmcd really good stuff, I am looking forward to seeing this landing. 🙇

internal/server/receiver_handlers.go

internal/server/cel.go

bigkevmcd · 2024-11-18T19:40:35Z

@pjbgf

+1 on leaning on k8s' default limits as a starting point. It might be worth just confirming whether the payload and expression sizes for this use case are equivalent, so that those limits would not be too forgiving.

https://github.com/kubernetes/kubernetes/blob/cf480a3a1a9cb22f3439c0a7922822d9f67f31b5/staging/src/k8s.io/apiserver/pkg/apis/cel/config.go#L40

That's 3MiB, I've definitely seen GitHub hook notifications that were > 1.5MiB.

matheuscscp

New batch of comments ❤️

api/v1/receiver_types.go

docs/spec/v1/receivers.md

api/v1/condition_types.go

matheuscscp

Still need to go over internal/server/receiver_handlers.go once again

docs/spec/v1/receivers.md

internal/controller/receiver_controller.go

internal/server/cel.go

internal/controller/receiver_controller.go

internal/server/cel.go

internal/server/cel_test.go

matheuscscp · 2025-01-31T14:48:49Z

@bigkevmcd we have released the fluxcd/pkg packages, please pull them in with:

go get github.com/fluxcd/pkg/apis/[email protected] for InvalidCELExpressionReason
go get github.com/fluxcd/pkg/[email protected] for the runtime/cel library

🙏

Edit: Like Stefan said below, just need to rebase.

stefanprodan · 2025-02-01T07:20:13Z

@bigkevmcd if you rebase with main, you'll find runtime/cel and the InvalidCELExpressionReason.

Signed-off-by: Kevin McDermott <[email protected]> Co-authored-by: Matheus Pimenta <[email protected]>

stefanprodan

🚀

matheuscscp · 2025-02-11T08:39:55Z

Thanks very much for this contribution @bigkevmcd ! 🚀 🚀 🚀

bigkevmcd force-pushed the cel-resource-filtering branch from 4969c46 to 1fe996b Compare October 10, 2024 05:40

github-advanced-security bot found potential problems Oct 10, 2024

View reviewed changes

internal/server/receiver_handlers.go Fixed Show fixed Hide fixed

bigkevmcd force-pushed the cel-resource-filtering branch 3 times, most recently from d87dbfd to b1c4e7e Compare October 10, 2024 19:03

stefanprodan changed the title ~~CEL Resource Filtering~~ Implement Receiver resource filtering with CEL Oct 11, 2024

bigkevmcd force-pushed the cel-resource-filtering branch from 45bd724 to 1e6929c Compare October 13, 2024 14:57

bigkevmcd marked this pull request as ready for review October 13, 2024 15:18

bigkevmcd marked this pull request as draft October 16, 2024 07:30

bigkevmcd force-pushed the cel-resource-filtering branch from 1e6929c to d658d4a Compare October 17, 2024 08:46

bigkevmcd marked this pull request as ready for review October 17, 2024 08:54

matheuscscp reviewed Oct 17, 2024

View reviewed changes

internal/server/receiver_handlers.go Show resolved Hide resolved

github-advanced-security bot found potential problems Oct 17, 2024

View reviewed changes

internal/server/receiver_handlers.go Fixed Show fixed Hide fixed

bigkevmcd force-pushed the cel-resource-filtering branch 4 times, most recently from 5a22c4f to bddfd56 Compare October 18, 2024 13:06

stefanprodan approved these changes Oct 28, 2024

View reviewed changes

matheuscscp approved these changes Oct 28, 2024

View reviewed changes

darkowlzz reviewed Oct 31, 2024

View reviewed changes

internal/server/cel.go Outdated Show resolved Hide resolved

internal/server/receiver_handlers.go Outdated Show resolved Hide resolved

api/v1/receiver_types.go Show resolved Hide resolved

bigkevmcd force-pushed the cel-resource-filtering branch 6 times, most recently from 8e892ef to 18c1f31 Compare November 1, 2024 14:34

stefanprodan reviewed Nov 1, 2024

View reviewed changes

api/v1/receiver_types.go Outdated Show resolved Hide resolved

bigkevmcd force-pushed the cel-resource-filtering branch from 8a9e435 to 57c36b0 Compare November 1, 2024 20:58

darkowlzz added the area/api API related issues and pull requests label Nov 4, 2024

github-advanced-security bot found potential problems Nov 10, 2024

View reviewed changes

internal/controller/receiver_controller.go Fixed Show fixed Hide fixed

bigkevmcd force-pushed the cel-resource-filtering branch from 348c595 to 2636e01 Compare November 10, 2024 16:52

darkowlzz reviewed Nov 13, 2024

View reviewed changes

bigkevmcd force-pushed the cel-resource-filtering branch from 891a42c to 757f96d Compare November 13, 2024 17:44

darkowlzz reviewed Nov 14, 2024

View reviewed changes

internal/server/cel.go Outdated Show resolved Hide resolved

pjbgf reviewed Nov 15, 2024

View reviewed changes

bigkevmcd force-pushed the cel-resource-filtering branch from 757f96d to 1645796 Compare December 12, 2024 20:03

bigkevmcd force-pushed the cel-resource-filtering branch from 1645796 to f79599e Compare January 12, 2025 17:55

This was referenced Jan 28, 2025

[RFC-0009] Add InvalidCELExpressionReason to apis/meta fluxcd/pkg#859

Merged

[RFC-0009] Add healthcheck expressions to kustomize types fluxcd/pkg#857

Merged

[RFC-0009] Add CEL library with custom healthchecks to runtime fluxcd/pkg#858

Merged

bigkevmcd force-pushed the cel-resource-filtering branch from f79599e to e438ca1 Compare January 28, 2025 13:45

matheuscscp self-requested a review January 28, 2025 17:12

matheuscscp reviewed Jan 28, 2025

View reviewed changes

matheuscscp reviewed Jan 29, 2025

View reviewed changes

bigkevmcd force-pushed the cel-resource-filtering branch from 15d0f16 to eb6f1ce Compare February 3, 2025 08:32

matheuscscp mentioned this pull request Feb 3, 2025

runtime: downgrade github.com/google/cel-go and cel.dev/expr fluxcd/pkg#865

Closed

bigkevmcd force-pushed the cel-resource-filtering branch from 7183c3c to fc59218 Compare February 3, 2025 19:01

matheuscscp force-pushed the cel-resource-filtering branch 2 times, most recently from 592c82e to be22b2b Compare February 10, 2025 16:03

Implement Receiver resource filtering with CEL

28deef9

Signed-off-by: Kevin McDermott <[email protected]> Co-authored-by: Matheus Pimenta <[email protected]>

matheuscscp force-pushed the cel-resource-filtering branch from be22b2b to 28deef9 Compare February 10, 2025 16:22

stefanprodan approved these changes Feb 11, 2025

View reviewed changes

stefanprodan merged commit 2eb3c35 into fluxcd:main Feb 11, 2025
6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Receiver resource filtering with CEL #948

Implement Receiver resource filtering with CEL #948

bigkevmcd commented Oct 10, 2024 •

edited by stefanprodan

Loading

bigkevmcd commented Oct 10, 2024 •

edited

Loading

matheuscscp left a comment •

edited

Loading

stefanprodan left a comment

matheuscscp left a comment

stefanprodan commented Nov 1, 2024

bigkevmcd commented Nov 13, 2024

darkowlzz left a comment

pjbgf left a comment

bigkevmcd commented Nov 18, 2024

matheuscscp left a comment

matheuscscp left a comment

matheuscscp commented Jan 31, 2025 •

edited

Loading

stefanprodan commented Feb 1, 2025

stefanprodan left a comment

matheuscscp commented Feb 11, 2025

Implement Receiver resource filtering with CEL #948

Implement Receiver resource filtering with CEL #948

Conversation

bigkevmcd commented Oct 10, 2024 • edited by stefanprodan Loading

bigkevmcd commented Oct 10, 2024 • edited Loading

matheuscscp left a comment • edited Loading

Choose a reason for hiding this comment

stefanprodan left a comment

Choose a reason for hiding this comment

matheuscscp left a comment

Choose a reason for hiding this comment

stefanprodan commented Nov 1, 2024

bigkevmcd commented Nov 13, 2024

darkowlzz left a comment

Choose a reason for hiding this comment

pjbgf left a comment

Choose a reason for hiding this comment

bigkevmcd commented Nov 18, 2024

matheuscscp left a comment

Choose a reason for hiding this comment

matheuscscp left a comment

Choose a reason for hiding this comment

matheuscscp commented Jan 31, 2025 • edited Loading

stefanprodan commented Feb 1, 2025

stefanprodan left a comment

Choose a reason for hiding this comment

matheuscscp commented Feb 11, 2025

bigkevmcd commented Oct 10, 2024 •

edited by stefanprodan

Loading

bigkevmcd commented Oct 10, 2024 •

edited

Loading

matheuscscp left a comment •

edited

Loading

matheuscscp commented Jan 31, 2025 •

edited

Loading