Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable Callbacks to change the result of dataset #15

Open
richardhathaway-8451 opened this issue Oct 18, 2024 · 2 comments
Open

Enable Callbacks to change the result of dataset #15

richardhathaway-8451 opened this issue Oct 18, 2024 · 2 comments

Comments

@richardhathaway-8451
Copy link

richardhathaway-8451 commented Oct 18, 2024

Enable callbacks to change the result of a dataset.

This is particularly useful for adding extra logic to the result of ALL implementations of an abstractdataset, like in an interface.

@interface(dispatch=Option("MODE"))
class A:
  @abstractdataset(callbacks = my_extra_logic)
  def number():
    ...

@pipeline_step
def my_extra_logic(df):
  # do extra logic and return

Callbacks are different from Effects. Callbacks are fixed at the time the dataset is defined and cannot be altered, but they can change the result of the dataset. Effects are not fixed at the time the dataset is defined, so they can be altered later, but they cannot affect the result of the dataset and the chain of evaluation.

@richardhathaway-8451
Copy link
Author

Another issue that a callback would solve is when a user wants to by default tack on additional logic to the end of a dataset (but also have the option to turn off that additional logic).

Currently this is being achieved with overloads using .default, but it is a bit strange to have an overload where the default implementation is to always use the overloaded version.

@dataset(dispatch = Option("TEST", True))
def x():
  return 1

@x.overload(alias = True)
def x2(val = x.default):
  return val + 1

@austinwarner-8451
Copy link
Collaborator

Added on feature/callback. The argument name is callback and takes a single callback, which can be either a plain callable or something that evaluates to a callable (e.g. a Pipeline or PipelineStep). Callbacks can be run in series by constructing a Pipeline via @dataset(callback=pipeline_step_1 + ... + pipeline_step_n) or @dataset(callback=Pipeline() + plain_step_1 + ... + plain_step_n)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants