Using Kedro Plugins for existing Python project

[noklam]

Description

Related to #410

I want to explore the ability to use Kedro-plugins with existing project. Let's say we want to expand our users group, one may only use DataCatalog and ConfigLoader. While some of the components could be use standalone, but they are not as useful as using it in a Kedro Project.

Outcome of this

• Is this possible currently? What is the minimal changes to make this work?
• Can a notebook user enjoy the benefit of kedro-plugins ecosystem?

I see this adds a lot value to users who wish to use Kedro partly, and potentially lead to more users to migrate to a full kedro project.

p.s. I think this may related to the milestone for using kedro components, but I cannot find the milestone maybe it's not created yet?

Context

• Growing our user bases beyond full kedro users

If this is possible, I can see that we don't even need to market Kedro itself, it will be simply enabling "data validation for notebook users". This will largely depends on the specific plugin though, not all plugins will make sense for non-Kedro project.

Possible Implementation

Possible Alternatives

[astrojuanlu]

If I understand correctly, pluggy triggers are mainly in KedroSession, am I right?

I don't see how automatically triggering plugins could work when using Kedro as a library, but I agree we could explore how one could use existing plugins in a more manual way.

[noklam]

@astrojuanlu Not all. For example before_dataset_loaded live inside runner.py
https://github.com/kedro-org/kedro/blob/main/kedro/runner/runner.py#L348

In addition it doesn't have to be automatically triggered. As long as it understand the structure(using kedro project), or there is a way to configure the structure (using component). i.e. Offline data check via DataCatalog, runtime data check (kedro run, this is only possible with a full kedro project).

[astrojuanlu]

Maybe this is only tangentially related but I contend that Hooks is a framework thing and not a library thing, and therefore Runner should not have any triggers.

Could be related to https://github.com/kedro-org/kedro/milestone/51

[astrojuanlu]

Turning this into a Discussion.