Allow for registering feature that doesn't have warehouse store

[pradithya]

Registering feature that doesn't have warehouse store will currently return error.
There is a use case that some feature doesn't need warehouse store, so the restriction could be removed.
Additionally, ingestion will need to be updated to do no operation for warehouse storing operation.

[woop]

So the question here is

• Do we allow toggling of the warehouse storage option at the feature level
• or at the system level (Feast deployment)
• or both?

I think we should have the option to both disable warehouse storage on specific features, as well as deploy a Feast instance without a warehouse (serving only)

[tims]

My pull request fixes a typo.. But basically ingestion already supports setting a "no operation" store, for either serving or ingestion. The core api would still need to implement sending it of course.

[woop]

My pull request fixes a typo.. But basically ingestion already supports setting a "no operation" store, for either serving or ingestion. The core api would still need to implement sending it of course.

So the next step is a PR which adds this functionality to core?

@zhilingc, possible to have a look at this?

[tims]

I'm also working on a PR to make ingestion accept an empty storage spec. Though in reality, that will interpreted as a StorageSpec.defaultInstance(). With empty string as the storeid. Ingestion will then treat this as a noop store.

[zhilingc]

Ok. You might want to remove the validation check that makes declaring a store mandatory when registering a feature as well.

[tims]

Aren't you worried people will register features and they will just all go to noop when they leave it out though? We do need to implement inheriting a system default first.

And then a user needs to be able to override the default to a noop.

Which now that I think about might be a pain. As if they set the store id to an empty string, it will look the same as the default instance, so how will you know when to override it with the default or not?

Remember protos not having optional values means that there's no such thing as a null store id.

[woop]

Aren't you worried people will register features and they will just all go to noop when they leave it out though? We do need to implement inheriting a system default first.

And then a user needs to be able to override the default to a noop.

Which now that I think about might be a pain. As if they set the store id to an empty string, it will look the same as the default instance, so how will you know when to override it with the default or not?

Remember protos not having optional values means that there's no such thing as a null store id.

Isn't a better approach simply to have an option to switch off either store? That we we don't have to look at the value and try to guess whether it should be on or off. I think the default should be inheritance.

[zhilingc]

When is the default declared? Should it be in the configuration when deploying core Feast? Or is it a necessary step that has to be done by the user before registering features (but after deployment)?

[woop]

When is the default declared? Should it be in the configuration when deploying core Feast? Or is it a necessary step that has to be done by the user before registering features (but after deployment)?

I think it should be done during initial installation. That way the user could theoretically have all data storage abstracted away from them.

[tims]

@zhilingc yes it should be a system default. So it should come with the system.

Users shouldn't even be creating new stores? I think we made store specs creatable accidentally to be honest, because all the others are. They should probably only ever be created by the person who deploys a feast instance.

[tims]

Can we close this issue now?

[tims]

Closing, but reopen if I missed something.

[pradithya]

Reopening the issue, core is still checking for presence of warehouse store. I am crafting a PR

[pradithya]

I am still having this error when trying to ingest data for feature having no warehouse store.
The feature spec looks like as follow:

id: "myentity.minute.feature_1"
name: "feature_1"
owner: "[email protected]"
description: "Test feature"
uri: "NA"
granularity: "MINUTE"
valueType: "DOUBLE"
entity: "myentity"
dataStores: 
  serving: 
    id: "REDIS1"

Exception:

Optional[Exception in thread "main" java.lang.IllegalArgumentException: no write transforms found
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
        at feast.ingestion.transform.SplitOutputByStore.expand(SplitOutputByStore.java:57)
        at feast.ingestion.transform.SplitOutputByStore.expand(SplitOutputByStore.java:45)
        at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
        at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
        at feast.ingestion.values.PFeatureRows.apply(PFeatureRows.java:109)
        at feast.ingestion.transform.WarehouseStoreTransform.expand(WarehouseStoreTransform.java:45)
        at feast.ingestion.transform.WarehouseStoreTransform.expand(WarehouseStoreTransform.java:30)
        at org.apache.beam.sdk.Pipeline.applyInternal(Pipeline.java:537)
        at org.apache.beam.sdk.Pipeline.applyTransform(Pipeline.java:488)
        at feast.ingestion.values.PFeatureRows.apply(PFeatureRows.java:109)
        at feast.ingestion.ImportJob.expand(ImportJob.java:184)
        at feast.ingestion.ImportJob.mainWithResult(ImportJob.java:126)
        at feast.ingestion.ImportJob.main(ImportJob.java:110)]

[pradithya]

Apparently the ingestion job succeed but all feature row didn't pass validation