Issues with Feast stack when deployed via Operator - due to using remote provider type

[tchughesiv]

Expected Behavior

When deployed with the Operator, Feast should be fully functional.

Current Behavior

Several methods failing... e.g.

$ feast apply
No project found in the repository. Using project name sample defined in feature_store.yaml
Applying changes for project sample
/feast/sdk/python/feast/feature_store.py:575: RuntimeWarning: On demand feature view is an experimental feature. This API is stable, but the functionality does not scale well for offline retrieval
  warnings.warn(
12/05/2024 02:35:01 PM feast.errors WARNING: Invalid error detail: : Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/usr/local/bin/feast", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/cli.py", line 760, in apply_total_command
    apply_total(repo_config, repo, skip_source_validation)
  File "/feast/sdk/python/feast/repo_operations.py", line 404, in apply_total
    apply_total_with_repo_instance(
  File "/feast/sdk/python/feast/repo_operations.py", line 351, in apply_total_with_repo_instance
    store.apply(all_to_apply, objects_to_delete=all_to_delete, partial=False)
  File "/feast/sdk/python/feast/feature_store.py", line 933, in apply
    self._make_inferences(
  File "/feast/sdk/python/feast/feature_store.py", line 613, in _make_inferences
    update_feature_views_with_inferred_features_and_entities(
  File "/feast/sdk/python/feast/inference.py", line 180, in update_feature_views_with_inferred_features_and_entities
    _infer_features_and_entities(
  File "/feast/sdk/python/feast/inference.py", line 230, in _infer_features_and_entities
    provider.get_table_column_names_and_types_from_data_source(
  File "/feast/sdk/python/feast/infra/passthrough_provider.py", line 524, in get_table_column_names_and_types_from_data_source
    return self.offline_store.get_table_column_names_and_types_from_data_source(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/infra/offline_stores/remote.py", line 420, in get_table_column_names_and_types_from_data_source
    table = _send_retrieve_remote(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/infra/offline_stores/remote.py", line 494, in _send_retrieve_remote
    return _call_get(client, command_descriptor)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/infra/offline_stores/remote.py", line 503, in _call_get
    reader = client.do_get(ticket)
             ^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/arrow_error_handler.py", line 20, in wrapper
    raise e
  File "/feast/sdk/python/feast/arrow_error_handler.py", line 15, in wrapper
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/infra/offline_stores/remote.py", line 53, in do_get
    return super().do_get(ticket, options)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_flight.pyx", line 1633, in pyarrow._flight.FlightClient.do_get
  File "pyarrow/_flight.pyx", line 58, in pyarrow._flight.check_flight_status
pyarrow._flight.FlightServerError: FlightServerBase.do_get must return a FlightDataStream. Detail: Python exception: Traceback (most recent call last):
  File "pyarrow/_flight.pyx", line 2185, in pyarrow._flight._do_get
TypeError: FlightServerBase.do_get must return a FlightDataStream
. gRPC client debug context: UNKNOWN:Error received from peer ipv4:172.30.141.142:80 {grpc_message:"FlightServerBase.do_get must return a FlightDataStream. Detail: Python exception: Traceback (most recent call last):\n  File \"pyarrow/_flight.pyx\", line 2185, in pyarrow._flight._do_get\nTypeError: FlightServerBase.do_get must return a FlightDataStream\n", grpc_status:2, created_time:"2024-12-05T14:35:01.244475681+00:00"}. Client context: IOError: Server never sent a data message. Detail: Internal 

$ python3 test_workflow.py
--- Run feast apply to setup feature store on Postgres ---
No project found in the repository. Using project name sample defined in feature_store.yaml
Applying changes for project sample
/feast/sdk/python/feast/feature_store.py:575: RuntimeWarning: On demand feature view is an experimental feature. This API is stable, but the functionality does not scale well for offline retrieval
  warnings.warn(
Deploying infrastructure for driver_hourly_stats_fresh
Deploying infrastructure for driver_hourly_stats--- Historical features for training ---
   driver_id     event_timestamp  label_driver_reported_satisfaction  val_to_add  val_to_add_2  conv_rate  acc_rate  avg_daily_trips
0       1001 2021-04-12 10:59:42                                   1           1            10   0.654609  0.089119               56
1       1002 2021-04-12 08:12:10                                   5           2            20   0.628438  0.045928              642
2       1003 2021-04-12 16:40:26                                   3           3            30   0.597835  0.370411              682--- Historical features for batch scoring ---
   driver_id            event_timestamp  label_driver_reported_satisfaction  val_to_add  val_to_add_2  conv_rate  acc_rate  avg_daily_trips
0       1001 2024-12-05 14:45:02.856623                                   1           1            10   0.258632  0.935114               59
1       1002 2024-12-05 14:45:02.856623                                   5           2            20   0.294726  0.869369              341
2       1003 2024-12-05 14:45:02.856623                                   3           3            30   0.759920  0.541704               69--- Load features into online store ---
Materializing 2 feature views to 2024-12-05 14:45:03+00:00 into the remote online store.driver_hourly_stats from 2024-12-05 14:44:04+00:00 to 2024-12-05 14:45:03+00:00:
0it [00:00, ?it/s]
driver_hourly_stats_fresh from 2024-12-05 14:44:04+00:00 to 2024-12-05 14:45:03+00:00:
0it [00:00, ?it/s]--- Online features ---
Traceback (most recent call last):
  File "/feast-init/sample/feature_repo/test_workflow.py", line 131, in <module>
    run_demo()
  File "/feast-init/sample/feature_repo/test_workflow.py", line 26, in run_demo
    fetch_online_features(store)
  File "/feast-init/sample/feature_repo/test_workflow.py", line 122, in fetch_online_features
    returned_features = store.get_online_features(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/feature_store.py", line 1696, in get_online_features
    return provider.get_online_features(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/infra/passthrough_provider.py", line 249, in get_online_features
    return self.online_store.get_online_features(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/infra/online_stores/online_store.py", line 179, in get_online_features
    ) = utils._prepare_entities_to_read_from_online_store(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/utils.py", line 1036, in _prepare_entities_to_read_from_online_store
    ) = _get_online_request_context(registry, project, features, full_feature_names)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/utils.py", line 959, in _get_online_request_context
    ) = _get_feature_views_to_use(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/utils.py", line 907, in _get_feature_views_to_use
    fv = registry.get_any_feature_view(name, project, allow_cache)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/feast/sdk/python/feast/infra/registry/remote.py", line 290, in get_any_feature_view 

Specifications

• Version: 0.42.0
• Platform: Kubernetes
• Subsystem: Feast deployed via Go Operator

Possible Solution

Move away from using remote provider for service interaction.