Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking β€œSign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: support various reductions in pyspark #1870

Merged
merged 9 commits into from
Jan 27, 2025
Prev Previous commit
split func name in maybe_evaluate as well
  • Loading branch information
FBruzzesi committed Jan 27, 2025
commit a93c0fa714844d9b65c1cd3b1d037c5f98d69c85
6 changes: 5 additions & 1 deletion narwhals/_spark_like/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,11 @@ def maybe_evaluate(df: SparkLikeLazyFrame, obj: Any, *, returns_scalar: bool) ->
msg = "Multi-output expressions (e.g. `nw.all()` or `nw.col('a', 'b')`) not supported in this context"
raise NotImplementedError(msg)
column_result = column_results[0]
if obj._returns_scalar and obj._function_name != "lit" and not returns_scalar:
if (
obj._returns_scalar
and obj._function_name.split("->", maxsplit=1)[0] != "lit"
and not returns_scalar
):
# Returns scalar, but overall expression doesn't.
# Let PySpark do its broadcasting
return column_result.over(Window.partitionBy(F.lit(1)))
Expand Down
Loading