-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Add Spark get_struct_field function #12166
base: main
Are you sure you want to change the base?
feat: Add Spark get_struct_field function #12166
Conversation
✅ Deploy Preview for meta-velox canceled.
|
eeebd22
to
713b61d
Compare
|
||
namespace facebook::velox::functions::sparksql { | ||
|
||
class GetStructFieldExpr : public SpecialForm { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe you need to extend class FunctionCallToSpecialForm
rather than SpecialForm
, so as the implementation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think there are different, one is for function register, the other is for function iteself
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm wondering if we can implement the get_struct_field as a vector function then register it as a special form. What's the specific consideration here to implement it by extending SpecialForm
?
@jinchengchenghh @rui-mo Comments addressed. Please help review this again when you are convenient |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks. Please also add documentation for this function.
VectorPtr ordinalVector; | ||
inputs_[0]->eval(rows, context, input); | ||
inputs_[1]->eval(rows, context, ordinalVector); | ||
VELOX_USER_CHECK(ordinalVector->isConstantEncoding()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I suppose we can move the checks inside constructSpecialForm
and extract the constant value there. Some example in decimal round:
velox/velox/functions/sparksql/specialforms/DecimalRound.cpp
Lines 229 to 245 in 04bfdff
VELOX_USER_CHECK_EQ( | |
args[1]->type()->kind(), | |
TypeKind::INTEGER, | |
"The second argument of decimal_round should be of integer type."); | |
auto constantExpr = std::dynamic_pointer_cast<exec::ConstantExpr>(args[1]); | |
VELOX_USER_CHECK_NOT_NULL( | |
constantExpr, | |
"The second argument of decimal_round should be constant expression."); | |
VELOX_USER_CHECK( | |
constantExpr->value()->isConstantEncoding(), | |
"The second argument of decimal_round should be wrapped in constant vector."); | |
auto constantVector = | |
constantExpr->value()->asUnchecked<ConstantVector<int32_t>>(); | |
VELOX_USER_CHECK( | |
!constantVector->isNullAt(0), | |
"The second argument of decimal_round is non-nullable."); | |
scale = constantVector->valueAt(0); |
|
||
namespace facebook::velox::functions::sparksql { | ||
|
||
class GetStructFieldExpr : public SpecialForm { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm wondering if we can implement the get_struct_field as a vector function then register it as a special form. What's the specific consideration here to implement it by extending SpecialForm
?
No description provided.