Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Work Towards Supporting SQL Delete #201

Merged
merged 2 commits into from
Sep 17, 2024
Merged

Work Towards Supporting SQL Delete #201

merged 2 commits into from
Sep 17, 2024

Conversation

shehabgamin
Copy link
Contributor

@shehabgamin shehabgamin commented Sep 17, 2024 •
edited
Loading

TODO:

  1. Implement ExecutionPlan for WriteOp::Delete.
  2. Filter condition not working (unable to resolve column name).

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 17, 2024 •
edited
Loading

Gold Data Report

Notes
  1. The tables below show the number of true positives (TP), true negatives (TN), false positives (FP), and false negatives (FN) in gold data input processing.
  2. A positive input is a valid test case, while a negative input is a test case that is expected to fail.

Commit Information

Commit Revision Branch
After a8abd56 refs/pull/201/merge
Before 447e547 main

Summary

Commit TP TN FP FN Total
After 1243 189 23 774 2229
Before 1241 189 23 776 2229

Details

Gold Data Metrics
Group File Commit TP TN FP FN Total
spark data_type.json After 42 5 0 0 47
Before 42 5 0 0 47
expression/case.json After 5 0 0 0 5
Before 5 0 0 0 5
expression/cast.json After 4 0 0 0 4
Before 4 0 0 0 4
expression/current.json After 2 0 0 0 2
Before 2 0 0 0 2
expression/date.json After 4 1 0 0 5
Before 4 1 0 0 5
expression/interval.json After 340 6 0 6 352
Before 340 6 0 6 352
expression/large.json After 0 0 0 2 2
Before 0 0 0 2 2
expression/like.json After 29 0 10 0 39
Before 29 0 10 0 39
expression/misc.json After 81 4 1 26 112
Before 81 4 1 26 112
expression/numeric.json After 28 6 1 2 37
Before 28 6 1 2 37
expression/string.json After 16 1 0 2 19
Before 16 1 0 2 19
expression/timestamp.json After 7 3 0 0 10
Before 7 3 0 0 10
expression/window.json After 72 0 1 1 74
Before 72 0 1 1 74
function/agg.json After 73 0 0 61 134
Before 73 0 0 61 134
function/array.json After 30 0 0 12 42
Before 30 0 0 12 42
function/bitwise.json After 4 0 0 7 11
Before 4 0 0 7 11
function/collection.json After 8 0 0 1 9
Before 8 0 0 1 9
function/conditional.json After 10 0 0 0 10
Before 10 0 0 0 10
function/conversion.json After 1 0 0 0 1
Before 1 0 0 0 1
function/csv.json After 0 0 0 5 5
Before 0 0 0 5 5
function/datetime.json After 36 0 0 88 124
Before 36 0 0 88 124
function/generator.json After 4 0 0 9 13
Before 4 0 0 9 13
function/hash.json After 2 0 0 5 7
Before 2 0 0 5 7
function/json.json After 0 0 0 20 20
Before 0 0 0 20 20
function/lambda.json After 0 0 0 31 31
Before 0 0 0 31 31
function/map.json After 1 0 0 14 15
Before 1 0 0 14 15
function/math.json After 62 0 0 58 120
Before 62 0 0 58 120
function/misc.json After 7 0 0 41 48
Before 7 0 0 41 48
function/predicate.json After 55 0 0 14 69
Before 55 0 0 14 69
function/string.json After 79 0 0 94 173
Before 79 0 0 94 173
function/struct.json After 2 0 0 0 2
Before 2 0 0 0 2
function/url.json After 0 0 0 5 5
Before 0 0 0 5 5
function/window.json After 6 0 0 3 9
Before 6 0 0 3 9
function/xml.json After 0 0 0 9 9
Before 0 0 0 9 9
plan/ddl_alter_table.json After 0 17 0 62 79
Before 0 17 0 62 79
plan/ddl_alter_view.json After 0 1 0 5 6
Before 0 1 0 5 6
plan/ddl_analyze_table.json After 0 6 0 16 22
Before 0 6 0 16 22
plan/ddl_cache.json After 0 1 0 4 5
Before 0 1 0 4 5
plan/ddl_create_index.json After 0 0 0 3 3
Before 0 0 0 3 3
plan/ddl_create_table.json After 23 23 3 2 51
Before 23 23 3 2 51
plan/ddl_delete_from.json After 2 1 0 0 3
Before 0 1 0 2 3
plan/ddl_describe.json After 0 0 0 4 4
Before 0 0 0 4 4
plan/ddl_drop_index.json After 0 0 0 2 2
Before 0 0 0 2 2
plan/ddl_drop_view.json After 5 0 0 0 5
Before 5 0 0 0 5
plan/ddl_insert_into.json After 12 2 0 4 18
Before 12 2 0 4 18
plan/ddl_insert_overwrite.json After 5 2 0 4 11
Before 5 2 0 4 11
plan/ddl_load_data.json After 0 0 0 4 4
Before 0 0 0 4 4
plan/ddl_merge_into.json After 0 7 0 8 15
Before 0 7 0 8 15
plan/ddl_misc.json After 4 0 0 9 13
Before 4 0 0 9 13
plan/ddl_replace_table.json After 20 18 0 2 40
Before 20 18 0 2 40
plan/ddl_select.json After 1 0 0 0 1
Before 1 0 0 0 1
plan/ddl_show_views.json After 0 0 0 7 7
Before 0 0 0 7 7
plan/ddl_uncache.json After 0 0 0 2 2
Before 0 0 0 2 2
plan/ddl_update.json After 2 1 0 0 3
Before 2 1 0 0 3
plan/error_alter_table.json After 0 4 0 0 4
Before 0 4 0 0 4
plan/error_analyze_table.json After 0 1 0 0 1
Before 0 1 0 0 1
plan/error_create_table.json After 0 3 0 0 3
Before 0 3 0 0 3
plan/error_describe.json After 0 1 0 0 1
Before 0 1 0 0 1
plan/error_join.json After 0 2 0 0 2
Before 0 2 0 0 2
plan/error_load_data.json After 0 1 0 0 1
Before 0 1 0 0 1
plan/error_misc.json After 0 11 0 0 11
Before 0 11 0 0 11
plan/error_order_by.json After 1 3 0 0 4
Before 1 3 0 0 4
plan/error_select.json After 0 10 0 0 10
Before 0 10 0 0 10
plan/error_with.json After 0 1 0 0 1
Before 0 1 0 0 1
plan/plan_alter_view.json After 0 2 0 0 2
Before 0 2 0 0 2
plan/plan_create_view.json After 0 2 0 0 2
Before 0 2 0 0 2
plan/plan_explain.json After 0 1 1 0 2
Before 0 1 1 0 2
plan/plan_group_by.json After 7 1 0 3 11
Before 7 1 0 3 11
plan/plan_hint.json After 25 0 3 0 28
Before 25 0 3 0 28
plan/plan_insert_into.json After 3 0 0 0 3
Before 3 0 0 0 3
plan/plan_insert_overwrite.json After 1 0 0 1 2
Before 1 0 0 1 2
plan/plan_join.json After 39 3 0 20 62
Before 39 3 0 20 62
plan/plan_misc.json After 6 6 0 16 28
Before 6 6 0 16 28
plan/plan_order_by.json After 2 3 0 13 18
Before 2 3 0 13 18
plan/plan_select.json After 40 17 2 55 114
Before 40 17 2 55 114
plan/plan_set_operation.json After 14 0 0 3 17
Before 14 0 0 3 17
plan/plan_with.json After 3 1 0 2 6
Before 3 1 0 2 6
plan/unpivot_join.json After 4 0 0 0 4
Before 4 0 0 0 4
plan/unpivot_select.json After 7 6 0 7 20
Before 7 6 0 7 20
table_schema.json After 7 5 1 0 13
Before 7 5 1 0 13

@github-actions GitHub Actions
Copy link

github-actions bot commented Sep 17, 2024 •
edited
Loading

Spark Test Report

Commit Information

Commit Revision Branch
After a8abd56 refs/pull/201/merge
Before 447e547 refs/heads/main

Test Summary

Suite Commit Failed Passed Skipped Warnings Time (s)
doctest-column After 4 29 3 5.70
Before 4 29 3 5.68
doctest-dataframe After 51 55 1 4 7.05
Before 52 54 1 4 7.08
doctest-functions After 225 178 6 8 10.62
Before 224 179 6 8 10.69
test-connect After 436 602 126 245 100.36
Before 437 601 126 245 100.00

Test Details

Error Counts 
(-1)      716 Total
(-1)      342 Total Unique
-------- ---- ----------------------------------------------------------------------------------------------------------
           35 DocTestFailure
           27 UnsupportedOperationException: map partitions
           23 UnsupportedOperationException: co-group map
           23 UnsupportedOperationException: group map
           18 PySparkAssertionError: [DIFFERENT_PANDAS_DATAFRAME] DataFrames are not almost equal:
           15 UnsupportedOperationException: streaming query manager command
           13 UnsupportedOperationException: inline user defined window function
           13 UnsupportedOperationException: lambda function
           10 UnsupportedOperationException: unsupported data source format: Some("text")
           10 handle add artifacts
            8 AssertionError: AnalysisException not raised
            8 PythonException: 
            8 PythonException:  KeyError: 0
            8 UnsupportedOperationException: hint
            7 AnalysisException: No field named "#3". Valid fields are "#0".
            7 AssertionError: False is not true
            7 SparkRuntimeException: type_coercion
            6 IllegalArgumentException: invalid argument: UDF function type must be Python UDF
            6 UnsupportedOperationException: function in table factor
            6 UnsupportedOperationException: write stream operation start
            5 IllegalArgumentException: invalid argument: expecting either join condition or using columns
            5 UnsupportedOperationException: corr
            5 UnsupportedOperationException: function: monotonically_increasing_id
            4 AnalysisException: Error during planning: view not found: v
            4 AnalysisException: Execution error: 'Utf8("INTERVAL '0 00:00:00.000123' DAY TO SECOND") = CAST(#1 AS...
            4 PySparkNotImplementedError: [NOT_IMPLEMENTED] rdd() is not implemented.
            4 UnsupportedOperationException: cov
            4 UnsupportedOperationException: function: window
            4 UnsupportedOperationException: interval unit: day-time
            4 UnsupportedOperationException: replace
            4 UnsupportedOperationException: sample
            4 UnsupportedOperationException: sample by
            4 UnsupportedOperationException: unknown function: hll_sketch_agg
            4 UnsupportedOperationException: unpivot
            3 AnalysisException: Error during planning: Error during planning: Coercion from [Int64, Boolean] to t...
            3 AnalysisException: Error during planning: Inconsistent data type across values list at row 2 column ...
            3 AnalysisException: Error during planning: Table function 'range' not found
            3 AnalysisException: Error during planning: The expression to get an indexed field is only valid for `...
            3 AnalysisException: Execution error: Date part 'D' not supported
            3 AssertionError: "[('a', [('b', 'c')])]" != "{'a': {'b': 'c'}}"
            3 AssertionError: PythonException not raised
            3 IllegalArgumentException: expected value at line 1 column 1
            3 PythonException:  ArrowException: Invalid argument error: must either specify a row count or at leas...
            3 SparkRuntimeException: Internal error: UDF returned a different number of rows than expected. Expect...
            3 UnsupportedOperationException: crosstab
            3 UnsupportedOperationException: expression: Tuple([Value(SingleQuotedString("value")), Value(Number("...
            3 UnsupportedOperationException: function: input_file_name
            3 UnsupportedOperationException: function: pmod
            3 UnsupportedOperationException: function: to_date
            3 UnsupportedOperationException: function: ~
            3 UnsupportedOperationException: handle analyze input files
            3 UnsupportedOperationException: to schema
            3 ValueError: Converting to Python dictionary is not supported when duplicate field names are present
            2 AnalysisException: Cannot cast from struct to other types except struct
            2 AnalysisException: Cannot cast list to non-list data types
            2 AnalysisException: Cannot cast to Decimal128(14, 7). Overflowing on NaN
            2 AnalysisException: Error during planning: The expression to get an indexed field is only valid for `...
            2 AnalysisException: Error during planning: cannot resolve attribute: ObjectName([Identifier("a.b")])
            2 AnalysisException: Error during planning: two values expected: [Alias(Alias { expr: Column(Column { ...
            2 AnalysisException: Invalid or Unsupported Configuration: Could not find config namespace "spark"
            2 AssertionError
            2 AssertionError: "TABLE_OR_VIEW_NOT_FOUND" does not match "Error during planning: cannot resolve attr...
            2 AssertionError: Lists differ: [Row([15 chars]lue=1.0)] != [Row([15 chars]lue=1), Row(key='count', va...
            2 AssertionError: Lists differ: [Row([22 chars](key=1, value='1'), Row(key=2, value='2'), Row[2402 cha...
            2 AssertionError: True is not false
            2 IllegalArgumentException: invalid argument: empty data source paths
            2 IllegalArgumentException: invalid argument: sql parser error: Expected: ), found: id at Line: 1, Col...
            2 IllegalArgumentException: invalid argument: sql parser error: Expected: ), found: id at Line: 1, Col...
            2 IllegalArgumentException: invalid argument: sql parser error: Expected: ), found: id at Line: 3, Col...
            2 IllegalArgumentException: invalid argument: sql parser error: Expected: ), found: id at Line: 5, Col...
            2 IllegalArgumentException: invalid argument: sql parser error: Expected: ), found: t at Line: 3, Colu...
            2 IllegalArgumentException: invalid argument: sql parser error: Expected: ), found: v at Line: 1, Colu...
            2 SparkRuntimeException: Internal error: start_from index out of bounds.
            2 UnsupportedOperationException: Only literal expr are supported in Python UDTFs for now, got expr: ma...
            2 UnsupportedOperationException: Only literal expr are supported in Python UDTFs for now, got expr: ma...
            2 UnsupportedOperationException: Only literal expr are supported in Python UDTFs for now, got expr: na...
            2 UnsupportedOperationException: Only literal expr are supported in Python UDTFs for now, got expr: ra...
            2 UnsupportedOperationException: approx quantile
            2 UnsupportedOperationException: collect metrics
            2 UnsupportedOperationException: decimal literal with precision or scale
            2 UnsupportedOperationException: describe
            2 UnsupportedOperationException: freq items
            2 UnsupportedOperationException: function: bitmap_bit_position
            2 UnsupportedOperationException: function: crc32
            2 UnsupportedOperationException: function: dayofweek
            2 UnsupportedOperationException: function: encode
            2 UnsupportedOperationException: function: format_number
            2 UnsupportedOperationException: function: from_csv
            2 UnsupportedOperationException: function: from_json
            2 UnsupportedOperationException: function: inline
            2 UnsupportedOperationException: function: least
            2 UnsupportedOperationException: function: map_from_arrays
            2 UnsupportedOperationException: function: map_keys
            2 UnsupportedOperationException: function: overlay
            2 UnsupportedOperationException: function: sec
            2 UnsupportedOperationException: function: shiftrightunsigned
            2 UnsupportedOperationException: function: timestamp_seconds
            2 UnsupportedOperationException: function: xxhash64
            2 UnsupportedOperationException: handle analyze same semantics
            2 UnsupportedOperationException: tail
            2 UnsupportedOperationException: unknown function: approx_count_distinct
            2 UnsupportedOperationException: unknown function: collect_set
            2 UnsupportedOperationException: unresolved regex
            2 UnsupportedOperationException: unsupported data source format: Some("orc")
            2 UnsupportedOperationException: user defined data type should only exist in a field
            2 handle artifact statuses
            1 AnalysisException: Cannot cast string 'abc' to value of Float64 type
            1 AnalysisException: Cannot cast value 'abc' to value of Boolean type
            1 AnalysisException: Error during planning: Error during planning: Coercion from [Int64, Boolean] to t...
            1 AnalysisException: Error during planning: Error during planning: Coercion from [Utf8, Boolean] to th...
            1 AnalysisException: Error during planning: Error during planning: Coercion from [Utf8, Boolean] to th...
            1 AnalysisException: Error during planning: Inconsistent data type across values list at row 1 column ...
            1 AnalysisException: Error during planning: Inconsistent data type across values list at row 2 column ...
            1 AnalysisException: Error during planning: Invalid qualifier b
            1 AnalysisException: Error during planning: No function matches the given name and argument types 'NTH...
            1 AnalysisException: Error during planning: The array_concat function can only accept list as the args...
            1 AnalysisException: Error during planning: cannot resolve attribute: ObjectName([Identifier("a.e")])
            1 AnalysisException: Error during planning: cannot resolve attribute: ObjectName([Identifier("df_as1.n...
            1 AnalysisException: Error during planning: three values expected: [Alias(Alias { expr: Column(Column ...
            1 AnalysisException: Error during planning: three values expected: [Literal(Int32(1)), Literal(Int32(3...
            1 AnalysisException: Error during planning: two values expected: [Alias(Alias { expr: Column(Column { ...
            1 AnalysisException: Error during planning: view not found: tab2
            1 AnalysisException: Execution error: 'Utf8("1970-01-01 00:00:00") = CAST(#1 AS Utf8) AS dt AS (1970-0...
            1 AnalysisException: Execution error: 'Utf8("2012-02-02 02:02:02") = CAST(?table?.#0 AS Utf8) AS a AS ...
            1 AnalysisException: Execution error: 'Utf8("INTERVAL '0 00:00:00.000123' DAY TO SECOND") = CAST(#1 AS...
            1 AnalysisException: Execution error: Error parsing timestamp from '1997-02-28 10:30:00' using format ...
            1 AnalysisException: Execution error: Error parsing timestamp from '2015-04-08' using format 'yyyy-MM-...
            1 AnalysisException: Execution error: Error parsing timestamp from '2023-01-01' using format 'dd-MM-yy...
            1 AnalysisException: Execution error: The UPPER function can only accept strings, but got Int64.
            1 AnalysisException: Execution error: Unable to find factory for TEXT
            1 AnalysisException: Execution error: array_reverse does not support type 'Utf8'.
            1 AnalysisException: Invalid or Unsupported Configuration: could not find config namespace for key "ig...
            1 AnalysisException: Invalid or Unsupported Configuration: could not find config namespace for key "li...
            1 AnalysisException: No field named "#3". Valid fields are "#0", "v % Int32(2)".
            1 AnalysisException: No field named "#3". Valid fields are "v % Int32(2)".
            1 AnalysisException: No field named "?table?"."#1". Valid fields are "?table?"."#0".
            1 AnalysisException: No field named tbl."#2". Valid fields are tbl."#3".
            1 AnalysisException: Schema contains duplicate unqualified field name "#0"
            1 AssertionError: "2000000" does not match "Internal error: raise_error expects a single UTF-8 string ...
(+1)        1 AssertionError: "Database 'memory:b84973d9-cb64-42cf-8f2a-08827659584d' dropped." does not match "in...
(+1)        1 AssertionError: "Database 'memory:f14dcb4b-7cc6-407d-9e93-68989464fc8f' dropped." does not match "in...
            1 AssertionError: "Exception thrown when converting pandas.Series" does not match "
            1 AssertionError: "Exception thrown when converting pandas.Series" does not match "expected value at l...
            1 AssertionError: "PickleException" does not match "
            1 AssertionError: "UDTF_ARROW_TYPE_CAST_ERROR" does not match "
            1 AssertionError: "[('a', 'b')]" != "{'a': 'b'}"
            1 AssertionError: "aggregate function.*argument.*aggregate function" does not match "No field named "#...
            1 AssertionError: "foobar" does not match "Internal error: raise_error expects a single UTF-8 string a...
            1 AssertionError: "timestamp values are not equal (timestamp='1969-01-01 09:01:01+08:00': data[0][1]='...
            1 AssertionError: '+---[17 chars]-----+\n|                        x|\n+--------[132 chars]-+\n' != '+-...
            1 AssertionError: 2 != 3
            1 AssertionError: AnalysisException not raised by <lambda>
            1 AssertionError: BinaryType() != NullType()
            1 AssertionError: DataFrame.iloc[:, 0] (column name="struct") are different
            1 AssertionError: Exception not raised
            1 AssertionError: Items in the second set but not the first:
            1 AssertionError: Lists differ: [Row([22 chars]e(2018, 12, 31, 16, 0), aware=datetime.datetim[16 chars...
            1 AssertionError: Lists differ: [Row([49 chars] 1), internal_value=-31532339000000000), Row(i[225 char...
            1 AssertionError: Lists differ: [Row(_c0='Joe', _c1=20, _c2='Hi,\nI am Jeo'), Row(_c0='Tom', [90 chars...
            1 AssertionError: Lists differ: [Row(ln(id)=-inf, ln(id)=-inf, struct(id, name)=Row(i[1232 chars]9'))]...
            1 AssertionError: Lists differ: [Row(name='Michael', age=None), Row(name='Andy[34 chars]=19)] != [Row(...
            1 AssertionError: Row(point='[1.0, 2.0]', pypoint='[3.0, 4.0]') != Row(point='(1.0, 2.0)', pypoint='[3...
            1 AssertionError: Row(res="[('personal', [('name', 'John'), ('city', 'New York')])]") != Row(res="{'pe...
            1 AssertionError: StorageLevel(False, True, True, False, 1) != StorageLevel(False, False, False, False...
            1 AssertionError: Struc[31 chars]stampNTZType(), True), StructField('val', Inte[13 chars]ue)]) != Stru...
            1 AssertionError: Struc[32 chars]e(), False), StructField('b', DoubleType(), Fa[158 chars]ue)]) != Str...
            1 AssertionError: Struc[40 chars]ue), StructField('val', ArrayType(DoubleType(), False), True)]) != St...
            1 AssertionError: YearMonthIntervalType(0, 1) != YearMonthIntervalType(0, 0)
            1 AssertionError: [1.0, 2.0] != ExamplePoint(1.0,2.0)
            1 AssertionError: {} != {'max_age': 5}
            1 AttributeError: 'DataFrame' object has no attribute '_ipython_key_completions_'
            1 AttributeError: 'DataFrame' object has no attribute '_joinAsOf'
(+1)        1 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpcnjpci18'
(+1)        1 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpxg170qu8'
            1 IllegalArgumentException: 83140 is too large to store in a Decimal128 of precision 4. Max is 9999
            1 IllegalArgumentException: invalid argument: hex requires a literal argument
            1 IllegalArgumentException: invalid argument: sql parser error: Expected: (, found: AS at Line: 1, Col...
            1 IllegalArgumentException: invalid argument: sql parser error: Expected: (, found: EOF
            1 KeyError: 'max'
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] foreach() is not implemented.
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] foreachPartition() is not implemented.
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] localCheckpoint() is not implemented.
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] sparkContext() is not implemented.
            1 PySparkNotImplementedError: [NOT_IMPLEMENTED] toJSON() is not implemented.
            1 PythonException:  ArrowTypeError: ("Expected dict key of type str or bytes, got 'int'", 'Conversion ...
            1 PythonException:  AttributeError: 'NoneType' object has no attribute 'partitionId'
            1 PythonException:  AttributeError: 'list' object has no attribute 'x'
            1 PythonException:  AttributeError: 'list' object has no attribute 'y'
            1 PythonException:  TypeError: 'NoneType' object is not subscriptable
            1 QueryExecutionException: Json error: Not valid JSON: EOF while parsing a list at line 1 column 1
            1 QueryExecutionException: Json error: Not valid JSON: expected value at line 1 column 2
            1 SparkRuntimeException: External error: This feature is not implemented: physical plan is not yet imp...
            1 SparkRuntimeException: Internal error: Failed to decode from base64: Invalid padding.
            1 SparkRuntimeException: Optimizer rule 'optimize_projections' failed
            1 UnsupportedOperationException: Insert into not implemented for this table
            1 UnsupportedOperationException: Physical plan does not support logical expression Sort(Sort { expr: C...
            1 UnsupportedOperationException: PlanNode::IsCached
            1 UnsupportedOperationException: SQL show functions
            1 UnsupportedOperationException: Unsupported statement: SHOW DATABASES
            1 UnsupportedOperationException: bucketing
            1 UnsupportedOperationException: call function
            1 UnsupportedOperationException: deduplicate within watermark
            1 UnsupportedOperationException: function exists
            1 UnsupportedOperationException: function: aes_decrypt
            1 UnsupportedOperationException: function: aes_encrypt
            1 UnsupportedOperationException: function: array_insert
            1 UnsupportedOperationException: function: array_sort
            1 UnsupportedOperationException: function: arrays_zip
            1 UnsupportedOperationException: function: bin
            1 UnsupportedOperationException: function: bit_count
            1 UnsupportedOperationException: function: bit_get
            1 UnsupportedOperationException: function: bitmap_bucket_number
            1 UnsupportedOperationException: function: bitmap_count
            1 UnsupportedOperationException: function: bround
            1 UnsupportedOperationException: function: conv
            1 UnsupportedOperationException: function: convert_timezone
            1 UnsupportedOperationException: function: csc
            1 UnsupportedOperationException: function: current_catalog
            1 UnsupportedOperationException: function: current_database
            1 UnsupportedOperationException: function: current_schema
            1 UnsupportedOperationException: function: current_timezone
            1 UnsupportedOperationException: function: date_from_unix_date
            1 UnsupportedOperationException: function: dayofmonth
            1 UnsupportedOperationException: function: dayofyear
            1 UnsupportedOperationException: function: decode
            1 UnsupportedOperationException: function: e
            1 UnsupportedOperationException: function: element_at
            1 UnsupportedOperationException: function: elt
            1 UnsupportedOperationException: function: format_string
            1 UnsupportedOperationException: function: from_unixtime
            1 UnsupportedOperationException: function: from_utc_timestamp
            1 UnsupportedOperationException: function: get_json_object
            1 UnsupportedOperationException: function: getbit
            1 UnsupportedOperationException: function: greatest
            1 UnsupportedOperationException: function: hash
            1 UnsupportedOperationException: function: hour
            1 UnsupportedOperationException: function: inline_outer
            1 UnsupportedOperationException: function: java_method
            1 UnsupportedOperationException: function: json_array_length
            1 UnsupportedOperationException: function: json_object_keys
            1 UnsupportedOperationException: function: json_tuple
            1 UnsupportedOperationException: function: last_day
            1 UnsupportedOperationException: function: localtimestamp
            1 UnsupportedOperationException: function: locate
            1 UnsupportedOperationException: function: make_dt_interval
            1 UnsupportedOperationException: function: make_interval
            1 UnsupportedOperationException: function: make_timestamp
            1 UnsupportedOperationException: function: make_timestamp_ltz
            1 UnsupportedOperationException: function: make_timestamp_ntz
            1 UnsupportedOperationException: function: make_ym_interval
            1 UnsupportedOperationException: function: map_concat
            1 UnsupportedOperationException: function: map_entries
            1 UnsupportedOperationException: function: map_from_entries
            1 UnsupportedOperationException: function: map_values
            1 UnsupportedOperationException: function: mask
            1 UnsupportedOperationException: function: minute
            1 UnsupportedOperationException: function: months_between
            1 UnsupportedOperationException: function: next_day
            1 UnsupportedOperationException: function: parse_url
            1 UnsupportedOperationException: function: position
            1 UnsupportedOperationException: function: printf
            1 UnsupportedOperationException: function: quarter
            1 UnsupportedOperationException: function: reflect
            1 UnsupportedOperationException: function: regexp_count
            1 UnsupportedOperationException: function: regexp_extract
            1 UnsupportedOperationException: function: regexp_extract_all
            1 UnsupportedOperationException: function: regexp_instr
            1 UnsupportedOperationException: function: regexp_substr
            1 UnsupportedOperationException: function: replace
            1 UnsupportedOperationException: function: schema_of_csv
            1 UnsupportedOperationException: function: schema_of_json
            1 UnsupportedOperationException: function: second
            1 UnsupportedOperationException: function: sentences
            1 UnsupportedOperationException: function: session_window
            1 UnsupportedOperationException: function: sha
            1 UnsupportedOperationException: function: sha1
            1 UnsupportedOperationException: function: soundex
            1 UnsupportedOperationException: function: spark_partition_id
            1 UnsupportedOperationException: function: split
            1 UnsupportedOperationException: function: split_part
            1 UnsupportedOperationException: function: stack
            1 UnsupportedOperationException: function: str_to_map
            1 UnsupportedOperationException: function: timestamp_micros
            1 UnsupportedOperationException: function: timestamp_millis
            1 UnsupportedOperationException: function: to_char
            1 UnsupportedOperationException: function: to_csv
            1 UnsupportedOperationException: function: to_json
            1 UnsupportedOperationException: function: to_number
            1 UnsupportedOperationException: function: to_unix_timestamp
            1 UnsupportedOperationException: function: to_utc_timestamp
            1 UnsupportedOperationException: function: to_varchar
            1 UnsupportedOperationException: function: try_add
            1 UnsupportedOperationException: function: try_aes_decrypt
            1 UnsupportedOperationException: function: try_divide
            1 UnsupportedOperationException: function: try_element_at
            1 UnsupportedOperationException: function: try_multiply
            1 UnsupportedOperationException: function: try_subtract
            1 UnsupportedOperationException: function: try_to_binary
            1 UnsupportedOperationException: function: try_to_number
            1 UnsupportedOperationException: function: try_to_timestamp
            1 UnsupportedOperationException: function: typeof
            1 UnsupportedOperationException: function: unhex
            1 UnsupportedOperationException: function: unix_micros
            1 UnsupportedOperationException: function: unix_millis
            1 UnsupportedOperationException: function: unix_seconds
            1 UnsupportedOperationException: function: url_decode
            1 UnsupportedOperationException: function: url_encode
            1 UnsupportedOperationException: function: weekday
            1 UnsupportedOperationException: function: weekofyear
            1 UnsupportedOperationException: function: width_bucket
            1 UnsupportedOperationException: function: xpath
            1 UnsupportedOperationException: function: xpath_boolean
            1 UnsupportedOperationException: function: xpath_double
            1 UnsupportedOperationException: function: xpath_float
            1 UnsupportedOperationException: function: xpath_int
            1 UnsupportedOperationException: function: xpath_long
            1 UnsupportedOperationException: function: xpath_number
            1 UnsupportedOperationException: function: xpath_short
            1 UnsupportedOperationException: function: xpath_string
            1 UnsupportedOperationException: handle analyze is local
            1 UnsupportedOperationException: handle analyze semantic hash
            1 UnsupportedOperationException: list functions
            1 UnsupportedOperationException: summary
            1 UnsupportedOperationException: unknown function: any_value
            1 UnsupportedOperationException: unknown function: approx_percentile
            1 UnsupportedOperationException: unknown function: bitmap_or_agg
            1 UnsupportedOperationException: unknown function: count_if
            1 UnsupportedOperationException: unknown function: count_min_sketch
            1 UnsupportedOperationException: unknown function: distributed_sequence_id
            1 UnsupportedOperationException: unknown function: grouping_id
            1 UnsupportedOperationException: unknown function: histogram_numeric
            1 UnsupportedOperationException: unknown function: kurtosis
            1 UnsupportedOperationException: unknown function: max_by
            1 UnsupportedOperationException: unknown function: min_by
            1 UnsupportedOperationException: unknown function: mode
            1 UnsupportedOperationException: unknown function: percentile
            1 UnsupportedOperationException: unknown function: percentile_approx
            1 UnsupportedOperationException: unknown function: product
            1 UnsupportedOperationException: unknown function: skewness
            1 UnsupportedOperationException: unknown function: try_avg
            1 UnsupportedOperationException: unknown function: try_sum
            1 internal error: unknown attribute in plan 229: a
(-1)        0 AssertionError: "Database 'memory:7c29f551-3fc5-4289-b705-faf0458de712' dropped." does not match "in...
(-1)        0 AssertionError: "Database 'memory:ad288d62-029f-42f8-b8d8-549786220b3d' dropped." does not match "in...
(-1)        0 AssertionError: Lists differ: [Row(t=3), Row(t=4)] != [Row(t=4), Row(t=3)]
(-1)        0 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp012kg3a2'
(-1)        0 FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpq2ti7bkx'
Passed Tests Diff 
--- before	2024-09-17 09:01:21.771529432 +0000
+++ after	2024-09-17 09:01:21.903530684 +0000
@@ -77,0 +78 @@
+.venvs/test/lib/python3.11/site-packages/pyspark/sql/dataframe.py::pyspark.sql.dataframe.DataFrame.withColumns
@@ -196 +196,0 @@
-.venvs/test/lib/python3.11/site-packages/pyspark/sql/functions.py::pyspark.sql.functions.ntile
@@ -519,0 +520 @@
+.venvs/test/lib/python3.11/site-packages/pyspark/sql/tests/connect/test_parity_arrow_python_udf.py::UDFParityTests::test_udf_with_aggregate_function

@shehabgamin shehabgamin changed the title Support SQL Delete Work Towards Supporting SQL Delete Sep 17, 2024
@shehabgamin shehabgamin marked this pull request as ready for review September 17, 2024 08:53
@shehabgamin shehabgamin merged commit 9b4a524 into main Sep 17, 2024
8 checks passed
@shehabgamin shehabgamin deleted the support-sql-delete branch September 17, 2024 09:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@shehabgamin