Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix DataFrame::with_column to handle creating column names with a period #3700

Merged
merged 3 commits into from
Oct 7, 2022
Merged

Fix DataFrame::with_column to handle creating column names with a period #3700

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
ce06029
Fix `DataFrame::with_column` to handle creating column names with a p…
alamb Oct 4, 2022
32a86dc
Merge remote-tracking branch 'apache/master' into alamb/fix_with_column
alamb Oct 6, 2022
e48b14d
Remove old comments
alamb Oct 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 45 additions & 3 deletions datafusion/core/src/dataframe.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ use crate::physical_plan::{execute_stream, execute_stream_partitioned, Execution
use crate::prelude::SessionContext;
use crate::scalar::ScalarValue;
use async_trait::async_trait;
use datafusion_common::Column;
use parking_lot::RwLock;
use parquet::file::properties::WriterProperties;
use std::any::Any;
Expand Down Expand Up @@ -672,7 +673,10 @@ impl DataFrame {
col_exists = true;
new_column.clone()
} else {
col(f.name())
Expr::Column(Column {
relation: None,
name: f.name().into(),
})
}
})
.collect();
Expand Down Expand Up @@ -827,6 +831,7 @@ mod tests {
use crate::physical_plan::ColumnarValue;
use crate::test_util;
use crate::{assert_batches_sorted_eq, execution::context::SessionContext};
use arrow::array::Int32Array;
use arrow::datatypes::DataType;
use datafusion_expr::{
avg, cast, count, count_distinct, create_udf, lit, max, min, sum,
Expand Down Expand Up @@ -1365,9 +1370,9 @@ mod tests {
ctx.register_batch("test", data)?;

let sql = r#"
SELECT
SELECT
COUNT(1)
FROM
FROM
test
GROUP BY
column_1"#;
Expand All @@ -1378,6 +1383,43 @@ mod tests {
Ok(())
}

#[tokio::test]
async fn with_column_name() -> Result<()> {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks to @hengfeiyang for this reproducer

// define data with a column name that has a "." in it:
let array: Int32Array = [1, 10].into_iter().collect();
let batch =
RecordBatch::try_from_iter(vec![("f.c1", Arc::new(array) as _)]).unwrap();

let ctx = SessionContext::new();
ctx.register_batch("t", batch).unwrap();

let df = ctx
.table("t")
.unwrap()
// try and create a column with a '.' in it
.with_column("f.c2", lit("hello"))
.unwrap();
// Note trying to select causes an error (todo file a separate ticket)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it turns it something still isn't right (select doesn't work) -- I plan to file an issue (or maybe someone else will have the time to debug it)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#3733

//.select_columns(&["f.c1", "f.c2"])
//.unwrap();

let df_results = df.collect().await.unwrap();

assert_batches_sorted_eq!(
vec![
"+------+-------+",
"| f.c1 | f.c2 |",
"+------+-------+",
"| 1 | hello |",
"| 10 | hello |",
"+------+-------+",
],
&df_results
);

Ok(())
}

#[tokio::test]
async fn cache_test() -> Result<()> {
let df = test_table()
Expand Down