Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Minor: Avoid cloning as many Ident during SQL planning #4534

Merged
merged 3 commits into from
Dec 12, 2022
Merged

Minor: Avoid cloning as many Ident during SQL planning #4534

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
895b4cd
Minor: Avoid cloning some Identifiers during planning
alamb Dec 6, 2022
fbd2cab
Merge remote-tracking branch 'apache/master' into alamb/remove_normal…
alamb Dec 11, 2022
0c0a2fc
Remove another copy
alamb Dec 11, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 7 additions & 3 deletions datafusion/expr/src/logical_plan/plan.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1195,13 +1195,17 @@ pub struct SubqueryAlias {
}

impl SubqueryAlias {
pub fn try_new(plan: LogicalPlan, alias: &str) -> datafusion_common::Result<Self> {
pub fn try_new(
plan: LogicalPlan,
alias: impl Into<String>,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change allows SubqueryAlias::try_new to take a String if the caller has one or a &str that will be copied into a new String if needed

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice implementation👍, copy just when call &str.

) -> datafusion_common::Result<Self> {
let alias = alias.into();
let schema: Schema = plan.schema().as_ref().clone().into();
let schema =
DFSchemaRef::new(DFSchema::try_from_qualified_schema(alias, &schema)?);
DFSchemaRef::new(DFSchema::try_from_qualified_schema(&alias, &schema)?);
Ok(SubqueryAlias {
input: Arc::new(plan),
alias: alias.to_string(),
alias,
schema,
})
}
Expand Down
36 changes: 17 additions & 19 deletions datafusion/sql/src/planner.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -72,9 +72,7 @@ use datafusion_expr::{
};

use crate::parser::{CreateExternalTable, DescribeTable, Statement as DFStatement};
use crate::utils::{
make_decimal_type, normalize_ident, normalize_ident_owned, resolve_columns,
};
use crate::utils::{make_decimal_type, normalize_ident, resolve_columns};

use super::{
parser::DFParser,
Expand Down Expand Up @@ -296,7 +294,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
..
} if with_options.is_empty() => {
let mut plan = self.query_to_plan(*query, &mut PlannerContext::new())?;
plan = self.apply_expr_alias(plan, &columns)?;
plan = self.apply_expr_alias(plan, columns)?;

Ok(LogicalPlan::CreateView(CreateView {
name: object_name_to_table_reference(name)?,
Expand Down Expand Up @@ -484,7 +482,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {

for cte in with.cte_tables {
// A `WITH` block can't use the same name more than once
let cte_name = normalize_ident(&cte.alias.name);
let cte_name = normalize_ident(cte.alias.name.clone());
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

previously normalize_indent always cloned. Now it only clones in a few places and most of the time can reuse the String in the sqlparser-ast directly

if planner_context.ctes.contains_key(&cte_name) {
return Err(DataFusionError::SQL(ParserError(format!(
"WITH query name {:?} specified more than once",
Expand Down Expand Up @@ -689,7 +687,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
.iter()
.any(|x| x.option == ColumnOption::Null);
fields.push(Field::new(
&normalize_ident(&column.name),
&normalize_ident(column.name),
Copy link
Contributor Author

@alamb alamb Dec 7, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is unfortunate to simply drop the String immediately, but Field::new requires a &str (it can't take the String). Filed upstream: https://github.com/apache/arrow-rs/pull/3288/files

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, Look like we can do improvement like above SubqueryAlias::try_new().

data_type,
allow_null,
));
Expand Down Expand Up @@ -894,7 +892,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
JoinConstraint::Using(idents) => {
let keys: Vec<Column> = idents
.into_iter()
.map(|x| Column::from_name(normalize_ident(&x)))
.map(|x| Column::from_name(normalize_ident(x)))
.collect();
LogicalPlanBuilder::from(left)
.join_using(&right, join_type, keys)?
Expand Down Expand Up @@ -977,16 +975,16 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
) -> Result<LogicalPlan> {
let apply_name_plan = LogicalPlan::SubqueryAlias(SubqueryAlias::try_new(
plan,
&normalize_ident(&alias.name),
normalize_ident(alias.name),
)?);

self.apply_expr_alias(apply_name_plan, &alias.columns)
self.apply_expr_alias(apply_name_plan, alias.columns)
}

fn apply_expr_alias(
&self,
plan: LogicalPlan,
idents: &Vec<Ident>,
idents: Vec<Ident>,
) -> Result<LogicalPlan> {
if idents.is_empty() {
Ok(plan)
Expand All @@ -999,7 +997,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
} else {
let fields = plan.schema().fields().clone();
LogicalPlanBuilder::from(plan)
.project(fields.iter().zip(idents.iter()).map(|(field, ident)| {
.project(fields.iter().zip(idents.into_iter()).map(|(field, ident)| {
col(field.name()).alias(normalize_ident(ident))
}))?
.build()
Expand Down Expand Up @@ -1618,7 +1616,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
from_schema,
&[select_expr.clone()],
)?;
let expr = Alias(Box::new(select_expr), normalize_ident(&alias));
let expr = Alias(Box::new(select_expr), normalize_ident(alias));
Ok(vec![normalize_col(expr, plan)?])
}
SelectItem::Wildcard => {
Expand Down Expand Up @@ -1946,13 +1944,13 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {

Ok(Expr::Column(Column {
relation: None,
name: normalize_ident(&id),
name: normalize_ident(id),
}))
}
}

SQLExpr::MapAccess { ref column, keys } => {
if let SQLExpr::Identifier(ref id) = column.as_ref() {
SQLExpr::MapAccess { column, keys } => {
if let SQLExpr::Identifier(id) = *column {
plan_indexed(col(&normalize_ident(id)), keys)
} else {
Err(DataFusionError::NotImplemented(format!(
Expand All @@ -1969,7 +1967,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {

SQLExpr::CompoundIdentifier(ids) => {
if ids[0].value.starts_with('@') {
let var_names: Vec<_> = ids.into_iter().map(|s| normalize_ident(&s)).collect();
let var_names: Vec<_> = ids.into_iter().map(normalize_ident).collect();
let ty = self
.schema_provider
.get_variable_type(&var_names)
Expand Down Expand Up @@ -2294,7 +2292,7 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> {
// (e.g. "foo.bar") for function names yet
function.name.to_string()
} else {
normalize_ident(&function.name.0[0])
normalize_ident(function.name.0[0].clone())
};

// first, check SQL reserved words
Expand Down Expand Up @@ -3060,7 +3058,7 @@ fn idents_to_table_reference(idents: Vec<Ident>) -> Result<OwnedTableReference>
impl IdentTaker {
fn take(&mut self) -> String {
let ident = self.0.pop().expect("no more identifiers");
normalize_ident_owned(ident)
normalize_ident(ident)
}
}

Expand Down Expand Up @@ -3103,7 +3101,7 @@ pub fn object_name_to_qualifier(sql_table_name: &ObjectName) -> String {
.rev()
.zip(columns)
.map(|(ident, column_name)| {
format!(r#"{} = '{}'"#, column_name, normalize_ident(ident))
format!(r#"{} = '{}'"#, column_name, normalize_ident(ident.clone()))
})
.collect::<Vec<_>>()
.join(" AND ")
Expand Down
10 changes: 1 addition & 9 deletions datafusion/sql/src/utils.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -539,16 +539,8 @@ pub(crate) fn make_decimal_type(
}
}

// Normalize an identifier to a lowercase string unless the identifier is quoted.
pub(crate) fn normalize_ident(id: &Ident) -> String {
match id.quote_style {
Some(_) => id.value.clone(),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here the value is always cloned which is not necessary for most uses when we already have an owned string

None => id.value.to_ascii_lowercase(),
}
}

// Normalize an owned identifier to a lowercase string unless the identifier is quoted.
pub(crate) fn normalize_ident_owned(id: Ident) -> String {
pub(crate) fn normalize_ident(id: Ident) -> String {
match id.quote_style {
Some(_) => id.value,
None => id.value.to_ascii_lowercase(),
Expand Down