Add DatabricksWorkflowTaskGroup

[pankajkoti]

This pull request introduces the DatabricksWorkflowTaskGroup
to the Airflow Databricks provider from the astro-provider-databricks
repository.
It marks another pull request aimed at contributing
operators and features from that repository into the Airflow
Databricks provider, the previous PR being #39178.

The task group launches a Databricks Workflow
and runs the notebook jobs from within it, resulting in a
75% cost reduction ($0.40/DBU for all-purpose compute,
$0.07/DBU for Jobs compute) when compared to executing
DatabricksNotebookOperator outside of DatabricksWorkflowTaskGroup.

There are a few advantages to defining your Databricks Workflows in Airflow:

	via Databricks	via Airflow
Authoring interface	Web-based via Databricks UI	Code via Airflow DAG
Workflow compute pricing	✅	✅
Notebook code in source control	✅	✅
Workflow structure in source control		✅
Retry from beginning	✅	✅
Retry single task		✅
Task groups within Workflows		✅
Trigger workflows from other DAGs		✅
Workflow-level parameters		✅

Below screenshots depict successful Airflow DAG runs and corresponding
successful Databricks job run

Airflow DAG view

Datarbricks job Graph view

Co-authored by: @dimberman @tatiana in the original repo.

Co-authored-by: Daniel Imberman [email protected]
Co-authored-by: Tatiana Al-Chueyr [email protected]

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[tatiana]

It's great to see how far you've advanced on this, @pankajkoti ! You're reimplementing the original feature in a better way. Thank you!

[tatiana]

@pankajkoti I left some minor comments, but it looks great. It's impressive how you were able to keep the original interfaces, while being consistent with the Airflow Databricks provider. Thank you!

[Lee-W]

Looks good! Left some minor suggestions, most of them are just annotation improvements

[pankajkoti]

@tatiana @Lee-W @phanikumv I have addressed all the review comments so far. Would appreciate another review please.

[Lee-W]

Thanks @pankajkoti ! overall it looks great! left one suggestion but we probably can do it in the next PR

[tatiana]

Hi @pankajkoti , thanks for addressing all the feedback, you made it better than the original implementation!

Some minor feedback that can be addressed in follow-up PRs:

1. When using DatabricksNotebookOperator from DatabricksWorkflowTaskGroup, the Databricks Workflow Job Tasks should not be prefixed with the DAG & TaskGroup. Using the example shown in the PR description:
   • The task (notebook identifier) is named workflow_notebook_1 in the Airflow Graph view
   • The task is currently named example_Databricks_workflow_test_workflow_root_1234_run_workflow_notebook_1 in the Databricks Graph view - unnecessarily since tasks are not displayed as jobs in the Databricks job list. The current name in Databricks makes it hard to understand how different notebooks relate to each other in the Databricks UI.
2. To have a clear bullet list or table showing how the concepts map between Airflow & Databricks:
   • Airflow DAG - Databricks Workflow Job name prefix
   • Airflow TaskGroup - Databricks Workflow Job
   • Airflow Notebook task - Databricks Workflow Job Task
3. Move a log line outside of the loop since it does not give meaningful data to be printed for each iteration of the for loop: Add DatabricksWorkflowTaskGroup #39771 (comment)
4. Switch Using token auth log from INFO to DEBUG (unrelated to the current PR, but it improves the log messages)
5. Refactor other parts of the Databricks provider to use the just introduced Enum RunLifeCycleState