Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show code problems found by workflow linter in the migration dashboard #1741

Merged
merged 16 commits into from
May 22, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ See [contributing instructions](CONTRIBUTING.md) to help improve this project.
* [Dependency CLI commands](#dependency-cli-commands)
* [Table Migration Workflow Tasks](#table-migration-workflow-tasks)
* [Other considerations](#other-considerations)
* [Jobs Static Code Analysis Workflow](#jobs-static-code-analysis-workflow)
* [Utility commands](#utility-commands)
* [`logs` command](#logs-command)
* [`ensure-assessment-run` command](#ensure-assessment-run-command)
Expand Down Expand Up @@ -469,6 +470,17 @@ There are 3 main table migration workflows, targeting different table types. All

[[back to top](#databricks-labs-ucx)]

## Jobs Static Code Analysis Workflow

> Please note that this is an experimental workflow.

The `experimental-workflow-linter` workflow lints accessible code belonging to all workflows/jobs present in the
workspace. The linting emits problems indicating what to resolve for making the code Unity Catalog compatible.

![code compatibility problems](docs/code_compatibility_problems.png)

[[back to top](#databricks-labs-ucx)]

# Utility commands

## `logs` command
Expand Down
Binary file added docs/code_compatibility_problems.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
-- widget title=Workflow migration problems, row=2, col=0, size_x=2, size_y=8

## 2 - Code compatibility problems

The table on the right assist with verifying if workflows are Unity Catalog compatible. It can be filtered on the path,
problem code and workflow name. The table:
- Points to a problem detected in the code using the code path, workflow & task reference and start/end line & column;
- Explains the problem with a human-readable message and a code.

The code compatibility problems are updated after running
[Jobs Static Code Analysis Workflow](https://github.com/databrickslabs/ucx/blob/main/README.md#workflow-linter-workflow).
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
-- viz type=table, name=Workflow migration problems, search_by=path,code,job_name, columns=path,code,message,workflow_id,workflow_name,task_key,start_line,start_col,end_line,end_col
-- widget title=Workflow migration problems, row=2, col=2, size_x=4, size_y=8
SELECT
path,
code,
message,
job_id AS workflow_id,
job_name AS workflow_name,
task_key,
start_line,
start_col,
end_line,
end_col
FROM $inventory.workflow_problems
5 changes: 5 additions & 0 deletions src/databricks/labs/ucx/source_code/workflows.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,3 +11,8 @@ def lint_all_workflows(self, ctx: RuntimeContext):
"""[EXPERIMENTAL] Analyses all jobs for source code compatibility problems. This is an experimental feature,
that is not yet fully supported."""
ctx.workflow_linter.refresh_report(ctx.sql_backend, ctx.inventory_database)

@job_task(dashboard="migration_main", depends_on=[lint_all_workflows])
def migration_report(self, ctx: RuntimeContext):
"""Refreshes the migration dashboard after all previous tasks have been completed. Note that you can access the
dashboard _before_ all tasks have been completed, but then only already completed information is shown."""
Loading