Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add databricks labs ucx migrate-tables and a related workflow #670

Closed
nfx opened this issue Dec 5, 2023 · 1 comment · Fixed by #1511
Closed

Add databricks labs ucx migrate-tables and a related workflow #670

nfx opened this issue Dec 5, 2023 · 1 comment · Fixed by #1511
Assignees
Labels
feat/migration-index mapping of databases to catalog or potentially other databases migrate/external go/uc/upgrade SYNC EXTERNAL TABLES step migrate/managed go/uc/upgrade Upgrade Managed Tables and Jobs

Comments

@nfx
Copy link
Collaborator

nfx commented Dec 5, 2023

databricks labs ucx migrate-tables to apply table mapping from CSV file with the following columns: src_workspace_name,dst_catalog,src_schema,dst_schema,src_table,dst_table . Consider the following stages:

Initial assessment
We scan all tables, generate an inventory, get all perms, etc.
We provide a mapping file the customer can customize, but seed the UC table name with something like the workspace name or ID as the catalog.
Customer can choose to modify the mapping or not

Initial migration
We use the mapping file to sync tables, apply perms, and persist the mapping data in inventory

Repeated migration - this is probably a job that runs on schedule and the user should be instructed to configure accordingly
We use the mapping data in inventory to re-sync any tables, but not re-apply permissions as these probably changed.

Fine-grained:

Upstream dependencies:

Next quarter:

@nfx nfx added migrate/external go/uc/upgrade SYNC EXTERNAL TABLES step migrate/managed go/uc/upgrade Upgrade Managed Tables and Jobs feat/migration-index mapping of databases to catalog or potentially other databases labels Dec 5, 2023
@github-project-automation github-project-automation bot moved this to Triage in UCX Dec 5, 2023
@FastLee
Copy link
Contributor

FastLee commented Dec 6, 2023

  1. We have to make sure that the mapping is validated and no conflicts were found prior to migration.
  2. Make sure the external locations required for the migration were created and that the user has created table permissions on these external locations.

@nfx nfx moved this from Triage to Month Backlog in UCX Dec 6, 2023
@pohlposition pohlposition moved this from Month Backlog to Active Backlog in UCX Apr 8, 2024
@qziyuan qziyuan linked a pull request Apr 24, 2024 that will close this issue
@nfx nfx closed this as completed in #1511 Apr 25, 2024
@github-project-automation github-project-automation bot moved this from Active Backlog to Archive in UCX Apr 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feat/migration-index mapping of databases to catalog or potentially other databases migrate/external go/uc/upgrade SYNC EXTERNAL TABLES step migrate/managed go/uc/upgrade Upgrade Managed Tables and Jobs
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

4 participants