Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Update databricks-labs-lsql requirement from <0.12,>=0.5 to >=0.5,<0.13 #2688

Merged
merged 1 commit into from
Sep 20, 2024
Merged

Update databricks-labs-lsql requirement from <0.12,>=0.5 to >=0.5,<0.13 #2688

merged 1 commit into from
Sep 20, 2024

Conversation

dependabot[bot]
Copy link
Contributor

@dependabot dependabot bot commented on behalf of github Sep 19, 2024 •
edited
Loading

⚠️ Dependabot is rebasing this PR ⚠️

Rebasing might not happen immediately, so don't worry if this takes some time.

Note: if you make any changes to this PR yourself, they will take precedence over the rebase.

Updates the requirements on databricks-labs-lsql to permit the latest version.

Release notes

Sourced from databricks-labs-lsql's releases.

v0.12.0

  • Added method to detect rows are written to the MockBackend (#292). In this commit, the MockBackend class in the 'backends.py' file has been updated with a new method, 'has_rows_written_for', which allows for differentiation between a table that has never been written to and one with zero rows. This method checks if a specific table has been written to by iterating over the table stubs in the _save_table attribute and returning True if the given full name matches any of the stub full names. Additionally, the class has been supplemented with the rows_written_for method, which takes a table name and mode as input and returns a list of rows written to that table in the given mode. Furthermore, several new test cases have been added to test the functionality of the MockBackend class, including checking if the has_rows_written_for method correctly identifies when there are no rows written, when there are zero rows written, and when rows are written after the first and second write operations. These changes improve the overall testing coverage of the project and aid in testing the functionality of the MockBackend class. The new methods are accompanied by documentation strings that explain their purpose and functionality.

Contributors: @​JCZuurmond

Changelog

Sourced from databricks-labs-lsql's changelog.

0.12.0

  • Added method to detect rows are written to the MockBackend (#292). In this commit, the MockBackend class in the 'backends.py' file has been updated with a new method, 'has_rows_written_for', which allows for differentiation between a table that has never been written to and one with zero rows. This method checks if a specific table has been written to by iterating over the table stubs in the _save_table attribute and returning True if the given full name matches any of the stub full names. Additionally, the class has been supplemented with the rows_written_for method, which takes a table name and mode as input and returns a list of rows written to that table in the given mode. Furthermore, several new test cases have been added to test the functionality of the MockBackend class, including checking if the has_rows_written_for method correctly identifies when there are no rows written, when there are zero rows written, and when rows are written after the first and second write operations. These changes improve the overall testing coverage of the project and aid in testing the functionality of the MockBackend class. The new methods are accompanied by documentation strings that explain their purpose and functionality.

0.11.0

  • Added filter spec implementation (#276). In this commit, a new FilterHandler class has been introduced to handle filter files with the suffix .filter.json, which can parse filter specifications in the header of the filter file and validate the filter columns and types. The commit also adds support for three types of filters: DATE_RANGE_PICKER, MULTI_SELECT, and DROPDOWN, which can be linked with multiple visualization widgets. Additionally, a FilterTile class has been added to the Tile class, which represents a filter tile in the dashboard and includes methods to validate the tile, create widgets, and generate filter encodings and queries. The DashboardMetadata class has been updated to include a new method get_datasets() to retrieve the datasets for the dashboard. These changes enhance the functionality of the dashboard by adding support for filtering data using various filter types and linking them with multiple visualization widgets, improving the customization and interactivity of the dashboard, and making it more user-friendly and efficient.
  • Bugfix: MockBackend wasn't mocking savetable properly when the mode is append (#289). This release includes a bugfix and enhancements for the MockBackend component, which is used to mock the SQLBackend. The .savetable() method failed to function as expected in append mode, writing all rows to the same table instead of accumulating them. This bug has been addressed, ensuring that rows accumulate correctly in append mode. Additionally, a new test function, test_mock_backend_save_table_overwrite(), has been added to demonstrate the corrected behavior of overwrite mode, showing that it now replaces only the existing rows for the given table while preserving other tables' contents. The type signature for .save_table() has been updated, restricting the mode parameter to accept only two string literals: "append" and "overwrite". The MockBackend behavior has been updated accordingly, and rows are now filtered to exclude any None or NULL values prior to saving. These improvements to the MockBackend functionality and test suite increase reliability when using the MockBackend as a testing backend for the system.
  • Changed filter spec to use YML instead of JSON (#290). In this release, the filter specification files have been converted from JSON to YAML format, providing a more human-readable format for the filter specifications. The schema for the filter file includes flags for column, columns, type, title, description, order, and id, with the type flag taking on values of DROPDOWN, MULTI_SELECT, or DATE_RANGE_PICKER. This change impacts the FilterHandler, is_filter method, and _from_dashboard_folder method, as well as relevant parts of the documentation. Additionally, the parsing methods have been updated to use yaml.safe_load instead of json.loads, and the is_filter method now checks for .filter.yml suffix. A new file, '00_0_date.filter.yml', has been added to the 'tests/integration/dashboards/filter_spec_basic' directory, containing a sample date filter definition. Furthermore, various tests have been added to validate filter specifications, such as checking for invalid type and both column and columns keys being present. These updates aim to enhance readability, maintainability, and ease of use for filter configuration.
  • Increase testing of generic types storage (#282). A new commit enhances the testing of generic types storage by expanding the test suite to include a list of structs, ensuring more comprehensive testing of the system. The Foo struct has been renamed to Nested for clarity, and two new structs, NestedWithDict and Nesting, have been added. The Nesting struct contains a Nested object, while NestedWithDict includes a string and an optional dictionary of strings. A new test case demonstrates appending complex types to a table by creating and saving a table with two rows, each containing a Nesting struct. The test then fetches the data and asserts the expected number of rows are returned, ensuring the proper functioning of the storage system with complex data types.
  • Minor Changes to avoid redundancy in code and follow code patterns (#279). In this release, we have made significant improvements to the dashboards.py file to make the code more concise, maintainable, and in line with the standard library's recommended usage. The export_to_zipped_csv method has undergone major changes, including the removal of the BytesIO module import and the use of StringIO for handling strings as files. The method no longer creates a separate ZIP file for the CSV files, instead using the provided export_path. Additionally, the method skips tiles that don't contain queries. We have also introduced a new method, dataclass_transform, which transforms a given dataclass into a new one with specific attributes and behavior. This method creates a new dataclass with a custom metaclass and adds a new method, to_dict(), which converts the instances of the new dataclass to dictionaries. These changes promote code reusability and reduce redundancy in the codebase, making it easier for software engineers to work with.
  • New example with bar chart in dashboards-as-code (#281). A new example of a dashboard featuring a bar chart has been added to the dashboards-as-code feature using the existing metadata overrides feature to support the new widget type, without bloating the TileMetadata structure. An integration test was added to demonstrate the creation of a bar chart, and the resulting dashboard can be seen in the attached screenshot. Additionally, a new SQL file has been added for the Product Sales dashboard, showcasing sales data for different product categories. This approach can potentially be used to support other widget types such as Bar, Pivot, Area, etc. The team is encouraged to provide feedback on this proposed solution.

0.10.0

  • Added Functionality to export any dashboards-as-code into CSV (#269). The DashboardMetadata class now includes a new method, export_to_zipped_csv, which enables exporting any dashboard as CSV files in a ZIP archive. This method accepts sql_backend and export_path as parameters and exports dashboard queries to CSV files in the specified ZIP archive by iterating through tiles and fetching dashboard queries if the tile is a query. To ensure the proper functioning of this feature, unit tests and manual testing have been conducted. A new test, test_dashboards_export_to_zipped_csv, has been added to verify the correct export of dashboard data to a CSV file.
  • Added support for generic types in SqlBackend (#272). In this release, we've added support for using rich dataclasses, including those with optional and generic types, in the SqlBackend of the StatementExecutionBackend class. The new functionality is demonstrated in the test_supports_complex_types unit test, which creates a Nested dataclass containing various complex data types, such as nested dataclasses, datetime objects, dict, list, and optional fields. This enhancement is achieved by updating the save_table method to handle the conversion of complex dataclasses to SQL statements. To facilitate type inference, we've introduced a new StructInference class that converts Python dataclasses and built-in types to their corresponding SQL Data Definition Language (DDL) representations. This addition simplifies data definition and manipulation operations while maintaining type safety and compatibility with various SQL data types.

0.9.3

  • Added documentation for exclude flag (#265). A new exclude flag has been added to the configuration file for our lab tool, allowing users to specify a path to exclude from formatting during lab execution. This release also includes corrections to grammatical errors in the descriptions of existing flags related to catalog and database settings, such as updating seperated to "separate". Additionally, the flag descriptions for publish and open-browser have been updated for clarification: publish now clearly controls whether the dashboard is published after creation, while open-browser controls whether the dashboard is opened in a web browser. These changes are aimed at improving user experience and ease of use for our lab tool.
  • Fixed dataclass field type in _row_to_sql (#266). In this release, we have addressed an issue related to #257 by fixing the dataclass field type in the _row_to_sql method of the backends.py file. Additionally, we have made updates to the _schema_for method to use a new _field_type class method. This change resolves a rare problem where the field.type is a string instead of a type and ensures compatibility with a pull request from an external repository (databrickslabs/ucx#2526). The new _field_type method attempts to load the type from __builtins__ if it's a string and logs a warning if it fails. The _row_to_sql method now consistently uses the _field_type method to get the field type. This ensures that the library functions seamlessly and consistently, avoiding any potential issues in the future.

0.9.2

  • Make hatch a prerequisite (#259). In this commit, Eric Vergnaud has introduced a change to make the installation of hatch version 1.9.4 a prerequisite for the project to avoid errors related to pip command recognition. The Makefile has been updated to handle the installation of hatch automatically, and the hatch env create command is now used instead of pip install hatch==1.7.0. This change ensures that the development environment is consistent and reliable by maintaining the correct version of hatch and automatically handling its installation. Additionally, the .venv/bin/python and dev targets have been updated accordingly to reflect these changes. This commit also formats all files using the make dev fmt command, which helps maintain consistent code formatting throughout the project.
  • add support for exclusions in fmt command (#263). In this release, we have added support for exclusions to the fmt command in the 'databricks/labs/lsql/cli.py' module. This feature allows users to specify a list of directories or files to exclude while formatting SQL files, which is particularly useful when verifying SQL notebooks in ucx. The fmt command now accepts a new optional parameter 'exclude', which accepts an iterable of strings that specify the relative paths to exclude. Any sql_file that is a descendant of any exclusion is skipped during formatting. The exclusions are implemented by converting the relative paths into Path objects. This change addresses the issue where single line comments are converted into inlined comments, causing misinterpretation. The added unit test is manually verified, and this pull request fixes issue #261. This feature was authored and co-authored by Eric Vergnaud.

0.9.1

  • Fixed dataclass field types (#257). This PR introduces a workaround to a Python bug affecting the dataclasses.fields() function, which sometimes returns field types as string type names instead of types. This can cause the ORM to malfunction. The workaround involves checking if the returned f.type is a string, and if so, converting it to a type by looking it up in the __builtins__ dictionary. This change is global and affects the _schema_for function in the backends.py file, which is responsible for creating a schema for a given dataclass, taking into account any necessary type conversions. This change ensures consistent and accurate type handling in the face of the Python bug, improving the reliability of our ORM.
  • Fixed missing EOL when formatting SQL files (#260). In this release, we have addressed an issue related to the inconsistent addition of end-of-line (EOL) characters in formatted SQL files. The QueryTile.format() method has been updated to ensure that an EOL character is always added, except when the input query already ends with a newline. This change enhances the reliability of the SQL formatting functionality, making the output format more predictable and improving the overall user experience. The new implementation is demonstrated in the test_query_format_preserves_eol() test case, and existing test cases have been updated to check for the presence of EOL characters, further ensuring consistent and correct formatting.
  • Fixed normalize case input in cli (#258). In this release, we have updated the fmt command in the cli.py file to allow users to specify whether they want to normalize the case of SQL files when formatting. The normalize_case parameter now defaults to the string "true" and checks if it is in the STRING_AFFIRMATIVES list to determine whether to normalize the case of SQL files. Additionally, we have introduced a new optional normalize_case parameter in the format method of the dashboards.py file in the Databricks CLI, which normalizes the identifiers in the query to lower case when set to True. We have also added support for a new normalize_case parameter in the QueryTile.format() method, which prevents the automatic normalization of string input to uppercase when set to False. This change allows for more flexibility in handling string input and ensures that the input string is preserved as-is. These updates improve the functionality and usability of the open-source library, providing more control to users over formatting and handling of string input.

0.9.0

  • Added design for filter file (#251). A new feature has been added to enable the creation of filters for multiple widgets in a dashboard using a .filter.json file. This file allows users to specify columns to be filtered, the filter type, title, description, order, and a unique ID for each filter. Both the column and columns flags are supported, with the former taking a single string and the latter taking a list of strings. The filter type can be set to a drop-down menu or another type as desired. The .filter.json file schema also supports optional title and description strings, as well as order and ID flags. An example of a .filter.json file is provided in the commit message. Additionally, the dashboard.yml file documentation has been updated to include information on how to use the new .filter.json file.
  • adding normalize-case option to databricks labs lsql fmt cmd (#254). In this open-source library release, the databricks labs lsql tool's fmt command now supports a new flag, normalize-case. This flag allows users to control the normalization of query text to lowercase, providing more flexibility when formatting SQL queries. By default, query text is still normalized to lowercase, but users can now prevent this behavior by setting the normalize-case flag to False. This change addresses an issue where some queries are case sensitive, such as those using map field keys in UCX dashboards. Additionally, a new parameter normalize_case has been added to the format method in the dashboards.py file, with updated method documentation. A new test function, test_query_formats_no_normalize(), has also been included to ensure consistent formatter behavior.

0.8.0

  • Removed deploy_dashboard method (#240). In this release, the deploy_dashboard method has been removed from the dashboards.py file and the legacy deployment method has been deprecated. The deploy_dashboard method was previously used to deploy a dashboard to a workspace, but it has been replaced with the create method of the lakeview attribute of the WorkspaceClient object. Additionally, the test_dashboards_creates_dashboard_via_legacy_method method has been removed. A new test has been added to ensure that the deploy_dashboard method is no longer being used, utilizing the deprecated_call function from pytest to verify that calling the method raises a deprecation warning. This change simplifies the code and improves the overall design of the system, resolving issue #232. The _with_better_names method and create_dashboard method remain unchanged.
  • Skip test that fails due to insufficient permission to create schema (#248). A new test function, test_dashboards_creates_dashboard_with_replace_database, has been added to the open-source library, but it is currently marked to be skipped due to missing permissions to create a schema. This function creates an instance of the Dashboards class with the ws parameter, creates a dashboard using the make_dashboard function, and performs various actions using the created dashboard, as well as functions such as tmp_path and sql_backend. This test function aims to ensure that the Dashboards class functions as expected when creating a dashboard with a replaced database. Once the necessary permissions for creating a schema are acquired, this test function can be enabled for further testing and validation.

... (truncated)

Commits

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

  • @dependabot rebase will rebase this PR
  • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
  • @dependabot merge will merge this PR after your CI passes on it
  • @dependabot squash and merge will squash and merge this PR after your CI passes on it
  • @dependabot cancel merge will cancel a previously requested merge and block automerging
  • @dependabot reopen will reopen this PR if it is closed
  • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
  • @dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency
  • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
  • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot
Update databricks-labs-lsql requirement from <0.12,>=0.5 to >=0.5,<0.13
291fe02 
Updates the requirements on [databricks-labs-lsql](https://github.com/databrickslabs/lsql) to permit the latest version.
- [Release notes](https://github.com/databrickslabs/lsql/releases)
- [Changelog](https://github.com/databrickslabs/lsql/blob/main/CHANGELOG.md)
- [Commits](databrickslabs/lsql@v0.5.0...v0.12.0)

---
updated-dependencies:
- dependency-name: databricks-labs-lsql
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
@dependabot dependabot bot requested a review from a team as a code owner September 19, 2024 15:39
@dependabot dependabot bot added dependencies python Pull requests that update Python code labels Sep 19, 2024
@JCZuurmond
Copy link
Member

Running CI manually: https://github.com/databrickslabs/ucx/actions/runs/10954818987

@JCZuurmond JCZuurmond mentioned this pull request Sep 20, 2024
Merged
1 task
@nfx nfx merged commit b6912eb into main Sep 20, 2024
6 of 7 checks passed
@nfx nfx deleted the dependabot/pip/databricks-labs-lsql-gte-0.5-and-lt-0.13 branch September 20, 2024 12:46
@nfx nfx mentioned this pull request Sep 26, 2024
Merged
nfx added a commit that referenced this pull request Sep 26, 2024
@nfx
Release v0.38.0
ba8edf1 
* Added Py4j implementation of tables crawler to retrieve a list of HMS tables in the assessment workflow ([#2579](#2579)). In this release, we have added a Py4j implementation of a tables crawler to retrieve a list of Hive Metastore tables in the assessment workflow. A new `FasterTableScanCrawler` class has been introduced, which can be used in the Assessment Job based on a feature flag to replace the old Scala code, allowing for better logging during table scans. The existing `assessment.crawl_tables` workflow now utilizes the new py4j crawler instead of the scala one. Integration tests have been added to ensure the functionality works correctly. The commit also includes a new method for listing table names in the specified database and improvements to error handling and logging mechanisms. The new Py4j tables crawler enhances the functionality of the assessment workflow by improving error handling, resulting in better logging and faster table scanning during the assessment process. This change is part of addressing issue [#2190](#2190) and was co-authored by Serge Smertin.
* Added `create-ucx-catalog` cli command ([#2694](#2694)). A new CLI command, `create-ucx-catalog`, has been added to create a catalog for migration tracking that can be used across multiple workspaces. The command creates a UCX catalog for tracking migration status and artifacts, and is created by running `databricks labs ucx create-ucx-catalog` and specifying the storage location for the catalog. Relevant user documentation, unit tests, and integration tests have been added for this command. The `assign-metastore` command has also been updated to allow for the selection of a metastore when multiple metastores are available in the workspace region. This change improves the migration tracking feature and enhances the user experience.
* Added experimental `migration-progress-experimental` workflow ([#2658](#2658)). This commit introduces an experimental workflow, `migration-progress-experimental`, which refreshes the inventory for various resources such as clusters, grants, jobs, pipelines, policies, tables, TableMigrationStatus, and UDFs. The workflow can be triggered using the `databricks labs ucx migration-progress` CLI command and uses a new implementation of a Scala-based crawler, `TablesCrawler`, which will eventually replace the current implementation. The new workflow is a duplicate of most of the `assessment` pipeline's functionality but with some differences, such as the use of `TablesCrawler`. Relevant user documentation has been added, along with unit tests, integration tests, and a screenshot of a successful staging environment run. The new workflow is expected to run on a schedule in the future. This change resolves [#2574](#2574) and progresses [#2074](#2074).
* Added handling for `InternalError` in `Listing.__iter__` ([#2697](#2697)). This release introduces improved error handling in the `Listing.__iter__` method of the `Generic` class, located in the `workspace_access/generic.py` file. Previously, only `NotFound` exceptions were handled, but now both `InternalError` and `NotFound` exceptions are caught and logged appropriately. This change enhances the robustness of the method, which is responsible for listing objects of a specific type and returning them as `GenericPermissionsInfo` objects. To ensure the correct functionality, we have added new unit tests and manual testing. The logging of the `InternalError` exception is properly handled in the `GenericPermissionsSupport` class when listing serving endpoints. This behavior is verified by the newly added test function `test_internal_error_in_serving_endpoints_raises_warning` and the updated `test_serving_endpoints_not_enabled_raises_warning`.
* Added handling for `PermissionDenied` when listing accessible workspaces ([#2733](#2733)). A new `can_administer` method has been added to the `Workspaces` class in the `workspaces.py` file, which allows for more fine-grained control over which users can administer workspaces. This method checks if the user has access to a given workspace and is a member of the workspace's `admins` group, indicating that the user has administrative privileges for that workspace. If the user does not have access to the workspace or is not a member of the `admins` group, the method returns `False`. Additionally, error handling in the `get_accessible_workspaces` method has been improved by adding a `PermissionDenied` exception to the list of exceptions that are caught and logged. New unit tests have been added for the `AccountWorkspaces` class of the `databricks.labs.blueprint.account` module to ensure that the new method is functioning as intended, specifically checking if a user is a workspace administrator based on whether they belong to the `admins` group. The linked issue [#2732](#2732) is resolved by this change. All changes have been manually and unit tested.
* Added static code analysis results to assessment dashboard ([#2696](#2696)). This commit introduces two new tasks, `assess_dashboards` and `assess_workflows`, to the existing assessment dashboard for identifying migration problems in dashboards and workflows. These tasks analyze embedded queries and notebooks for migration issues and collect direct filesystem access patterns requiring attention. Upon completion, the results are stored in the inventory database and displayed on the Migration dashboard. Additionally, two new widgets, job/query problem widgets and directfs access widgets, have been added to enhance the dashboard's functionality by providing additional information related to code compatibility and access control. Integration tests using mock data have been added and manually tested to ensure the proper functionality of these new features. This update improves the overall assessment and compatibility checking capabilities of the dashboard, making it easier for users to identify and address issues related to Unity Catalog compatibility in their workflows and dashboards.
* Added unskip CLI command to undo a skip on schema or a table ([#2727](#2727)). This pull request introduces a new CLI command, "unskip", which allows users to reverse a previously applied `skip` on a schema or table. The `unskip` command accepts a required `--schema` parameter and an optional `--table` parameter. A new function, also named "unskip", has been added, which takes the same parameters as the `skip` command. The function checks for the required `--schema` parameter and creates a new WorkspaceContext object to call the appropriate method on the table_mapping object. Two new methods, `unskip_schema` and "unskip_table_or_view", have been added to the HiveMapping class. These methods remove the skip mark from a schema or table, respectively, and handle exceptions such as NotFound and BadRequest. The get_tables_to_migrate method has been updated to consider the unskipped tables or schemas. Currently, the feature is tested manually and has not been added to the user documentation.
* Added unskip CLI command to undo a skip on schema or a table ([#2734](#2734)). A new `unskip` CLI command has been added to the project, which allows users to remove the `skip` mark set by the existing `skip` command on a specified schema or table. This command takes an optional `--table` flag, and if not provided, it will unskip the entire schema. The new functionality is accompanied by a unit test and relevant user documentation, and addresses issue [#1938](#1938). The implementation includes the addition of the `unskip_table_or_view` method, which generates the appropriate `ALTER TABLE/VIEW` statement to remove the skip marker, and updates to the `unskip_schema` method to include the schema name in the `ALTER SCHEMA` statement. Additionally, exception handling has been updated to include `NotFound` and `BadRequest` exceptions. This feature simplifies the process of undoing a skip on a schema, table, or view in the Hive metastore, which previously required manual editing of the Hive metastore properties.
* Assess source code as part of the assessment ([#2678](#2678)). This commit introduces enhancements to the assessment workflow, including the addition of two new tasks for evaluating source code from SQL queries in dashboards and from notebooks/files in jobs and tasks. The existing `databricks labs install ucx` command has been modified to incorporate linting during the assessment. The `QueryLinter` class has been updated to accept an additional argument for linting source code. These changes have been thoroughly tested through integration tests to ensure proper functionality. Co-authored by Eric Vergnaud.
* Bump astroid version, pylint version and drop our f-string workaround ([#2746](#2746)). In this update, we have bumped the versions of astroid and pylint to 3.3.1 and removed workarounds related to f-string inference limitations in previous versions of astroid (< 3.3). These workarounds were necessary for handling issues such as uninferrable sys.path values and the lack of f-string inference in loops. We have also updated corresponding tests to reflect these changes and improve the overall code quality and maintainability of the project. These changes are part of a larger effort to update dependencies and simplify the codebase by leveraging the latest features of updated tools and removing obsolete workarounds.
* Delete temporary files when running solacc ([#2750](#2750)). This commit includes changes to the `solacc.py` script to improve the linting process for the `solacc` repository, specifically targeting the issue of excessive temporary files that were exceeding CI storage capacity. The modifications include linting the repository on a per-top-level `solution` basis, where each solution resides within the top folders and is independent of others. Post-linting, temporary files and directories registered in `PathLookup` are deleted to enhance storage efficiency. Additionally, this commit prepares for improving false positive detection and introduces a new `SolaccContext` class that tracks various aspects of the linting process, providing more detailed feedback on the linting results. This change does not introduce new functionality or modify existing functionality, but rather optimizes the linting process for the `solacc` repository, maintaining CI storage capacity levels within acceptable limits.
* Don't report direct filesystem access for API calls ([#2689](#2689)). This release introduces enhancements to the Direct File System Access (DFSA) linter, resolving false positives in API call reporting. The `ws.api_client.do` call previously triggered inaccurate direct filesystem access alerts, which have been addressed by adding new methods to identify HTTP call parameters and specific API calls. The linter now disregards DFSA patterns within known API calls, eliminating false positives with relative URLs and duplicate advice from SparkSqlPyLinter. Additionally, improvements in the `python_ast.py` and `python_infer.py` files include the addition of `is_instance_of` and `is_from_module` methods, along with safer inference methods to prevent infinite recursion and enhance value inference. These changes significantly improve the DFSA linter's accuracy and effectiveness when analyzing code containing API calls.
* Enables cli cmd `databricks labs ucx create-catalog-schemas` to apply catalog/schema acl from legacy hive_metastore ([#2676](#2676)). The new release introduces a `databricks labs ucx create-catalog-schemas` command, which applies catalog/schema Access Control List (ACL) from a legacy hive_metastore. This command modifies the existing `table_mapping` method to include a new `grants_crawler` parameter in the `CatalogSchema` constructor, enabling the application of ACLs from the legacy hive_metastore. A corresponding unit test is included to ensure proper functionality. The `CatalogSchema` class in the `databricks.labs.ucx.hive_metastore.catalog_schema` module has been updated with a new argument `hive_acl` and the integration of the `GrantsCrawler` class. The `GrantsCrawler` class is responsible for crawling the Hive metastore and retrieving grants for catalogs, schemas, and tables. The `prepare_test` function has been updated to include the `hive_acl` argument and the `test_catalog_schema_acl` function has been updated to test the new functionality, ensuring that the correct grant statements are generated for a wider range of principals and catalogs/schemas. These changes improve the functionality and usability of the `databricks labs ucx create-catalog-schemas` command, allowing for a more seamless transition from a legacy hive metastore.
* Fail `make test` on coverage below 90% ([#2682](#2682)). A new change has been introduced to the pyproject.toml file to enhance the codebase's quality and robustness by ensuring that the test coverage remains above 90%. This has been accomplished by adding the `--cov-fail-under=90` flag to the `test` and `coverage` scripts in the `[tool.hatch.envs.default.scripts]` section. This flag will cause the `make test` command to fail if the coverage percentage falls below the specified value of 90%, ensuring that all new changes are thoroughly tested and that the codebase maintains a minimum coverage threshold. This is a best practice for maintaining code coverage and improving the overall quality and reliability of the codebase.
* Fixed DFSA false positives from f-string fragments ([#2679](#2679)). This commit addresses false positive DataFrame API Scanning Antipattern (DFSA) reports in Python code, specifically in f-string fragments containing forward slashes and curly braces. The linter has been updated to accurately detect DFSA paths while avoiding false positives, and it now checks for `JoinedStr` fragments in string constants. Additionally, the commit rectifies issues with duplicate advices reported by `SparkSqlPyLinter`. No new features or major functionality changes have been introduced; instead, the focus has been on improving the reliability and accuracy of DFSA detection. Co-authored by Eric Vergnaud, this commit includes new unit tests and refinements to the DFSA linter, specifically addressing false positive patterns like `f"/Repos/{thing1}/sdk-{thing2}-{thing3}"`. To review these changes, consult the updated tests in the `tests/unit/source_code/linters/test_directfs.py` file, such as the new test case for the f-string pattern causing false positives. By understanding these improvements, you'll ensure your project adheres to the latest updates, maintaining quality and accurate DFSA detection.
* Fixed failing integration tests that perform a real assessment ([#2736](#2736)). In this release, we have made significant improvements to the integration tests in the `assessment` workflow, by reducing the scope of the assessment and improving efficiency and reliability. We have removed several object creation functions and added a new function `populate_for_linting` for linting purposes. The `populate_for_linting` function adds necessary information to the installation context, and is used to ensure that the integration tests still have the required data for linting. We have also added a pytest fixture `populate_for_linting` to set up a minimal amount of data in the workspace for linting purposes. These changes have been implemented in the `test_workflows.py` file in the integration/assessment directory. This will help to ensure that the tests are not unnecessarily extensive, and that they are able to accurately assess the functionality of the library.
* Fixed sqlglot crasher with 'drop schema ...' statement ([#2758](#2758)). In this release, we have addressed a crash issue in the `sqlglot` library caused by the `drop schema` statement. A new method, `_unsafe_lint_expression`, has been introduced to prevent the crash by checking if the current expression is a `Use`, `Create`, or `Drop` statement and updating the `schema` attribute accordingly. The library now correctly handles the `drop schema` statement and returns a `Deprecation` warning if the table being processed is in the `hive_metastore` catalog and has been migrated to the Unity Catalog. Unit tests have been added to ensure the correct behavior of this code, and the linter for `from table` SQL has been updated to parse and handle the `drop schema` statement without raising any errors. These changes improve the library's overall reliability and stability, allowing it to operate smoothly with the `drop schema` statement.
* Fixed test failure: `test_table_migration_job_refreshes_migration_status[regular-migrate-tables]` ([#2625](#2625)). In this release, we have addressed two issues ([#2621](#2621) and [#2537](#2537)) and fixed a test failure in `test_table_migration_job_refreshes_migration_status[regular-migrate-tables]`. The `index` and `index_full_refresh` methods in `table_migrate.py` have been updated to accept a new `force_refresh` flag. When set to `True`, these methods will ensure that the migration status is up-to-date. This change also affects the `ViewsMigrationSequencer` class, which now passes `force_refresh=True` to the `index` method. Additionally, we have fixed a test failure by reusing the `force_refresh` flag to ensure the migration status is up-to-date. The `TableMigrationStatus` class in `table_migration_status.py` has been modified to accept an optional `force_refresh` parameter in the `index` method, and a unit test has been updated to assert the correct behavior when updating the migration status.
* Fixes error message ([#2759](#2759)). The `load` method of the `mapping.py` file in the `databricks/labs/ucx/hive_metastore` package has been updated to correct an error message displayed when a `NotFound` exception is raised. The previous message suggested running an incorrect command, which has been updated to the correct one: "Please run: databricks labs ucx create-table-mapping". This change does not add any new methods or alter existing functionality, but instead focuses on improving the user experience by providing accurate information when an error occurs. The scope of this change is limited to updating the error message, and no other modifications have been made.
* Fixes issue of circular dependency of migrate-location ACL ([#2741](#2741)). In this release, we have resolved two issues ([#274](#274)
* Fixes source table alias dissapearance during migrate_views ([#2726](#2726)). This release introduces a fix to preserve the alias for the source table during the conversion of CREATE VIEW SQL from the legacy Hive metastore to the Unity Catalog. The issue was addressed by adding a new test case, `test_migrate_view_alias_test`, to verify the correct handling of table aliases during migration. The changes also include a fix for the SQL conversion and new test cases to ensure the correct handling of table aliases, reflected in accurate SQL conversion. A new parameter, `alias`, has been added to the Table class, and the `apply` method in the `from_table.py` file has been updated. The migration process has been updated to retain the original alias of the table. Unit tests have been added and thoroughly tested to confirm the correctness of the changes, including handling potential intermittent failures caused by external dependencies.
* Py4j table crawler: suggestions/fixes for describing tables ([#2684](#2684)). This release introduces significant improvements and fixes to the Py4J-based table crawler, enhancing its capability to describe tables effectively. The code for fetching table properties over the bridge has been updated, and error tracing has been improved through individual fetching of each table property and providing python backtrace on JVM side errors. Scala `Option` values unboxing issues have been resolved, and a small optimization has been implemented to detect partitioned tables without materializing the collection. The table's `.viewText()` property is now properly handled as a Scala `Option`. The `catalog` argument is now explicitly verified to be `hive_metastore`, and a new static method `_option_as_python` has been introduced for safely extracting values from Scala `Option`. The `_describe` method has been refactored to handle exceptions more gracefully and improved code readability. These changes result in better functionality, error handling, logging, and performance when describing tables within a specified catalog and database. The linked issues [#2658](#2658) and [#2579](#2579) are progressed through these updates, and appropriate testing has been conducted to ensure the improvements' effectiveness.
* Speedup assessment workflow by making DBFS root table size calculation parallel ([#2745](#2745)). In this release, the assessment workflow for calculating DBFS root table size has been optimized through the parallelization of the calculation process, resulting in improved performance. This has been achieved by updating the `pipelines_crawler` function in `src/databricks/labs/ucx/contexts/workflow_task.py`, specifically the `cached_property table_size_crawler`, to include an additional argument `self.config.include_databases`. The `TablesCrawler` class has also been modified to include a generic type parameter `Table`, enabling type hinting and more robust type checking. Furthermore, the unit test file `test_table_size.py` in the `hive_metastore` directory has been updated to handle corrupt tables and invalid delta format errors more effectively. Additionally, a new entry `databricks-pydabs` has been added to the "known.json" file, potentially enabling better integration with the `databricks-pydabs` library or providing necessary configuration information for parallel processing. Overall, these changes improve the efficiency and scalability of the codebase and optimize the assessment workflow for calculating DBFS root table size.
* Updated databricks-labs-blueprint requirement from <0.9,>=0.8 to >=0.8,<0.10 ([#2747](#2747)). In this update, the requirement for `databricks-labs-blueprint` has been updated to version `>=0.8,<0.10` in the `pyproject.toml` file. This change allows the project to utilize the latest features and bug fixes included in version 0.9.0 of the `databricks-labs-blueprint` library. Notable updates in version 0.9.0 consist of the addition of Databricks CLI version as part of routed command telemetry and support for Unicode Byte Order Mark (BOM) in file upload and download operations. Additionally, various bug fixes and improvements have been implemented for the `WorkspacePath` class, including the addition of `stat()` methods and improved compatibility with different versions of Python.
* Updated databricks-labs-lsql requirement from <0.12,>=0.5 to >=0.5,<0.13 ([#2688](#2688)). In this update, the version requirement of the `databricks-labs-lsql` library has been changed from a version greater than or equal to 0.5 and less than 0.12 to a version greater than or equal to 0.5 and less than 0.13. This allows the project to utilize the latest version of 'databricks-labs-lsql', which includes new methods for differentiating between a table that has never been written to and one with zero rows in the MockBackend class. Additionally, the update adds support for various filter types and improves testing coverage and reliability. The release notes and changelog for the updated library are provided in the commit message for reference.
* Updated documentation to explain the usage of collections and eligible commands ([#2738](#2738)). The latest update to the Databricks Labs Unified CLI (UCX) tool introduces the `join-collection` command, which enables users to join two or more workspaces into a collection, allowing for streamlined and consolidated command execution across multiple workspaces. This feature is available to Account admins on the Databricks account, Workspace admins on the workspaces to be joined, and requires UCX installation on the workspace. To run collection-eligible commands, users can simply pass the `--run-as-collection=True` flag. This enhancement enhances the UCX tool's functionality, making it easier to manage and execute commands on multiple workspaces.
* Updated sqlglot requirement from <25.22,>=25.5.0 to >=25.5.0,<25.23 ([#2687](#2687)). In this pull request, we have updated the version requirement for the `sqlglot` library in the pyproject.toml file. The previous requirement specified a version greater than or equal to 25.5.0 and less than 25.22, but we have updated it to allow for versions greater than or equal to 25.5.0 and less than 25.23. This change allows us to use the latest version of 'sqlglot', while still ensuring compatibility with other dependencies. Additionally, this pull request includes a detailed changelog from the `sqlglot` repository, which provides information on the features, bug fixes, and changes included in each version. This can help us understand the scope of the update and how it may impact our project.
* [DOCUMENTATION] Improve documentation on using account profile for `sync-workspace-info` cli command ([#2683](#2683)). The `sync-workspace-info` CLI command has been added to the Databricks Labs UCX package, which uploads the workspace configuration to all workspaces in the Databricks account where the `ucx` tool is installed. This feature requires Databricks Account Administrator privileges and is necessary to create an immutable default catalog mapping for the table migration process. It also serves as a prerequisite for the `create-table-mapping` command. To utilize this command, users must configure the Databricks CLI profile with access to the Databricks account console, available at "accounts.cloud.databricks.com" or "accounts.azuredatabricks.net". Additionally, the documentation for using the account profile with the `sync-workspace-info` command has been enhanced, addressing issue [#1762](#1762).
* [DOCUMENTATION] Improve documentation when installing UCX from a machine with restricted internet access ([#2690](#2690)). "A new section has been added to the `ADVANCED` installation section of the UCX library documentation, providing detailed instructions for installing UCX with a company-hosted PyPI mirror. This feature is intended for environments with restricted internet access, allowing users to bypass the public PyPI index and use a company-controlled mirror instead. Users will need to add all UCX dependencies to the company-hosted PyPI mirror and set the `PIP_INDEX_URL` environment variable to the mirror URL during installation. The solution also includes a prompt asking the user if their workspace blocks internet access. Additionally, the documentation has been updated to clarify that UCX requires internet access to connect to GitHub for downloading the tool, specifying the necessary URLs that need to be accessible. This update aims to improve the installation process for users with restricted internet access and provide clear instructions and prompts for installing UCX on machines with limited internet connectivity."

Dependency updates:

 * Updated sqlglot requirement from <25.22,>=25.5.0 to >=25.5.0,<25.23 ([#2687](#2687)).
 * Updated databricks-labs-lsql requirement from <0.12,>=0.5 to >=0.5,<0.13 ([#2688](#2688)).
 * Updated databricks-labs-blueprint requirement from <0.9,>=0.8 to >=0.8,<0.10 ([#2747](#2747)).
nfx added a commit that referenced this pull request Sep 26, 2024
@nfx
Release v0.38.0 (#2762)
d14a2cf 
* Added Py4j implementation of tables crawler to retrieve a list of HMS
tables in the assessment workflow
([#2579](#2579)). In this
release, we have added a Py4j implementation of a tables crawler to
retrieve a list of Hive Metastore tables in the assessment workflow. A
new `FasterTableScanCrawler` class has been introduced, which can be
used in the Assessment Job based on a feature flag to replace the old
Scala code, allowing for better logging during table scans. The existing
`assessment.crawl_tables` workflow now utilizes the new py4j crawler
instead of the scala one. Integration tests have been added to ensure
the functionality works correctly. The commit also includes a new method
for listing table names in the specified database and improvements to
error handling and logging mechanisms. The new Py4j tables crawler
enhances the functionality of the assessment workflow by improving error
handling, resulting in better logging and faster table scanning during
the assessment process. This change is part of addressing issue
[#2190](#2190) and was
co-authored by Serge Smertin.
* Added `create-ucx-catalog` cli command
([#2694](#2694)). A new CLI
command, `create-ucx-catalog`, has been added to create a catalog for
migration tracking that can be used across multiple workspaces. The
command creates a UCX catalog for tracking migration status and
artifacts, and is created by running `databricks labs ucx
create-ucx-catalog` and specifying the storage location for the catalog.
Relevant user documentation, unit tests, and integration tests have been
added for this command. The `assign-metastore` command has also been
updated to allow for the selection of a metastore when multiple
metastores are available in the workspace region. This change improves
the migration tracking feature and enhances the user experience.
* Added experimental `migration-progress-experimental` workflow
([#2658](#2658)). This
commit introduces an experimental workflow,
`migration-progress-experimental`, which refreshes the inventory for
various resources such as clusters, grants, jobs, pipelines, policies,
tables, TableMigrationStatus, and UDFs. The workflow can be triggered
using the `databricks labs ucx migration-progress` CLI command and uses
a new implementation of a Scala-based crawler, `TablesCrawler`, which
will eventually replace the current implementation. The new workflow is
a duplicate of most of the `assessment` pipeline's functionality but
with some differences, such as the use of `TablesCrawler`. Relevant user
documentation has been added, along with unit tests, integration tests,
and a screenshot of a successful staging environment run. The new
workflow is expected to run on a schedule in the future. This change
resolves [#2574](#2574) and
progresses [#2074](#2074).
* Added handling for `InternalError` in `Listing.__iter__`
([#2697](#2697)). This
release introduces improved error handling in the `Listing.__iter__`
method of the `Generic` class, located in the
`workspace_access/generic.py` file. Previously, only `NotFound`
exceptions were handled, but now both `InternalError` and `NotFound`
exceptions are caught and logged appropriately. This change enhances the
robustness of the method, which is responsible for listing objects of a
specific type and returning them as `GenericPermissionsInfo` objects. To
ensure the correct functionality, we have added new unit tests and
manual testing. The logging of the `InternalError` exception is properly
handled in the `GenericPermissionsSupport` class when listing serving
endpoints. This behavior is verified by the newly added test function
`test_internal_error_in_serving_endpoints_raises_warning` and the
updated `test_serving_endpoints_not_enabled_raises_warning`.
* Added handling for `PermissionDenied` when listing accessible
workspaces ([#2733](#2733)).
A new `can_administer` method has been added to the `Workspaces` class
in the `workspaces.py` file, which allows for more fine-grained control
over which users can administer workspaces. This method checks if the
user has access to a given workspace and is a member of the workspace's
`admins` group, indicating that the user has administrative privileges
for that workspace. If the user does not have access to the workspace or
is not a member of the `admins` group, the method returns `False`.
Additionally, error handling in the `get_accessible_workspaces` method
has been improved by adding a `PermissionDenied` exception to the list
of exceptions that are caught and logged. New unit tests have been added
for the `AccountWorkspaces` class of the
`databricks.labs.blueprint.account` module to ensure that the new method
is functioning as intended, specifically checking if a user is a
workspace administrator based on whether they belong to the `admins`
group. The linked issue
[#2732](#2732) is resolved
by this change. All changes have been manually and unit tested.
* Added static code analysis results to assessment dashboard
([#2696](#2696)). This
commit introduces two new tasks, `assess_dashboards` and
`assess_workflows`, to the existing assessment dashboard for identifying
migration problems in dashboards and workflows. These tasks analyze
embedded queries and notebooks for migration issues and collect direct
filesystem access patterns requiring attention. Upon completion, the
results are stored in the inventory database and displayed on the
Migration dashboard. Additionally, two new widgets, job/query problem
widgets and directfs access widgets, have been added to enhance the
dashboard's functionality by providing additional information related to
code compatibility and access control. Integration tests using mock data
have been added and manually tested to ensure the proper functionality
of these new features. This update improves the overall assessment and
compatibility checking capabilities of the dashboard, making it easier
for users to identify and address issues related to Unity Catalog
compatibility in their workflows and dashboards.
* Added unskip CLI command to undo a skip on schema or a table
([#2727](#2727)). This pull
request introduces a new CLI command, "unskip", which allows users to
reverse a previously applied `skip` on a schema or table. The `unskip`
command accepts a required `--schema` parameter and an optional
`--table` parameter. A new function, also named "unskip", has been
added, which takes the same parameters as the `skip` command. The
function checks for the required `--schema` parameter and creates a new
WorkspaceContext object to call the appropriate method on the
table_mapping object. Two new methods, `unskip_schema` and
"unskip_table_or_view", have been added to the HiveMapping class. These
methods remove the skip mark from a schema or table, respectively, and
handle exceptions such as NotFound and BadRequest. The
get_tables_to_migrate method has been updated to consider the unskipped
tables or schemas. Currently, the feature is tested manually and has not
been added to the user documentation.
* Added unskip CLI command to undo a skip on schema or a table
([#2734](#2734)). A new
`unskip` CLI command has been added to the project, which allows users
to remove the `skip` mark set by the existing `skip` command on a
specified schema or table. This command takes an optional `--table`
flag, and if not provided, it will unskip the entire schema. The new
functionality is accompanied by a unit test and relevant user
documentation, and addresses issue
[#1938](#1938). The
implementation includes the addition of the `unskip_table_or_view`
method, which generates the appropriate `ALTER TABLE/VIEW` statement to
remove the skip marker, and updates to the `unskip_schema` method to
include the schema name in the `ALTER SCHEMA` statement. Additionally,
exception handling has been updated to include `NotFound` and
`BadRequest` exceptions. This feature simplifies the process of undoing
a skip on a schema, table, or view in the Hive metastore, which
previously required manual editing of the Hive metastore properties.
* Assess source code as part of the assessment
([#2678](#2678)). This
commit introduces enhancements to the assessment workflow, including the
addition of two new tasks for evaluating source code from SQL queries in
dashboards and from notebooks/files in jobs and tasks. The existing
`databricks labs install ucx` command has been modified to incorporate
linting during the assessment. The `QueryLinter` class has been updated
to accept an additional argument for linting source code. These changes
have been thoroughly tested through integration tests to ensure proper
functionality. Co-authored by Eric Vergnaud.
* Bump astroid version, pylint version and drop our f-string workaround
([#2746](#2746)). In this
update, we have bumped the versions of astroid and pylint to 3.3.1 and
removed workarounds related to f-string inference limitations in
previous versions of astroid (< 3.3). These workarounds were necessary
for handling issues such as uninferrable sys.path values and the lack of
f-string inference in loops. We have also updated corresponding tests to
reflect these changes and improve the overall code quality and
maintainability of the project. These changes are part of a larger
effort to update dependencies and simplify the codebase by leveraging
the latest features of updated tools and removing obsolete workarounds.
* Delete temporary files when running solacc
([#2750](#2750)). This
commit includes changes to the `solacc.py` script to improve the linting
process for the `solacc` repository, specifically targeting the issue of
excessive temporary files that were exceeding CI storage capacity. The
modifications include linting the repository on a per-top-level
`solution` basis, where each solution resides within the top folders and
is independent of others. Post-linting, temporary files and directories
registered in `PathLookup` are deleted to enhance storage efficiency.
Additionally, this commit prepares for improving false positive
detection and introduces a new `SolaccContext` class that tracks various
aspects of the linting process, providing more detailed feedback on the
linting results. This change does not introduce new functionality or
modify existing functionality, but rather optimizes the linting process
for the `solacc` repository, maintaining CI storage capacity levels
within acceptable limits.
* Don't report direct filesystem access for API calls
([#2689](#2689)). This
release introduces enhancements to the Direct File System Access (DFSA)
linter, resolving false positives in API call reporting. The
`ws.api_client.do` call previously triggered inaccurate direct
filesystem access alerts, which have been addressed by adding new
methods to identify HTTP call parameters and specific API calls. The
linter now disregards DFSA patterns within known API calls, eliminating
false positives with relative URLs and duplicate advice from
SparkSqlPyLinter. Additionally, improvements in the `python_ast.py` and
`python_infer.py` files include the addition of `is_instance_of` and
`is_from_module` methods, along with safer inference methods to prevent
infinite recursion and enhance value inference. These changes
significantly improve the DFSA linter's accuracy and effectiveness when
analyzing code containing API calls.
* Enables cli cmd `databricks labs ucx create-catalog-schemas` to apply
catalog/schema acl from legacy hive_metastore
([#2676](#2676)). The new
release introduces a `databricks labs ucx create-catalog-schemas`
command, which applies catalog/schema Access Control List (ACL) from a
legacy hive_metastore. This command modifies the existing
`table_mapping` method to include a new `grants_crawler` parameter in
the `CatalogSchema` constructor, enabling the application of ACLs from
the legacy hive_metastore. A corresponding unit test is included to
ensure proper functionality. The `CatalogSchema` class in the
`databricks.labs.ucx.hive_metastore.catalog_schema` module has been
updated with a new argument `hive_acl` and the integration of the
`GrantsCrawler` class. The `GrantsCrawler` class is responsible for
crawling the Hive metastore and retrieving grants for catalogs, schemas,
and tables. The `prepare_test` function has been updated to include the
`hive_acl` argument and the `test_catalog_schema_acl` function has been
updated to test the new functionality, ensuring that the correct grant
statements are generated for a wider range of principals and
catalogs/schemas. These changes improve the functionality and usability
of the `databricks labs ucx create-catalog-schemas` command, allowing
for a more seamless transition from a legacy hive metastore.
* Fail `make test` on coverage below 90%
([#2682](#2682)). A new
change has been introduced to the pyproject.toml file to enhance the
codebase's quality and robustness by ensuring that the test coverage
remains above 90%. This has been accomplished by adding the
`--cov-fail-under=90` flag to the `test` and `coverage` scripts in the
`[tool.hatch.envs.default.scripts]` section. This flag will cause the
`make test` command to fail if the coverage percentage falls below the
specified value of 90%, ensuring that all new changes are thoroughly
tested and that the codebase maintains a minimum coverage threshold.
This is a best practice for maintaining code coverage and improving the
overall quality and reliability of the codebase.
* Fixed DFSA false positives from f-string fragments
([#2679](#2679)). This
commit addresses false positive DataFrame API Scanning Antipattern
(DFSA) reports in Python code, specifically in f-string fragments
containing forward slashes and curly braces. The linter has been updated
to accurately detect DFSA paths while avoiding false positives, and it
now checks for `JoinedStr` fragments in string constants. Additionally,
the commit rectifies issues with duplicate advices reported by
`SparkSqlPyLinter`. No new features or major functionality changes have
been introduced; instead, the focus has been on improving the
reliability and accuracy of DFSA detection. Co-authored by Eric
Vergnaud, this commit includes new unit tests and refinements to the
DFSA linter, specifically addressing false positive patterns like
`f"/Repos/{thing1}/sdk-{thing2}-{thing3}"`. To review these changes,
consult the updated tests in the
`tests/unit/source_code/linters/test_directfs.py` file, such as the new
test case for the f-string pattern causing false positives. By
understanding these improvements, you'll ensure your project adheres to
the latest updates, maintaining quality and accurate DFSA detection.
* Fixed failing integration tests that perform a real assessment
([#2736](#2736)). In this
release, we have made significant improvements to the integration tests
in the `assessment` workflow, by reducing the scope of the assessment
and improving efficiency and reliability. We have removed several object
creation functions and added a new function `populate_for_linting` for
linting purposes. The `populate_for_linting` function adds necessary
information to the installation context, and is used to ensure that the
integration tests still have the required data for linting. We have also
added a pytest fixture `populate_for_linting` to set up a minimal amount
of data in the workspace for linting purposes. These changes have been
implemented in the `test_workflows.py` file in the
integration/assessment directory. This will help to ensure that the
tests are not unnecessarily extensive, and that they are able to
accurately assess the functionality of the library.
* Fixed sqlglot crasher with 'drop schema ...' statement
([#2758](#2758)). In this
release, we have addressed a crash issue in the `sqlglot` library caused
by the `drop schema` statement. A new method, `_unsafe_lint_expression`,
has been introduced to prevent the crash by checking if the current
expression is a `Use`, `Create`, or `Drop` statement and updating the
`schema` attribute accordingly. The library now correctly handles the
`drop schema` statement and returns a `Deprecation` warning if the table
being processed is in the `hive_metastore` catalog and has been migrated
to the Unity Catalog. Unit tests have been added to ensure the correct
behavior of this code, and the linter for `from table` SQL has been
updated to parse and handle the `drop schema` statement without raising
any errors. These changes improve the library's overall reliability and
stability, allowing it to operate smoothly with the `drop schema`
statement.
* Fixed test failure:
`test_table_migration_job_refreshes_migration_status[regular-migrate-tables]`
([#2625](#2625)). In this
release, we have addressed two issues
([#2621](#2621) and
[#2537](#2537)) and fixed a
test failure in
`test_table_migration_job_refreshes_migration_status[regular-migrate-tables]`.
The `index` and `index_full_refresh` methods in `table_migrate.py` have
been updated to accept a new `force_refresh` flag. When set to `True`,
these methods will ensure that the migration status is up-to-date. This
change also affects the `ViewsMigrationSequencer` class, which now
passes `force_refresh=True` to the `index` method. Additionally, we have
fixed a test failure by reusing the `force_refresh` flag to ensure the
migration status is up-to-date. The `TableMigrationStatus` class in
`table_migration_status.py` has been modified to accept an optional
`force_refresh` parameter in the `index` method, and a unit test has
been updated to assert the correct behavior when updating the migration
status.
* Fixes error message
([#2759](#2759)). The `load`
method of the `mapping.py` file in the
`databricks/labs/ucx/hive_metastore` package has been updated to correct
an error message displayed when a `NotFound` exception is raised. The
previous message suggested running an incorrect command, which has been
updated to the correct one: "Please run: databricks labs ucx
create-table-mapping". This change does not add any new methods or alter
existing functionality, but instead focuses on improving the user
experience by providing accurate information when an error occurs. The
scope of this change is limited to updating the error message, and no
other modifications have been made.
* Fixes issue of circular dependency of migrate-location ACL
([#2741](#2741)). In this
release, we have resolved two issues
([#274](#274)
* Fixes source table alias dissapearance during migrate_views
([#2726](#2726)). This
release introduces a fix to preserve the alias for the source table
during the conversion of CREATE VIEW SQL from the legacy Hive metastore
to the Unity Catalog. The issue was addressed by adding a new test case,
`test_migrate_view_alias_test`, to verify the correct handling of table
aliases during migration. The changes also include a fix for the SQL
conversion and new test cases to ensure the correct handling of table
aliases, reflected in accurate SQL conversion. A new parameter, `alias`,
has been added to the Table class, and the `apply` method in the
`from_table.py` file has been updated. The migration process has been
updated to retain the original alias of the table. Unit tests have been
added and thoroughly tested to confirm the correctness of the changes,
including handling potential intermittent failures caused by external
dependencies.
* Py4j table crawler: suggestions/fixes for describing tables
([#2684](#2684)). This
release introduces significant improvements and fixes to the Py4J-based
table crawler, enhancing its capability to describe tables effectively.
The code for fetching table properties over the bridge has been updated,
and error tracing has been improved through individual fetching of each
table property and providing python backtrace on JVM side errors. Scala
`Option` values unboxing issues have been resolved, and a small
optimization has been implemented to detect partitioned tables without
materializing the collection. The table's `.viewText()` property is now
properly handled as a Scala `Option`. The `catalog` argument is now
explicitly verified to be `hive_metastore`, and a new static method
`_option_as_python` has been introduced for safely extracting values
from Scala `Option`. The `_describe` method has been refactored to
handle exceptions more gracefully and improved code readability. These
changes result in better functionality, error handling, logging, and
performance when describing tables within a specified catalog and
database. The linked issues
[#2658](#2658) and
[#2579](#2579) are
progressed through these updates, and appropriate testing has been
conducted to ensure the improvements' effectiveness.
* Speedup assessment workflow by making DBFS root table size calculation
parallel ([#2745](#2745)).
In this release, the assessment workflow for calculating DBFS root table
size has been optimized through the parallelization of the calculation
process, resulting in improved performance. This has been achieved by
updating the `pipelines_crawler` function in
`src/databricks/labs/ucx/contexts/workflow_task.py`, specifically the
`cached_property table_size_crawler`, to include an additional argument
`self.config.include_databases`. The `TablesCrawler` class has also been
modified to include a generic type parameter `Table`, enabling type
hinting and more robust type checking. Furthermore, the unit test file
`test_table_size.py` in the `hive_metastore` directory has been updated
to handle corrupt tables and invalid delta format errors more
effectively. Additionally, a new entry `databricks-pydabs` has been
added to the "known.json" file, potentially enabling better integration
with the `databricks-pydabs` library or providing necessary
configuration information for parallel processing. Overall, these
changes improve the efficiency and scalability of the codebase and
optimize the assessment workflow for calculating DBFS root table size.
* Updated databricks-labs-blueprint requirement from <0.9,>=0.8 to
>=0.8,<0.10
([#2747](#2747)). In this
update, the requirement for `databricks-labs-blueprint` has been updated
to version `>=0.8,<0.10` in the `pyproject.toml` file. This change
allows the project to utilize the latest features and bug fixes included
in version 0.9.0 of the `databricks-labs-blueprint` library. Notable
updates in version 0.9.0 consist of the addition of Databricks CLI
version as part of routed command telemetry and support for Unicode Byte
Order Mark (BOM) in file upload and download operations. Additionally,
various bug fixes and improvements have been implemented for the
`WorkspacePath` class, including the addition of `stat()` methods and
improved compatibility with different versions of Python.
* Updated databricks-labs-lsql requirement from <0.12,>=0.5 to
>=0.5,<0.13
([#2688](#2688)). In this
update, the version requirement of the `databricks-labs-lsql` library
has been changed from a version greater than or equal to 0.5 and less
than 0.12 to a version greater than or equal to 0.5 and less than 0.13.
This allows the project to utilize the latest version of
'databricks-labs-lsql', which includes new methods for differentiating
between a table that has never been written to and one with zero rows in
the MockBackend class. Additionally, the update adds support for various
filter types and improves testing coverage and reliability. The release
notes and changelog for the updated library are provided in the commit
message for reference.
* Updated documentation to explain the usage of collections and eligible
commands ([#2738](#2738)).
The latest update to the Databricks Labs Unified CLI (UCX) tool
introduces the `join-collection` command, which enables users to join
two or more workspaces into a collection, allowing for streamlined and
consolidated command execution across multiple workspaces. This feature
is available to Account admins on the Databricks account, Workspace
admins on the workspaces to be joined, and requires UCX installation on
the workspace. To run collection-eligible commands, users can simply
pass the `--run-as-collection=True` flag. This enhancement enhances the
UCX tool's functionality, making it easier to manage and execute
commands on multiple workspaces.
* Updated sqlglot requirement from <25.22,>=25.5.0 to >=25.5.0,<25.23
([#2687](#2687)). In this
pull request, we have updated the version requirement for the `sqlglot`
library in the pyproject.toml file. The previous requirement specified a
version greater than or equal to 25.5.0 and less than 25.22, but we have
updated it to allow for versions greater than or equal to 25.5.0 and
less than 25.23. This change allows us to use the latest version of
'sqlglot', while still ensuring compatibility with other dependencies.
Additionally, this pull request includes a detailed changelog from the
`sqlglot` repository, which provides information on the features, bug
fixes, and changes included in each version. This can help us understand
the scope of the update and how it may impact our project.
* [DOCUMENTATION] Improve documentation on using account profile for
`sync-workspace-info` cli command
([#2683](#2683)). The
`sync-workspace-info` CLI command has been added to the Databricks Labs
UCX package, which uploads the workspace configuration to all workspaces
in the Databricks account where the `ucx` tool is installed. This
feature requires Databricks Account Administrator privileges and is
necessary to create an immutable default catalog mapping for the table
migration process. It also serves as a prerequisite for the
`create-table-mapping` command. To utilize this command, users must
configure the Databricks CLI profile with access to the Databricks
account console, available at "accounts.cloud.databricks.com" or
"accounts.azuredatabricks.net". Additionally, the documentation for
using the account profile with the `sync-workspace-info` command has
been enhanced, addressing issue
[#1762](#1762).
* [DOCUMENTATION] Improve documentation when installing UCX from a
machine with restricted internet access
([#2690](#2690)). "A new
section has been added to the `ADVANCED` installation section of the UCX
library documentation, providing detailed instructions for installing
UCX with a company-hosted PyPI mirror. This feature is intended for
environments with restricted internet access, allowing users to bypass
the public PyPI index and use a company-controlled mirror instead. Users
will need to add all UCX dependencies to the company-hosted PyPI mirror
and set the `PIP_INDEX_URL` environment variable to the mirror URL
during installation. The solution also includes a prompt asking the user
if their workspace blocks internet access. Additionally, the
documentation has been updated to clarify that UCX requires internet
access to connect to GitHub for downloading the tool, specifying the
necessary URLs that need to be accessible. This update aims to improve
the installation process for users with restricted internet access and
provide clear instructions and prompts for installing UCX on machines
with limited internet connectivity."

Dependency updates:

* Updated sqlglot requirement from <25.22,>=25.5.0 to >=25.5.0,<25.23
([#2687](#2687)).
* Updated databricks-labs-lsql requirement from <0.12,>=0.5 to
>=0.5,<0.13 ([#2688](#2688)).
* Updated databricks-labs-blueprint requirement from <0.9,>=0.8 to
>=0.8,<0.10 ([#2747](#2747)).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
python Pull requests that update Python code
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@JCZuurmond @nfx