[CT-2081] [Feature] Switch to black23 (or potentially adjust some other package version pins?)

[ckdake]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

dbt-core is currently pinned to black==22.10.0 in dev-requirements.txt. We're currently attempting to migrate our repo that contains our dbt-projects to black 23 and the dependency tree is unresolvable due to this pin in dbt-core.

#6425 is around for 22.10.0 -> 22.12.0, but there is not an automated one to 23 yet. (dependabot notes that it will make one once this PR is merged).

EDIT: this may not be black, it may be some other version pins. Discussion below.

Thanks!

Describe alternatives you've considered

No response

Who will this benefit?

Anyone keeping their dependencies up to date, with python code outside of just what is within dbt.

Are you interested in contributing this feature?

Sure! It'll be a messy commit and may involve rebasing a lot of inflight work due to style changes, so I don't want to do it in a vaccum.

Anything else?

No response

[dbeatty10]

Thanks for reaching out @ckdake !

I am not aware of any reasons why we couldn't use the newest version of black in our dev-requirements.txt, but I also wouldn't expect this to affect versioned installs of dbt-core from PyPI at all. e.g., I'd expect that you could use dbt projects just fine side-by-side with whichever version of black you want to install.

Could you tell me a little bit more how you are installing dbt-core? Are you installing from PyPI, installing from source in GitHub, or something else entirely?

I'd expect that a stand-alone install of dbt-core would look like the following, which would not specify or require any versions of black:

python -m pip install dbt-core==1.4.1

But it sounds like you are also installing the requirements within dev-requirements.txt, which I'd typically only expect if you are running dbt-core in development (like contributing a pull request).

Hopefully once I understand a bit more about your environment and how you are installing dbt we can figure out a path forward.

[ckdake]

Hmm, this may not be black then, but something else in the pile of dependency trees. dbt-core does look to be the one with hard dependency pins that exclude the most recent current versions of a handful of packages. Here is the dev-requirements.txt from our repo:

sqlfluff==2.0.0a4
sqlfluff-templater-dbt==2.0.0a4
psycopg2==2.9.5
seaborn==0.12.2
black==22.12.0
flake8==6.0.0
ipdb==0.13.11
pytest==7.2.1
pandas-stubs==1.5.3.230203
pylint==2.16.1
mypy==1.0.0
yamllint==1.29.0

and this is the error I'm getting when trying to bump up black to 23.1.0:

The conflict is caused by:
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.4.1 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.4.0 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.3.2 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.3.1 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.3.0 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.2.4 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.2.3 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.2.2 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
    pytest 7.2.1 depends on packaging
    dbt-core 1.2.1 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.2.0 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.1.3 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.1.2 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.1.1 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.1.0 depends on packaging<22.0 and >=20.9
    sqlfluff 2.0.0a4 depends on typing-extensions
    mypy 1.0.0 depends on typing-extensions>=3.10
    astroid 2.14.1 depends on typing-extensions>=4.0.0; python_version < "3.11"
    dbt-core 1.0.9 depends on typing-extensions<3.11 and >=3.7.4
    sqlfluff 2.0.0a4 depends on typing-extensions
    mypy 1.0.0 depends on typing-extensions>=3.10
    astroid 2.14.1 depends on typing-extensions>=4.0.0; python_version < "3.11"
    dbt-core 1.0.8 depends on typing-extensions<3.11 and >=3.7.4
    sqlfluff 2.0.0a4 depends on typing-extensions
    mypy 1.0.0 depends on typing-extensions>=3.10
    astroid 2.14.1 depends on typing-extensions>=4.0.0; python_version < "3.11"
    dbt-core 1.0.7 depends on typing-extensions<3.11 and >=3.7.4
    sqlfluff 2.0.0a4 depends on typing-extensions
    mypy 1.0.0 depends on typing-extensions>=3.10
    astroid 2.14.1 depends on typing-extensions>=4.0.0; python_version < "3.11"
    dbt-core 1.0.6 depends on typing-extensions<3.11 and >=3.7.4
    sqlfluff 2.0.0a4 depends on typing-extensions
    mypy 1.0.0 depends on typing-extensions>=3.10
    astroid 2.14.1 depends on typing-extensions>=4.0.0; python_version < "3.11"
    dbt-core 1.0.5 depends on typing-extensions<3.11 and >=3.7.4
    sqlfluff 2.0.0a4 depends on typing-extensions
    mypy 1.0.0 depends on typing-extensions>=3.10
    astroid 2.14.1 depends on typing-extensions>=4.0.0; python_version < "3.11"
    dbt-core 1.0.4 depends on typing-extensions<3.11 and >=3.7.4
    sqlfluff 2.0.0a4 depends on typing-extensions
    mypy 1.0.0 depends on typing-extensions>=3.10
    astroid 2.14.1 depends on typing-extensions>=4.0.0; python_version < "3.11"
    dbt-core 1.0.3 depends on typing-extensions<3.11 and >=3.7.4
    sqlfluff 2.0.0a4 depends on typing-extensions
    mypy 1.0.0 depends on typing-extensions>=3.10
    astroid 2.14.1 depends on typing-extensions>=4.0.0; python_version < "3.11"
    dbt-core 1.0.2 depends on typing-extensions<3.11 and >=3.7.4
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.0.1 depends on packaging<22.0 and >=20.9
    black 23.1.0 depends on packaging>=22.0
    pytest 7.2.1 depends on packaging
    dbt-core 1.0.0 depends on packaging<22.0 and >=20.9

[dbeatty10]

@ckdake I've got three responses for you!

1. Specific issue
2. General problem
3. Workaround in the meantime

Specific issue

Giving this a quick glance, it looks like the actual issue you are running into is this dependency in dbt-core 1.4.1:

• packaging<22.0

This conflicts with this one from black 23.1.0:

• packaging>=22.0

We've got an open issue and PR for that specific bound for packaging here:

• [CT-1688] [Feature] Remove packaging <22 version cap #6461
• remove packaging<22 cap #6463

When that issue is solved, it will help future you, but not current you.

General problem

In general, we are also looking to minimize these types of conflicts in the future:

• Revise our approach to pinning dependencies #6495

We'd welcome you in that discussion!

Workaround in the meantime

black is a CLI that can installed in a virtual environment with pipx.

So you can do something like the following:

1. Install pipx
2. Install black using pipx
3. Install your dbt adapter(s) of choice using pip
4. Check the versions of black and dbt

Install pipx

https://pypa.github.io/pipx/installation/

python3 -m pip install --user pipx
python3 -m pipx ensurepath

Install black using pipx

https://pypa.github.io/pipx/getting-started/

Latest version of black:

pipx install black

Or if you want a specific version:

pipx install black==22.12.0

Note: in this case, you'd remove black from your requirements.txt and just rely upon the pipx installation instead.

Install your dbt adapter(s) of choice using pip

We're just installing dbt-postgres here, but you can add whichever adapter(s) are relevant to your situation. All dbt-* adapters will include dbt-core as a dependency.

python -m pip install dbt-postgres

Check the versions of black and dbt

black --version
dbt --version

So I think we have paths forward for both the specific thing you are experiencing as well as the general problem. In the meantime, please let me know if that workaround helps you at all!

[ckdake]

Thanks for the detailed response! Good to see the # 1 and # 2 above as in-flight, both are good things to have.

pipx might add a little more complexity to our stack than this "getting fixed upstream" will, and it's not super time-sensitive. I'll close this issue out, keep an eye on your links, and chase down the pipx approach if this becomes more urgent on our side.

[gshank]

I removed the packaging pin in pr #6838 last week. Do we want to backport it?

[dbeatty10]

I removed the packaging pin in pr #6838 last week. Do we want to backport it?

@gshank Ultimately, I think some combination of @jtcohen6 @leahwicz @nathaniel-may would need to approve backporting it.