fix colorlog import

[maurerle]

Reason for the proposed changes

I get the error that colorlog is not valid.
This seems to have change in one of the latest versions of the package.

This issue occurs when installing with pip install -e .

INTEGRATION_TESTS_BRANCH=master
GSY_FRAMEWORK_BRANCH=master

[fievelk]

Thanks for the PR!
I believe that the issue here is mostly that gsy-e-sdk and gsy-e are enforcing two different versions of colorlog.

• In gsy-e-sdk we don't specify constraints: https://github.com/gridsingularity/gsy-e-sdk/blob/b41e7ed543a16c5cd03bc6858ef6bd336ae67eba/requirements/base.in#L4
• in gsy-e we use colorlog==4.7.2:

  gsy-e/requirements/base.in

  Line 6 in 54ec132

  colorlog==4.7.2

So, before we make this change, I think we should either bump colorlog to the latest version (6.6.0) or set gsy-e-sdk to use 4.7.2 too. I personally suggest to bump the version to the latest release and check if this creates incompatibilities between repos. I'll report the issue on slack.

[maurerle]

Well, wouldn't it be useful to losen those hard constraints anyway to allow flexible integration with other packages?
We had a lot of worries with the strict dependencies.

I'd remove the fixed dependencies so that an pip install -e . works either for users - where its needed you can still create the fixed state with fixed versions then, but other users have less problems using this project.

[fievelk]

Well, wouldn't it be useful to losen those hard constraints anyway to allow flexible integration with other packages? We had a lot of worries with the strict dependencies.

I'd remove the fixed dependencies so that an pip install -e . works either for users - where its needed you can still create the fixed state with fixed versions then, but other users have less problems using this project.

@maurerle I'd personally keep some form of constraint. If we loosen them completely, once the external library implements breaking changes it will cause exceptions on our end (also on production). What I would suggest is in-between: we can bump using a "compatible release" specifier (see PEP 440). For example, colorlog~=6.6.0 would allow us to automatically bump the micro version. We could even try with ~=6.6 and bump the minor version.

[maurerle]

Sounds good, but this is AFAIK not possible using pip-compile from pip-tools.

Maybe you can look at our fork to see how we managed to be flexible when needed by installing from base.in:
https://github.com/bc4p/gsy-e/blob/master/setup.py#L10
but also provide a fixed version with our additional dependencies as a feature.

I would love to integrate this into the upstream GSY.

Furthermore, if you update gsy-framework, you have to directly update the repositories depending on it accordingly - as you are installing gsy-framework directly from master as a dependency. This is a major source for breaking.
Especially, as gsy-framework also needs fixed dependencies.
This becomes unmanagable if you try to install gsy-e from an old commit, as the latest gsy-framework is still to be used.

I would suggest to use git submodules there - as this allows to specify an explicit commit-id.
This is quite generic and does not include the repository name if done right:
https://github.com/bc4p/gsy-e/blob/master/.gitmodules

What do you think about this?

[fievelk]

Sounds good, but this is AFAIK not possible using pip-compile from pip-tools.

@maurerle in my experience, this is possible. We should just update base.in here

gsy-e/requirements/base.in

Line 6 in 531423a

colorlog==4.7.2

to be

colorlog~=6.6.0 

Then running fab compile (or using pip-tools directly) should work. Of course, these changes should have been made on gsy-framework too, otherwise there might be other compatibility issues. Have you already given it a try?

Furthermore, if you update gsy-framework, you have to directly update the repositories depending on it accordingly - as you are installing gsy-framework directly from master as a dependency. This is a major source for breaking.

I personally agree with this, but there might be other more straightforward ways to deal with it than using submodules. One way could be using tags or commits' hashes to point to the relevant gsy-framework commit. But this requires some time and effort on the team. I've raised the issue and created a ticket for it.