cekit build fails with AttributeError: 'NoneType' object has no attribute 'keys' #48

vorburger · 2018-12-07T00:48:50Z

On Fedora 28:

dnf copr enable @cekit/cekit
sudo dnf copr enable @cekit/cekit
git clone [email protected]:jboss-container-images/openjdk.git
cd openjdk
cekit build
2018-12-07 01:45:14,980 cekit        INFO     Generating files for docker engine.
2018-12-07 01:45:14,993 cekit        INFO     Initializing image descriptor...
2018-12-07 01:45:16,894 cekit        WARNING  The way of defining repository 'jboss-os' is deprecated. Convert it to an URL based repository object. Consult Cekit docs, for more details.
Traceback (most recent call last):
  File "/usr/bin/cekit", line 11, in <module>
    load_entry_point('cekit==2.2.2', 'console_scripts', 'cekit')()
  File "/usr/lib/python3.6/site-packages/cekit/cli.py", line 289, in run
    Cekit().parse().run()
  File "/usr/lib/python3.6/site-packages/cekit/cli.py", line 223, in run
    discover_modules(os.path.join(self.args.target, 'repo'))
  File "/usr/lib/python3.6/site-packages/cekit/module.py", line 92, in discover_modules
    'module.yaml'))))
  File "/usr/lib/python3.6/site-packages/cekit/descriptor/module.py", line 31, in __init__
    self._prepare()
  File "/usr/lib/python3.6/site-packages/cekit/descriptor/image.py", line 89, in _prepare
    self._descriptor['packages'] = Packages(self._descriptor['packages'])
  File "/usr/lib/python3.6/site-packages/cekit/descriptor/packages.py", line 59, in __init__
    self._prepare()
  File "/usr/lib/python3.6/site-packages/cekit/descriptor/packages.py", line 63, in _prepare
    for x in self._descriptor.get('repositories', [])]
  File "/usr/lib/python3.6/site-packages/cekit/descriptor/packages.py", line 63, in <listcomp>
    for x in self._descriptor.get('repositories', [])]
  File "/usr/lib/python3.6/site-packages/cekit/descriptor/packages.py", line 77, in __init__
    descriptor = self._create_repo_object(descriptor)
  File "/usr/lib/python3.6/site-packages/cekit/descriptor/packages.py", line 112, in _create_repo_object
    descriptor['url']['repository'] = self._get_repo_url(descriptor)
  File "/usr/lib/python3.6/site-packages/cekit/descriptor/packages.py", line 121, in _get_repo_url
    configured_repository_names = configured_repositories.keys()
AttributeError: 'NoneType' object has no attribute 'keys'

The text was updated successfully, but these errors were encountered:

rcernich · 2018-12-12T17:52:29Z

@jmtd, we should work on removing package.repositories from all modules. In addition to that, I've created cekit/cekit#344.

rcernich · 2018-12-12T19:16:30Z

To workaround the issue, you can create a ~/.cekit/config file which contains:

[repositories]
jboss-os=dummy-url
jboss-rhscl=dummy-url

That said, the next hurdle will be locating community artifacts for the following:

hawkular-javaagent-1.0.1.Final-redhat-2-shaded.jar
jolokia-jvm-1.5.0.redhat-1-agent.jar
jmx_prometheus_javaagent-0.3.1.redhat-00006.jar

It looks like community artifacts can all be downloaded from Maven Central: hawkular-javaagent-1.0.1.Final-shaded.jar, jolokia-jvm-1.5.0-agent.jar, jmx_prometheus_javaagent-0.3.1.jar.

As a workaround, adding and using a community-artifacts-overrides.yaml file should work:

artifacts:
- name: hawkular-javaagent-1.0.1.Final-redhat-2-shaded.jar
  url: https://repo1.maven.org/maven2/org/hawkular/agent/hawkular-javaagent/1.0.1.Final/hawkular-javaagent-1.0.1.Final-shaded.jar
  md5: 1f55173d84e4ac52d0ee542e251d8b02
- name: jolokia-jvm-1.5.0.redhat-1-agent.jar
  url: https://repo1.maven.org/maven2/org/jolokia/jolokia-jvm/1.5.0/jolokia-jvm-1.5.0-agent.jar
  md5: 64cb886a4a1cf5906e0f33afd76243a8
- name: jmx_prometheus_javaagent-0.3.1.redhat-00006.jar
  url: https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.3.1/jmx_prometheus_javaagent-0.3.1.jar
  md5: 4f08e57afb4088469904956237c0cf8e

Then running with:
$ cekit --overrides-file community-artifacts-overrides.yaml build

Note, using this method, the files in the image will still be named with -redhat- in them, but they will be the community versions (you can verify by expanding the jars and looking at the META-INF/maven/... files).

@jmtd, if we're going to make the default versions of the image.yaml file community oriented, we should probably do the same with the upstream modules (jolokia, hawkular, jmx_exporter). That said, there's probably some work to be done in cekit to help ensure we're using the "proper" file names for these types of resources (e.g. jolokia-jvm-1.5.0-agent.jar instead of jolokia-jvm-1.5.0.redhat-1-agent.jar). @goldmann, @dbecvarik, you're probably interested in this as well, from a cekit perspective.

goldmann · 2018-12-13T08:53:47Z

Great comment, thanks @rcernich.

I was always thinking that the best way to approach is to use the name and target keys in a smart way. The name key should be just a unique identifier of the artifact across all of them.

Let's take Jolokia as the example. It's unlikely that we will have two Jolokia JARs to be added to the image, so the name key could be just: jolokia. But in this case the artifact fetched will be saved under the jolokia name too, which is something we don't want.

This is where the target key steps in and helps. Consider the example below:

artifacts:
- name: jolokia
  target: jolokia-jvm-agent.jar
  url: https://repo1.maven.org/maven2/org/jolokia/jolokia-jvm/1.5.0/jolokia-jvm-1.5.0-agent.jar
  md5: 64cb886a4a1cf5906e0f33afd76243a8

In such case, the module could use/expect the jolokia-jvm-agent.jar filename and it would be "stable". Additionally, the artifact would be easy to override, since we could use the jolokia name/key.

The only one caveat I can think of is using a filename in the image that is different from the name it was release under. Considering pros and cons of above solution I think this is something we could trade for the ease of maintenance.

If the original artifact provides a name (via http, or via metadata in Brew), we could implement adding a symlink to the artifact itself. I would not be used by the module, but it would be there, if someone would like to poke in the image to understand how it is built.

jmtd · 2018-12-17T10:50:48Z

The build issue has been fixed in cekit 2.2.4 I believe (and I cannot reproduce this problem with cekit 2.2.4 locally), closing accordingly. Please re-open if this does still happen for you with ≥ 2.2.4.

There's lots of other relevant TODO bits in mentioned in comments in this issue that I will break out into separate issues soon.

Thanks

pennpeng · 2018-12-21T06:54:48Z

@jmtd

# cekit build
2018-12-21 14:53:58,252 cekit        INFO     Generating files for docker engine.
2018-12-21 14:53:58,271 cekit        INFO     Initializing image descriptor...
2018-12-21 14:53:58,272 cekit        INFO     Preparing resource 'cct_module'
2018-12-21 14:54:05,411 cekit        WARNING  The way of defining repository 'jboss-os' is deprecated. Convert it to an URL based repository object. Consult Cekit docs, for more details.
Traceback (most recent call last):
  File "/usr/bin/cekit", line 9, in <module>
    load_entry_point('cekit==2.2.4', 'console_scripts', 'cekit')()
  File "/usr/lib/python2.7/site-packages/cekit/cli.py", line 289, in run
    Cekit().parse().run()
  File "/usr/lib/python2.7/site-packages/cekit/cli.py", line 223, in run
    discover_modules(os.path.join(self.args.target, 'repo'))
  File "/usr/lib/python2.7/site-packages/cekit/module.py", line 92, in discover_modules
    'module.yaml'))))
  File "/usr/lib/python2.7/site-packages/cekit/descriptor/module.py", line 31, in __init__
    self._prepare()
  File "/usr/lib/python2.7/site-packages/cekit/descriptor/image.py", line 89, in _prepare
    self._descriptor['packages'] = Packages(self._descriptor['packages'])
  File "/usr/lib/python2.7/site-packages/cekit/descriptor/packages.py", line 59, in __init__
    self._prepare()
  File "/usr/lib/python2.7/site-packages/cekit/descriptor/packages.py", line 63, in _prepare
    for x in self._descriptor.get('repositories', [])]
  File "/usr/lib/python2.7/site-packages/cekit/descriptor/packages.py", line 77, in __init__
    descriptor = self._create_repo_object(descriptor)
  File "/usr/lib/python2.7/site-packages/cekit/descriptor/packages.py", line 112, in _create_repo_object
    descriptor['url']['repository'] = self._get_repo_url(descriptor)
  File "/usr/lib/python2.7/site-packages/cekit/descriptor/packages.py", line 121, in _get_repo_url
    configured_repository_names = configured_repositories.keys()
AttributeError: 'NoneType' object has no attribute 'keys'
[root@okd311-1 openjdk]# cekit --version
2.2.4

goldmann · 2019-01-04T13:33:20Z

What is the repository/branch you use? I cannot reporoduce it with the openjdk repository and the develop branch (https://github.com/jboss-container-images/openjdk):

➜  openjdk git:(develop) ✗ cekit build               
2019-01-04 14:32:09,145 cekit        INFO     Generating files for docker engine.
2019-01-04 14:32:09,153 cekit        INFO     Initializing image descriptor...
2019-01-04 14:32:09,154 cekit        INFO     Preparing resource 'cct_module'
2019-01-04 14:32:11,074 cekit        WARNING  The way of defining repository 'jboss-os' is deprecated. Convert it to an URL based repository object. Consult Cekit docs, for more details.
2019-01-04 14:32:11,297 cekit        INFO     Handling artifacts...
2019-01-04 14:32:11,297 cekit        INFO     Preparing resource 'hawkular-javaagent-1.0.1.Final-redhat-2-shaded.jar'
2019-01-04 14:32:11,316 cekit        INFO     Preparing resource 'jolokia-jvm-1.5.0.redhat-1-agent.jar'
2019-01-04 14:32:11,317 cekit        INFO     Preparing resource 'jmx_prometheus_javaagent-0.3.1.redhat-00006.jar'
2019-01-04 14:32:11,363 cekit        INFO     Rendering Dockerfile...
2019-01-04 14:32:11,497 cekit        INFO     Using Docker builder to build the image.
2019-01-04 14:32:11,497 cekit        INFO     Building container image...
2019-01-04 14:32:11,629 cekit        INFO     Docker: Step 1/58 : FROM centos:7
2019-01-04 14:32:11,630 cekit        INFO     Docker: ---> 75835a67d134
2019-01-04 14:32:11,630 cekit        INFO     Docker: Step 2/58 : USER root

[SNIP]
➜  openjdk git:(develop) ✗ cekit --version
2.2.4

goldmann · 2019-01-04T13:34:50Z

One thing that comes to my mind is the difference in the ~/.cekit/config configuration file. Do you have [repositories] section there with the jboss-os key?

vorburger · 2019-02-13T02:27:25Z

@jmtd nope, this is still NOK with today's cekit --version = 2.2.5, for me...

To workaround the issue, you can create a ~/.cekit/config file which contains:

whatever that black magic does, it does indeed get around the original probably.

As a workaround, adding and using a community-artifacts-overrides.yaml file should work:
$ cekit --overrides-file community-artifacts-overrides.yaml build

nope, when applied exactly as documented above that was still NOK, but I figured renames...

How about #58 to make it possible for anyone without a Ph.D. in cekit-oloy to "just build" this project?

vorburger · 2019-02-13T17:32:18Z

From my side, we could close this issue when #58 is merged.

jmtd · 2019-02-19T15:05:35Z

I managed to reproduce this with cekit 2.2.5 (I suspect I had stuff in ~/.cekit/config which was why I didn't hit the problem last time), and then 2.2.6 appears to fix it. Can you please retry with 2.2.6? We could avoid merging the short-term-fix #58 in that case.

jmtd · 2019-09-09T13:06:02Z

I think this was fixed in cekit 2.2.6 as per my last comment; I've had no further reports of problems (and we're on cekit 3.4.0 now). Please re-open if still a problem

vorburger mentioned this issue Dec 7, 2018

build failed #44

Closed

vorburger changed the title ~~cekit build with AttributeError: 'NoneType' object has no attribute 'keys'~~ cekit build fails with AttributeError: 'NoneType' object has no attribute 'keys' Dec 7, 2018

rcernich mentioned this issue Dec 12, 2018

Process repository definitions after initializing image.yaml (generate.init()) cekit/cekit#344

Closed

jmtd closed this as completed Dec 17, 2018

jmtd reopened this Dec 26, 2018

vorburger mentioned this issue Jan 22, 2019

Merge fabric8io-images/s2i project with jboss-container-images/openjdk fabric8io-images/s2i#218

Open

vorburger mentioned this issue Feb 12, 2019

artefacts in RHEL7-based images prevent easily building the images #57

Closed

vorburger added a commit to vorburger/openjdk that referenced this issue Feb 13, 2019

add overrides.yaml and update README (rh-openjdk#48)

4d569fa

vorburger added a commit to vorburger/openjdk that referenced this issue Feb 13, 2019

add overrides.yaml and update README (rh-openjdk#48)

4c4eab6

This was referenced Feb 13, 2019

add overrides.yaml and update README (#48) #58

Closed

Avoid requiring a weird ~/.cekit/config with dummy-url (which cause a WARNING) #60

Closed

vorburger added a commit to vorburger/openjdk that referenced this issue Feb 13, 2019

add artefact-overrides.yaml and update README (rh-openjdk#48)

3caafd7

vorburger mentioned this issue Feb 13, 2019

Long term solution to build CE images without overrides #67

Closed

jmtd closed this as completed Sep 9, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cekit build fails with AttributeError: 'NoneType' object has no attribute 'keys' #48

cekit build fails with AttributeError: 'NoneType' object has no attribute 'keys' #48

vorburger commented Dec 7, 2018

rcernich commented Dec 12, 2018

rcernich commented Dec 12, 2018

goldmann commented Dec 13, 2018

jmtd commented Dec 17, 2018

pennpeng commented Dec 21, 2018 •

edited

Loading

goldmann commented Jan 4, 2019

goldmann commented Jan 4, 2019

vorburger commented Feb 13, 2019

vorburger commented Feb 13, 2019

jmtd commented Feb 19, 2019

jmtd commented Sep 9, 2019

cekit build fails with AttributeError: 'NoneType' object has no attribute 'keys' #48

cekit build fails with AttributeError: 'NoneType' object has no attribute 'keys' #48

Comments

vorburger commented Dec 7, 2018

rcernich commented Dec 12, 2018

rcernich commented Dec 12, 2018

goldmann commented Dec 13, 2018

jmtd commented Dec 17, 2018

pennpeng commented Dec 21, 2018 • edited Loading

goldmann commented Jan 4, 2019

goldmann commented Jan 4, 2019

vorburger commented Feb 13, 2019

vorburger commented Feb 13, 2019

jmtd commented Feb 19, 2019

jmtd commented Sep 9, 2019

pennpeng commented Dec 21, 2018 •

edited

Loading