ERROR! Unexpected Exception, this is probably a bug: No module named 'msrest'

[Akasurde]

From @kenorb on Jul 12, 2020 21:32

SUMMARY

Ansible fails with the error that module doesn't exist.

I've tried to reinstall Ansible, but it didn't help.

% pip3 install --force-reinstall ansible

ISSUE TYPE

• Bug Report

COMPONENT NAME

• azure_rm

ANSIBLE VERSION

% ansible --version
ansible 2.9.10
  config file = ansible.cfg
  configured module search path = ['/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.8/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.8.3 (default, Jul  7 2020, 13:04:12) [Clang 11.0.3 (clang-1103.0.32.62)]

CONFIGURATION

[inventory]
enable_plugins = azure_rm, constructed, ini, script

OS / ENVIRONMENT

macOS 10.15.5

STEPS TO REPRODUCE

% ansible help -vvv
ERROR! Unexpected Exception, this is probably a bug: No module named 'msrest'

EXPECTED RESULTS

No errors.

ACTUAL RESULTS

Traceback (most recent call last):
  File "/usr/local/bin/ansible", line 123, in <module>
    exit_code = cli.run()
  File "/usr/local/lib/python3.8/site-packages/ansible/cli/adhoc.py", line 97, in run
    loader, inventory, variable_manager = self._play_prereqs()
  File "/usr/local/lib/python3.8/site-packages/ansible/cli/__init__.py", line 469, in _play_prereqs
    inventory = InventoryManager(loader=loader, sources=options['inventory'])
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 162, in __init__
    self.parse_sources(cache=True)
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 220, in parse_sources
    parse = self.parse_source(source, cache=cache)
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 266, in parse_source
    for plugin in self._fetch_inventory_plugins():
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 199, in _fetch_inventory_plugins
    plugin = inventory_loader.get(name)
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/loader.py", line 552, in get
    self._module_cache[path] = self._load_module_source(name, path)
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/loader.py", line 525, in _load_module_source
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/inventory/azure_rm.py", line 186, in <module>
    from msrest import ServiceClient, Serializer, Deserializer
ModuleNotFoundError: No module named 'msrest'

Then after installing msrest (pip3 install msrest), there is another error after that:

Traceback (most recent call last):
  File "/usr/local/bin/ansible", line 123, in <module>
    exit_code = cli.run()
  File "/usr/local/lib/python3.8/site-packages/ansible/cli/adhoc.py", line 97, in run
    loader, inventory, variable_manager = self._play_prereqs()
  File "/usr/local/lib/python3.8/site-packages/ansible/cli/__init__.py", line 469, in _play_prereqs
    inventory = InventoryManager(loader=loader, sources=options['inventory'])
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 162, in __init__
    self.parse_sources(cache=True)
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 220, in parse_sources
    parse = self.parse_source(source, cache=cache)
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 266, in parse_source
    for plugin in self._fetch_inventory_plugins():
  File "/usr/local/lib/python3.8/site-packages/ansible/inventory/manager.py", line 199, in _fetch_inventory_plugins
    plugin = inventory_loader.get(name)
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/loader.py", line 552, in get
    self._module_cache[path] = self._load_module_source(name, path)
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/loader.py", line 525, in _load_module_source
    spec.loader.exec_module(module)
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/usr/local/lib/python3.8/site-packages/ansible/plugins/inventory/azure_rm.py", line 187, in <module>
    from msrestazure import AzureConfiguration
ModuleNotFoundError: No module named 'msrestazure'

WORKAROUND

This solves the issue:

pip3 install msrest msrestazure

However Ansible shouldn't handle installing missing modules it-self?

Copied from original issue: ansible/ansible#70586

[Fred-sun]

@Akasurde Thanks for you contribution this issue. If you install ansible, of course, you also need to install his dependency package. Thank you very much!

[Fred-sun]

Another: This is may not a bug and should be closed. Thank you!

[kenorb]

Thanks for the explanation that this is by design. I'm happy for @Akasurde to close this issue.

[Fred-sun]

Thanks for the explanation that this is by design. I'm happy for @Akasurde to close this issue.

Thanks for your reply! I will close it!

[scrain777]

I cannot imagine what @Fred-sun is thinking: it is very broken for a Python package to not properly reference its dependencies so that they are automatically installed. Requiring the user to manually install dependencies is not an acceptable situation. This issue should be reopened.

[TheRojam]

I cannot imagine what @Fred-sun is thinking: it is very broken for a Python package to not properly reference its dependencies so that they are automatically installed. Requiring the user to manually install dependencies is not an acceptable situation. This issue should be reopened.

he is loves ms and i think he does not know how good software works ;)

[abelal83]

This is a joke right? How is someone new to ansible suppose to know deps need to be installed when it's not even documented? For example https://docs.ansible.com/ansible/latest/collections/azure/azcollection/azure_rm_storageaccount_module.html#ansible-collections-azure-azcollection-azure-rm-storageaccount-module just says to install using galaxy.

In comparison terraform just works our of the box.

[paultaiton]

This is a joke right? How is someone new to ansible suppose to know deps need to be installed when it's not even documented? For example https://docs.ansible.com/ansible/latest/collections/azure/azcollection/azure_rm_storageaccount_module.html#ansible-collections-azure-azcollection-azure-rm-storageaccount-module just says to install using galaxy.

In comparison terraform just works our of the box.

This issue was reported right in the middle of Ansible undergoing a transition from monolithic to modular architecture. Previously all of these modules were included with the base package, but that was split out into collections primarily to uncouple the development and release cycles of collections and the core engine.

https://galaxy.ansible.com/azure/azcollection

If you look at the install instructions, one of the very first things is to 'pip install -r requirements-azure.txt', that is what pulls in the python dependencies.

If you have recommendations on how Ansible should automatically pull in dependencies of installed collections, you can report that on the Ansible github repo, but this repo is not the right place for it since it's just a collection and is implementing the Ansible design.

Also, Terraform is MUCH more limited in scope and does not have to account for all of the use cases that Ansible supports which it cannot. Because Ansible is a general purpose automation tool and not limited to infrastructure components, it naturally requires that a different and unpredictable set of dependencies will need to be installed for different use cases. They're hardly appropriate for comparison.

[abelal83]

Thanks for responding Paul. I genuinely was not expecting a response as this issue has been closed. In fairness I had been struggling with ansible for days based on the docs which simply states to install collection. I had no idea there was git repo with further instructions on deps. My response here was out of frustration over the time I had spent and was not meant to cause offense. I apologise if I have. It would help incrediblely if the docs were updated to say go to this github repo and install requirements first, it certainly would have saved me hours. Most of the error mesages I got when googling indicated missing packages which I then used pip to install only to be met with more missing packages.

[paultaiton]

Thanks for responding Paul. I genuinely was not expecting a response as this issue has been closed. In fairness I had been struggling with ansible for days based on the docs which simply states to install collection. I had no idea there was git repo with further instructions on deps. My response here was out of frustration over the time I had spent and was not meant to cause offense. I apologise if I have. It would help incrediblely if the docs were updated to say go to this github repo and install requirements first, it certainly would have saved me hours. Most of the error mesages I got when googling indicated missing packages which I then used pip to install only to be met with more missing packages.

Understandable. It does make sense that the "NOTE" banner on the module docs should indicate that the link inside contains further instructions on dependencies. However I have no idea where that banner comes from. Since it appears in the docs for all of the collections (for instance AWS has the same message,) it looks to be coming from Ansible core. I'll raise the issue over there and see if it can be made more verbose.

[paultaiton]

@abelal83
I opened an issue ansible/ansible#74195 with the ansible core project. I don't know if that's the right place for it, but I asked for a modification that makes it more explicit that further dependencies may need to be satisfied.

[abelal83]

Thank you @paultaiton, really appreciate it. It should hopefully save others from the same frustration I had.