Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

juniper.device.sofware: MX vmhost device software upgrade with reboot option fails with error raise EzErrors.RpcTimeoutError(self, rpc_cmd_e.tag, self.timeout) #705

Open
chidanandpujar opened this issue Dec 10, 2024 · 1 comment
Open

juniper.device.sofware: MX vmhost device software upgrade with reboot option fails with error raise EzErrors.RpcTimeoutError(self, rpc_cmd_e.tag, self.timeout) #705

chidanandpujar opened this issue Dec 10, 2024 · 1 comment
Assignees
@chidanandpujar

Comments

@chidanandpujar
Copy link
Collaborator

Issue Type

  • Bug Report

Module Name

juniper.device.software with reboot.

juniper.device collection and Python libraries version

~/ansible_docker_test1/ansible-junos-stdlib/tests$ sudo podman run -it --rm ansible:v1.0.6 pip list
Executing given commands
Package               Version
--------------------- -------
ansible               11.1.0
ansible-core          2.18.1
bcrypt                4.2.1
cffi                  1.17.1
colorama              0.4.6
configparser          7.1.0
cryptography          44.0.0
future                1.0.0
icdiff                1.9.1
Jinja2                3.1.4
jsnapy                1.3.8
junos-eznc            2.7.2
jxmlease              1.0.3
lxml                  5.3.0
MarkupSafe            3.0.2
ncclient              0.6.15
packaging             24.2
paramiko              3.5.0
pip                   24.3.1
pipdeptree            2.24.0
pycparser             2.22
PyNaCl                1.5.0
pyparsing             3.2.0
pyserial              3.5
PyYAML                6.0.2
resolvelib            1.0.1
scp                   0.15.0
setuptools            75.6.0
six                   1.17.0
transitions           0.9.2
xmltodict             0.14.2
yamlordereddictloader 0.4.2

~/ansible_docker_test1/ansible-junos-stdlib/tests$ sudo podman run -it --rm ansible:v1.0.6 ansible-galaxy collection list
Executing given commands

# /root/.ansible/collections/ansible_collections
Collection                               Version
---------------------------------------- -------
juniper.device                           1.0.6  

# /usr/local/lib/python3.12/site-packages/ansible_collections
Collection                               Version
---------------------------------------- -------
amazon.aws                               9.0.0  
ansible.netcommon                        7.1.0  
ansible.posix                            1.6.2  
ansible.utils                            5.1.2  
ansible.windows                          2.5.0  
arista.eos                               10.0.1 
awx.awx                                  24.6.1 
azure.azcollection                       3.1.0  
check_point.mgmt                         6.2.1  
chocolatey.chocolatey                    1.5.3  
cisco.aci                                2.10.1 
cisco.asa                                6.0.0  
cisco.dnac                               6.25.0 
cisco.intersight                         2.0.20 
cisco.ios                                9.0.3  
cisco.iosxr                              10.2.2 
cisco.ise                                2.9.6  
cisco.meraki                             2.18.3 
cisco.mso                                2.9.0  
cisco.nxos                               9.2.1  
cisco.ucs                                1.14.0 
cloud.common                             4.0.0  
cloudscale_ch.cloud                      2.4.0  
community.aws                            9.0.0  
community.ciscosmb                       1.0.9  
community.crypto                         2.22.3 
community.digitalocean                   1.27.0 
community.dns                            3.1.0  
community.docker                         4.1.0  
community.general                        10.1.0 
community.grafana                        2.1.0  
community.hashi_vault                    6.2.0  
community.hrobot                         2.0.2  
community.library_inventory_filtering_v1 1.0.2  
community.libvirt                        1.3.0  
community.mongodb                        1.7.8  
community.mysql                          3.11.0 
community.network                        5.1.0  
community.okd                            4.0.0  
community.postgresql                     3.9.0  
community.proxysql                       1.6.0  
community.rabbitmq                       1.3.0  
community.routeros                       3.1.0  
community.sap_libs                       1.4.2  
community.sops                           2.0.0  
community.vmware                         5.2.0  
community.windows                        2.3.0  
community.zabbix                         3.2.0  
containers.podman                        1.16.2 
cyberark.conjur                          1.3.1  
cyberark.pas                             1.0.30 
dellemc.enterprise_sonic                 2.5.1  
dellemc.openmanage                       9.9.0  
dellemc.powerflex                        2.5.0  
dellemc.unity                            2.0.0  
f5networks.f5_modules                    1.32.1 
fortinet.fortimanager                    2.8.2  
fortinet.fortios                         2.3.8  
google.cloud                             1.4.1  
grafana.grafana                          5.6.0  
hetzner.hcloud                           4.2.2  
ibm.qradar                               4.0.0  
ibm.spectrum_virtualize                  2.0.0  
ibm.storage_virtualize                   2.5.0  
ieisystem.inmanage                       3.0.0  
infinidat.infinibox                      1.4.5  
infoblox.nios_modules                    1.7.1  
inspur.ispim                             2.2.3  
junipernetworks.junos                    9.1.0  
kaytus.ksmanage                          2.0.0  
kubernetes.core                          5.0.0  
kubevirt.core                            2.1.0  
lowlydba.sqlserver                       2.3.4  
microsoft.ad                             1.7.1  
netapp.cloudmanager                      21.24.0
netapp.ontap                             22.13.0
netapp.storagegrid                       21.13.0
netapp_eseries.santricity                1.4.1  
netbox.netbox                            3.20.0 
ngine_io.cloudstack                      2.5.0  
openstack.cloud                          2.3.0  
ovirt.ovirt                              3.2.0  
purestorage.flasharray                   1.32.0 
purestorage.flashblade                   1.19.1 
sensu.sensu_go                           1.14.0 
splunk.es                                4.0.0  
telekom_mms.icinga_director              2.2.1  
theforeman.foreman                       4.2.0  
vmware.vmware                            1.7.1  
vmware.vmware_rest                       4.3.0  
vultr.cloud                              1.13.0 
vyos.vyos                                5.0.0  
wti.remote                               1.0.10

OS / Environment

JUNOS
MX vmhost.

Summary

MX device upgrade with reboot option fails with error.
The full traceback is:
File "/tmp/ansible_juniper.device.software_payload_3egabt4c/ansible_juniper.device.software_payload.zip/ansible_collections/juniper/device/plugins/modules/software.py", line 779, in main
got = junos_module.sw.reboot(0, None, all_re, None, install_params.get('vmhost'))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/jnpr/junos/utils/sw.py", line 1248, in reboot
raise err
File "/usr/local/lib/python3.12/site-packages/jnpr/junos/utils/sw.py", line 1246, in reboot
return self._system_operation(cmd, in_min, at, all_re, other_re, vmhost)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/jnpr/junos/utils/sw.py", line 1169, in _system_operation
raise err
File "/usr/local/lib/python3.12/site-packages/jnpr/junos/utils/sw.py", line 1149, in _system_operation
rsp = self.rpc(cmd, ignore_warning=True, normalize=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/jnpr/junos/rpcmeta.py", line 386, in call
return self._junos.execute(rpc_cmd, **kvargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/jnpr/junos/decorators.py", line 63, in wrapper
result = function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/jnpr/junos/decorators.py", line 31, in wrapper
return function(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/jnpr/junos/device.py", line 843, in execute
raise EzErrors.RpcTimeoutError(self, rpc_cmd_e.tag, self.timeout)

Steps to reproduce

---
- name: Test juniper.device.software module
  hosts: all
  gather_facts: false
  vars:
    wait_time: 3600
    pkg_dir: /var/tmp/
    os_version: 22.2R3
    os_package: junos-vmhost-install-mx-x86-64-22.2R3-S3.18.tgz 
    log_dir: /var/log/

  tasks:
    - name: Checking NETCONF connectivity
      ansible.builtin.wait_for:
        host: "{{ ansible_ssh_host }}"
        port: 830
        timeout: 5
    - name: Install Junos OS package
      juniper.device.software:
        vmhost: true
        reboot: true
        no_copy: true
        all_re: false
        version: false
        validate: false
        package: "{{ pkg_dir }}/{{ os_package }}"
        logfile: "{{ log_dir }}/software.log"
      register: test1
      notify:
        - Wait_reboot

    - name: Print response
      ansible.builtin.debug:
        var: test1

    - name: Check TEST - 1
      ansible.builtin.assert:
        that:
          - test1.failed == false

  handlers:
    - name: Wait_reboot
      ansible.builtin.wait_for:
        host: "{{ ansible_ssh_host }}"
        port: 830
        timeout: "{{ wait_time }}"
      when: not test1.check_mode


sudo podman run -it --rm -v $PWD:/project  ansible:v1.0.6 ansible-playbook -i inventory pb.juniper_junos_software.yml -vvvvv

Actual results

HDD01 (SFSA100GM3AA4TO-C-LB-64A-JUN)\nBoot0002* ETH00 (1C-9C-8C-18-C8-60)\neth-lldp-disable.sh already present\neth-lldp-stop.sh already present\neth-link-down-on-close-enable.sh already present\nmk-mtre-downgrade.sh already present\ndowngrade_cleanup_file_list  already present\n0\n/tmp/partdisk-0h6Rqe/unpack/tpm-upgrade_install.sh: line 59: /usr/sbin/TPMFactoryUpd: No such file or directory\n... upgrade complete.\nA REBOOT IS REQUIRED TO LOAD THIS SOFTWARE CORRECTLY.\nUse the 'request vmhost reboot' command to reboot the system.\n Reboot failed. It may not have been initiated."
}

PLAY RECAP ****************************************************************************************************************************************************************
local_connection_testcases : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
@chidanandpujar chidanandpujar self-assigned this Dec 10, 2024
@chidanandpujar
Copy link
Collaborator Author

chidanandpujar commented Feb 7, 2025 •
edited
Loading

issue is not reproducible now and playbook works fine.


---
- name: Test juniper.device.software module
  hosts: all
  gather_facts: false
  vars:
    wait_time: 3600
    pkg_dir: /var/tmp/
    os_version: 
    os_package: 
    log_dir: /var/log/

  tasks:
    - name: Checking NETCONF connectivity
      ansible.builtin.wait_for:
        host: "{{ ansible_ssh_host }}"
        port: 830
        timeout: 5
    - name: Install Junos OS package
      juniper.device.software:
        vmhost: true
        reboot: True 
        no_copy: true
        all_re: false
        validate: false
        package: "{{ pkg_dir }}/{{ os_package }}"
        logfile: "{{ log_dir }}/software.log"
      register: test1
      ignore_errors: true
      notify:
        - Wait_reboot

    - name: Print response
      ansible.builtin.debug:
        var: test1

    - name: Check TEST - 1
      ansible.builtin.assert:
        that:
          - test1.failed == false

  handlers:
    - name: Wait_reboot
      ansible.builtin.wait_for:
        host: "{{ ansible_ssh_host }}"
        port: 830
        timeout: "{{ wait_time }}"
        delay: 300 
      when: not test1.check_mode

Logs:

###########################################\nDone with pfe_app Install...\nCreating moint points for JUNOSFS and VMFS: /junos, /vm\nUpdating boot partition...\nsh /tmp/pkg-KRY/unpack/install///scripts/vmhost-platform-grub-install.sh /tmp/partdisk-7L5YUp/espfs/EFI/BOOT/grub.cfg.b /tmp/pkg-KRY/unpack/install// jrootb_P 8 /tmp/partdisk-7L5YUp/espfs/EFI/BOOT\nMX10001\nUpgrade complete.\nRestore Last Known good BIOS..\nRestoring Last Known Good\nBootNext: 0000\nBootCurrent: 0000\nTimeout: 5 seconds\nBootOrder: 0000,0001,0002\nBoot0000* HDD00 (SFSA100GM3AA4TO-C-LB-64A-JUN)\nBoot0001* HDD01 (SFSA100GM3AA4TO-C-LB-64A-JUN)\nBoot0002* ETH00 (1C-9C-8C-18-C8-60) \neth-lldp-disable.sh already present\nmk-mtre-downgrade.sh already present\ndowngrade_cleanup_file_list  already present\n0\n... upgrade complete.\nA REBOOT IS REQUIRED TO LOAD THIS SOFTWARE CORRECTLY.\nUse the 'request vmhost reboot' command to reboot the system.\n Reboot successfully initiated. Reboot message: \nInitiating vmhost reboot...  \n\n\nshutdown: [pid 23571]\nShutdown NOW!\n"
    }
}

TASK [Check TEST - 1] *****************************************************************************************************************************************************
task path: ansible_docker_test1/ansible-junos-stdlib/tests/pb.juniper_junos_software_reboot.yml:36
ok: [local_connection_testcases] => {
    "changed": false,
    "msg": "All assertions passed"
}
NOTIFIED HANDLER Wait_reboot for local_connection_testcases

RUNNING HANDLER [Wait_reboot] *********************************************************************************************************************************************
task path: ansible_docker_test1/ansible-junos-stdlib/tests/pb.juniper_junos_software_reboot.yml:42
<10.219.23.164> ESTABLISH LOCAL CONNECTION FOR USER: root
<10.219.23.164> EXEC /bin/sh -c 'echo ~root && sleep 0'
<10.219.23.164> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /root/.ansible/tmp `"&& mkdir "` echo /root/.ansible/tmp/ansible-tmp-1738906760.4638624-60604-146840035269477 `" && echo ansible-tmp-1738906760.4638624-60604-146840035269477="` echo /root/.ansible/tmp/ansible-tmp-1738906760.4638624-60604-146840035269477 `" ) && sleep 0'
Using module file /usr/local/lib/python3.10/dist-packages/ansible/modules/wait_for.py
<10.219.23.164> PUT /root/.ansible/tmp/ansible-local-60544mzm252k3/tmpvt9gp11m TO /root/.ansible/tmp/ansible-tmp-1738906760.4638624-60604-146840035269477/AnsiballZ_wait_for.py
<10.219.23.164> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1738906760.4638624-60604-146840035269477/ /root/.ansible/tmp/ansible-tmp-1738906760.4638624-60604-146840035269477/AnsiballZ_wait_for.py && sleep 0'
<10.219.23.164> EXEC /bin/sh -c '/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1738906760.4638624-60604-146840035269477/AnsiballZ_wait_for.py && sleep 0'
<10.219.23.164> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1738906760.4638624-60604-146840035269477/ > /dev/null 2>&1 && sleep 0'
ok: [local_connection_testcases] => {
    "changed": false,
    "elapsed": 300,
    "invocation": {
        "module_args": {
            "active_connection_states": [
                "ESTABLISHED",
                "FIN_WAIT1",
                "FIN_WAIT2",
                "SYN_RECV",
                "SYN_SENT",
                "TIME_WAIT"
            ],
            "connect_timeout": 5,
            "delay": 300,
            "exclude_hosts": null,
            "host": "x.x.x.x",
            "msg": null,
            "path": null,
            "port": 830,
            "search_regex": null,
            "sleep": 1,
            "state": "started",
            "timeout": 3600
        }
    },
    "match_groupdict": {},
    "match_groups": [],
    "path": null,
    "port": 830,
    "search_regex": null,
    "state": "started"
}

PLAY RECAP ****************************************************************************************************************************************************************
local_connection_testcases : ok=5    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

/home/labroot/ansible_docker_test1/ansible-junos-stdlib/tests#

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@chidanandpujar chidanandpujar

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@chidanandpujar