Enhancement to callback_plugins/jsnapy.py

[jnpr-bowen]

Using jsnapy 1.2 to get fields from the updated test_results structure.
For my project I have made a change to jsnapy.py that tries to provide an 'operator friendly' output to running of jsnapy tests in Ansible and added a keyword to the junos_jsnapy module that controls whether you get error, or error+info messages from JSNAPy. Below is sample output generated by running a test playbook. Is there an interest for me to make this a pull request to the ansbile-junos-stdlib?
'reporting' is a var in my playbook and sets the value for "callback" on my modified junos_jsnapy module. Callback, if not set, leaves the old callback output, error gives only error messages from test, info gives both error and info messages.
The examples below do not show the 'coloring' the callback applies. "Results for:" header is Blue, and error messages are Red, all other text is b/w. "-- Test: " header comes from the rpc/cli that is executed (AKA the 'test_name' key for the list of results for that rpc/cli in the test_results json).
"Results for" and "Errors for" all come from 'test_name' key that is inside the test_name list. This is due to something like 'ping' may be executed multiple times for different destinations. Ping will only be a key in the test_results one time, but the list could have unique names for test_name if provided (look at error+info output below).

Running for errors:
$ ansible-playbook -i lab_role.inv -l lab-chi-spn02 test_run.pb.yml -e reporting=error

Results for: lab-chi-spn02 *****************************************************
 -- Test: get_software_information
 -- Test: ping
 -- Test: get_bfd_session_information
 -- Test: get_isis_adjacency_information
 -- Test: get_environment_information
	Errors for test: Verify_Environmentals
		'Power Supply 3 is Check.'
		'Power Supply 4 is Check.'
		'Power Supply 5 is Check.'
 -- Test: get_bgp_summary_information
	Errors for test: Check_BGP_sessions
		'BGP to 192.0.2.1 AS 65001 is Idle.'
		'BGP to 192.0.2.1 AS 65001 is Idle.'

Running for errors+info:
$ ansible-playbook -i lab_role.inv -l lab-chi-spn02 test_run.pb.yml -e reporting=info

 -- Test: get_software_information
	Results for test: version_check
		'Correct version of Junos found.'
 -- Test: ping
	Results for test: Ping Host 192.0.2.1
		'All pings of 192.0.2.1 succeeded.'
	Results for test: Ping Host 192.0.2.2
		'All pings of 192.0.2.2 succeeded.'
	Results for test: Ping Host 192.0.2.3
		'All pings of 192.0.2.3 succeeded.'
	Results for test: Ping Host 192.0.2.4
		'All pings of 192.0.2.4 succeeded.'
	Results for test: Ping Host 192.0.2.5
		'All pings of 192.0.2.5 succeeded.'
 -- Test: get_bfd_session_information
	Results for test: bfd_sessions
		'Good: There are 13 sessions established.'
	Results for test: bfd_sessions
		'Good: Session to BFD peer 10.70.70.5 with detection time of 0.030 is Up.'
		'Good: Session to BFD peer 10.70.70.7 with detection time of 0.030 is Up.'
		'Good: Session to BFD peer 192.168.17.0 with detection time of 0.750 is Up.'
		'Good: Session to BFD peer 192.168.22.0 with detection time of 0.750 is Up.'
		'Good: Session to BFD peer 192.168.60.0 with detection time of 0.450 is Up.'
		'Good: Session to BFD peer 192.168.61.0 with detection time of 0.600 is Up.'
		'Good: Session to BFD peer 192.168.61.1 with detection time of 0.600 is Up.'
		'Good: Session to BFD peer 192.168.62.0 with detection time of 0.450 is Up.'
		'Good: Session to BFD peer 192.168.62.1 with detection time of 0.450 is Up.'
		'Good: Session to BFD peer 203.0.113.3 with detection time of 0.600 is Up.'
		'Good: Session to BFD peer 203.0.113.15 with detection time of 0.600 is Up.'
		'Good: Session to BFD peer 203.0.113.27 with detection time of 0.450 is Up.'
		'Good: Session to BFD peer 203.0.113.39 with detection time of 0.450 is Up.'
 -- Test: get_isis_adjacency_information
	Results for test: ISIS_Status
		'Adjacency established to lab-msr11 on ae1.947.'
		'Adjacency established to lab-msr12 on ae2.947.'
 -- Test: get_environment_information
	Results for test: Verify_Environmentals
		'Power Supply 0 is OK.'
		'Power Supply 1 is OK.'
		'Power Supply 2 is OK.'
	Errors for test: Verify_Environmentals
		'Power Supply 3 is Check.'
		'Power Supply 4 is Check.'
		'Power Supply 5 is Check.'
 -- Test: get_bgp_summary_information
	Results for test: Check_BGP_sessions
		'BGP to 192.168.17.0 AS 64500 is OK/Established.'
		'BGP to 192.168.22.0 AS 64500 is OK/Established.'
		'BGP to 192.168.60.0 AS 64500 is OK/Established.'
		'BGP to 192.168.61.0 AS 64500 is OK/Established.'
		'BGP to 192.168.61.1 AS 64500 is OK/Established.'
		'BGP to 192.168.62.0 AS 64500 is OK/Established.'
		'BGP to 192.168.62.1 AS 64500 is OK/Established.'
		'BGP to 198.51.100.3 AS 65001 is OK/Established.'
		'BGP to 198.51.100.15 AS 65002 is OK/Established.'
		'BGP to 198.51.100.27 AS 65003 is OK/Established.'
		'BGP to 198.51.100.39 AS 65004 is OK/Established.'
	Errors for test: Check_BGP_sessions
		'BGP to 198.51.100.7 AS 65001 is Idle.'
		'BGP to 198.51.100.11 AS 65001 is Idle.'

[jnpr-bowen]

The code change is now in pull request #252

[jnpr-bowen]

There is a condition that my pull request deals with but the message may not be "the best", looking for comments/suggestions about changing the output message.
The situation is if a test executes an RPC for which the the device is not configured, like something IS-IS related and the device is not running IS-IS. The result is that the JSNAPy result structure says one test failed, but there is no 'message' field in the failed results list. In my pull request the message printed is hard coded into jsnapy.py and looks like:

 -- Test: get_isis_adjacency_information
	Errors for test: ISIS_Status
		Test failed without providing message.

The RPC response won't match any xpath and can't be put in the JSNAPy error message as it is just:

<output>IS-IS instance is not running</output>

After showing this test to the customer, they suggested "Test invalid for node." be printed, but I don't know if that will be 100% true. Should the message also say something about checking the RPC response file?

[rahkumar651991]

Based on the discussion on #252 , closing this.