Optserv 1269 part 2 servox async logic hardening

[linkous8]

• refactor connector initialization to be member method load_connectors of servo instead of procedural assembly logic
• update servo to store connector type routes loaded from config yam/default values
• fix servo attach() and detach() referencing servo module
• refactor servo shutdown to send gootbye event, add reason param to shutdown to include in request
• use TaskGroup for checks rememdy processing to avoid task leaks
• Move AssemblyRunner run() args poll, interactive to class properties set on init
• fix incorrected type hint on _default_context_manager
• Update progress handler to support creating its queue processor task with an external task group tied to lifetime of the AssemblyRunner
• Update progress handler sink to require servo from extra or context and include in progress dict
• alphebetize progress dict by key
• move servo to api connection details from ServoRunner to servo
• Update ServoRunner to utilize a task group to organize tasks that used to be created and stored as properties with additional management boilerplate
• Coallesce main_loop into run_main_loop
• get rid of pointlress _reraise_if_neccessary method cluttering stack trace
• remove hello event logic from run and set it to run in servo.startup()
• fix main_loop_task not set to main loop of ServoRunner
• move goodbye event logic to servo shutdown, call assembly.remove_servo() to send it, use _main_loop_task and _task_group for cleanup and auditing
• Update AssemblyRunner to kick off and store single root task from synchronous invocation
• Move most of run() logic into async version _run()
• Update AssemblyRunner to use two task group for tracking tasks that used to be spawned from create_task without storing any refernces to them
• _task_group contextualizes all tasks run within the AssemblyRunner._run() function to ensure the lifetime of the Progress handler is tied to the lifetime of the running servos and gets cancelled by crash errors thrown by them
• _runners_task_group holds only the servo tasks. Due to the progressHandlers queue_processor running an infinite loop, _task_group would run forever if allowed to. _runners_task_group allows us to determine if all servo runner tasks have completed and gracefully exit the _task_group when that is the case
• move _report_progress from dynamic method to class method
• move _handle_progress_exception from dynamic method to class method
• add AssemblyRunner.shutdown guard for re-entrant shutdown
• Update AssemblyRunner.shutdown to attempt to shutdown gracefully and kill the root_task instead of killing all tasks. This should propagate a cancellation down the chain of task groups this agent is now structured under
• Update _handle_progress_exception to shutdown and restart only the individual servo that threw the error instead of all_tasks. Makes use of previously dead code Assembly.add/remove_servo as well as added task groups
• Update Servo connectors lists init to be self contained
• Fix type hint error warnings caused by DNSSubdomainName, DNSLabelName, ContainerTagName
• Disable unused pubsub startup in prometheus connector
• fix unresolved import errors in IDE for devtools.debug
• Fix captured_logs type hint error
• Fix type hints for minikube fixture
• Use task group for FakeAPI server
• Use task group for kubectl_ports_forwarded
• Remove any remaining logic that clobbers asyncio.all_tasks (only tests left at this point)
• fix servo_test failures
• replace prometheus test tasks with task group

[leokalev]

The use of 'connectors' vs 'connectors' is not easy to understand by looking at the code. If I got it right, the connectors is exclusively for the 'events' mixin and is a copy of 'connectors', except it also includes 'self'.

A short comment explaining the use might be helpful here.

[linkous8]

I don't understand it 100% either but I think one is used for the event bus (includes servo) for dispatching events and the other is used by the servo (excludes servo self reference) for non-event collector handling

[leokalev]

The markdown let me down here - I meant __connectors__ not connectors. Anyway, this is optional - but if you have an idea what to put in a docstring, it will be great.

[linkous8]

It should go in the events mixin but really there should be some logic to unify the variable so that its less confusing and the mixin is self-contained instead of having keyword arg details spilling out into other components.

servox/servo/events.py

Line 646 in f3ade34

__connectors__: List[Mixin] = None,

[leokalev]

note: (follow up on the comment about changing 'black' to a larger line_length):
if you decide to adopt this, please don't do it as part of the current PR - that would be messy >:-0