Add lm-eval correctness test

[dbarbuzzi]

Note

The PR description has been rewritten for clarity based on current state.

The main goal of this PR is to update the actions added by @mgoin in #166 and translate one of them into an end-to-end test building off of the server framework from #200.

This also updates some of the actions and workflows to ensure they are running leanly and at appropriate times:

• an initial draft of a release workflow is added (runs only on-demand/manually)
• the ‘build-test’ workflow is updated to support a 'WEEKLY' category
  • the lm-eval smoke test is run nightly (Mon-Sat)
  • the lm-eval full test is run weekly (Sun)
• the lm-eval smoke flow/action is updated to use the prebuilt wheel
• the lm-eval full flow/action is updated to use the prebuilt wheel and run the new test version

Pending

Per a conversation with @robertgshaw2-neuralmagic, the problematic marlin models have been disabled while work is underway to reduce their non-determinism. They can be re-evaluated/re-enabled at a later point.

There are currently a few models that may occasionally fail the test which may relate to either non-determinism of marlin, implementation issues, or perhaps a combination:

• neuralmagic/llama-2-7b-chat-marlin has failed with an rtol as high as 0.2
• neuralmagic/phi-2-super-marlin has failed occasionally with rtol=0.05

Separately, mistralai/Mixtral-8x7B-Instruct-v0.1 is untested due to the significant hardware required.

[derekk-nm]

it might be nice to collect results for all of the metrics, and maybe all of the tasks, that are not close, then assert if there are any in error. that way all of the problems are reported, rather than having a developer fix one issue then getting an error on the next that they didn't fix yet.

[andy-neuma]

looking good