data store: waiting jobs

[wxtim]

Spotted in the wild, "waiting jobs":

{
  "data": {
    "workflows": [
      {
        "taskProxies": [
          {
            "id": "~tpilling/mi-bb964/run2//19600101T0000Z/coupled",
            "state": "waiting",
            "jobs": []
          },
          {
            "id": "~tpilling/mi-bb964/run2//19600101T0000Z/fcm_make2_drivers",
            "state": "waiting",
            "jobs": [
              {
                "id": "~tpilling/mi-bb964/run2//19600101T0000Z/fcm_make2_drivers/01",
                "state": "waiting"
              },
              {
                "id": "~tpilling/mi-bb964/run2//19600101T0000Z/fcm_make2_drivers/03",
                "state": "waiting"
              },
              {
                "id": "~tpilling/mi-bb964/run2//19600101T0000Z/fcm_make2_drivers/02",
                "state": "waiting"
              }
            ]
          },

The "waiting" state is a task state, not a job state. It shouldn't be possible to get a waiting job.

Steps to reproduce:
create a workflow where submission fails after trying 2 hosts.

[hjoliver]

Ping @dwsutherland

[dwsutherland]

Have reproduce this:

[platforms]
    [[boss]]
        hosts = fdasfeds, fdawsf

    [[foo]]
        inherit = FAM3
        platform = boss
        submission retry delays = PT0S, PT30S, PT1M
        [[[meta]]]
            description = "some task foo"
        [[[environment]]]
            GREETING = "Hello from foo!"

query {
  workflows (ids: ["linear/run1"]) {
    id
    taskProxies {
      id
      state
      jobs (sort: {keys: ["submitNum"], reverse: true}) {
        id
        state
      }
    }
  }
}

{
  "data": {
    "workflows": [
      {
        "id": "~sutherlander/linear/run1",
        "taskProxies": [
          {
            "id": "~sutherlander/linear/run1//20210201T00/foo",
            "state": "waiting",
            "jobs": []
          },
          {
            "id": "~sutherlander/linear/run1//20210101T00/foo",
            "state": "submit-failed",
            "jobs": [
              {
                "id": "~sutherlander/linear/run1//20210101T00/foo/04",
                "state": "submit-failed"
              },
              {
                "id": "~sutherlander/linear/run1//20210101T00/foo/03",
                "state": "waiting"
              },
              {
                "id": "~sutherlander/linear/run1//20210101T00/foo/02",
                "state": "waiting"
              },
              {
                "id": "~sutherlander/linear/run1//20210101T00/foo/01",
                "state": "waiting"
              }
            ]
          },
          {
            "id": "~sutherlander/linear/run1//20210101T00/bar",
            "state": "waiting",
            "jobs": []
          }
        ]
      }
    ]
  }
}

will look into this... I think it's because they adopt the task's state initially.. I suppose it should be preparing -> submit-failed?

[dwsutherland]

Found the issue, the platform submission failure is happening before the job is inserted into the data-store (and DB):

2022-06-29T02:48:00Z CRITICAL - [20210101T00/foo preparing job:01 flows:1] submission failed
~sutherlander/linear/run1//20210101T00/foo/01 ...... submit-failed
JOB OR STATE NOT FOUND
JOB DOES NOT EXIST
2022-06-29T02:48:00Z INFO - [20210101T00/foo preparing job:01 flows:1] => waiting
2022-06-29T02:48:00Z WARNING - [20210101T00/foo waiting job:01 flows:1] retrying in P0Y (after 2022-06-29T02:48:00Z)
preparing
INSERTING: ~sutherlander/linear/run1//20210101T00/foo/01, STATUS: preparing

just need to find out where this failure is getting triggered .. and I'll put up a fix

[dwsutherland]

Fix up