[FEATURE] Migrate Get Detector using Job Scheduler to SDK using SDKRestClient #451

owaiskazi19 · 2023-02-14T18:46:41Z

Is your feature request related to a problem?

Part of opensearch-project/OpenSearch#5224

What solution would you like?

Migrate the Get Detector Rest API to Extensions SDK using SDKRestClient.

Use the implementation of #353 as a reference. Add new issues for any steps you skip.

Implement the following API endpoints:

GET _plugins/_anomaly_detection/detectors/<detectorId>?job=true
GET _plugins/_anomaly_detection/detectors/<detectorId>?task=true

The text was updated successfully, but these errors were encountered:

joshpalis · 2023-05-01T21:40:24Z

Get Detector for Realtime Single Entity Detector (job=true)

curl -X GET "localhost:9200/_extensions/_ad-extension/detectors/t-g-2YcBegkemgyzl2YC?job=true"
{
  "_id": "t-g-2YcBegkemgyzl2YC",
  "_version": 1,
  "_primary_term": 1,
  "_seq_no": 0,
  "anomaly_detector": {
    "name": "test-detector",
    "description": "Test detector",
    "time_field": "@timestamp",
    "indices": [
      "server_log"
    ],
    "filter_query": {
      "match_all": {
        "boost": 1
      }
    },
    "detection_interval": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "shingle_size": 8,
    "schema_version": 0,
    "feature_attributes": [
      {
        "feature_id": "BLM-2YcBoF3GEVXabW2v",
        "feature_name": "test",
        "feature_enabled": true,
        "aggregation_query": {
          "test": {
            "avg": {
              "field": "jvmGcTime"
            }
          }
        }
      }
    ],
    "last_update_time": 1682976970492,
    "detector_type": "SINGLE_ENTITY"
  },
  "anomaly_detector_job": {
    "name": "t-g-2YcBegkemgyzl2YC",
    "schedule": {
      "interval": {
        "start_time": 1682977003130,
        "period": 1,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "enabled": true,
    "enabled_time": 1682977003130,
    "last_update_time": 1682977003130,
    "lock_duration_seconds": 60
  }
}

Get Detector for Realtime Single Entity Detector (task=true)

curl -X GET "localhost:9200/_extensions/_ad-extension/detectors/OzpF2YcB0jqKNXhVvqnE?task=true"
{
  "_id": "OzpF2YcB0jqKNXhVvqnE",
  "_version": 1,
  "_primary_term": 1,
  "_seq_no": 0,
  "anomaly_detector": {
    "name": "test-detector",
    "description": "Test detector",
    "time_field": "@timestamp",
    "indices": [
      "server_log"
    ],
    "filter_query": {
      "match_all": {
        "boost": 1
      }
    },
    "detection_interval": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "shingle_size": 8,
    "schema_version": 0,
    "feature_attributes": [
      {
        "feature_id": "GBVF2YcBrcqcQ-5_lYlp",
        "feature_name": "test",
        "feature_enabled": true,
        "aggregation_query": {
          "test": {
            "avg": {
              "field": "jvmGcTime"
            }
          }
        }
      }
    ],
    "last_update_time": 1682977439421,
    "detector_type": "SINGLE_ENTITY"
  },
  "realtime_detection_task": {
    "task_id": "PTpG2YcB0jqKNXhVJamy",
    "last_update_time": 1682977526248,
    "error": "No RCF models are available either because RCF models are not ready or all nodes are unresponsive or the system might have bugs.",
    "state": "CREATED",
    "detector_id": "OzpF2YcB0jqKNXhVvqnE",
    "task_progress": 0,
    "init_progress": 0,
    "execution_start_time": 1682977465774,
    "is_latest": true,
    "task_type": "REALTIME_SINGLE_ENTITY",
    "coordinating_node": "ad-extension",
    "detector": {
      "name": "test-detector",
      "description": "Test detector",
      "time_field": "@timestamp",
      "indices": [
        "server_log"
      ],
      "filter_query": {
        "match_all": {
          "boost": 1
        }
      },
      "detection_interval": {
        "period": {
          "interval": 1,
          "unit": "Minutes"
        }
      },
      "window_delay": {
        "period": {
          "interval": 1,
          "unit": "Minutes"
        }
      },
      "shingle_size": 8,
      "schema_version": 0,
      "feature_attributes": [
        {
          "feature_id": "GBVF2YcBrcqcQ-5_lYlp",
          "feature_name": "test",
          "feature_enabled": true,
          "aggregation_query": {
            "test": {
              "avg": {
                "field": "jvmGcTime"
              }
            }
          }
        }
      ],
      "last_update_time": 1682977439421,
      "detector_type": "SINGLE_ENTITY"
    }
  },
  "historical_analysis_task": null
}

Get Detector for Realtime Multi Entity Detector (job=true)

curl -X GET "localhost:9200/_extensions/_ad-extension/detectors/4hRK2YcBvUrjdfHQ1NZI?job=true"
{
  "_id": "4hRK2YcBvUrjdfHQ1NZI",
  "_version": 1,
  "_primary_term": 1,
  "_seq_no": 0,
  "anomaly_detector": {
    "name": "test-multi-detector",
    "description": "Test detector",
    "time_field": "@timestamp",
    "indices": [
      "server_log"
    ],
    "filter_query": {
      "match_all": {
        "boost": 1
      }
    },
    "detection_interval": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "shingle_size": 8,
    "schema_version": 0,
    "feature_attributes": [
      {
        "feature_id": "vIBK2YcBA3RarfGOqm7c",
        "feature_name": "test",
        "feature_enabled": true,
        "aggregation_query": {
          "test": {
            "avg": {
              "field": "cpuTime"
            }
          }
        }
      }
    ],
    "last_update_time": 1682977772610,
    "category_field": [
      "process"
    ],
    "detector_type": "MULTI_ENTITY"
  },
  "anomaly_detector_job": {
    "name": "4hRK2YcBvUrjdfHQ1NZI",
    "schedule": {
      "interval": {
        "start_time": 1682977800384,
        "period": 1,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "enabled": true,
    "enabled_time": 1682977800384,
    "last_update_time": 1682977800384,
    "lock_duration_seconds": 60
  }
}

Get Detector for Realtime Multi Entity Detector (task=true)

curl -X GET "localhost:9200/_extensions/_ad-extension/detectors/4hRK2YcBvUrjdfHQ1NZI?task=true"
{
  "_id": "4hRK2YcBvUrjdfHQ1NZI",
  "_version": 1,
  "_primary_term": 1,
  "_seq_no": 0,
  "anomaly_detector": {
    "name": "test-multi-detector",
    "description": "Test detector",
    "time_field": "@timestamp",
    "indices": [
      "server_log"
    ],
    "filter_query": {
      "match_all": {
        "boost": 1
      }
    },
    "detection_interval": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "shingle_size": 8,
    "schema_version": 0,
    "feature_attributes": [
      {
        "feature_id": "vIBK2YcBA3RarfGOqm7c",
        "feature_name": "test",
        "feature_enabled": true,
        "aggregation_query": {
          "test": {
            "avg": {
              "field": "cpuTime"
            }
          }
        }
      }
    ],
    "last_update_time": 1682977772610,
    "category_field": [
      "process"
    ],
    "detector_type": "MULTI_ENTITY"
  },
  "realtime_detection_task": {
    "task_id": "5BRL2YcBvUrjdfHQQdap",
    "last_update_time": 1682977860621,
    "error": "",
    "state": "INIT",
    "detector_id": "4hRK2YcBvUrjdfHQ1NZI",
    "task_progress": 0,
    "init_progress": 0,
    "execution_start_time": 1682977800613,
    "is_latest": true,
    "task_type": "REALTIME_HC_DETECTOR",
    "coordinating_node": "ad-extension",
    "detector": {
      "name": "test-multi-detector",
      "description": "Test detector",
      "time_field": "@timestamp",
      "indices": [
        "server_log"
      ],
      "filter_query": {
        "match_all": {
          "boost": 1
        }
      },
      "detection_interval": {
        "period": {
          "interval": 1,
          "unit": "Minutes"
        }
      },
      "window_delay": {
        "period": {
          "interval": 1,
          "unit": "Minutes"
        }
      },
      "shingle_size": 8,
      "schema_version": 0,
      "feature_attributes": [
        {
          "feature_id": "vIBK2YcBA3RarfGOqm7c",
          "feature_name": "test",
          "feature_enabled": true,
          "aggregation_query": {
            "test": {
              "avg": {
                "field": "cpuTime"
              }
            }
          }
        }
      ],
      "last_update_time": 1682977772610,
      "category_field": [
        "process"
      ],
      "detector_type": "MULTI_ENTITY"
    },
    "estimated_minutes_left": 32
  },
  "historical_analysis_task": null
}

joshpalis · 2023-05-01T21:55:33Z

Note : Historical Detection does not have an associated job, therefore it is only necessary to check for historical batch tasks

Get Detector for Historical Single Entity Detector (task=true)

curl -X GET "localhost:9200/_extensions/_ad-extension/detectors/8YhR2YcBmwguAcPCmxcl?task=true"
{
  "_id": "8YhR2YcBmwguAcPCmxcl",
  "_version": 1,
  "_primary_term": 1,
  "_seq_no": 0,
  "anomaly_detector": {
    "name": "test-detector",
    "description": "Test detector",
    "time_field": "@timestamp",
    "indices": [
      "server_log"
    ],
    "filter_query": {
      "match_all": {
        "boost": 1
      }
    },
    "detection_interval": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "shingle_size": 8,
    "schema_version": 0,
    "feature_attributes": [
      {
        "feature_id": "wAxR2YcBuOGmIVmCcfS1",
        "feature_name": "test",
        "feature_enabled": true,
        "aggregation_query": {
          "test": {
            "avg": {
              "field": "jvmGcTime"
            }
          }
        }
      }
    ],
    "last_update_time": 1682978216735,
    "detector_type": "SINGLE_ENTITY"
  },
  "realtime_detection_task": null,
  "historical_analysis_task": {
    "task_id": "8ohT2YcBmwguAcPClBf4",
    "last_update_time": 1682978358356,
    "state": "RUNNING",
    "detector_id": "8YhR2YcBmwguAcPCmxcl",
    "task_progress": 0.42016807,
    "init_progress": 1,
    "current_piece": 1682939760000,
    "execution_start_time": 1682978346228,
    "is_latest": true,
    "task_type": "HISTORICAL_SINGLE_ENTITY",
    "coordinating_node": "ad-extension",
    "worker_node": "ad-extension",
    "detector": {
      "name": "test-detector",
      "description": "Test detector",
      "time_field": "@timestamp",
      "indices": [
        "server_log"
      ],
      "filter_query": {
        "match_all": {
          "boost": 1
        }
      },
      "detection_interval": {
        "period": {
          "interval": 1,
          "unit": "Minutes"
        }
      },
      "window_delay": {
        "period": {
          "interval": 1,
          "unit": "Minutes"
        }
      },
      "shingle_size": 8,
      "schema_version": 0,
      "feature_attributes": [
        {
          "feature_id": "wAxR2YcBuOGmIVmCcfS1",
          "feature_name": "test",
          "feature_enabled": true,
          "aggregation_query": {
            "test": {
              "avg": {
                "field": "jvmGcTime"
              }
            }
          }
        }
      ],
      "last_update_time": 1682978216735,
      "detector_type": "SINGLE_ENTITY"
    },
    "detection_date_range": {
      "start_time": 1682819780000,
      "end_time": 1683119770000
    }
  }
}

Get Detector for Historical Multi Entity Detector (task=true)

curl -X GET "localhost:9200/_extensions/_ad-extension/detectors/c8eC3YcB0_8fGnWCRC5m?task=true"
{
  "_id": "c8eC3YcB0_8fGnWCRC5m",
  "_version": 1,
  "_primary_term": 1,
  "_seq_no": 0,
  "anomaly_detector": {
    "name": "test-multi-detector",
    "description": "Test detector",
    "time_field": "@timestamp",
    "indices": [
      "server_log"
    ],
    "filter_query": {
      "match_all": {
        "boost": 1
      }
    },
    "detection_interval": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "window_delay": {
      "period": {
        "interval": 1,
        "unit": "Minutes"
      }
    },
    "shingle_size": 8,
    "schema_version": 0,
    "feature_attributes": [
      {
        "feature_id": "hiSC3YcBhde1L1ODG3Ne",
        "feature_name": "test",
        "feature_enabled": true,
        "aggregation_query": {
          "test": {
            "avg": {
              "field": "cpuTime"
            }
          }
        }
      }
    ],
    "last_update_time": 1683048514656,
    "category_field": [
      "process"
    ],
    "detector_type": "MULTI_ENTITY"
  },
  "realtime_detection_task": null,
  "historical_analysis_task": {
    "task_id": "dMeD3YcB0_8fGnWCES6y",
    "last_update_time": 1683048567214,
    "state": "CREATED",
    "detector_id": "c8eC3YcB0_8fGnWCRC5m",
    "task_progress": 0,
    "init_progress": 0,
    "execution_start_time": 1683048567214,
    "is_latest": true,
    "task_type": "HISTORICAL_HC_DETECTOR",
    "coordinating_node": "ad-extension",
    "detector": {
      "name": "test-multi-detector",
      "description": "Test detector",
      "time_field": "@timestamp",
      "indices": [
        "server_log"
      ],
      "filter_query": {
        "match_all": {
          "boost": 1
        }
      },
      "detection_interval": {
        "period": {
          "interval": 1,
          "unit": "Minutes"
        }
      },
      "window_delay": {
        "period": {
          "interval": 1,
          "unit": "Minutes"
        }
      },
      "shingle_size": 8,
      "schema_version": 0,
      "feature_attributes": [
        {
          "feature_id": "hiSC3YcBhde1L1ODG3Ne",
          "feature_name": "test",
          "feature_enabled": true,
          "aggregation_query": {
            "test": {
              "avg": {
                "field": "cpuTime"
              }
            }
          }
        }
      ],
      "last_update_time": 1683048514656,
      "category_field": [
        "process"
      ],
      "detector_type": "MULTI_ENTITY"
    },
    "detection_date_range": {
      "start_time": 1682889993000,
      "end_time": 1683189983000
    }
  }
}

joshpalis · 2023-05-01T22:19:54Z

Tasks to complete

Test job=true for single entity real time detectors
Test job=true for multi entity real time detectors
Test task=true for single entity real time detectors
Test task=true for multi entity real time detectors
Test task=true for single entity historical detectors
Test task=true for multi-entity historical detectors (Dependent on Migrate Profile Detector to SDK using SDKRestClient #381)

joshpalis · 2023-05-02T17:31:58Z

All APIs have been tested successfully

owaiskazi19 added enhancement New feature or request untriaged labels Feb 14, 2023

joshpalis mentioned this issue Feb 14, 2023

[META] Integrate the Job Scheduler Communication Mechanism with Anomaly Detection Extension to enable extensions to schedule jobs #427

Closed

7 tasks

owaiskazi19 assigned joshpalis Feb 14, 2023

minalsha removed the untriaged label Mar 7, 2023

This was referenced Mar 23, 2023

[META] Migrate AD backend plugin to AD Extension opensearch-project/OpenSearch#5224

Open

[META] Implement Get Detector for AnomalyDetector Extension #211

Closed

owaiskazi19 mentioned this issue Apr 28, 2023

[Feature/Extensions] Profile detector opensearch-project/anomaly-detection#882

Merged

joshpalis closed this as completed May 2, 2023

joshpalis mentioned this issue May 2, 2023

[FEATURE] Integrate historical Anomaly Detection analysis #636

Closed

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[FEATURE] Migrate Get Detector using Job Scheduler to SDK using SDKRestClient #451

[FEATURE] Migrate Get Detector using Job Scheduler to SDK using SDKRestClient #451

owaiskazi19 commented Feb 14, 2023 •

edited by joshpalis

Loading

joshpalis commented May 1, 2023 •

edited

Loading

joshpalis commented May 1, 2023 •

edited

Loading

joshpalis commented May 1, 2023 •

edited

Loading

joshpalis commented May 2, 2023

[FEATURE] Migrate Get Detector using Job Scheduler to SDK using SDKRestClient #451

[FEATURE] Migrate Get Detector using Job Scheduler to SDK using SDKRestClient #451

Comments

owaiskazi19 commented Feb 14, 2023 • edited by joshpalis Loading

Is your feature request related to a problem?

What solution would you like?

joshpalis commented May 1, 2023 • edited Loading

Get Detector for Realtime Single Entity Detector (job=true)

Get Detector for Realtime Single Entity Detector (task=true)

Get Detector for Realtime Multi Entity Detector (job=true)

Get Detector for Realtime Multi Entity Detector (task=true)

joshpalis commented May 1, 2023 • edited Loading

Get Detector for Historical Single Entity Detector (task=true)

Get Detector for Historical Multi Entity Detector (task=true)

joshpalis commented May 1, 2023 • edited Loading

Tasks to complete

joshpalis commented May 2, 2023

owaiskazi19 commented Feb 14, 2023 •

edited by joshpalis

Loading

joshpalis commented May 1, 2023 •

edited

Loading

joshpalis commented May 1, 2023 •

edited

Loading

joshpalis commented May 1, 2023 •

edited

Loading