Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reapply "[Response Ops][Alerting] Backfill actions schema changes for intermediate release #204657

Merged
merged 7 commits into from
Jan 7, 2025

Conversation

ymao1
Copy link
Contributor

@ymao1 ymao1 commented Dec 17, 2024

Summary

Redoing this PR which had to be reverted. This should not be merged until this update to the task manager v1 schema is released.

To verify

  1. Set xpack.task_manager.unsafe.exclude_task_types: ['ad_hoc_run-backfill', 'actions:*'] in your Kibana config.
  2. Run Kibana on main and create some detection rules that run frequently, with actions.
  3. Schedule a manual run for your detection rules.
    • Because of the config, the action_task_params SO and the ad_hoc_run_task_params SO will get written but not read yet.
  4. Remove the exclude_task_types config and "upgrade" to this PR branch and verify that rules continue to run and that the actions are triggered and the manual rule runs go through
  5. Re-add the exclude_task_types config and let the rule run again to schedule action. Schedule another manual rule run.
  6. Remove the exclude_task_types config and "downgrade" back to main and verify that rules continue to run, the action gets triggered and manual rule runs go through.

@ymao1 ymao1 self-assigned this Dec 17, 2024
@ymao1 ymao1 added Feature:Alerting Feature:Task Manager Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v9.0.0 backport:prev-minor Backport to (9.0) the previous minor version (i.e. one version back from main) v8.18.0 labels Dec 17, 2024
@ymao1 ymao1 marked this pull request as ready for review January 6, 2025 18:46
@ymao1 ymao1 requested review from a team as code owners January 6, 2025 18:46
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@ymao1 ymao1 added the release_note:skip Skip the PR/issue when compiling release notes label Jan 6, 2025
Copy link
Member

@pmuellr pmuellr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Test instructions in the PR got a little more involved :-), worked as described.

Copy link
Contributor

@jloleysens jloleysens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mappings changes LGTM, did not test locally!

@ymao1
Copy link
Contributor Author

ymao1 commented Jan 7, 2025

@elasticmachine merge upstream

@ymao1
Copy link
Contributor Author

ymao1 commented Jan 7, 2025

In addition to the verification instructions, I did the following locally.

Running serverless locally with .kibana_alerting_cases index that had the same version conflict issue that QA project cc5d had and already included the new mappings in the action_task_param type and the task type (in the .kibana_task_manager index)

.kibana_alerting_cases/_mappings
{
    ".kibana_alerting_cases_1": {
        "mappings": {
            "dynamic": "strict",
            "_meta": {
                "migrationState": {
                    "convertingDocuments": false
                },
                "docVersions": {
                    "cases-rules": "10.1.0",
                    "connector_token": "10.1.0",
                    "rules-settings": "10.1.0",
                    "cases": "10.1.0",
                    "action_task_params": "10.2.0",
                    "cases-connector-mappings": "10.0.0",
                    "maintenance-window": "10.1.0",
                    "ad_hoc_run_params": "10.2.0",
                    "api_key_pending_invalidation": "10.1.0",
                    "cases-comments": "10.0.0",
                    "cases-telemetry": "10.0.0",
                    "cases-configure": "10.0.0",
                    "alert": "10.3.0",
                    "cases-user-actions": "10.0.0",
                    "action": "10.1.0"
                },
                "mappingVersions": {
                    "cases-rules": "10.1.0",
                    "connector_token": "10.1.0",
                    "rules-settings": "10.1.0",
                    "cases": "10.1.0",
                    "action_task_params": "10.2.0",
                    "cases-connector-mappings": "10.0.0",
                    "maintenance-window": "10.1.0",
                    "ad_hoc_run_params": "10.2.0",
                    "api_key_pending_invalidation": "10.1.0",
                    "cases-comments": "10.0.0",
                    "cases-telemetry": "10.0.0",
                    "cases-configure": "10.0.0",
                    "alert": "10.3.0",
                    "cases-user-actions": "10.0.0",
                    "action": "10.1.0"
                }
            },
            "properties": {
                "action": {
                    "dynamic": "false",
                    "properties": {
                        "actionTypeId": {
                            "type": "keyword"
                        },
                        "name": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword"
                                }
                            }
                        }
                    }
                },
                "action_task_params": {
                    "dynamic": "false",
                    "properties": {
                        "apiKeyId": {
                            "type": "keyword"
                        }
                    }
                },
                "ad_hoc_run_params": {
                    "dynamic": "false",
                    "properties": {
                        "apiKeyId": {
                            "type": "keyword"
                        },
                        "createdAt": {
                            "type": "date"
                        },
                        "end": {
                            "type": "date"
                        },
                        "rule": {
                            "properties": {
                                "alertTypeId": {
                                    "type": "keyword"
                                },
                                "consumer": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "start": {
                            "type": "date"
                        }
                    }
                },
                "alert": {
                    "dynamic": "false",
                    "properties": {
                        "actions": {
                            "type": "nested",
                            "dynamic": "false",
                            "properties": {
                                "actionRef": {
                                    "type": "keyword"
                                },
                                "actionTypeId": {
                                    "type": "keyword"
                                },
                                "group": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "alertTypeId": {
                            "type": "keyword"
                        },
                        "consumer": {
                            "type": "keyword"
                        },
                        "createdAt": {
                            "type": "date"
                        },
                        "createdBy": {
                            "type": "keyword"
                        },
                        "enabled": {
                            "type": "boolean"
                        },
                        "executionStatus": {
                            "properties": {
                                "error": {
                                    "properties": {
                                        "message": {
                                            "type": "keyword"
                                        },
                                        "reason": {
                                            "type": "keyword"
                                        }
                                    }
                                },
                                "lastDuration": {
                                    "type": "long"
                                },
                                "lastExecutionDate": {
                                    "type": "date"
                                },
                                "numberOfTriggeredActions": {
                                    "type": "long"
                                },
                                "status": {
                                    "type": "keyword"
                                },
                                "warning": {
                                    "properties": {
                                        "message": {
                                            "type": "keyword"
                                        },
                                        "reason": {
                                            "type": "keyword"
                                        }
                                    }
                                }
                            }
                        },
                        "lastRun": {
                            "properties": {
                                "alertsCount": {
                                    "properties": {
                                        "active": {
                                            "type": "float"
                                        },
                                        "ignored": {
                                            "type": "float"
                                        },
                                        "new": {
                                            "type": "float"
                                        },
                                        "recovered": {
                                            "type": "float"
                                        }
                                    }
                                },
                                "outcome": {
                                    "type": "keyword"
                                },
                                "outcomeOrder": {
                                    "type": "float"
                                }
                            }
                        },
                        "legacyId": {
                            "type": "keyword"
                        },
                        "mapped_params": {
                            "properties": {
                                "risk_score": {
                                    "type": "float"
                                },
                                "severity": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "monitoring": {
                            "properties": {
                                "run": {
                                    "properties": {
                                        "calculated_metrics": {
                                            "properties": {
                                                "p50": {
                                                    "type": "long"
                                                },
                                                "p95": {
                                                    "type": "long"
                                                },
                                                "p99": {
                                                    "type": "long"
                                                },
                                                "success_ratio": {
                                                    "type": "float"
                                                }
                                            }
                                        },
                                        "last_run": {
                                            "properties": {
                                                "metrics": {
                                                    "properties": {
                                                        "duration": {
                                                            "type": "long"
                                                        },
                                                        "gap_duration_s": {
                                                            "type": "float"
                                                        },
                                                        "total_alerts_created": {
                                                            "type": "float"
                                                        },
                                                        "total_alerts_detected": {
                                                            "type": "float"
                                                        },
                                                        "total_indexing_duration_ms": {
                                                            "type": "long"
                                                        },
                                                        "total_search_duration_ms": {
                                                            "type": "long"
                                                        }
                                                    }
                                                },
                                                "timestamp": {
                                                    "type": "date"
                                                }
                                            }
                                        }
                                    }
                                }
                            }
                        },
                        "muteAll": {
                            "type": "boolean"
                        },
                        "mutedInstanceIds": {
                            "type": "keyword"
                        },
                        "name": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword",
                                    "normalizer": "lowercase"
                                }
                            }
                        },
                        "notifyWhen": {
                            "type": "keyword"
                        },
                        "params": {
                            "type": "flattened",
                            "ignore_above": 4096
                        },
                        "revision": {
                            "type": "long"
                        },
                        "running": {
                            "type": "boolean"
                        },
                        "schedule": {
                            "properties": {
                                "interval": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "scheduledTaskId": {
                            "type": "keyword"
                        },
                        "snoozeSchedule": {
                            "type": "nested",
                            "properties": {
                                "duration": {
                                    "type": "long"
                                },
                                "id": {
                                    "type": "keyword"
                                },
                                "skipRecurrences": {
                                    "type": "date",
                                    "format": "strict_date_time"
                                }
                            }
                        },
                        "tags": {
                            "type": "keyword"
                        },
                        "throttle": {
                            "type": "keyword"
                        },
                        "updatedAt": {
                            "type": "date"
                        },
                        "updatedBy": {
                            "type": "keyword"
                        }
                    }
                },
                "api_key_pending_invalidation": {
                    "properties": {
                        "apiKeyId": {
                            "type": "keyword"
                        },
                        "createdAt": {
                            "type": "date"
                        }
                    }
                },
                "cases": {
                    "dynamic": "false",
                    "properties": {
                        "assignees": {
                            "properties": {
                                "uid": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "category": {
                            "type": "keyword"
                        },
                        "closed_at": {
                            "type": "date"
                        },
                        "closed_by": {
                            "properties": {
                                "email": {
                                    "type": "keyword"
                                },
                                "full_name": {
                                    "type": "keyword"
                                },
                                "profile_uid": {
                                    "type": "keyword"
                                },
                                "username": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "connector": {
                            "properties": {
                                "fields": {
                                    "properties": {
                                        "key": {
                                            "type": "text"
                                        },
                                        "value": {
                                            "type": "text"
                                        }
                                    }
                                },
                                "name": {
                                    "type": "text"
                                },
                                "type": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "created_at": {
                            "type": "date"
                        },
                        "created_by": {
                            "properties": {
                                "email": {
                                    "type": "keyword"
                                },
                                "full_name": {
                                    "type": "keyword"
                                },
                                "profile_uid": {
                                    "type": "keyword"
                                },
                                "username": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "customFields": {
                            "type": "nested",
                            "properties": {
                                "key": {
                                    "type": "keyword"
                                },
                                "type": {
                                    "type": "keyword"
                                },
                                "value": {
                                    "type": "keyword",
                                    "fields": {
                                        "boolean": {
                                            "type": "boolean",
                                            "ignore_malformed": true
                                        },
                                        "date": {
                                            "type": "date",
                                            "ignore_malformed": true
                                        },
                                        "ip": {
                                            "type": "ip",
                                            "ignore_malformed": true
                                        },
                                        "number": {
                                            "type": "long",
                                            "ignore_malformed": true
                                        },
                                        "string": {
                                            "type": "text"
                                        }
                                    }
                                }
                            }
                        },
                        "description": {
                            "type": "text"
                        },
                        "duration": {
                            "type": "unsigned_long"
                        },
                        "external_service": {
                            "properties": {
                                "connector_name": {
                                    "type": "keyword"
                                },
                                "external_id": {
                                    "type": "keyword"
                                },
                                "external_title": {
                                    "type": "text"
                                },
                                "external_url": {
                                    "type": "text"
                                },
                                "pushed_at": {
                                    "type": "date"
                                },
                                "pushed_by": {
                                    "properties": {
                                        "email": {
                                            "type": "keyword"
                                        },
                                        "full_name": {
                                            "type": "keyword"
                                        },
                                        "profile_uid": {
                                            "type": "keyword"
                                        },
                                        "username": {
                                            "type": "keyword"
                                        }
                                    }
                                }
                            }
                        },
                        "owner": {
                            "type": "keyword"
                        },
                        "settings": {
                            "properties": {
                                "syncAlerts": {
                                    "type": "boolean"
                                }
                            }
                        },
                        "severity": {
                            "type": "short"
                        },
                        "status": {
                            "type": "short"
                        },
                        "tags": {
                            "type": "keyword"
                        },
                        "title": {
                            "type": "text",
                            "fields": {
                                "keyword": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "total_alerts": {
                            "type": "integer"
                        },
                        "total_comments": {
                            "type": "integer"
                        },
                        "updated_at": {
                            "type": "date"
                        },
                        "updated_by": {
                            "properties": {
                                "email": {
                                    "type": "keyword"
                                },
                                "full_name": {
                                    "type": "keyword"
                                },
                                "profile_uid": {
                                    "type": "keyword"
                                },
                                "username": {
                                    "type": "keyword"
                                }
                            }
                        }
                    }
                },
                "cases-comments": {
                    "dynamic": "false",
                    "properties": {
                        "actions": {
                            "properties": {
                                "type": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "alertId": {
                            "type": "keyword"
                        },
                        "comment": {
                            "type": "text"
                        },
                        "created_at": {
                            "type": "date"
                        },
                        "created_by": {
                            "properties": {
                                "username": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "externalReferenceAttachmentTypeId": {
                            "type": "keyword"
                        },
                        "owner": {
                            "type": "keyword"
                        },
                        "persistableStateAttachmentTypeId": {
                            "type": "keyword"
                        },
                        "pushed_at": {
                            "type": "date"
                        },
                        "type": {
                            "type": "keyword"
                        },
                        "updated_at": {
                            "type": "date"
                        }
                    }
                },
                "cases-configure": {
                    "dynamic": "false",
                    "properties": {
                        "closure_type": {
                            "type": "keyword"
                        },
                        "created_at": {
                            "type": "date"
                        },
                        "owner": {
                            "type": "keyword"
                        }
                    }
                },
                "cases-connector-mappings": {
                    "dynamic": "false",
                    "properties": {
                        "owner": {
                            "type": "keyword"
                        }
                    }
                },
                "cases-rules": {
                    "dynamic": "false",
                    "properties": {
                        "counter": {
                            "type": "unsigned_long"
                        },
                        "createdAt": {
                            "type": "date"
                        },
                        "rules": {
                            "properties": {
                                "id": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "updatedAt": {
                            "type": "date"
                        }
                    }
                },
                "cases-telemetry": {
                    "type": "object",
                    "dynamic": "false"
                },
                "cases-user-actions": {
                    "dynamic": "false",
                    "properties": {
                        "action": {
                            "type": "keyword"
                        },
                        "created_at": {
                            "type": "date"
                        },
                        "created_by": {
                            "properties": {
                                "username": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "owner": {
                            "type": "keyword"
                        },
                        "payload": {
                            "dynamic": "false",
                            "properties": {
                                "assignees": {
                                    "properties": {
                                        "uid": {
                                            "type": "keyword"
                                        }
                                    }
                                },
                                "comment": {
                                    "properties": {
                                        "externalReferenceAttachmentTypeId": {
                                            "type": "keyword"
                                        },
                                        "persistableStateAttachmentTypeId": {
                                            "type": "keyword"
                                        },
                                        "type": {
                                            "type": "keyword"
                                        }
                                    }
                                },
                                "connector": {
                                    "properties": {
                                        "type": {
                                            "type": "keyword"
                                        }
                                    }
                                }
                            }
                        },
                        "type": {
                            "type": "keyword"
                        }
                    }
                },
                "connector_token": {
                    "dynamic": "false",
                    "properties": {
                        "connectorId": {
                            "type": "keyword"
                        },
                        "tokenType": {
                            "type": "keyword"
                        }
                    }
                },
                "coreMigrationVersion": {
                    "type": "keyword"
                },
                "created_at": {
                    "type": "date"
                },
                "created_by": {
                    "type": "keyword"
                },
                "maintenance-window": {
                    "dynamic": "false",
                    "properties": {
                        "enabled": {
                            "type": "boolean"
                        },
                        "events": {
                            "type": "date_range",
                            "format": "epoch_millis||strict_date_optional_time"
                        }
                    }
                },
                "managed": {
                    "type": "boolean"
                },
                "namespace": {
                    "type": "keyword"
                },
                "namespaces": {
                    "type": "keyword"
                },
                "originId": {
                    "type": "keyword"
                },
                "references": {
                    "type": "nested",
                    "properties": {
                        "id": {
                            "type": "keyword"
                        },
                        "name": {
                            "type": "keyword"
                        },
                        "type": {
                            "type": "keyword"
                        }
                    }
                },
                "rules-settings": {
                    "dynamic": "false",
                    "properties": {
                        "flapping": {
                            "type": "object"
                        }
                    }
                },
                "type": {
                    "type": "keyword"
                },
                "typeMigrationVersion": {
                    "type": "version"
                },
                "updated_at": {
                    "type": "date"
                },
                "updated_by": {
                    "type": "keyword"
                }
            }
        }
    }
}
.kibana_task_manager/_mappings
{
    ".kibana_task_manager_1": {
        "mappings": {
            "dynamic": "strict",
            "_meta": {
                "migrationState": {
                    "convertingDocuments": false
                },
                "docVersions": {
                    "task": "10.3.0",
                    "background-task-node": "10.1.0"
                },
                "mappingVersions": {
                    "task": "10.3.0",
                    "background-task-node": "10.1.0"
                }
            },
            "properties": {
                "background-task-node": {
                    "dynamic": "false",
                    "properties": {
                        "id": {
                            "type": "keyword"
                        },
                        "last_seen": {
                            "type": "date"
                        }
                    }
                },
                "coreMigrationVersion": {
                    "type": "keyword"
                },
                "created_at": {
                    "type": "date"
                },
                "created_by": {
                    "type": "keyword"
                },
                "managed": {
                    "type": "boolean"
                },
                "namespace": {
                    "type": "keyword"
                },
                "namespaces": {
                    "type": "keyword"
                },
                "originId": {
                    "type": "keyword"
                },
                "references": {
                    "type": "nested",
                    "properties": {
                        "id": {
                            "type": "keyword"
                        },
                        "name": {
                            "type": "keyword"
                        },
                        "type": {
                            "type": "keyword"
                        }
                    }
                },
                "task": {
                    "dynamic": "false",
                    "properties": {
                        "attempts": {
                            "type": "integer"
                        },
                        "enabled": {
                            "type": "boolean"
                        },
                        "ownerId": {
                            "type": "keyword"
                        },
                        "partition": {
                            "type": "integer"
                        },
                        "priority": {
                            "type": "integer"
                        },
                        "retryAt": {
                            "type": "date"
                        },
                        "runAt": {
                            "type": "date"
                        },
                        "schedule": {
                            "properties": {
                                "interval": {
                                    "type": "keyword"
                                }
                            }
                        },
                        "scheduledAt": {
                            "type": "date"
                        },
                        "scope": {
                            "type": "keyword"
                        },
                        "status": {
                            "type": "keyword"
                        },
                        "taskType": {
                            "type": "keyword"
                        }
                    }
                },
                "type": {
                    "type": "keyword"
                },
                "typeMigrationVersion": {
                    "type": "version"
                },
                "updated_at": {
                    "type": "date"
                },
                "updated_by": {
                    "type": "keyword"
                }
            }
        }
    }
}

I did the following:

  1. Started up Kibana, observed the mapping conflict error.
  2. Updated the mappingVersions for .kibana_alerting_cases, started up Kibana with just the migrator node, observed the migration error.
  3. Updated the docVersions for .kibana_alerting_cases, started up Kibana with just the migrator node, startup 👍, started up Kibana with [bg,ui] roles, startup 👍
  4. Switched to this branch, started up Kibana with just the migrator node, startup 👍, started up Kibana with [bg,ui] roles, startup 👍
  5. After this "upgrade", I see the docVersion and mappingVersion for action_task_params and ad_hoc_run_params updated back to 10.2.0

@ymao1
Copy link
Contributor Author

ymao1 commented Jan 7, 2025

@elasticmachine merge upstream

@elasticmachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

  • [job] [logs] FTR Configs #3 / Rules Management - Prebuilt Rules - Prebuilt Rules Management @ess @serverless @skipInServerlessMKI Bootstrap Prebuilt Rules should skip installing fleet packages if they are already installed

Metrics [docs]

✅ unchanged

History

cc @ymao1

@ymao1 ymao1 merged commit 11e3a0b into elastic:main Jan 7, 2025
8 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12659747737

@ymao1 ymao1 deleted the backfill-actions-schema-updates-again branch January 7, 2025 21:37
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Jan 7, 2025
… intermediate release (elastic#204657)

## Summary

Redoing [this PR](elastic#203184) which
had to be [reverted](elastic#204218).
This should not be merged until [this update to the task manager v1
schema](elastic#204413) is released.

## To verify
1. Set `xpack.task_manager.unsafe.exclude_task_types:
['ad_hoc_run-backfill', 'actions:*']` in your Kibana config.
2. Run Kibana on main and create some detection rules that run
frequently, with actions.
3. Schedule a manual run for your detection rules.
- Because of the config, the `action_task_params` SO and the
`ad_hoc_run_task_params` SO will get written but not read yet.
4. Remove the `exclude_task_types` config and "upgrade" to this PR
branch and verify that rules continue to run and that the actions are
triggered and the manual rule runs go through
5. Re-add the `exclude_task_types` config and let the rule run again to
schedule action. Schedule another manual rule run.
6. Remove the `exclude_task_types` config and "downgrade" back to main
and verify that rules continue to run, the action gets triggered and
manual rule runs go through.

Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit 11e3a0b)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kowalczyk-krzysztof pushed a commit to kowalczyk-krzysztof/kibana that referenced this pull request Jan 7, 2025
… intermediate release (elastic#204657)

## Summary

Redoing [this PR](elastic#203184) which
had to be [reverted](elastic#204218).
This should not be merged until [this update to the task manager v1
schema](elastic#204413) is released.

## To verify
1. Set `xpack.task_manager.unsafe.exclude_task_types:
['ad_hoc_run-backfill', 'actions:*']` in your Kibana config.
2. Run Kibana on main and create some detection rules that run
frequently, with actions.
3. Schedule a manual run for your detection rules.
- Because of the config, the `action_task_params` SO and the
`ad_hoc_run_task_params` SO will get written but not read yet.
4. Remove the `exclude_task_types` config and "upgrade" to this PR
branch and verify that rules continue to run and that the actions are
triggered and the manual rule runs go through
5. Re-add the `exclude_task_types` config and let the rule run again to
schedule action. Schedule another manual rule run.
6. Remove the `exclude_task_types` config and "downgrade" back to main
and verify that rules continue to run, the action gets triggered and
manual rule runs go through.

Co-authored-by: Elastic Machine <[email protected]>
kibanamachine added a commit that referenced this pull request Jan 7, 2025
…changes for intermediate release (#204657) (#205821)

# Backport

This will backport the following commits from `main` to `8.x`:
- [Reapply &quot;[Response Ops][Alerting] Backfill actions schema
changes for intermediate release
(#204657)](#204657)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ying
Mao","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-01-07T21:37:01Z","message":"Reapply
\"[Response Ops][Alerting] Backfill actions schema changes for
intermediate release (#204657)\n\n## Summary\r\n\r\nRedoing [this
PR](#203184) which\r\nhad to be
[reverted](https://github.com/elastic/kibana/pull/204218).\r\nThis
should not be merged until [this update to the task manager
v1\r\nschema](#204413) is
released.\r\n\r\n## To verify\r\n1. Set
`xpack.task_manager.unsafe.exclude_task_types:\r\n['ad_hoc_run-backfill',
'actions:*']` in your Kibana config.\r\n2. Run Kibana on main and create
some detection rules that run\r\nfrequently, with actions.\r\n3.
Schedule a manual run for your detection rules.\r\n- Because of the
config, the `action_task_params` SO and the\r\n`ad_hoc_run_task_params`
SO will get written but not read yet.\r\n4. Remove the
`exclude_task_types` config and \"upgrade\" to this PR\r\nbranch and
verify that rules continue to run and that the actions are\r\ntriggered
and the manual rule runs go through\r\n5. Re-add the
`exclude_task_types` config and let the rule run again to\r\nschedule
action. Schedule another manual rule run.\r\n6. Remove the
`exclude_task_types` config and \"downgrade\" back to main\r\nand verify
that rules continue to run, the action gets triggered and\r\nmanual rule
runs go through.\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"11e3a0b77ef2e15bcd0eacf0572f27eeed60ea69","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["Feature:Alerting","release_note:skip","Feature:Task
Manager","Team:ResponseOps","v9.0.0","backport:prev-minor","v8.18.0"],"title":"Reapply
\"[Response Ops][Alerting] Backfill actions schema changes for
intermediate
release","number":204657,"url":"https://github.com/elastic/kibana/pull/204657","mergeCommit":{"message":"Reapply
\"[Response Ops][Alerting] Backfill actions schema changes for
intermediate release (#204657)\n\n## Summary\r\n\r\nRedoing [this
PR](#203184) which\r\nhad to be
[reverted](https://github.com/elastic/kibana/pull/204218).\r\nThis
should not be merged until [this update to the task manager
v1\r\nschema](#204413) is
released.\r\n\r\n## To verify\r\n1. Set
`xpack.task_manager.unsafe.exclude_task_types:\r\n['ad_hoc_run-backfill',
'actions:*']` in your Kibana config.\r\n2. Run Kibana on main and create
some detection rules that run\r\nfrequently, with actions.\r\n3.
Schedule a manual run for your detection rules.\r\n- Because of the
config, the `action_task_params` SO and the\r\n`ad_hoc_run_task_params`
SO will get written but not read yet.\r\n4. Remove the
`exclude_task_types` config and \"upgrade\" to this PR\r\nbranch and
verify that rules continue to run and that the actions are\r\ntriggered
and the manual rule runs go through\r\n5. Re-add the
`exclude_task_types` config and let the rule run again to\r\nschedule
action. Schedule another manual rule run.\r\n6. Remove the
`exclude_task_types` config and \"downgrade\" back to main\r\nand verify
that rules continue to run, the action gets triggered and\r\nmanual rule
runs go through.\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"11e3a0b77ef2e15bcd0eacf0572f27eeed60ea69"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/204657","number":204657,"mergeCommit":{"message":"Reapply
\"[Response Ops][Alerting] Backfill actions schema changes for
intermediate release (#204657)\n\n## Summary\r\n\r\nRedoing [this
PR](#203184) which\r\nhad to be
[reverted](https://github.com/elastic/kibana/pull/204218).\r\nThis
should not be merged until [this update to the task manager
v1\r\nschema](#204413) is
released.\r\n\r\n## To verify\r\n1. Set
`xpack.task_manager.unsafe.exclude_task_types:\r\n['ad_hoc_run-backfill',
'actions:*']` in your Kibana config.\r\n2. Run Kibana on main and create
some detection rules that run\r\nfrequently, with actions.\r\n3.
Schedule a manual run for your detection rules.\r\n- Because of the
config, the `action_task_params` SO and the\r\n`ad_hoc_run_task_params`
SO will get written but not read yet.\r\n4. Remove the
`exclude_task_types` config and \"upgrade\" to this PR\r\nbranch and
verify that rules continue to run and that the actions are\r\ntriggered
and the manual rule runs go through\r\n5. Re-add the
`exclude_task_types` config and let the rule run again to\r\nschedule
action. Schedule another manual rule run.\r\n6. Remove the
`exclude_task_types` config and \"downgrade\" back to main\r\nand verify
that rules continue to run, the action gets triggered and\r\nmanual rule
runs go through.\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"11e3a0b77ef2e15bcd0eacf0572f27eeed60ea69"}},{"branch":"8.x","label":"v8.18.0","branchLabelMappingKey":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Ying Mao <[email protected]>
crespocarlos pushed a commit to crespocarlos/kibana that referenced this pull request Jan 8, 2025
… intermediate release (elastic#204657)

## Summary

Redoing [this PR](elastic#203184) which
had to be [reverted](elastic#204218).
This should not be merged until [this update to the task manager v1
schema](elastic#204413) is released.

## To verify
1. Set `xpack.task_manager.unsafe.exclude_task_types:
['ad_hoc_run-backfill', 'actions:*']` in your Kibana config.
2. Run Kibana on main and create some detection rules that run
frequently, with actions.
3. Schedule a manual run for your detection rules.
- Because of the config, the `action_task_params` SO and the
`ad_hoc_run_task_params` SO will get written but not read yet.
4. Remove the `exclude_task_types` config and "upgrade" to this PR
branch and verify that rules continue to run and that the actions are
triggered and the manual rule runs go through
5. Re-add the `exclude_task_types` config and let the rule run again to
schedule action. Schedule another manual rule run.
6. Remove the `exclude_task_types` config and "downgrade" back to main
and verify that rules continue to run, the action gets triggered and
manual rule runs go through.

Co-authored-by: Elastic Machine <[email protected]>
CAWilson94 pushed a commit to CAWilson94/kibana that referenced this pull request Jan 13, 2025
… intermediate release (elastic#204657)

## Summary

Redoing [this PR](elastic#203184) which
had to be [reverted](elastic#204218).
This should not be merged until [this update to the task manager v1
schema](elastic#204413) is released.

## To verify
1. Set `xpack.task_manager.unsafe.exclude_task_types:
['ad_hoc_run-backfill', 'actions:*']` in your Kibana config.
2. Run Kibana on main and create some detection rules that run
frequently, with actions.
3. Schedule a manual run for your detection rules.
- Because of the config, the `action_task_params` SO and the
`ad_hoc_run_task_params` SO will get written but not read yet.
4. Remove the `exclude_task_types` config and "upgrade" to this PR
branch and verify that rules continue to run and that the actions are
triggered and the manual rule runs go through
5. Re-add the `exclude_task_types` config and let the rule run again to
schedule action. Schedule another manual rule run.
6. Remove the `exclude_task_types` config and "downgrade" back to main
and verify that rules continue to run, the action gets triggered and
manual rule runs go through.

Co-authored-by: Elastic Machine <[email protected]>
viduni94 pushed a commit to viduni94/kibana that referenced this pull request Jan 23, 2025
… intermediate release (elastic#204657)

## Summary

Redoing [this PR](elastic#203184) which
had to be [reverted](elastic#204218).
This should not be merged until [this update to the task manager v1
schema](elastic#204413) is released.

## To verify
1. Set `xpack.task_manager.unsafe.exclude_task_types:
['ad_hoc_run-backfill', 'actions:*']` in your Kibana config.
2. Run Kibana on main and create some detection rules that run
frequently, with actions.
3. Schedule a manual run for your detection rules.
- Because of the config, the `action_task_params` SO and the
`ad_hoc_run_task_params` SO will get written but not read yet.
4. Remove the `exclude_task_types` config and "upgrade" to this PR
branch and verify that rules continue to run and that the actions are
triggered and the manual rule runs go through
5. Re-add the `exclude_task_types` config and let the rule run again to
schedule action. Schedule another manual rule run.
6. Remove the `exclude_task_types` config and "downgrade" back to main
and verify that rules continue to run, the action gets triggered and
manual rule runs go through.

Co-authored-by: Elastic Machine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:prev-minor Backport to (9.0) the previous minor version (i.e. one version back from main) Feature:Alerting Feature:Task Manager release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v8.18.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants