Making snapshot name to scripted input in template #77

thalurur · 2021-06-10T17:26:52Z

Signed-off-by: Ravi Thaluru [email protected]

Issue #, if available:
https://discuss.opendistrocommunity.dev/t/universal-snapshot-policy-for-all-indexes/4815

Description of changes:
Processing snapshot field as a script instead of string so snapshot name can be derived from run time context - the two attributes supported during runtime are index and indexuuid

CheckList:
[X] Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

src/test/resources/mappings/cached-opendistro-ism-config.json

cryptk · 2021-09-20T22:35:17Z

Is there something that is holding this up? This feature would be greatly appreciated

…ed from the index or any runtime attributes instead of being constant Signed-off-by: Ravi Thaluru <[email protected]>

codecov-commenter · 2021-10-01T17:01:21Z

Codecov Report

Merging #77 (096aff4) into main (dafd3bb) will decrease coverage by 0.06%.
The diff coverage is 93.75%.

@@             Coverage Diff              @@
##               main      #77      +/-   ##
============================================
- Coverage     75.88%   75.81%   -0.07%     
- Complexity     1922     1924       +2     
============================================
  Files           260      260              
  Lines         10992    11002      +10     
  Branches       1720     1723       +3     
============================================
  Hits           8341     8341              
- Misses         1699     1710      +11     
+ Partials        952      951       -1

Impacted Files	Coverage Δ
...atemanagement/step/snapshot/AttemptSnapshotStep.kt	`89.02% <92.85%> (+0.13%)`	⬆️
...ment/indexstatemanagement/action/SnapshotAction.kt	`46.15% <100.00%> (ø)`
...atemanagement/model/action/SnapshotActionConfig.kt	`82.85% <100.00%> (ø)`
...nt/indexstatemanagement/model/destination/Slack.kt	`40.90% <0.00%> (-13.64%)`	⬇️
...anagement/indexstatemanagement/model/Transition.kt	`63.07% <0.00%> (-4.62%)`	⬇️
...arch/indexmanagement/rollup/RollupSearchService.kt	`57.40% <0.00%> (-3.71%)`	⬇️
...ent/indexstatemanagement/util/ManagedIndexUtils.kt	`77.92% <0.00%> (-1.30%)`	⬇️
...earch/indexmanagement/transform/model/Transform.kt	`87.00% <0.00%> (-0.50%)`	⬇️
...nt/indexstatemanagement/model/destination/Chime.kt	`54.54% <0.00%> (+13.63%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dafd3bb...096aff4. Read the comment docs.

...lin/org/opensearch/indexmanagement/indexstatemanagement/step/snapshot/AttemptSnapshotStep.kt

dbbaughe · 2021-10-01T18:45:50Z

...lin/org/opensearch/indexmanagement/indexstatemanagement/step/snapshot/AttemptSnapshotStep.kt

@@ -159,6 +173,7 @@ class AttemptSnapshotStep(
    }

    companion object {
+        val validTopContextFields = setOf("index", "indexUuid")


Just to confirm is this all users have asked for? (Access to index/indexUuid in the snapshot name)

the users only asked for indexName in the issue, I have added indexUuid too since its readily available

dbbaughe · 2021-10-01T18:47:04Z

...lin/org/opensearch/indexmanagement/indexstatemanagement/step/snapshot/AttemptSnapshotStep.kt

-                        .format(DateTimeFormatter.ofPattern("uuuu.MM.dd-HH:mm:ss.SSS", Locale.ROOT))
-                )
+            val snapshotScript = Script(ScriptType.INLINE, Script.DEFAULT_TEMPLATE_LANG, config.snapshot, mapOf())
+            snapshotName = compileTemplate(snapshotScript, managedIndexMetaData, indexName).plus(snapshotNamePrefix)


Any behavior differences for users w/ existing snapshot actions or they should continue to work (and behave) as before this fix?

i.e. their snapshot names should not change somehow because of this change if they are not using the ctx variable and are just using a string.

They should work the same way, the mustache template compiler will return the same string value if there is no placeholder fields

dbbaughe · 2021-10-01T18:48:58Z

...lin/org/opensearch/indexmanagement/indexstatemanagement/step/snapshot/AttemptSnapshotStep.kt

+        val compiledValue = scriptService.compile(template, TemplateScript.CONTEXT)
+            .newInstance(template.params + mapOf("ctx" to contextMap))
+            .execute()
+        return if (compiledValue.isBlank()) defaultValue else compiledValue


When does compiledValue.isBlank() occur?

if someone puts in a template with just a replaceable field and that is not valid e.g. {{ctx.name}} or {{ctx.anything_not_allowed}} this will be empty string

If someone specifies the template as {{ctx.anything_not_allowed}}_snapshot then the compiled value is just _snapshot

If someone specifies the template as {ctx.anything_not_allowed} since this is not a valid template placeholder it will be compiled as {ctx.anything_not_allowed}

Hmm can someone specify an empty string as the snapshot name currently or do we block that?
i.e. if currently sameone has "" and it's allowed then instead of "" + the postfix stuff it'll change to $indexName + postfix stuff (since it looks like we add the index name as the default to use if this returns blank)

good point, there is no check for it being non empty at the moment the field needs to be present in the policy but it can be empty string.

I can update the default value to empty string in that case - i.e if the original name is empty string

Signed-off-by: Ravi Thaluru <[email protected]>

* Add integTest script to the repo (#94) Signed-off-by: Peter Zhu <[email protected]> * Removing Usages of Action Get Call and using listeners (#100) Signed-off-by: Aditya Jindal <[email protected]> * Enhance ISM template (#105) Signed-off-by: bowenlan-amzn <[email protected]> * Explain response still use old opendistro policy id (#109) * Explain response still use old opendistro policy id * Use hardcoded policyid setting in tests for explain response * Trying to fix flaky tests * Storing user information as part of the job when security plugin is installed (#113) Signed-off-by: Ravi Thaluru <[email protected]> * ISM/Notification channel support (#117) * Updates NAME of transport actions * Upgrades Kotlin version, updates dependencies on other OS plugins, adds notification plugin as a test resource and includes it in test clusters * Adds support for Channels in error notifications and notification actions * Adds support for sending notifications to channels * Adds support for publishing notifications to the legacy destinations through the Notification plugin and some cleanup * Removes notification alerting jar dependency * Adds compile only dep on commons codec for digest utils sha1 method in ism rollup * Updates Error Notification to make channel/destination nullable, and adds helper methods for publish calls * Constructs URL for legacy custom webhook Signed-off-by: Drew Baugher <[email protected]> * Fixes Feature enum and dep Signed-off-by: Drew Baugher <[email protected]> * Trying something else Signed-off-by: Drew Baugher <[email protected]> * Addresses comments Signed-off-by: Drew Baugher <[email protected]> * Storing user object in all APIs and enabling filter of response based on user (#115) Signed-off-by: Ravi Thaluru <[email protected]> * Upgrade dependencies to 1.1 and build snapshot by default. (#121) Signed-off-by: dblock <[email protected]> * Security improvements (#126) Signed-off-by: Ravi Thaluru <[email protected]> * Removes support for notification plugin (#136) Signed-off-by: Drew Baugher <[email protected]> * Updating security filtering logic (#137) Signed-off-by: Ravi Thaluru <[email protected]> * Release note for 1.1.0.0 release. (#139) * Release note for 1.1.0.0 release. Signed-off-by: bowenlan-amzn <[email protected]> * Correct copyright notices * Uses published daily snapshot dependencies (#141) Signed-off-by: Drew Baugher <[email protected]> * License header check (#142) * Provide default copyright header using IDE feature Signed-off-by: bowenlan-amzn <[email protected]> * Address #103 history write index is rolled over even if the history indices are disabled * Removed integtest.sh. (#148) Signed-off-by: dblock <[email protected]> * Adds mavenLocal back to repositories (#158) Signed-off-by: Drew Baugher <[email protected]> * Making snapshot name to scripted input in template (#77) Signed-off-by: Ravi Thaluru <[email protected]> * Fix issues with security changes in rollup runnner (#161) * Updates index management version to 1.2 (#157) * Updates index management version to 1.2 * Updates job scheduler snapshot to 1.2 in test resources Signed-off-by: Robert Downs <[email protected]> * Adds setting to search all rollup jobs on a target index (#165) * Adds cluster setting to search all rollup jobs Signed-off-by: Clay Downs <[email protected]> * Adds implementation for the delay feature in rollup jobs (#147) * Adds delay implementation for rollup jobs * Removes non-continuous delay implementation * Adds additional rollup delay tests Signed-off-by: Clay Downs <[email protected]> * Updates testCompile mockito version, adds AwaitsFix annotation to MetadataRegressionIT tests (#168) * Updates testCompile mockito version to match OpenSearch changes * AwaitsFix on the failing MetadataRegressionIT tests Signed-off-by: Robert Downs <[email protected]> * Adds cluster setting to configure index state management jitter (#153) * Adds jitter cluster setting, sets jitter to 0 for ISM tests Signed-off-by: Clay Downs <[email protected]> * Allows out of band rollovers on an index without causing ISM to fail (#180) * Allows out of band rollovers on an index without causing ISM to fail Signed-off-by: Drew Baugher <[email protected]> * Fixes detekt issue Signed-off-by: Drew Baugher <[email protected]> * Remove policy API on read only indices (#182) * In explain API not showing the total count to all users (#185) Co-authored-by: Peter Zhu <[email protected]> Co-authored-by: Aditya Jindal <[email protected]> Co-authored-by: Bowen Lan <[email protected]> Co-authored-by: Ravi <[email protected]> Co-authored-by: Drew Baugher <[email protected]> Co-authored-by: Daniel Doubrovkine (dB.) <[email protected]>

…t#77) Signed-off-by: Ravi Thaluru <[email protected]>

thalurur marked this pull request as draft June 10, 2021 17:27

thalurur mentioned this pull request Jun 10, 2021

Making snapshot name a scripted field opendistro-for-elasticsearch/index-management#425

Closed

thalurur changed the title ~~Making snapshot name to script template so snapshot name can be derived from the index or any runtime attributes instead of being constant~~ Making snapshot name to scripted input in template Jun 10, 2021

dbbaughe reviewed Jun 10, 2021

View reviewed changes

src/test/resources/mappings/cached-opendistro-ism-config.json Outdated Show resolved Hide resolved

thalurur added 2 commits October 1, 2021 09:27

Making snapshot name to script template so snapshot name can be deriv…

87df7ea

…ed from the index or any runtime attributes instead of being constant Signed-off-by: Ravi Thaluru <[email protected]>

Not changing mapping type of snapshot name

6636c68

thalurur force-pushed the rsnap branch from b25a9fd to 6636c68 Compare October 1, 2021 16:28

Correctly building mustache template script in the attempt step

c7af351

thalurur marked this pull request as ready for review October 1, 2021 16:38

dbbaughe reviewed Oct 1, 2021

View reviewed changes

...lin/org/opensearch/indexmanagement/indexstatemanagement/step/snapshot/AttemptSnapshotStep.kt Outdated Show resolved Hide resolved

dbbaughe reviewed Oct 1, 2021

View reviewed changes

thalurur added 2 commits October 1, 2021 12:59

PR comments

5b7b8c3

Signed-off-by: Ravi Thaluru <[email protected]>

Honoring empty snapshot names

096aff4

dbbaughe approved these changes Oct 1, 2021

View reviewed changes

thalurur merged commit 905c16e into opensearch-project:main Oct 1, 2021

wuychn pushed a commit to ochprince/index-management that referenced this pull request Mar 16, 2023

Making snapshot name to scripted input in template (opensearch-projec…

3958175

…t#77) Signed-off-by: Ravi Thaluru <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Making snapshot name to scripted input in template #77

Making snapshot name to scripted input in template #77

thalurur commented Jun 10, 2021 •

edited

Loading

cryptk commented Sep 20, 2021

codecov-commenter commented Oct 1, 2021 •

edited

Loading

dbbaughe Oct 1, 2021

thalurur Oct 1, 2021

dbbaughe Oct 1, 2021

dbbaughe Oct 1, 2021

thalurur Oct 1, 2021

dbbaughe Oct 1, 2021

thalurur Oct 1, 2021 •

edited

Loading

dbbaughe Oct 1, 2021

thalurur Oct 1, 2021 •

edited

Loading

Making snapshot name to scripted input in template #77

Making snapshot name to scripted input in template #77

Conversation

thalurur commented Jun 10, 2021 • edited Loading

cryptk commented Sep 20, 2021

codecov-commenter commented Oct 1, 2021 • edited Loading

Codecov Report

dbbaughe Oct 1, 2021

Choose a reason for hiding this comment

thalurur Oct 1, 2021

Choose a reason for hiding this comment

dbbaughe Oct 1, 2021

Choose a reason for hiding this comment

dbbaughe Oct 1, 2021

Choose a reason for hiding this comment

thalurur Oct 1, 2021

Choose a reason for hiding this comment

dbbaughe Oct 1, 2021

Choose a reason for hiding this comment

thalurur Oct 1, 2021 • edited Loading

Choose a reason for hiding this comment

dbbaughe Oct 1, 2021

Choose a reason for hiding this comment

thalurur Oct 1, 2021 • edited Loading

Choose a reason for hiding this comment

thalurur commented Jun 10, 2021 •

edited

Loading

codecov-commenter commented Oct 1, 2021 •

edited

Loading

thalurur Oct 1, 2021 •

edited

Loading

thalurur Oct 1, 2021 •

edited

Loading