Revision tree repair tool #2857
Assignees
Comments
|
|
@adamcfraser here's what I have so far: release/1.3.1.3...feature/issue_2847_repair_tool Can you take a quick pass to make sure we're on the same page? In particular the RepairBucket method What's still pending:
|
tleyden
added a commit
that referenced
this issue
Aug 31, 2017
* TestRepairRevsHistoryWithCycles() * Add Repair() — unit test passes now * Add repair bucket (in progress) * Test passes * Fix dry run * TestRepairBucketRevTreeCycles() passes * Add _repair endpoint * Fix invocation of InitFrom * Refactor RepairBucket() to return repaired docs to enable more test assertions * WriteRepairedDocsToDisk by default to make diagnosis of repair tool easier * Run gofmt + goimports * PR feedback, remove redundant repair_job * DocTransformer takes raw bytes instead of the marshalled document * Return RepairBucketResult with doc and and repair job * Update _repair endpoint to marshal result to response * Add WriteRepairedDocsToBucket(), fix super nasty dev bug along the way * Fix bug in WriteRepairedDocsToBucket() * Change to 24 hours and fix up _sync: doc id * gofmt * Return DryRun and BackupOrDryRunDocId in results * Repair() -> RepairCycles() * More test docs * Handle bucket.Update err * Add TestRepairBucketDryRun() * Use bucket.GetRaw() * Gofmt + goimports * TestRepairBucket had wrong number of docs in assertion + saw error on drone that made me think there is interference between walrus buckets w/ same name. * Fix compile error
tleyden
added a commit
that referenced
this issue
Aug 31, 2017
* TestRepairRevsHistoryWithCycles() * Add Repair() — unit test passes now * Add repair bucket (in progress) * Test passes * Fix dry run * TestRepairBucketRevTreeCycles() passes * Add _repair endpoint * Fix invocation of InitFrom * Refactor RepairBucket() to return repaired docs to enable more test assertions * WriteRepairedDocsToDisk by default to make diagnosis of repair tool easier * Run gofmt + goimports * PR feedback, remove redundant repair_job * DocTransformer takes raw bytes instead of the marshalled document * Return RepairBucketResult with doc and and repair job * Update _repair endpoint to marshal result to response * Add WriteRepairedDocsToBucket(), fix super nasty dev bug along the way * Fix bug in WriteRepairedDocsToBucket() * Change to 24 hours and fix up _sync: doc id * gofmt * Return DryRun and BackupOrDryRunDocId in results * Repair() -> RepairCycles() * More test docs * Handle bucket.Update err * Add TestRepairBucketDryRun() * Use bucket.GetRaw() * Gofmt + goimports * TestRepairBucket had wrong number of docs in assertion + saw error on drone that made me think there is interference between walrus buckets w/ same name. * Fix compile error
tleyden
added a commit
that referenced
this issue
Aug 31, 2017
* TestRepairRevsHistoryWithCycles() * Add Repair() — unit test passes now * Add repair bucket (in progress) * Test passes * Fix dry run * TestRepairBucketRevTreeCycles() passes * Add _repair endpoint * Fix invocation of InitFrom * Refactor RepairBucket() to return repaired docs to enable more test assertions * WriteRepairedDocsToDisk by default to make diagnosis of repair tool easier * Run gofmt + goimports * PR feedback, remove redundant repair_job * DocTransformer takes raw bytes instead of the marshalled document * Return RepairBucketResult with doc and and repair job * Update _repair endpoint to marshal result to response * Add WriteRepairedDocsToBucket(), fix super nasty dev bug along the way * Fix bug in WriteRepairedDocsToBucket() * Change to 24 hours and fix up _sync: doc id * gofmt * Return DryRun and BackupOrDryRunDocId in results * Repair() -> RepairCycles() * More test docs * Handle bucket.Update err * Add TestRepairBucketDryRun() * Use bucket.GetRaw() * Gofmt + goimports * TestRepairBucket had wrong number of docs in assertion + saw error on drone that made me think there is interference between walrus buckets w/ same name. * Fix compile error
tleyden
added a commit
that referenced
this issue
Aug 31, 2017
* Fixes #2857: Revision tree repair tool (#2866) * TestRepairRevsHistoryWithCycles() * Add Repair() — unit test passes now * Add repair bucket (in progress) * Test passes * Fix dry run * TestRepairBucketRevTreeCycles() passes * Add _repair endpoint * Fix invocation of InitFrom * Refactor RepairBucket() to return repaired docs to enable more test assertions * WriteRepairedDocsToDisk by default to make diagnosis of repair tool easier * Run gofmt + goimports * PR feedback, remove redundant repair_job * DocTransformer takes raw bytes instead of the marshalled document * Return RepairBucketResult with doc and and repair job * Update _repair endpoint to marshal result to response * Add WriteRepairedDocsToBucket(), fix super nasty dev bug along the way * Fix bug in WriteRepairedDocsToBucket() * Change to 24 hours and fix up _sync: doc id * gofmt * Return DryRun and BackupOrDryRunDocId in results * Repair() -> RepairCycles() * More test docs * Handle bucket.Update err * Add TestRepairBucketDryRun() * Use bucket.GetRaw() * Gofmt + goimports * TestRepairBucket had wrong number of docs in assertion + saw error on drone that made me think there is interference between walrus buckets w/ same name. * Fix compile error * Fix null return value when no repairs done * Change from Warn -> Crud
|
@tleyden I went through #2847 about dangling parent. I am more interested in identifying documents have revision tree issues. |
|
@ArihantRk the docs are here: https://github.com/couchbase/sync_gateway/wiki/Issue-2857-Repair-Tool If you do a dry run, it will show you which docs it would repair (but not actually make any changes) |
|
@ArihantRk The tool is only intended to be used in conjunction with data affected by a specific maintenance patches, and only if that data can't be otherwise rolled back. It's not intended for general detection of 'revision tree issues'. If you feel like you're in a situation that requires this tool, you should review w/ support. |
This was referenced Sep 11, 2017
tleyden
added a commit
that referenced
this issue
Sep 29, 2017
* TestRepairRevsHistoryWithCycles() * Add Repair() — unit test passes now * Add repair bucket (in progress) * Test passes * Fix dry run * TestRepairBucketRevTreeCycles() passes * Add _repair endpoint * Fix invocation of InitFrom * Refactor RepairBucket() to return repaired docs to enable more test assertions * WriteRepairedDocsToDisk by default to make diagnosis of repair tool easier * Run gofmt + goimports * PR feedback, remove redundant repair_job * DocTransformer takes raw bytes instead of the marshalled document * Return RepairBucketResult with doc and and repair job * Update _repair endpoint to marshal result to response * Add WriteRepairedDocsToBucket(), fix super nasty dev bug along the way * Fix bug in WriteRepairedDocsToBucket() * Change to 24 hours and fix up _sync: doc id * gofmt * Return DryRun and BackupOrDryRunDocId in results * Repair() -> RepairCycles() * More test docs * Handle bucket.Update err * Add TestRepairBucketDryRun() * Use bucket.GetRaw() * Gofmt + goimports * TestRepairBucket had wrong number of docs in assertion + saw error on drone that made me think there is interference between walrus buckets w/ same name. * Fix compile error
tleyden
added a commit
that referenced
this issue
Oct 3, 2017
* TestRepairRevsHistoryWithCycles() * Add Repair() — unit test passes now * Add repair bucket (in progress) * Test passes * Fix dry run * TestRepairBucketRevTreeCycles() passes * Add _repair endpoint * Fix invocation of InitFrom * Refactor RepairBucket() to return repaired docs to enable more test assertions * WriteRepairedDocsToDisk by default to make diagnosis of repair tool easier * Run gofmt + goimports * PR feedback, remove redundant repair_job * DocTransformer takes raw bytes instead of the marshalled document * Return RepairBucketResult with doc and and repair job * Update _repair endpoint to marshal result to response * Add WriteRepairedDocsToBucket(), fix super nasty dev bug along the way * Fix bug in WriteRepairedDocsToBucket() * Change to 24 hours and fix up _sync: doc id * gofmt * Return DryRun and BackupOrDryRunDocId in results * Repair() -> RepairCycles() * More test docs * Handle bucket.Update err * Add TestRepairBucketDryRun() * Use bucket.GetRaw() * Gofmt + goimports * TestRepairBucket had wrong number of docs in assertion + saw error on drone that made me think there is interference between walrus buckets w/ same name. * Fix compile error
adamcfraser
pushed a commit
that referenced
this issue
Oct 3, 2017
…go into infinite loop + rev tree repair tool) (#2869) * Fixes #2847 Getting doc history can go into infinite loop: Backport for 1.4.1.2 (#2858) * Fixes #2847 Getting doc history can go into infinite loop: Backport for 1.3.1.2 (#2856) * Repro case for #2847 - Getting doc history can go into infinite loop #2847 * Add revTree.Validate() method * Invoke rawDoc.History.RenderGraphvizDot(), change timeout checking * Add new test * Cherry pick commit from feature/issue_2847_cycles Cherry pick commit d8feb1d from feature/issue_2847_cycles, which is based on the master branch * Fixes issue #2847 by wwitching the dangling parent check in pruneRevisions to happen after branches are deleted #2847 (comment) * Comments on test * Fixes issue #2847 by fixing marshalJSON to better handle dangling parents during marshal #2847 (comment) * Replace getHistory with getValidatedHistory TestRevsHistoryInfiniteLoop now passes * Rename getValidatedHistory -> getHistory * Gofmt * Remove parent rev from history string * Remove unneeded fmt.Sprintf() * Run sg-accel tests * Try pointing to specific commit to fix build failure * Use git url instead of ssh for sga accel repo * Revert "Use git url instead of ssh for sga accel repo" This reverts commit 1c5e061. * Revert "Try pointing to specific commit to fix build failure" This reverts commit dd3f9d9. * Revert "Run sg-accel tests" This reverts commit d5cc940. * Remove rawDoc.History.Validate(). Does not help test catch any issues. * Remove commented log * Remove unneeded import # Conflicts: # db/revtree_test.go * Fix compile errors # Conflicts: # db/crud.go # db/revtree_test.go * Fixes #2857: Revision tree repair tool (#2866) * TestRepairRevsHistoryWithCycles() * Add Repair() — unit test passes now * Add repair bucket (in progress) * Test passes * Fix dry run * TestRepairBucketRevTreeCycles() passes * Add _repair endpoint * Fix invocation of InitFrom * Refactor RepairBucket() to return repaired docs to enable more test assertions * WriteRepairedDocsToDisk by default to make diagnosis of repair tool easier * Run gofmt + goimports * PR feedback, remove redundant repair_job * DocTransformer takes raw bytes instead of the marshalled document * Return RepairBucketResult with doc and and repair job * Update _repair endpoint to marshal result to response * Add WriteRepairedDocsToBucket(), fix super nasty dev bug along the way * Fix bug in WriteRepairedDocsToBucket() * Change to 24 hours and fix up _sync: doc id * gofmt * Return DryRun and BackupOrDryRunDocId in results * Repair() -> RepairCycles() * More test docs * Handle bucket.Update err * Add TestRepairBucketDryRun() * Use bucket.GetRaw() * Gofmt + goimports * TestRepairBucket had wrong number of docs in assertion + saw error on drone that made me think there is interference between walrus buckets w/ same name. * Fix compile error * Fix null return value when no repairs done * Change from Warn -> Crud * Disable _repair endpoint * Fix test compile errors * Fixes #2892 repair tool efficiency (#2893) * Fixes #2892 - Repair tool efficiency improvements Initial first pass at iterating over the view w/ paging * Add documentation * More comments * Comment regarding min pageSIzeViewResult * pull ViewQueryPageSize out to a constant. Enhance tests to add more docs to excercise the view iteration * Address PR feedback * Update comment * Change constant to DefaultViewQueryPageSize * Run gofmt # Conflicts: # base/constants.go # db/repair_bucket_test.go * Adds support for custom TTL of repaired docs and ViewQueryPageSize deafults (#2902) Adds support for custom TTL of repaired docs and ViewQueryPageSize defaults * correctly apply _repair parameter value for params.RepairedFileTTL (#2909) * Fixes #2919 Revtree repair tool gets stuck if node points to itself a… (#2920) * Fixes #2919 Revtree repair tool gets stuck if node points to itself as parent * PR feedback
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
To repair the revision trees with cycles caused by #2847
Instructions
See https://github.com/couchbase/sync_gateway/wiki/Issue-2857-Repair-Tool
Test Instructions
data-bucketbucketwget https://gist.githubusercontent.com/tleyden/bc4d9cae6ac8bfe408c373e673ff58e9/raw/659700bc6287ed44cdd8158a9e8a82a33cb80e08/data_for_testcbc cp data_for_test -U couchbase://localhost/data-bucketcurl http://{{host}}:{{port_admin}}/{{db}}/data_for_test?revs=trueand looking for"reason": "Internal error: getHistory found cycle in revision treeThe text was updated successfully, but these errors were encountered: