Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix remove of azure files #18571

Merged
merged 1 commit into from
May 27, 2016
Merged

Fix remove of azure files #18571

merged 1 commit into from
May 27, 2016

Conversation

dadoonet
Copy link
Member

Probably when we updated Azure SDK, we introduced a regression.
Actually, we are not able to remove files anymore.

For example, if you register a new azure repository, the snapshot service tries to create a temp file and then remove it.
Removing does not work and you can see in logs:

[2016-05-18 11:03:24,914][WARN ][org.elasticsearch.cloud.azure.blobstore] [azure] can not remove [tests-ilmRPJ8URU-sh18yj38O6g/] in container {elasticsearch-snapshots}: The specified blob does not exist.

This fix deals with that. It now list all the files in a flatten mode, remove in the full URL the server and the container name.

As an example, when you are removing a blob which full name is https://dpi24329.blob.core.windows.net/elasticsearch-snapshots/bar/test you need to actually call Azure SDK with bar/test as the path, elasticsearch-snapshots is the container.

Related to #16472.
Related to #18436.

Backport of #18451 in 2.x branch

To test it, I ran some manual tests:

On my laptop, create a file /path/to/azure/config/elasticsearch.yml:

cloud.azure.storage.default.account: ACCOUNT
cloud.azure.storage.default.key: KEY

Run AzureRepositoryF#main() with -Des.cluster.routing.allocation.disk.threshold_enabled=false -Des.path.home=/path/to/azure/ options.

Then run:

curl -XDELETE localhost:9200/foo?pretty
curl -XDELETE localhost:9200/_snapshot/my_backup1?pretty
curl -XPUT localhost:9200/foo/bar/1?pretty -d '{
 "foo": "bar"
}'
curl -XPOST localhost:9200/foo/_refresh?pretty
curl -XGET localhost:9200/foo/_count?pretty
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
   "type": "azure"
}'

curl -XPOST "localhost:9200/_snapshot/my_backup1/snap1?pretty&wait_for_completion=true"
curl -XDELETE localhost:9200/foo?pretty
curl -XPOST "localhost:9200/_snapshot/my_backup1/snap1/_restore?pretty&wait_for_completion=true"
curl -XGET localhost:9200/foo/_count?pretty

Then check files we have on azure platform using the console.
Then run:

curl -XDELETE localhost:9200/_snapshot/my_backup1/snap1?pretty

Then check files we have on azure platform using the console and verify that everything has been cleaned.

Probably when we updated Azure SDK, we introduced a regression.
Actually, we are not able to remove files anymore.

For example, if you register a new azure repository, the snapshot service tries to create a temp file and then remove it.
Removing does not work and you can see in logs:

```
[2016-05-18 11:03:24,914][WARN ][org.elasticsearch.cloud.azure.blobstore] [azure] can not remove [tests-ilmRPJ8URU-sh18yj38O6g/] in container {elasticsearch-snapshots}: The specified blob does not exist.
```

This fix deals with that. It now list all the files in a flatten mode, remove in the full URL the server and the container name.

As an example, when you are removing a blob which full name is `https://dpi24329.blob.core.windows.net/elasticsearch-snapshots/bar/test` you need to actually call Azure SDK with `bar/test` as the path, `elasticsearch-snapshots` is the container.

Related to elastic#16472.
Related to elastic#18436.

Backport of elastic#18451 in 2.x branch

To test it, I ran some manual tests:

On my laptop, create a file `/path/to/azure/config/elasticsearch.yml`:

```yml
cloud.azure.storage.default.account: ACCOUNT
cloud.azure.storage.default.key: KEY
```

Run `AzureRepositoryF#main()` with `-Des.cluster.routing.allocation.disk.threshold_enabled=false -Des.path.home=/path/to/azure/` options.

Then run:

```sh
curl -XDELETE localhost:9200/foo?pretty
curl -XDELETE localhost:9200/_snapshot/my_backup1?pretty
curl -XPUT localhost:9200/foo/bar/1?pretty -d '{
 "foo": "bar"
}'
curl -XPOST localhost:9200/foo/_refresh?pretty
curl -XGET localhost:9200/foo/_count?pretty
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
   "type": "azure"
}'

curl -XPOST "localhost:9200/_snapshot/my_backup1/snap1?pretty&wait_for_completion=true"
curl -XDELETE localhost:9200/foo?pretty
curl -XPOST "localhost:9200/_snapshot/my_backup1/snap1/_restore?pretty&wait_for_completion=true"
curl -XGET localhost:9200/foo/_count?pretty
```

Then check files we have on azure platform using the console.
Then run:

```
curl -XDELETE localhost:9200/_snapshot/my_backup1/snap1?pretty
```

Then check files we have on azure platform using the console and verify that everything has been cleaned.
@dadoonet
Copy link
Member Author

@imotov I backported #18451 for 2.x branch. Would you mind giving a quick look?

@imotov
Copy link
Contributor

imotov commented May 27, 2016

LGTM

@imotov imotov removed their assignment May 27, 2016
@imotov imotov removed the review label May 27, 2016
@dadoonet dadoonet merged commit 92382d8 into elastic:2.x May 27, 2016
@clintongormley clintongormley added :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs and removed :Plugin Cloud Azure labels Feb 14, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
>bug :Distributed Coordination/Snapshot/Restore Anything directly related to the `_snapshot/*` APIs v2.4.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants