API to download export results #9075

Marishka17 · 2025-02-07T12:10:30Z

Motivation and context

This PR introduces the following changes:

Several server API endpoints have been deprecated or no longer handle the process of exporting specific resources:
- [API] GET /api/projects/id/dataset?action=import_status is deprecated
- [API] GET /api/projects/id/dataset(no action parameter or action != import_status) returns 410 status code - API endpoint no longer used to handle export datasets flow
- [API] GET /api/projects/id/annotations returns 410 status code - API endpoint no longer used to handle export annotations flow
- [API] GET /api/projects/id/backup returns 410 status code - API endpoint no longer used to handle export backups flow
- [API] GET /api/tasks/id/backup returns 410 status code - API endpoint no longer used to handle export backups flow
- [API] GET /api/tasks/id/annotations?format= returns 410 status code - API endpoint no longer used to handle export annotations flow
- [API] GET /api/tasks/d/dataset returns 410 status code - API endpoint no longer used to handle export datasets flow
- [API] GET /api/jobs/id/annotations?format= returns 410 status code - API endpoint no longer used to handle export annotations flow
- [API] GET /api/jobs/id/dataset returns 410 status code - API endpoint no longer used to handle export datasets flow
Introduced new "private" endpoints to download prepared files (they are hidden in the generated server schema):
- [API] GET /api/projects/id/dataset/download?rq_id=rq_id
- [API] GET /api/projects/id/backup/download?rq_id=rq_id
- [API] GET /api/tasks/id/dataset/download?rq_id=rq_id
- [API] GET /api/tasks/id/backup/download?rq_id=rq_id
- [API] GET /api/jobs/id/dataset/download?rq_id=rq_id
Changed permissions used when downloading a prepared file (now admin and RQ job owner have rights to download a prepared file related to a specific background job (it's a questionable topic whether should be resource access checked also))
Added result filename saving to RQ metadata for export-related RQ jobs:

{
    ...
    "result_filename": str // is used as the final filename when downloading files/uploading to cloud storage
}

Final file name is taken from meta when uploading a file to cloud storage (key and key_pattern args have been removed)

How has this been tested?

Checklist

I submit my changes into the develop branch
I have created a changelog fragment
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues (see GitHub docs)

TODO:

Extract unrelated fixes/changes into separate PRs

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.

Marishka17 · 2025-02-27T15:20:45Z

/check

cvat/apps/engine/views.py

Marishka17 · 2025-03-07T10:29:56Z

@zhiltsov-max, Could you please check that deprecated API is not used in Human Protocol oracles (and update if needed)?

cvat/apps/engine/views.py

SpecLad · 2025-03-11T16:53:51Z

cvat/apps/engine/views.py

        """),
        parameters=[
+            # --- Deprecated params section ---


I think these parameters are removed, not just deprecated.

I did not remove these parameters from the specification because they are still used to determine whether a request is made to export the dataset or just to get raw annotations. So, I think it would be better to remove them from the schema once the endpoint no longer returns a 410 response.

I disagree. By listing these parameters in the schema you are making the claim that they are supported, whereas in fact they are not. I can see the utility in keeping them around for a short while, so that people who read the schema can understand what happened to them; but in this case you should change the description to something accurate, e.g. "this parameter is no longer supported".

FWIW, I also think we should keep the logic that returns 410 if these parameters are present indefinitely, or at least for a long time. It costs very little, and it ensures that clients that use the old API get an error. OTOH, we probably don't need to keep these parameters in the schema for very long.

cvat/apps/engine/views.py

SpecLad · 2025-03-11T17:11:14Z

cvat/apps/engine/tests/utils.py

+            response = self.client.put(url, data=data)
+        return response
+
+    def _wait_request_to_be_finished(


It's odd to see this here. IIRC, for unit tests we run RQ jobs in synchronous mode, so no waiting should be necessary - the job should be complete immediately after the initiating request.

it's true, but I believe there are still several advantages of adding a waiting step:

We do not need to update related tests if we run these tests in asynchronous mode

This way we test that /api/requests/rq_id API returns the expected code (200, 401, 403)

if we run these tests in asynchronous mode

Is it possible?

This way we test that /api/requests/rq_id API returns the expected code (200, 401, 403)

I've nothing against calling this API, I just don't think we need to do it in a loop.

Is it possible?

Warum nicht?

Warum nicht?

Well... how would you do it? AFAICS, cvat.settings.testing forces all queues to use sync mode. Moreover, it also uses a fake Redis implementation, so it wouldn't be possible to run RQ workers anyway.

cvat/apps/engine/tests/utils.py

sonarqubecloud · 2025-03-13T12:44:19Z

Quality Gate passed

Issues
5 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
2.1% Duplication on New Code

See analysis details on SonarQube Cloud

Marishka17 added 3 commits February 7, 2025 12:56

Add new API && remove deprecated && refactor working with rq meta

2a2dc86

Remove outdated tests

e3d6da9

Resolve conflicts

a01a8a8

Marishka17 added the server label Feb 7, 2025

Marishka17 requested review from zhiltsov-max and SpecLad as code owners February 7, 2025 12:10

Rename method

0f71c08

This was referenced Feb 7, 2025

Drop support for the GET /api/tasks/<id>/status API #9076

Open

[engine] Fix some type hints when request is passed #9077

Merged

Marishka17 added 4 commits February 9, 2025 17:59

Split RQMeta into several classes && small fixes

a22e10f

Black code

19a322a

f

f535d32

Fix typo

3c9895b

Marishka17 mentioned this pull request Feb 10, 2025

Refactor way of working with RQ meta #9082

Merged

6 tasks

Marishka17 added 13 commits February 14, 2025 14:19

Resolve conflicts

a896dfd

Fix merge

5aceaf4

Resolve conflicts

f05f591

Fix missing import

1ea77dc

Store result filename in RQ meta

24d1c87

Fix func signature

f503d49

Update permissions

69b1f9f

Fix typos

b909919

Do not disable request card after downloading file

347251f

Update tests

0c1e928

Fix typos

2688ae4

Revert back checking dataset import status by old API

5f52083

Add redis migration

af4ad59

Marishka17 requested a review from bsekachev as a code owner February 27, 2025 15:07

Remove unused imports

9bfe8b6

SpecLad reviewed Mar 6, 2025

View reviewed changes

cvat/apps/engine/views.py Outdated Show resolved Hide resolved

Marishka17 added 4 commits March 7, 2025 11:34

Update changelog

ea498ae

Apply comments

d1ee24b

Fix typo and pylint issue

c42385b

Fix server schema

3946e6f

SpecLad reviewed Mar 11, 2025

View reviewed changes

Marishka17 added 6 commits March 11, 2025 19:30

Small fixes in server schema

22c4783

Reduce code duplication in tests && small fixes

397939c

Merge branch 'develop' into mk/api_to_download_export_results

d14d047

Fix unit tests

8222969

Cleanup code

d321c01

Fix redis migration

4dee705

Marishka17 force-pushed the mk/api_to_download_export_results branch from 9418de4 to 4dee705 Compare March 12, 2025 10:49

Marishka17 changed the title ~~API to download export results~~ [do not merge] API to download export results Mar 12, 2025

Marishka17 changed the title ~~[do not merge] API to download export results~~ API to download export results Mar 12, 2025

Marishka17 force-pushed the mk/api_to_download_export_results branch 2 times, most recently from 4ef5485 to 5d01bc9 Compare March 12, 2025 11:34

Refactor redis migration a bit

4b01053

Marishka17 force-pushed the mk/api_to_download_export_results branch from 5d01bc9 to 4b01053 Compare March 12, 2025 11:35

Marishka17 added 5 commits March 12, 2025 13:39

Remove outdated comment

13507c2

Remove <> from response

1ad1f80

Update description of deprecated query params

54558fc

Do not wait requests to be finished in tests

570b60f

Merge branch 'develop' into mk/api_to_download_export_results

10cd428

SpecLad approved these changes Mar 13, 2025

View reviewed changes

SpecLad merged commit 950f9b6 into develop Mar 13, 2025
34 checks passed

SpecLad deleted the mk/api_to_download_export_results branch March 13, 2025 13:50

cvat-bot bot mentioned this pull request Mar 24, 2025

Release v2.32.0 #9248

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

API to download export results #9075

API to download export results #9075

Marishka17 commented Feb 7, 2025 •

edited

Loading

Marishka17 commented Feb 27, 2025

Marishka17 commented Mar 7, 2025

SpecLad Mar 11, 2025

Marishka17 Mar 11, 2025

SpecLad Mar 13, 2025

SpecLad Mar 11, 2025

Marishka17 Mar 11, 2025

SpecLad Mar 13, 2025

Marishka17 Mar 13, 2025

SpecLad Mar 13, 2025

sonarqubecloud bot commented Mar 13, 2025

API to download export results #9075

API to download export results #9075

Conversation

Marishka17 commented Feb 7, 2025 • edited Loading

Motivation and context

How has this been tested?

Checklist

License

Marishka17 commented Feb 27, 2025

Marishka17 commented Mar 7, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Mar 13, 2025

Quality Gate passed

Marishka17 commented Feb 7, 2025 •

edited

Loading