Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable stream import for COCO data format #1091

Merged
merged 28 commits into from
Jul 12, 2023
Merged

Enable stream import for COCO data format #1091

merged 28 commits into from
Jul 12, 2023

Conversation

vinnamkim
Copy link
Contributor

@vinnamkim vinnamkim commented Jul 11, 2023 •
edited
Loading

Summary

  • ticket no. 114761
  • Implement COCOPageMapper
  • Enable the COCO extractor to generate DatasetItem using COCOPageMapper in the stream manner

How to test

I parametrized the existing tests for this change.

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added the description of my changes into CHANGELOG.​
  • I have updated the documentation accordingly

License

  • I submit my code changes under the same MIT License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2023 Intel Corporation
#
# SPDX-License-Identifier: MIT

vinnamkim added 21 commits July 4, 2023 15:36
@vinnamkim
Refactor dataset.py to seperate DatasetStorage
4ccb5c1 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Add StreamDatasetStorage and StreamDataset
1412e62 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Merge remote-tracking branch 'upstream/develop' into feature/stream-d…
7be138a 
…ataset

Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Update CHANGELOG.md
75cd172 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Fix typing error
a080382 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Enable stream import for VOC and YOLO
3a09d02 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Merge branch 'develop' into feature/stream-importer-voc-and-yolo
67cd067
@vinnamkim
Fix duplication
02603fc 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Update CHANGELOG.md
d0ff715 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Update specs.json
505f782 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Refactor first
2edea68 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Implement streamable DatumaroBase
56215ca 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Update tests
f82d070 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Merge branch 'releases/1.4.0' into feature/stream-importer-datum
798d7fe
@vinnamkim
Update CHANGELOG.md
34f1e56 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Revert unused change
758f1ca 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Update skip comment
3df5f19 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Fix Windows test errors
ee79511 
 - Specify encoding type (utf-8)

Signed-off-by: Vinnam Kim <[email protected]>
@vinnamkim
Rename and update broken is_stream propagation
8d50797 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Add COCOPageMapper
47c464b 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim
Update COCOImporter to introduce streaming
5803c78 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim vinnamkim added this to the 1.4.0 milestone Jul 11, 2023
@vinnamkim vinnamkim added the FEATURE New feature & functionality label Jul 11, 2023
@vinnamkim
Update CHANGELOG.md
c8767c1 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim vinnamkim marked this pull request as ready for review July 11, 2023 10:00
@vinnamkim vinnamkim requested review from a team as code owners July 11, 2023 10:00
@vinnamkim vinnamkim requested review from jihyeonyi and removed request for a team July 11, 2023 10:00
jihyeonyi
jihyeonyi reviewed Jul 12, 2023
View reviewed changes
tests/unit/data_formats/coco/test_roboflow.py Outdated
@@ -2,12 +2,15 @@
#
# SPDX-License-Identifier: MIT

from typing import Any, Dict, Optional, Type
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Type is imported but not used. Doesn't the commit-hook apply to the test code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The "tests" directory is excluded during the ruff check.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Anyway I'll revert it with fixing the errors on Windows

@vinnamkim
Rename
ab88cf9 
Signed-off-by: Kim, Vinnam <[email protected]>
wonjuleee
wonjuleee reviewed Jul 12, 2023
View reviewed changes
src/datumaro/plugins/data_formats/coco/page_mapper.py Outdated
end: int = -1

def reset(self):
self.flush = 0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comment: self.flush = False for reset

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 3a8af6d

wonjuleee
wonjuleee approved these changes Jul 12, 2023
View reviewed changes
Copy link
Contributor

@wonjuleee wonjuleee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@vinnamkim
Nit fix
3a8af6d 
Signed-off-by: Kim, Vinnam <[email protected]>
@vinnamkim vinnamkim merged commit 3c7364d into openvinotoolkit:releases/1.4.0 Jul 12, 2023
@vinnamkim vinnamkim deleted the feature/stream-importer-coco branch July 12, 2023 05:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jihyeonyi jihyeonyi jihyeonyi approved these changes

@wonjuleee wonjuleee wonjuleee approved these changes

Assignees
No one assigned
Labels
FEATURE New feature & functionality
Projects
None yet
Milestone
1.4.0
Development

Successfully merging this pull request may close these issues.
3 participants
@vinnamkim @jihyeonyi @wonjuleee