Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Digital typhoon dataset #1748

Merged
merged 61 commits into from
Aug 29, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
61 commits
Select commit Hold shift + click to select a range
c82a5e7
analysis task dataset
nilsleh Nov 13, 2023
1da1fa1
merge main
nilsleh Nov 30, 2023
c31f73a
implement sequence sampling
nilsleh Nov 30, 2023
a4bde5e
add outline datamodule
nilsleh Nov 30, 2023
e2a37a5
add datamodule with two way splitting capabilities
nilsleh Dec 1, 2023
a2881af
add plotting function
nilsleh Dec 1, 2023
49254ad
download and verify
nilsleh Dec 1, 2023
bcaaed9
add unit tests but they fail
nilsleh Dec 1, 2023
6272191
fix tests
nilsleh Dec 2, 2023
4477cf6
fix style
nilsleh Dec 2, 2023
0028a22
trainer testing yaml
nilsleh Dec 2, 2023
a939656
test split logic
nilsleh Dec 2, 2023
ba94b79
fix tests
nilsleh Dec 2, 2023
6963138
fix tests2
nilsleh Dec 2, 2023
aeec4dd
found bug
nilsleh Dec 2, 2023
171fed8
merge main
nilsleh Dec 2, 2023
82c5bed
try to fix mypy
nilsleh Dec 7, 2023
407d50f
h5py error docs
nilsleh Dec 7, 2023
48cb869
fix docs
nilsleh Dec 7, 2023
f6959a4
merge main
nilsleh Dec 11, 2023
44146eb
fix one mypy error
nilsleh Dec 13, 2023
4b3f46d
mypy on test file
nilsleh Dec 13, 2023
1f6f359
fix coverage
nilsleh Dec 13, 2023
ceaa7e6
fix tests for trainers
nilsleh Dec 13, 2023
0870131
fix mypy
nilsleh Dec 13, 2023
072c8fc
merge main
nilsleh Dec 13, 2023
2b40f5e
try typed dict
nilsleh Dec 13, 2023
4cbdb17
try to fix docs
nilsleh Dec 21, 2023
3465139
fix pytest
nilsleh Dec 21, 2023
d3576d7
linters
nilsleh Dec 21, 2023
23f6433
suggested changes and normalization procedure
nilsleh Feb 8, 2024
73138ff
merge
nilsleh Feb 8, 2024
81664fb
merge main
nilsleh Feb 8, 2024
71907af
regression target normalization
nilsleh Feb 8, 2024
204fb0c
update dataset splitting
nilsleh Feb 8, 2024
dace184
fix test
nilsleh Feb 8, 2024
90eb006
Merge branch 'main' into digital_typhoon
nilsleh Aug 20, 2024
c4d35be
quotes
nilsleh Aug 20, 2024
309a364
strings
nilsleh Aug 20, 2024
007c5c3
ruff
nilsleh Aug 20, 2024
dc8686d
quotes
nilsleh Aug 20, 2024
77e9fe3
ruff format on all
nilsleh Aug 20, 2024
9a91c4e
docs
nilsleh Aug 20, 2024
82af758
lazy import
nilsleh Aug 20, 2024
35668fb
h5py
nilsleh Aug 20, 2024
c6850cf
h5py datamodule
nilsleh Aug 20, 2024
b9e0db0
typo
nilsleh Aug 20, 2024
5fb6074
tests
nilsleh Aug 20, 2024
a5afd95
review
nilsleh Aug 21, 2024
bb35ecf
pass tests
nilsleh Aug 21, 2024
036f526
fix tests
nilsleh Aug 27, 2024
751c475
list -> tuple
adamjstewart Aug 27, 2024
9e2ca7a
mypy fix
adamjstewart Aug 27, 2024
0013418
rename
nilsleh Aug 28, 2024
d4c16a9
Merge branch 'main' into digital_typhoon
nilsleh Aug 28, 2024
836681e
tests
nilsleh Aug 28, 2024
87ebb44
Remove Analysis
adamjstewart Aug 28, 2024
2b40528
min pandas 2.2.0
nilsleh Aug 28, 2024
b555e09
pull
nilsleh Aug 28, 2024
37a74a8
resolve tests
nilsleh Aug 29, 2024
5512500
Merge branch 'main' into digital_typhoon
nilsleh Aug 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
try to fix mypy
  • Loading branch information
nilsleh committed Dec 7, 2023
commit 82c5bed0c8a8aefd5e9d068a368d1f0ad1a7acb0
6 changes: 2 additions & 4 deletions torchgeo/datamodules/digital_typhoon.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,6 @@

from typing import Any, Union

from torch.utils.data import Dataset

from ..datasets import DigitalTyphoonAnalysis
from .geo import NonGeoDataModule
from .utils import group_shuffle_split
Expand Down Expand Up @@ -43,8 +41,8 @@ def __init__(
self.split_by = split_by

def split_dataset(
self, dataset: Dataset
) -> tuple[list[dict[str, Union[str, list]]]]:
self, dataset: DigitalTyphoonAnalysis
) -> tuple[list[int, dict[str, Union[str, list[int]]]]]:
"""Split dataset into two parts.

Args:
Expand Down
9 changes: 9 additions & 0 deletions torchgeo/datasets/digital_typhoon.py
Original file line number Diff line number Diff line change
Expand Up @@ -279,6 +279,15 @@ def _load_image(self, sample_df: pd.DataFrame) -> Tensor:
"""

def load_image_tensor(id: str, filepath: str) -> Tensor:
"""Load a single image tensor from a h5 file.

Args:
id: typhoon id
filepath: path to the h5 file

Returns:
image tensor
"""
full_path = os.path.join(self.root, self.data_root, "image", id, filepath)
with h5py.File(full_path, "r") as h5f:
# tensor with added channel dimension
Expand Down