Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add marine infrastructure dataset and model config #49

Merged
merged 69 commits into from
Feb 24, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
69 commits
Select commit Hold shift + click to select a range
6f7ce55
wind turbine: compare new split vs old split performance
favyen2 Sep 26, 2024
c2ec853
try with 384x384 patches plus freeze the model for the first couple e…
favyen2 Sep 27, 2024
90b26fe
fix freezing code
favyen2 Sep 27, 2024
726ad8e
remove unused old config for wind turbine training
favyen2 Sep 30, 2024
9b0d4bb
add webmercator version of the wind turbine dataset
favyen2 Sep 30, 2024
91dd7f9
Add script to add the bounds metadata for layers using SingleImageRas…
favyen2 Sep 30, 2024
f03249c
update name for the 384x384 experiment
favyen2 Sep 30, 2024
036bf0b
Fix wind turbine webmercator training (labels were not being populated)
favyen2 Sep 30, 2024
d6f8b02
Use six Sentinel-2 images from diverse months of the year
uakfdotb Oct 16, 2024
8cc663a
Add marine infrastructure dataset config and model config.
favyen2 Oct 18, 2024
2b29949
add files needed by the model config for training
favyen2 Oct 18, 2024
82a9149
maybe close to same performance
favyen2 Oct 22, 2024
a5953a6
Merge remote-tracking branch 'origin/master' into favyen/turbine-2024…
favyen2 Oct 22, 2024
24e6093
json formatting
favyen2 Oct 22, 2024
cb04f2d
fix missing sections in config.json
favyen2 Oct 22, 2024
6a42948
Merge remote-tracking branch 'origin/master' into favyen/marine-infra
favyen2 Oct 23, 2024
91274c1
Add job launcher and prediction pipeline for Satlas applications.
favyen2 Oct 23, 2024
5249d27
Merge remote-tracking branch 'origin/master' into favyen/marine-infra
favyen2 Nov 6, 2024
ac8eb32
marine infra updates
favyen2 Nov 6, 2024
4fb0a69
gcp rtree index not working after august 2024 ...
favyen2 Nov 11, 2024
01deb1f
Merge remote-tracking branch 'origin/master' into favyen/marine-infra
favyen2 Dec 5, 2024
7fccfb3
latest changes
favyen2 Dec 5, 2024
f6bbc78
sync
favyen2 Dec 10, 2024
07565da
Merge remote-tracking branch 'origin/master' into favyen/marine-infra
favyen2 Dec 12, 2024
7085a2f
sync
favyen2 Dec 12, 2024
480473a
Merge branch 'favyen/turbine-20240926' into favyen/marine-infra
favyen2 Dec 12, 2024
cf57d21
sync
favyen2 Dec 19, 2024
c193308
remove unneeded marine infra model configs
favyen2 Dec 19, 2024
cba9484
clarify that convert_satlas_webmercator_to_rslearn is intended to be …
favyen2 Dec 19, 2024
9f94514
add readme for wind turbine configs
favyen2 Dec 19, 2024
0f1a7db
add readme for worker system
favyen2 Dec 19, 2024
31421ad
update readme
favyen2 Dec 19, 2024
cd07cfe
add azure configs
favyen2 Jan 7, 2025
f916d5c
upgrade solar farm ingestion
favyen2 Jan 9, 2025
463fa94
remove debug configs
favyen2 Jan 9, 2025
976def9
add documentation and test (wip)
favyen2 Jan 9, 2025
0bd47f5
fix test
favyen2 Jan 9, 2025
f8ee3ec
move convert_satlas_webmercator_to_rslean to one_off_porjects
favyen2 Jan 9, 2025
693f9a0
update readme
favyen2 Jan 9, 2025
522c210
Merge branch 'master' of github.com:allenai/rslearn_projects into fav…
favyen2 Jan 9, 2025
4d200c8
fix
favyen2 Jan 13, 2025
31576ab
fix
favyen2 Jan 14, 2025
298de6a
only start satlas jobs that weren't already completed
favyen2 Jan 17, 2025
3c9083c
enable satlas prediction pipeline to run on jupiter (using /data disk)
uakfdotb Jan 31, 2025
1efd544
add solar farm config
uakfdotb Jan 31, 2025
8be2dd9
Merge branch 'master' into favyen/marine-infra
favyen2 Feb 5, 2025
9a3d757
add documentation about viterbi smoothing step
favyen2 Feb 6, 2025
0d922d6
Merge branch 'favyen/marine-infra' of github.com:allenai/rslearn_proj…
favyen2 Feb 6, 2025
ec95814
add documentation
favyen2 Feb 6, 2025
ead3e22
remove unused launch_worker
favyen2 Feb 6, 2025
a71f3f6
add test for bkt
favyen2 Feb 7, 2025
1eb9d78
add test for rslp/common/worker.py
favyen2 Feb 7, 2025
c4af2a5
add doc string
favyen2 Feb 7, 2025
df45d4c
add tests
favyen2 Feb 7, 2025
731cd0c
add test for apply_nms
favyen2 Feb 7, 2025
98d0925
add merge_points test
favyen2 Feb 7, 2025
d7810bf
remove unused nms stuff
favyen2 Feb 7, 2025
fed1b51
fix test
favyen2 Feb 7, 2025
eeb63e4
add tests for smoothing
uakfdotb Feb 8, 2025
df3341d
refactor prediction pipeline
favyen2 Feb 10, 2025
9b86ca4
fix tests
favyen2 Feb 11, 2025
cbf1f1e
working tests
uakfdotb Feb 12, 2025
a4d4ebf
Merge remote-tracking branch 'origin/master' into favyen/marine-infra
favyen2 Feb 13, 2025
a56c3f5
fix broken Dockerfile (missing wget)
favyen2 Feb 18, 2025
b23985b
fix import for updated planetary computer data source
favyen2 Feb 18, 2025
436bc00
fix tests
favyen2 Feb 20, 2025
8c713ca
fix tests x2
favyen2 Feb 20, 2025
f73761d
don't run bkt test in ci since bigtable instance is expensive to keep…
favyen2 Feb 24, 2025
4a251f8
Merge remote-tracking branch 'origin/master' into favyen/marine-infra
favyen2 Feb 24, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
fix
  • Loading branch information
favyen2 committed Jan 14, 2025
commit 31576ab3f3ecbed5aaec0246e14bfd28f4ee6f4f
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@
geometry["coordinates"] = (
np.array(geometry["coordinates"]) - [window.bounds[0], window.bounds[1]]
).tolist()
shapes.append((geometry, 255))
shapes.append((geometry, 1))
if shapes:
mask = rasterio.features.rasterize(
shapes,
Expand Down
2 changes: 1 addition & 1 deletion rslp/common/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,4 @@ Then you can launch the worker. To test on one machine:

And to launch 100 workers on Beaker:

python -m rslp.main common launch skylight-proto-1 rslp-job-queue-YOURNAME-sub 100 --gpus 1 --shared_memory 256GiB
python -m rslp.main common launch BEAKER_IMAGE_NAME skylight-proto-1 rslp-job-queue-YOURNAME-sub 100 --gpus 1 --shared_memory 256GiB
7 changes: 5 additions & 2 deletions rslp/common/worker.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,8 @@
)
from google.cloud import pubsub_v1, storage

from rslp.launch_beaker import BUDGET, DEFAULT_WORKSPACE, IMAGE_NAME, get_base_env_vars
from rslp.launch_beaker import BUDGET, DEFAULT_WORKSPACE
from rslp.launcher_lib import get_base_env_vars
from rslp.log_utils import get_logger
from rslp.main import run_workflow

Expand Down Expand Up @@ -196,6 +197,7 @@ def launch_worker(project_id: str, subscription_id: str) -> None:


def launch_workers(
image_name: str,
project_id: str,
subscription_id: str,
num_workers: int,
Expand All @@ -207,6 +209,7 @@ def launch_workers(
"""Start workers for the prediction jobs.

Args:
image_name: the Beaker image name to use for the jobs.
project_id: the Google Cloud project ID.
subscription_id: the Pub/Sub subscription ID.
num_workers: number of workers to launch
Expand All @@ -224,7 +227,7 @@ def launch_workers(
spec = ExperimentSpec.new(
budget=BUDGET,
description="worker",
beaker_image=IMAGE_NAME,
beaker_image=image_name,
priority=priority,
command=["python", "-m", "rslp.main"],
arguments=[
Expand Down
Loading