This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Imagery is always reprojected, even if it is already correctly projected and tile aligned #431
Comments
Plantain
changed the title
|
@Plantain I'm not sure to really follow. TiTiler uses rio-tiler which use rasterio WarpedVRT https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/reader.py#L83 which is why you see This is tested in https://github.com/cogeotiff/rio-cogeo/blob/master/tests/test_web.py#L135-L212. You can also debug this using https://github.com/developmentseed/tilebench that's is said, if there is an issue this should be in rio-tiler |
From my experience there may be small rounding errors that cause GDAL to slightly misalign each block in the image, requiring web-optimized readers to send multiple requests to fetch a single tile. This depends a lot on your input data and how much warping GDAL needs to perform to reproject to 3857. The fix I have used for this in the past is to "pre-align" your data by first creating a Warped VRT that is aligned to 3857 before creating your COG. This gives you a little bit more control over what GDAL is doing:
What immediately stands out to me is your spatial resolution - First ran into this writing https://github.com/geospatial-jeff/aiocogeo. Note that it has been a while since I've created COGs without pre-aligning the data first, so this may have been fixed. As @vincentsarago mentioned, I would highly recommend using https://github.com/developmentseed/tilebench to debug as it shows you alignment between COG blocks and XYZ tiles. |
|
Thanks both for the context and ideas - I did suspect this probably wasn't a titiler issue but I wasn't sure where to start below it. I'm not in control of that spatial resolution or the X/Y directions when using TILING_SCHEME=GoogleMapsCompatible - so I guess this is still an issue in GDAL (using 3.3.2, will try 3.4.1...) I have tried to debug this with tilebench and I'm not sure if I am interpreting the results correctly - it looks like in most rows a tile fetch is 2 GET's, but every third row inexplicably is 3 GET's. If everything was aligned correctly it would be 1 GET or 2 for a tile? tilebench viz http://static2.skysight.io/demo.tiff --config GDAL_INGESTED_BYTES_AT_OPEN=200000 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR |
|
@Plantain for your file https://github.com/OSGeo/gdal/blob/35c07b18316b4b6d238f6d60b82c31e25662ad27/gcore/tilematrixset.cpp#L79-L108 vs https://github.com/developmentseed/morecantile/blob/master/morecantile/data/WebMercatorQuad.json the differences in ☝️ is really small but Morecantile use the values provided by OGC 🤷♂️ can I ask which version of GDAL you used to create the COG? and also what was the command you used? also @geospatial-jeff raises a valid point about the resolution being different in x and y 🤷 |
|
I'm making them with satpy, which wraps GDAL - see
pytroll/satpy#1882
scn.save_dataset(composite, filename=outname, driver='COG',
tiling_scheme='GoogleMapsCompatible', compress="JPEG",
warp_resampling="lanczos", overview_resampling="lanczos",
overview_quality=90, quality=90, num_threads="32",
zoom_level_strategy="UPPER", fill_value=0)
GDAL 3.3.2, released 2021/09/01
I saw the same behaviour building it from the commandline too though.
…On Mon, Feb 14, 2022 at 8:27 PM Vincent Sarago ***@***.***> wrote:
@Plantain <https://github.com/Plantain> for your file
http://static2.skysight.io/demo.tiff only the raw resolution (level 0) is
aligned with the mercator grid.
As for the 3 GET, this is due to the fact that GDAL and rio-tiler
(morecantile) use a slightly different way to define the TileMatrixSet used
to get the tile bounding box.
https://github.com/OSGeo/gdal/blob/35c07b18316b4b6d238f6d60b82c31e25662ad27/gcore/tilematrixset.cpp#L79-L108
vs
https://github.com/developmentseed/morecantile/blob/master/morecantile/data/WebMercatorQuad.json
the differences in ☝️ is really small but Morecantile use the values
provided by OGC 🤷♂️
can I ask which version of GDAL you used to create the COG? and also what
was the command you used?
also @geospatial-jeff <https://github.com/geospatial-jeff> raises a valid
point about the resolution being different in x and y 🤷
$ tilebench profile http://static2.skysight.io/demo.tiff --tile 7-70-58 --config GDAL_INGESTED_BYTES_AT_OPEN=200000 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR | jq
{
"LIST": {
"count": 0
},
"HEAD": {
"count": 1
},
"GET": {
"count": 3,
"bytes": 262144,
"ranges": [
"0-229375",
"46661632-46678015",
"47595520-47611903"
]
},
"Timing": 0.41567397117614746
}
# use rio cogeo COG
$ rio cogeo create demo.tiff demo_cog.tif -w -p jpeg --blocksize 256 --co JPEG_QUALITY=90
$ tilebench profile http://0.0.0.0:8000/demo_cog.tif --tile 7-70-58 --config GDAL_INGESTED_BYTES_AT_OPEN=200000 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR | jq
{
"LIST": {
"count": 0
},
"HEAD": {
"count": 1
},
"GET": {
"count": 2,
"bytes": 245760,
"ranges": [
"0-229375",
"48955392-48971775"
]
},
"Timing": 0.13192009925842285
}
—
Reply to this email directly, view it on GitHub
<#431 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACODF527SQK7IVH2LQBO33U3DKKRANCNFSM5ODJ3VGA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
The original data file (or rather, a file out of satpy without any
projection) looks like http://static2.skysight.io/himawari.tiff , which
unfortunately rio isn't very happy with, even though titiler does handle it
directly - and serving that directly with titiler and not needing to do any
reprojection in advance would be my preferred choice, but the latency is
just too high, it takes >1s per tile, even with the imagery local. Maybe
related to davenquinn/mars-tiler#2
$ rio cogeo create himawari.tiff demo_cog.tif -w -p jpeg --blocksize 256
--co JPEG_QUALITY=90
/home/rasp/.local/lib/python3.8/site-packages/rio_cogeo/cogeo.py:161:
UserWarning: Nodata/Alpha band will be translated to an internal mask band.
warnings.warn(
Traceback (most recent call last):
File "/usr/local/bin/rio", line 8, in <module>
sys.exit(main_group())
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1128,
in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1053,
in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1659,
in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1659,
in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 1395,
in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.8/dist-packages/click/core.py", line 754, in
invoke
return __callback(*args, **kwargs)
File
"/home/rasp/.local/lib/python3.8/site-packages/rio_cogeo/scripts/cli.py",
line 250, in create
cog_translate(
File "/home/rasp/.local/lib/python3.8/site-packages/rio_cogeo/cogeo.py",
line 212, in cog_translate
params = utils.get_web_optimized_params(
File "/home/rasp/.local/lib/python3.8/site-packages/rio_cogeo/utils.py",
line 94, in get_web_optimized_params
transform_bounds(
File "/usr/local/lib/python3.8/dist-packages/rasterio/warp.py", line 161,
in transform_bounds
return _transform_bounds(
File "rasterio/_warp.pyx", line 1566, in rasterio._warp._transform_bounds
File "rasterio/_base.pyx", line 1500, in rasterio._base._transform
File "rasterio/_err.pyx", line 191, in rasterio._err.exc_wrap_int
rasterio._err.CPLE_AppDefinedError: Reprojection failed, err = 2050,
further errors will be suppressed on the transform object.
On Mon, Feb 14, 2022 at 9:24 PM Matthew Scutter ***@***.***>
wrote:
… I'm making them with satpy, which wraps GDAL - see
pytroll/satpy#1882
scn.save_dataset(composite, filename=outname, driver='COG',
tiling_scheme='GoogleMapsCompatible', compress="JPEG",
warp_resampling="lanczos", overview_resampling="lanczos",
overview_quality=90, quality=90, num_threads="32",
zoom_level_strategy="UPPER", fill_value=0)
GDAL 3.3.2, released 2021/09/01
I saw the same behaviour building it from the commandline too though.
On Mon, Feb 14, 2022 at 8:27 PM Vincent Sarago ***@***.***>
wrote:
> @Plantain <https://github.com/Plantain> for your file
> http://static2.skysight.io/demo.tiff only the raw resolution (level 0)
> is aligned with the mercator grid.
> As for the 3 GET, this is due to the fact that GDAL and rio-tiler
> (morecantile) use a slightly different way to define the TileMatrixSet used
> to get the tile bounding box.
>
>
> https://github.com/OSGeo/gdal/blob/35c07b18316b4b6d238f6d60b82c31e25662ad27/gcore/tilematrixset.cpp#L79-L108
> vs
> https://github.com/developmentseed/morecantile/blob/master/morecantile/data/WebMercatorQuad.json
>
> the differences in ☝️ is really small but Morecantile use the values
> provided by OGC 🤷♂️
>
> can I ask which version of GDAL you used to create the COG? and also what
> was the command you used?
>
> also @geospatial-jeff <https://github.com/geospatial-jeff> raises a
> valid point about the resolution being different in x and y 🤷
>
> $ tilebench profile http://static2.skysight.io/demo.tiff --tile 7-70-58 --config GDAL_INGESTED_BYTES_AT_OPEN=200000 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR | jq
>
> {
>
> "LIST": {
>
> "count": 0
>
> },
>
> "HEAD": {
>
> "count": 1
>
> },
>
> "GET": {
>
> "count": 3,
>
> "bytes": 262144,
>
> "ranges": [
>
> "0-229375",
>
> "46661632-46678015",
>
> "47595520-47611903"
>
> ]
>
> },
>
> "Timing": 0.41567397117614746
>
> }
>
>
>
> # use rio cogeo COG
>
> $ rio cogeo create demo.tiff demo_cog.tif -w -p jpeg --blocksize 256 --co JPEG_QUALITY=90
>
> $ tilebench profile http://0.0.0.0:8000/demo_cog.tif --tile 7-70-58 --config GDAL_INGESTED_BYTES_AT_OPEN=200000 --config GDAL_DISABLE_READDIR_ON_OPEN=EMPTY_DIR | jq
>
> {
>
> "LIST": {
>
> "count": 0
>
> },
>
> "HEAD": {
>
> "count": 1
>
> },
>
> "GET": {
>
> "count": 2,
>
> "bytes": 245760,
>
> "ranges": [
>
> "0-229375",
>
> "48955392-48971775"
>
> ]
>
> },
>
> "Timing": 0.13192009925842285
>
> }
>
>
> —
> Reply to this email directly, view it on GitHub
> <#431 (comment)>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACODF527SQK7IVH2LQBO33U3DKKRANCNFSM5ODJ3VGA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
|
I though this was fixed in rasterio 1.3a3 🤷 . but then there is another issue because the bounds rio-cogeo will get here https://github.com/cogeotiff/rio-cogeo/blob/master/rio_cogeo/utils.py#L93-L97 is wrong (I'll fix this) GEOS projection is really hard to handle 😭 |
|
I can provide a sample of all the most common GEOS projections for unit
testing if you'd like?
…On Tue, 15 Feb 2022, 12:07 am Vincent Sarago, ***@***.***> wrote:
rasterio._err.CPLE_AppDefinedError: Reprojection failed, err = 2050,
further errors will be suppressed on the transform object.
I though this was fixed in rasterio 1.3a3 🤷 . but then there is another
issue because the bounds rio-cogeo will get here
https://github.com/cogeotiff/rio-cogeo/blob/master/rio_cogeo/utils.py#L93-L97
is wrong (I'll fix this)
GEOS projection is really hard to handle 😭
—
Reply to this email directly, view it on GitHub
<#431 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AACODF6CBHNSQMTTBVWX4G3U3D37NANCNFSM5ODJ3VGA>
.
Triage notifications on the go with GitHub Mobile for iOS
<https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
or Android
<https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
This is in fact not true because we have a test in rio-tiler to make sure we aligned the bounds with the internal tile to avoid rounding issue https://github.com/cogeotiff/rio-tiler/blob/master/rio_tiler/utils.py#L265-L283 The more I look at this the more I think there is something going on with the Y resolution in GDAL. |
Yeah I remember we used to have problem with tile alignment when first building morecantile until we implemented this snapping logic.
This indicates to me that the issue is happening in the Y direction (not the X), and probably related to the slightly different Y resolution. Each row from the origin (top-left corner) will have a little bit of error because of the Y resolution, that error propagates to an extra tile read every third row. A little extra error in the Y direction would cause 3 GET requests instead of 2. 1 for the header, 1 for the tile in question, and 1 for the tile's southern neighbor. If there was any error in the X direction we would see 5 GET requests, as we'd also be fetching the lower right and right neighbor. And the only thing different between X/Y in this case is resolution 🤷 |
|
I believe the real reason it that you have not used the Try to reach a result like here and try again |
|
@jratike80 we see anormal GET request even for the highest resolution (were we make sure we are not requesting the overviews) |
|
Pity, it would have been so easy fix. But your example was obviously not hitting the full resolution |
|
@jratike80 can you explain why you believe this from the kernels? |
|
Sorry, just bad thinking I guess. But I would be interested in understanding what this means |
GDAL WebOptimized COGRio-cogeo Web-Optimized COG☝️ I don't think the kernels tell we are reading the overview. the As Even, said the error is maybe on the |
This issue was moved to a discussion.
You can continue the conversation there. Go to discussion →
Expected behaviour: If a COG is created with TILING_SCHEME=GoogleMapsCompatible, and the serving environment is EPSG:3857, because the COG's tiles are aligned and the projection already matches, titiler should be able to do everything it needs with two reads (metadata + tile) and zero reprojection. It should be very fast because it's only two reads and no reprojection.
Actual behaviour:
I suspected that some reprojection was happening because it appeared as though there was some loss in fidelity between source and the browser, and that it was just as slow (~1-2 seconds...) as if the source imagery was in a totally different projection.
Turning all the debugging up to the maximum, it looks like it is doing multiple reads and projection.
Problem:
Does GDAL not align the COG properly, or does titiler read more than it needs to?
Details on GoogleMapsCompatible:
https://gdal.org/drivers/raster/cog.html
TILING_SCHEME=CUSTOM/GoogleMapsCompatible/other: Default value: CUSTOM. If set to a value different than CUSTOM, the definition of the specified tiling scheme will be used to reproject the dataset to its CRS, select the resolution corresponding to the closest zoom level and align on tile boundaries at this resolution.
Demo file at: http://static2.skysight.io/demo.tiff
The text was updated successfully, but these errors were encountered: