HTTP referer ignored with XYZ Tiles connection

[MatzFan]

What is the bug or the crash?

I have an XYZ Tiles connection to an ArcGIS server which requires a referer header. Despite setting the referer in the relevant field it is ignored in HTTP requests made to the server. Consequently the tile service does not work as 403 errors result from all requests.

Note I have just upgraded from QGIS v3.22 and the referer header was respected in that version.

Steps to reproduce the issue

Set up an new XYZ Tiles connection as follows:-

• URL: https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Basemaps/Jersey_Basemap_Colour_wgs84/MapServer/tile/{z}/{y}/{x}
• Min Zoom level: 0
• Max Zoom level: 23
• Referer: https://www.gov.je/

Open the Debugging panel, select one of the failed HTTP requests and copy as cURL. Observer that the Accept and User-Agent headers are set, but not Referer. E.g:
curl 'https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Basemaps/Jersey_Basemap_Colour_wgs84/MapServer/tile/12/1404/2027' -H 'Accept: */*' -H 'User-Agent: QGIS/33801/LMDE 6 (faye)' --compressed

Versions

<style type="text/css"> p, li { white-space: pre-wrap; } </style>

QGIS version	3.38.1-Grenoble	QGIS code revision	3d4177a
Qt version	5.15.8
Python version	3.11.2
GDAL/OGR version	3.6.2
PROJ version	9.1.1
EPSG Registry database version	v10.076 (2022-08-31)
GEOS version	3.11.1-CAPI-1.17.1
SQLite version	3.40.1
PostgreSQL client version	15.7 (Debian 15.7-0+deb12u1)
SpatiaLite version	5.0.1
QWT version	6.1.4
QScintilla2 version	2.13.3
OS version	LMDE 6 (faye)
Active Python plugins
grassprovider	2.12.99
db_manager	0.1.20
processing	2.12.99
MetaSearch	0.3.6

QGIS version 3.38.1-Grenoble QGIS code revision [3d4177a](https://github.com/qgis/QGIS/commit/3d4177afc6b) Qt version 5.15.8 Python version 3.11.2 GDAL/OGR version 3.6.2 PROJ version 9.1.1 EPSG Registry database version v10.076 (2022-08-31) GEOS version 3.11.1-CAPI-1.17.1 SQLite version 3.40.1 PostgreSQL client version 15.7 (Debian 15.7-0+deb12u1) SpatiaLite version 5.0.1 QWT version 6.1.4 QScintilla2 version 2.13.3 OS version LMDE 6 (faye)

Active Python plugins
grassprovider
2.12.99
db_manager
0.1.20
processing
2.12.99
MetaSearch
0.3.6

Supported QGIS version

• I'm running a supported QGIS version according to the roadmap.

New profile

• I tried with a new QGIS profile

Additional context

No response

[agiudiceandrea]

@MatzFan, thanks for reporting. I cannot replicate the described behavior using QGIS 3.38.1 (OSGeo4W) on Windows 10:

{
  "Bytes Received": 54,
  "Bytes Total": 54,
  "Cache (control)": "Load from cache if available, otherwise load from network",
  "Cache (save)": "Can store result in cache",
  "Headers": {
    "Accept": "*/*",
    "User-Agent": "Mozilla/5.0 QGIS/33801/Windows 10 Version 1903",
    "referer": "https://www.gov.je/"
  },
  "ID": "src\\providers\\wms\\qgswmsprovider.cpp:4678 (QgsWmsTiledImageDownloadHandler::QgsWmsTiledImageDownloadHandler)",
  "Initiator": "QgsWmsTiledImageDownloadHandler",
  "Operation": "GET",
  "Query": {
    "https://maps.gov.je/arcgis/rest/services/Basemaps/Jersey_Basemap_Colour_wgs84/MapServer/tile/5/12/17": null
  },
  "Replies": 2,
  "Reply": {
    "Cache (result)": "Read from network",
    "Error": "Error transferring https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Basemaps/Jersey_Basemap_Colour_wgs84/MapServer/tile/5/12/17 - server replied: ",
    "Error Code": "299",
    "Headers": {
      "Cache-Control": "private",
      "Content-Length": "54",
      "Content-Type": "text/html",
      "Date": "Sat, 27 Jul 2024 11:01:54 GMT",
      "Server": "Microsoft-IIS/10.0, Microsoft-IIS/10.0,",
      "Set-Cookie": "ARRAffinity=261d951d69f0428930fe62a6cf964d040d0ec192dcfa9759226384e5a10b87f8;Path=/;HttpOnly;Secure;Domain=gojdippmaps.azurewebsites.net\nARRAffinitySameSite=261d951d69f0428930fe62a6cf964d040d0ec192dcfa9759226384e5a10b87f8;Path=/;HttpOnly;SameSite=None;Secure;Domain=gojdippmaps.azurewebsites.net",
      "Vary": "Origin",
      "X-AspNet-Version": "4.0.30319",
      "X-Content-Type-Options": "nosniff",
      "X-Powered-By": "ASP.NET, ASP.NET",
      "X-XSS-Protection": "1; mode=block"
    },
    "Status": "498"
  },
  "Thread": "0x00000215d8038010",
  "Total time (ms)": 748,
  "URL": "https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Basemaps/Jersey_Basemap_Colour_wgs84/MapServer/tile/5/12/17"
}

curl 'https://gojdippmaps.azurewebsites.net/proxy.ashx?https://maps.gov.je/arcgis/rest/services/Basemaps/Jersey_Basemap_Colour_wgs84/MapServer/tile/5/12/17' -H 'referer: https://www.gov.je/' -H 'Accept: */*' -H 'User-Agent: Mozilla/5.0 QGIS/33801/Windows 10 Version 1903'  --compressed

[MatzFan]

Thanks for your quick response, perhaps this is Linux-specific. Below is a screenshot to confirm my settings and the result.

[MatzFan]

And here is the same project, same system, settings unchanged in v3.22.16, complete with referer:

[MatzFan]

No change with new profile, but I fixed it by removing the layer from my project and then adding it back. Referer header now present again. Perhaps the issue was caused by me upgrading from 3.22 to 3.38. I'll leave this open in case you want to investigate whether this is an upgrade problem, but otherwise I consider the issue resolved. Thanks for your help.

[agiudiceandrea]

The issue occurs when a project, created by QGIS 3.22 and containing an XYZ layer for which the "referer" HTTP header was set, is opened using QGIS >= 3.26.0 (QGIS 3.34.9 and QGIS 3.38.1 included): in such case the referer URL is not sent in the HTTP request to the XYZ server.

Thus QGIS >= 3.26 cannot correctly read the referer URL set for an XYZ (maybe other ones?) layer contained in a QGIS 3.22 project.

It seems to me the issue is due to the fact that in QGIS 3.22 the referer URL is added to the layer source as referer=referer_URL, while it looks like QGIS >= 3.26 only recognize the referer URL in the layer source if in the form http-header:referer=referer_URL.

@benoitdm-oslandia, may this be due to #48578?

[benoitdm-oslandia]

Hi, I will have a look asap ;)

[benoitdm-oslandia]

@MatzFan can you provide both project files (in 3.22 and >=3.26)?

[agiudiceandrea]

@benoitdm-oslandia please find both the QGIS project created using QGIS 3.22.0 and the one created using QGIS 3.34.9 in the attached zip archive xyzreferer.zip.

As you can see inspecting them, or even just looking at the Layer Properties / Information panel, the layer created using QGIS 3.22 has the referer stored in the layer's source as referer=https://www.gov.je/ while the layer created using QGIS 3.34 has the referer stored in the layer's source as http-header:referer=https://www.gov.je/&referer=https://www.gov.je/.

QGIS 3.22 recognises the referer opening either the project created using QGIS 3.22 or the project created using QGIS 3.34.
QGIS 3.34 only recognises the referer opening the project created using QGIS 3.34, while it doesn't recognise the referer opening the project created using QGIS 3.22.

It looks like that while QGIS 3.34 stores the referer in the layer source both as http-header:referer= and referer= (probably for forward compatibility), it only recognises the referer stored as http-header:referer= and not as referer= being not backward compatible...

[MatzFan]

@benoitdm-oslandia do you still need project files from me?

[benoitdm-oslandia]

@MatzFan @agiudiceandrea I pushed a fix and if the CI works properly you could have a windows to test with!

Regards.