feat: Programmatic download of valid ZIP files returns HTML responses for certain producer URLs #524
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary:
Added
User-Agent
to the header of the requests, resolving the issue for all feeds specified in #502. To achieve this, I mimicked the equivalent curl request from the browser network tab and systematically filtered out each header to identify that only theUser-Agent
was necessary.Testing Tips:
datasets-batch-deployer-dev
from the branchfeat/502
to deploy the current behavior.FEEDS_LIMIT
variable in thebatch-datasets-dev
cloud function to cover more feeds.mdb-232
,mdb-919
, andmdb-683
have been properly processed. If these feeds are not included in the processing, you may need to further increase theFEEDS_LIMIT
variable.Example for
![image-fotor-20240703131338](https://private-user-images.githubusercontent.com/60586858/345545051-78d5ce85-6b01-4718-b115-2289477a58eb.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3Mzk1MDg0MDksIm5iZiI6MTczOTUwODEwOSwicGF0aCI6Ii82MDU4Njg1OC8zNDU1NDUwNTEtNzhkNWNlODUtNmIwMS00NzE4LWIxMTUtMjI4OTQ3N2E1OGViLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTAyMTQlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwMjE0VDA0NDE0OVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI4ZmQ5Yjg4M2Y3ZTc4ZWMxYTY4ZmU3M2YxZGMxZmM4MmNiYTRhMzc3MjQ3ZDY0NWUzZjIyNGY1YmFlMDI3MWYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.2aXaH09M5uwCk-D5MP57FdE1PehzN16zGPlK-bX_byI)
mdb-683
:Please make sure these boxes are checked before submitting your pull request - thanks!
./scripts/api-tests.sh
to make sure you didn't break anything