Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Programmatic download of valid ZIP files returns HTML responses for certain producer URLs #524

Merged
merged 3 commits into from
Jul 3, 2024

Conversation

cka-y
Copy link
Contributor

@cka-y cka-y commented Jul 3, 2024

Summary:
Added User-Agent to the header of the requests, resolving the issue for all feeds specified in #502. To achieve this, I mimicked the equivalent curl request from the browser network tab and systematically filtered out each header to identify that only the User-Agent was necessary.

Testing Tips:

  • Run datasets-batch-deployer-dev from the branch feat/502 to deploy the current behavior.
  • Adjust the FEEDS_LIMIT variable in the batch-datasets-dev cloud function to cover more feeds.
  • Filter logs to validate that feeds mdb-232, mdb-919, and mdb-683 have been properly processed. If these feeds are not included in the processing, you may need to further increase the FEEDS_LIMIT variable.

Example for mdb-683:
image-fotor-20240703131338

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@davidgamez
Copy link
Member

Some source systems prefer that we send a customized user agent to identify us as downloaders (see MobilityData/gtfs-validator#1562). I'm okay with this solution for now; let's just keep in mind that this issue can appear again, and we might need a less simple solution.

@cka-y cka-y merged commit 0c85a41 into main Jul 3, 2024
2 checks passed
@cka-y cka-y deleted the feat/502 branch July 3, 2024 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Programmatic download of valid ZIP files returns HTML responses for certain producer URLs
2 participants