Skip to content
This repository has been archived by the owner on Dec 7, 2023. It is now read-only.

Document how any admin can manually browse dataflow logs #146

Closed
cisaacstern opened this issue Sep 28, 2022 · 1 comment
Closed

Document how any admin can manually browse dataflow logs #146

cisaacstern opened this issue Sep 28, 2022 · 1 comment

Comments

@cisaacstern
Copy link
Member

Lifting this into its own issue, because it's really important.

Any admin (regardless of whether they are a member of the pangeo-forge-4967 GCP project) can query logs for Dataflow jobs using the secrets/dataflow-job-submission.json service account key contained in this repo.

This is an essential tool for supporting recipe contributors, e.g. on staged-recipes, when jobs fail.

#145 (comment) describes how to do this.

Knowing the job_id is key here, so fixing #145 will make this easier. Once #145 is fixed, the process described in #145 (comment) should be added to docs/README.md

cc @andersy005

@cisaacstern
Copy link
Member Author

cisaacstern commented Sep 29, 2022

In addition to gcloud logging CLI, @yuvipanda's logs.py linked in #145 (comment) and explained further in #145 (comment) should also be described in this doc.

Note that I'm not certain that the secrets/dataflow-job-submission.json service account creds will work for all operations supported in this logs.py, but this service account's roles could be adjusted if necessary, to ensure that Pangeo Forge Orchestrator admins can use these creds for calling Yuvi's logs.py, even if they are not members of the pangeo-forge-4967 GCP project.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant