Skip to content

Latest commit

 

History

History
42 lines (26 loc) · 1.08 KB

File metadata and controls

42 lines (26 loc) · 1.08 KB

Azure Dataset Processing Pipeline

architecture

Setup and deployment

Make sure nix package manager is available on host os and run the following command to setting up the re-producible environment:

nix-shell

it will setup local environments with Azure CLI, Python and requirements for jobs (scripts) to be tested on local.

  • add required python modules to requirements.txt for scripts in the scripts directory.

Deployment

Steps:

  • Deploy all resources to Azure
  • Generate Connection string for blob account
  • Copy Scripts to the scripts blob
  • Run Pipeline

Debug

  • while working on the project, I was in need to monitor steps and deubg/make sure everything is correctly set up. for further information, check commands in Makefile with debug prefix.

Issues

Documentation