How To?

prerequisites

• docker
• docker-compose

Commands

docker-compose build 
docker-compose up -d

URLS

Airflow url - http://0.0.0.0:19000/home Spark-cluster - http://0.0.0.0:9010/

Screen shots

Airflow UI

Spark UI

Variable list

oracle password you can get from dbt_integration/.dbt/profiles.yml

sender_email and sender_password is configured from google smtp settings

Connection list

dbt-container ssh connection

spark-master ssh connection

oracle db ssh connection

Spark connection

Postman Connection