From 8948109683134671fadffda0c2f41cc3695e1711 Mon Sep 17 00:00:00 2001 From: Kenneth Kilty Date: Tue, 14 Jan 2025 19:00:39 -0500 Subject: [PATCH 1/3] added apache airflow blog --- .../2025-01-20-annouce-airflow-howto.md | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) create mode 100644 blog/_posts/2025-01-20-annouce-airflow-howto.md diff --git a/blog/_posts/2025-01-20-annouce-airflow-howto.md b/blog/_posts/2025-01-20-annouce-airflow-howto.md new file mode 100644 index 00000000..4529eee9 --- /dev/null +++ b/blog/_posts/2025-01-20-annouce-airflow-howto.md @@ -0,0 +1,27 @@ +--- +title: "Apache Airflow Guidance for AKS" +description: "Learn how to set up an AKS cluster, deploy Airflow, and explore the Airflow UI running on AKS." +date: 2025-01-20 +author: Kenneth Kilty +categories: general +--- + +## Apache Airflow on AKS + +We're pleased to share [new guidance](https://learn.microsoft.com/azure/aks/airflow-overview) on deploying open-source Apache Airflow on Azure Kubernetes Service (AKS). + +Apache Airflow is an open-source platform for orchestrating complex workflows and data pipelines. It allows users to define, schedule, and monitor workflows using Python. Airflow supports numerous integrations such as Azure Blob Storage or Azure Postgres SQL and can scale to handle large data volumes. The web-based Airflow UI provides a visual representation of your workflows, making it easier to track progress and troubleshoot issues. Airflow is widely used for ETL processes, data engineering, and managing machine learning pipelines. + +This [new AKS how-to guide](https://learn.microsoft.com/azure/aks/airflow-overview) will walk you through the entire process, from setting up your AKS cluster with Airflow secretes in Azure Key Vault and DAG logs in Azure Blog Storage installing Apache Airflow using Helm. Considerations for Airflow distributed architecture for production are included within the guide. + +You will also explore the Apache Airflow UI, where you can monitor and manage your workflows in Airflow running on AKS. Whether you're new to Airflow or looking to deploy your existing setup on AKS, this guide has something for everyone. + +## Apache Airflow on Astro, Powered by AKS + +For customers needing commercial support for Airflow on AKS, look to our partner [Astro](https://learn.microsoft.com/azure/partner-solutions/astronomer/overview) available today as an [Azure Native ISV Service](https://learn.microsoft.com/azure/partner-solutions/) powered on Azure by AKS! + +Astro by [Astronomer.io](https://www.astronomer.io/) is an industry-leading DataOps platform. Powered by Apache Airflow, Astro dramatically reduces costs, increases productivity, and reliably powers customers most critical data pipelines. Astronomer also drives all Apache Airflow releases and has contributed over 55% of Apache Airflow open source code. + +You can find Apache Airflow on Astro – An Azure Native ISV Service in the [Azure portal](https://ms.portal.azure.com/?Azure_Marketplace_Astronomer_assettypeoptions=%7B%22Astronomer%22%3A%7B%22options%22%3A%22%22%7D%7D#browse/Astronomer.Astro%2Forganizations) or get it on [Azure Marketplace](https://azuremarketplace.microsoft.com/marketplace/apps/astronomer1591719760654.astronomer?tab=Overview0) + +Go ahead and dive in and unlock the full potential of Apache Airflow on AKS! From 5ee03a4f697067aeb6ee0d84d9b061b3c402ee5e Mon Sep 17 00:00:00 2001 From: Kenneth Kilty Date: Tue, 14 Jan 2025 19:06:04 -0500 Subject: [PATCH 2/3] added reference to Astro learning material --- blog/_posts/2025-01-20-annouce-airflow-howto.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/blog/_posts/2025-01-20-annouce-airflow-howto.md b/blog/_posts/2025-01-20-annouce-airflow-howto.md index 4529eee9..e0415ab1 100644 --- a/blog/_posts/2025-01-20-annouce-airflow-howto.md +++ b/blog/_posts/2025-01-20-annouce-airflow-howto.md @@ -16,11 +16,11 @@ This [new AKS how-to guide](https://learn.microsoft.com/azure/aks/airflow-overvi You will also explore the Apache Airflow UI, where you can monitor and manage your workflows in Airflow running on AKS. Whether you're new to Airflow or looking to deploy your existing setup on AKS, this guide has something for everyone. -## Apache Airflow on Astro, Powered by AKS +## Apache Airflow on Astro, Powered on Azure by AKS For customers needing commercial support for Airflow on AKS, look to our partner [Astro](https://learn.microsoft.com/azure/partner-solutions/astronomer/overview) available today as an [Azure Native ISV Service](https://learn.microsoft.com/azure/partner-solutions/) powered on Azure by AKS! -Astro by [Astronomer.io](https://www.astronomer.io/) is an industry-leading DataOps platform. Powered by Apache Airflow, Astro dramatically reduces costs, increases productivity, and reliably powers customers most critical data pipelines. Astronomer also drives all Apache Airflow releases and has contributed over 55% of Apache Airflow open source code. +Astro by [Astronomer.io](https://www.astronomer.io/) is an industry-leading DataOps platform. Powered by Apache Airflow, Astro dramatically reduces costs, increases productivity, and reliably powers customers most critical data pipelines. Astronomer also drives all Apache Airflow releases and has contributed over 55% of Apache Airflow open source code. Astronomer also maintains learning material for Airflow connectors to Azure from Astro such as [Azure Blob Storage](https://www.astronomer.io/docs/learn/connections/azure-blob-storage/). You can find Apache Airflow on Astro – An Azure Native ISV Service in the [Azure portal](https://ms.portal.azure.com/?Azure_Marketplace_Astronomer_assettypeoptions=%7B%22Astronomer%22%3A%7B%22options%22%3A%22%22%7D%7D#browse/Astronomer.Astro%2Forganizations) or get it on [Azure Marketplace](https://azuremarketplace.microsoft.com/marketplace/apps/astronomer1591719760654.astronomer?tab=Overview0) From 55f86915fa50d61e9421d0ca069e1c1a6e4ecaa8 Mon Sep 17 00:00:00 2001 From: Kenneth Kilty Date: Tue, 21 Jan 2025 12:14:31 -0500 Subject: [PATCH 3/3] Update blog/_posts/2025-01-20-annouce-airflow-howto.md Co-authored-by: Paul Yu --- blog/_posts/2025-01-20-annouce-airflow-howto.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/blog/_posts/2025-01-20-annouce-airflow-howto.md b/blog/_posts/2025-01-20-annouce-airflow-howto.md index e0415ab1..339cf34c 100644 --- a/blog/_posts/2025-01-20-annouce-airflow-howto.md +++ b/blog/_posts/2025-01-20-annouce-airflow-howto.md @@ -22,6 +22,6 @@ For customers needing commercial support for Airflow on AKS, look to our partner Astro by [Astronomer.io](https://www.astronomer.io/) is an industry-leading DataOps platform. Powered by Apache Airflow, Astro dramatically reduces costs, increases productivity, and reliably powers customers most critical data pipelines. Astronomer also drives all Apache Airflow releases and has contributed over 55% of Apache Airflow open source code. Astronomer also maintains learning material for Airflow connectors to Azure from Astro such as [Azure Blob Storage](https://www.astronomer.io/docs/learn/connections/azure-blob-storage/). -You can find Apache Airflow on Astro – An Azure Native ISV Service in the [Azure portal](https://ms.portal.azure.com/?Azure_Marketplace_Astronomer_assettypeoptions=%7B%22Astronomer%22%3A%7B%22options%22%3A%22%22%7D%7D#browse/Astronomer.Astro%2Forganizations) or get it on [Azure Marketplace](https://azuremarketplace.microsoft.com/marketplace/apps/astronomer1591719760654.astronomer?tab=Overview0) +You can find Apache Airflow on Astro – An Azure Native ISV Service in the [Azure portal](https://portal.azure.com/#view/Microsoft_Azure_Marketplace/GalleryItemDetailsBladeNopdl/id/astronomer1591719760654.astro) or get it on [Azure Marketplace](https://azuremarketplace.microsoft.com/marketplace/apps/astronomer1591719760654.astronomer?tab=Overview0) Go ahead and dive in and unlock the full potential of Apache Airflow on AKS!