Sunbird portal deployment queries

[SameerT-protean]

@ravismula
After reviewing the easy installer GitHub repo. We have few queries and doubts regarding the sunbird portal deployment.
Kindly provide update on below mentioned points.

1. Which module from the Terraform folder in the easy installer GitHub repo should be deployed first?
2. Is there a recommended order or dependency graph for deploying the Terraform modules within the easy installer repository?
3. Can the Sunbird team provide more detailed documentation or readme files for the
   repositories?
4. Could Sunbird provide a high-level overview of the Sunbird platform's architecture, including the key components and their interactions?
5. What is the typical deployment pipeline for Sunbird? Are there any specific tools or methodologies used for continuous integration and continuous deployment (CI/CD)?
6. What are the key application connections and how do they interconnect on the portal?
7. Where can we find the Docker image locations, and what are the specific environment variables and VPC setup required?
8. What are the critical environment variables that need to be set for the Sunbird components?
9. Can the Sunbird team provide a overview of the entire deployment process and the interconnectivity of the applications?

[ravismula]

Hi @SameerT-protean

1. Which module from the Terraform folder in the easy installer GitHub repo should be deployed first?

Start with release-7.0.0 branch.

Inside the terraform/azure/_common directory, you will find all the modules that create the infrastructure.

• The network.hcl file is used to create the network infrastructure.
• The aks.hcl file is used to create the AKS cluster.

To create the AKS cluster, the network module is used. To find out the dependency, In the hcl file look for inputs where some values will start dependency and the value will be the output of the module. For example, in the aks.hcl file, you will find the below code:

inputs = {
  environment                        = local.environment
  building_block             = local.building_block
  resource_group_name        = dependency.network.outputs.resource_group_name
  vnet_subnet_id             = dependency.network.outputs.aks_subnet_id
}

Here, the aks module is dependent on the network module. The aks_subnet_id and resource_group_name are from the output of the network module

Simlarly, you can find the dependency of other modules. The one that doesn't have any dependency is the first module to be created. Terraform will create the infrastructure in the order of dependency and will parallelize the creation of resources where possible.

2. Is there a recommended order or dependency graph for deploying the Terraform modules within the easy installer repository?

The order of creation of resources is driven using install.sh in the template directory. When deploying the infra, create a copy of the template directory say demo as described in the README.md file here https://github.com/project-sunbird/sunbird-ed-installer/blob/release-7.0.0/README.md. The lines 238-249 in the install.sh file will create the resources in the order of dependency.

3. Can the Sunbird team provide more detailed documentation or readme files for the repositories?

Each repository has a README.md file that explains the purpose of the repository and how to use it. For example https://github.com/Sunbird-Ed/SunbirdEd-portal/blob/master/README.md has detailed information on Sunbird Ed Portal. If you are unable to find information about a particular repository or looking for more details, please feel free to ask in more detail.

4. Could Sunbird provide a high-level overview of the Sunbird platform's architecture, including the key components and their interactions?

The Ed Microsite has the details of the entire Ed Portal which includes Architecture Overview of Sunbird Ed Building Block and its dependencies on other building block. It also has youtube videos that explain the architecture and its components.

5. What is the typical deployment pipeline for Sunbird? Are there any specific tools or methodologies used for continuous integration and continuous deployment (CI/CD)?

CI/CD integration is via the traditional flow, which is not releavant to sunbird-ed-installer. The part involving building images and making them available on Azure Container Registry is via Jenkins, which is currenlty maintained by the Sunbird team.

6. What are the key application connections and how do they interconnect on the portal?

The architecture diagram along with the documentation/videos should answer this question. Ref Ans 4

7. Where can we find the Docker image locations, and what are the specific environment variables and VPC setup required?

Each module in the helmcharts has the information about the images that are in use. For example as a part of values.yaml in edbb has the information about the images that are used in this module. In cases where only the image tag is specified, the respective chart will have the detailed information about the image. Example. We adopt nested helmchart structure, where a bundle is a collection of helmcharts. Each bundle has its own values.yaml which overrides the values of the individual helmcharts that are part of the bundle.

8. What are the critical environment variables that need to be set for the Sunbird components?

Refer to terraform/azure/template/environment.hcl and terraform/azure/template/global-values.yaml for the values that have to be overridden for each installation.

9. Can the Sunbird team provide a overview of the entire deployment process and the interconnectivity of the applications?

The deployment process is explained in the README.md file of the sunbird-ed-installer repository. The interconnectivity of the applications is explained in the architecture diagram and the documentation available on the Ed Microsite.

Hope these answers help you move forward with the deployment. If you have any more questions, feel free to post it here!

Cheers!

[SameerT-protean]

@ravismula
Are there any terraform script written for Google cloud ?

[ravismula]

@SameerT-protean - No, we don't have scripts for Google Cloud.

[SameerT-protean]

@ravismula
Migration from Azure to Google Cloud for Project Sunbird_ (Final Stage of Deployment).pdf

attached is a pdf file for the errors we are getting while deploying the application in GCP env.
there are some hardcoded values in the code that are pointing to Azure services. These hardcoded references are causing the deployment to fail. To resolve this, we need to remove or replace these hardcoded values to ensure a smooth and successful deployment process. For your reference, I have attached a pdf with more detailed information.

[ravismula]

@SameerT-protean

the changes have to made in the helm charts to support GCP

[SameerT-protean]

@ravismula

There are few more queries and issues were are facing.

We spent our time working on the deployment of inquirybb service. We were able to complete all the modifications required for the msa to run on GCP cloud. Apart from the mention of azure acr repositories, there are no mention of Azure/AWS/S3 anywhere in that code.

But after the deployment of the code, it is still failing. It’s either because it’s still looking for S3 FS or there are no libraries/modules present in the code itself for gs or GCP usage.

There is no FS present for gs file system, due to which the flink service is failing repeatedly.

We guess that the root cause is that the Flink msa lacks the necessary Hadoop libraries or configurations to interact with Google Cloud Storage.
And the solution to that would be to include the Hadoop-GCS connector in the Flink msa which will allow the Flink to interact with GCS via Hadoop’s filesystem API.

This is purely from code end, and we cannot look into this. Need your help in here.

Even though, there is no mention of S3 or Azure FS, the services are still looking for that because it’s coming from the pulled image used to roll up the msa, again a part where a developer needs to intervene and look into it.

[SameerT-protean]

@ravismula
Please find below update and few of the queries that needs to be addressed.

update-
Completed Services:
The following services have been successfully completed:
EDBB, Inquirybb, Monitoring, and Library.
Pending Items:
EDBB Service: Two pods in the EDBB service are still pending. The Sunbird team is aware of the issue, and their developers are working to resolve it.
Inquirybb Service: Two pods are failing due to the unavailability of the flink-config-map. Additionally, the flink-config-map is not defined in the GitHub repository. We need to reach out to the Sunbird team to confirm where this ConfigMap is being created.
Artifact Download:
We have successfully downloaded the required artifacts from Azure and stored them in our Google Cloud Storage.
Obsrvbb Service:
The Obsrvbb service is utilizing an AWS S3 bucket for storage. But we were told only about azure need to confirm with sunbird team whether we are using AWS also.
Other Services:
All remaining services are using Flink. It would be great if @ravismula you could assist us on this.