Fix #10344 - Use ingestionPipeline retries when deploying

[pmbrull]

Describe your changes:

Fixes #10344

When creating the ingestion, we should allow users to configure the number of retries if the workflow fails. By default, it will be 0.

We need to pass this information in the ingestionPipeline.airflowConfig.retries field. We have already updated the ingestion framework to use it.

We'll do the changes in the Collate repo accordingly

Screen.Recording.2023-08-21.at.5.54.05.PM.mov

Type of change:

• Bug fix
• Improvement
• New feature
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• Documentation

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

[cypress]

Passing run #28269 ↗︎

0

321

37

3

0

Details:

Merge branch 'main' into issue-10344
Project: openmetadata	Commit: 45572ad31e
Status: Passed	Duration: 33:43 💡
Started: Aug 22, 2023 5:38 AM	Ended: Aug 22, 2023 6:12 AM

_{This comment has been generated by cypress-bot as a result of this project's GitHub integration settings.}

[sonarqubecloud]

[open-metadata-ui] Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

0.0% Coverage
0.0% Duplication

The version of Java (11.0.14.1) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

[sonarqubecloud]

[open-metadata-airflow-apis] Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication

[sonarqubecloud]

[open-metadata-ingestion] Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
No Duplication information