ED-1232: OCI support for Build/DataPipeline/CoreDataProducts #172

ddevadat · 2023-05-03T13:01:21Z

updated cloud-store-sdk to 1.4.6 along with associated class file changes

Signed-off-by: Deepak Devadathan <[email protected]>

…JAPACCOE-2821-release-5.0.0_RC1-release-5.1.1 Signed-off-by: Deepak Devadathan <[email protected]>

Signed-off-by: Deepak Devadathan <[email protected]>

anandp504

@ddevadat Please address the PR comments

anandp504 · 2023-05-23T09:40:57Z

.circleci/config.yml

-              cd sunbird-analytics-core && git checkout $target_branch
-          fi
+          git clone https://github.com/ocisunbird/sunbird-analytics-core.git
+          cd sunbird-analytics-core && git checkout oci-5.1.0-JAPACCOE-2839-release-5.0.0_RC2


@ddevadat This looks like a release tag is hard coded in the .circleci/config.yml. This doesn't look right.

@anandp504 you are right..lets park this PR as this needs some special attention.

The issue here is circle ci test runs. We have upgraded the cloud-store-sdk version in the analytics core repo, those changes are yet to merged in the release-5.1.1

Till that happens , everytime circle ci runs for core-dataproducts it will try to build from release-5.1.1 which doesnt have cloud-store-sdk:1.4.6 yet and will fail.

@ddevadat - Did we merge the analytics-core with the latest SDK changes? If yes, can we revert these changes back?

reverted the changes as the analytics core pr is merged

Signed-off-by: Deepak Devadathan <[email protected]>

ddevadat · 2023-05-26T03:35:39Z

@manjudr
we raised spark version to 3.2.1 to support oracle bds cluster. I change the jackson core version to 2.12.0. With that the analytics-core ci tests are running fine.

The problem is core-dataproducts.
I changed the spark version to 3.2.1, but i am getting errors like

(ip-10-0-178-138.ec2.internal executor driver): java.lang.NoSuchMethodError: 'org.apache.hadoop.fs.FutureDataInputStreamBuilder org.apache.hadoop.fs.FileSystem.openFile(org.apache.hadoop.fs.Path)'
	at org.apache.hadoop.mapred.LineRecordReader.<init>(LineRecordReader.java:111)
	at org.apache.hadoop.mapred.TextInputFormat.getRecordReader(TextInputFormat.java:67)
	at org.apache.spark.rdd.HadoopRDD$$anon$1.liftedTree1$1(HadoopRDD.scala:286)
	at org.apache.spark.rdd.HadoopRDD$$anon$1.<init>(HadoopRDD.scala:285)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:243)
	at org.apache.spark.rdd.HadoopRDD.compute(HadoopRDD.scala:96)

Any idea what could be causing this

Signed-off-by: Deepak Devadathan <[email protected]>

exclusion for hadoop-client-api

sonarqubecloud · 2023-05-26T05:30:58Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

0.0% Coverage
0.0% Duplication

Signed-off-by: Deepak Devadathan <[email protected]>

manjudr · 2023-06-19T11:31:07Z

@ddevadat - Are you still facing the java.lang.NoSuchMethodError: 'org.apache.hadoop.fs.FutureDataInputStreamBuilder org.apache.hadoop.fs.FileSystem.openFile error?

ddevadat · 2023-06-19T23:50:14Z

@ddevadat - Are you still facing the java.lang.NoSuchMethodError: 'org.apache.hadoop.fs.FutureDataInputStreamBuilder org.apache.hadoop.fs.FileSystem.openFile error?

no this error is not seen in the last circle ci build and test run

manjudr · 2023-08-03T06:18:39Z

batch-models/src/main/scala/org/ekstep/analytics/util/BaseDruidQueryProcessor.scala

@@ -29,6 +29,9 @@ trait BaseDruidQueryProcessor {
      case "s3" =>
        sc.hadoopConfiguration.set("fs.s3n.awsAccessKeyId", AppConf.getConfig(accountKey.getOrElse("aws_storage_key")));
        sc.hadoopConfiguration.set("fs.s3n.awsSecretAccessKey", AppConf.getConfig(accountSecret.getOrElse("aws_storage_secret")));
+      case "oci" =>


@ddevadat - We are making some CSP refactor changes. We might be required to revise these changes again.

Will do it from our side but you might be required to validate with the OCI environment once.

sonarqubecloud · 2023-08-03T07:57:11Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
2 Code Smells

0.0% Coverage
0.0% Duplication

The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more here

ddevadat added 24 commits January 17, 2023 23:19

updated cloud-store-sdk version 1.4.5

ef143f3

Signed-off-by: Deepak Devadathan <[email protected]>

included endpoint while initiating jobmanager storage

d41f584

Signed-off-by: Deepak Devadathan <[email protected]>

endpoint type as option

95b77e9

Signed-off-by: Deepak Devadathan <[email protected]>

updated the endpoint config lookup

92dab69

Signed-off-by: Deepak Devadathan <[email protected]>

testing with endpoint definition

e481404

Signed-off-by: Deepak Devadathan <[email protected]>

removed endpoint from storage config for testing

290e053

Signed-off-by: Deepak Devadathan <[email protected]>

added back storage endpoint

2e0dd7b

Signed-off-by: Deepak Devadathan <[email protected]>

updated cloud-store-sdk:1.4.6

6f8e3a5

Signed-off-by: Deepak Devadathan <[email protected]>

removed endpoint from jobmanager

7769f68

Signed-off-by: Deepak Devadathan <[email protected]>

Added the endpoint variable

ffe5455

Signed-off-by: Deepak Devadathan <[email protected]>

passing region flag

0fa66f8

Signed-off-by: Deepak Devadathan <[email protected]>

added endpoint for s3 spark configuration

de54886

Signed-off-by: Deepak Devadathan <[email protected]>

updated report storage configuration for oci

f466ad2

Signed-off-by: Deepak Devadathan <[email protected]>

updated sprk hd confi

7657b9b

Signed-off-by: Deepak Devadathan <[email protected]>

updated jets3t version

11c9ea4

Signed-off-by: Deepak Devadathan <[email protected]>

added debug lines

3e267ae

Signed-off-by: Deepak Devadathan <[email protected]>

added oci support

bea59ae

Signed-off-by: Deepak Devadathan <[email protected]>

Update .gitignore

54d0108

added endpoint in resources for testing

05eed9e

Signed-off-by: Deepak Devadathan <[email protected]>

Merge remote-tracking branch 'upstream/release-5.1.1' into oci-5.1.0-…

41cc8c7

…JAPACCOE-2821-release-5.0.0_RC1-release-5.1.1 Signed-off-by: Deepak Devadathan <[email protected]>

removed debug lines

3b012c5

Signed-off-by: Deepak Devadathan <[email protected]>

testing circle ci build

7469ecc

Signed-off-by: Deepak Devadathan <[email protected]>

testing with a modified circle ci build

bfdd80c

Signed-off-by: Deepak Devadathan <[email protected]>

update the branch for analy core checkout for circleci

9a07686

Signed-off-by: Deepak Devadathan <[email protected]>

anandp504 requested changes May 23, 2023

View reviewed changes

anandp504 mentioned this pull request May 23, 2023

ED-1232 - Add oci oss support for Build/DataPipeline/CoreDataProducts #171

Closed

5 tasks

ddevadat added 4 commits May 26, 2023 09:48

testing with spark 3.2.1

42b756b

Signed-off-by: Deepak Devadathan <[email protected]>

changed spark-cassandra-connector to 3.2.0

7a3c76f

Signed-off-by: Deepak Devadathan <[email protected]>

with spark-cassandra-connector-assembly

9f1506e

Signed-off-by: Deepak Devadathan <[email protected]>

reverted spark cassandra connector version

7a29c31

Signed-off-by: Deepak Devadathan <[email protected]>

ddevadat added 2 commits May 26, 2023 13:48

exclusion for hadoop-client-api

0d5381c

Signed-off-by: Deepak Devadathan <[email protected]>

Merge pull request #1 from ocisunbird/oci-release-5.1.1-test-pr

4a41a50

exclusion for hadoop-client-api

revereted to original circle ci

02d6347

Signed-off-by: Deepak Devadathan <[email protected]>

ddevadat mentioned this pull request Jun 26, 2023

ED-2087: Add OCI support for LernDataproducts Sunbird-Lern/data-products#52

Merged

manjudr reviewed Aug 3, 2023

View reviewed changes

This comment was marked as outdated.

Sign in to view

manjudr approved these changes Aug 3, 2023

View reviewed changes

manjudr merged commit f39c2fb into Sunbird-Obsrv:release-5.1.1 Aug 3, 2023

manjudr mentioned this pull request Aug 4, 2023

Revert "ED-1232: OCI support for Build/DataPipeline/CoreDataProducts " #179

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ED-1232: OCI support for Build/DataPipeline/CoreDataProducts #172

ED-1232: OCI support for Build/DataPipeline/CoreDataProducts #172

ddevadat commented May 3, 2023

anandp504 left a comment

anandp504 May 23, 2023

ddevadat May 24, 2023

manjudr Aug 3, 2023

ddevadat Aug 3, 2023

ddevadat commented May 26, 2023

sonarqubecloud bot commented May 26, 2023

manjudr commented Jun 19, 2023

ddevadat commented Jun 19, 2023

manjudr Aug 3, 2023

manjudr Aug 3, 2023 •

edited

Loading

This comment was marked as outdated.

sonarqubecloud bot commented Aug 3, 2023

ED-1232: OCI support for Build/DataPipeline/CoreDataProducts #172

ED-1232: OCI support for Build/DataPipeline/CoreDataProducts #172

Conversation

ddevadat commented May 3, 2023

anandp504 left a comment

Choose a reason for hiding this comment

anandp504 May 23, 2023

Choose a reason for hiding this comment

ddevadat May 24, 2023

Choose a reason for hiding this comment

manjudr Aug 3, 2023

Choose a reason for hiding this comment

ddevadat Aug 3, 2023

Choose a reason for hiding this comment

ddevadat commented May 26, 2023

sonarqubecloud bot commented May 26, 2023

manjudr commented Jun 19, 2023

ddevadat commented Jun 19, 2023

manjudr Aug 3, 2023

Choose a reason for hiding this comment

manjudr Aug 3, 2023 • edited Loading

Choose a reason for hiding this comment

This comment was marked as outdated.

sonarqubecloud bot commented Aug 3, 2023

manjudr Aug 3, 2023 •

edited

Loading