DHS e2e example changes #1585

bsrikan · 2018-11-27T23:35:25Z

This PR contains changes to the DHS e2e example. Renaming flows so they make sense.
Formatting and adding more info in the README to look up inserted docs etc.

sbayatpur · 2018-11-28T00:05:11Z

examples/DHS-e2e/README.md

-
-
-___You can verify via REST port 8004 for ingested/harmonized docs___
+# Running Data Hub Service end-end #


Suggested change

# Running Data Hub Service end-end #

# Running Data Hub Service end-to-end #

sbayatpur · 2018-11-28T00:11:21Z

examples/DHS-e2e/README.md

-___You can verify via REST port 8004 for ingested/harmonized docs___
+# Running Data Hub Service end-end #
+
+There are 2 projects one for each DHS and DSF. DHS is the project to run DHF flows in the stack spun up by DHS. Whereas DSF is the project that consumes curated data in the FINAL database. Commands to be executed in sequence are as under:


Suggested change

There are 2 projects one for each DHS and DSF. DHS is the project to run DHF flows in the stack spun up by DHS. Whereas DSF is the project that consumes curated data in the FINAL database. Commands to be executed in sequence are as under:

There are two projects — one for running DHF flows in Data Hub Service (DHS), and one for consuming data from the final database with Data Services (DSF).

sbayatpur · 2018-11-28T00:12:15Z

examples/DHS-e2e/README.md

+
+There are 2 projects one for each DHS and DSF. DHS is the project to run DHF flows in the stack spun up by DHS. Whereas DSF is the project that consumes curated data in the FINAL database. Commands to be executed in sequence are as under:
+
+## Pre-req ##


Suggested change

## Pre-req ##

## Prerequisites ##

sbayatpur · 2018-11-28T00:12:47Z

examples/DHS-e2e/README.md

+There are 2 projects one for each DHS and DSF. DHS is the project to run DHF flows in the stack spun up by DHS. Whereas DSF is the project that consumes curated data in the FINAL database. Commands to be executed in sequence are as under:
+
+## Pre-req ##
+Gradle 4.x+ installed globally.


Suggested change

Gradle 4.x+ installed globally.

Gradle 4.x+ is installed globally.

I don't think that you should use a global gradle install step. gradlew is much safer to use and less invasive. I actually have never installed gradle on my laptop.

OK. I added it because I was running into issues with gradle version, since DHF expect 3.4+ and DSF expected 3.8+ I think. So I installed a 4.x+ and just used that and thought it will be easier. I can include the wrapper within these projects so folks trying this out there dont have to install gradle.

DHF requires a minimum of 3.4, but we support 4.x. Packaging one would be even easier, I agree, and help insulate the DHS from 'DHF-esque' deployment specifics.

sbayatpur · 2018-11-28T00:19:11Z

examples/DHS-e2e/README.md

+  * `export PATH=$PATH:<unzipped dir>/gradle4.2.1/bin`
+  * source ~/.bash_profile
+
+There is a gradle task “importAllCustomers” to ingest source documents. So you can either run that task or use your locally installed mlcp.sh as described later.


Suggested change

There is a gradle task “importAllCustomers” to ingest source documents. So you can either run that task or use your locally installed mlcp.sh as described later.

Install MarkLogic Content Pump (mlcp). Note: This example uses mlcp. If you prefer not to use mlcp, you can use the gradle task “importAllCustomers” to ingest source documents.

sbayatpur · 2018-11-28T00:19:57Z

examples/DHS-e2e/README.md

+
+## Pre-req ##
+Gradle 4.x+ installed globally.
+* wget https://services.gradle.org/distributions/gradle-4.2.1-bin.zip


Does 'wget' mean something or is it a typo?

wget is a common utility to get things from the web, alternate to curl

sbayatpur · 2018-11-28T00:20:18Z

examples/DHS-e2e/README.md

+## Pre-req ##
+Gradle 4.x+ installed globally.
+* wget https://services.gradle.org/distributions/gradle-4.2.1-bin.zip
+* unzip to a dir of choice


Suggested change

* unzip to a dir of choice

* Unzip to a directory of your choice

sbayatpur · 2018-11-28T00:21:08Z

examples/DHS-e2e/README.md

+Gradle 4.x+ installed globally.
+* wget https://services.gradle.org/distributions/gradle-4.2.1-bin.zip
+* unzip to a dir of choice
+* update env var PATH in bash_profile


Suggested change

* update env var PATH in bash_profile

* Update the PATH environment variable in bash_profile

I appreciate that you documented everything so well, but this is specific to bash, and .bash_profile is specific so some operating systems. not important of course.

sbayatpur · 2018-11-28T00:22:27Z

examples/DHS-e2e/README.md

+* unzip to a dir of choice
+* update env var PATH in bash_profile
+  * `export PATH=$PATH:<unzipped dir>/gradle4.2.1/bin`
+  * source ~/.bash_profile


Is this a command they have to run or something else?

Yes its a command to run so the changes in .bash_profile can take effect.

sbayatpur · 2018-11-28T00:23:19Z

examples/DHS-e2e/README.md

+  * source ~/.bash_profile
+
+
+After creating stack in AWS make sure you have created users with appropriate roles. Update DHS/gradle.properties and DSF/gradle.properties to use them.


Suggested change

After creating stack in AWS make sure you have created users with appropriate roles. Update DHS/gradle.properties and DSF/gradle.properties to use them.

After creating your stack in AWS, make sure you have created users with appropriate roles. Update DHS/gradle.properties and DSF/gradle.properties to use them.

sbayatpur · 2018-11-28T00:28:33Z

examples/DHS-e2e/README.md

+
+After creating stack in AWS make sure you have created users with appropriate roles. Update DHS/gradle.properties and DSF/gradle.properties to use them.
+
+* For DHS project


Suggested change

* For DHS project

* For the DHS project

sbayatpur · 2018-11-28T00:28:45Z

examples/DHS-e2e/README.md

+* For DHS project
+  * User with flowDeveloper role can create, load into modules DB and run flows
+  * User with flowOperator role can only run flows
+* For DSF project


Suggested change

* For DSF project

* For the DSF project

sbayatpur · 2018-11-28T00:29:55Z

examples/DHS-e2e/README.md

+After creating stack in AWS make sure you have created users with appropriate roles. Update DHS/gradle.properties and DSF/gradle.properties to use them.
+
+* For DHS project
+  * User with flowDeveloper role can create, load into modules DB and run flows


Suggested change

* User with flowDeveloper role can create, load into modules DB and run flows

* Users with the flowDeveloper role can load flows into the modules database and run them

sbayatpur · 2018-11-28T00:30:11Z

examples/DHS-e2e/README.md

+
+* For DHS project
+  * User with flowDeveloper role can create, load into modules DB and run flows
+  * User with flowOperator role can only run flows


Suggested change

* User with flowOperator role can only run flows

* Users with the flowOperator role can only run flows

sbayatpur · 2018-11-28T00:32:46Z

examples/DHS-e2e/README.md

+  * User with flowDeveloper role can create, load into modules DB and run flows
+  * User with flowOperator role can only run flows
+* For DSF project
+  * User with endpointDeveloper role can load into modules DB and call the DSF API


Suggested change

* User with endpointDeveloper role can load into modules DB and call the DSF API

* Users with endpointDeveloper role can load documents into the modules database and call the DSF API

sbayatpur · 2018-11-28T00:33:42Z

examples/DHS-e2e/README.md

+  * User with flowOperator role can only run flows
+* For DSF project
+  * User with endpointDeveloper role can load into modules DB and call the DSF API
+  * User with endpointUser role can only call the DSF API


Suggested change

* User with endpointUser role can only call the DSF API

* Users with the endpointUser role can only call the DSF API

sbayatpur · 2018-11-28T00:34:36Z

examples/DHS-e2e/README.md

+
+
+## Assumptions and things to note ##
+* This project assumes that DHS environment is already provisioned. All the app servers, databases are provisioned and required roles created as described in pre-req above


Suggested change

* This project assumes that DHS environment is already provisioned. All the app servers, databases are provisioned and required roles created as described in pre-req above

* This project assumes that DHS environment is already provisioned. All the app servers and databases should be provisioned and required roles created as described in prerequisites above

sbayatpur · 2018-11-28T00:35:45Z

examples/DHS-e2e/README.md

+## Assumptions and things to note ##
+* This project assumes that DHS environment is already provisioned. All the app servers, databases are provisioned and required roles created as described in pre-req above
+* This example was executed from bastion host
+* DHS enables you to configure the endpoints to be private or public. If they are public, you can run this project from your laptop. If the endpoints are private, then these hosts are only accessible from the VPC that is peered to the MarkLogic VPC (This can be accessed from your local by ssh tunneling). In either case please update mlHost in DHS/gradle.properties and DSF/gradle.properties to use Flows endpoint.


Is ssh tunneling in addition to using a bastion host?

Not an addition but an alternate to bastion host, so you can run things from your laptop.

grechaw · 2018-11-28T00:32:44Z

examples/DHS-e2e/DHS/build.gradle

+task importAllCustomers(type: com.marklogic.gradle.task.MlcpTask) {
+	doFirst {
+		classpath = configurations.mlcp
+		input_file_path = "/Users/sbalasub/git/marklogic-data-hub/examples/DHS-e2e/DHS/input/json/customers/"


How will you accommodate this use of your home directory?

I have asked folks to edit the input file path in the README.

grechaw · 2018-11-28T00:33:22Z

examples/DHS-e2e/DSF/src/main/java/com/marklogic/APIs/testCustomer.java

+import com.fasterxml.jackson.databind.JsonNode;
+
+
+public class testCustomer {


In Java always capitalize class names

grechaw · 2018-11-28T00:34:54Z

examples/DHS-e2e/README.md

+There are 2 projects one for each DHS and DSF. DHS is the project to run DHF flows in the stack spun up by DHS. Whereas DSF is the project that consumes curated data in the FINAL database. Commands to be executed in sequence are as under:
+
+## Pre-req ##
+Gradle 4.x+ installed globally.


I don't think that you should use a global gradle install step. gradlew is much safer to use and less invasive. I actually have never installed gradle on my laptop.

grechaw · 2018-11-28T00:35:18Z

examples/DHS-e2e/README.md

+
+## Pre-req ##
+Gradle 4.x+ installed globally.
+* wget https://services.gradle.org/distributions/gradle-4.2.1-bin.zip


wget is a common utility to get things from the web, alternate to curl

grechaw · 2018-11-28T00:36:08Z

examples/DHS-e2e/README.md

+Gradle 4.x+ installed globally.
+* wget https://services.gradle.org/distributions/gradle-4.2.1-bin.zip
+* unzip to a dir of choice
+* update env var PATH in bash_profile


I appreciate that you documented everything so well, but this is specific to bash, and .bash_profile is specific so some operating systems. not important of course.

sbayatpur · 2018-11-28T00:49:34Z

examples/DHS-e2e/README.md

+* DHS enables you to configure the endpoints to be private or public. If they are public, you can run this project from your laptop. If the endpoints are private, then these hosts are only accessible from the VPC that is peered to the MarkLogic VPC (This can be accessed from your local by ssh tunneling). In either case please update mlHost in DHS/gradle.properties and DSF/gradle.properties to use Flows endpoint.
+
+
+# Steps to run end to end #


Suggested change

# Steps to run end to end #

# Steps #

sbayatpur · 2018-11-28T00:50:35Z

examples/DHS-e2e/README.md

+## Getting your flows in DHS ##
+1. cd `<path to DHS>`
+2. Install data-hub core MODULES
+    1. gradle hubInstallModules


Suggested change

1. gradle hubInstallModules

1. Run gradle hubInstallModules

sbayatpur · 2018-11-28T00:51:16Z

examples/DHS-e2e/README.md

+1. cd `<path to DHS>`
+2. Install data-hub core MODULES
+    1. gradle hubInstallModules
+    2. data-hub-MODULES should have 133 documents. You can verify this in your browser:


Suggested change

2. data-hub-MODULES should have 133 documents. You can verify this in your browser:

2. Verify that data-hub-MODULES has 133 documents from your browser:

sbayatpur · 2018-11-28T00:51:31Z

examples/DHS-e2e/README.md

+    2. data-hub-MODULES should have 133 documents. You can verify this in your browser:
+        ___http://CURATION_ENDPOINT:8004/v1/search?database=data-hub-MODULES___
+3. Load your modules for input/harmoninzation flows
+    1. gradle mlLoadModules


Suggested change

1. gradle mlLoadModules

1. Run gradle mlLoadModules

Its a command so didnt want to include it in a sentence. The sub bullet is the thing to run to get the thing described in main bullet done.

sbayatpur · 2018-11-28T00:52:25Z

examples/DHS-e2e/README.md

+        ___http://CURATION_ENDPOINT:8004/v1/search?database=data-hub-MODULES___
+3. Load your modules for input/harmoninzation flows
+    1. gradle mlLoadModules
+    2. data-hub-MODULES should now have 145 documents. You can verify this in your browser:


Suggested change

2. data-hub-MODULES should now have 145 documents. You can verify this in your browser:

2. Verify that data-hub-MODULES has 145 documents from your browser:

sbayatpur · 2018-11-28T00:52:48Z

examples/DHS-e2e/README.md

+    1. gradle mlLoadModules
+    2. data-hub-MODULES should now have 145 documents. You can verify this in your browser:
+        ___http://CURATION_ENDPOINT:8004/v1/search?database=data-hub-MODULES___
+4. Run input flow


Suggested change

4. Run input flow

4. Run the input flow

sbayatpur · 2018-11-28T00:55:39Z

examples/DHS-e2e/README.md

+        ___Alternately you can run___
+
+    2. gradle importAllCustomers
+        1. Ensure to update path to input documents in DHS/build.gradle where the task is defined


Suggested change

1. Ensure to update path to input documents in DHS/build.gradle where the task is defined

- Make sure to update the path to the input documents in DHS/build.gradle, where the task is defined

sbayatpur · 2018-11-28T00:56:08Z

examples/DHS-e2e/README.md

+
+    2. gradle importAllCustomers
+        1. Ensure to update path to input documents in DHS/build.gradle where the task is defined
+    3. Post ingestion, there should be 11 documents in data-hub-STAGING:


Suggested change

3. Post ingestion, there should be 11 documents in data-hub-STAGING:

3. Post ingestion, verify there are 11 documents in data-hub-STAGING:

sbayatpur · 2018-11-28T00:56:22Z

examples/DHS-e2e/README.md

+        1. Ensure to update path to input documents in DHS/build.gradle where the task is defined
+    3. Post ingestion, there should be 11 documents in data-hub-STAGING:
+        ___http://CURATION_ENDPOINT:8004/v1/search?database=data-hub-STAGING___
+5. Run harmonization flow


Suggested change

5. Run harmonization flow

5. Run the harmonization flow

sbayatpur · 2018-11-28T00:56:45Z

examples/DHS-e2e/README.md

+    3. Post ingestion, there should be 11 documents in data-hub-STAGING:
+        ___http://CURATION_ENDPOINT:8004/v1/search?database=data-hub-STAGING___
+5. Run harmonization flow
+    1. gradle hubRunFlow -PentityName=Customer -PflowName=customerHarmonize


Suggested change

1. gradle hubRunFlow -PentityName=Customer -PflowName=customerHarmonize

1. Run gradle hubRunFlow -PentityName=Customer -PflowName=customerHarmonize

sbayatpur · 2018-11-28T00:57:19Z

examples/DHS-e2e/README.md

+        ___http://CURATION_ENDPOINT:8004/v1/search?database=data-hub-STAGING___
+5. Run harmonization flow
+    1. gradle hubRunFlow -PentityName=Customer -PflowName=customerHarmonize
+    2. Post harmonization, there should be 11 documents in data-hub-FINAL:


Suggested change

2. Post harmonization, there should be 11 documents in data-hub-FINAL:

2. Verify there are 11 documents in data-hub-FINAL:

sbayatpur · 2018-11-28T00:57:39Z

examples/DHS-e2e/README.md

+    2. Post harmonization, there should be 11 documents in data-hub-FINAL:
+        ___http://CURATION_ENDPOINT:8004/v1/search?database=data-hub-FINAL___
+
+## Consuming curated data from FINAL database ##


Suggested change

## Consuming curated data from FINAL database ##

## Consuming curated data from the final database ##

sbayatpur · 2018-11-28T00:58:46Z

examples/DHS-e2e/README.md

+        ___http://CURATION_ENDPOINT:8004/v1/search?database=data-hub-FINAL___
+
+## Consuming curated data from FINAL database ##
+1. cd `<path to DSF>`


Suggested change

1. cd `<path to DSF>`

1. Navigate to the DSF directory (cd `<path to DSF>`)

sbayatpur · 2018-11-28T00:59:03Z

examples/DHS-e2e/README.md

+
+## Consuming curated data from FINAL database ##
+1. cd `<path to DSF>`
+2. Load your APIs into data-hub-MODULES database


Suggested change

2. Load your APIs into data-hub-MODULES database

2. Load your APIs into the data-hub-MODULES database

sbayatpur · 2018-11-28T00:59:21Z

examples/DHS-e2e/README.md

+## Consuming curated data from FINAL database ##
+1. cd `<path to DSF>`
+2. Load your APIs into data-hub-MODULES database
+    1. gradle mlLoadModules


Suggested change

1. gradle mlLoadModules

1. Run gradle mlLoadModules

sbayatpur · 2018-11-28T01:00:40Z

examples/DHS-e2e/README.md

+1. cd `<path to DSF>`
+2. Load your APIs into data-hub-MODULES database
+    1. gradle mlLoadModules
+        1. If you are using same project to run against another AWS stack, delete `module-timestamps.properties` under `build/ml-javaclient-util` dir


Should they do this after or before running gradle mlLoadModules?

sbayatpur · 2018-11-28T01:01:05Z

examples/DHS-e2e/README.md

+2. Load your APIs into data-hub-MODULES database
+    1. gradle mlLoadModules
+        1. If you are using same project to run against another AWS stack, delete `module-timestamps.properties` under `build/ml-javaclient-util` dir
+        2. data-hub-MODULES should now have 152 documents. You can verify this in your browser:


Suggested change

2. data-hub-MODULES should now have 152 documents. You can verify this in your browser:

2. Verify that data-hub-MODULES should has 152 documents from your browser:

sbayatpur · 2018-11-28T01:01:49Z

examples/DHS-e2e/README.md

+        2. data-hub-MODULES should now have 152 documents. You can verify this in your browser:
+            ___http://CURATION_ENDPOINT:8004/v1/search?database=data-hub-MODULES___
+3. Call the API. The API runs a query on FINAL database to return all the Customers who have "Sales" in their title
+    1. gradle runMain


Is there a way to indent without adding a bullet point or number in front of 'gradle runMain'? Also, for all these commands, can we format them so they appear differently than normal text?

sbayatpur · 2018-11-28T01:09:03Z

examples/DHS-e2e/README.md

+4. Run input flow
+    1. mlcp.sh import -mode "local" -host "`Ingest/Flows endpoint`" -port "8006" -username "xx" -password "yy" -input_file_path "`path to DHS/input/json/customers/`" -input_file_type "documents" -output_collections "Customer,DHS" -output_permissions "rest-reader,read,rest-writer,update" -output_uri_replace "`path to DHS/input/json`,''" -document_type "json" -transform_module "/data-hub/4/transforms/mlcp-flow-transform.sjs" -transform_namespace "http://marklogic.com/data-hub/mlcp-flow-transform" -transform_param "entity-name=Customer,flow-name=customerInput" -restrict_hosts true
+
+        ___Alternately you can run___


Suggested change

___Alternately you can run___

___Alternatively you can run___

sbayatpur · 2018-11-28T01:10:46Z

examples/DHS-e2e/README.md

+
+        ___Alternately you can run___
+
+    2. gradle importAllCustomers


This shouldn't be #2 if it's not a sequential step...you can try adding a bullet underneath 1. instead

aebadirad · 2018-12-21T16:56:36Z

@bsrikan Are we good on this?

…rything

bsrikan · 2018-12-21T21:31:15Z

@aebadirad All ready to merge.

bsrikan requested review from aebadirad, sbayatpur and eltesoro-ml November 27, 2018 23:35

sbayatpur reviewed Nov 28, 2018

View reviewed changes

grechaw suggested changes Nov 28, 2018

View reviewed changes

sbayatpur reviewed Nov 28, 2018

View reviewed changes

bsrikan force-pushed the develop branch 2 times, most recently from a29b8c5 to 593ee3d Compare December 5, 2018 23:35

bsrikan force-pushed the develop branch 4 times, most recently from 203898d to 4fab171 Compare December 20, 2018 17:52

bsrikan added 9 commits December 21, 2018 12:04

Formatted and made some edits to DHS-e2e examples readme

f127e82

Using DHF 4.0.3 in example and making necessary changes to DSF files

dd7891a

Renamed Customer input and harmonization flows

7f4c45c

Deleting old input/harm flows with cryptic names

002573c

Creating gradle wrappers so users dont have to install gradle

7dbf6f7

Capitalizing class name

e3540a0

Incorporated review comments for README

c231438

Updated import task so it takes values from gradle.properties for eve…

4b69c7f

…rything

Commits with 4.1 updates

dec8248

bsrikan force-pushed the develop branch from 4fab171 to dec8248 Compare December 21, 2018 21:30

aebadirad approved these changes Dec 21, 2018

View reviewed changes

aebadirad merged commit af5dddd into Marklogic-retired:develop Dec 21, 2018



		___You can verify via REST port 8004 for ingested/harmonized docs___
		# Running Data Hub Service end-end #

	# Running Data Hub Service end-end #
	# Running Data Hub Service end-to-end #

	There are 2 projects one for each DHS and DSF. DHS is the project to run DHF flows in the stack spun up by DHS. Whereas DSF is the project that consumes curated data in the FINAL database. Commands to be executed in sequence are as under:
	There are two projects — one for running DHF flows in Data Hub Service (DHS), and one for consuming data from the final database with Data Services (DSF).


		There are 2 projects one for each DHS and DSF. DHS is the project to run DHF flows in the stack spun up by DHS. Whereas DSF is the project that consumes curated data in the FINAL database. Commands to be executed in sequence are as under:

		## Pre-req ##

	Gradle 4.x+ installed globally.
	Gradle 4.x+ is installed globally.

	There is a gradle task “importAllCustomers” to ingest source documents. So you can either run that task or use your locally installed mlcp.sh as described later.
	Install MarkLogic Content Pump (mlcp). Note: This example uses mlcp. If you prefer not to use mlcp, you can use the gradle task “importAllCustomers” to ingest source documents.

	* unzip to a dir of choice
	* Unzip to a directory of your choice

	* update env var PATH in bash_profile
	* Update the PATH environment variable in bash_profile

		* source ~/.bash_profile


		After creating stack in AWS make sure you have created users with appropriate roles. Update DHS/gradle.properties and DSF/gradle.properties to use them.

	After creating stack in AWS make sure you have created users with appropriate roles. Update DHS/gradle.properties and DSF/gradle.properties to use them.
	After creating your stack in AWS, make sure you have created users with appropriate roles. Update DHS/gradle.properties and DSF/gradle.properties to use them.


		After creating stack in AWS make sure you have created users with appropriate roles. Update DHS/gradle.properties and DSF/gradle.properties to use them.

		* For DHS project

	* User with flowDeveloper role can create, load into modules DB and run flows
	* Users with the flowDeveloper role can load flows into the modules database and run them

	* User with flowOperator role can only run flows
	* Users with the flowOperator role can only run flows

	* User with endpointDeveloper role can load into modules DB and call the DSF API
	* Users with endpointDeveloper role can load documents into the modules database and call the DSF API

	* User with endpointUser role can only call the DSF API
	* Users with the endpointUser role can only call the DSF API



		## Assumptions and things to note ##
		* This project assumes that DHS environment is already provisioned. All the app servers, databases are provisioned and required roles created as described in pre-req above

		import com.fasterxml.jackson.databind.JsonNode;


		public class testCustomer {

		* DHS enables you to configure the endpoints to be private or public. If they are public, you can run this project from your laptop. If the endpoints are private, then these hosts are only accessible from the VPC that is peered to the MarkLogic VPC (This can be accessed from your local by ssh tunneling). In either case please update mlHost in DHS/gradle.properties and DSF/gradle.properties to use Flows endpoint.


		# Steps to run end to end #

	2. data-hub-MODULES should have 133 documents. You can verify this in your browser:
	2. Verify that data-hub-MODULES has 133 documents from your browser:

	2. data-hub-MODULES should now have 145 documents. You can verify this in your browser:
	2. Verify that data-hub-MODULES has 145 documents from your browser:

DHS e2e example changes #1585

DHS e2e example changes #1585

Conversation

bsrikan commented Nov 27, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbayatpur Nov 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sbayatpur Nov 28, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aebadirad commented Dec 21, 2018

bsrikan commented Dec 21, 2018

sbayatpur Nov 28, 2018 •

edited

Loading

sbayatpur Nov 28, 2018 •

edited

Loading

	1. Ensure to update path to input documents in DHS/build.gradle where the task is defined
	- Make sure to update the path to the input documents in DHS/build.gradle, where the task is defined

	3. Post ingestion, there should be 11 documents in data-hub-STAGING:
	3. Post ingestion, verify there are 11 documents in data-hub-STAGING:

	1. gradle hubRunFlow -PentityName=Customer -PflowName=customerHarmonize
	1. Run gradle hubRunFlow -PentityName=Customer -PflowName=customerHarmonize

	2. Post harmonization, there should be 11 documents in data-hub-FINAL:
	2. Verify there are 11 documents in data-hub-FINAL:

	## Consuming curated data from FINAL database ##
	## Consuming curated data from the final database ##

	1. cd `<path to DSF>`
	1. Navigate to the DSF directory (cd `<path to DSF>`)

	2. Load your APIs into data-hub-MODULES database
	2. Load your APIs into the data-hub-MODULES database

	2. data-hub-MODULES should now have 152 documents. You can verify this in your browser:
	2. Verify that data-hub-MODULES should has 152 documents from your browser:

	___Alternately you can run___
	___Alternatively you can run___