diff --git a/bin/pull_message b/bin/pull_message index 3237ff9d4e..d16c6703dc 100755 --- a/bin/pull_message +++ b/bin/pull_message @@ -4,6 +4,7 @@ # while true; do bin/pull_message PROJECT_ID; done # +ROOT=$(realpath $(dirname $0)/..) subscription=$USER-debug out_file=/tmp/captured.json @@ -21,13 +22,19 @@ subType=$(jq -r '.[0].message.attributes.subType' $out_file 2> /dev/null) subFolder=$(jq -r '.[0].message.attributes.subFolder' $out_file 2> /dev/null) deviceId=$(jq -r '.[0].message.attributes.deviceId' $out_file 2> /dev/null) timestamp=$(jq -r '.[0].message.data' $out_file 2>/dev/null | base64 --decode | jq -r .timestamp 2> /dev/null) || true +data=$(jq -r .[0].message.data /tmp/captured.json | base64 --decode) + +out_folder=$ROOT/out/devices/$deviceId +mkdir -p $out_folder if [[ $subType == null ]]; then subType=event fi if [[ $subFolder != null ]]; then - echo $subFolder $subType for $deviceId at $timestamp + echo $subFolder $subType $deviceId $timestamp + out_file=$out_folder/${timestamp}_${subFolder}_${subType}.json + echo $data | jq . > $out_file else echo No matching messages found. fi diff --git a/docs/cloud/gcp/cloud_setup.md b/docs/cloud/gcp/cloud_setup.md index e50d9ad553..290ec2979c 100644 --- a/docs/cloud/gcp/cloud_setup.md +++ b/docs/cloud/gcp/cloud_setup.md @@ -28,3 +28,4 @@ in order to use the the validator and registar tools * IoT Core Provisioner * Pub/Sub Editor +6. Install [Cloud Functions](functions.md) diff --git a/docs/cloud/gcp/functions.md b/docs/cloud/gcp/functions.md new file mode 100644 index 0000000000..089fa3211e --- /dev/null +++ b/docs/cloud/gcp/functions.md @@ -0,0 +1,60 @@ +[**UDMI**](../../../) / [**Docs**](../../) / [**Cloud**](../) / [**GCP**](./) / [Functions](#) + +# Cloud Functions + +## Deploy + +Run `udmis/deploy_udmis_gcloud ${project_id}` + +## Basic Architecture + +The basic UDMI function architecture allows for data flow through the system and integration +with cloud-based agents. + +External Components +* **Agent**: External (to UDMI core) agent that processes messages and updates config. Could be + something like a ML Agent for setpoint optimization, or a Discovery Mapping agent. Each agent will + typically have its own dedicated PubSub subscription. +* **IoT Core**: Primary connection point for external (on-site) devices. Represents an authentication + point and (nominally) MQTT connect endpoint. Devices particular to a given site are grouped + a dedicated _site registry_. + +Cloud Topics/Functions +* _udmi\_target_: Core tap point for all transactions. +* _udmi\_state_: Simple shunt for adding attributes to state messages. +* _udmi\_config_: Configuration update handling. + +```mermaid +graph LR + %%{wrap}%% + I[IoT Core
registries/site_name
devices/device_name] + T[topic/function:
udmi_target] + C[topic/function:
udmi_config] + S[topic/function:
udmi_state] + J[subscription:
agent] + A[external
agent
] + + I -.-> |event| T + I -.-> |state| S + S -.-> T + + C --> I + + T --> J + J --> A + A -.-> |config| C + C -.-> T +``` +(solid lines are direct API calls, dashed lines are PubSub messages) + +## Reflector Architecture + +The UDMI reflector setup is designed to allow an external (to the GCP project) _utility_ to interact +with the system through a channel authenticated as an IoT Device. This is functionally equivalent +to an _agent_ connected over PubSub, but has a different authenticaiton/connection paradigm. Specifically, +a PubSub connection will rely on a GCP-based IAM auth to provide access to PubSub resources, +while a _reflector_ connection relies on a device-specific credential and connects over MQTT. + +Utilities connect to the `UDMS-REFLECT` IoT Core registry, backed by the _udmi__reflect_ topic/function. +The 'device' within the `UDMS-REFLECT` registry is actually the _site\_name_ of the site (so not +an actual device), corresponding to the IoT Core _registry_ containing the actual device connections. diff --git a/docs/cloud/gcp/readme.md b/docs/cloud/gcp/readme.md index 9ae48b0fe1..e1347faa0c 100644 --- a/docs/cloud/gcp/readme.md +++ b/docs/cloud/gcp/readme.md @@ -3,7 +3,7 @@ # GCP - [Cloud Setup](cloud_setup.md): Basic cloud project and IoT core configuration -- [UDMIS Infrastructure Setup](udmis.md): UDMIS Infrastructure (cloud functions) - [UDMI terraform configuration files](terraform.md): Automate the creation of UDMI related infrastructure on GCP -- [Pubber Swarm](swarm.md): Simulate 1000's of devices connecting into IoT Core +- [UDMIS Infrastructure Setup](udmis.md): UDMIS Infrastructure (cloud functions) - [UDMIF Deployment](../../../udmif/deploy.md): UDMIF Dashboard deployment instructions +- [Pubber Swarm](swarm.md): Simulate 1000's of devices connecting into IoT Core diff --git a/udmis/deploy_udmis_gcloud b/udmis/deploy_udmis_gcloud index 0005be069f..4b500a471e 100755 --- a/udmis/deploy_udmis_gcloud +++ b/udmis/deploy_udmis_gcloud @@ -38,5 +38,4 @@ done echo Waiting for all deployments to complete... wait -echo Skipped deploy of udmi_update b/c it requires firebase. echo Done with deploying functions $PUBSUB_FUNCTIONS