From bc19b1cba02e5d9579ed503769da8939fb92acd8 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Thu, 11 Jul 2024 15:30:13 -0700 Subject: [PATCH 01/58] Documentation cleanup --- docs/cloud/gcp/cloud_setup.md | 31 ------------------------------- docs/cloud/gcp/readme.md | 1 - docs/guides/development.md | 32 -------------------------------- docs/messages/config.md | 9 +-------- docs/messages/envelope.md | 3 +-- docs/messages/readme.md | 10 +++------- docs/specs/connecting.md | 2 +- docs/specs/gateway.md | 6 ------ docs/specs/message_walk.md | 7 ------- docs/specs/tech_stack.md | 30 +++++++----------------------- docs/tools/gcloud.md | 2 +- docs/tools/pubber.md | 13 ------------- docs/tools/sequencer.md | 4 ++-- docs/tools/setup.md | 6 ------ docs/tools/sharding.md | 2 +- docs/tools/validator.md | 9 +-------- 16 files changed, 18 insertions(+), 149 deletions(-) delete mode 100644 docs/cloud/gcp/cloud_setup.md diff --git a/docs/cloud/gcp/cloud_setup.md b/docs/cloud/gcp/cloud_setup.md deleted file mode 100644 index 14dcb13863..0000000000 --- a/docs/cloud/gcp/cloud_setup.md +++ /dev/null @@ -1,31 +0,0 @@ -[**UDMI**](../../../) / [**Docs**](../../) / [**Cloud**](../) / [**GCP**](./) / [Cloud Setup](#) - -# Cloud Setup - -1. Identify an existing GCP to host the system, or acquire a new one if necessary. - * Ensure [billing](https://cloud.google.com/billing/docs/how-to/modify-project) - has been enabled -2. Identify a GCP IoT Core registry you would like to use, or create a new one if necessary - * Refer to [GCP IoT Core Getting Started](https://cloud.google.com/iot/docs/how-tos/getting-started) - for additional information - * Search for IoT Core in the GCP Console - * Click `Enable` - * Click to create a new Registry -3. Assign PUB/SUB topics to the registry as described in the [Message Walk Guidance](../../specs/message_walk.md). - If the topics do not exist, create them - * Set the default _Cloud Pub/Sub topic_ for the registry to the topic `udmi_target` - * Set the _Device state topic_ (may be hidden under _Advanced Options_) to `udmi_state` -4. Install the Google Cloud SDK in order to be able to use some of the tools included, - such as the `registrar` tool - * Follow the guidance on [Google Cloud SDK](https://cloud.google.com/sdk/docs/install) - installation documentation - * Once installed, configure the - [application default credentials](https://cloud.google.com/sdk/gcloud/reference/auth/application-default) - using end-user credentials (`gcloud auth application-default login`). - * Select the project your registry resides in using - `gcloud config set project ` -5. Ensure the authenticated user has at minimum the following IAM permissions - in order to use the the `validator` and `registrar` tools - * IoT Core Provisioner - * Pub/Sub Editor -6. Install [Cloud Functions](functions.md) diff --git a/docs/cloud/gcp/readme.md b/docs/cloud/gcp/readme.md index e1347faa0c..8e8a156cee 100644 --- a/docs/cloud/gcp/readme.md +++ b/docs/cloud/gcp/readme.md @@ -2,7 +2,6 @@ # GCP -- [Cloud Setup](cloud_setup.md): Basic cloud project and IoT core configuration - [UDMI terraform configuration files](terraform.md): Automate the creation of UDMI related infrastructure on GCP - [UDMIS Infrastructure Setup](udmis.md): UDMIS Infrastructure (cloud functions) - [UDMIF Deployment](../../../udmif/deploy.md): UDMIF Dashboard deployment instructions diff --git a/docs/guides/development.md b/docs/guides/development.md index 77b92dcb4d..06b7cb5dc9 100644 --- a/docs/guides/development.md +++ b/docs/guides/development.md @@ -56,38 +56,6 @@ The below files need to be updated. Do not blindly copy! Inspect all diffs and c * `rm -rf tests/sites/*/**/*/out && rm -rf tests/sites/*/out` to remove out files from `bin/test_registrar` - -## Configuring Cloud CI Tests - -To enable the CI tests, there first needs to be a dedicated GCP Project with an IoT Core -registry which mirrors the [example site model](https://github.com/faucetsdn/udmi_site_model). -A Github variable must also configured to point to the GCP project - -They key steps to setup the dedicated project are as follows: -1. Setup up a GCP Project and IoT Core Registry. The - [cloud setup](../cloud/gcp/cloud_setup.md) and [UDMIS (cloud functions) setup](../cloud/gcp/udmis.md) - documents give guidance on this. If GCP Cloud SDK and re-authentication may be required. The registry name - and cloud region are as follows: - - **Registry Name**: `ZZ-TRI-FECTA` - - **Cloud Region**: us-central1 -2. Setup the site model by cloning the - [example site model](https://github.com/faucetsdn/udmi_site_model) - in the udmi root directory and running the [registrar](../tools/registrar.md) - tool to configure the site model in the IoT Core Registry. - - `git clone https://github.com/faucetsdn/udmi_site_model.git` - - `bin/registar udmi_site_model` -2. Set up the [sequence tests](../tools/sequencer.md). The public key used for the - virtual device in the IoT Core registry is the public key from - [`udmi_site_model/devices/AHU-1/rsa_public.pem`](https://raw.githubusercontent.com/faucetsdn/udmi_site_model/master/devices/AHU-1/rsa_public.pem). - A `validator_config.json` configuration file is not needed (this is - generated automatically during the CI test) - - The registry name is `ZZ-TRI-FECTA`. -4. A Github variable needs to be added to the project, accessed from the - project's _Settings_ page. - - **Name**: GCP_TARGET_PROJECT - - **Value**: _GCP Project ID_ -5. Enable Github Actions - The workflow can be tested with an empty commit (`git commit --allow-empty -m "Blank commit to trigger CI"; git push`). On an unmodified branch, these tests should pass if correctly configured diff --git a/docs/messages/config.md b/docs/messages/config.md index c31d6cf5dd..fbf8ed62c6 100644 --- a/docs/messages/config.md +++ b/docs/messages/config.md @@ -5,15 +5,8 @@ **Schema Definition:** [config.json](../../schema/config.json) ([_🧬View_](../../gencode/docs/config.html)) -The UDMI config block specifies the -[Cloud IoT Core Config](https://cloud.google.com/iot/docs/how-tos/config/configuring-devices) -block that controls a device's intended behavior. - -Unless a config message has an [expiry](../specs/sequences/writeback.md#value-expiration), the latest -config message is always considered present. - +The UDMI config block specifies information that controls a device's intended behavior. It is composed of specific sub-entries for each sub-system { _system_, _pointset_, _gateway_, etc... }. - This [working example](../../tests/schemas/config/example.json) shows how a typical `config` message is constructed. diff --git a/docs/messages/envelope.md b/docs/messages/envelope.md index ba5f693f92..298dcb9b62 100644 --- a/docs/messages/envelope.md +++ b/docs/messages/envelope.md @@ -6,8 +6,7 @@ The UDMI `envelope` is not a message itself, but the attributes and other information that is delivered along with a message. There is no direct awareness of it at the device level: it is -added and manipulated in the back-end only. Primarily based on the -[Cloud IoT Core message attributes](https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#publishing_telemetry_events). +added and manipulated in the back-end only. Primarily based on the message attributes. From the device side, the `subFolder` property of the MQTT topic is passed along to the message envelope, which allows the back-end to properly categorize and process data. A device does not need diff --git a/docs/messages/readme.md b/docs/messages/readme.md index 1612940779..3c84cbc2b8 100644 --- a/docs/messages/readme.md +++ b/docs/messages/readme.md @@ -36,15 +36,11 @@ flow, but also the treatment of the message as it moves through the system: The properties and uses of the four types fall out from this accordingly: * _state_: Sticky to the cloud information from the device, including information such as the overall -status of the device, and any errors or conditions from, e.g., writing points. See -[IoT Core State Docs](https://cloud.google.com/iot/docs/how-tos/config/getting-state) for more information. +status of the device, and any errors or conditions from, e.g., writing points. * _events_: The canonical streaming telemetry messages from the device, usually containing things -like temperature readings or system memory utilization. See -[IoT Core Event Docs](https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#publishing_telemetry_events) for more information. -* _config_: Ability to control the behavior of a device, e.g. for key rotation, writeback, etc... See -[IoT Core Config Docs](https://cloud.google.com/iot/docs/how-tos/config/configuring-devices) for more information. +like temperature readings or system memory utilization. +* _config_: Ability to control the behavior of a device, e.g. for key rotation, writeback, etc... * _command_: Direct but transitory messages to a device, e.g. to install a new auth key or perform diagnostic operations. -See [IoT Core Command Docs](https://cloud.google.com/iot/docs/how-tos/commands) for more information. Individual blocks (folders), such as _system_ or _pointset_ will have their own semantic uses for the various types. See the individual block folder documentation to learn more about those aspects. diff --git a/docs/specs/connecting.md b/docs/specs/connecting.md index 8b2c94b9b8..ef107ad7e6 100644 --- a/docs/specs/connecting.md +++ b/docs/specs/connecting.md @@ -26,7 +26,7 @@ On-prem Cloud * **Cloud**: The logical cloud representation of that device * _Prem-to-Cloud Connectivity_ (ascii `*` in the diagram) * **Network**: Basic IP network connectivity, but could be whatever works - * **Transport**: Typically MQTT & GCP IoT Core, but could be something else + * **Transport**: Typically MQTT, but could be something else ## Connection Models diff --git a/docs/specs/gateway.md b/docs/specs/gateway.md index 9647daaf8c..ce26ec1b29 100644 --- a/docs/specs/gateway.md +++ b/docs/specs/gateway.md @@ -7,12 +7,6 @@ or traditional devices that do not communicate directly to the cloud using a MQTT/UDMI connection. For example, an older [BACnet](http://www.bacnet.org/) based system could use a gateway to translate on-prem communications into UDMI. -The -[Google Cloud IoT Core Gateway Documentation](https://cloud.google.com/iot/docs/how-tos/gateways) -provides an overview of the cloud-side implementation of a gateway. UDMI, then, -specifies an additional layer of specification around the associated -message formats. - Conceptually, there are two types of entities involved: the _gateway device_, and the _proxied device_. Both of these are 'devices' in the sense that they have an entry in a cloud registry diff --git a/docs/specs/message_walk.md b/docs/specs/message_walk.md index dec4adafa1..e1a0e2197e 100644 --- a/docs/specs/message_walk.md +++ b/docs/specs/message_walk.md @@ -19,13 +19,6 @@ model. It's important for system testing to isolate the setup from anything exte if Pubber doesn't work then there's something wrong with the code or GCP configuration, otherwise it indicates a problem with the real/actual device. -## IoT Core - -[IoT Core](https://cloud.google.com/iot/docs/) is the externally-facing endpoint for on-prem devices. -It organizes a particular device into a { _project_, _registry_ } grouping. The important -configuration parameters for a given _registry_ are the _Default telemetry_ and _Device state_ topic -types, which should be set to _udmi\_target_ and _udmi\_state_, respectively. - ## PubSub Topics [PubSub](https://cloud.google.com/pubsub/docs/) is used as the primary communication mechanism for diff --git a/docs/specs/tech_stack.md b/docs/specs/tech_stack.md index 04def97219..bee185fa37 100644 --- a/docs/specs/tech_stack.md +++ b/docs/specs/tech_stack.md @@ -7,11 +7,11 @@ technology stack for compliant IoT devices. # Core Requirements -* [Google Cloud's MQTT Protocol Bridge](https://cloud.google.com/iot/docs/how-tos/mqtt-bridge). - * This is _not_ the same as a generic MQTT Broker, but it is compatible with standard client-side libraries. - * Other transports (non-Google MQTT, CoAP, etc...) are acceptable with prior approval. - * Connected to a specific Cloud IoT Registry designated for each site-specific project. -* Utilizes the MQTT Topic table listed below. +* MQTT broker or bridge + * Availabile with a local setup using the standard `mosquitto` + * Cloud-based solutions such as [ClearBlade IoT Core](https://www.clearblade.com/iot-core/) + * Anything else that works... (it's OSS so go crazy!) +* Utilizes the MQTT Topic table listed below (can be customized) * JSON encoding following the core schema definition, specifying the semantic structure of the data. # MQTT Topic Suffix Table @@ -23,11 +23,11 @@ technology stack for compliant IoT devices. | pointset | event | pointset | `{topic_prefix}/events/pointset` | pointset.json | | system | event | system | `{topic_prefix}/events/system` | system.json | -For GCP implementations the full topic would be `/devices/{device_id}/{suffix}` +For many implementations the full topic would be `/devices/{device_id}/{suffix}` # Backend Systems -Any backend system (in a GCP project) should adhere to the following guidelines: +Any backend system should adhere to the following guidelines: * All messages to/from the devices should conform to the UDMI schema payloads (pass validation). * All exchanges with the devices should go through a PubSub topic: * The _state_ and _events_ messages are published to a topic configured through the IoT Core registry. @@ -44,19 +44,3 @@ gcloud pubsub topics publish target \ The reason for the redirection of any data through a PubSub topic is so that the Cloud IoT registry, if necessary, can be housed in a different cloud project from the backend applications. - -## Types and Topics - -When using the -[GCP Cloud IoT Core MQTT Bridge](https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#publishing_telemetry_events) -there are multiple ways the specific schema used during validation is chosen. -* All messages have their attributes validated against the `.../attributes.json` schema. These attributes are -automatically defined server-side by the MQTT Client ID and Topic, and are not explicitly included in any message payload. -* A [device event message](https://cloud.google.com/iot/docs/how-tos/mqtt-bridge#publishing_telemetry_events) -is validated against the sub-schema indicated by the MQTT topic `subFolder`. E.g., the MQTT -topic `/devices/{device-id}/events/pointset` will be validated against `.../pointset.json`. -* [Device state messages](https://cloud.google.com/iot/docs/how-tos/config/getting-state#reporting_device_state) -are validated against the `.../state.json` schema on `/devices/{device-id}/state` MQTT topic. -* (There currently is no stream validation of -[device config messages](https://cloud.google.com/iot/docs/how-tos/config/configuring-devices#mqtt), which are sent on the -`/devices/{device-id}/config` topic.) diff --git a/docs/tools/gcloud.md b/docs/tools/gcloud.md index 9db3693fb1..194d2eb50c 100644 --- a/docs/tools/gcloud.md +++ b/docs/tools/gcloud.md @@ -15,7 +15,7 @@ them to work out-of-the-box without a deeper understanding of what's going on! `gcloud --project=$project_id functions logs read udmi_config --sort-by=time_utc --limit=1000` -# Update a device's GCP IoT Core configuration +# Update a device's configuration `bin/reset_config` diff --git a/docs/tools/pubber.md b/docs/tools/pubber.md index 3bc33f28c0..2671e30f11 100644 --- a/docs/tools/pubber.md +++ b/docs/tools/pubber.md @@ -143,19 +143,6 @@ INFO daq.pubber.Pubber - 2022-05-24T15:26:23Z State update: INFO daq.pubber.Pubber - 2022-05-24T15:26:32Z 2022-05-24T15:26:32Z sending test message #0 ``` - -## Cloud Setup - -To use Pubber, there needs to be a cloud-side device entry configured in a GCP project configured to -use [Cloud IoT](https://cloud.google.com/iot/docs/). The -[Creating or Editing a Device](https://cloud.google.com/iot/docs/how-tos/devices#creating_or_editing_a_device) -section of the documentation describe how to create a simple device and key-pair (see next section for -a helper script). You can/should substitute the relevant values in the configuration below for your -specific setup. The relevant bits of configuration are the information in the local/pubber.json -file (see above), and the generated public key (also see above). - -Alternatively, you can use the [registrar tool](registrar.md) to automate device registration. - ## Troubleshooting When using the `messageTrace` option to pubber and running the sequence tester it will include the sequence diff --git a/docs/tools/sequencer.md b/docs/tools/sequencer.md index d4db1b8d85..0a04fe43f9 100644 --- a/docs/tools/sequencer.md +++ b/docs/tools/sequencer.md @@ -11,8 +11,8 @@ A [guide is given for using sequencer for complete device testing.](../guides/de # Sequencer Setup -1. Ensure you have [deployed the necessary cloud functions](../cloud/gcp/udmis.md) to your GCP project -2. Add a new GCP IoT Core registry with a registry ID of `UDMI-REFLECT`. +1. Ensure you have a suitable MQTT broker setup. +2. Add a new registry with a registry ID of `UDMI-REFLECT`. * Use `udmi_reflect` as the Pub/Sub topic for both the **device telemetry and **device state** * This serves as a _reflector_ of the MAIN IoT registry combining all messages published into a single stream. diff --git a/docs/tools/setup.md b/docs/tools/setup.md index 595ba39f7e..15a1b8c13e 100644 --- a/docs/tools/setup.md +++ b/docs/tools/setup.md @@ -2,12 +2,6 @@ # UDMI Tools Setup -## Baseline Requirements - -Most interactions work with [Cloud IoT Core](https://cloud.google.com/iot/docs/) -and [PubSub](https://cloud.google.com/pubsub/docs), so a familiarity with those -topics is assumed throughout the UDMI documentation. - ## Software Prerequisites As a minimum, to run or deploy the included tools, the following software will need diff --git a/docs/tools/sharding.md b/docs/tools/sharding.md index 33589624ad..99aa9ff8b1 100644 --- a/docs/tools/sharding.md +++ b/docs/tools/sharding.md @@ -25,7 +25,7 @@ a system with a registry suffix setting of `_A` would show: ## Setup -- Manual creation of GCP IoT Core registries: +- Manual creation of registries: - Create shadow site_model registry (e.g. registry `ZZ-TRI-FECTA_A`). - Create shadow alternate registry (e.g. registry `ZZ-REDIRECT-NA_A`). - Create shadow reflector device entry (e.g. device `ZZ-TRI-FECTA_A` in the `UDMI-REFLECT` registry). diff --git a/docs/tools/validator.md b/docs/tools/validator.md index add04b3ed7..819ccbcfbb 100644 --- a/docs/tools/validator.md +++ b/docs/tools/validator.md @@ -102,8 +102,7 @@ with the tool itself. These give some direction on how to resolve it (e.g. missing parameters, missing components, authentication errors). Authentication errors are frequently encountered when the user authenticated -onto the gcloud SDK does not have [sufficient permissions](../cloud/gcp/cloud_setup.md) -or the wrong GCP project is set. +onto the gcloud SDK does not have or the wrong GCP project is set. If no error message is given, but there are no results or the expected results are not observed, then the following troubleshooting steps are suggested to @@ -119,10 +118,6 @@ identify the problem. been sent or config messages received - Enable debug logging for the device on IoT Core and check the Stackdriver log for any errors -- Ensure the [cloud functions have been deployed](../cloud/gcp/udmis.md), the cloud functions - are running with a service account with [sufficient permissions](../cloud/gcp/cloud_setup.md) - and the [Pub/Sub subscriptions are configured](../cloud/gcp/cloud_setup.md). For use with the - validator, the subscription should be to the udmi_target topic. - Check the Pub/sub subscription to check messages are being sent to the cloud - Check the payload is as expected @@ -142,6 +137,4 @@ Unknown schema subFolder 'events_unknown' for XXX-1 and not the `udmi_state` topic or any others **Missing messages** _or_ **messages not appearing in validator output** -- Ensure the cloud functions are running with a service account with [sufficient - permissions](../cloud/gcp/cloud_setup.md) - Ensure subscribing to the right topic From 8d3aff7c8c0c21248469a5895adb47de893d7c25 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Thu, 11 Jul 2024 15:33:23 -0700 Subject: [PATCH 02/58] Fix spelling --- .wordlist.txt | 3 ++- docs/specs/tech_stack.md | 2 +- 2 files changed, 3 insertions(+), 2 deletions(-) diff --git a/.wordlist.txt b/.wordlist.txt index c3661f322b..d38c986f66 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -18,8 +18,8 @@ breakpoint buildingsiot centric CLA +ClearBlade CLI -Clearblade CloudRun CoAP comms @@ -99,6 +99,7 @@ nterface OIDC onboarding Onboarding +OSS pagent parameterization parsers diff --git a/docs/specs/tech_stack.md b/docs/specs/tech_stack.md index bee185fa37..869aabc762 100644 --- a/docs/specs/tech_stack.md +++ b/docs/specs/tech_stack.md @@ -8,7 +8,7 @@ technology stack for compliant IoT devices. # Core Requirements * MQTT broker or bridge - * Availabile with a local setup using the standard `mosquitto` + * Available with a local setup using the standard `mosquitto` * Cloud-based solutions such as [ClearBlade IoT Core](https://www.clearblade.com/iot-core/) * Anything else that works... (it's OSS so go crazy!) * Utilizes the MQTT Topic table listed below (can be customized) From 6bcf585ed1dc7e77ed5532b6023c7b52e5e4a479 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Thu, 11 Jul 2024 15:51:36 -0700 Subject: [PATCH 03/58] Spelling fix --- docs/tools/registrar.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/tools/registrar.md b/docs/tools/registrar.md index 718829d561..51e412e0c4 100644 --- a/docs/tools/registrar.md +++ b/docs/tools/registrar.md @@ -47,8 +47,8 @@ bin/registrar config_file * `config_file`: Path to a configuration file which contains configuration options; * `site_path`: The _directory_ containing the site model, or a model-with-project _file_ directly. * `project_id`: The project ID that contains the target registry. The project ID can be prepended with iot_provider: - * `//clearblade/PROJECT_ID` for a public Clearblade project. - * `//gbos/PROJECT_ID` for a Google operated Clearblade project. + * `//clearblade/PROJECT_ID` for a public ClearBlade project. + * `//gbos/PROJECT_ID` for a Google operated ClearBlade project. * `options`: Various options to impact behavior: * `-u` Update. * `-d` Delete all device in the site model from the registry (combine with `-x` to delete all devices from the registry) From e79f07757879d11727fea595759266433f6ed387 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Thu, 11 Jul 2024 21:41:27 -0700 Subject: [PATCH 04/58] Purge images --- .gitignore | 1 + bin/container | 5 +++++ udmis/Dockerfile.udmis | 2 ++ udmis/bin/build | 4 ++++ 4 files changed, 12 insertions(+) diff --git a/.gitignore b/.gitignore index a564f419b4..949b23672b 100644 --- a/.gitignore +++ b/.gitignore @@ -23,6 +23,7 @@ credentials.json /udmis/build/ /udmis/out/ /udmis/var/ +/udmis/sbin/ /udmis/profile/ /udmis/.idea/libraries/ /selfie/build/ diff --git a/bin/container b/bin/container index 37ba0a82ff..41f522ccc2 100755 --- a/bin/container +++ b/bin/container @@ -110,6 +110,11 @@ EOF fi if [[ -n $build ]]; then + echo Cleaning old images... + docker rmi $target:latest || true + images=$(docker images | fgrep $REPOSITORY | awk '{print $3}') || true + [[ -n $images ]] && docker rmi $images + echo Building Dockerfile.$target docker build -f Dockerfile.$target -t $target . docker tag $target $udmi_ref diff --git a/udmis/Dockerfile.udmis b/udmis/Dockerfile.udmis index ca68e1455e..416104fb06 100644 --- a/udmis/Dockerfile.udmis +++ b/udmis/Dockerfile.udmis @@ -11,6 +11,8 @@ ADD build/libs/udmis-1.0-SNAPSHOT-all.jar build/libs/udmis-1.0-SNAPSHOT-all.jar ADD bin/ bin/ +ADD sbin/ bin/ + ADD var/ var/ ENV CLEARBLADE_CONFIGURATION=/udmi/clearblade.json diff --git a/udmis/bin/build b/udmis/bin/build index 71bf8734bd..f36c1d2334 100755 --- a/udmis/bin/build +++ b/udmis/bin/build @@ -19,6 +19,10 @@ ROOT=$(dirname $0)/.. BASE=.. cd $ROOT +echo Mirroring udmi/bin... +rm -rf sbin +cp -a ../bin/ sbin/ + jarfile=build/libs/udmis-1.0-SNAPSHOT-all.jar newer_files=$([[ -f $jarfile ]] && find src/ $BASE/gencode/java -type f -newer $jarfile || echo missing) From 8543fe418dab28a582ac31af419010b25a77251e Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Fri, 12 Jul 2024 05:30:38 -0700 Subject: [PATCH 05/58] Basic containerization --- bin/container | 2 +- bin/start_local | 6 ++++++ etc/shell_common.sh | 6 +++--- udmis/Dockerfile.udmis | 2 +- udmis/bin/build | 5 +++-- 5 files changed, 14 insertions(+), 7 deletions(-) diff --git a/bin/container b/bin/container index 41f522ccc2..77e9b288b2 100755 --- a/bin/container +++ b/bin/container @@ -113,7 +113,7 @@ if [[ -n $build ]]; then echo Cleaning old images... docker rmi $target:latest || true images=$(docker images | fgrep $REPOSITORY | awk '{print $3}') || true - [[ -n $images ]] && docker rmi $images + [[ -n $images ]] && docker rmi -f $images echo Building Dockerfile.$target docker build -f Dockerfile.$target -t $target . diff --git a/bin/start_local b/bin/start_local index 9788f76cbc..4f25a888f9 100755 --- a/bin/start_local +++ b/bin/start_local @@ -3,6 +3,12 @@ UDMI_ROOT=$(dirname $0)/.. cd $UDMI_ROOT +if [[ ! -d etc/ ]]; then + echo Migrating support scripts to etc... + mkdir etc/ + mv var/*.sh etc/ +fi + source $UDMI_ROOT/etc/shell_common.sh [[ $# == 1 ]] || fail Usage: $0 project_spec diff --git a/etc/shell_common.sh b/etc/shell_common.sh index 0cb673886c..61439f9414 100644 --- a/etc/shell_common.sh +++ b/etc/shell_common.sh @@ -34,9 +34,9 @@ UDMI_JAR=$UDMI_ROOT/validator/build/libs/validator-1.0-SNAPSHOT-all.jar udmi_version=$(cd $UDMI_ROOT; git describe --dirty) || true -if [[ -z $udmi_version ]]; then - udmi_version=git-$(cd $UDMI_ROOT; git describe --dirty --always) -fi +[[ -z $udmi_version ]] && udmi_version=git-$(cd $UDMI_ROOT; git describe --dirty --always) || true + +[[ -z $udmi_version ]] && udmi_version=unknown export UDMI_ROOT export UDMI_TOOLS=$udmi_version diff --git a/udmis/Dockerfile.udmis b/udmis/Dockerfile.udmis index 416104fb06..4bb2096535 100644 --- a/udmis/Dockerfile.udmis +++ b/udmis/Dockerfile.udmis @@ -2,7 +2,7 @@ FROM alpine:latest WORKDIR /root -RUN apk add bash openjdk17 gcompat curl jq +RUN apk add bash openjdk17 gcompat curl jq sudo # Workaround for https://github.com/grpc/grpc-java/issues/8751 ENV LD_PRELOAD=/lib/libgcompat.so.0 diff --git a/udmis/bin/build b/udmis/bin/build index f36c1d2334..1bf364f340 100755 --- a/udmis/bin/build +++ b/udmis/bin/build @@ -19,9 +19,10 @@ ROOT=$(dirname $0)/.. BASE=.. cd $ROOT -echo Mirroring udmi/bin... -rm -rf sbin +echo Mirroring bin/ and etc/... +rm -rf sbin/ cp -a ../bin/ sbin/ +cp ../etc/*.sh var/ jarfile=build/libs/udmis-1.0-SNAPSHOT-all.jar From fb04adf73d547343550ec67703ee19e519d5d940 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Fri, 12 Jul 2024 05:37:11 -0700 Subject: [PATCH 06/58] mango docker runner --- contrib/mango/mango_docker | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100755 contrib/mango/mango_docker diff --git a/contrib/mango/mango_docker b/contrib/mango/mango_docker new file mode 100755 index 0000000000..531c3260af --- /dev/null +++ b/contrib/mango/mango_docker @@ -0,0 +1,13 @@ +#!/bin/sh +set -o errexit -o nounset + +MANGO_VERSION=5.1.4 +mkdir -p mango/data && cd mango +curl -sSL https://store.mango-os.com/downloads/m2m2-udmi-${MANGO_VERSION}.zip -o udmi.zip +unzip udmi.zip -d udmi && rm udmi.zip + +docker run --rm \ + -p 8443:8443 \ + -v data:/opt/mango-data \ + -v udmi:/opt/mango/web/modules/udmi \ + ghcr.io/radixiot/mango:${MANGO_VERSION} From 73b9bfce2d49d2d0d05a048a3a5b9f986d27700a Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Fri, 12 Jul 2024 05:49:38 -0700 Subject: [PATCH 07/58] Adding etcd basics --- bin/start_etcd | 14 +++++++------- udmis/Dockerfile.udmis | 10 ++++++---- 2 files changed, 13 insertions(+), 11 deletions(-) diff --git a/bin/start_etcd b/bin/start_etcd index e1485a08a2..0b80c63f33 100755 --- a/bin/start_etcd +++ b/bin/start_etcd @@ -1,13 +1,13 @@ #!/bin/bash -e IMAGE=quay.io/coreos/etcd:v3.5.13 +BINDIR=/tmp/etcd -docker kill etcd || true +#docker run -v /usr/share/ca-certificates/:/etc/ssl/certs \ -docker run -v /usr/share/ca-certificates/:/etc/ssl/certs \ - -p 2379:2379 --rm --name etcd ${IMAGE} etcd --version +udmis/bin/etcdctl version -docker run -d -v /usr/share/ca-certificates/:/etc/ssl/certs \ - -p 2379:2379 --rm --name etcd ${IMAGE} etcd \ - -listen-client-urls=http://0.0.0.0:2379 \ - -advertise-client-urls=http://127.0.0.1:2379 +$BINDIR/etcd -version + +$BINDIR/etcd -listen-client-urls=http://0.0.0.0:2379 \ + -advertise-client-urls=http://127.0.0.1:2379 diff --git a/udmis/Dockerfile.udmis b/udmis/Dockerfile.udmis index 4bb2096535..b7f7546457 100644 --- a/udmis/Dockerfile.udmis +++ b/udmis/Dockerfile.udmis @@ -4,15 +4,17 @@ WORKDIR /root RUN apk add bash openjdk17 gcompat curl jq sudo +ADD bin/ bin/ + +ADD sbin/ bin/ + +RUN bin/etcdctl version + # Workaround for https://github.com/grpc/grpc-java/issues/8751 ENV LD_PRELOAD=/lib/libgcompat.so.0 ADD build/libs/udmis-1.0-SNAPSHOT-all.jar build/libs/udmis-1.0-SNAPSHOT-all.jar -ADD bin/ bin/ - -ADD sbin/ bin/ - ADD var/ var/ ENV CLEARBLADE_CONFIGURATION=/udmi/clearblade.json From e6558fd7030d0ac1a044d886e10848f1e38f234c Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Fri, 12 Jul 2024 07:42:38 -0700 Subject: [PATCH 08/58] Don't hardcode local site model --- .github/workflows/testing.yml | 8 ++++---- bin/start_local | 24 ++++++++++++------------ 2 files changed, 16 insertions(+), 16 deletions(-) diff --git a/.github/workflows/testing.yml b/.github/workflows/testing.yml index 911ce81c7e..9628cce113 100644 --- a/.github/workflows/testing.yml +++ b/.github/workflows/testing.yml @@ -161,7 +161,7 @@ jobs: - name: base setup run: bin/run_tests install_dependencies - name: local setup - run: bin/start_local $TARGET_PROJECT + run: bin/start_local sites/udmi_site_model $TARGET_PROJECT - name: registrar clean run: bin/test_regclean $TARGET_PROJECT - name: sequence tests clean @@ -218,7 +218,7 @@ jobs: - name: base setup run: bin/run_tests install_dependencies - name: local setup - run: bin/start_local $TARGET_PROJECT + run: bin/start_local sites/udmi_site_model $TARGET_PROJECT - name: registrar clean run: bin/test_regclean solo $TARGET_PROJECT - name: telemetry validator @@ -262,7 +262,7 @@ jobs: bin/clone_model bin/registrar sites/udmi_site_model - name: local setup - run: bin/start_local //mqtt/localhost + run: bin/start_local sites/udmi_site_model //mqtt/localhost - name: bin/test_etcd run: bin/test_etcd - name: bin/test_mosquitto @@ -313,7 +313,7 @@ jobs: - name: base setup run: bin/run_tests install_dependencies - name: local setup - run: bin/start_local $TARGET_PROJECT + run: bin/start_local sites/udmi_site_model $TARGET_PROJECT - name: regclean run: bin/test_regclean $TARGET_PROJECT - name: bin/test_proxy diff --git a/bin/start_local b/bin/start_local index 4f25a888f9..5e1c22599a 100755 --- a/bin/start_local +++ b/bin/start_local @@ -1,32 +1,32 @@ #!/bin/bash -e UDMI_ROOT=$(dirname $0)/.. -cd $UDMI_ROOT - -if [[ ! -d etc/ ]]; then - echo Migrating support scripts to etc... - mkdir etc/ - mv var/*.sh etc/ -fi - source $UDMI_ROOT/etc/shell_common.sh -[[ $# == 1 ]] || fail Usage: $0 project_spec +[[ $# == 2 ]] || fail Usage: $0 site_model project_spec -project_spec=$1 +site_model=$(realpath $1) +project_spec=$2 shift echo "export TARGET_PROJECT=${project_spec:-}" echo "export UDMI_REGISTRY_SUFFIX=${UDMI_REGISTRY_SUFFIX:-}" echo "export UDMI_ALT_REGISTRY=${UDMI_ALT_REGISTRY:-}" +cd $UDMI_ROOT + +if [[ ! -d etc/ ]]; then + echo Migrating support scripts to etc... + mkdir etc/ + mv var/*.sh etc/ +fi + if [[ ! $project_spec =~ ^//mqtt/ ]]; then echo Not a local setup, doing nothing! exit 0 fi -site_model=sites/udmi_site_model -site_config=sites/udmi_site_model/cloud_iot_config.json +site_config=$site_model/cloud_iot_config.json registry_id=$(jq -r .registry_id $site_config)${UDMI_REGISTRY_SUFFIX:-} bin/start_etcd From 8a2053d05a9605c7eb2bb99ccb8551033270a670 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Mon, 15 Jul 2024 10:40:16 -0700 Subject: [PATCH 09/58] Progressing on docker udmis --- bin/docker_udmis | 20 ++++++++++++++++++++ bin/start_etcd | 2 +- bin/start_local | 18 +++++++++++------- etc/shell_common.sh | 2 +- udmis/Dockerfile.udmis | 2 +- 5 files changed, 34 insertions(+), 10 deletions(-) create mode 100755 bin/docker_udmis diff --git a/bin/docker_udmis b/bin/docker_udmis new file mode 100755 index 0000000000..8e2953eb7b --- /dev/null +++ b/bin/docker_udmis @@ -0,0 +1,20 @@ +#!/bin/bash -e + +UDMI_ROOT=$(dirname $0)/.. +source $UDMI_ROOT/etc/shell_common.sh + +cd $UDMI_ROOT + +docker rm udmis || true + +echo bin/start_local site/ //mqtt/localhost + +cmd="docker run -d --name udmis \ + -v $PWD/sites/udmi_site_model:/root/site \ + -v $PWD/var/etcd:/root/default.etcd \ + -v $PWD/var/mosquitto:/etc/mosquitto \ + udmis:latest tail -f /dev/null" + +echo exec: $cmd +$cmd + diff --git a/bin/start_etcd b/bin/start_etcd index 0b80c63f33..bc4e6f506c 100755 --- a/bin/start_etcd +++ b/bin/start_etcd @@ -5,7 +5,7 @@ BINDIR=/tmp/etcd #docker run -v /usr/share/ca-certificates/:/etc/ssl/certs \ -udmis/bin/etcdctl version +udmis/bin/etcdctl version || bin/etcdctl version $BINDIR/etcd -version diff --git a/bin/start_local b/bin/start_local index 5e1c22599a..bce17c1433 100755 --- a/bin/start_local +++ b/bin/start_local @@ -1,6 +1,16 @@ #!/bin/bash -e UDMI_ROOT=$(dirname $0)/.. + +if [[ ! -d $UDMI_ROOT/etc/ ]]; then + echo Migrating support scripts to etc... + ( + cd $UDMI_ROOT + mkdir etc/ + mv var/*.sh etc/ + ) +fi + source $UDMI_ROOT/etc/shell_common.sh [[ $# == 2 ]] || fail Usage: $0 site_model project_spec @@ -15,12 +25,6 @@ echo "export UDMI_ALT_REGISTRY=${UDMI_ALT_REGISTRY:-}" cd $UDMI_ROOT -if [[ ! -d etc/ ]]; then - echo Migrating support scripts to etc... - mkdir etc/ - mv var/*.sh etc/ -fi - if [[ ! $project_spec =~ ^//mqtt/ ]]; then echo Not a local setup, doing nothing! exit 0 @@ -29,7 +33,7 @@ fi site_config=$site_model/cloud_iot_config.json registry_id=$(jq -r .registry_id $site_config)${UDMI_REGISTRY_SUFFIX:-} -bin/start_etcd +bin/start_etcd 2>&1 > etcd.log & bin/setup_ca $site_model bin/start_mosquitto diff --git a/etc/shell_common.sh b/etc/shell_common.sh index 61439f9414..68544d4956 100644 --- a/etc/shell_common.sh +++ b/etc/shell_common.sh @@ -36,7 +36,7 @@ udmi_version=$(cd $UDMI_ROOT; git describe --dirty) || true [[ -z $udmi_version ]] && udmi_version=git-$(cd $UDMI_ROOT; git describe --dirty --always) || true -[[ -z $udmi_version ]] && udmi_version=unknown +[[ $udmi_version == git- ]] && udmi_version=unknown export UDMI_ROOT export UDMI_TOOLS=$udmi_version diff --git a/udmis/Dockerfile.udmis b/udmis/Dockerfile.udmis index b7f7546457..d10af1fe86 100644 --- a/udmis/Dockerfile.udmis +++ b/udmis/Dockerfile.udmis @@ -2,7 +2,7 @@ FROM alpine:latest WORKDIR /root -RUN apk add bash openjdk17 gcompat curl jq sudo +RUN apk add bash openjdk17 gcompat curl jq sudo git mosquitto mosquitto-clients ADD bin/ bin/ From 058c528f3d9567956561aba52a80e1529435846b Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Mon, 15 Jul 2024 16:33:19 -0700 Subject: [PATCH 10/58] Iterating on docker container --- bin/container | 6 +++--- bin/docker_udmis | 2 -- bin/start_local | 8 +++++--- bin/start_mosquitto | 11 +++++++++-- udmis/Dockerfile.udmis | 2 +- udmis/bin/build | 4 ++++ 6 files changed, 22 insertions(+), 11 deletions(-) diff --git a/bin/container b/bin/container index 77e9b288b2..fa397d35be 100755 --- a/bin/container +++ b/bin/container @@ -111,9 +111,9 @@ fi if [[ -n $build ]]; then echo Cleaning old images... - docker rmi $target:latest || true - images=$(docker images | fgrep $REPOSITORY | awk '{print $3}') || true - [[ -n $images ]] && docker rmi -f $images + #docker rmi $target:latest || true + #images=$(docker images | fgrep $REPOSITORY | awk '{print $3}') || true + #[[ -n $images ]] && docker rmi -f $images echo Building Dockerfile.$target docker build -f Dockerfile.$target -t $target . diff --git a/bin/docker_udmis b/bin/docker_udmis index 8e2953eb7b..8c38650ac2 100755 --- a/bin/docker_udmis +++ b/bin/docker_udmis @@ -7,8 +7,6 @@ cd $UDMI_ROOT docker rm udmis || true -echo bin/start_local site/ //mqtt/localhost - cmd="docker run -d --name udmis \ -v $PWD/sites/udmi_site_model:/root/site \ -v $PWD/var/etcd:/root/default.etcd \ diff --git a/bin/start_local b/bin/start_local index bce17c1433..26d2cdc964 100755 --- a/bin/start_local +++ b/bin/start_local @@ -7,7 +7,7 @@ if [[ ! -d $UDMI_ROOT/etc/ ]]; then ( cd $UDMI_ROOT mkdir etc/ - mv var/*.sh etc/ + mv var/*.sh var/*.conf etc/ ) fi @@ -33,12 +33,14 @@ fi site_config=$site_model/cloud_iot_config.json registry_id=$(jq -r .registry_id $site_config)${UDMI_REGISTRY_SUFFIX:-} -bin/start_etcd 2>&1 > etcd.log & +bin/start_etcd 2&>1 > etcd.log & + +source $UDMI_ROOT/etc/mosquitto_ctrl.sh +mkdir -p $CERT_DIR bin/setup_ca $site_model bin/start_mosquitto -source $UDMI_ROOT/etc/mosquitto_ctrl.sh $MOSQUITTO_CTRL deleteClient $SERV_USER $MOSQUITTO_CTRL createClient $SERV_USER -p $SERV_PASS # No client_id to allow multiple backend connections. diff --git a/bin/start_mosquitto b/bin/start_mosquitto index 58331284e2..dcaab27f97 100755 --- a/bin/start_mosquitto +++ b/bin/start_mosquitto @@ -1,10 +1,15 @@ -#!/bin/bash -eu +#!/bin/bash -eux UDMI_ROOT=$(dirname $0)/.. source $UDMI_ROOT/etc/shell_common.sh source $UDMI_ROOT/etc/mosquitto_ctrl.sh +# This is not already set-up when running inside of docker... +if [[ ! -f $ETC_DIR/mosquitto.conf ]]; then + cp var/mosquitto.conf $ETC_DIR/ +fi + GROUP=mosquitto UDMI_FILE=$ETC_DIR/conf.d/udmi.conf PASS_FILE=$ETC_DIR/mosquitto.passwd @@ -12,8 +17,10 @@ DYN_FILE=$ETC_DIR/dynamic_security.json if [[ ! -f $UDMI_FILE ]]; then echo Creating new $UDMI_FILE from template... + mkdir -p $(dirname $UDMI_FILE) sudo cp etc/mosquitto_udmi.conf $UDMI_FILE - PLUGIN_FILE=$(whereis -b mosquitto_dynamic_security.so | awk '{print $2}') + PLUGIN_FILE=/usr/lib/mosquitto_dynamic_security.so + [[ -f $PLUGIN_FILE ]] || PLUGIN_FILE=$(whereis -b mosquitto_dynamic_security.so | awk '{print $2}') ls -l "$PLUGIN_FILE" echo Configuring dynamic security plugin $PLUGIN_FILE sudo sed -i "s%plugin dynsec%plugin $PLUGIN_FILE%" $UDMI_FILE diff --git a/udmis/Dockerfile.udmis b/udmis/Dockerfile.udmis index d10af1fe86..cad62dc1a5 100644 --- a/udmis/Dockerfile.udmis +++ b/udmis/Dockerfile.udmis @@ -2,7 +2,7 @@ FROM alpine:latest WORKDIR /root -RUN apk add bash openjdk17 gcompat curl jq sudo git mosquitto mosquitto-clients +RUN apk add bash openjdk17 gcompat curl jq sudo git mosquitto mosquitto-clients openssl ADD bin/ bin/ diff --git a/udmis/bin/build b/udmis/bin/build index 1bf364f340..7013525350 100755 --- a/udmis/bin/build +++ b/udmis/bin/build @@ -23,6 +23,10 @@ echo Mirroring bin/ and etc/... rm -rf sbin/ cp -a ../bin/ sbin/ cp ../etc/*.sh var/ +cp ../etc/*.conf var/ + +echo Copying mosquitto config files... +cp /etc/mosquitto/mosquitto.conf var/ jarfile=build/libs/udmis-1.0-SNAPSHOT-all.jar From e394639ed137ebc5d8a871e0a0669d3c70f53a98 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Mon, 15 Jul 2024 18:37:29 -0700 Subject: [PATCH 11/58] More stuff --- bin/docker_udmis | 2 +- bin/start_local | 5 ++--- bin/start_mosquitto | 19 +++++++++++++++---- bin/start_udmis | 2 +- 4 files changed, 19 insertions(+), 9 deletions(-) diff --git a/bin/docker_udmis b/bin/docker_udmis index 8c38650ac2..29994f77e5 100755 --- a/bin/docker_udmis +++ b/bin/docker_udmis @@ -11,7 +11,7 @@ cmd="docker run -d --name udmis \ -v $PWD/sites/udmi_site_model:/root/site \ -v $PWD/var/etcd:/root/default.etcd \ -v $PWD/var/mosquitto:/etc/mosquitto \ - udmis:latest tail -f /dev/null" + udmis:latest bin/start_mosquitto site/ //mqtt/localhost" echo exec: $cmd $cmd diff --git a/bin/start_local b/bin/start_local index 26d2cdc964..74da17567c 100755 --- a/bin/start_local +++ b/bin/start_local @@ -7,7 +7,7 @@ if [[ ! -d $UDMI_ROOT/etc/ ]]; then ( cd $UDMI_ROOT mkdir etc/ - mv var/*.sh var/*.conf etc/ + cp var/*.sh var/*.conf etc/ ) fi @@ -33,7 +33,7 @@ fi site_config=$site_model/cloud_iot_config.json registry_id=$(jq -r .registry_id $site_config)${UDMI_REGISTRY_SUFFIX:-} -bin/start_etcd 2&>1 > etcd.log & +bin/start_etcd >etcd.log 2>&1 & source $UDMI_ROOT/etc/mosquitto_ctrl.sh mkdir -p $CERT_DIR @@ -41,7 +41,6 @@ mkdir -p $CERT_DIR bin/setup_ca $site_model bin/start_mosquitto - $MOSQUITTO_CTRL deleteClient $SERV_USER $MOSQUITTO_CTRL createClient $SERV_USER -p $SERV_PASS # No client_id to allow multiple backend connections. $MOSQUITTO_CTRL addClientRole $SERV_USER service diff --git a/bin/start_mosquitto b/bin/start_mosquitto index dcaab27f97..1f73391fe8 100755 --- a/bin/start_mosquitto +++ b/bin/start_mosquitto @@ -1,4 +1,4 @@ -#!/bin/bash -eux +#!/bin/bash -eu UDMI_ROOT=$(dirname $0)/.. source $UDMI_ROOT/etc/shell_common.sh @@ -15,6 +15,8 @@ UDMI_FILE=$ETC_DIR/conf.d/udmi.conf PASS_FILE=$ETC_DIR/mosquitto.passwd DYN_FILE=$ETC_DIR/dynamic_security.json +mkdir -p /var/log/mosquitto/ + if [[ ! -f $UDMI_FILE ]]; then echo Creating new $UDMI_FILE from template... mkdir -p $(dirname $UDMI_FILE) @@ -25,13 +27,14 @@ if [[ ! -f $UDMI_FILE ]]; then echo Configuring dynamic security plugin $PLUGIN_FILE sudo sed -i "s%plugin dynsec%plugin $PLUGIN_FILE%" $UDMI_FILE sudo sed -i "s%plugin_opt_config_file dynsec%plugin_opt_config_file $DYN_FILE%" $UDMI_FILE + [[ $(whoami) == root ]] && echo user root >> $UDMI_FILE fi if [[ ! -f $DYN_FILE ]]; then echo Creating new $DYN_FILE echo Configuring MQTT user: $AUTH_USER sudo mosquitto_ctrl dynsec init $DYN_FILE $AUTH_USER $AUTH_PASS - sudo chgrp $GROUP $DYN_FILE + [[ $(whoami) != root ]] && sudo chgrp $GROUP $DYN_FILE sudo chmod 0660 $DYN_FILE fi @@ -39,11 +42,19 @@ if [[ ! -f $PASS_FILE ]]; then echo Creating $PASS_FILE sudo touch $PASS_FILE sudo chmod 0640 $PASS_FILE - sudo chgrp $GROUP $PASS_FILE + [[ $(whoami) != root ]] && sudo chgrp $GROUP $PASS_FILE sudo mosquitto_passwd -b ${PASS_FILE} ${AUTH_USER} ${AUTH_PASS} fi -sudo systemctl restart mosquitto +if [[ -n $(which systemctl) ]]; then + sudo systemctl restart mosquitto +else + echo Starting mosquitto in the background... + mosquitto -c /etc/mosquitto/mosquitto.conf > mosquitto.log 2>&1 & + MOSQUITTO_PID=$! + sleep 10 + [[ -d /proc/$MOSQUITTO_PID ]] || (cat mosquitto.log && fail starting mosquitto) +fi $MOSQUITTO_CTRL createRole device $MOSQUITTO_CTRL addRoleACL device subscribePattern '/r/+/d/+/#' allow diff --git a/bin/start_udmis b/bin/start_udmis index 8d09bd34fd..ce37b1879f 100755 --- a/bin/start_udmis +++ b/bin/start_udmis @@ -1,4 +1,4 @@ -#!/bin/bash -e +#!/bin/bash -ex UDMI_ROOT=$(dirname $0)/.. source $UDMI_ROOT/etc/shell_common.sh From cd012a714193bc264a80670b9f88d15f5c14e850 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Mon, 15 Jul 2024 22:24:30 -0700 Subject: [PATCH 12/58] Working local dockerfile --- .gitignore | 2 ++ bin/docker_udmis | 4 +++- bin/start_etcd | 4 +++- bin/start_local | 17 ++++++----------- bin/start_udmis | 11 ++++++++--- udmis/Dockerfile.udmis | 4 +++- udmis/bin/build | 11 ++++++----- 7 files changed, 31 insertions(+), 22 deletions(-) diff --git a/.gitignore b/.gitignore index 949b23672b..2e1a49e497 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,8 @@ credentials.json /udmis/sbin/ /udmis/profile/ /udmis/.idea/libraries/ +/udmis/udmi_bin/ +/udmis/udmi_etc/ /selfie/build/ /pubber/build/ /pubber/out/ diff --git a/bin/docker_udmis b/bin/docker_udmis index 29994f77e5..efe2e7339c 100755 --- a/bin/docker_udmis +++ b/bin/docker_udmis @@ -11,7 +11,9 @@ cmd="docker run -d --name udmis \ -v $PWD/sites/udmi_site_model:/root/site \ -v $PWD/var/etcd:/root/default.etcd \ -v $PWD/var/mosquitto:/etc/mosquitto \ - udmis:latest bin/start_mosquitto site/ //mqtt/localhost" + udmis:latest tail -f /dev/null" + +#bin/start_local site/ //mqtt/localhost" echo exec: $cmd $cmd diff --git a/bin/start_etcd b/bin/start_etcd index bc4e6f506c..4cc8acb1ff 100755 --- a/bin/start_etcd +++ b/bin/start_etcd @@ -5,7 +5,9 @@ BINDIR=/tmp/etcd #docker run -v /usr/share/ca-certificates/:/etc/ssl/certs \ -udmis/bin/etcdctl version || bin/etcdctl version +echo pwd: $(pwd) + +udmis/bin/etcdctl version || ../bin/etcdctl version $BINDIR/etcd -version diff --git a/bin/start_local b/bin/start_local index 74da17567c..b14acac906 100755 --- a/bin/start_local +++ b/bin/start_local @@ -1,16 +1,6 @@ #!/bin/bash -e UDMI_ROOT=$(dirname $0)/.. - -if [[ ! -d $UDMI_ROOT/etc/ ]]; then - echo Migrating support scripts to etc... - ( - cd $UDMI_ROOT - mkdir etc/ - cp var/*.sh var/*.conf etc/ - ) -fi - source $UDMI_ROOT/etc/shell_common.sh [[ $# == 2 ]] || fail Usage: $0 site_model project_spec @@ -23,6 +13,11 @@ echo "export TARGET_PROJECT=${project_spec:-}" echo "export UDMI_REGISTRY_SUFFIX=${UDMI_REGISTRY_SUFFIX:-}" echo "export UDMI_ALT_REGISTRY=${UDMI_ALT_REGISTRY:-}" +# Check for missing etc/ dir for docker execution. +if [[ -d var/ && ! -d etc/ ]]; then + ln -s var etc +fi + cd $UDMI_ROOT if [[ ! $project_spec =~ ^//mqtt/ ]]; then @@ -33,7 +28,7 @@ fi site_config=$site_model/cloud_iot_config.json registry_id=$(jq -r .registry_id $site_config)${UDMI_REGISTRY_SUFFIX:-} -bin/start_etcd >etcd.log 2>&1 & +bin/start_etcd > etcd.log 2>&1 & source $UDMI_ROOT/etc/mosquitto_ctrl.sh mkdir -p $CERT_DIR diff --git a/bin/start_udmis b/bin/start_udmis index ce37b1879f..2f5fe60c99 100755 --- a/bin/start_udmis +++ b/bin/start_udmis @@ -20,20 +20,25 @@ if [[ ${1:-} == stop ]]; then exit 0 fi -bin/container udmis prep --no-check +bin/container udmis prep --no-check || true +mkdir -p out LOGFILE=out/udmis.log +date > $LOGFILE export ETCD_CLUSTER=localhost export SSL_SECRETS_DIR=/etc/mosquitto/certs -sudo PATH=$PATH -E udmis/bin/run udmis/etc/local_pod.json > $LOGFILE 2>&1 & +UDMIS_DIR=udmis +[[ -d $UDMIS_DIR ]] || UDMIS_DIR=.. + +sudo PATH=$PATH -E $UDMIS_DIR/bin/run $UDMIS_DIR/etc/local_pod.json > $LOGFILE 2>&1 & PID=$! WAITING=30 for i in `seq 1 $WAITING`; do - if [[ -f $POD_READY ]]; then + if [[ -f $POD_READY || ! -d /proc/$PID ]]; then break fi echo Waiting for udmis startup $((WAITING - i))... diff --git a/udmis/Dockerfile.udmis b/udmis/Dockerfile.udmis index cad62dc1a5..6a61524309 100644 --- a/udmis/Dockerfile.udmis +++ b/udmis/Dockerfile.udmis @@ -6,7 +6,7 @@ RUN apk add bash openjdk17 gcompat curl jq sudo git mosquitto mosquitto-clients ADD bin/ bin/ -ADD sbin/ bin/ +ADD udmi_bin/ udmi/bin/ RUN bin/etcdctl version @@ -17,6 +17,8 @@ ADD build/libs/udmis-1.0-SNAPSHOT-all.jar build/libs/udmis-1.0-SNAPSHOT-all.jar ADD var/ var/ +ADD udmi_etc/ udmi/etc/ + ENV CLEARBLADE_CONFIGURATION=/udmi/clearblade.json ENV SHUNT_CONFIGURATION=/udmi/shunt-config.json diff --git a/udmis/bin/build b/udmis/bin/build index 7013525350..ffcd7ca86f 100755 --- a/udmis/bin/build +++ b/udmis/bin/build @@ -19,11 +19,12 @@ ROOT=$(dirname $0)/.. BASE=.. cd $ROOT -echo Mirroring bin/ and etc/... -rm -rf sbin/ -cp -a ../bin/ sbin/ -cp ../etc/*.sh var/ -cp ../etc/*.conf var/ +echo Mirroring udim/bin to udmi_bin... +rm -rf udmi_bin/ +cp -a ../bin/ udmi_bin/ +echo Mirroring udmi/etc to udmi_etc... +rm -rf udmi_etc/ +cp -a ../etc udmi_etc/ echo Copying mosquitto config files... cp /etc/mosquitto/mosquitto.conf var/ From 273374e0f3eceaa1d2bae7dad6c6bcf2ad249c73 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 06:49:58 -0700 Subject: [PATCH 13/58] Cleanup debugging --- bin/docker_udmis | 5 +---- bin/mosquctl_log | 2 +- bin/setup_chromedriver | 2 +- bin/start_udmis | 2 +- 4 files changed, 4 insertions(+), 7 deletions(-) diff --git a/bin/docker_udmis b/bin/docker_udmis index efe2e7339c..506e27af26 100755 --- a/bin/docker_udmis +++ b/bin/docker_udmis @@ -11,10 +11,7 @@ cmd="docker run -d --name udmis \ -v $PWD/sites/udmi_site_model:/root/site \ -v $PWD/var/etcd:/root/default.etcd \ -v $PWD/var/mosquitto:/etc/mosquitto \ - udmis:latest tail -f /dev/null" - -#bin/start_local site/ //mqtt/localhost" + udmis:latest udmi/bin/start_local site/ //mqtt/localhost" echo exec: $cmd $cmd - diff --git a/bin/mosquctl_log b/bin/mosquctl_log index 7ae6a0dd90..7686e16b49 100755 --- a/bin/mosquctl_log +++ b/bin/mosquctl_log @@ -1,4 +1,4 @@ -#!/bin/bash -ex +#!/bin/bash -e UDMI_ROOT=$(dirname $0)/.. cd $UDMI_ROOT diff --git a/bin/setup_chromedriver b/bin/setup_chromedriver index 326f5523b5..9d530d41de 100755 --- a/bin/setup_chromedriver +++ b/bin/setup_chromedriver @@ -1,4 +1,4 @@ -#!/bin/bash -ex +#!/bin/bash -e echo "deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main" | sudo tee -a /etc/apt/sources.list.d/google.list diff --git a/bin/start_udmis b/bin/start_udmis index 2f5fe60c99..f0a453f074 100755 --- a/bin/start_udmis +++ b/bin/start_udmis @@ -1,4 +1,4 @@ -#!/bin/bash -ex +#!/bin/bash -e UDMI_ROOT=$(dirname $0)/.. source $UDMI_ROOT/etc/shell_common.sh From d926744ab577573d5b51b9a8a1094c048b88aa60 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 08:19:15 -0700 Subject: [PATCH 14/58] Working local docker startup --- bin/container | 5 ++--- bin/docker_udmis | 15 ++++++++++++--- bin/start_etcd | 10 +++++++++- bin/start_local | 4 +++- bin/start_mosquitto | 5 ++--- 5 files changed, 28 insertions(+), 11 deletions(-) diff --git a/bin/container b/bin/container index fa397d35be..c226c9d8bb 100755 --- a/bin/container +++ b/bin/container @@ -111,9 +111,8 @@ fi if [[ -n $build ]]; then echo Cleaning old images... - #docker rmi $target:latest || true - #images=$(docker images | fgrep $REPOSITORY | awk '{print $3}') || true - #[[ -n $images ]] && docker rmi -f $images + images=$(docker images | fgrep $REPOSITORY | awk '{print $1":"$2}') || true + [[ -n $images ]] && docker rmi $images echo Building Dockerfile.$target docker build -f Dockerfile.$target -t $target . diff --git a/bin/docker_udmis b/bin/docker_udmis index 506e27af26..f9cafcfe58 100755 --- a/bin/docker_udmis +++ b/bin/docker_udmis @@ -3,15 +3,24 @@ UDMI_ROOT=$(dirname $0)/.. source $UDMI_ROOT/etc/shell_common.sh +[[ $# == 1 ]] || usage site_model + +site_model=$(realpath $1) +shift + cd $UDMI_ROOT docker rm udmis || true -cmd="docker run -d --name udmis \ - -v $PWD/sites/udmi_site_model:/root/site \ +cmd="docker run -d --name udmis -p 8883:8883 \ + -v $site_model:/root/site \ -v $PWD/var/etcd:/root/default.etcd \ -v $PWD/var/mosquitto:/etc/mosquitto \ - udmis:latest udmi/bin/start_local site/ //mqtt/localhost" + udmis:latest udmi/bin/start_local site //mqtt/localhost" echo exec: $cmd $cmd + +echo + +docker ps diff --git a/bin/start_etcd b/bin/start_etcd index 4cc8acb1ff..4da9bb41e6 100755 --- a/bin/start_etcd +++ b/bin/start_etcd @@ -12,4 +12,12 @@ udmis/bin/etcdctl version || ../bin/etcdctl version $BINDIR/etcd -version $BINDIR/etcd -listen-client-urls=http://0.0.0.0:2379 \ - -advertise-client-urls=http://127.0.0.1:2379 + -advertise-client-urls=http://127.0.0.1:2379 \ + > etcd.log 2>&1 & +ETCD_PID=$! + +echo Waiting 10s for etcd to start... +sleep 10 +[[ -d /proc/$ETCD_PID ]] || (cat etcd.log && fail starting etcd) + +echo Completed etcd startup. diff --git a/bin/start_local b/bin/start_local index b14acac906..41fa7141c7 100755 --- a/bin/start_local +++ b/bin/start_local @@ -28,7 +28,7 @@ fi site_config=$site_model/cloud_iot_config.json registry_id=$(jq -r .registry_id $site_config)${UDMI_REGISTRY_SUFFIX:-} -bin/start_etcd > etcd.log 2>&1 & +bin/start_etcd source $UDMI_ROOT/etc/mosquitto_ctrl.sh mkdir -p $CERT_DIR @@ -52,3 +52,5 @@ sudo chmod a+r /var/log/mosquitto/mosquitto.log bin/start_udmis echo Done with local server setup. +echo Blocking until termination. +tail -f /dev/null diff --git a/bin/start_mosquitto b/bin/start_mosquitto index 1f73391fe8..b5cff60e73 100755 --- a/bin/start_mosquitto +++ b/bin/start_mosquitto @@ -49,11 +49,12 @@ fi if [[ -n $(which systemctl) ]]; then sudo systemctl restart mosquitto else - echo Starting mosquitto in the background... mosquitto -c /etc/mosquitto/mosquitto.conf > mosquitto.log 2>&1 & MOSQUITTO_PID=$! + echo Waiting 10s for background mosquitto to start... sleep 10 [[ -d /proc/$MOSQUITTO_PID ]] || (cat mosquitto.log && fail starting mosquitto) + echo Completed mosquitto startup. fi $MOSQUITTO_CTRL createRole device @@ -69,5 +70,3 @@ if [[ $clients =~ ${AUTH_USER} ]]; then else fail Improper client: $clients fi - -echo use: systemctl status mosquitto From ec5b4f57396e9cf97ba3c60a534e9e1bcbb79519 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 10:23:01 -0700 Subject: [PATCH 15/58] Fix udmis build --- udmis/bin/build | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/udmis/bin/build b/udmis/bin/build index ffcd7ca86f..8f7fbd910e 100755 --- a/udmis/bin/build +++ b/udmis/bin/build @@ -26,7 +26,8 @@ echo Mirroring udmi/etc to udmi_etc... rm -rf udmi_etc/ cp -a ../etc udmi_etc/ -echo Copying mosquitto config files... +echo Copying default mosquitto config files... +mkdir -p var/ cp /etc/mosquitto/mosquitto.conf var/ jarfile=build/libs/udmis-1.0-SNAPSHOT-all.jar From 73dae45ff387eafa70d68e6505d822dd13110ab2 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 10:23:46 -0700 Subject: [PATCH 16/58] Conditional copy --- udmis/bin/build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/udmis/bin/build b/udmis/bin/build index 8f7fbd910e..7c090f6131 100755 --- a/udmis/bin/build +++ b/udmis/bin/build @@ -28,7 +28,7 @@ cp -a ../etc udmi_etc/ echo Copying default mosquitto config files... mkdir -p var/ -cp /etc/mosquitto/mosquitto.conf var/ +cp /etc/mosquitto/mosquitto.conf var/ || true jarfile=build/libs/udmis-1.0-SNAPSHOT-all.jar From 7ef04c9773aeb689df996f6dad4f8b87d5976fdc Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 10:33:48 -0700 Subject: [PATCH 17/58] Fix startup stuff --- bin/docker_udmis | 13 +++++++++++-- bin/start_local | 11 ++++++++--- bin/start_udmis | 1 + 3 files changed, 20 insertions(+), 5 deletions(-) diff --git a/bin/docker_udmis b/bin/docker_udmis index f9cafcfe58..61d0796724 100755 --- a/bin/docker_udmis +++ b/bin/docker_udmis @@ -12,15 +12,24 @@ cd $UDMI_ROOT docker rm udmis || true +[[ -f $site_model/cloud_iot_config.json ]] || fail missing $site_model/cloud_iot_config.json + cmd="docker run -d --name udmis -p 8883:8883 \ -v $site_model:/root/site \ -v $PWD/var/etcd:/root/default.etcd \ -v $PWD/var/mosquitto:/etc/mosquitto \ - udmis:latest udmi/bin/start_local site //mqtt/localhost" + udmis:latest udmi/bin/start_local block site/ //mqtt/localhost" echo exec: $cmd $cmd -echo +echo Waiting 30s for container startup... +sleep 30 + +echo :::::::::::::: docker logs udmis +docker logs udmis +echo :::::::::::::: docker ps docker ps + +echo :::::::::::::: diff --git a/bin/start_local b/bin/start_local index 41fa7141c7..dc0f28c83d 100755 --- a/bin/start_local +++ b/bin/start_local @@ -3,7 +3,12 @@ UDMI_ROOT=$(dirname $0)/.. source $UDMI_ROOT/etc/shell_common.sh -[[ $# == 2 ]] || fail Usage: $0 site_model project_spec +if [[ ${1:-} == block ]]; then + block=$1 + shift +fi + +[[ $# == 2 ]] || usage [block] site_model project_spec site_model=$(realpath $1) project_spec=$2 @@ -52,5 +57,5 @@ sudo chmod a+r /var/log/mosquitto/mosquitto.log bin/start_udmis echo Done with local server setup. -echo Blocking until termination. -tail -f /dev/null + +[[ -n $block ]] && echo Blocking until termination. && tail -f /dev/null diff --git a/bin/start_udmis b/bin/start_udmis index f0a453f074..295ab2a125 100755 --- a/bin/start_udmis +++ b/bin/start_udmis @@ -45,6 +45,7 @@ for i in `seq 1 $WAITING`; do sleep 1 done +echo ::::::::: tail $LOGFILE tail -n 30 $LOGFILE [[ -f $POD_READY ]] || fail pod_ready.txt not found. From af68e2662f7e2bb055b94e583aa74cc2ffa580cd Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 11:00:45 -0700 Subject: [PATCH 18/58] Fix startup --- bin/docker_udmis | 5 ++++- bin/start_mosquitto | 6 ++++-- docs/udmis/local_docker.md | 22 ++++++++++++++++++++++ docs/udmis/readme.md | 2 +- 4 files changed, 31 insertions(+), 4 deletions(-) create mode 100644 docs/udmis/local_docker.md diff --git a/bin/docker_udmis b/bin/docker_udmis index 61d0796724..a142b5412c 100755 --- a/bin/docker_udmis +++ b/bin/docker_udmis @@ -14,11 +14,14 @@ docker rm udmis || true [[ -f $site_model/cloud_iot_config.json ]] || fail missing $site_model/cloud_iot_config.json +IMAGE_TAG=ghcr.io/faucetsdn/udmi:latest +docker inspect -f '{{.Id}}' --type=image udmis:latest && IMAGE_TAG=udmis:latest + cmd="docker run -d --name udmis -p 8883:8883 \ -v $site_model:/root/site \ -v $PWD/var/etcd:/root/default.etcd \ -v $PWD/var/mosquitto:/etc/mosquitto \ - udmis:latest udmi/bin/start_local block site/ //mqtt/localhost" + $IMAGE_TAG udmi/bin/start_local block site/ //mqtt/localhost" echo exec: $cmd $cmd diff --git a/bin/start_mosquitto b/bin/start_mosquitto index b5cff60e73..9a4de3142e 100755 --- a/bin/start_mosquitto +++ b/bin/start_mosquitto @@ -7,7 +7,8 @@ source $UDMI_ROOT/etc/mosquitto_ctrl.sh # This is not already set-up when running inside of docker... if [[ ! -f $ETC_DIR/mosquitto.conf ]]; then - cp var/mosquitto.conf $ETC_DIR/ + echo Installing base mosquitto.conf + cp ../var/mosquitto.conf $ETC_DIR/ fi GROUP=mosquitto @@ -49,7 +50,8 @@ fi if [[ -n $(which systemctl) ]]; then sudo systemctl restart mosquitto else - mosquitto -c /etc/mosquitto/mosquitto.conf > mosquitto.log 2>&1 & + # Raw mode for running (e.g.) in a docker container + mosquitto -c $ETC_DIR/mosquitto.conf > mosquitto.log 2>&1 & MOSQUITTO_PID=$! echo Waiting 10s for background mosquitto to start... sleep 10 diff --git a/docs/udmis/local_docker.md b/docs/udmis/local_docker.md new file mode 100644 index 0000000000..878581586c --- /dev/null +++ b/docs/udmis/local_docker.md @@ -0,0 +1,22 @@ +[**UDMI**](../../) / [**Docs**](../) / [UDMIS](.) / [Local Docker](#) + +# Local docker UDMIS setup and execution + +* Identify a site model and parameters + * For default reference udmi site model: + * `bin/clone_model` + * `site_model=sites/udmi_site_model` + * `device_id=AHU-1` + * `serial_no=21874812` +* docker run `bin/docker_udmis ${site_model}` +* db files `ls -l var/` +* registrar `bin/registrar ${site_model} //mqtt/localhost` +* pubber `bin/pubber ${site_model} //mqtt/localhost ${device_id}` + +# Container build + +General notes on how to build/push the upstream docker image. + +* `bin/container udmis push` +* `docker tag udmis:latest ghcr.io/faucetsdn/udmi:latest` +* `docker push ghcr.io/faucetsdn/udmi:latest` diff --git a/docs/udmis/readme.md b/docs/udmis/readme.md index 35a42c9c4c..fb7a575a9b 100644 --- a/docs/udmis/readme.md +++ b/docs/udmis/readme.md @@ -1,4 +1,4 @@ [**UDMI**](../../) / [**Docs**](../) / [UDMIS](#) * [Class Overview](class_overview.md) - +* [Local docker](local_docker.md) From c6a17916374f0e928a39a57f42c523fb553acbdc Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 11:03:09 -0700 Subject: [PATCH 19/58] Docs update --- docs/udmis/local_docker.md | 12 ++++++++---- 1 file changed, 8 insertions(+), 4 deletions(-) diff --git a/docs/udmis/local_docker.md b/docs/udmis/local_docker.md index 878581586c..2d75b7306d 100644 --- a/docs/udmis/local_docker.md +++ b/docs/udmis/local_docker.md @@ -8,10 +8,14 @@ * `site_model=sites/udmi_site_model` * `device_id=AHU-1` * `serial_no=21874812` -* docker run `bin/docker_udmis ${site_model}` -* db files `ls -l var/` -* registrar `bin/registrar ${site_model} //mqtt/localhost` -* pubber `bin/pubber ${site_model} //mqtt/localhost ${device_id}` +* Run the standard docker image: + * `bin/docker_udmis ${site_model}` +* Persistant DB files are mapped locally + * `ls -l var/etcd var/mosquitto` +* Register your site with the server + * `bin/registrar ${site_model} //mqtt/localhost` +* Optionally test with pubber instance + * `bin/pubber ${site_model} //mqtt/localhost ${device_id} ${serial_no}` # Container build From 14e6a534342171cba99708214f0004e565e1cbb6 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 11:15:49 -0700 Subject: [PATCH 20/58] Cleanup --- bin/start_etcd | 2 -- 1 file changed, 2 deletions(-) diff --git a/bin/start_etcd b/bin/start_etcd index 4da9bb41e6..055c37dc36 100755 --- a/bin/start_etcd +++ b/bin/start_etcd @@ -3,8 +3,6 @@ IMAGE=quay.io/coreos/etcd:v3.5.13 BINDIR=/tmp/etcd -#docker run -v /usr/share/ca-certificates/:/etc/ssl/certs \ - echo pwd: $(pwd) udmis/bin/etcdctl version || ../bin/etcdctl version From 1e98bfd8fc3377693ecfacfee9915bcd1ebb6bc1 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 13:51:59 -0700 Subject: [PATCH 21/58] Fix cert check for pubber --- .../java/com/google/udmi/util/CertManager.java | 16 ++++++++++++---- docs/udmis/local_docker.md | 1 + 2 files changed, 13 insertions(+), 4 deletions(-) diff --git a/common/src/main/java/com/google/udmi/util/CertManager.java b/common/src/main/java/com/google/udmi/util/CertManager.java index b692e606ba..92737dcae8 100644 --- a/common/src/main/java/com/google/udmi/util/CertManager.java +++ b/common/src/main/java/com/google/udmi/util/CertManager.java @@ -1,5 +1,6 @@ package com.google.udmi.util; +import static com.google.common.base.Preconditions.checkState; import static com.google.udmi.util.GeneralUtils.sha256; import static java.lang.String.format; @@ -34,6 +35,7 @@ public class CertManager { public static final String TLS_1_2_PROTOCOL = "TLSv1.2"; + public static final String CA_CERT_FILE = "ca.crt"; private static final String BOUNCY_CASTLE_PROVIDER = "BC"; private static final String X509_FACTORY = "X.509"; private static final String X509_ALGORITHM = "X509"; @@ -42,7 +44,6 @@ public class CertManager { private static final String CA_CERT_ALIAS = "ca-certificate"; private static final String CLIENT_CERT_ALIAS = "certificate"; private static final String PRIVATE_KEY_ALIAS = "private-key"; - public static final String CA_CERT_FILE = "ca.crt"; private final File caCrtFile; private final File keyFile; private final File crtFile; @@ -62,8 +63,7 @@ public CertManager(File caCrtFile, File clientDir, Transport transport, isSsl = Transport.SSL.equals(transport); if (isSsl) { - File rsaCrtFile = new File(clientDir, "rsa_private.crt"); - String prefix = rsaCrtFile.exists() ? "rsa" : "ec"; + String prefix = keyPrefix(clientDir); crtFile = new File(clientDir, prefix + "_private.crt"); keyFile = new File(clientDir, prefix + "_private.pem"); this.password = passString.toCharArray(); @@ -78,6 +78,14 @@ public CertManager(File caCrtFile, File clientDir, Transport transport, } } + private String keyPrefix(File clientDir) { + File rsaCrtFile = new File(clientDir, "rsa_private.crt"); + File ecCrtFile = new File(clientDir, "ec_private.crt"); + checkState(rsaCrtFile.exists() || ecCrtFile.exists(), + "no .crt found for device in " + clientDir.getAbsolutePath()); + return rsaCrtFile.exists() ? "rsa" : "ec"; + } + /** * Get a certificate-backed socket factory. */ @@ -120,7 +128,7 @@ public SSLSocketFactory getCertSocketFactory() throws Exception { clientKeyStore.load(null, null); clientKeyStore.setCertificateEntry(CLIENT_CERT_ALIAS, clientCert); clientKeyStore.setKeyEntry(PRIVATE_KEY_ALIAS, privateKey, password, - new java.security.cert.Certificate[] {clientCert}); + new java.security.cert.Certificate[]{clientCert}); KeyManagerFactory keyManagerFactory = KeyManagerFactory.getInstance( KeyManagerFactory.getDefaultAlgorithm()); keyManagerFactory.init(clientKeyStore, password); diff --git a/docs/udmis/local_docker.md b/docs/udmis/local_docker.md index 2d75b7306d..3c567fdc96 100644 --- a/docs/udmis/local_docker.md +++ b/docs/udmis/local_docker.md @@ -15,6 +15,7 @@ * Register your site with the server * `bin/registrar ${site_model} //mqtt/localhost` * Optionally test with pubber instance + * `sudo bin/keygen CERT ${site_model}/devices/${device_id}/` * `bin/pubber ${site_model} //mqtt/localhost ${device_id} ${serial_no}` # Container build From f4f48945d43a780a46d119a2342fa65bf8cbc1fb Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 14:45:24 -0700 Subject: [PATCH 22/58] FIx startup conditions --- bin/start_local | 1 + docs/udmis/local_docker.md | 2 +- 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/bin/start_local b/bin/start_local index dc0f28c83d..6c96a0b732 100755 --- a/bin/start_local +++ b/bin/start_local @@ -3,6 +3,7 @@ UDMI_ROOT=$(dirname $0)/.. source $UDMI_ROOT/etc/shell_common.sh +block= if [[ ${1:-} == block ]]; then block=$1 shift diff --git a/docs/udmis/local_docker.md b/docs/udmis/local_docker.md index 3c567fdc96..c1c858ad7a 100644 --- a/docs/udmis/local_docker.md +++ b/docs/udmis/local_docker.md @@ -10,7 +10,7 @@ * `serial_no=21874812` * Run the standard docker image: * `bin/docker_udmis ${site_model}` -* Persistant DB files are mapped locally +* Persistent DB files are mapped locally * `ls -l var/etcd var/mosquitto` * Register your site with the server * `bin/registrar ${site_model} //mqtt/localhost` From c268ebfcc3cb1f18fb94423c29676ceffddfc6fb Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 14:50:05 -0700 Subject: [PATCH 23/58] Fix local block logic --- bin/start_local | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/bin/start_local b/bin/start_local index 6c96a0b732..03bf8988b1 100755 --- a/bin/start_local +++ b/bin/start_local @@ -59,4 +59,4 @@ bin/start_udmis echo Done with local server setup. -[[ -n $block ]] && echo Blocking until termination. && tail -f /dev/null +[[ -z $block ]] || (echo Blocking until termination. && tail -f /dev/null) From db5046a522bd05aff28478ff92530e143746e394 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 14:51:53 -0700 Subject: [PATCH 24/58] Remove udmis/sbin --- .gitignore | 1 - 1 file changed, 1 deletion(-) diff --git a/.gitignore b/.gitignore index 2e1a49e497..0753e3b6d4 100644 --- a/.gitignore +++ b/.gitignore @@ -23,7 +23,6 @@ credentials.json /udmis/build/ /udmis/out/ /udmis/var/ -/udmis/sbin/ /udmis/profile/ /udmis/.idea/libraries/ /udmis/udmi_bin/ From 3e5d412de1a63db6fc3cdb52bb0895e50b2f83bc Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Tue, 16 Jul 2024 18:24:39 -0700 Subject: [PATCH 25/58] Don't validate validation events --- etc/validator.out | 68 ------------------- .../google/daq/mqtt/validator/Validator.java | 13 +++- 2 files changed, 10 insertions(+), 71 deletions(-) diff --git a/etc/validator.out b/etc/validator.out index 903ec69b12..aed41c5c98 100644 --- a/etc/validator.out +++ b/etc/validator.out @@ -1,7 +1,6 @@ sites/udmi_site_model/out/devices/AHU-1/config.json sites/udmi_site_model/out/devices/AHU-1/events_pointset.json sites/udmi_site_model/out/devices/AHU-1/events_system.json -sites/udmi_site_model/out/devices/AHU-1/events_validation.json sites/udmi_site_model/out/devices/AHU-1/persistent_data.json sites/udmi_site_model/out/devices/AHU-1/state.json sites/udmi_site_model/out/devices/AHU-1/state_localnet.json @@ -9,7 +8,6 @@ sites/udmi_site_model/out/devices/AHU-1/state_pointset.json sites/udmi_site_model/out/devices/AHU-1/state_system.json sites/udmi_site_model/out/devices/AHU-22/events_pointset.json sites/udmi_site_model/out/devices/AHU-22/events_system.json -sites/udmi_site_model/out/devices/AHU-22/events_validation.json sites/udmi_site_model/out/devices/GAT-123/config.json sites/udmi_site_model/out/devices/GAT-123/events_blobset.json sites/udmi_site_model/out/devices/GAT-123/events_discovery.json @@ -17,7 +15,6 @@ sites/udmi_site_model/out/devices/GAT-123/events_error.json sites/udmi_site_model/out/devices/GAT-123/events_invalid.json sites/udmi_site_model/out/devices/GAT-123/events_pointset.json sites/udmi_site_model/out/devices/GAT-123/events_system.json -sites/udmi_site_model/out/devices/GAT-123/events_validation.json sites/udmi_site_model/out/devices/GAT-123/persistent_data.json sites/udmi_site_model/out/devices/GAT-123/state.json sites/udmi_site_model/out/devices/GAT-123/state_gateway.json @@ -26,7 +23,6 @@ sites/udmi_site_model/out/devices/GAT-123/state_pointset.json sites/udmi_site_model/out/devices/GAT-123/state_system.json sites/udmi_site_model/out/devices/SNS-4/events_pointset.json sites/udmi_site_model/out/devices/SNS-4/events_system.json -sites/udmi_site_model/out/devices/SNS-4/events_validation.json :::::::::::::: sites/udmi_site_model/out/devices/AHU-1/config.out :::::::::::::: @@ -100,22 +96,6 @@ sites/udmi_site_model/out/devices/AHU-1/events_system.out } ] } :::::::::::::: -sites/udmi_site_model/out/devices/AHU-1/events_validation.out -:::::::::::::: -{ - "timestamp" : "REDACTED_TIMESTAMP", - "version" : "1.5.1", - "sub_folder" : "validation", - "sub_type" : "events", - "status" : { - "message" : "Successful validation", - "category" : "validation.device.receive", - "timestamp" : "REDACTED_TIMESTAMP", - "level" : 200 - }, - "errors" : [ ] -} -:::::::::::::: sites/udmi_site_model/out/devices/AHU-1/state.out :::::::::::::: { @@ -261,22 +241,6 @@ sites/udmi_site_model/out/devices/AHU-22/events_system.out "errors" : [ ] } :::::::::::::: -sites/udmi_site_model/out/devices/AHU-22/events_validation.out -:::::::::::::: -{ - "timestamp" : "REDACTED_TIMESTAMP", - "version" : "1.5.1", - "sub_folder" : "validation", - "sub_type" : "events", - "status" : { - "message" : "Successful validation", - "category" : "validation.device.receive", - "timestamp" : "REDACTED_TIMESTAMP", - "level" : 200 - }, - "errors" : [ ] -} -:::::::::::::: sites/udmi_site_model/out/devices/GAT-123/config.out :::::::::::::: { @@ -428,22 +392,6 @@ sites/udmi_site_model/out/devices/GAT-123/events_system.out "errors" : [ ] } :::::::::::::: -sites/udmi_site_model/out/devices/GAT-123/events_validation.out -:::::::::::::: -{ - "timestamp" : "REDACTED_TIMESTAMP", - "version" : "1.5.1", - "sub_folder" : "validation", - "sub_type" : "events", - "status" : { - "message" : "Successful validation", - "category" : "validation.device.receive", - "timestamp" : "REDACTED_TIMESTAMP", - "level" : 200 - }, - "errors" : [ ] -} -:::::::::::::: sites/udmi_site_model/out/devices/GAT-123/state.out :::::::::::::: { @@ -576,19 +524,3 @@ sites/udmi_site_model/out/devices/SNS-4/events_system.out }, "errors" : [ ] } -:::::::::::::: -sites/udmi_site_model/out/devices/SNS-4/events_validation.out -:::::::::::::: -{ - "timestamp" : "REDACTED_TIMESTAMP", - "version" : "1.5.1", - "sub_folder" : "validation", - "sub_type" : "events", - "status" : { - "message" : "Successful validation", - "category" : "validation.device.receive", - "timestamp" : "REDACTED_TIMESTAMP", - "level" : 200 - }, - "errors" : [ ] -} diff --git a/validator/src/main/java/com/google/daq/mqtt/validator/Validator.java b/validator/src/main/java/com/google/daq/mqtt/validator/Validator.java index c246a0617f..38610d7bfd 100644 --- a/validator/src/main/java/com/google/daq/mqtt/validator/Validator.java +++ b/validator/src/main/java/com/google/daq/mqtt/validator/Validator.java @@ -517,12 +517,19 @@ private void sendInitializationQuery() { } private boolean handleSystemMessage(Map attributes, Object object) { - if (SubFolder.UDMI.value().equals(attributes.get(SUBFOLDER_PROPERTY_KEY)) - && SubType.CONFIG.value().equals(attributes.get(SUBTYPE_PROPERTY_KEY))) { + + String subFolderRaw = attributes.get(SUBFOLDER_PROPERTY_KEY); + String subTypeRaw = attributes.get(SUBTYPE_PROPERTY_KEY); + + if (SubFolder.UDMI.value().equals(subFolderRaw) && SubType.CONFIG.value().equals(subTypeRaw)) { handleUdmiConfig(convertTo(UdmiConfig.class, object)); return true; } - return false; + + // Don't validate validation messages. Not really a problem, but sometimes validation messages + // aren't reflected back (when not using PubSub), so for consistency just reject everything. + return SubFolder.VALIDATION.value().equals(subFolderRaw) + && SubType.EVENTS.value().equals(subTypeRaw); } private void handleUdmiConfig(UdmiConfig udmiConfig) { From 45f9bbcd3d72b3060ab9359db7b8fe637bb85552 Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Wed, 17 Jul 2024 08:22:30 -0700 Subject: [PATCH 26/58] Cleanup dockerfiles --- .gitignore | 2 -- bin/container | 2 +- pubber/Dockerfile.pubber | 11 +++++++++++ .../etc/{Dockerfile.pubber => Dockerfile.puller} | 0 udmis/Dockerfile.udmis | 4 ++-- udmis/bin/build | 14 +++++++------- validator/bin/build | 8 +++++++- 7 files changed, 28 insertions(+), 13 deletions(-) create mode 100644 pubber/Dockerfile.pubber rename pubber/etc/{Dockerfile.pubber => Dockerfile.puller} (100%) diff --git a/.gitignore b/.gitignore index 0753e3b6d4..a564f419b4 100644 --- a/.gitignore +++ b/.gitignore @@ -25,8 +25,6 @@ credentials.json /udmis/var/ /udmis/profile/ /udmis/.idea/libraries/ -/udmis/udmi_bin/ -/udmis/udmi_etc/ /selfie/build/ /pubber/build/ /pubber/out/ diff --git a/bin/container b/bin/container index c226c9d8bb..05137134f7 100755 --- a/bin/container +++ b/bin/container @@ -22,7 +22,7 @@ git_repo=${git_prefix#*.com:} GCP_PROJECT=$(gcloud config get project) REPOSITORY=ghcr.io/${git_repo} -TEMPLATES=$(cd etc; ls k8s_*.yaml) +TEMPLATES=$(cd etc; ls k8s_*.yaml) || true NOCHECK= [[ ${1:-} == "--no-check" ]] && NOCHECK=1 && shift diff --git a/pubber/Dockerfile.pubber b/pubber/Dockerfile.pubber new file mode 100644 index 0000000000..eba32992fe --- /dev/null +++ b/pubber/Dockerfile.pubber @@ -0,0 +1,11 @@ +FROM alpine:latest + +WORKDIR /root + +RUN apk add openjdk17 bash gcompat curl jq python3 git + +ADD build/ pubber/build/ + +ADD bin/ pubber/bin/ + +CMD ["/root/pubber/bin/run"] diff --git a/pubber/etc/Dockerfile.pubber b/pubber/etc/Dockerfile.puller similarity index 100% rename from pubber/etc/Dockerfile.pubber rename to pubber/etc/Dockerfile.puller diff --git a/udmis/Dockerfile.udmis b/udmis/Dockerfile.udmis index 6a61524309..b94889363e 100644 --- a/udmis/Dockerfile.udmis +++ b/udmis/Dockerfile.udmis @@ -6,7 +6,7 @@ RUN apk add bash openjdk17 gcompat curl jq sudo git mosquitto mosquitto-clients ADD bin/ bin/ -ADD udmi_bin/ udmi/bin/ +ADD build/udmi_bin/ udmi/bin/ RUN bin/etcdctl version @@ -17,7 +17,7 @@ ADD build/libs/udmis-1.0-SNAPSHOT-all.jar build/libs/udmis-1.0-SNAPSHOT-all.jar ADD var/ var/ -ADD udmi_etc/ udmi/etc/ +ADD build/udmi_etc/ udmi/etc/ ENV CLEARBLADE_CONFIGURATION=/udmi/clearblade.json ENV SHUNT_CONFIGURATION=/udmi/shunt-config.json diff --git a/udmis/bin/build b/udmis/bin/build index 7c090f6131..15bd3e1c4a 100755 --- a/udmis/bin/build +++ b/udmis/bin/build @@ -19,12 +19,13 @@ ROOT=$(dirname $0)/.. BASE=.. cd $ROOT -echo Mirroring udim/bin to udmi_bin... -rm -rf udmi_bin/ -cp -a ../bin/ udmi_bin/ -echo Mirroring udmi/etc to udmi_etc... -rm -rf udmi_etc/ -cp -a ../etc udmi_etc/ +echo Cleaning $ROOT/build/ +rm -rf build +mkdir build + +echo Mirroring udim/bin and udmi/etc +cp -a ../bin/ build/udmi_bin/ +cp -a ../etc/ build/udmi_etc/ echo Copying default mosquitto config files... mkdir -p var/ @@ -43,7 +44,6 @@ export JAVA_HOME=$JAVA_HOME_11_X64 echo Building udmis in $PWD -rm -rf build ./gradlew shadow $check $covg $* ls -l $jarfile diff --git a/validator/bin/build b/validator/bin/build index 23a16f0350..c9e9317aa9 100755 --- a/validator/bin/build +++ b/validator/bin/build @@ -19,6 +19,13 @@ ROOT=$(dirname $0)/.. BASE=.. cd $ROOT +echo Cleaning $ROOT/build/ +rm -rf build/ +mkdir build/ + +echo Importing udmi/bin... +cp -a ../bin build/udmi_bin/ + jarfile=build/libs/validator-1.0-SNAPSHOT-all.jar newest=$(ls -rt `find src/ $BASE/gencode/java -type f` | tail -n 1) @@ -36,7 +43,6 @@ echo Java version $(java --version) echo Building validator in $PWD -rm -rf build ./gradlew shadow $check $covg $* ls -l $jarfile From c65f55fac9fca903e4d9124e48d99484e3b3fcde Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Wed, 17 Jul 2024 11:00:29 -0700 Subject: [PATCH 27/58] Basic validator container --- bin/container | 2 +- bin/toolrun | 2 +- udmis/bin/build | 3 ++- validator/Dockerfile.validator | 10 +++++++--- validator/bin/build | 4 +++- 5 files changed, 14 insertions(+), 7 deletions(-) diff --git a/bin/container b/bin/container index 05137134f7..e35560888f 100755 --- a/bin/container +++ b/bin/container @@ -91,7 +91,7 @@ if [[ -n $prep ]]; then rm -rf var tmp && mkdir -p var tmp cp -f etc/*.json var/ || true - cp -a ../schema tmp/ + cp -a ../schema build/ bin/build diff --git a/bin/toolrun b/bin/toolrun index b81112fd59..b6070ed0da 100755 --- a/bin/toolrun +++ b/bin/toolrun @@ -11,7 +11,7 @@ OUT_DIR=$UDMI_ROOT/out mkdir -p $OUT_DIR rm -f $OUT_DIR/$util_name.log -$UDMI_ROOT/validator/bin/build +[[ ! -d $UDMI_ROOT/validator/src ]] || $UDMI_ROOT/validator/bin/build JAVA_CLASS=com.google.daq.mqtt.util.Dispatcher diff --git a/udmis/bin/build b/udmis/bin/build index 15bd3e1c4a..b14d33700e 100755 --- a/udmis/bin/build +++ b/udmis/bin/build @@ -23,9 +23,10 @@ echo Cleaning $ROOT/build/ rm -rf build mkdir build -echo Mirroring udim/bin and udmi/etc +echo Mirroring udim files... cp -a ../bin/ build/udmi_bin/ cp -a ../etc/ build/udmi_etc/ +cp -a ../schema build/schema/ echo Copying default mosquitto config files... mkdir -p var/ diff --git a/validator/Dockerfile.validator b/validator/Dockerfile.validator index 6d13f676e2..d750d11495 100644 --- a/validator/Dockerfile.validator +++ b/validator/Dockerfile.validator @@ -11,10 +11,14 @@ ENV PATH=$PATH:/root/google-cloud-sdk/bin # Workaround for https://github.com/grpc/grpc-java/issues/8751 ENV LD_PRELOAD=/lib/libgcompat.so.0 -ADD tmp/schema/ schema/ +ADD bin/ validator/bin/ -ADD build/ validator/build/ +ADD build/schema/ schema/ -ADD bin/ validator/bin/ +ADD build/libs validator/build/libs/ + +ADD build/udmi_etc/ etc/ + +ADD build/udmi_bin/ bin/ CMD ["/root/validator/bin/automate"] diff --git a/validator/bin/build b/validator/bin/build index c9e9317aa9..791f7fbc89 100755 --- a/validator/bin/build +++ b/validator/bin/build @@ -23,8 +23,10 @@ echo Cleaning $ROOT/build/ rm -rf build/ mkdir build/ -echo Importing udmi/bin... +echo Mirroring udim files... cp -a ../bin build/udmi_bin/ +cp -a ../etc build/udmi_etc/ +cp -a ../schema build/schema/ jarfile=build/libs/validator-1.0-SNAPSHOT-all.jar From 0289061643a5b39df7da16f8e826d3f8c71d19bf Mon Sep 17 00:00:00 2001 From: Trevor Pering Date: Thu, 18 Jul 2024 08:17:28 -0700 Subject: [PATCH 28/58] Working registrar docker --- docs/udmis/local_docker.md | 31 +++++++++++++++++-- udmif/web/.husky/pre-commit | 5 --- validator/.idea/jarRepositories.xml | 5 +++ .../bos/iot/core/proxy/MqttPublisher.java | 2 +- 4 files changed, 34 insertions(+), 9 deletions(-) delete mode 100755 udmif/web/.husky/pre-commit diff --git a/docs/udmis/local_docker.md b/docs/udmis/local_docker.md index c1c858ad7a..21909b50e1 100644 --- a/docs/udmis/local_docker.md +++ b/docs/udmis/local_docker.md @@ -2,6 +2,27 @@ # Local docker UDMIS setup and execution +* `docker network create udminet` +* `newgrp docker` + +``` +site_model=$PWD/sites/udmi_site_model +``` + +``` +docker run -d --rm --net udminet --name udmis -p 8883:8883 \ + -v $site_model:/root/site \ + -v $PWD/var/etcd:/root/default.etcd \ + -v $PWD/var/mosquitto:/etc/mosquitto \ + ghcr.io/faucetsdn/udmi:udmis-latest udmi/bin/start_local block site/ //mqtt/localhost +``` + +``` +docker run --rm --net udminet --name validator \ + -v $site_model:/root/site \ + ghcr.io/faucetsdn/udmi:validator-latest bin/registrar site/ //mqtt/udmis +``` + * Identify a site model and parameters * For default reference udmi site model: * `bin/clone_model` @@ -22,6 +43,10 @@ General notes on how to build/push the upstream docker image. -* `bin/container udmis push` -* `docker tag udmis:latest ghcr.io/faucetsdn/udmi:latest` -* `docker push ghcr.io/faucetsdn/udmi:latest` +``` +for image in udmis validator pubber; do + bin/container $image push + docker tag $image:latest ghcr.io/faucetsdn/udmi:$image-latest + docker push ghcr.io/faucetsdn/udmi:$image-latest +done +``` diff --git a/udmif/web/.husky/pre-commit b/udmif/web/.husky/pre-commit deleted file mode 100755 index eb534aaf23..0000000000 --- a/udmif/web/.husky/pre-commit +++ /dev/null @@ -1,5 +0,0 @@ -#!/bin/sh -. "$(dirname "$0")/_/husky.sh" - -cd udmif/web -npx lint-staged diff --git a/validator/.idea/jarRepositories.xml b/validator/.idea/jarRepositories.xml index 1da9a525d2..5ecc50afc0 100644 --- a/validator/.idea/jarRepositories.xml +++ b/validator/.idea/jarRepositories.xml @@ -31,5 +31,10 @@