feat: Add OpenRPC API docs and associated JSON Schemas (#534)

# Description

This pull request includes the following features:

- [x] Add OpenRPC API document and JSON schema generator
  - [x] Generate schemas in CI 
- [x] Add OpenRPC API document
  - [x] Add document
  - [x] Add `rpc.discover` endpoint for service discovery
- [x] Add JSON Schemas
  - [x] Workflow schema
  - [x] Receipt schema
  - [x] Receipt notification schema
  - [x] Health check schema
  - [x] Node info schema
  - [x] Metrics schema
  - [x] Network notifications
- [x] Refactor network notifications with structs to better support JSON
Schemas
- [x] Add rendezvous and mDNS network notifications
  - [x] Update discovery tests to use notifications instead of timeouts 
- [x] Convert event cache to async implementation
- [x] Update pre-commit hook to allow no newline at the end of JSON
files
- [x] Add `subscribe_network_events` test utility
- [x] Update `dependabot_pr` GitHub action check for modified files 

## Link to issue

Closes #449 

## Type of change

- [x] New feature (non-breaking change that adds functionality)
- [x] Refactor (non-breaking change that updates existing functionality)
- [x] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [x] This change requires a documentation update (document the OpenRPC
API, API discovery endpoint, and schemas in README)
- [x] Comments have been added/updated

The network notifications and node info response contain breaking
changes for Control Panel.

## Test plan (required)

The following test suites were added or updated:

- [x] Refactored [network notification
tests](https://github.com/ipvm-wg/homestar/blob/b5ffd8181b09c585525be57c9e2bace86768bf76/homestar-runtime/src/event_handler/notification/network.rs#L2173-L2671)
- [x] OpenRPC discover endpoint test
- [x] Separated out node info test
- [x] Refactored Rendezvous, mDNS, and connection tests to rely on new
notifications
- [x] Consolidated rendezvous connect and disconnect tests
- [x] Consolidated mDNS connect and disconnect tests
- [x] Consolidated connection and disconnection tests

---------

Co-authored-by: Zeeshan Lakhani <[email protected]>
Co-authored-by: bgins <[email protected]>

.dockerignore

@@ -10,7 +10,9 @@
 !**/migrations
 !diesel.toml
 !**/wit
+!**/schemas/api.json
 
 examples
 homestar-functions
 homestar-workspace-hack
+homestar-schemas

.github/workflows/audit.yml

@@ -2,7 +2,7 @@ name: 🛡 Audit-Check
 
 on:
   schedule:
-    - cron: '0 0 * * *'
+    - cron: "0 0 * * *"
 
 jobs:
   security-audit:

.github/workflows/coverage.yml

@@ -2,10 +2,10 @@ name: ☂ Code Coverage
 
 on:
   push:
-    branches: [ main ]
+    branches: [main]
 
   pull_request:
-    branches: [ '*' ]
+    branches: ["*"]
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}

.github/workflows/dependabot_pr.yaml

@@ -35,7 +35,7 @@ jobs:
 
       - name: Check for modified files
         id: git-check
-        run: echo modified=$(if git diff-index --quiet HEAD --; then echo "false"; else echo "true"; fi) >> $GITHUB_OUTPUT
+        run: echo modified=$(if [[ $(git diff) ]]; then echo "true"; else echo "false"; fi) >> $GITHUB_OUTPUT
 
       - name: Push changes
         if: steps.git-check.outputs.modified == 'true'

.github/workflows/schemas.yml

@@ -0,0 +1,72 @@
+name: 📄 Schemas
+
+on:
+  push:
+    branches: [main]
+    # branches: [main, "**"]
+
+permissions:
+  contents: write
+  pull-requests: write
+
+jobs:
+  changes:
+    runs-on: ubuntu-latest
+    outputs:
+      rust: ${{ steps.filter.outputs.rust }}
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+      - uses: dorny/paths-filter@v3
+        id: filter
+        with:
+          filters: |
+            rust:
+              - 'homestar-invocation/src/**'
+              - 'homestar-runtime/src/**'
+              - 'homestar-schemas/src/**'
+              - 'homestar-workflow/src/**'
+
+  schemas:
+    needs: changes
+    if: ${{ needs.changes.outputs.rust == 'true' }}
+    env:
+      SCCACHE_GHA_ENABLED: "true"
+      RUSTC_WRAPPER: "sccache"
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+          token: ${{ secrets.HOMESTAR_UPDATE_TOKEN }}
+
+      - name: Install Rust Toolchain
+        uses: dtolnay/rust-toolchain@stable
+
+      - name: Cache Project
+        uses: Swatinem/rust-cache@v2
+        with:
+          cache-on-failure: true
+          shared-key: test-all-stable-ubuntu-latest
+          save-if: ${{ github.event_name == 'push' }}
+
+      - name: Sccache
+        uses: mozilla-actions/[email protected]
+
+      - name: Run generate schemas
+        run: cargo run -p homestar-schemas
+
+      - name: Check for modified schemas
+        id: git-check
+        shell: bash
+        run: echo modified=$(if [[ $(git diff homestar-runtime/schemas/) ]]; then echo "true"; else echo "false"; fi) >> $GITHUB_OUTPUT
+
+      - name: Push changes
+        if: steps.git-check.outputs.modified == 'true'
+        run: |
+          git config user.name "${GITHUB_ACTOR}"
+          git config user.email "${GITHUB_ACTOR}@users.noreply.github.com"
+          git remote set-url origin https://x-access-token:${GITHUB_TOKEN}@github.com/${GITHUB_REPOSITORY}.git
+          git commit -am "chore(schemas): update OpenRPC API doc and JSON schemas"
+          git push

.pre-commit-config.yaml

@@ -79,7 +79,7 @@ repos:
       - id: check-merge-conflict
       - id: trailing-whitespace
       - id: end-of-file-fixer
-        exclude: \.txt$
+        exclude: \.(txt|json)$
       - id: check-yaml
       - id: check-json
       - id: check-added-large-files

Cargo.toml

@@ -4,6 +4,7 @@ members = [
   "homestar-functions/*",
   "homestar-invocation",
   "homestar-runtime",
+  "homestar-schemas",
   "homestar-wasm",
   "homestar-workflow",
   "homestar-workspace-hack",
@@ -34,6 +35,7 @@ chrono = { version = "0.4", default-features = false, features = [
   "clock",
   "std",
 ] }
+const_format = "0.2"
 enum-assoc = "1.1"
 enum-as-inner = "0.6"
 faststr = { version = "0.2", default-features = false, features = ["serde"] }
@@ -51,7 +53,11 @@ libsqlite3-sys = { version = "0.27", default-features = false, features = [
   "bundled",
 ] }
 rand = { version = "0.8", default-features = false }
+schemars = { version = "0.8", features = ["arrayvec", "url"] }
 serde = { version = "1.0", default-features = false, features = ["derive"] }
+serde_json = { version = "1.0", default-features = false, features = [
+  "raw_value",
+] }
 serde_ipld_dagcbor = { version = "0.4", default-features = false, features = [
   "std",
 ] }

README.md

@@ -154,6 +154,10 @@ represents the `Homestar` runtime. We recommend diving into each package's own
   writing and compiling [Wasm component][wasm-component] modules using
   [wit-bindgen][wit-bindgen].
 
+- [homestar-schemas](./homestar-schemas)
+
+`homestar-schemas` is a crate for generating OpenRPC docs and JSON Schemas that document the [homestar-runtime](./homestar-runtime) JSON-RPC API, workflows, and receipts.
+
 - [examples/*](./examples)
 
   `examples` contains examples and demos showcasing `Homestar` packages

docker/Dockerfile

@@ -31,9 +31,11 @@ RUN cargo init --lib homestar-invocation && \
 
 RUN echo "fn main() {}" > ./homestar-runtime/src/main.rs
 
-RUN mkdir -p ./homestar-runtime/src/test_utils/proc_macro ./homestar-runtime/migrations ./examples ./homestar-functions
+RUN mkdir -p ./homestar-runtime/src/test_utils/proc_macro ./homestar-runtime/migrations \
+    ./examples ./homestar-functions ./homestar-schemas
 RUN bash -c 'pushd ./examples && cargo init dummy-app-examples && popd'
 RUN bash -c 'pushd ./homestar-functions && cargo init dummy-app-fns && popd'
+RUN cargo init homestar-schemas
 RUN cargo init --lib homestar-workspace-hack
 
 # copy cargo.*
@@ -43,6 +45,7 @@ COPY ../homestar-workflow/Cargo.toml ./homestar-workflow/
 COPY ../homestar-wasm/Cargo.toml ./homestar-wasm/
 COPY ../homestar-runtime/Cargo.toml ./homestar-runtime/
 COPY ../homestar-runtime/migrations ./homestar-runtime/migrations
+COPY ../homestar-runtime/schemas/api.json ./homestar-runtime/schemas/api.json
 COPY ../homestar-runtime/src/test_utils/proc_macro ./homestar-runtime/src/test_utils/proc_macro
 
 ENTRYPOINT ["/bin/bash"]
@@ -91,6 +94,7 @@ WORKDIR /home/runner
 
 COPY --chown=homestar:homestar diesel.toml ./
 COPY --chown=homestar:homestar ../homestar-runtime/migrations ./migrations
+COPY --chown=homestar:homestar ../homestar-runtime/schemas ./schemas
 COPY --chown=homestar:homestar --from=builder /usr/local/bin/homestar-runtime ./homestar
 COPY --chown=homestar:homestar --from=builder /usr/local/bin/diesel /usr/local/bin/diesel
 COPY --chown=homestar:homestar --from=builder  /etc/*.db ./