Add MessageUpgrader to validator #286

grafnu · 2022-04-01T15:56:27Z

No description provided.

noursaidi · 2022-04-03T23:14:50Z

The validator doesn't appear to be process state messages with this change - when a state message is published, nothing appears in the command line output or output directory.

I have compared against the current master (checksum 29530a7c8e265230d5127501feeee7b49811b1d5) and this does process state messages, so my message flow seems to be working. In both cases I was manually triggering state messages from an older instance of pubber so I'm certain there were state messages being sent

grafnu · 2022-04-04T03:36:01Z

Did you update your cloud functions? There's been a bunch of changes there that need to go along with this. Do I have access to the project? What is it? Would just make it easier for me to poke at it!

…

On Sun, Apr 3, 2022 at 4:15 PM Noureddine ***@***.***> wrote: The validator doesn't appear to be process state messages with this change - when a state message is published, nothing appears in the command line output or output directory. I have compared against the current master (checksum 29530a7) and this does process state messages, so my message flow seems to be working. In both cases I was manually triggering state messages from an older instance of pubber so I'm certain there were state messages being sent — Reply to this email directly, view it on GitHub <#286 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIEPDY5H74ZZOIJJQLCH6LVDIQ7LANCNFSM5SJDQ5ZQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

noursaidi · 2022-04-04T10:35:19Z

I did deploy the cloud functions - I had to manually add the GCP project ID as an environment variable in the dashboard_deploy_gcloud script as it was coming up undefined (same issue as Francesco)

You have access to my GCP project. You run the validator with the following command
bin/validator <any site model> daq1-273309 udmi_target

I have configured a script to publish a static state payload every minute under device AHU-9 (you do have access to my site model but the payload I'm publishing doesn't match up with the site model)

grafnu · 2022-04-04T15:23:22Z

Ok, I can reproduce the problem... I think I was just so focused on the config and event messages that status kinda for lost :-).

…

On Mon, Apr 4, 2022 at 3:35 AM Noureddine ***@***.***> wrote: I did deploy the cloud functions <https://console.cloud.google.com/functions/list?project=daq1-273309> - I had to manually add the GCP project ID as an environmental variable in the dashboard_deploy_gcloud script as it was coming up undefined (same issue as Francesco) You have access to my GCP project. You run the validator with the following command bin/validator <any site model> daq1-273309 udmi_target I have configured a script to publish a static state payload every minute under device AHU-9 <https://console.cloud.google.com/iot/locations/us-central1/registries/registrar_test/devices/details/AHU-9/history?project=daq1-273309> (you do have access to my site model <https://github.com/noursaidi/nes-site-model> but the payload I'm publishing doesn't match up with the site model) — Reply to this email directly, view it on GitHub <#286 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIEPD7MZQWW6PPGGWIR43TVDLAXDANCNFSM5SJDQ5ZQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

grafnu · 2022-04-04T16:10:11Z

Ok, fixed. It was an update to the cloud functions that was required -- so the daq1 project is updated and working now (at least I tested against it). Also, I noticed that you had tried to set a UDMI_GCP_PROJECT env variable... but that wouldn't work, it would need to be GCP_PROJECT. I did update the gcloud deploy command to automatically set the GCP_PROJECT variable appropriately. GCP changed some things about their default env variables with the node runtime so it was confusing a few things.

…

On Mon, Apr 4, 2022 at 8:23 AM Trevor Pering ***@***.***> wrote: Ok, I can reproduce the problem... I think I was just so focused on the config and event messages that status kinda for lost :-). On Mon, Apr 4, 2022 at 3:35 AM Noureddine ***@***.***> wrote: > I did deploy the cloud functions > <https://console.cloud.google.com/functions/list?project=daq1-273309> - > I had to manually add the GCP project ID as an environmental variable in > the dashboard_deploy_gcloud script as it was coming up undefined (same > issue as Francesco) > > You have access to my GCP project. You run the validator with the > following command > bin/validator <any site model> daq1-273309 udmi_target > > I have configured a script to publish a static state payload every minute > under device AHU-9 > <https://console.cloud.google.com/iot/locations/us-central1/registries/registrar_test/devices/details/AHU-9/history?project=daq1-273309> > (you do have access to my site model > <https://github.com/noursaidi/nes-site-model> but the payload I'm > publishing doesn't match up with the site model) > > — > Reply to this email directly, view it on GitHub > <#286 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAIEPD7MZQWW6PPGGWIR43TVDLAXDANCNFSM5SJDQ5ZQ> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

noursaidi · 2022-04-04T16:59:02Z

I did add const PROJECT_ID = process.env.UDMI_GCP_PROJECT in the cloud function, I was testing something with Firebase deploy which is also not working for me.

Though for this, FIREBASE_CONFIG is set and within this there is a field projectId with the project ID which can be used in place of process.env.GCLOUD_PROJECT for this deploy method

It is now processing state messages and old payloads aren't triggering an error - but behaviour has changed. There used to be separate state_pointset.out and state_system.out etc, but now there's a single state.out - was this expected? It's also still not validating (against metadata) the points in pointset (issue #193)

grafnu · 2022-04-04T17:38:26Z

Expected... kinda? I wasn't going into the changes with a specific target in mind, but more of a "trying to make it logically consistent." Right now it's validating the entire "state.json" message, which is why there is one "state.out" file -- it's not individually validating the sub-blocks (state_pointset.json) -- the sequence tester does that (well, and records both), but it's operating in a different way. This PR was primarily about adding the MessageUpgrader to the validator, so it didn't try to validate against metadata -- that would be something else.

…

On Mon, Apr 4, 2022 at 9:59 AM Noureddine ***@***.***> wrote: I did add const PROJECT_ID = process.env.UDMI_GCP_PROJECT in the cloud function, I was testing something with Firebase deploy which is also not working for me. Though for this, FIREBASE_CONFIG is set and within this there is a field projectId with the project ID which can be used in place of process.env.GCLOUD_PROJECT for this deploy method It is now processing state messages and old payloads aren't triggering an error - but behaviour has changed. There used to be separate state_pointset.out and state_system.out etc, but now there's a single state.out - was this expected? It's also still not validating (against metadata) the points in pointset (issue #193 <#193>) — Reply to this email directly, view it on GitHub <#286 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIEPD72YKRNT3K7ECDPKN3VDMNWDANCNFSM5SJDQ5ZQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

grafnu · 2022-04-04T17:42:26Z

Note that one change that IS for sure expected is the location of the output files. They used to be rooted in the udmi/out/ directory, but now they should appear in the {site_model}/out/ directory -- moving towards the general expectation that the udmi directory would be "system level" so all site-specific things would be kept differently. Validator and sequencer should work this way now, but I think pubber is till in the old model. On Mon, Apr 4, 2022 at 10:38 AM Trevor Pering ***@***.***> wrote:

…

Expected... kinda? I wasn't going into the changes with a specific target in mind, but more of a "trying to make it logically consistent." Right now it's validating the entire "state.json" message, which is why there is one "state.out" file -- it's not individually validating the sub-blocks (state_pointset.json) -- the sequence tester does that (well, and records both), but it's operating in a different way. This PR was primarily about adding the MessageUpgrader to the validator, so it didn't try to validate against metadata -- that would be something else. On Mon, Apr 4, 2022 at 9:59 AM Noureddine ***@***.***> wrote: > I did add const PROJECT_ID = process.env.UDMI_GCP_PROJECT in the cloud > function, I was testing something with Firebase deploy which is also not > working for me. > > Though for this, FIREBASE_CONFIG is set and within this there is a field > projectId with the project ID which can be used in place of > process.env.GCLOUD_PROJECT for this deploy method > > It is now processing state messages and old payloads aren't triggering an > error - but behaviour has changed. There used to be separate > state_pointset.out and state_system.out etc, but now there's a single > state.out - was this expected? It's also still not validating (against > metadata) the points in pointset (issue #193 > <#193>) > > — > Reply to this email directly, view it on GitHub > <#286 (comment)>, or > unsubscribe > <https://github.com/notifications/unsubscribe-auth/AAIEPD72YKRNT3K7ECDPKN3VDMNWDANCNFSM5SJDQ5ZQ> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

johnrandolph · 2022-04-04T16:28:34Z

pubber/src/main/java/daq/pubber/Pubber.java

@@ -379,7 +379,6 @@ private synchronized void cancelExecutor() {
    if (scheduledFuture != null) {
      try {
        scheduledFuture.cancel(false);
-        scheduledFuture.get();


Was this creating problems or is it just cleanup to not wait? Sometimes these small changes are tricky to gauge.

It actually causes problems -- once the Future is cancelled (previous line) the get() fails... Just never noticed it until I started to trace down errors...

johnrandolph · 2022-04-04T18:01:05Z

validator/src/main/java/com/google/daq/mqtt/validator/Validator.java

@@ -473,7 +484,7 @@ private File makeDeviceDir(String deviceId) {
    return deviceDir;
  }

-  private String makeSchemaName(Map<String, String> attributes) {
+  private String mungeAndExtractSchema(Map<String, String> attributes) {


It isn't immediately obvious why we are crossing subFolder and subType values.

Can we either make it more clear with comments, or, reference the doc? envelope.md doesn't explain it ... ?

Ok, you shamed me into fixing the underlying confusion, so the crossing the values is no longer required. It did involve some additional changes, but it's now a lot cleaner overall, I think.

grafnu · 2022-04-05T15:20:23Z

I'm submitting this now even though there might be further tweaks necessary, since it at least cleans up a bunch of underlying issues that might be impacting other threads. If there's any additional comments/changes, just post them and I can address in another PR.

grafnu added 30 commits March 21, 2022 08:23

Make gateway results more flexible

619bfee

Gencode

21f32d0

Single test running capability

9cddf4d

Basic cleanup of test runner

1cc8d1c

Cleanup

a39b0e7

Limit test class

81ca9a7

Test method target checking

08c5f2b

Disable test method

7a55b50

Linty

47c2785

linty

3f72cf8

Single test on command line

2693c4a

Diagnostics

f93b174

Use . to mean all tests

5e2610a

Merge branch 'master' into gatewayerror

6d7d029

Single test fix

0f5d45a

Cleanup gateway.out

9ee92ed

Update out/ directory to be under the site model

f20fbf4

Cleanup of useFirestore

78bb691

Fix config timestamp/version generation

b95f4e4

Add log level filtering

34beb8f

Adding logLevel flag

6b5fc70

Chaing info level for a few things

a6447bc

Notification cleanup

6b5802e

Debug test output

1960244

Fix test out location

fda67ac

Fixing validator.out

665c39d

Upgrade validator messages

46f41a8

Trying to fix validations

c6f9f9a

Merge branch 'master' into gatewayerror

beff2f9

Fix for non-firebase world

dd60b5d

grafnu added 6 commits April 1, 2022 15:34

Fix serial number test

8cfb01c

Adding in config update delay

973dc80

Add config update send delay

ed7bf47

Change order of validator test

939489c

Fix out order

7aae9dd

Fix out order

699cdef

Fix states handling

e49dfa1

noursaidi approved these changes Apr 4, 2022

View reviewed changes

johnrandolph reviewed Apr 4, 2022

View reviewed changes

grafnu added 8 commits April 4, 2022 13:37

Clean up update handling

a353068

Fixup JS code

1c688d3

Fix update handling

7bbef70

Fix typo

31a127d

Fixing state query

0614dd1

Fix some update details

e397a8a

Fix sequencer

44c5edc

Fix validator

407532c

grafnu requested review from noursaidi and johnrandolph April 5, 2022 05:10

grafnu merged commit d573d5f into faucetsdn:master Apr 5, 2022

grafnu deleted the validator branch April 5, 2022 15:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add MessageUpgrader to validator #286

Add MessageUpgrader to validator #286

grafnu commented Apr 1, 2022

noursaidi commented Apr 3, 2022

grafnu commented Apr 4, 2022 via email

noursaidi commented Apr 4, 2022 •

edited

Loading

grafnu commented Apr 4, 2022 via email

grafnu commented Apr 4, 2022 via email

noursaidi commented Apr 4, 2022

grafnu commented Apr 4, 2022 via email

grafnu commented Apr 4, 2022 via email

johnrandolph Apr 4, 2022

grafnu Apr 5, 2022

johnrandolph Apr 4, 2022

grafnu Apr 5, 2022

grafnu commented Apr 5, 2022

Add MessageUpgrader to validator #286

Add MessageUpgrader to validator #286

Conversation

grafnu commented Apr 1, 2022

noursaidi commented Apr 3, 2022

grafnu commented Apr 4, 2022 via email

noursaidi commented Apr 4, 2022 • edited Loading

grafnu commented Apr 4, 2022 via email

grafnu commented Apr 4, 2022 via email

noursaidi commented Apr 4, 2022

grafnu commented Apr 4, 2022 via email

grafnu commented Apr 4, 2022 via email

johnrandolph Apr 4, 2022

Choose a reason for hiding this comment

grafnu Apr 5, 2022

Choose a reason for hiding this comment

johnrandolph Apr 4, 2022

Choose a reason for hiding this comment

grafnu Apr 5, 2022

Choose a reason for hiding this comment

grafnu commented Apr 5, 2022

noursaidi commented Apr 4, 2022 •

edited

Loading