Delayed signing of notification from processing to the redis saving #2463

jimleroyer · 2025-02-18T19:56:31Z

Summary | Résumé

Delayed signing of notification from processing to the redis saving
Refactored many types as well to reflect the notification's lifecycle, as well as removed unnecessary fields.

Related Issues | Cartes liées

Add X-Ray Trace ID to Redis Cluster

Test instructions | Instructions pour tester la modification

TODO: Fill in test instructions for the reviewer.

Release Instructions | Instructions pour le déploiement

None.

Reviewer checklist | Liste de vérification du réviseur

This PR does not break existing functionality.
This PR does not violate GCNotify's privacy policies.
This PR does not raise new security concerns. Refer to our GC Notify Risk Register document on our Google drive.
This PR does not significantly alter performance.
Additional required documentation resulting of these changes is covered (such as the README, setup instructions, a related ADR or the technical documentation).

⚠ If boxes cannot be checked off before merging the PR, they should be moved to the "Release Instructions" section with appropriate steps required to verify before release. For example, changes to celery code may require tests on staging to verify that performance has not been affected.

Refactored many types as well to reflect the notification's lifecycle, as well as removed unnecessary fields.

jimleroyer · 2025-02-18T19:58:15Z

app/encryption.py

-SignedNotifications = NewType("SignedNotifications", List[SignedNotification])
-
-
-class NotificationDictToSign(TypedDict):


This was moved into the types module and became PendingNotification to represent the early lifecycle stage of the notification processing (when it hits the API and prior to be saved into the DB).

jimleroyer · 2025-02-18T19:58:24Z

app/encryption.py

-from typing_extensions import NotRequired  # type: ignore
-
-SignedNotification = NewType("SignedNotification", str)
-SignedNotifications = NewType("SignedNotifications", List[SignedNotification])


This was moved into the types module.

app/annotations.py

+    @wraps(func)
+    def wrapper(*args, **kwargs):
+        from app import signer_notification
+        from app.queue import QueueMessage


app/annotations.py

@@ -22,6 +57,9 @@

    @wraps(func)
    def wrapper(*args, **kwargs):
+        from app import signer_notification
+        from app.types import SignedNotification, SignedNotifications


app/queue.py

@@ -8,6 +8,7 @@
 from flask import current_app
 from redis import Redis

+from app.annotations import sign_param


app/types.py

 from app.models import Job, NotificationType, Service
+from app.queue import QueueMessage


tests/app/test_annotations.py

@@ -101,3 +134,18 @@

        signed = func_to_sign_return()
        assert signed == 1
+
+    def test_unsign_and_return(self, mocker):
+        from tests.app import test_annotations


jimleroyer · 2025-02-18T19:59:17Z

app/encryption.py

@@ -42,22 +18,22 @@ def init_app(self, app: Any, secret_key: str | List[str], salt: str) -> None:
        self.serializer = URLSafeSerializer(secret_key)
        self.salt = salt

-    def sign(self, to_sign: str | NotificationDictToSign) -> str | bytes:


Uncoupling the notification model from the CryptoSigner. It will only have a str to sign when that is called.

jimleroyer · 2025-02-18T19:59:58Z

app/notifications/process_notifications.py

-            template_id=notification.get("template_id"),
-            template_version=notification.get("template_version"),
+            template_id=verifiedNotification.template_id,
+            template_version=verifiedNotification.template_version,


This is now a data class so it has proper fields instead of dictionary entries.

jimleroyer · 2025-02-18T20:01:50Z

app/notifications/process_notifications.py

+            #   created_by_id: this is the user who created the notification and will be set on sending time, used by one off or admin UI uploads
+            # reference=verifiedNotification.reference,  # type: ignore
+            # created_by_id=verifiedNotification.created_by_id,  # type: ignore
+            # billable_units=verifiedNotification.billable_units,  # type: ignore


I think we can remove these:

reference: a notification will only have a reference once SES or Pinpoint is called. The exception is letters where a random reference will be generated but this function does not handle these, not GCNotify at large.

created_by_id: Not set when received by the API AFAIK. Other code paths are used by the one-off send or bulk send, where the created_by_id might be set.

billable_units: most likely not set at this stage yet.

jimleroyer · 2025-02-18T20:03:53Z

app/notifications/process_notifications.py

+            # reference=verifiedNotification.reference,  # type: ignore
+            # created_by_id=verifiedNotification.created_by_id,  # type: ignore
+            # billable_units=verifiedNotification.billable_units,  # type: ignore
+            status=NOTIFICATION_CREATED,  # type: ignore


We do not receive a status from the notification, and should we? We can safely say the notification is created at this point, prior to save it into the database. I checked the other code paths for the persist_notification function (the one off save -- this one is for multiple) and it sets the notification to created status by default.

jimleroyer · 2025-02-18T20:05:15Z

app/notifications/process_notifications.py

+            # created_by_id=verifiedNotification.created_by_id,  # type: ignore
+            # billable_units=verifiedNotification.billable_units,  # type: ignore
+            status=NOTIFICATION_CREATED,  # type: ignore
+            reply_to_text=verifiedNotification.reply_to_text,


The reply_to_text field can be set via the API by providing the email_reply_to_id JSON field in the POST request. So that should be present in the PendingNotification | VerifiedNotification type and passed down the stream to the database.

jimleroyer · 2025-02-18T20:05:39Z

app/notifications/process_notifications.py

+            case models.EMAIL_TYPE:
+                notification_obj.normalised_to = format_email_address(notification_recipient)
+            # case models.LETTER_TYPE:
+            #     notification_obj.postage = verifiedNotification.postage  # or verifiedNotification.template_postage


Ah yeah.. I thought this is so useless and annoying..

jimleroyer · 2025-02-18T20:06:11Z

app/types.py

+    api_key: str
+    key_type: str  # should be ApiKeyType but can't import that here
+    client_reference: Optional[str]
+    reply_to_text: Optional[str]


The reply_to_text field can be set via the API by providing the email_reply_to_id JSON field in the POST request. So that should be present in the PendingNotification | VerifiedNotification type and passed down the stream to the database.

jimleroyer · 2025-02-18T20:06:28Z

app/types.py

-class VerifiedNotification(NotificationDictToSign):
+
+@dataclass
+class PendingNotification(QueueMessage):


Previously known as NotificationDictToSign.

jimleroyer · 2025-02-18T20:08:37Z

app/v2/notifications/post_notifications.py


-    signed_notification_data = signer_notification.sign(_notification)


This layer does not signed the notification anymore: we want to delay the signing in the Redis queue because we need to add extra metadata in there, some coupled to queue management. Hence we removed the signing and Redis will do it itself, via the new signing annotations.

jimleroyer · 2025-02-18T20:09:28Z

app/queue.py

@@ -193,8 +219,9 @@ def acknowledge(self, receipt: UUID) -> bool:
        put_batch_saving_inflight_processed(self.__metrics_logger, self, 1)
        return True

-    def publish(self, message: str):
-        self._redis_client.rpush(self._inbox, message)
+    @sign_param


This will sign the parameter automatically, using the app.signer_notification object.

smcmurtry

I like these changes so far!

smcmurtry · 2025-02-19T20:02:55Z

app/celery/tasks.py

-            "job_id": _notification.get("job", None),
-            "job_row_number": _notification.get("row_number", None),
-        }
+        notification = VerifiedNotification.from_dict(


Nice! having a from_dict method is a much better way to do this.

Should we rename notification -> verified_notification to make this consistent with the code below?

smcmurtry · 2025-02-19T20:21:23Z

app/notifications/process_notifications.py

            if redis_store.get(redis.daily_limit_cache_key(service_id)):
                redis_store.incr(redis.daily_limit_cache_key(service_id))

        current_app.logger.info(
            "{} {} created at {}".format(
-                notification.get("notification_type"),
-                notification.get("notification_id"),
-                notification.get("notification_created_at"),  # type: ignore


Was this just a bug? I don't think this key was actually there.

smcmurtry · 2025-02-19T20:29:48Z

app/types.py

+    @classmethod
+    def from_dict(cls, data: dict):
+        """Create an object from a dictionary."""
+        return cls(**data)


Looking at the doc for dataclasses: https://docs.python.org/3/library/dataclasses.html

It looks like that decorater will add an __init__ method where you can pass in all the data you have defined above. So you could create an instance like this:

pending_notification = PendingNotification(id=x.id, template=x.template, etc )

instead of:

pending_notification = PendingNotification.from_dict({"id": x.id, "template": x.template, etc })

if you want.

But we do often pass in a dict that looks like {**notification, "id": new_id} so having a from_dict method is probably best

smcmurtry · 2025-02-19T20:31:40Z

app/v2/notifications/post_notifications.py

-        "client_reference": form.get("reference", None),
-        "reply_to_text": reply_to_text,
-    }
+    _notification = PendingNotification.from_dict(


could rename to

Suggested change

_notification = PendingNotification.from_dict(

pending_notification = PendingNotification.from_dict(

smcmurtry · 2025-02-19T20:42:31Z

app/annotations.py

-from app import signer_notification
-from app.encryption import SignedNotification, SignedNotifications
+
+def sign_param(func):


This seems like a great idea but I can't tell if it works just by looking at the code. Could you add some instructions for how to test if this is working?

Delayed signing of notification from processing to the redis saving.

a111b12

Refactored many types as well to reflect the notification's lifecycle, as well as removed unnecessary fields.

jimleroyer requested review from smcmurtry and sastels February 18, 2025 19:56

jimleroyer self-assigned this Feb 18, 2025

jimleroyer commented Feb 18, 2025

View reviewed changes

github-advanced-security bot found potential problems Feb 18, 2025

View reviewed changes

jimleroyer commented Feb 18, 2025

View reviewed changes

jimleroyer mentioned this pull request Feb 19, 2025

Add X-Ray Trace ID to Redis Cluster cds-snc/notification-planning-core#476

Open

10 tasks

smcmurtry reviewed Feb 19, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Delayed signing of notification from processing to the redis saving #2463

Delayed signing of notification from processing to the redis saving #2463

jimleroyer commented Feb 18, 2025

jimleroyer Feb 18, 2025

jimleroyer Feb 18, 2025

jimleroyer Feb 18, 2025

jimleroyer Feb 18, 2025

jimleroyer Feb 18, 2025

jimleroyer Feb 18, 2025

jimleroyer Feb 18, 2025

jimleroyer Feb 18, 2025

jimleroyer Feb 18, 2025

jimleroyer Feb 18, 2025

jimleroyer Feb 18, 2025

jimleroyer Feb 18, 2025 •

edited

Loading

smcmurtry left a comment

smcmurtry Feb 19, 2025

smcmurtry Feb 19, 2025

smcmurtry Feb 19, 2025

smcmurtry Feb 19, 2025

smcmurtry Feb 19, 2025 •

edited

Loading

smcmurtry Feb 19, 2025

smcmurtry Feb 19, 2025

		SignedNotifications = NewType("SignedNotifications", List[SignedNotification])


		class NotificationDictToSign(TypedDict):

		from app.models import Job, NotificationType, Service
		from app.queue import QueueMessage


		signed_notification_data = signer_notification.sign(_notification)

	_notification = PendingNotification.from_dict(
	pending_notification = PendingNotification.from_dict(

Delayed signing of notification from processing to the redis saving #2463

Are you sure you want to change the base?

Delayed signing of notification from processing to the redis saving #2463

Conversation

jimleroyer commented Feb 18, 2025

Summary | Résumé

Related Issues | Cartes liées

Test instructions | Instructions pour tester la modification

Release Instructions | Instructions pour le déploiement

Reviewer checklist | Liste de vérification du réviseur

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimleroyer Feb 18, 2025 • edited Loading

Choose a reason for hiding this comment

smcmurtry left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

smcmurtry Feb 19, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jimleroyer Feb 18, 2025 •

edited

Loading

smcmurtry Feb 19, 2025 •

edited

Loading