Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Docs/async blog #744

Merged
merged 4 commits into from
Jul 29, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
publish:
name: Publish docs

if: github.event_name == 'push' && startsWith(github.event.ref, 'refs/tags/v')
if: github.ref == 'refs/heads/master'

needs: build
runs-on: ubuntu-latest
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -15,16 +15,16 @@ In this blog post, I will delve into how this is all achieved. From explaining h

<!-- more -->

## Briding Python and Binary Libraries
## Bridging Python and Binary Libraries

Python, known for its dynamic typing and automated memory management, is fundamentally an interpreted language. Despite not having innate capabilities to directly interact with binary libraries, most Python interpreters bridge this gap efficiently. For instance, CPython—the principal interpreter—enables the creation of binary extensions[^1] and similarly, PyPy—a widely-used alternative—offers comparable functionalities[^2].
Python, known for its dynamic typing and automated memory management, is fundamentally an interpreted language. Despite not having innate capabilities to directly interact with binary libraries, most Python interpreters bridge this gap efficiently. For instance, CPython—the principal interpreter—enables the creation of binary extensions[^binary_extension] and similarly, PyPy—a widely-used alternative—offers comparable functionalities[^pypy].

[^1]: You can find extensive documentation on building extensions for CPython [here](https://docs.python.org/3/extending/extending.html).
[^2]: PyPy extension-building documentation is available [here](https://doc.pypy.org/en/latest/extending.html).
[^binary_extension]: You can find extensive documentation on building extensions for CPython [here](https://docs.python.org/3/extending/extending.html).
[^pypy]: PyPy extension-building documentation is available [here](https://doc.pypy.org/en/latest/extending.html).

However, each interpreter has a distinct API tailored for crafting these binary extensions, which unfortunately leads to a lack of universal solutions across different environments. Furthermore, interpreters like [Jython](https://jython.org) and [Pyodide](https://pyodide.org/en/stable/), which are based on Java and WebAssembly respectively, present unique challenges that often preclude the straightforward use of such extensions due to their distinct runtime architectures.[^3]
However, each interpreter has a distinct API tailored for crafting these binary extensions, which unfortunately leads to a lack of universal solutions across different environments. Furthermore, interpreters like [Jython](https://jython.org) and [Pyodide](https://pyodide.org/en/stable/), which are based on Java and WebAssembly respectively, present unique challenges that often preclude the straightforward use of such extensions due to their distinct runtime architectures.[^pyodide]

[^3]: It would appear that Pyodide can support C extensions as explained [here](https://pyodide.org/en/stable/development/new-packages.html), though by and large Pyodide appears to be intended for pure Python packages.
[^pyodide]: It would appear that Pyodide can support C extensions as explained [here](https://pyodide.org/en/stable/development/new-packages.html), though by and large Pyodide appears to be intended for pure Python packages.

While it is possible for the extension to contain all the logic, our specific requirement is merely to provide a bridge between Python and the Rust core library. This is the niche that [Python C Foreign Function Interface (CFFI)](https://cffi.readthedocs.io/en/stable/) fills. By parsing a C header file, CFFI automates the generation of extension code needed for Python to interface with the binary library. Consequently, this library can be imported into Python as if it were any standard module—streamlining development and potentially improving performance by leveraging optimized native code.

Expand Down Expand Up @@ -175,10 +175,10 @@ class OwnedString(str):
lib.pactffi_string_delete(self._ptr)
```

The `__del__` method is called[^4] when the object is about to be deallocated[^5], allowing us to free the memory associated with the string. This ensures that memory is managed correctly and prevents potential memory leaks.
The `__del__` method is called[^del_exceptions] when the object is about to be deallocated[^del_no_guarantee], allowing us to free the memory associated with the string. This ensures that memory is managed correctly and prevents potential memory leaks.

[^4]: There are some unique circumstances where `__del__` may not be called, such as when the Python interpreter is shutting down.
[^5]: Python does not provide guarantees on when `__del__` will be called, so it is not recommended to rely on it for critical cleanup tasks. Instead, the `__enter__` and `__exit__` methods should be used to guarantee timely cleanup.
[^del_exceptions]: There are some unique circumstances where `__del__` may not be called, such as when the Python interpreter is shutting down.
[^del_no_guarantee]: Python does not provide guarantees on when `__del__` will be called, so it is not recommended to rely on it for critical cleanup tasks. Instead, the `__enter__` and `__exit__` methods should be used to guarantee timely cleanup.

## Conclusion

Expand Down
185 changes: 185 additions & 0 deletions docs/blog/posts/2024/07-26 asynchronous message support.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,185 @@
---
authors:
- JP-Ellis
date:
created: 2024-07-26
---

# Asynchronous Message Support

We are excited to announce that support for verifying asynchronous message interactions has been added in the recent [release of Pact Python version 2.2.1](https://github.com/pact-foundation/pact-python/releases/tag/v2.2.1). To explore this feature, use the [`pact.v3`][pact.v3] module. A huge shoutout goes to [Val Kolovos](https://github.com/valkolovos) who contributed this feature across two very large PRs ([#714](https://github.com/pact-foundation/pact-python/pull/714) and [#725](https://github.com/pact-foundation/pact-python/pull/725)). This represents a significant step forward in the capabilities of Pact Python and on the road to full support for the Pact specification.

Asynchronous messages play a crucial role in building resilient and scalable systems. They allow services to communicate with each other without blocking, which can be particularly useful when the sender and receiver are not always available at the same time. However, verifying these interactions is challenging due to the wide variety of messaging systems and protocols.

Pact simplifies this process by focusing on the content of the messages rather than their transport mechanisms. This approach allows defining expected message exchanges and verifying their adherence independently of messaging systems and protocols. For a more comprehensive view of non-HTTP contract testing, have a look over at [docs.pact.io](https://docs.pact.io/getting_started/how_pact_works#non-http-testing-message-pact). The Pact specification provides a way to verify these interactions, but until now, Pact Python support for this feature was incomplete at best.

<!-- more -->

We are thrilled about this new feature and eager to see how our community will leverage it in their projects! Please try out asynchronous message support while it's still in preview mode, as your feedback is invaluable in shaping its final release.

Your feedback will help us refine and prefect this feature. You can provide feedback through any of these channels:

- Report issues on our GitHub page: [Pact Python Issues](https://github.com/pact-foundation/pact-python/issues).
- Join discussions on GitHub: [Pact Python Discussions](https://github.com/pact-foundation/pact-python/discussions).
- Connect with us on Slack: [Pact Foundation Slack](https://slack.pact.io/).

Thank you for your continued support!

## Consumer Example

Pact is a consumer-driven contract testing tool, and so the consumer defines the expectations of the message. Within the context of asynchronous messages, the consumer is the service that processes the message and might be referred to as the _subscriber_.

Consider an example where a consumer service is responsible for asynchronously processing requests to delete a user from the database and delete associated files. The Python client might listen for messages from AWS SQS and process them using a function like this:

```python
from typing import Any

import boto3

QUEUE_URL = "https://sqs.us-east-1.amazonaws.com/123456789012/MyQueue"


def delete_user(user_id: str) -> bool:
# Delete user from database
# Delete associated files
return True


def process_message(message: dict[str, Any]) -> bool:
if message.get("action") == "delete_user":
user_id = message["user_id"]
return delete_user(user_id)
return False


def main():
sqs = boto3.client("sqs")

response = sqs.receive_message(QueueUrl=queue_url)
for message in response.get("Messages", []):
if process_message(message):
sqs.delete_message(
QueueUrl=queue_url,
ReceiptHandle=message["ReceiptHandle"],
)
```

In this example, the `process_message` function processes messages from an SQS queue and calls the `delete_user` function to delete the user from the database and associated files. The `main` function listens for messages from the SQS queue and processes them using the `process_message` function.

Here’s an example of a Pact test for this consumer:

```python
import json

from pact.v3 import Pact

from my_consumer import process_message

def handler(body: str | bytes | None, metadata: dict[str, Any]) -> None:
message = json.loads(body)
process_message(message)

pact = Pact(
consumer="deleteUserService",
provider="someProvider",
).with_specification("V3") # (1)

(
pact
.upon_receiving("a request to delete a user", "Async")
.with_body(
json.dumps({
"action": "delete_user",
"user_id": "123",
})
) # (2)
)

pact.verify(handler, "Async")
```

1. Support for asynchronous messages starts in version 3 of the Pact specification.
2. No `will_respond_with` method exists for asynchronous messages since there’s no response expected.

This example highlights how the verification of asynchronous messages differs from HTTP interaction. As the transport layer is abstracted away, a `handler` function is required to parse the raw message string or bytes, and pass it to the underlying function that processes the message.

The `handler` would also typically be responsible for mocking the underlying systems that the consumer interacts with, such as the database or file system. This allows the consumer to be tested in isolation, without relying on external services. Furthermore, the mocked systems can then be inspected to verify that the consumer has performed the expected actions.

## Provider Example

For context of asynchronous messages, the provider is the service that sends the message and might be referred to as the _publisher_ or _producer_. Since the contract is defined by the consumer, the Pact provider test simply has to verify that the messages it sends meet the expectations of the consumer.

As the underlying protocol is abstracted away, Pact uses a local HTTP server to receive the messages that the provider sends. The provider test for the above consumer might look something like this:

```python
from pact.v3 import Verifier

class Provider:
"""
A simple HTTP provider that sends messages to the consumer.

This would typically use the same underlying functions that would generate messages, except that instead of being sent into the message queue, they are sent to the consumer's HTTP server.
"""

provider = Provider()

(
Verifier()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth mentioning that this is a departure from the existing interfaces MessageProvider

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I personally want the blog post to just focus on what's coming, as opposed to how things were.

I would like to create a more thorough upgrade guide once we are much closer to v3 though.

.set_info("someProvider", url=provider.url) # (1)
.set_source("/path/to/pacts")
.set_state(provider.state_url) # (2)
.add_transport( # (3)
protocol="message",
path="/_pact/message",
)
)
```

1. The provider URL is required, but is only used if the Pact being verified contains both HTTP and message interactions. It is not used for message interactions, and should the Pact not contain any HTTP interactions, the endpoint need not be active.
2. The provider state URL is required to ensure the provider is in the correct state. If the provider is entirely stateless, this can be omitted.
3. This path is used by Pact to ensure that the provider is in the correct state before sending the message.

Those familiar with HTTP interactions will notice that the process is very similar, with the key difference of the additional `add_transport` method. This configures a simple HTTP endpoint which Pact can use to prompt the provider to send a specific message. The following sequence diagram illustrates the flow of the provider test:

```mermaid
sequenceDiagram
participant Pact as Pact
participant T as Test
participant P as Provider

Pact->>Pact: Read source(s)
Pact->>T: Set provider state(s)
Pact->>T: Trigger message generation
T->>+P: Call provider
P->>T: Generate message
T->>Pact: Forward message<br/>over HTTP
Pact->>Pact: Verify message
```

At present, it is the responsibility of the end user to set up the provider endpoint middle-man to access the message triggers; however, future versions of Pact Python will abstract this away thereby reducing the test boilerplate required. The payloads are:

1. Trigger from Pact to the provider to generate a message:

```http
POST /_pact/message HTTP/1.1
Content-Type: application/json

{
"description": "a request to delete a user",
}
```

2. Response expected from the provider:

```http
HTTP/1.1 200 OK
Content-Type: application/json
Pact-Message-Metadata: <optional base64 encoded JSON object>

{
"action": "delete_user",
"user_id": "123",
}
```

Some queueuing systems allow for metadata to be attached to messages and may be required as part of the Pact. If that is the case, the metadata generated by the provider can be passed through the `Pact-Message-Metadata` header as a base-64 encoded string of the underlying JSON object.
Loading